• huginn@feddit.it
    link
    fedilink
    English
    arrow-up
    2
    arrow-down
    2
    ·
    10 months ago

    Here’s a white paper explicitly proving:

    No emergent properties (illusory due to bad measures)

    Predictable linear progress with model size

    https://arxiv.org/abs/2304.15004

    The field changes fast, I understand it is hard to keep up

      • huginn@feddit.it
        link
        fedilink
        English
        arrow-up
        2
        arrow-down
        1
        ·
        10 months ago
        1. Emergence is the whole being greater than the sum of its parts. That’s the original meaning of emergent properties, which is laid out in the first paragraph of the article. It’s the scholarly usage as well, and what the claims of observed emergence are using as the base of their claim.

        2. The article very explicitly demonstrated that only about 10% of any of the measures for LLMs displayed any emergence and that illusory emergence was the result of overly rigid metrics. Swapping to edit distance as an approximately close metric causes the sharp spikes to disappear for obvious reasons: no longer having a sharp yes/no allows for linear progression to reappear. It was always there, merely masked by flawed statistics.

        If you can’t be bothered to read here’s a very easy to understand video by one of the authors: https://www.youtube.com/watch?v=ypKwNrmuuPM