While looking into artificial intelligence "behavior," researchers affirmed that yes, OpenAI's GPT-4 appeared to be getting dumber.

While looking into artificial intelligence “behavior,” researchers affirmed that yes, OpenAI’s GPT-4 appeared to be getting dumber.

  • Skua@kbin.social
    2·
    1 year ago

    Surely it can’t just be a case of the LM saying a hard yes or no to any question of “is this prime” with the data they have, though? The results are a significant majority one way or the other in each case, but never 100%. Of the 500 each time, GPT3.5 has 37 answers go against the majority in March and 66 in July. That doesn’t seem like a hard one answer to any primality query to me, though that does come with the caveat that I’m by no means actually well studied on the topic

    • mal099@kbin.social
      2·
      1 year ago

      True, GPT does not return a “yes” or “no” 100% of the time in either case, but that’s not the point. The point is that it’s impossible to say if GPT has actually gotten better or worse at predicting prime numbers with their test set. Since the test set is composed of only prime numbers, we do not know if GPT is more likely to call a number “prime” when it actually is a prime number than when it isn’t. All we know is that it was very likely to answer “yes” to the question “is this number prime?” in March, and very likely to answer “no” in July. We do not know if the number makes a difference.