LLMs performed best on questions related to legal systems and social complexity, but they struggled significantly with topics such as discrimination and social mobility.
“The main takeaway from this study is that LLMs, while impressive, still lack the depth of understanding required for advanced history,” said del Rio-Chanona. “They’re great for basic facts, but when it comes to more nuanced, PhD-level historical inquiry, they’re not yet up to the task.”
Among the tested models, GPT-4 Turbo ranked highest with 46% accuracy, while Llama-3.1-8B scored the lowest at 33.6%.
Make sure to read a bit further, they usually get about 10000 attempts. And unfortunately most tests are just about recalling stuff, understanding is not something a text predictor does. It can’t actually think.
This. Everything about every ‘ai solves’ article I’ve read basically boils down to the equivalent of monkeys on typewriters, only with statistical guidance. Millions of iterations to solve what is essentially an intuitive solution for a reasonably intelligent being. I passed my driving test on the first try, after maybe a couple of weeks in school. What current ai ai doing is not intelligent in the least, and hardly efficient.
A friend’s mother was a doctor. Long ago, back in the 90s, she was talking about how there was some medical certification test that non-English speakers were passing simply by noticing key words in the question and correct answer.
About as useful as driver’s licenses in some places