LLMs performed best on questions related to legal systems and social complexity, but they struggled significantly with topics such as discrimination and social mobility.
“The main takeaway from this study is that LLMs, while impressive, still lack the depth of understanding required for advanced history,” said del Rio-Chanona. “They’re great for basic facts, but when it comes to more nuanced, PhD-level historical inquiry, they’re not yet up to the task.”
Among the tested models, GPT-4 Turbo ranked highest with 46% accuracy, while Llama-3.1-8B scored the lowest at 33.6%.
ai cant understand anything, that is why they will lie wildly if they dont know something. It can’t know that it doesn’t know. It would likely require somekind of consciousness for it to actually understand something. Expecting anything more than what it currently does well is just stupid until it proves it can do more. Yet people keep being surprised, but that isnt anything new.