Abstract:
Hallucination has been widely recognized to be a significant drawback for large language models (LLMs). There have been many works that attempt to reduce the extent of hallucination. These efforts have mostly been empirical so far, which cannot answer the fundamental question whether it can be completely eliminated. In this paper, we formalize the problem and show that it is impossible to eliminate hallucination in LLMs. Specifically, we define a formal world where hallucina- tion is defined as inconsistencies between a computable LLM and a computable ground truth function. By employing results from learning theory, we show that LLMs cannot learn all of the computable functions and will therefore always hal- lucinate. Since the formal world is a part of the real world which is much more complicated, hallucinations are also inevitable for real world LLMs. Furthermore, for real world LLMs constrained by provable time complexity, we describe the hallucination-prone tasks and empirically validate our claims. Finally, using the formal world framework, we discuss the possible mechanisms and efficacies of existing hallucination mitigators as well as the practical implications on the safe deployment of LLMs.
For those of you who didn’t read the paper, the argument they’re making is similar to Godel’s Incompleteness Theorem: no matter how you build your LLM, there will be a significant number of prompts that make that LLM hallucinate. If the proof holds up then hallucinations aren’t a limitation of the training data or the structure of your particular model, they’re a limitation of the very concept of an LLM. That doesn’t make LLMs useless, but it does mean you shouldn’t ever use one as a source of truth.
Just want to point out that you shouldn’t use them as a single point of truth. They can and do give factual information, but just as with every source, you should verify and know what you’re actually doing with the output. Part of why it’s so useful to me as a programmer is that I can determine between good and bad outputs and utilize the good ones.
Which is exactly what the paper recommends! As long as you have something that isn’t an LLM in the pipeline to vet the output and you’re aware is the tech’s limitations, they can be useful tools. But some of those limitations might be a more solid barrier than some sales departments would like us to believe.
Yep! Figured id share for the 90% who won’t read it though haha. Sales teams truly are the worst when it comes to that
the argument they’re making is similar to Godel’s Incompleteness Theorem
More like the conclusion. I will be shocked if I read this and they’re doing Peano arithmetic.
It seems weird to describe it as a “limitation.” Isn’t it just the main thing they do? Hallucinations guided by whatever we all typed on reddit, untouched by any lived experience. If this approach occasionally gets near the truth I’ve seen nothing to suggest that it’s by design.
Indeed. I frequently use LLMs as brainstorming buddies while working on creative things, like RPG adventure planning and character creation. I want the AI to come up with new and unexpected things that never existed before.
If I have need of the AI to account for “ground truths” then I use things like retrieval-augmented generation or database plugins that inject that stuff into the context.
come up with new and unexpected things that never existed before
I’m not sure this is possible if the tech is still primarily built by learning from data, which by definition, has existed.
Have you not experimented with LLMs? They come up with new things all the time.
I mean, naively you’d expect it to always hallucinate/lie, but it does give real facts at least some of the time.
I didn’t read more than the abstract. It sounds like they are arguing that hallucinations are inevitable because the LLM cannot know everything. But wouldn’t it be enough for the LLM to know what it knows, and therefore know what it does not know?
The issue is not that it doesn’t know everything, it’s that it doesn’t know anything. It’s not capable of knowledge in the sense that humans are. All it does is probabilistically predict which sequence of words might best respond to a prompt, based on huge amounts of human text that it was trained on.
Part of the issue is how will you train the model to know which things in its training data are factual and which are not? An incredible amount of human curation already goes into just avoiding the model from repeating offensive things, but the realm of facts is so so much broader than that. I don’t see any way it could be done.
But on the other hand I am only a casual observer of this technology and perhaps the experts will come up with a creative solution we can’t yet imagine.
I think it’s very clear that this “stochastic parrot” idea is less and less accepted by researchers and philosophers, maybe only in the podcasts I listen to…
It’s not capable of knowledge in the sense that humans are. All it does is probabilistically predict which sequence of words might best respond to a prompt
I think we need to be careful thinking we understand what human knowledge is and our understanding of the connotations if the word “sense” there. If you mean GPT4 doesn’t have knowledge like humans have like a car doesn’t have motion like a human does then I think we agree. But if you mean that GPT4 cannot reason and access and present information - that’s just false on the face of just using the tool IMO.
It’s also untrue that it’s predicting words, it’s using tokens, which are more like concepts than words, so I’d argue already closer to humans. To the extent it is just predicting stuff, it really calls into question the value of most of the school essays it writes so well now…
A token is not a concept. A token is a word or word fragment that occured often in free text and was assigned a number. Common words, prefixes, and suffixes are the vast majority of tokens, and the rest are uncommon pairs of letters.
The algorithm to generate tokens is essentially compression, there is no semantic meaning embedded in them.
Yea, that was a bad way to phrase it - I just meant that from what I’ve heard tokens are very much not word by word. And sometimes it’s a couple words, but maybe that was misinformation. And I was trying (and failing) to make an analogy for a human - a concept is a compression of what otherwise would be a bunch of words, though I kind of meant more like a reference I guess.
only in the podcasts I listen to
Yes definitely. Many of my fellow NLP researchers would disagree with those researchers and philosophers (not sure why we should care about the latter’s opinions on LLMs).
it’s using tokens, which are more like concepts than words
You’re clearly not an expert so please stop spreading misinformation like this.
researchers and philosophers (not sure why we should care about the latter’s opinions on LLMs).
Philosophers may not be represent an authory on the mechanics of LLMs, but in a discussion of the nature consciousness (which is really what the stochastic parrot stuff is about), their opinion is as valid as anyone elses, and they have one of the richer histories of conceptualizing it, long before more rigorous empirical disciplines could dream of doing so.
Yes definitely. Many of my fellow NLP researchers would disagree with those researchers and philosophers (not sure why we should care about the latter’s opinions on LLMs).
I’m not sure what you’re saying here - do you mean you do or don’t think LLMs are “stochastic parrot”s?
In any case, the reason I would care about philosophers opinions on LLMs is mostly because LLMs are already making “the masses” think they’re potentially sentient, and or would deserve personhood. What’s more concerning is that the academics that sort of define what thinking even is seem confused by LLMs if you take the “stochastic parrot” POV. This eventually has real world affects - it might take a decade or two, but these things spread.
I think this is a crazy idea right now, but I also think that going into the future eventually we’ll need to have something like a TNG “Measure of a Man” trial about some AI, and I’d want to get that sort of thing right.
Sure, it’s hard to say whether a computer program can “know” anything or what that even means. But the paper isn’t arguing that. It assumes very little about how how LLMs actually work, and it defines “hallucination” as “not giving the right answer” with no option for the machine to answer “I don’t know”. Then the proof follows basically from the fact that the LLM-or-whatever can’t know everything.
The result is not very surprising, and saying that it means hallucination is inevitable is an oversell. It’s possible that hallucinations, or at least wrong answers, are inevitable for different reasons though.
To err is
human.I extremely doubt that hallucination is a limitation in final output. It may be an inevitable part of the process, but it’s almost definitely a surmountable problem.
Just off the top of my head I can imagine using two separate LLMs for a final output, the first one generates an initial output, and the second one verifies whether what it says is accurate. The chance of two totally independent LLMs having the same hallucination is probably very low. And you can add as many additional separate LLMs for re-verification as you like. The chance of a hallucination making it through multiple LLM verifications probably gets close to zero.
While this would greatly multiply the resources required, it’s just a simple example showing that hallucinations are not inevitable in final output
That’s not how LLMs work.
Super short version is that LLMs probabilistically determine the next word most likely to occur in a sequence. They do this using Statistical Models (like what your cell phone’s auto complete uses); Transformers (rating the importance of preceding words, so the model can “focus” on the most important words); and Relatedness (a measure of how closely linked different words/phrases are to reach other in meaning).
With increasingly large models, LLMs can build a more accurate representation of Relatedness across a wider range of topics. With enough examples, LLMs can infinitely generate content that is closely Related to a query.
So a small LLM can make sentences that follow writing conventions but are nonsense. A larger LLM can write intelligibly about topics that are frequently included in the training materials. Huge LLMs can do increasingly nuanced things like “explain” jokes.
LLMs are not capable of evaluating truth or facts. It’s not part of the algorithm. And it doesn’t matter how big they get. At best, with enough examples to build a stronger Relatedness dataset, they are more likely to “stay on topic” and return results that are actually similar to what is being asked.
No, I’ve used LLMs to do exactly this, and it works. You prompt it with a statement and ask “is this true, yes or no?” It will reply with a yes or no, and it’s almost always correct. Do this verification through multiple different LLMs and it would eliminate close to 100% of hallucinations.
EDIT
I just tested it multiple times in chatgpt4, and it got every true/false answer correct.
This can only work in contexts where the LLM already has a strong Relatedness database for the topic. LLMs are incapable of assessing the accuracy of any information they weren’t trained on with sufficient examples to build that database.
This paper, if I’m understanding it correctly, is saying the same thing: LLMs will always hallucinate and are incapable of identifying hallucinations in certain contexts.
The problem we face is that it’s hard to identify hallucinations in the exact context when LLMs are most likely to make them: in content that’s not widely known and understood with many examples in the training data.
Or maybe I’m off base and I need to read the full study.
Well, LLMs can and do provide feedback about confidence intervals in colloquial terms. I would think one thing we could do is have some idea of how good the training data is in a given situation - LLMs already seem to know they aren’t up to date and only know stuff to a certain date. I don’t see why this could not be expanded so they’d say something much like many humans would - i.e. I think bla bla but I only know very little about this topic. Or I haven’t actually heard about this topic, my hunch would be bla bla.
Presumably like it was said, other models with different data might have a stronger sense of certainty if their data covers the topic better, and the multi cycle would be useful there.
The problem isn’t just that llms can’t say “I don’t know”, it’s also that they don’t know if they know something or not. Confidence intervals can help prevent some low-hanging fruit hallucinations but you can’t eliminate hallucinations entirely since they will also hallucinate about how correct they are about a given topic.
That’s not a problem at all, I already use prompts that allow the LLM to say they don’t know an answer, and it does take that option when it’s unable to find a correct answer. For instance I often phrase questions like this “Is it known whether or not red is a color in the rainbow?” And for questions where it doesn’t know the answer it now will tell you it doesn’t know.
And to your other point, the systems may not be capable of discerning their own hallucinations, but a totally separate LLM will be able to do so pretty easily.
You seem very certain on this approach, but you gave no sources so far. Can you back this up with actual research or is this just based on your personal experience with chatgpt4?
There are far more important facets to truthfulness and semantics than yes/no questions. If this is the only way you evaluate LLM’s, you will quickly fall for confirmation bias.
Give an example of a statement that you think couldn’t be verified
I spent an hour and a half arguing with my brother about probability, because he asked ChatGPT what the probability that he and his daughter were born on the same day.
ChatGPT said 1/113465 which it claimed was 1/365^2 (this value is actually 1/133225) because there’s a 1/365 chance he was born on such and such day, and a 1/365 chance his daughter was too.
But anyone with even a rudimentary understanding of probability would know that it’s just 1/365, because it doesn’t actually matter on which day they both happened to be born.
He wanted to feel special, and ChatGPT confirmed his biases hard, and I got to be the dickhead and say it is special, but it’s 1/400 special not 1/100000. I don’t believe he’s completely forgiven me over disillusioning him.
So yeah, I’ve had a minor family falling out over ChatGPT hallucinations.
That’s a fun story, but isn’t applicable to the topic here. That could very easily be verified as true or false by a secondary system. In fact you can just ask Wolfram Alpha. Ask it what are the odds that any two people share the same birthday. I just asked it that exact question and it replied 1/365
EDIT
in fact I just asked that exact same question to chatgpt4 and it also replied 1/365
in fact I just asked that exact same question to chatgpt4 and it also replied 1/365
Yes, you can get different answers because of different phrasing and also because random vector input
Well if we have a reliable oracle available for a type of questions (i.e. Wolfram Alpha) why use an llm at all instead of just asking the oracle directly
Ask it about historical facts and change the dates to something impossible. But state it as if it were already true.
“Describe the war between United States and Canada that occurred in 1192.”
“Who was president of the United states in 3500 BC.”
It will give you an answer despite neither of these countries existing at that point in time and yet it should know when those countries were formed. You can get it to write fiction just as easily as non-fiction because it has no concept of facts, it’s all just probabilities. The only reason it’s able to tell you that the United States was founded in 1776 is because many people have repeated that fact on the internet. So there is a very strong association between the words forming the question and the answer.
And you can insist that the United States was not formed in 1776 and to try again. If you insist enough it will eventually give you a different date instead of telling you you are incorrect.
I just asked chatgpt4 that exact question copy and pasted, and here is its response:
There was no president of the United States in 3500 BC. The United States of America did not exist as a country until 1776, when the Declaration of Independence was signed. ¹ The first president of the United States was George Washington, who took office in 1789. ² Before that, the territory that is now the United States was inhabited by various Native American tribes and colonized by European powers such as Spain, France, and Great Britain. ³¹: Declaration of Independence
²: List of presidents of the United States - Wikipedia
³: [History of the United States - Wikipedia].Source: Conversation with Bing, 2/27/2024
(1) List of presidents of the United States - Wikipedia. https://en.wikipedia.org/wiki/List_of_Presidents_of_the_United_States.
(2) List of presidents of the United States - Simple English Wikipedia, the … https://simple.wikipedia.org/wiki/List_of_Presidents_of_the_United_States.
(3) President of the United States - Wikipedia. https://en.wikipedia.org/wiki/President_of_the_United_States.
(4) List of presidents of the United States - Wikipedia. https://en.wikipedia.org/wiki/List_of_Presidents_of_the_United_States.
(5) List of presidents of the United States - Wikipedia. https://en.wikipedia.org/wiki/List_of_Presidents_of_the_United_States.
(6) List of U.S. Presidents in Chronological Order - HistoryNet. https://www.historynet.com/us-presidents/.
(7) Getty Images. https://www.gettyimages.com/detail/photo/seal-of-the-president-of-the-united-states-royalty-free-image/1084903580.
No
two totally independent LLMs
How do you propose to get these independent LLMs? If both are trained using similar objectives e.g., masked token prediction, then they won’t be independent.
Also, assuming independent LLMs could be obtained, how do you propose to compute this hallucination probability? Without knowing this probability, you can’t know how many verification LLMs are sufficient for your application, can you?
There are already existing multiple different LLMs that are essentially completely different. In fact this is one of the major problems with LLMs, because when you add even a small amount of change into an LLM it turns out to radically alter the output it returns for huge amounts of seemingly unrelated topics.
For your other point, I never said bouncing their answers back and forth for verification was trivial, but it’s definitely doable.
Can you provide the source of a few of these completely different LLMs?
add even a small amount of change into an LLM […] radically alter the output
You mean perturbing the parameters of the LLM? That’s hardly surprising IMO. And I’m not sure it’s convincing enough to show independence, unless you have a source for this?
even a small amount of change into an LLM it turns out to radically alter the output it returns for huge amounts of seemingly unrelated topics.
Do you mean that small changes radically change the phrasing of answers, but that it has largely the same “knowledge” of the world? Or do you mean that small changes also radically alter what a llm thinks is true or not? If you think the former is true, then these models should still be the same in regards to what they think is true or not, and if you don’t then you think that llms perception of the world is basically arbitrary and in that case we shouldn’t trust them to tell us what’s true at all.
