No Stupid Questions

It's a really well-trained parrot. It responds to what you say, and then it responds to what it hears itself say.

But despite knowing which sounds go together based on which sounds it heard, it doesn't actually speak English.

I am an LLM researcher at MIT, and hopefully this will help.

As others have answered, LLMs have only learned the ability to autocomplete given some input, known as the prompt. Functionally, the model is strictly predicting the probability of the next word^+^, called tokens, with some randomness injected so the output isn’t exactly the same for any given prompt.

The probability of the next word comes from what was in the model’s training data, in combination with a very complex mathematical method to compute the impact of all previous words with every other previous word and with the new predicted word, called self-attention, but you can think of this like a computed relatedness factor.

This relatedness factor is very computationally expensive and grows exponentially, so models are limited by how many previous words can be used to compute relatedness. This limitation is called the Context Window. The recent breakthroughs in LLMs come from the use of very large context windows to learn the relationships of as many words as possible.

This process of predicting the next word is repeated iteratively until a special stop token is generated, which tells the model go stop generating more words. So literally, the models builds entire responses one word at a time from left to right.

Because all future words are predicated on the previously stated words in either the prompt or subsequent generated words, it becomes impossible to apply even the most basic logical concepts, unless all the components required are present in the prompt or have somehow serendipitously been stated by the model in its generated response.

This is also why LLMs tend to work better when you ask them to work out all the steps of a problem instead of jumping to a conclusion, and why the best models tend to rely on extremely verbose answers to give you the simple piece of information you were looking for.

From this fundamental understanding, hopefully you can now reason the LLM limitations in factual understanding as well. For instance, if a given fact was never mentioned in the training data, or an answer simply doesn’t exist, the model will make it up, inferring the next most likely word to create a plausible sounding statement. Essentially, the model has been faking language understanding so much, that even when the model has no factual basis for an answer, it can easily trick a unwitting human into believing the answer to be correct.

—-

^+^more specifically these words are tokens which usually contain some smaller part of a word. For instance, understand and able would be represented as two tokens that when put together would become the word understandable.

It's your phone's 'predictive text', but if it were trained on the internet.

It can guess what the next word should be a lot of the time, but it's also easy for it to go off the rails.

Harry Frankfurt's influential 2005 book (based on his influential 1986 essay), On Bullshit, offered a description of what bullshit is.

When we say a speaker tells the truth, that speaker says something true that they know is true.

When we say a speaker tells a lie, that speaker says something false that they know is false.

But bullshit is when the speaker says something to persuade, not caring whether the underlying statement is true or false. The goal is to persuade the listener of that underlying fact.

The current generation of AI chat bots are basically optimized for bullshit. The underlying algorithms reward the models for sounding convincing, not necessarily for being right.

idea that "it makes convincing sentences, but it doesn't know what it's talking about"

Like a teenager who has come into a new group and is now trying so hard to fit in :-)

You could maybe just share a meme like this one.

Some folks in the comments there share actual LLM results, a few of which are sensible but plenty that aren't far off from the joke.

Not an ELI5, sorry. I'm an AI PhD, and I want to push back against the premises a lil bit.

Why do you assume they don't know? Like what do you mean by "know"? Are you taking about conscious subjective experience? or consistency of output? or an internal world model?

There's lots of evidence to indicate they are not conscious, although they can exhibit theory of mind. Eg: https://arxiv.org/pdf/2308.08708.pdf

For consistency of output and internal world models, however, their is mounting evidence to suggest convergence on a shared representation of reality. Eg this paper published 2 days ago: https://arxiv.org/abs/2405.07987

The idea that these models are just stochastic parrots that only probabilisticly repeat their training data isn't correct, although it is often repeated online for some reason.

A little evidence that comes to my mind is this paper showing models can understand rare English grammatical structures even if those structures are deliberately withheld during training: https://arxiv.org/abs/2403.19827

It's just fancy predictive text like while texting on your phone. It guesses what the next word should be for a lot more complex topics.

Imagine you were asked to start speaking a new language, eg Chinese. Your brain happens to work quite differently to the rest of us. You have immense capabilities for memorization and computation but not much else. You can't really learn Chinese with this kind of mind, but you have an idea that plays right into your strengths. You will listen to millions of conversations by real Chinese speakers and mimic their patterns. You make notes like "when one person says A, the most common response by the other person is B", or "most often after someone says X, they follow it up with Y". So you go into conversations with Chinese speakers and just perform these patterns. It's all just sounds to you. You don't recognize words and you can't even tell from context what's happening. If you do that well enough you are technically speaking Chinese but you will never have any intent or understanding behind what you say. That's basically LLMs.

There’s the Chinese Room argument, which is a bit related:

https://en.wikipedia.org/wiki/Chinese_room

A 5 year old repeating daddy's swear words without knowing what it is.

That analogy is hard to come up with because the question of whether it even comprehends meaning requires first answering the unanswerable question of what meaning actually is and whether or not humans are also just spicy pattern predictors / autocompletes, since predicting patterns is like the whole point of evolving intelligence, being able to connect cause and effect in patterns and anticipate the future just helps with not starving. The line is far blurrier than most are willing to admit and ultimately hinges on our experience of sapience rather than being able to strictly define knowledge and meaning.

Instead it's far better to say that ML models are not sentient, they are like a very big brain that's switched off, but we can access it by stimulating it with a prompt.

Imagine that you have a random group of people waiting in line at your desk. You have each one read the prompt, and the response so far, and then add a word themself. Then they leave and the next person in line comes and does it.

This is why "why did you say ?" questions are nonsensical to AI. The code answering it is not the code that wrote it and there is no communication coordination or anything between the different word answerers.

The idea that “it makes convincing sentences, but it doesn’t know what it’s talking about” is a difficult concept to convey or wrap your head around.

I see the people you talk to aren't familiar with politicians?

Imagine making a whole chicken out of chicken-nugget goo.

It will look like a roast chicken. It will taste alarmingly like chicken. It absolutely will not be a roast chicken.

The sad thing is that humans do a hell of a lot of this, a hell of a lot of the time. Look how well a highschooler who hasn't actually read the book can churn out a book report. Flick through, soak up the flavour and texture of the thing, read the blurb on the back to see what it's about, keep in mind the bloated over-flowery language that teachers expect, and you can bullshit your way to an A.

Only problem is, you can't use the results for anything productive, which is what people try to use GenAI for.

The Chinese Room by Searle

It's all just fancy statistics. It turns words into numbers. Then it finds patterns in those numbers. When you enter a prompt, it finds numbers that are similar and spits out an answer.

You can get into vectors and back propagation and blah blah blah but essentially it's a math formula. We call it AI but it's not fundamentally different than solving 2x + 4 = 10 for x.

After reading some of the comments and pondering this question myself, I think I may have thought of a good analogy that atleast helps me (even though I know fairly well how LLM's work)

An LLM is like a car on the road. It can follow all the rules, like breaking in front of a red light, turning, signaling etc. However, a car has NO understanding of any of the traffic rules it follows.

A car can even break those rules, even if its behaviour is intended (if you push the gas pedal at a red light, the car is not in the wrong because it doesn't KNOW the rules, it just acts on it).

Why this works for me is that when I give examples of human behaviour or animal behaviour, I automatically ascribe some sort of consciousness. An LLM has no conscious (as far as I know for now). This idea is exactly what I want to convey. If I think of a car and rules, it is obvious to me that a car has no concept of rules, but still is part of those rules somehow.

• Compression algorithms can reduce most written text to about 20–25% of its original size—implying that that’s the amount of actual unique information it contains, while the rest is predictable filler.

• Empirical studies have found that chimps and human infants, when looking at test patterns, will ignore patterns that are too predictable or too unpredictable—with the sweet spot for maximizing attention being patterns that are about 80% predictable.

• AI researchers have found that generating new text by predicting the most likely continuation of the given input results in text that sounds monotonous and obviously robotic. Through trial and error, they found that, instead of choosing the most likely result, choosing one with around an 80% likelihood threshold produces results judged most interesting and human-like.

The point being: AI has stumbled on a method of mimicking the presence of meaning by imitating the ratio of novelty to predictability that characterizes real human thought. But we know that the actual content of that novelty is randomly chosen, rather than being a deliberate message.

So there's two different things to what you are asking.

(1) They don't know what (i.e. semantically) they are talking about.

This is probably not the case, and there's very good evidence over the past year in research papers and replicated projects that transformer models do pick up world models from the training data such that they are aware and integrating things at a more conceptual level.

For example, even a small toy GPT model trained only on chess moves builds an internal structure of the whole board and tracks "my pieces" and "opponent pieces."

(2) Why do they say dumb shit that's clearly wrong and don't know.

They aren't knowledge memorizers. They are very advanced pattern extenders.

Where the answer to a question is part of the pattern they can successfully extend, they get the answer correct. But if it isn't, they confabulate an answer in a similar way to stroke patients who don't know that they don't know the answer to something and make it up as they go along. Similar to stroke patients, you can even detect when this is happening with a similar approach (ask 10x and see how consistent the answer is or if it changes each time).

They aren't memorizing the information like a database. They are building ways to extend input into output in ways that match as much information as they can be fed. In this, they are beyond exceptional. But they've been kind of shoehorned into the initial tech demo usecase of "knowledgeable chatbot" which is a less than ideal use. The fact they were even good at information recall was a surprise to most researchers.

All of this also touches upon an interesting topic. What it really means to understand something? Just because you know stuff and may even be able to apply it in flexible ways, does that count as understanding? I’m not a philosopher, so I don’t even know how to approach something like this.

Anyway, I think the main difference is the lack of personal experience about the real world. With LLMs, it’s all second hand knowledge. A human could memorize facts like how water circulates between rivers, lakes and clouds, and all of that information would be linked to personal experiences, which would shape the answer in many ways. An LLM doesn’t have such experiences.

Another thing would be reflecting on your experiences and knowledge. LLMs do none of that. They just speak whatever “pops in their mind”, whereas humans usually think before speaking… Well at least we are capable of doing that even though we may not always take advantage of this super power. Although, the output of an LLM can be monitored and abruptly deleted as soon as it crosses some line. It’s sort of like mimicking the thought processes you have inside your head before opening your mouth.

Example: Explain what it feels like to have an MRI taken of your head. If you haven’t actually experienced that yourself, you’ll have to rely on second hand information. In that case, the explanation will probably be a bit flimsy. Imagine you also read all the books, blog posts and and reddit comments about it, and you’re able to reconstruct a fancy explanation regardless.

This lack of experience may hurt the explanation a bit, but an LLM doesn’t have any experiences of anything in the real world. It has only second hand descriptions of all those experiences, and that will severely hurt all explanations and reasoning.

It's like talking to a republican!

The short hand answer I’d try to give people is ‘it’s statistics’. Based on training data, there’s a certain chance of certain words being in proximity of each other. There’s no reasoning behind placement, other than whatever pattern is discernible from known situation.

Someone once described is as T9 on steroids. It's like your mobile keyboard suggesting follow up words, just a lot more complex in size.

If an AI says something malicious, our brain immediatly jumps to "it has intent". How can we explain this away?

The more you understand the underlying concept of LLMs, the more the magic fades away. LLMs are certainly cool and can be fun but the hype around them seems very artificial and they're certainly not what I'd describe as "AI". To me, an AI would be something that actually has some form of consciousness, something that actually can form its own thoughts and learn things on its own through observation or experimentation. LLMs can't do any of those things. They're static and always wait for your input to even do anything. For text generation you can even just regenerate an answer to the same previous text and the replies can and will vary greatly. If they say something mean or malicious, it's simply because it is based on whatever they were trained on and whatever parameters they are following (like if you told them to roleplay a mean person).

Part of the problem is hyperactive agency detection - the same biological bug/feature that fuels belief in the divine.

If a twig snaps, it could be nothing or someone. If it's nothing and we react as if it was someone, no biggie. If it was someone and we react as if it was nothing, potential biggie. So our brains are bias towards assuming agency where there is none, to keep us alive.

Like a kid trying very hard to sound like everyone else. "Eloquent bullshit generator"

have you played that game where everyone write a subjet and put it on a stack of paper, then everyone puts a verb on a different stack of paper, then everyone put an object on a third stack of paper, and you can even add a place or whatever on the next stack of paper. You end-up with fun sentences like A cat eat Kevin's brain on the beach. It's the kind of stuff (pre-)teen do to have a good laugh.

Chat GPT somehow works the same way, except that instead of having 10 paper in 5 stack, it has millions of paper in thousands of stack and depending on the "context" will choose which stack it draws paper from (To take an ELI5 analogy)

The way I've explained it before is that it's like the autocomplete on your phone. Your phone doesn't know what you're going to write, but it can predict that after word A, it is likelly word B will appear, so it suggests it. LLMs are just the same as that, but much more powerful and trained on the writing of thousands of people. The LLM predicts that after prompt X the most likelly set of characters to follow it is set Y. No comprehension required, just prediction based on previous data.