How LLMs Work: Tokens & Prediction

1. The World Before Transformers

Before 2017, the best AI models we had were called RNNs (Recurrent Neural Networks). An LLM built on RNNs used to be like someone reading a book but being awful at remembering what it had read before.

These legacy models had two fatal flaws that prevented them from scaling:

1. The "One-by-One" Processing: RNNs read sequentially—one word at a time, from left to right. To understand the 100th word, they had to finish reading the previous 99 words first. They couldn't skip ahead or look at the whole page at once. This made training them incredibly slow because you couldn't use powerful computer chips (GPUs) to do multiple things at the same time.
2. The "Goldfish Memory" Problem: This was the bigger issue. As the model read, it tried to pass information from one word to the next. But by the time it got to the end of a long paragraph, the signal from the beginning had faded away. This is technically known as the vanishing gradient problem.

The Problem with Memory: If a sentence started with "The boy, who lived in a house across the street..." and ended 50 words later with "...was happy," the RNN would often forget who was happy. Was it the boy? The house? The street? The connection was lost in the distance.

2. The Transformer Breakthrough

In 2017, researchers at Google published "Attention Is All You Need", introducing the Transformer. This architecture changed the fundamental physics of how computers process language.

Instead of reading one word at a time like a human (or an RNN), the Transformer ingests the entire sentence at once.

The diagram on the right illustrates this machine's blueprint. It is divided into two main parts:

• The Encoder (left stack): Analyzes the input text to build a rich understanding of its meaning.
• The Decoder (right stack): Uses that understanding to generate a response, one token at a time.

Inside these stacks are layers of Multi-Head Attention that allow the model to focus on different parts of the sentence simultaneously. This is Parallelism, and it is the reason GPT-4 can exist. Because it sees everything at once, we can throw thousands of GPUs at the training process, scaling up to datasets that include nearly the entire internet.

Unlike the student reading word-by-word, the Transformer is like an alien with a thousand eyes. It looks at every single word in a book simultaneously. The first word and the last word are just as "close" to the Transformer as two words sitting right next to each other.

3. Self-Attention: The Intelligence Mechanism

If the model reads everything at once, how does it know which words relate to each other? The secret sauce is Self-Attention.

Think of Attention as a spotlight. When the model looks at a specific word, it can dim the lights on irrelevant words and shine a bright spotlight on the words that give it context.

Let's revisit the ambiguity problem:

"The cat drank the milk because it was hungry."

"The cat drank the milk because it was sweet."

To a simple computer, the word "it" is vague. But the Transformer calculates "attention scores" (technically, using Query, Key, and Value vectors) between every word pair:

• In the first sentence, when looking at "it", the model assigns a high relevance score to "cat" (because hunger relates to living things) and a low score to "milk."
• In the second sentence, when looking at "it", the model assigns a high relevance score to "milk" (because sweetness relates to food) and a low score to "cat."

Multi-Head Attention

You'll often hear about "Multi-Head Attention." This just means the model does this spotlight trick multiple times in parallel.

• Head 1 might focus on syntax (e.g., matching verbs to nouns).
• Head 2 might focus on semantics (e.g., matching "King" to "Queen").
• Head 3 might focus on sentiment (e.g., noticing that "amazing" makes the whole sentence positive).

By layering these "heads," the model builds a nuanced, multi-dimensional understanding of the text.

4. Tokens, Embeddings, and The "Factory Floor"

We've talked about "words", but LLMs don't actually see words. They see Tokens and Embeddings.

Tokens ≠ Words

The model breaks text down into chunks called distinct units called Tokens.

• Common words like "apple" might be a single token.
• Complex or rare words like "Transformerification" would be broken into multiple tokens: Trans, former, ifi, cation.
• Even spaces and punctuation are tokens.

Token 1: " The" (ID: 464)
Token 2: "qui"  (ID: 23145)
Token 3: "ck"   (ID: 402)
Token 4: " brown" (ID: 7586)

Why Your LLM Breaks: This tokenization is often why models fail at simple tasks like "spelling words backwards" or specific coding formatting. To a model, the token " The" (with a leading space) is completely different from the token "The" (no space). They have different IDs and different embeddings. If the model hasn't seen enough examples of one variation, it might behave unexpectedly.

Embeddings: The Meaning Vectors

Once tokenized, each token is converted into an Embedding. This is a long list of numbers (a vector) that represents the meaning of that token.

• In this mathematical space, the numbers for "King" minus "Man" plus "Woman" results in a vector very close to "Queen".
• This allows the model to perform "math" on concepts, not just match strings.

Positional Encoding

Since the Transformer sees all tokens at once (parallel), it has no inherent concept of order. "Man bites dog" looks the same as "Dog bites man". To fix this, we inject Positional Encodings—essentially adding a "timestamp" signature to each token embedding so the model knows its position in the sequence.

5. From Architecture to Behavior: The Autoregressive Loop

How does this architecture actually generate the text you see on screen? It uses an Autoregressive Loop.

1. Input: You give the model a prompt: "The sky is"
2. Process: The model processes these tokens, applies attention, and outputs a probability distribution for every possible next token in its vocabulary.
3. Predict: It selects the most likely next token (e.g., "blue").
4. Loop: It takes "The sky is blue" and feeds it back into itself as the new input.
5. Repeat: It predicts the next token based on this new sequence.

Decoding Strategies (Temperature)

If the model always picked the #1 most likely token (this is called "Greedy Decoding" or Temperature = 0), the text would often feel robotic and repetitive. To fix this, we introduce Temperature:

• Low Temperature (0.1): The model is conservative and factual. It sticks to the most likely path.
• High Temperature (0.8+): The model takes risks, occasionally picking the 2nd or 3rd most likely word. This makes the output feel more "creative" or diverse, but increases the risk of going off-track.

6. Why LLMs Hallucinate (It's Not a Bug)

Understanding the autoregressive mechanism explains Hallucinations.

An LLM is not a database of facts; it is a pattern completion engine. When you ask a question, it doesn't "look up" the answer; it predicts the most probable text that would follow your question based on the patterns it saw during training.

If you ask for a biography of a fake person, the model will often hallucinate a very convincing one. Why? Because in its training data, biographies follow a strict pattern: Name -> Date of Birth -> Early Life -> Career. The model fulfills the pattern with high statistical confidence, even if the facts are entirely invented.

Hallucinations are simply the model prioritizing probabilistic coherence over frequency correctness.

7. Production Economics: Tokens, Latency, & Cost

For developers and system architects, understanding the "physics" of this system is crucial for production.

1. Input Tokens vs. Output Tokens

• Input is processed in parallel (Fast & Cheap). The model ingests your whole prompt at once.
• Output is generated serially (Slow & Expensive). The model must run the entire massive neural network once for every single token it generates.

2. The KV Cache (Memory Bottleneck) You might wonder: "If the model has to re-read the whole conversation for every new word it generates, isn't that incredibly inefficient?" Yes! To solve this, engineers use the KV Cache. We save the "Key" and "Value" vectors for past tokens so we don't have to re-compute them. However, this cache grows linearly with context length. Long conversations eat up massive amounts of GPU memory (VRAM), which is often the hard limit on "Context Window" size.