Chinchilla: Everyone Was Training the Wrong-Shaped Model

Say you have exactly one hour of oven time. You can bake one big cake, or two small ones. You cannot have both, because the oven is the oven.

Training an AI model works the same way. You have a fixed budget — money, electricity, weeks of machine time — and you spend it on two things: how big the model is, and how much text it reads. Spend more on one, and you have less for the other.

Between 2020 and 2022, everyone chose the big cake. The largest models grew from 175 billion parameters to 530 billion, while the amount of text they read barely moved — all of them read roughly the same 300 billion words' worth. Size was the headline. Reading was a constant nobody thought to vary.

Then DeepMind ran the experiment properly and found that everybody had it backwards.

See the research paper

Training Compute-Optimal Large Language Models, by Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Laurent Sifre and colleagues, appeared in March 2022. The model they trained to prove the point is called Chinchilla, and its argument is now the default assumption behind essentially every training run.

1. Two words you need

Parameters are the adjustable numbers inside the model — its capacity to store patterns. More parameters means more room to learn. This is the size of the cake.

Tokens are chunks of text, roughly three-quarters of a word each. The number of tokens is how much the model reads during training. This is the baking time.

Now the budget. For a model like this, the total work required is close to:

$$C \approx 6ND$$

where $C$ is the total computing effort, $N$ is the parameters and $D$ is the tokens. The six comes from counting the work done for each parameter on each token, going forwards and then backwards through the model.

That equation is the whole setup. If $C$ is fixed — and it always is, because it is money and calendar time — then $N$ and $D$ trade directly against each other. Double the size, halve the reading.

So the real question is not "how big should the model be." It is: for a fixed budget, which combination of size and reading gives the best model? Everything else in the paper is machinery for answering that.

2. What everyone believed before

The prevailing advice came from Kaplan and colleagues in 2020 — a landmark paper that established this whole way of thinking and remains hugely influential. Its analysis suggested that when your budget grows, most of the extra should go into making the model bigger.

People followed it, reasonably. That is exactly why the leading models of the era all read roughly the same amount while varying wildly in size. Everyone was turning the dial the advice told them to turn.

The Chinchilla team re-ran the experiment much more broadly, and found one methodological detail that turned out to matter enormously.

When you train a model, you gradually reduce how big its learning steps are, on a schedule set in advance for the planned length of the run. In the earlier work, models trained for a shorter time were still using a schedule built for a longer one — so they were stopped while still taking large, clumsy steps, and ended up worse than they should have been.

Which means the comparison was rigged without anyone intending it. "Small model that reads more" was handicapped against "big model," and the apparent advantage of size was, in part, an artefact of the schedule.

3. Three separate routes to the same answer

The paper's credibility comes from not answering the question once. The team trained over 400 models, from 70 million to more than 16 billion parameters, reading anywhere from 5 to 500 billion tokens — and analysed the results three independent ways.

Route one — fix the sizes, vary the reading. Take a family of models and train each for many different durations. Plot how good each got against how much it cost. The line hugging the bottom of all those curves is the best you can do at each budget.

Route two — fix the budget, vary the size. Pick a budget and try many model sizes within it, so bigger models automatically get less reading. Plot the results and you get a clear valley — too small is bad, too big is bad, and the bottom is the best choice for that budget. Repeat for nine different budgets and join up the low points.

Route three — fit a formula. Describe every run's result as a mathematical function of size and reading, then solve for the best combination directly.

These three differ in what they assume and where they could go wrong. They agree with each other. That agreement is what makes the conclusion hard to argue with, and it is a template worth copying for any claim that matters.

4. The answer: grow both, equally

Stated in one line in the paper: for the best use of a budget, model size and reading should be scaled equally. Double your budget, and you should get about 1.4 times the parameters and 1.4 times the reading — not twice the parameters and the same reading.

Applied to the models of the day, that was blunt. Gopher — 280 billion parameters, 300 billion tokens — was the wrong shape for what it cost. The right shape was about four times smaller, reading about four times more.

So they built exactly that. Chinchilla: 70 billion parameters, 1.4 trillion tokens, on the same budget as Gopher.

That ratio became the field's rule of thumb, and it is just division: 1.4 trillion divided by 70 billion is 20 tokens per parameter.

5. What Chinchilla did

It beat Gopher, a model four times its size — and beat it broadly, not on a cherry-picked test. It also beat GPT-3 (175B), Jurassic-1 (178B) and Megatron-Turing NLG (530B), which is more than seven times larger.

On MMLU, a broad general-knowledge exam, it scored 67.5% — a 7% improvement over Gopher and, at the time, the best result anyone had.

The second consequence is the one that reshaped the industry, and the paper says it plainly: a smaller model is cheaper to run. Gopher and Chinchilla cost the same to train, once. Chinchilla costs a quarter as much every single time anyone uses it, forever. When a model is trained once and used billions of times, the running bill dwarfs the training bill — so the better-shaped model is not merely as good, it is dramatically cheaper to own.

6. Doing the arithmetic yourself

The value of $C \approx 6ND$ is that you can sanity-check a plan on paper before spending anything.

Suppose your budget is $10^{22}$ units of work. The rule says aim for $D \approx 20N$. Substitute it in:

$$10^{22} = 6N(20N) = 120N^2 \quad \Rightarrow \quad N \approx 9.1 \times 10^{9}$$

So: roughly a 9-billion-parameter model reading about 180 billion tokens.

Now look at the two ways to get it wrong with the same budget.

Build a 30-billion model instead, and it only gets about 56 billion tokens — under 2 tokens per parameter. Badly under-read. It will lose to the 9-billion model despite being three times the size.

Build a 1-billion model, and it gets 1.7 trillion tokens — 1,700 tokens per parameter. Extremely well-read and simply too small to use what it read.

Both sit on the same budget line, and both are up the sides of the valley. That it is a valley rather than a slope is the whole finding: there is a wrong answer in both directions, and the field had spent three years walking steadily up one side.

7. What the ratio does and does not mean

The 20-to-1 number has been repeated so often that it has hardened into a law. It is not one. It is the arithmetic of a single model, from a study of one model family, one dataset and one recipe, at scales that now look modest.

Several things move it. Better text raises the value of each token. Re-reading the same text lowers it, because a second pass is worth less than the first. Newer model designs change the relationship the estimate assumes.

What survives all of that is the structural finding, which is worth far more than the constant: there is a best answer, it is not at either extreme, and the field before 2022 was a long way off it in a consistent direction.

Anyone doing a serious training run should work out their own valley, using route two — the most practical of the three.

8. Why it mattered

Chinchilla ended the parameter-count era. Before it, size was the number in the press release. After it, the interesting questions became the quality of the text, the quantity of it, and how far past the best-to-train point you should go for cheaper running. Every open model since — trained small, trained long, trained on trillions of tokens — comes downstream of this paper.

It is also a rare case of a field correcting itself cleanly. A widely-adopted result was wrong in an identifiable way. Somebody ran the experiment carefully enough to find the flaw, published the correction with three independent analyses, and trained a model to prove it.

That is how the process is meant to work, and it does not always.

The transferable lesson: when you have a dial and a fixed budget, the interesting question is never how far you can turn it. It is which dial, and in what proportion — and that is a question you have to actually go and measure.

Sources

• Hoffmann et al. — Training Compute-Optimal Large Language Models, 2022
• Kaplan et al. — Scaling Laws for Neural Language Models, 2020
• Rae et al. — Scaling Language Models: Methods, Analysis & Insights from Training Gopher, 2021