LoRA: Why a Rank-8 Matrix Is Enough

Suppose you own an enormous, beautifully detailed map of a country, and you want a version marked up for cycling. You could photocopy the entire map and draw on the copy. Now you own two enormous maps. Want a hiking version too? Three.

Or you could keep one map and buy a stack of clear plastic sheets. Draw the cycling routes on one sheet, the hiking routes on another. Lay a sheet over the map when you need it. The map never changes, and each sheet weighs almost nothing.

That is LoRA. The map is a pre-trained AI model, and the sheets are a few megabytes each.

See the research paper

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang and Weizhu Chen published LoRA: Low-Rank Adaptation of Large Language Models in June 2021.

1. The problem with photocopying the map

Fine-tuning means taking a finished model and adjusting it for a specific job. Done the obvious way, you take the original numbers $W_0$ and adjust every one of them, producing $W_0 + \Delta W$. That $\Delta W$ — the difference — is the same size as the entire model.

For a model with 175 billion numbers, adapting it for one job means storing another 175 billion. Nine jobs, nine copies. By 2021 that was the main practical obstacle to using big models for anything specific.

Three costs follow, and only the first gets discussed.

Storage: one full model per job.

Training memory: worse than the model itself. The training process keeps two extra running notes per number, so it needs several times the model's size in memory. This is usually what actually stops you.

Serving: many jobs means either keeping many models loaded, or swapping models on every request.

So the question LoRA asks is whether $\Delta W$ really needs to be that big — whether the change carries as much information as its size suggests.

2. The bet: the change is simpler than it looks

Here is the intuition. Pre-training already taught the model language, facts and reasoning. Your specific job does not add much of that. It mostly reweights what is already there — pay more attention to this, phrase things like that.

That is a structurally simple adjustment, even though it is written across a very large grid of numbers.

If it really is simple, you should be able to describe it compactly. A big grid whose contents follow a simple pattern can be rebuilt from two much thinner grids multiplied together. So instead of learning the whole thing, learn the two thin ones:

$$\Delta W = BA, \qquad B \in \mathbb{R}^{d \times r},\ A \in \mathbb{R}^{r \times k},\ r \ll \min(d,k)$$

That $r$ is the rank — how thin the thin grids are, and the one number you choose. Think of it as how many independent adjustments you are allowing.

The model then computes:

$$h = W_0 x + \Delta W x = W_0 x + BAx$$

$W_0$ is frozen. Never updated, never given running notes. Only $A$ and $B$ train.

The count drops from $d \times k$ numbers to $r(d + k)$. For a typical $4096 \times 4096$ grid at $r = 8$: 16.8 million becomes 65,536. About 0.4%, from one line of algebra.

3. Starting from zero, on purpose

$A$ starts with small random values. $B$ starts at exactly zero — so at the very first step, $BA$ is zero and the adapted model is identical to the original.

This matters more than it looks. Training begins from a model that already works, rather than from a slightly damaged version of one. Every early step is spent learning your job, not repairing the damage that random initialisation would have caused.

The adjustment is also scaled by $\alpha / r$, where $\alpha$ is a constant you pick. This keeps the size of each step roughly the same when you change the rank, so you can try different ranks without re-tuning everything else. Many people set $\alpha = r$ and stop thinking about it, which is why you so often see them equal.

4. The property that made it win

Other methods existed before LoRA. Most worked by inserting small new pieces between the model's existing layers. They worked — but they made the model deeper, so every use of it took longer. In a live product that cost never goes away.

LoRA has a structural advantage, and it is why it displaced the alternatives.

Because the adjustment is added to an existing grid rather than inserted as a new step, you can simply work out

$$W = W_0 + BA$$

once, before deploying, and ship $W$. The result is an ordinary grid of the original shape. The model's structure is unchanged. Its speed is unchanged. There is no trace of LoRA left in what you deployed.

The paper says it plainly: unlike adapters, no additional inference latency.

Better still, you can undo it. Keep $W_0$ and swap sheets at runtime — subtract one, add another. Serving fifty different fine-tunes becomes one model in memory plus fifty small files, instead of fifty full models.

That is the property that multi-tenant AI serving runs on today.

5. Where to put the sheets

Not every grid in the model needs one. The paper tests which parts of the attention mechanism benefit, and finds something slightly counter-intuitive.

Given a fixed budget of extra numbers, spreading a low rank across more grids beats concentrating a higher rank in fewer. Adapting two parts at rank 4 outperforms adapting one part at rank 8.

Coverage matters more than capacity. The common defaults, which adapt four parts of the attention block at a modest rank, follow directly from this.

6. The reported numbers

From the abstract, against GPT-3 175B fine-tuned the traditional way:

• 10,000× fewer numbers to train
• 3× less memory needed on the graphics card
• Faster training, and no extra time when actually using the model
• Quality on par with or better than full fine-tuning on RoBERTa, DeBERTa, GPT-2 and GPT-3

That "or better" is not a rounding artefact. On smaller datasets, restricting the change to a simple pattern acts as a discipline — there is simply not enough room to memorise the training examples. Constrained adaptation can generalise better than unconstrained.

Fewer numbers producing a better model is unusual enough to be worth pausing on.

7. The knobs that actually matter

Four settings account for most of the difference between a run that works and one that does not.

Rank. Start at 8 or 16. Raise it only if training stalls at a quality you cannot accept — that is the signal of too little room. Raising rank to fix a model that is memorising makes it worse, and a rank of 128 on a small dataset is a common and expensive mistake.

Which parts to adapt. The attention parts at minimum. Adding the model's feed-forward layers helps when the job needs genuinely new behaviour rather than reweighted behaviour, at roughly triple the file size. Follow the paper: spread wider before going deeper.

$\alpha$. Setting it equal to the rank is a sensible default. Setting it to twice the rank amounts to bigger steps on the adapter. It is a knob, not a law, and it interacts with the learning rate — do not change both at once.

Learning rate. Higher than full fine-tuning, usually about ten times higher, because you are training a small fresh module rather than nudging a settled one. Using a full-fine-tuning learning rate is why some people conclude LoRA does not work.

8. QLoRA, and why this is now normal

The obvious follow-on: if the original model is frozen, its numbers only have to be precise enough to use, not precise enough to adjust. So store them more coarsely.

QLoRA does exactly that — it loads the frozen model at very low precision and trains the sheets on top at normal precision. That brought fine-tuning a serious model within reach of a single consumer graphics card.

This is the recipe most people run today whether they name it or not. The small model I keep on my own laptop — a coding model adapted to write shell commands — is a low-precision frozen base with sheets trained on top, and the whole run finished in under an hour on one machine.

The reason a student can now fine-tune a useful model on a laptop is a chain that starts with this paper.

9. The idea worth keeping

Strip away the machine learning and LoRA is a claim about where information lives.

A big model's numbers encode a great deal. The difference between the general model and a specialised one encodes very little. So writing that difference down at full size is enormous waste — and the compression available turns out to be around four orders of magnitude.

That reframing keeps paying out. It is why adapter files are megabytes, why serving hundreds of fine-tunes from one model is routine, and why the practical question about fine-tuning moved from can I afford this to do I have the examples.

Most of the time, the change you need is far smaller than the thing you are changing.

Sources

• Hu, Shen, Wallis, Allen-Zhu, Li, Wang, Wang, Chen — LoRA: Low-Rank Adaptation of Large Language Models, 2021
• Dettmers, Pagnoni, Holtzman, Zettlemoyer — QLoRA: Efficient Finetuning of Quantized LLMs, 2023
• Aghajanyan, Zettlemoyer, Gupta — Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning, 2020