derpx06Notes on systems, models & learning12 things I only understood after drawing them. Each one is a single idea, one diagram, and as few words as I could manage.
Getting several computers to agree on one ordered list of events, when any of them might crash at any moment, is the hardest easy-sounding problem in computing. Raft's contribution was making the answer explainable.
One approach keeps a perfect transcript and pays dearly for it. The other keeps running notes and forgets things. Mamba is a set of notes that finally learned to choose what to write down.
Storing an AI model's numbers at a quarter of their usual precision ought to ruin it. Mostly it does not — and the times it does are specific, well understood, and worth knowing before you pick one.
Finding the closest match among millions of AI-generated points defeats every classical index. The structure that won works like a friend-of-a-friend network, and you navigate it by always stepping closer.
A small model guesses the next few words, a big model checks them all at once, and a clever rule guarantees the result is exactly what the big model would have written alone.
AI models do not read letters. They read numbers handed to them by a compression trick from 1994 — and a surprising share of their strangest failures are that trick's fault, not the model's.
Teaching a huge AI model a new skill used to mean storing a second huge model. LoRA shrank that by ten thousand times, by storing only what changed.
For three years the field built bigger and bigger AI models and fed them the same amount of reading. DeepMind trained over 400 models to show that this was throwing most of the effort away.
For years the field tried to make attention cheaper by doing less arithmetic. It turns out the arithmetic was never the slow part — the walking back and forth was.
Why predicting every single pixel is a waste of time, and how Meta's new architecture learns to see the big picture.
How a small team of engineers shattered AI's most expensive assumption and triggered a trillion-dollar market reckoning.
An in-depth analysis of 'Attention Is All You Need' and the genesis of Large Language Models.
What I read, built and got wrong.