Local-First AI Is a Systems Problem, Not a Model Problem

Getting an AI model running on your own laptop now takes about four minutes. Install a runtime, download the model, type a question. It answers. The demo works instantly, and it feels like somebody else solved the hard problem and handed you the finished thing.

Then you put it inside a real application that does more than one thing at a time, and you find out the model was never the hard part.

The hard part is that you have just introduced a single, slow, memory-hungry, uninterruptible thing into a codebase written on the assumption that talking to other services is cheap and can happen in parallel.

The difference between renting and owning

When you call an AI through somebody's API, it behaves like a very well-mannered helper. Send it eight requests at once and eight things happen at once. If it gets busy, they add more machines. Somebody else runs the queue, and your code can be completely naive about it, because that elasticity is bought for you.

A model on your own machine is the opposite in every way.

There is exactly one of it. Not a fleet. One.

It holds several gigabytes of memory the entire time, if you want it to stay fast.

It works on one thing at a time, and once it starts producing an answer you cannot interrupt it politely.

So sending it eight requests at once does not give you eight answers at once. It gives you eight answers that trip over each other, evict each other's working memory, and finish later in total than if you had simply done them one after another.

I ran into this building Outreach AI, a tool where several AI agents — one researching, one profiling, one drafting, one criticising the draft — all want the same local model. Written the obvious way, where each agent just waits for its own answer, the whole thing ran slower than doing it in order, and the laptop's fans told the story.

Nothing was wrong with the model. Everything was wrong with my assumption that doing things at the same time was free.

Own the queue, on purpose

Once you accept there is one worker, the design follows.

Put an actual queue in front of the model. Give it a maximum length. Decide what happens when it fills up — turn work away, drop the least important, or make callers wait — and make that decision visible in your code rather than discovering it as a memory leak at three in the morning.

A queue with a limit is not a restriction you are working around. It is the thing that keeps the system honest. A queue with no limit does not make anything faster. It converts a fast, understandable failure — we are full, try later — into a slow, baffling one, where everything gets gradually slower for twenty minutes and then the whole program is killed by the operating system with no useful message.

Order matters more here than in a rented system, too. In Outreach AI the critic checking a draft and the researcher fetching background are not equally urgent — one is holding up something a person is waiting for, the other is filling a cache for later. With one worker, choosing who goes first is the only lever you have. It is a surprisingly big one.

Memory is what you are really rationing

The obvious guess is that you are rationing computing power. You are not. On a laptop you are rationing memory, and the numbers are less forgiving than cloud AI lets you believe.

The model's weights are the fixed cost — that is the several gigabytes, and shrinking them is what makes the model fit at all.

The variable cost is the model's short-term memory of the current conversation. It grows steadily with the length of the conversation and with how many conversations are happening at once. That is the one that kills the program. Two long conversations at the same time can need more of it than the machine has left after the weights, and the failure is not graceful.

This is also why swapping models deserves real thought. Unloading a four-gigabyte model and loading a different one costs seconds, and the person waiting feels every one of them. A pipeline that alternates between a small model for one step and a bigger one for another can spend more time swapping than answering.

Sometimes the right call is to use one slightly-worse model for both jobs. A model that is 10% weaker and always in memory beats a model that is 10% stronger and being loaded from disk.

Streaming is not decoration

With a local model, the time until the first word appears and the time until the last word appears are very far apart. That gap is the entire experience.

A rented model producing a short answer in under a second can get away with a spinner. A local model writing three paragraphs takes long enough that a spinner is unpleasant and a stream of words is delightful — not because streaming is faster, but because it turns an unknown wait into visible progress. Same speed. Completely different feeling.

The engineering consequence is that the stream has to survive the whole journey. If your agent framework waits for the complete answer before parsing it, or your web layer waits for a finished object, you have thrown the benefit away at the very last step. Designing for a stream that runs from the model all the way to the screen is a structural choice, and retrofitting it later hurts far more than starting with it.

What running locally actually buys

Worth being precise here, because the case usually gets made badly.

A local model is not smarter. It is normally meaningfully worse than the frontier alternative, and pretending otherwise just sets up the disappointment.

What it buys is a set of properties that have nothing to do with quality. No cost per use, so you can be wasteful on purpose. No network, so the speed is predictable and it works on a plane. No third party, so private data stays where it is. No vendor, so nothing changes underneath you.

Those are the reasons to do it. Every single one is a plumbing property.

Which is the point. Choosing to run locally is choosing to take on the queueing, the memory management and the patience that a rented API was quietly handling on your behalf. That work does not vanish. It moves into your codebase, where it becomes your problem — and pleasingly, it is the kind of problem that already has known answers.

The model was never the hard part. The queue in front of it always was.