Async & High-Throughput Patterns

1. The Synchronous Bottleneck

In a prototype, you write: response = llm.invoke("Hi") This code blocks the CPU for 2 seconds. If you have 100 users, User #100 waits 200 seconds (3 minutes).

Synchronous LLM calls are death for scale. Production systems must be Asynchronous and Parallel.

2. Async in LangChain

Python's asyncio is the key. Instead of invoke, use ainvoke. Instead of waiting, the CPU switches context to handle another request while the GPU computes.

import asyncio

async def handle_requests(inputs):
  # This runs 5 calls in PARALLEL.
  # Total time = Max(Time), not Sum(Time).
  tasks = [chain.ainvoke(i) for i in inputs]
  results = await asyncio.gather(*tasks)
  return results

3. Rate Limits & Backpressure

If you parallelize too much, you hit the API Rate Limit (e.g., 5000 TPM). You need a Semaphore to control concurrency.

semaphore = asyncio.Semaphore(10) (Max 10 active requests).

4. Batching for Throughput

GPUs hate single requests. They love matrices. Serving 10 requests of 100 tokens separately takes 10x time. Serving a batch of 10 requests takes ~1.2x time.

In LangChain, use .batch(). Behind the scenes, it optimizes the thread pool for maximum throughput.

5. Summary

Throughput (Requests Per Second) is different from Latency (Seconds Per Request). To maximize Throughput, you must saturate the IO with Async patterns.

Key Intuition: "Don't line up single file. Walk through the door together."