Handling Errors: Rate Limits & Overflow

1. Why LLM Errors Are Inevitable in Production

If you are coming from traditional software engineering, working with Large Language Models (LLMs) requires a fundamental shift in mindset. In traditional dev, APIs are deterministic: if you send valid JSON to a payment gateway, it succeeds 99.99% of the time, and errors are usually signifiers of a bug.

LLM APIs are different. They are probabilistic and resource-constrained by design.

• Probabilistic: The model might refuse a request today that it accepted yesterday because the "safety filter" rolled a different probability on your input.
• Resource Intensity: Generating text requires massive GPU memory and compute. Providers like OpenAI and Anthropic aggressively throttle traffic to keep their systems stable.

2. Rate Limits and API Throttling (429 Errors) ⚠️

The most common error you will encounter is 429 Too Many Requests. Providers enforce limits on:

1. RPM (Requests Per Minute): How many times you hit the API.
2. TPM (Tokens Per Minute): How much text you are processing.

Common causes include traffic bursts, aggressive retry logic, or parallel agents spinning up too many threads at once.

The Retry Storm (And How to Avoid It)

Naive code retries immediately upon failure.

• Attempt 1 Fails.
• Retry immediately -> Fails again (because you are still rate limited).
• Retry immediately -> Fails again.

This creates a "Retry Storm" that hammers the API provider, guaranteeing you get blocked for longer.

Solution: Exponential Backoff with Jitter

The standard solution is Exponential Backoff: wait a short time, then double the wait time for each subsequent failure (1s, 2s, 4s, 8s). Crucially, you must add Jitter (randomness). If 1,000 users all fail at 12:00:00 and all retry exactly at 12:00:01, they will just DDOS the server again. Jitter spreads out the retries.

import time
import random

def call_llm_with_backoff(prompt, max_retries=5):
  for attempt in range(max_retries):
      try:
          return api.generate(prompt)
      except RateLimitError:
          # Exponential Backoff: 2^attempt
          # Jitter: random float between 0 and 1
          delay = (2 ** attempt) + random.uniform(0, 1)
          
          print(f"Rate limited. Retrying in {delay:.2f}s...")
          time.sleep(delay)
          
  raise Exception("Max retries exceeded")

3. Context Window Limits and Overflow 📦

Every model has a strict Context Window—the maximum amount of text (tokens) it can hold in its "working memory" at once.

• GPT-4-Turbo: ~128k tokens.
• Claude 3: ~200k tokens.

This is a hard limit. You cannot "buy" more context for a specific request. If you send 129k tokens to a 128k model, the API will throw a 400 Bad Request or simply truncate (cut off) the end of your text silently.

4. Strategies for Handling Large Inputs 🔁

When you have a 500-page PDF and a 10-page context limit, you need architectural patterns to handle the overflow.

Strategy A: Sliding Window (Chunking)

Break the document into smaller pieces (chunks) with overlap.

• Chunk 1: Pages 1-5.
• Chunk 2: Pages 4-8 (Overlap ensures context isn't lost at the seam).
• Chunk 3: Pages 7-11.

Pros: Preserves local detail. Great for "Find X in this document" tasks. Cons: Loses global context. Chunk 3 doesn't know what happened in Chunk 1.

Strategy B: Summary-Refine

This preserves global context by maintaining a running summary.

1. Summarize Chunk 1.
2. Feed [Summary of 1] + [Chunk 2] into the model. Ask it to update the summary.
3. Feed [Summary of 1+2] + [Chunk 3] into the model.

Pros: Excellent for coherency and "big picture" understanding. Cons: Slow (serial processing). Errors in the early summary accumulate and poison the final result.

5. Compounding Errors and Failure Propagation 🧠

In Agentic workflows, LLMs often feed into each other.

• Agent A writes code -> Agent B reviews code -> Agent C deploys code.

If Agent A creates a subtle bug, Agent B might hallucinate that it's "correct," and Agent C crashes production. This is Failure Propagation.

Key Insight: Confidence ≠ Correctness. An LLM will often be most confident when it is hallucinating. You cannot rely on the model to "self-correct" reliably without external validation tools (like running the code, checking syntax, or verifying against a database).

6. Designing for Reliability, Not Perfection ⚙️

Since you cannot eliminate errors, you must design for them.

7. Observability and Debugging LLM Systems 🔍

Debugging LLMs is notoriously hard because the "bug" might be a nondeterministic roll of the dice. To survive in production, you need aggressive Observability.

You must log:

• The Exact Prompt: What exactly was sent? (Variables expanded).
• Token Counts: Are you inching closer to the limit?
• Latency: Did a 2-second task suddenly take 30 seconds?
• Tracing: In agent loops, you need a trace of every "Thought" and "Action" step to see where the logic derailed.

A Systems Mindset

LLMs are powerful, but they are bounded systems. They are constrained by math, memory, and network physics. Reliable applications emerge when those bounds are designed for—not ignored.