Latency & TTFT Optimization
The psychology of waiting. Streaming vs Bulk.
Course outline & contents
1. Time To First Token (TTFT)
User A waits 5 seconds for a full paragraph. User B waits 0.5 seconds for the first word, then watches the rest appear.
User A thinks the system is broken. User B thinks it is fast. TTFT (Time To First Token) is the most important metric for User Experience.
2. Streaming
In LangChain, never use .invoke() for a chatbot.
Use .stream().
It yields tokens as they are generated.
for chunk in chain.stream("Why represents the sky?"):
print(chunk, end="", flush=True)
# The user sees progress instantly.
3. Optimizing Prompt Size (Pre-fill Time)
Latency = Prefill Time + Generation Time.
- Prefill: Time to process your input prompt.
- Generation: Time to write the answer.
If you shove 100 documents into the context "just in case," your Prefill Time explodes. Pruning retrieval (Contextual Compression) saves latency.
4. Caching
If two users ask "What is the capital of France?", why compute it twice?
Semantic Caching stores (Question_Embedding -> Answer).
If a new question is 99% similar to a cached question, return the cached answer instantly (0ms latency).
5. Summary
Latency is physics (Model Size) + engineering (Streaming/Caching). You cannot change physics. You must master engineering.
Key Intuition: "Don't make them wait. Make them watch."