Lazy Loading & Generators

1. The Memory Wall

If you have 1 million documents. loader.load() tries to pull all 1 million into a Python list in RAM. Your server crashes. OOM KilleD.

2. Lazy Loading (Generators)

LangChain loaders expose .lazy_load(). This returns a Generator, not a List. It reads one file, yields it, and forgets it.

loader = DirectoryLoader("./huge_dataset")

# Bad (Eager)
# docs = loader.load() # Consumes 10GB RAM

# Good (Lazy)
for doc in loader.lazy_load():
  # Process one doc at a time
  vector_store.add_documents([doc])
  # RAM stays constant

3. Streaming Ingestion

In production pipelines, identifying "Lazy" operations is critical. Your ingestion script should be a pipeline of generators: Generator(Disk) -> Generator(Splitter) -> Generator(Embedder) -> Vector Store.

4. Summary

Don't buy more RAM. Write better pipelines. Lazy loading is the difference between a toy script and a production ETL job.

Key Intuition: "If it fits in RAM, it's not Big Data."