The Document Object & Metadata

1. Why Ingestion Is a System Problem

Most beginners treat RAG as a "Search" problem. Experienced engineers treat RAG as a "Data" problem.

If you load garbage into your vector store, no amount of prompt engineering will fix the output. Common failures include:

• Context Contamination: Indexing navigation menus ("Home > About > Contact") as if they were content.
• Loss of Hierarchy: Flattening a structured PDF into a stream of characters, losing the distinction between a "Section Header" and "Body Text."
• Ghost Sources: Retrieving a paragraph but losing the URL it came from.

Data Ingestion is strict pipeline engineering. It determines the ceiling of your system's intelligence.

2. The Atomic Unit: Document

In LangChain (and LlamaIndex), text is never just a string. It is wrapped in a Document object.

class Document:
  page_content: str
  metadata: dict

This separation is critical.

• page_content: The vector embedding is generated from this. It is what we search against.
• metadata: This is filtered before or after the search. It is never embedded (usually).

3. The Role of Metadata 🏷️

Metadata is not "extra" info. It is the control plane of your retrieval system.

A. Provenance (Trust)

If an LLM says "Revenue is up 20%," the user asks "According to what?" Metadata provides the citation.

metadata = {
  "source": "https://company.com/report-2023.pdf",
  "page": 14,
  "author": "CFO Office"
}

B. Filtering (Precision)

A user might ask: "What did the CEO say about AI in 2023?" If you rely only on vector similarity, you might retrieve the 2021 report because the words are similar.

With metadata, you enforce constraints: filter = { "year": 2023, "author": "CEO" }

This converts a "fuzzy" semantic search into a precise database query.

4. Summary

Never ingest raw strings. Ingest Documents. Your pipeline's first job is to extract content and attach rich, structural metadata.

Key Intuition: "Content is for the model. Metadata is for the system."