The Failure of Flat Retrieval

Traditional Retrieval Augmented Generation (RAG) relies on "chunking," which breaks documents into arbitrary segments (e.g., 500 words or paragraphs) and converts them into vectors for similarity search. While efficient for simple queries, this approach destroys the inherent structure of complex documents. When a document is flattened, headings are separated from their content, tables are detached from their explanatory text, and the logical relationship between sections is lost. This forces the LLM to guess how disconnected fragments relate, often leading to hallucinations or incomplete answers when a query requires information spanning multiple sections.

Structural Navigation as an Alternative

Instead of flattening documents, you can treat them as trees that mirror their original authoring structure (titles, headers, sections, and subsections). By using an agent to navigate this tree, you mimic how a human reads: checking the table of contents, jumping to relevant sections, and following cross-references.

This "Chunkless RAG" approach offers two primary advantages:

  • Context Preservation: Because the agent traverses the tree, it maintains the hierarchy. When reading a paragraph, the model knows exactly which section and subsection it belongs to, as the path (including headings) is preserved.
  • Holistic Reasoning: The agent can hold its place, navigate to different branches of the document, and synthesize information across disparate sections, which is nearly impossible with flat vector similarity.

Implementation and Trade-offs

To implement structural retrieval, you must first reconstruct the document's hierarchy from raw formats like PDFs, which typically lack semantic structure. Tools like Docling are designed to parse these files into structured trees where reading order is preserved and tables remain intact.

However, this approach introduces specific trade-offs:

  • Engineering Complexity: Parsing real-world documents into clean, usable trees is a significant technical challenge.
  • Latency and Cost: Unlike a single vector lookup, structural navigation requires multiple passes and back-and-forth reasoning with the model, increasing both latency and API usage.

Ultimately, structural retrieval is not a universal replacement for chunk-based RAG. Similarity search remains the superior tool for fuzzy, large-scale retrieval across millions of documents. The structural approach is most effective for long, highly organized documents where precision is critical and the relationships between parts of the document are the core of the user's query. In production systems, the most effective strategy is often a hybrid: using similarity search to locate the correct document, and structural navigation to extract the answer from within it.