Architecting for Scale: Hybrid Indexing and Retrieval

Managing a corpus of 10 million documents requires moving beyond simple vector search. To maintain performance and accuracy, the pipeline utilizes a hybrid indexing strategy. By storing chunks as both dense vectors and sparse BM25 postings in LanceDB, the system captures both semantic meaning and keyword-specific relevance.

To optimize retrieval, the pipeline employs Reciprocal Rank Fusion (RRF) to combine results from both search methods, effectively mitigating the weaknesses of each. The system retrieves a large candidate pool (150 chunks) and then uses a reranking model to narrow this down to the top 20 most relevant pieces of context. This multi-stage approach ensures that the LLM receives the highest-quality information while keeping the context window manageable.

Ensuring Accuracy: The 'Retrieve, Constrain, Verify, Abstain' Framework

As the document corpus grows, the risk of hallucination increases. The core strategy to combat this is a strict verification loop. The agent is constrained to answer only using the provided context and is required to cite specific sources for every claim made.

Key components of this verification include:

• Normalization and Deduplication: Using MinHash LSH to remove near-duplicates, which prevents the model from being biased by redundant information.
• Structure-Aware Chunking: Adding context prefixes to chunks to ensure the model understands the document hierarchy.
• Calibrated Abstention: If the retrieved context does not contain sufficient information to answer the query, the system is programmed to abstain rather than guess. This is achieved by routing and decomposing complex questions into smaller, verifiable sub-tasks, ensuring that the model only generates responses when it has high-confidence evidence.