The Failure of Fixed Chunking

Most RAG systems rely on a fixed chunk size (e.g., 512 tokens) chosen at indexing time. This is fundamentally flawed because the optimal chunk size is query-dependent, not data-dependent. Small chunks excel at retrieving specific, localized facts (e.g., a name or date), while larger chunks are necessary for retrieving thematic or contextual information (e.g., character relationships or narrative arcs). Because developers must commit to a single size at indexing—before they know what the queries will be—they inevitably lose information. Experiments show that an "oracle" that selects the perfect chunk size per query outperforms any single fixed-size strategy by 20% to 40% in recall.

Multiscale Indexing and Rank Fusion

Rather than attempting to find the "perfect" chunk size, the solution is to embrace multiple scales. The proposed approach involves:

  1. Multiscale Indexing: Duplicate the corpus and index it at several different window sizes (e.g., 50, 100, 200, 500, 1000, 2000 tokens).
  2. Document-Level Retrieval: To make results from different chunk sizes commensurable, retrieve the parent document rather than the specific chunk. This allows the system to compare "apples to apples" across different indexing strategies.
  3. Reciprocal Rank Fusion (RRF): Aggregate the rankings from all index scales using RRF. This simple, non-model-based script acts as a voting mechanism, combining the strengths of different window sizes to surface the most relevant documents.

Trade-offs and Implementation

This method treats retrieval as infrastructure plumbing rather than a model-tuning problem. It requires no complex training or architectural overhaul, only a simple script to merge results. The primary trade-off is storage: indexing at multiple scales increases memory usage by a factor of 2x to 5x. However, because the retrieval calls can be executed in parallel, there is almost no added latency to the user experience. Future work aims to optimize the number and selection of chunk sizes to reduce the storage overhead while maintaining the recall gains.