The Failure of Traditional Sharding

Legora initially managed search by sharding document chunks across thousands of Postgres partitions. This approach failed when "hot" active projects and "cold" inactive projects were co-located on the same partitions. Querying a single active project forced the database to pull massive partitions into memory, evicting others and causing cache thrashing. This resulted in P99 latencies spiking from 100ms to 20 seconds.

The Namespace-as-Atom Architecture

To solve this, Legora migrated to Turbopuffer, which treats the "namespace" (a project) as the atomic unit of storage. By mapping one namespace to one directory on object storage (S3), the system eliminates the need for complex sharding. Inactive projects simply reside in object storage at near-zero cost, while active projects are "puffed" into memory/NVMe caches as needed. This architecture provides:

  • Physical Isolation: Each namespace can reside in its own bucket and be encrypted with unique, customer-managed keys, meeting strict enterprise requirements for data sovereignty.
  • Cost Efficiency: By disabling the NVMe cache for specific multi-tenant workloads, they found that memory-only caching provided sufficient performance, further simplifying the infrastructure.

Optimizing for the Memory Hierarchy

Turbopuffer optimizes search by minimizing round trips to object storage, which typically incurs a 200ms latency.

  • Vector Search: Instead of graph-based navigation—which suffers from high latency when traversing nodes on disk—Turbopuffer organizes vectors into a tree structure. Upper-level centroids (the "root") are kept in DRAM, while leaf nodes containing actual document data reside on SSDs or object storage. This minimizes the number of round trips required to narrow down search results.
  • Full-Text Search: The system treats text search as a hashmap intersection problem. The core challenge is minimizing memory bandwidth during set intersection and optimizing dictionary retrieval to avoid unnecessary round trips. The team notes that at web scale, full-text search is often more computationally expensive than vector search.