Decoupling Metadata and Binary Storage

To maintain performance as the repository grew from 20,000 to 3 million models, Hugging Face separates concerns by storing only metadata in MongoDB Atlas, while model artifacts (tokenizer files, weights, datasets) reside in AWS S3. This separation allows the team to scale compute and storage independently, ensuring that heavy binary transfers do not impact the responsiveness of the metadata-heavy API.

Optimizing Search with Apache Lucene

As the catalog grew, traditional regex-based queries became a performance bottleneck. The team transitioned to using MongoDB’s $search operator, which utilizes Apache Lucene under the hood. To ensure instant results, they pre-tokenize model names during the insertion process, storing them in arrays within the document. This allows the search engine to perform efficient autocomplete and full-text lookups without the overhead of runtime tokenization.

Multi-Layered Scaling and Traffic Management

To keep the Hub responsive under the load of 14 million users, the architecture employs several strategies:

  • Database Distribution: A seven-node MongoDB cluster distributes read traffic across secondaries. A "hidden" analytics node is used for heavy reporting and ad-hoc queries, ensuring production traffic remains unaffected by resource-intensive operations.
  • Horizontal Scaling: The Hub API runs on Kubernetes, scaling from 10 to 500 pods. While standard Horizontal Pod Autoscalers (HPA) monitor CPU and memory, the team is migrating to KEDA (Kubernetes Event-driven Autoscaling) to scale based on application-specific metrics like request queue depth and event loop utilization.
  • Infrastructure Provisioning: When Kubernetes lacks capacity for new pods, CastAI automatically provisions additional infrastructure nodes to prevent scheduling bottlenecks.
  • Future-Proofing: To handle growth beyond a single replica set, the team is implementing database sharding, which horizontally partitions data across multiple shards to scale CPU, memory, and I/O throughput.