The Case for Small, Task-Specific Models

Small models (those fitting on older, affordable GPUs) are no longer inferior to frontier models. For specific tasks—such as OCR on specialized documents, named entity recognition, or SQL generation—a smaller, fine-tuned model often outperforms a generalized 750B parameter model. The strategy is to slice complex agentic workflows into discrete tasks and assign each to a model trained specifically for that domain. This approach allows for massive cost savings and latency improvements, often achieving low tens of milliseconds for embedding tasks compared to hundreds of milliseconds on managed endpoints.

Solving the Serving Bottleneck

Standard open-source serving tools often require significant research and tuning to reach production-grade performance. A major failure point in current infrastructure is the "top-down router," which attempts to manage worker state and batching centrally. These routers struggle with small, high-frequency requests, often stalling GPU utilization at 20-30% because their view of worker state is perpetually stale.

Superlinked proposes an inverted architecture to solve this:

  • Gateway Annotation: Instead of parsing the entire request, the gateway attaches metadata and drops the request into a shared, high-throughput queue (e.g., NATS JetStream).
  • Pull-Based Workers: Workers independently pull tasks from the queue and form their own batches. This allows workers to saturate themselves based on real-time capacity, effectively doubling cluster throughput.
  • Rust Sidecar: A Rust-based sidecar abstracts the complexity of different runtimes (PyTorch, Candle, SGLang), allowing the system to handle over 50 different model adapters without forcing infrastructure engineers to manually tune every deployment.

Optimizing for Velocity

To maintain organizational velocity, the infrastructure must decouple AI engineering from infrastructure engineering. By using a binary format like MessagePack instead of base64-encoded JSON, the system handles multi-modal data efficiently without clogging the internal queue. Furthermore, implementing an "auto-research" loop that automatically ships tuned configurations for new models allows teams to deploy LoRAs and fine-tunes without constant back-and-forth between teams. This approach turns model adaptation into a seamless process, enabling rapid iteration—such as deploying a LoRA that improved German legal text retrieval by 18% for a cost of 80 cents.