Tag: llama-cpp

Summaries

Towards AI

Apr 15, 2026

Ollama Crumbles in Production: Scale with vLLM or llama.cpp

Ollama, with 52M downloads, fails under load (3s to 1min+ responses for 40 users, collapses at 5 concurrent); vLLM and llama.cpp handle production better despite setup complexity.

__oneoff__

TurboQuant: 3-Bit KV Cache Slash Memory in llama.cpp

Google's TurboQuant quantizes KV cache to 2.67 bits/value with <1% perplexity loss, enabling 110K+ contexts on consumer GPUs; llama.cpp community forks deliver CUDA/ROCm support and 5x compression.