Efficient LLM inference hinges on mmap for lazy memory loading (e.g., <10s startup on llama.cpp) and quantization like GGUF K-Quants or AWQ/EXL2 to shrink 15GB models while preserving quality via salient weights and mixed precision.
Inference engines like llama.cpp use mmap to load 15GB models in <10s by lazily pulling weights from SSD to RAM/GPU, avoiding duplication. Quantize to GGUF Q4_K_M for best speed-quality on 32GB RAM GPUs, balancing compression and perplexity.
Quantize LLMs to Q4_K_M for ~0.56 bytes/param, fitting 8B models in 5GB total VRAM (weights +1GB overhead); MoE loads all params but activates subset for speed.
Google's TurboQuant quantizes KV cache to 2.67 bits/value with <1% perplexity loss, enabling 110K+ contexts on consumer GPUs; llama.cpp community forks deliver CUDA/ROCm support and 5x compression.
