SGLang: Fast LLM Serving on 400k+ GPUs
SGLang enables low-latency, high-throughput LLM inference from single GPUs to clusters, powering trillions of daily tokens for xAI, NVIDIA, AMD, and 400,000+ GPUs worldwide.
Delivering Production-Grade LLM Inference
SGLang serves large language models (LLMs) and multimodal models with low latency and high throughput across setups from one GPU to distributed clusters. It has 11,684 commits, active releases (49 total), and runs on PyPI with monthly downloads tracked. Benchmarks and performance details appear in release blogs like v0.2 (optimized for Llama3), v0.3, v0.4, large-scale expert parallelism, GB200 rack-scale, and GB300 long context. Use it to handle massive inference loads without performance drops in production.
Massive Adoption Drives Reliability
Deployed at scale generating trillions of tokens daily, SGLang runs on over 400,000 GPUs globally as the de facto open-source inference standard. Trusted by xAI, AMD, NVIDIA, Intel, LinkedIn, Cursor, Oracle Cloud, Google Cloud, Microsoft Azure, AWS, and universities like MIT, UCLA, Stanford, UC Berkeley, Tsinghua. Hosted by non-profit LMSYS, it resolves issues quickly (badges show high closure rates, low open issues). Enterprises contact sglang@lmsys.org for scaled deployments or sponsorships; contributors get coding agent perks like Cursor or Claude Code.
Quick Setup and Ecosystem Integration
Start via PyPI (pip install sglang), with folders for benchmarks, docs, examples, Python code, Docker, tests, and kernels. Join Slack, weekly dev meetings, roadmap, or docs at docs.sglang.io. Draws from Guidance, vLLM, LightLLM, FlashInfer, Outlines, LMQL for design and code reuse. Repo includes dev containers, pre-commit hooks, and AMD 3rdparty support for broad hardware compatibility.