dalao.ai

SGLang

SGLang is a high-performance serving framework for large language models and multimodal models.Reference profileOpen source

About SGLang

SGLang is an open-source serving framework for large language and multimodal models, hosted by LMSYS. It turns open-weight models into high-throughput, low-latency API services, from a single GPU up to large clusters. It is Apache-2.0 licensed. As of 2026-10-30 the GitHub repo has about 38.4k stars and 10k forks. The latest published release is v0.5.22 (2026-10-19); a v0.5.23 tag was created on 2026-10-30 but has no release notes yet.

EditorialUpdated 4 min read

What problem it solves

Downloading an open-weight model gets you weight files, not a service. Serving many users at once means scheduling batches, reusing the KV cache, splitting work across GPUs and supporting each vendor's hardware. SGLang is that layer: you start one server process, it exposes an OpenAI-compatible API, and it keeps the GPUs busy.

The README pitches it at agentic workloads, large-scale serving and RL rollouts. The same Python package also ships SGLang Diffusion for image and video generation.

SGLang powers large-scale production deployments, generating trillions of tokens each day across more than 400,000 GPUs worldwide.

SGLang documentation home page (the project's own claim)

Core capabilities

  • Prefix caching: the docs put RadixAttention and prefix caching at the center of the design, so multi-turn chats and agents that resend the same system prompt reuse KV cache that is already computed.
  • Compatible APIs: OpenAI-compatible endpoints (chat completions, completions, vision, embeddings), the Anthropic Messages API (/v1/messages) and an Ollama-compatible API, plus an offline engine API.
  • Built for scale: the docs cover prefill/decode (PD) disaggregation, expert and pipeline parallelism, hierarchical KV caching (HiCache), speculative decoding, structured outputs and LoRA serving.
  • Hardware: the README lists NVIDIA, AMD Instinct, Google TPU, Intel, Apple Silicon, Huawei Ascend and Moore Threads.
  • Release pace: v0.5.21 (2026-10-02) shipped 779 PRs from 227 contributors; v0.5.22 (2026-10-19) shipped 950 PRs from 245.

Who uses it

The README says SGLang runs production workloads at AI labs, cloud platforms, enterprises and universities, but backs this with a logo image rather than a list you can check.

One user on the public record is Cloudflare. Its October 2026 incident report says the GLM-5.3 deployment on Workers AI runs on SGLang, and that Cloudflare had been flagging this class of bug to SGLang since v0.5.10. RL frameworks including slime, AReaL and verl are listed in the README as using SGLang for rollouts.

How it compares

vLLM is the closest peer: also an Apache-2.0 open-source serving framework. Their public benchmarks come from the projects themselves or from third parties under different test setups, and are not directly comparable.

The llama.cpp and Ollama READMEs both lead with running models locally (while also mentioning the cloud) and offer installs for macOS, Windows and Linux, so local and single-machine use is their main ground. SGLang's quickstart assumes Linux with an NVIDIA GPU, and its docs center on datacenter GPUs and multi-node serving.

Who it's for / Not for

Good fit:

  • teams running their own inference service on their own GPUs;
  • platforms that need PD disaggregation or expert parallelism;
  • research groups that need high-throughput rollouts for RL training.

Poor fit:

  • individuals who just want to try a model on a laptop (the quickstart assumes Linux with an NVIDIA GPU);
  • small teams with nobody to track releases and security fixes.

Getting started

The quickstart assumes Linux, an NVIDIA GPU and a CUDA 13-compatible driver. Run docker pull lmsysorg/sglang:latest, start a server with sglang serve <model id> --host 0.0.0.0 --port 30000, then send one request to check it works. You can also install the Python package with uv.

For launch flags tuned to a specific model and GPU, see the official Cookbook (cookbook.sglang.io).

Pricing & license

The software is free. The LICENSE file at the repository root is the Apache License 2.0 (as of 2026-10-30).

The README and docs list no official hosted inference service, so your costs are mainly your own GPUs and operations. The README lists sglang@lmsys.org for enterprise deployment, consulting and sponsorship inquiries, without saying whether that support is paid.

Risks & limits

Tenant isolation. Cloudflare reports that between 2026-08-27 and 09-09, 155 responses from GLM-5.3 on Workers AI contained data from other customers' requests, all on prompts longer than 681,280 tokens. One cause was SGLang failing to reject oversized requests in PD-disaggregated mode. Fixes began in v0.5.20, and Cloudflare recommends that all operators update their SGLang dependency to v0.5.22 or later. SGLang's own write-up says single-instance deployments are not affected by this cross-engine transfer race.

A safeguard off by default. SGLang's write-up says the new PD KV checksum check (#39500, flag --disaggregation-enable-kv-checksum) is off by default because, when on, every transfer pays for an extra GPU read of the KV cache and an extra synchronization point.

Maintainer load. As of 2026-10-30 the repo has about 6,279 open issues and pull requests. Issue #30233, the bug report behind this incident, was auto-closed by the github-actions bot as inactive on 2026-09-05.

The project is still on 0.x, release notes regularly include behavior changes, and the README and docs state no backward-compatibility commitment.

Our verdict

SGLang is an open-source datacenter inference engine with about 38.4k GitHub stars (2026-10-30): permissive license, broad hardware support, very fast releases. The trade-off is that operations and security stay with you. Anyone running a multi-tenant service on it has to keep up with releases and check their PD-disaggregation setup against the checklists Cloudflare and SGLang have published.

Sources

Official

  1. SGLang READMEGitHub · Accessed Oct 30, 2026
  2. sgl-project/sglang on GitHubGitHub · Accessed Oct 30, 2026
  3. SGLang LICENSE (Apache-2.0)GitHub · Accessed Oct 30, 2026
  4. SGLang releases on GitHubGitHub · Accessed Oct 30, 2026
  5. SGLang tags on GitHubGitHub · Accessed Oct 30, 2026
  6. SGLang v0.5.22 release notesGitHub · Accessed Oct 30, 2026
  7. SGLang v0.5.21 release notesGitHub · Accessed Oct 30, 2026
  8. SGLang documentationdocs.sglang.io · Accessed Oct 30, 2026

Updates & reading

Long readEditorial

In Cloudflare's AI Data Leak, Isolation Fell Between the Cloud and the Open-Source Engine

An SGLang bug report that spelled out the root cause and a patch was closed by a bot for inactivity three days before Cloudflare heard about the leak. Cloudflare assumed the engine would reject prompts that didn't fit; in disaggregated mode it didn't. Small teams that self-host inference or pick a provider face the same gap.

NewsCloudflare

Cloudflare discloses cross-customer data exposure on Workers AI, traced to an SGLang bug

Cloudflare says 155 responses from the GLM-5.3 model on Workers AI between Aug 27 and Sep 9 contained data from other customers' requests, all for prompts over 681,280 tokens. A mismatch between the context window and the KV cache pool triggered a bug in SGLang's disaggregated prefill/decode mode. Cloudflare says it found no evidence of deliberate exploitation; SGLang published a coordinated fix.

2 sources · 1 post on X