An issue nobody answered
On July 6, a GitHub user filed a bug report against SGLang, the open-source inference engine, that read almost like a finished diagnosis. In prefill-decode disaggregated deployments, the report said, a request whose input was too long would be marked as aborted and have its prompt swapped for a one-token placeholder. But the request still went through the prefill queue, which sent just one token of KV cache to the decode side. The decode side had allocated room for the full prompt, so it "starts decoding from incomplete/uninitialized KV cache, producing garbage output." The report included a root-cause walk-through and a proposed patch.
Issue #30233 drew one reply, from the github-actions bot. On Sept. 5, the bot closed it "due to inactivity."
Three days later, Gadi Evron, chief executive of security firm Knostic, told Cloudflare through HackerOne that long prompts sent to the GLM-5.3 model on Cloudflare Workers AI were sometimes answered with text from other users' sessions. Cloudflare's incident report, published Oct. 26, confirmed that 155 responses were affected between Aug. 27 and Sept. 9, all to prompts longer than 681,280 tokens, and that their content came from other customers' recent requests to the same model. One of the root causes was the code path described in #30233. Cloudflare wrote that the bug "had been reported to SGLang by another user as issue #30233," and that it noticed this while investigating.
The 155 matters less than what the incident shows about who does what. Open-source inference engines now sit under cloud providers' multi-tenant services, but the guarantee that one customer's data never reaches another doesn't clearly belong to either side. The upstream project moves on features and speed, and a bug report can be closed by a bot. The cloud operator builds on assumptions about how the engine behaves, and when an assumption is wrong, nothing raises an error. The same gap exists for small teams that run their own inference or pick a provider, and for them it's harder to see.
Two reasonable assumptions
Cloudflare says two problems combined. The first was its own configuration. GLM-5.3 was offered with a 1,048,576-token context window, but the KV cache pool on the prefill GPUs held only 681,280 tokens. Cloudflare wrote that it knew about the mismatch before launch: "We expected the engine to return an error for any prompt it could not process, and our long-context tests returned successful responses, so we did not believe this to be an issue."
The second was the SGLang bug. On a single node, a rejected request returns an error. In disaggregated mode, the admission queues on both sides checked whether a request had finished, not whether it had been rejected, so the rejected request went through the whole handoff. The decode side got a "transfer complete" signal and generated a reply over cache pages that mostly still held someone else's data. The client got HTTP 200, a prompt_tokens count matching its long prompt, and a coherent reply.
Each assumption holds up on its own. The operator trusted the engine to reject what didn't fit, and on a single node the engine does. The failure sat in the deployment shape where the two meet. Cloudflare explained that the pool size isn't a configuration value: SGLang sizes it at startup from whatever GPU memory is left after loading weights, while the advertised context length is set separately. "Nothing forces the two to agree, and it is our job to check that they do." It lists this first among its recommendations to other operators and adds: "This is the check we did not enforce."
Users outside saw another layer. Knostic's write-up, dated Oct. 27, says the anomaly first appeared during Evron's ordinary work through OpenRouter, where output contained file paths from a machine other than his and Chinese-language tutorials. After triage on Sept. 7 pointed to Cloudflare as the provider, Knostic tested with its own Workers AI account. A 662,162-token request took 68.9 seconds and echoed back the test markers. An 812,377-token request took 5.0 seconds, lost the markers, returned content unrelated to the prompt and was billed in full, at 103,604.64 neurons. Knostic also says Cloudflare's documentation at the time listed a 1,310,720-token window, and that Cloudflare later told it this was a documentation error and the enforced limit was 1,048,576.
Patches, versions and "no reliable way to tell"
The fix history can be checked line by line in public records. #36988, which Cloudflare says stops the exposure in this incident and which originally targeted a related failure path, was merged to main on Sept. 6, two days before the report, but hadn't shipped. GitHub's release comparisons show it first appeared in v0.5.20, released Sept. 18. #38935, which stops rejected requests earlier, was merged Sept. 16 and shipped in v0.5.21 on Oct. 2. Cloudflare recommends that operators running disaggregated deployments move to v0.5.22, released Oct. 19, or later. SGLang's guidance lands on v0.5.22 in practice too, because the fix for the MORI backend and the release metrics both ship in that version.
The SGLang team's companion post on the LMSYS blog covers more ground than the Cloudflare incident. It describes a second bug in the same family: when a request is aborted mid-transfer, RDMA writes already in flight can't be recalled, and if the decode side hands those pages to another request right away, the late writes land in someone else's cache. SGLang says default disaggregated configurations were exposed to this race from the v0.4.x line in early 2025 until v0.5.21 (v0.5.22 for the MORI backend), and that the fixes haven't been backported to earlier release branches. On whether an operator can tell if leakage already happened, it says: "A deployer has no reliable way to tell, because the corruption leaves no error or log line."
The defenses are still incomplete. The new transfer checksum, --disaggregation-enable-kv-checksum, is off by default because each transfer pays for an extra GPU read and a synchronization point. Cloudflare says the checksum and --kv-canary don't yet cover every model architecture; GLM-5.3, GLM-5.3 Flash and DeepSeek V4 are among those still to be added. Cache pages aren't zeroed between requests, which Cloudflare calls common among inference engines and necessary for the best performance. Its patch to zero KV pages on release, #46435, was still a draft as of Oct. 28.
The other reading
There's a case that this is the system working. Cloudflare's timeline shows it took the model offline within 40 minutes of declaring an incident on Sept. 9 and restored service the same day. Its report lists 20 pull requests merged to SGLang main for this incident and related race conditions, with RadixArk, the company behind SGLang's maintainers, part of the work. Cloudflare also says it "had been flagging these types of issues with SGLang since v0.5.10" and that the incident sped up work it had already started with the SGLang team. After excluding test traffic, Cloudflare says, the timing and volume of affected requests matched normal long-context use, and it found no evidence of deliberate exploitation. Evron's write-up praises how Cloudflare handled the report. SGLang closes its post by saying the maintainers treat this class of report with priority. And because the code is open, outsiders can check when each patch merged and which release carried it.
The performance trade-offs are real too. Cloudflare says not zeroing pages is needed for the best performance; SGLang says the checksum is off by default because of its extra read and sync. Disaggregated mode itself is still getting faster: the v0.5.22 release notes credit one change with up to 27% more throughput on long-prompt workloads. Each added safety check has to be weighed against numbers like those.
That's why the question of who checks matters. An earlier piece on DeepSeek's open-source inference engine argued that once engine code reaches parity, competition moves to the operational layer that isn't open-sourced. This incident puts tenant isolation in that layer. Cloudflare's longer-term plan includes tagging every KV page with its owner and verifying ownership on read, along with formal verification of the RDMA transfer protocol. A prototype it described in August cost under 1% but was disabled for reliability problems; it is now working with SGLang on a version that works across models.
For small teams, the incident leaves at least two facts to check against. Teams running SGLang in disaggregated mode themselves will find the checklist in the two public reports: compare the KV pool size with the advertised context length, run v0.5.22 or later, turn on the checksum where it's supported and watch the release metrics. SGLang says a single-instance deployment without the cross-engine transfer path isn't affected by this race. Teams reaching open models through a router such as OpenRouter should note that Knostic needed triage to confirm the provider was Cloudflare, and which engine and version a provider runs is up to the provider to disclose. Knostic's test was simple: put unique markers at the start and end of a prompt, lengthen it step by step and see whether the markers come back.
On the day the bot closed the bug report from the opening (#30233), GLM-5.3 had been running on Workers AI for nine days. The fix that report proposed was to stop aborted prefill requests from transferring KV cache and to tell the decode side to abort too. The patch merged two months later went in the same direction.