Buoyant open-sources Kinference, a Kubernetes control plane for vLLM that routes by cached GPU context
Buoyant has open-sourced Kinference, a Kubernetes control plane for serving open-weight models with vLLM. Clients send OpenAI- or Anthropic-compatible traffic to its HTTP endpoint; Kinference then decides which GPU node should handle each request. Beyond per-tenant rate limits and usage tracking, it routes using queue depth and the context each GPU has already processed, so traffic that shares a prompt prefix can reuse that work instead of starting cold. In Buoyant's own benchmarks against plain Kubernetes load balancing, the wait before the first word appears fell sharply on one workload, and the system handled substantially more requests within the same latency target when many prompts shared a prefix. The project is Apache-2.0.
Why it matters
Running open-weight models in-house usually needs more than an engine: a scheduler that sees what each GPU has already processed and enforces per-tenant limits.
Sources1
Written by dalao.ai: the title, summary and why it matters are our own words. The original reports are the record; their publishers keep authorship.