Together AI launches Together Compiler, which has teams of AI agents keep rewriting the inference code for open models
Inference cloud company Together AI has released Together Compiler (TKCC), which uses teams of AI agents to rewrite the GPU code behind model inference and re-optimize it as customer traffic changes. The key idea is a "persistent engine": the inference engine starts once, and agents hot-reload each new candidate for testing without a restart. Together's table shows setup time per test falling from 467 seconds to 9 seconds. Agents split work through a shared message board, and a cost model picks the parallelism setup. Self-reported by Together: on SemiAnalysis's InferenceX benchmark, throughput rises by up to 2.6 times. The system already runs on many Together endpoints, and customers can join a limited beta for dedicated deployments.
Why it matters
Open model architectures keep diverging and hand-tuning each one does not scale; keeping agents inside a running engine to test changes quickly is a reusable approach to inference optimization.
Sources1
Written by dalao.ai: the title, summary and why it matters are our own words. The original reports are the record; their publishers keep authorship.