BeNice! paper: KV-cache sharing across loops in looped Transformers cuts GPU memory by 66.2% on a 1.3B model
"BeNice! Closing Looping’s Inference-Memory Gap via KV-Sharing"
Looped Transformers reuse the same layers multiple times to improve performance without adding parameters, but their KV-cache memory still grows with every loop.
This paper proposes a training method that lets different loops reuse the same KV-cache without losing accuracy, by combining stochastic-depth pretraining with a fine-tuning objective that makes earlier loops compatible with the final loop's cache.
On a 1.3B model, KV-sharing reduces total GPU memory usage by 66.2% in a tested configuration. In simulated long-context workloads, it also improves throughput by up to 23.6% and reduces 90pt time-to-first-token by 83.6%, while preserving model quality.
alphaxiv.org/abs/2610.31633XSources2
- @askalphaxivX"BeNice! Closing Looping’s Inference-Memory Gap via KV-Sharing" Looped Transformers reuse the same layers multiple times to improve performance without adding parameters, but their KV-cache memory still grows with every loop. This paper proposes a training method that lets different loops reuse the same KV-cache without losing accuracy, by combining stochastic-depth pretraining with a fine-tuning objective that makes earlier loops compatible with the final loop's cache. On a 1.3B model, KV-sharing reduces total GPU memory usage by 66.2% in a tested configuration. In simulated long-context workloads, it also improves throughput by up to 23.6% and reduces 90pt time-to-first-token by 83.6%, while preserving model quality. https://alphaxiv.org/abs/2610.31633View original ·
arXivResearchBeNice! Closing Looping's Inference-Memory Gap via KV-SharingView original ·
Headline by dalao.ai. The quoted post is the record; its author keeps authorship.