LlamaIndex details how it made its document parsing model OCRia-1 nearly 10 times faster on a single B200
LlamaIndex has explained how it sped up OCRia-1, its document parsing model that turns pages into text while keeping tables, charts and formatting. The team stored less sensitive parts of the model at lower precision, used a small model to guess several upcoming tokens that the main model then checks in one pass, and rewrote a number of GPU and CPU steps. LlamaIndex self-reports that on a single NVIDIA B200, generation went from about 285 to about 2800 tokens per second, nearly 10 times faster. The team plans to contribute the general improvements to vLLM, the open-source inference framework.
Why it matters
Most of these speed-up techniques apply to other models and are planned for vLLM, so teams deploying models can reuse them directly.
Sources1
Written by dalao.ai: the title, summary and why it matters are our own words. The original reports are the record; their publishers keep authorship.