ERNIEFrom the builder
FastDeploy 2.0: A Large-Scale Model Inference and Deployment Toolkit with Native Support for ERNIE 4.5
Baidu introduced FastDeploy 2.0, a PaddlePaddle-based toolkit for serving large models that is compatible with the OpenAI API and aligned with vLLM's interface, bundling quantization, CUDA Graph, speculative decoding, context caching and prefill/decode disaggregation. It runs on Nvidia GPUs as well as Chinese accelerators from Kunlunxin, Iluvatar, Hygon and Enflame. Baidu says 2-bit quantization shrinks the 300B ERNIE-4.5 weights from 600 GB to 89 GB, and 4-bit weights allow deployment on as few as four GPUs.
Under the condition of 1.1K input sequence length, the A3B model on an H800 GPU maintained TPOT within 25 ms, while the 0.3B and 0.6B models on an A30 GPU kept TPOT within 16 ms.
Saved . A shortened excerpt from the official blog; the original publisher retains authorship.