dalao.ai

llama.cpp

LLM inference in pure C/C++Open source

About llama.cpp

llama.cpp is a large-model inference engine written in C/C++ with almost no third-party dependencies. It runs quantized open models on laptops, desktops, phones and servers. It's a command-line tool, an OpenAI-compatible local server, and the birthplace of the GGUF model format and the ggml tensor library; Ollama, LM Studio and many other local AI apps use it underneath. In February 2026 the company behind it, ggml.ai, joined Hugging Face, and the project remains MIT-licensed.

EditorialUpdated 11 min read

The llama.cpp interface

What it does

Large models usually assume expensive data-center GPUs and a full Python, PyTorch and CUDA stack. llama.cpp goes after a different problem: running models on hardware people already own. The README states the goal: minimal setup and state-of-the-art performance on as much hardware as possible.

It gets there three ways. It's plain C/C++ with no external dependencies, so a build produces a handful of executables. It uses integer quantization, at 1.5, 2, 3, 4, 5, 6 and 8 bits, to shrink model size and memory use at some cost in accuracy. And it covers a range of hardware. Apple silicon is a first-class citizen, optimized with ARM NEON, Accelerate and Metal. On x86 it supports AVX, AVX2, AVX512 and AMX. NVIDIA GPUs get custom CUDA kernels, AMD goes through HIP and Moore Threads through MUSA, and there are Vulkan, SYCL, CANN, OpenCL and WebGPU backends too. When VRAM runs short, CPU+GPU hybrid inference keeps the layers that don't fit in system memory.

Underneath is the ggml tensor library, maintained by the same team, and the model format is GGUF. Many open models on Hugging Face ship GGUF builds, and a single flag downloads and runs one straight from Hugging Face.

On March 10, 2023, developer Georgi Gerganov published the first version. That day's README set the goal of running LLaMA with 4-bit quantization on a MacBook, and wrote: 'This was hacked in an evening - I have no idea if it works correctly.' That version ran on CPU only with mixed F16/F32 precision; the author had tested just the 7B model and noted that output typically degraded after 30 to 40 tokens. The project later moved from a personal account to the ggml-org organization.

ggml is the foundation under all of it. The ggml.ai site describes it as a tensor library for AI at the edge that powers both llama.cpp and the speech-recognition project whisper.cpp. Its stated principles: a low-level cross-platform implementation, integer quantization, broad hardware support, no third-party dependencies, zero memory allocations at runtime, and keeping the codebase as small and simple as possible. The site also encourages contributors to try crazy ideas and build wild demos.

Who's behind it: per the ggml.ai site, Georgi Gerganov founded the company in 2023 to support ggml and llama.cpp, with pre-seed funding from Nat Friedman and Daniel Gross. On February 20, 2026, ggml.ai announced on GitHub Discussions and the Hugging Face blog that it was joining Hugging Face, and said the project stays open source and is still led by its original maintainer. The announcement says the team keeps full autonomy and leadership over technical direction and that the project will stay '100% open-source and community driven'. The repo's LICENSE file is still MIT. The ggml.ai site itself now says the company was 'acquired by Hugging Face in 2026'. Georgi Gerganov now works at Hugging Face (GitHub profile). The blog post's authors also include Hugging Face engineers Xuan-Son Nguyen and Aleksander Grygier, both already core contributors.

The release cadence changed in 2026. Before that, the project shipped only b-numbered development builds. The team calls them nightlies, though they are not one a day: the number after the b is the commit's position on the master branch, and a merge can trigger a build. b11869, from 2026-10-25, is the 11,869th commit on master. With v0.1.0 on August 17, 2026, the project began adding semantic-version release tags, each placed on the same commit as some nightly (v0.7.0 and b11669 point to one commit). Releases as of 2026-10-25: v0.1.0 and v0.1.1 (both Aug 17, tags only, no Release entry), v0.1.2 (Aug 18, marked Pre-release), v0.2.0 (Aug 22), v0.3.0 (Aug 25), v0.4.0 (Sep 5), v0.4.1 (Sep 15), v0.5.0 (Sep 24), v0.6.0 (Oct 6), and the latest, v0.7.0, on October 16.

The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

llama.cpp README

Key features

One command to download and run. After installing, llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF pulls a model from Hugging Face and starts a chat; llama serve starts an OpenAI-compatible server with a built-in web chat UI. In v0.7.0 the web UI gained a full models manager, the server got a refactored chat session API and improved slot and prompt-cache restore behavior, and ggml moved to v0.27.0.

Many ways to install. The README lists an install script (curl -LsSf https://llama.app/install.sh | sh, or a PowerShell script on Windows), winget, Homebrew, Docker, prebuilt binaries and building from source.

Quantization and GGUF. Integer quantization from 1.5 to 8 bits plus the single-file GGUF format let 7B and 13B-class models run on an ordinary laptop. Lower bit widths mean smaller files and more accuracy loss.

Many backends. Besides CPU, the README lists Metal, CUDA, HIP, MUSA, Vulkan, SYCL, CANN, Hexagon, OpenCL, WebGPU, ZenDNN, zDNN, VirtGPU, BLAS and RPC, with OpenVINO in progress. The RPC backend spreads inference across several machines.

Faster decoding. v0.7.0 added LiliCorr speculative decoding (a small draft transformer that selects candidates on top of DFlash) and MTP speculative decoding for GLM5-Next. The MoE expert cache can be shared across multiple GPUs.

An official desktop app. llama.app is a Mac menu-bar and Windows tray app labeled 'by the llama.cpp team and Hugging Face'. It's about a 1 MB download, free, open source and works offline. It suggests models that fit the machine, keeps them in the Hugging Face cache shared with other tools, and loads them only when something asks, unloading after 5 minutes idle. On the site, machines with 16 GB of memory or more are pointed to Gemma 4 as a starting model. It also serves an OpenAI-compatible API at localhost:9931 that coding agents, editors and scripts can use directly.

How it compares

Ollama: a local model runner written in Go, MIT-licensed, with about 185k GitHub stars (2026-10-25). It wraps model downloading, running and the API.

LM Studio: a GUI desktop app from Element Labs, whose site says its runtime uses MLX and llama.cpp under the hood. It offers a graphical interface and model browser; llama.cpp's code is open source and can be embedded in other products.

vLLM: Apache-2.0, mostly Python, about 95k GitHub stars, built for high-throughput inference on GPU servers. vLLM targets heavy concurrent traffic in data centers; llama.cpp targets personal machines, edge devices and non-NVIDIA hardware.

MLX (mlx-lm): Apple's machine-learning framework and its language-model tooling, MIT-licensed and Apple silicon only. It targets Mac-only setups that use MLX-format models or fine-tuning; llama.cpp covers Mac, Windows and Linux with one workflow.

The official llama.app: a GUI that still tracks upstream llama.cpp closely. Its menu shows the llama.cpp build it runs (b9726 in the site's example), and it shares a model cache with the command-line version. Server deployments, scripting and embedding remain the job of the CLI and the library.

Ollama, LM Studio, vLLM, MLX and llama.app are all free and open source or free to download.

Who uses it, and who it's for

Adoption (as of 2026-10-25): the main GitHub repo has 132,254 stars and 24,772 forks. Including anonymous committers, GitHub's contributors API returns 2,169 entries, and 446 are tied to a GitHub account; GitHub's docs say the endpoint links only the first 500 author emails in a repo to accounts and treats the rest as anonymous. The repo was created on 2023-03-10 and was still being pushed to on 2026-10-25. The maintainer-PRs link at the top of the README lists the maintainers' GitHub accounts, founder Georgi Gerganov among them. The official client's two repos, Llama-macOS and Llama-Windows, show about 1.7K and 270 stars on the llama.app page.

Who it's for

Individuals who want to run open models offline on their own machine and care about privacy. On Apple silicon Macs, Metal acceleration works out of the box.

Developers embedding models in their own products. It exposes a C API with no Python dependency, which makes it easy to ship inside desktop, mobile or embedded software.

People without NVIDIA hardware. AMD, Intel, Moore Threads, Huawei Ascend (CANN), Qualcomm Hexagon and Vulkan-only GPUs all have a backend, and RISC-V chips get optimizations for RVV and other vector extensions, which helps for embedded and edge experiments.

Anyone short on VRAM who still wants a big model. Hybrid inference keeps some layers in system memory, and the --moe-cache-mib flag added in v0.7.0 caches host-memory MoE experts on the GPU; the project says that speeds up decoding by up to about 2.2x.

Researchers and hobbyists who want new models fast. New architectures tend to land quickly; v0.7.0 alone added K2 Horizon, Prism Bonsai 2 27B, EmbeddingGemma-2, MiniCPM-V 4.7 and Nemotron models.

Not for

Teams serving many concurrent users from a data center. High-throughput batched inference is usually better served by GPU-cluster engines like vLLM.

Casual users who just want to click and go. llama.cpp is still mainly a command-line tool; the official llama.app client offers a route that skips the terminal.

Anyone unwilling to manage their own security boundary. The RPC backend and the server have had several serious vulnerabilities.

Getting started

The quickest route is the official script: on macOS or Linux run curl -LsSf https://llama.app/install.sh | sh, and on Windows run irm https://llama.app/install.ps1 | iex in PowerShell. Homebrew, winget, Docker and the prebuilt binaries on the Releases page also work. The llama.app client offers a route that skips the terminal.

After installation, try a small model with llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF, then move to a larger one that fits in memory. To serve other apps, run llama serve -hf with a model name and point the app's OpenAI endpoint at the local server. Since v0.7.0 the default port is 9931, not 8080 (Docker images still use 8080 inside the container), so older tutorials that use 8080 are out of date.

Any chat front end that speaks the OpenAI-compatible API can connect to the llama serve endpoint.

On the two numbering schemes: b-numbered builds are the development builds the team calls nightlies. The number is the master commit count, and builds can change behavior from one to the next. v-numbered tags are releases; v0.7.0 sits on the same commit as b11669. docs/release.md describes where things are headed: the install script fetching binaries built from release tags, and package managers consuming the tags. As of 2026-10-25: run without a version, the official install script reads the latest pointer in a Hugging Face bucket, which on 2026-10-25 was b11852, not v0.7.0. ggml discussion #1579 says llama.app and Winget publish nightlies rather than releases, the Homebrew and Debian packages link against a system ggml, and building the Homebrew formula from release tags is still on the to-do list.

The install script accepts a LLAMA_VERSION environment variable for pinning a version, but only as a b-number (the script checks the build number the binary reports), and only for builds the bucket actually holds. On 2026-10-25 the bucket held b11429, the commit behind v0.6.0; b11669, the one behind v0.7.0, was not, so LLAMA_VERSION=b11669 fails to download. v0.7.0 is still available as prebuilt files on the b11669 page under Releases, or by building the v0.7.0 tag from source; -DLLAMA_BUILD_IS_DEV=OFF gives a clean version string.

Pricing & license

llama.cpp is free and MIT-licensed: it permits commercial use, modification and distribution in closed-source products, provided the copyright notice is kept. The official llama.app client is also free and open source. The Open Core section of the ggml.ai site says the company may develop commercially licensed extensions in the future; as of 2026-10-25 there is no paid product. Models run with it carry their own licenses, which govern commercial use separately.

Risks & limits

First, security. The repo's security advisories page lists 13 advisories. The two most severe involve the RPC backend: CVE-2024-42479 in August 2024 (write-what-where in rpc_server, 9.8) and CVE-2026-34159 in March 2026 (a check bypass leading to unauthenticated remote code execution, 9.8). Several more are heap overflows in GGUF parsing and the tokenizer (CVE-2026-33298 and CVE-2025-49847, for example); CVE-2026-27940 in March 2026 was a bypass of the 2025 fix for CVE-2025-53630, and llama-server itself had an out-of-bounds write in January 2026 (CVE-2026-21869), rated as follows: GitHub's advisory marks it Low, yet its own text says a remote attacker could crash the process or go on to execute code, and NVD scores it 9.8, critical. Fixes ship in b-numbered builds, such as b3561 for CVE-2024-42479 and b5662 for CVE-2025-49847. The RPC server and llama-server flaws can be triggered remotely, and fixes arrive only in newer builds.

Second, changing defaults. v0.7.0 changed the server's default port. Semantic versioning started in August 2026.

Third, the docs and the Releases page. docs/release.md says releases are not published to GitHub Releases, only tagged, yet every release from v0.1.2 on appears there, while v0.1.0 and v0.1.1 are tags only; the entries for v0.2.0 through v0.7.0 each carry just a text file that records the build number. The docs also say the install script and package managers use releases, while what they actually install is a nightly.

Fourth, direction after the ownership change. Since joining Hugging Face, the announcement says the team keeps full autonomy over technical direction and the project stays 100% open source, and llama.app already shares the Hugging Face cache.

Our verdict

What sets llama.cpp apart: plain C/C++ with no external dependencies, integer quantization from 1.5 to 8 bits with the GGUF format, backends covering CPU, Metal, CUDA, HIP, Vulkan and more, an MIT license, and the official llama.app desktop client. Its limits: it is still mainly a command-line tool; semantic versioning started in August 2026; the RPC backend has had two remote flaws scored 9.8, llama-server's out-of-bounds write is rated 9.8 by NVD but Low by GitHub's advisory, and model-file parsing has had repeated heap overflows.

Why it stands out

llama.cpp is the reason a large language model can run on the laptop you already own. Georgi Gerganov hacked the first version together in an evening, in plain C and C++ (his first README said exactly that), and the project has since defined the quantization formats, the file format, and the performance expectations for local inference.

Many desktop AI apps use it for local inference or borrow its ideas. In February 2026 ggml.ai, the company Georgi founded to support ggml and llama.cpp, joined Hugging Face; the ggml team keeps technical leadership and the project stays 100% open source under the MIT license. The repository is still readable, still moving weekly, and maintained by a group who care about a Raspberry Pi as much as an H100.

A caveat: the server and the RPC backend have had serious vulnerabilities. CVE-2026-21869, an out-of-bounds write in llama-server, is marked Low in the GitHub advisory but scored 9.8 by NVD, and the advisory says it can crash the process or lead to remote code execution. Stay current and isolate it before exposing it to a network.

It defined the quantization and file formats used for local inference, is MIT licensed, and is maintained for hardware from a Raspberry Pi to an H100.

Sources

Official

  1. ggml-org/llama.cpp (GitHub)GitHub · Accessed Oct 26, 2026
  2. llama.cpp READMEGitHub · Accessed Oct 26, 2026
  3. llama.cpp/README.md at 18ebda34d67c05f4f5584a9209e7efb949f5fd56 · ggml-org/llama.cppGitHub
  4. llama.cpp LICENSE (MIT)GitHub · Accessed Oct 26, 2026
  5. llama.cpp ReleasesGitHub · Accessed Oct 26, 2026
  6. Release v0.7.0 · ggml-org/llama.cppGitHub
  7. Release b11669 · ggml-org/llama.cppGitHub · Accessed Oct 26, 2026
  8. dist : releases and versioning of `ggml-org` projects · ggml-org/ggml · Discussion #1579GitHub

Updates & reading