llama.cpp
The C++ engine underneath most local inference. Ugly flags, best-in-class CPU performance, runs a 1B model in 2 GB of RAM.
llama.cpp is the project that made CPU inference practical, and it is what Ollama calls when you type ollama run. Using it directly buys you two things: a smaller footprint, and control over exactly how the memory is spent.
The direct route is one binary. llama-cli for a quick test, llama-server when you want an HTTP endpoint:
llama-server -m model-q4_k_m.gguf -c 2048 --host 0.0.0.0 --port 8080-c 2048 is the context length, and it is the flag people forget. Context costs memory too — roughly 100–200 MB per 1K tokens on a 7B model depending on the KV cache type. Setting a huge context on a small machine is a reliable way to get swapped to death.
Where it beats Ollama on small hardware
Three concrete cases:
- 4 GB total. Ollama's daemon plus a resident model does not leave room for anything else. A one-shot
llama-cliinvocation frees everything when it exits. - 2-bit quantisation. Q2_K is available and Ollama's registry rarely ships it. It costs quality, but it runs.
- Embeddings only. If all you need is a small embedding model for search, llama.cpp does it in a few hundred MB with no server at all.
Where it loses
Everything operational. If the process dies at 3 a.m. nothing restarts it; if you want to switch models you re-download a GGUF and change a flag. Most people should install Ollama first and come here when the RAM runs out — that is not a downgrade, it is the same engine with fewer opinions.
Pros
- Runs comfortably where Ollama cannot — 1B to 3B models in 2 to 4 GB of RAM
- No daemon, no background memory use; the process exits when the request is done
- `llama-server` gives you an OpenAI-compatible endpoint without the rest of the stack
- Quantisation options down to 2-bit, which is the only way some boards will ever run a 7B
Cons
- You manage GGUF files yourself; there is no model registry
- Flags change between releases, so old tutorials rot fast
- No chat UI, no model switching UI, no service supervision
Head-to-head
Alternatives worth a look
Ollama
The default way to run a local LLM. One binary, a model library, and an OpenAI-compatible endpoint on port 11434.
LocalAI
One OpenAI-compatible API that covers text, embeddings, speech-to-text and image generation — all on CPU, no GPU required.
Last verified 2026-10-17 (docs). Prices and licence terms change — check the vendor page before you buy. Spotted an error?