LocalAI
One OpenAI-compatible API that covers text, embeddings, speech-to-text and image generation — all on CPU, no GPU required.
LocalAI is the "one API to replace them all" option. Point a client's base_url at it and the same code that called OpenAI now runs a local model — chat, embeddings, transcription and image generation all behind the same shape of request.
That breadth is the selling point and the trap. Each loaded model is resident memory. A chat model plus an embedding model plus a whisper model on one 4 GB board is a swap storm, not a pipeline. The honest sizing rule is: budget the sum of the models you load at once, not the largest one.
Where it genuinely wins
If you are building something — a small internal tool, a script that needs embeddings and a summary — having a single service that answers /v1/chat/completions and /v1/embeddings without three daemons is worth real money in maintenance time. The Docker image is self-contained and the model definitions are plain YAML, so the whole thing is reproducible in a compose file.
Where Ollama is the better default
For a single chat model behind a UI, LocalAI is more configuration for the same result. Ollama's registry and one-liner are hard to beat for the 90% case. Come to LocalAI when you actually need several kinds of model on one port.
One practical note: keep the image tag pinned. The project moves quickly and the model gallery format has changed across releases; :latest plus an old config directory is a reliable way to spend an evening reading logs.
Pros
- One endpoint replacing OpenAI for chat, embeddings, whisper and image generation
- Multi-model: several backends can be loaded at once and routed by model name
- First-class Docker and Kubernetes deployment, no Python environment
Cons
- Multi-model loading is exactly what a 4 GB machine cannot afford
- Configuration is a YAML directory of model files; more knobs than Ollama, more ways to be wrong
- Docs assume you already know which backend you want
Alternatives worth a look
Ollama
The default way to run a local LLM. One binary, a model library, and an OpenAI-compatible endpoint on port 11434.
llama.cpp
The C++ engine underneath most local inference. Ugly flags, best-in-class CPU performance, runs a 1B model in 2 GB of RAM.
Last verified 2026-10-17 (docs). Prices and licence terms change — check the vendor page before you buy. Spotted an error?