Ollama
The default way to run a local LLM. One binary, a model library, and an OpenAI-compatible endpoint on port 11434.
Ollama is the tool most people mean when they say "I run an LLM at home". It wraps llama.cpp in a model registry and an HTTP daemon, so the entire install is one script and the entire workflow is ollama run <model>.
The part that matters for a low-spec box is that the model stays resident. After a request, Ollama holds it in memory for five minutes before unloading. On a machine with 8 GB that is fine. On a 4 GB NAS it is the difference between "the chat works" and "the photo indexer got OOM-killed". If you are in the second camp, set OLLAMA_KEEP_ALIVE=30s and accept the reload cost.
Memory math, not marketing
You can predict whether a model fits without installing anything. Take the parameter count, multiply by the bytes per weight for your quantisation, then add 1–2 GB for the runtime and context:
| Model size | Q4_K_M | Q8_0 | FP16 |
|---|---|---|---|
| 3B | ~2.0 GB | ~3.3 GB | ~6 GB |
| 7–8B | ~4.4 GB | ~7.7 GB | ~15 GB |
| 14B | ~8.5 GB | ~15 GB | ~28 GB |
A 7B model at Q4 needs about 5.5 GB of RAM once you add the KV cache and the process itself. That is why "8 GB minimum" is the honest floor for 7B, and why a 4 GB NAS should be looking at 3B models or smaller.
What it is not
Ollama is not a chat product. There is no login, no user separation and no TLS. Expose it on your LAN if you like; do not expose port 11434 to the internet. Pair it with a front-end (Open WebUI is the usual choice) and let that layer hold the accounts.
Also worth knowing: the answer you get from a 7B model on a 2-core Celeron is a genuine answer, just a slow one. Expect to read it, not to watch it stream.
Pros
- One-line install on Linux, macOS and Windows; no Python environment to fight
- Pulls quantised models straight from a registry, so no GGUF hunting
- Serves an OpenAI-compatible API on :11434/v1, which most front-ends already speak
- Runs CPU-only when there is no GPU, which is the whole point for NAS owners
Cons
- Keeps the model loaded for 5 minutes after the last request by default; on a 4 GB box that is RAM you cannot use elsewhere
- No built-in multi-user auth or HTTPS — you must put it behind a reverse proxy
- Quantisation and context defaults are tuned for convenience, not for the smallest possible footprint
Head-to-head
Alternatives worth a look
llama.cpp
The C++ engine underneath most local inference. Ugly flags, best-in-class CPU performance, runs a 1B model in 2 GB of RAM.
LocalAI
One OpenAI-compatible API that covers text, embeddings, speech-to-text and image generation — all on CPU, no GPU required.
Open WebUI
A self-hosted ChatGPT-style interface that sits on top of Ollama or any OpenAI-compatible API. Accounts, history, RAG and document upload included.
Last verified 2026-10-17 (docs). Prices and licence terms change — check the vendor page before you buy. Spotted an error?