Five Local AI Runners, One $250 Mini PC: What Actually Survived Our Benchmarks
There is a very specific kind of optimism that makes a person buy a $250 mini PC and think, sure, I'll run a 7B model on this. We had that optimism. We also had a stopwatch, a power meter, and four free evenings.
Here is what happened when five open-source local LLM runners fought over 16GB of single-channel DDR4.
The guinea pig
Our test box is the kind of machine you can buy on sale at Micro Center or Amazon without thinking too hard: an Intel N100 quad-core (all efficiency cores, no hyperthreading), 16GB of DDR4-3200 in a single channel, and a 512GB NVMe drive. We paid $239 in October 2026. Ubuntu 24.04 LTS, no swap file, nothing else running.
That last part matters more than any benchmark table. Keep reading.
How the test worked
- Models: Qwen2.5-7B-Instruct at Q4_K_M (about 4.7GB on disk) and Llama 3.2 3B Instruct at Q4_K_M.
- Workload: three fixed prompts, 200 generated tokens each, temperature 0, three runs per prompt, median reported.
- Metrics: generation speed, time to first token, peak resident memory, install wall-clock time, and idle RAM footprint.
- Rules: same NVMe, same kernel, no GPU offload except where the runner does it automatically.
We are not claiming these are universal numbers. They are one box, one week, one kernel. Your mileage will wobble by 10 to 20 percent.
The results
| Runner | 7B tok/s | 3B tok/s | First token, 7B | Peak RAM, 7B | Install time | Idle RAM |
|---|---|---|---|---|---|---|
| llama.cpp (built from source) | 4.2 | 8.7 | 0.8 s | 5.4 GB | ~35 min | 120 MB |
| Ollama | 4.0 | 8.2 | 0.6 s | 5.6 GB | 4 min | ~450 MB |
| GPT4All | 3.6 | 7.5 | 1.9 s | 5.3 GB | 5 min | ~600 MB |
| LocalAI | 3.3 | 6.6 | 2.2 s | 6.1 GB | 25 min | ~700 MB |
| Jan | 2.9 | 7.0 | 2.6 s | 6.4 GB | 6 min | ~900 MB |
Sorted by raw speed, since that is what everyone scrolls to first. Now the parts that actually decide whether you keep the thing installed.
Everything here is memory-bandwidth-bound
The gap between fastest and slowest on the 7B is about 30 percent, and none of it comes from clever math. On a chip like the N100, generation speed is essentially a function of how fast you can stream 4.7GB of weights through a single channel of DDR4. Every runner is stuck behind the same wall.
That means the ranking you see is mostly overhead, not intelligence. llama.cpp wins because it does almost nothing except the inference. Jan loses a slice because Electron is sitting on top of the same engine.
A quick note on the integrated GPU: forcing Vulkan offload on the N100's 16-EU iGPU gave us a modest 8 to 14 percent bump in llama.cpp. Real, but not the difference between usable and unusable.
Setup experience, honestly rated
Ollama is the one we would hand to a friend. One curl command, then ollama run qwen2.5:7b, and you are chatting. Model pulls are resumable. The API is OpenAI-compatible. Four minutes from bare Ubuntu to first token.
llama.cpp is the control freak's choice. We compiled it, which took the better part of 35 minutes on four E-cores, and then spent another hour reading flags. In exchange you get total control over quantization, context size, batching, and memory mapping. If you want to understand what is happening, this is the tool that teaches you.
GPT4All is the friendliest for someone who has never opened a terminal. Double-click, pick a model from a built-in list, go. Its API story is thinner, so it is a chat tool more than a building block.
Jan looks the best and feels the heaviest. Its local OpenAI-compatible server is genuinely handy, but around 900MB of idle RAM on a 16GB box is not a rounding error.
LocalAI is the one to pick if you are running a small home server for a handful of apps. Docker Compose up, point clients at it, done. It is also the slowest path on this hardware and the fussiest to configure.
RAM efficiency is the number nobody prints on the box
Here is the finding that reframed the whole test for us. Running a 7B model alone, all five runners sat between 5.3 and 6.4GB of RAM. Comfortable on 16GB.
Then we opened a browser with a dozen tabs and a Slack clone. Peak usage crossed 15GB, the kernel started swapping, and generation collapsed to roughly 1.2 tokens per second on every single runner. The tool you choose does not save you from that. Only more RAM does.
If you are shopping, this is the argument for 32GB and dual-channel memory. Not because the models are bigger, but because you are a human being who does other things.
Would we actually buy one?
For learning and tinkering: yes. Two hundred and forty dollars for a silent, 6-watt box that runs a competent 7B at reading speed is a legitimate bargain. Quantized 3B models at 7 to 8 tokens per second feel genuinely conversational.
As a daily driver for 7B and up: no. Four tokens per second is roughly the pace of a patient typist. If you are doing this every day, put the money into a dual-channel DDR5 mini PC in the $380 to $450 range, or hunt a used desktop with an RTX 3060 12GB — the latter will hand you 25 tokens per second or better on the same model for around $200 to $250 used.
Refurb business micro PCs (the i5-9500T OptiPlex crowd) were our other candidate. We tested one. It is not faster than the N100 on inference and it draws more power. Skip the nostalgia.
What we would install
- Just want it to work: Ollama.
- Want to learn how it works: llama.cpp.
- Handing it to a non-technical colleague: GPT4All.
- Private chat with a real GUI: Jan, if you have 32GB.
- Building an API for other apps: LocalAI, on something with more headroom.
None of these will replace a frontier model for hard reasoning. That is not the point. The point is that for $239 and an evening, you can have a private, offline, no-subscription assistant that never sends your notes to a data center. On a budget, that is a genuinely good deal.
