Nov · 18 · 2026 · permalink
LAB-01 · Local AI on my PC: Q4 vs Q8
A tool (llmfit) told me Qwen3-8B would fit on my PC. I wanted to check it and, while at it, see what changes between two versions of the same model: Q4, more compressed, and Q8, more precise and heavier. Do they load? How fast do they answer? Do they solve a hard problem better?
How I measured it. Hardware: RTX 5070 Ti, 16 GB. Models: Qwen3-8B in Q4_K_M and Q8_0. Tool: Ollama 0.40.16, on a local server.
- Three rounds per version, with the same context (4096), the same three tasks and the same order. I saved all 18 answers unedited.
- Temperature 0 and seed 42; I measured it with a script, and every figure is the median of the three rounds.
- For the hard problem I wrote a new puzzle: 5 people, 5 days and 5 conditions. A program checked by brute force that it has exactly one solution.
- I wrote it 5 different ways. Each model tried it without reasoning and with reasoning.
What loading showed. Both versions loaded and answered the three tasks in all three rounds, 100% on the GPU (37/37 layers).
| Measure | Q4_K_M | Q8_0 |
|---|---|---|
| Minimum estimated by llmfit (GB) | 5.81 | 9.66 |
| VRAM allocated by Ollama (GB) | 5.59 | 8.90 |
| Card memory, loaded minus idle (GB) | 5.81 | 9.10 |
| Preload (s) | 2.15 | 2.40 |
| Task | Q4_K_M | Q8_0 |
|---|---|---|
| Explain RAM | 114.65 | 76.50 |
| Order robots | 121.49 | 79.00 |
| Summarize a paragraph | 119.89 | 79.51 |
The puzzle. Without reasoning, Q4 solved 1 of 5 wordings and Q8, 0 of 5. With reasoning, both solved 4 of 5; on the fifth they ran out of room and gave no answer. Without reasoning they took under 1 s; with reasoning, Q4 took 13 to 16.5 s and Q8 took 20 to 29.5 s.
| Mode | Q4_K_M | Q8_0 |
|---|---|---|
| Without reasoning | 1 of 5 | 0 of 5 |
| With reasoning | 4 of 5 | 4 of 5 |
Q4 solving it without reasoning only happened with the original wording: with the others, both failed. In a chat-style repeat (factory settings, 3 rounds, 15 attempts per cell) the pattern held: without reasoning, Q4 solved 3 of 15 and Q8, 0 of 15; with reasoning, both solved 12 of 15.
Something visible at a glance: when ordering three robots, Q8 put Ciro at one end and broke a condition. And when asked for a two-sentence summary, Q4 gave four. Neither version came out clean.
What it does not prove. That llmfit is right on other machines, or that its memory figure is exact: only that these two setups loaded and answered here. It is not a quality score: it is one puzzle, five wordings, one machine and one setup. It does not say Q4 is for «everyday» and Q8 for «hard» work either: this test does not back that idea. A model fitting on the card does not mean it does your task well.