Jul · 02 · 2026 · permalink
A small LLM doesn't know what it doesn't know
In VaultOS, the command bar uses a local 3B model for phrases like "give me the first password I saved" or "my weakest password" — and it kept returning the same item. The cause: the model has no clock and no notion of password strength, and when the data doesn't exist an LLM doesn't say "I don't know" — it guesses.
The fix wasn't prompt engineering, it was determinism: I detect the intent with rules, order the candidates so item 1 is the right answer by construction, and quality phrases resolve against the real security audit — with an honest null when nothing matches. The model only picks from a list that no longer lets it be wrong.
