TrainO's photo food analysis got almost 2x slower without us shipping a single deploy. The code didn't change. The model provider's behavior did: it started returning pretty-printed JSON. On multi-item plates, the response started hitting the 1024 output-token limit, and every truncated response triggered a retry on the more expensive model. Result: escalations went from 0% to 29% in a few days.

The lesson was simple: an output-token limit is not necessarily a cost-saving mechanism. Squeeze it too tight and you don't reduce cost: you just turn an incomplete response into a full second attempt — slower and more expensive.

The fix was less glamorous, but more correct: we raised the output cap, forced minified JSON (30–50% fewer output tokens), and stopped discarding truncated responses — we now salvage the complete items before deciding whether escalation is needed. We also added per-record traceability: timings, outcome, model used, fallback applied and truncation cause. Next time, the diagnosis won't be intuition. It will be a SQL query.