85 real coding tasks, three model tiers, 305 logged API calls. Every number traces to a raw run JSON. The cheap tier won — and not by a little.
This is a field benchmark, not a vendor slide deck. One operator, one machine, the same model family at three settings, graded by executing the code — not by reading it.
thinking:high scored 94.1% vs 97.6% for the cheap setting, ran 25–52% slower, and burned 24–46% more tokens. Across six batches, not a single problem where the expensive tier genuinely won.
"Hard with a known recipe" — lock ordering, decorators, deque tricks — the small model passes at 80–100%. What kills it is medium no-template work: regexes, CSV quoting, constraint lists. Twenty medium problems. It passed four.
Asked to patch a working slugify, the small model met the new requirement and silently dropped the old behavior — slugify('Hello World') returned 'helloworld'. Clean code, invisible at review, caught only by running tests.
Which tier for which task type, plus three prompt/config rules that close the gap entirely. Take the tree, keep your subscription money.
Six programmatic batches, one blind-knowledge track, graded by execution. Ambiguous problems removed from the denominator — publicly, with a paper trail.
| Config | Overall | Latency | Tokens | Verdict |
|---|---|---|---|---|
| glm-5.3 · thinking:low | 83/85 = 97.6% | baseline | baseline | Run this by default |
| glm-5.3 · thinking:high | 80/85 = 94.1% | +25–52% slower | +24–46% more | Pay more, get less |
| glm-4-flash | 47/85 = 55.3% | fastest | cheapest | Fine with a verification step |
Chapters 1, 5, and 9: the executive summary, the small-model failure map, and the decision tree. No email required.
I ran 85 gradable coding tasks (plus 10 multi-turn patch jobs and a blind-knowledge track) through the same model family at three settings. The cheap setting won. Not one problem across six batches where the expensive tier genuinely beat it.
The interesting result is where flash dies. It isn't on the hard problems — it passes the classic interview material at 80–100%. What kills it is medium-difficulty work with constraints that have no template. The failure mode isn't stupidity; it's that flash skims the constraint list the way a tired senior skims a ticket at 6pm.
And the failure that should scare you doesn't show up in any single-run score: the model you trusted to patch your code un-patches something you already shipped. The code looked clean. Only running the tests caught it.
You have a coding task. Start here.
│
├─ One-file script, function, or small tool?
│ └─ YES → cheap tier (thinking:low). Medium-and-below was 100% here.
│
├─ Hard refactor, multi-constraint concurrency, gnarly logic?
│ └─ Still low. Hard tier: 94%. Expensive costs more, runs slower, scored worse.
│
├─ Some rule forces you onto the high tier?
│ └─ Add: "Output a single complete code block only."
│
└─ Batch work on a small model?
└─ Never without a verification step.
If the data doesn't change how you configure your tools, email within 30 days for a full refund.
No. It tests one model family at three settings with full methodology. The findings generalize about tiers and verification habits, not brand rankings. What wasn't tested is labeled UNTESTED, not hidden.
By execution. The grader extracts the code block and runs it — behavior checks, not string matching. Every grader passes a two-way self-check before grading a batch, and every non-PASS gets human review.
Developers and eng managers choosing AI coding tools or tiers, benchmark-curious engineers, and anyone burned by "it looked right."
Yes — we run a 40% affiliate program through Gumroad. See the affiliate page.