EXHIBIT 01
Both outputs look plausible
Modern models are all fluent. On open-ended prompts, weaknesses hide inside well-formatted, confident prose — the failure isn't on the surface where a quick read can find it.
Case File 02 · Methodology · Declassified
How I compare models: tasks that fail in visible, specific ways — not side-by-side answers and a gut feeling. The method, a worked head-to-head, and the real spending decision it produced.
INTERNAL METHODOLOGY — BUILT AND USED FOR MY OWN MODEL-STACK DECISIONS · EVERY RESULT BELOW IS FROM MY OWN TESTING
Before
A paid GLM subscription justified by general impressions — and no task-specific evidence for which model actually earned its place in which workflow.
After
Head-to-head discriminator tasks with visible failure modes, and a routing rule applied across my stack: the cheapest model that clears the quality bar for each task class.
Measurable Outcome
GLM subscription cancelled; GLM moved to per-call OpenRouter access only; budget caps enforced on all API keys. [AMY: fill in — any cost-per-task or monthly API spend numbers you're willing to publish]
The Problem
The default way teams compare models — give two of them the same prompt, read both answers, pick the one that "feels better" — reliably produces confident conclusions and bad decisions:
Modern models are all fluent. On open-ended prompts, weaknesses hide inside well-formatted, confident prose — the failure isn't on the surface where a quick read can find it.
If you already pay for a model, you're motivated to see its answer as better. "Feels better" is exactly the kind of judgment confirmation bias eats alive.
A single lucky or unlucky generation swings the verdict. Without a task where failure is objective, you can't tell signal from variance.
"Model A seems nicer" doesn't tell you what to route where, what to pay for, or what to cancel. An evaluation that can't change a line item isn't an evaluation.
The Method
A discriminator task is a test chosen so that the models' difference shows up as a visible, specific failure — something you can watch happen, not something you debate. Instead of "whose essay is better," the question becomes "whose billiard ball falls through the floor."
The task has to be hard enough to separate the models, and it has to exercise the capability you're actually buying. When it works, the result doesn't need a rubric discussion — the failure points at the exact capability gap, and the decision follows from it.
WHAT MAKES A GOOD DISCRIMINATOR TASK
Worked Example
Both models got the same brief: build a gravity billiards game. Physics is an unforgiving discriminator — collisions, gravity, and object behavior are either right or visibly wrong within seconds of playing. There is no "both answers seem fine" zone.
Same brief, Claude's implementation. Judge the gravity and collisions yourself.
Open Exhibit A →Same brief, GLM's implementation. The comparison takes about thirty seconds.
Open Exhibit B →| Evaluation dimension | Claude Opus 4.8 | GLM 5.2 |
|---|---|---|
| Physics correctness (gravity, collisions, ball behavior) | [AMY: fill in] | [AMY: fill in] |
| Playability of the result | [AMY: fill in] | [AMY: fill in] |
| Completeness on first pass | [AMY: fill in] | [AMY: fill in] |
| Iterations needed to reach playable | [AMY: fill in] | [AMY: fill in] |
Claude's physics implementation was meaningfully superior.
The gap was visible in the running game — not a matter of taste, and not close enough to argue about.
The Framework
Benchmarks only matter if they change what you route and what you pay for. The framework I apply after every evaluation:
Extraction, navigation, drafting, judgment — models get evaluated per class of work, never "in general."
What does failure look like for this class, and who sees the output? A cover letter and a page scrape have very different bars.
Test candidate models on tasks where clearing the bar — or missing it — is visible and specific.
The cheapest model that clears the bar wins the task class. Expensive models only where quality is the bottleneck.
This is the framework running live inside my agent pipeline: Claude Sonnet is reserved for cover letters where quality is the bottleneck, while DeepSeek V4 Flash/Pro handles page extraction and navigation where volume is the bottleneck.
Business Outcome
The point of the method is that evaluations terminate in spending decisions. Here's what this one changed:
The GLM subscription didn't survive contact with the evidence. It was cancelled after the head-to-head.
GLM moved to per-call access through OpenRouter — still available when it's the right tool for a cheap call, no longer a fixed monthly cost.
API budget caps enforced on all keys, so routing decisions stay decisions — not suggestions that drift. [AMY: fill in — cost-per-task or monthly API spend numbers, if publishing]
For a team, the same method answers the questions that stall AI adoption: which model do we standardize on, for which tasks, and what should we actually pay for? Benchmarked answers replace brand loyalty — and the eval suite becomes an asset you re-run every time a new model ships.
I build discriminator-task evaluations for your actual workload — and turn the results into a routing and spending plan.