Picking a model per task, with evals to prove it

Every model provider will tell you their model is the right one for your workload. They cannot all be right. Inside a single enterprise agent we often run three different models, each doing the job it is measurably best at.

Being model-agnostic is not a slogan. It is an operating practice, and it rests on one habit: never choose a model by vibes.

Different steps, different models

Take a prior-authorization agent in a health system. It has three model calls with very different demands.

The first step classifies an incoming request and pulls structured fields from a form. This is a high-volume, low-ambiguity task. A small, fast model does it as accurately as a frontier model at a fraction of the cost and latency.

The second step reads two years of encounter notes and assembles clinical evidence against payer criteria. This is the hard part. Long context, careful reasoning, real consequences. It gets the strongest model available, and the cost is justified because the step replaces hours of clinician time.

The third step drafts a plain-language patient message. Mid-tier model, tuned for tone, cheap enough to run on every case.

Same agent, three models, each earning its seat.

The eval harness that keeps it honest

Choices like these rot if nobody re-checks them. Models improve, prices drop, and last quarter's right answer becomes this quarter's waste. So every agent ships with a small eval set: thirty to a hundred real cases with known-good outputs, scored automatically.

When a new model version lands, we run the set. The report says, per step: accuracy, latency, cost per thousand runs. Swapping a model becomes a one-line change with evidence attached, and the customer sees the report before anything changes in production.

This is also what makes the security conversation easier. "We chose this model because it scored 96 on your own cases, here is the report" lands better than any benchmark chart.

Three rules we follow

• Route by step, not by agent. The unit of model choice is the individual task inside the workflow.
• Let cost count. A model that is 1 percent better and 8 times more expensive loses on high-volume steps.
• Re-run evals on every provider release. Ten minutes of compute beats a quarter of drift.

The result is agents that get cheaper and better over time, without anyone betting the deployment on a single vendor's roadmap.