Model evaluation (bake-off)
Testing AI models on examples like your real inputs, with answers checked by hand, before choosing one, rather than trusting benchmarks or price alone.
Updated October 2026 · with examples from our own builds
What does model evaluation mean in practice?
An evaluation runs candidate models over the same set of test inputs and scores each answer against a known correct one. It measures what matters for your feature, such as accuracy per field, failures, speed and cost per call, on data that looks like yours.
Evidence replaces opinion. Teams routinely expect the cheapest model to be good enough and find it isn't, or expect to need the most expensive and find a cheaper one passes.
Keep the test set and re-run it when providers release new models or change their prices, and write down your predictions before the run so you can see where your assumptions were wrong.
Model evaluation at a glance
- Test on
- Inputs like your users' real ones
- Score
- Each field that matters, plus cost and speed
- Re-run when
- Models or prices change
Where has model evaluation come up in our builds?
We built a test set of 29 freely licensed images, including Tamil and Hindi cards, photos of physical cards, foreign controls and two images that aren't cards. We had predicted the cheapest model would read 90% of clear cards correctly; it got 57% of names.
A 15-form benchmark covering menus, bookings, quizzes, job applications, RSVPs and more checks the AI pipeline end to end; the main model built all 15 cleanly.
Common questions about model evaluation
How many examples does an AI evaluation need?
Enough to cover the kinds of input your users will send, awkward ones included. A few dozen hand-checked examples can already separate models clearly, as a 29-image set did for BogarInfra.
Should I rely on public AI benchmarks to pick a model?
Use them to shortlist, not to decide. Public benchmarks test general skills; your feature depends on your inputs, so the deciding test should use them.
Related terms
Planning a product and not sure where model evaluation fits? Ask us, or browse all 116 terms in the glossary.