At LERETA — the second-largest property tax payment processor in the United States, handling roughly $18 billion in annual transactions — we faced a version of this problem before AI model releases were a weekly event. The company had acquired a Texas firm with similar technology and tried to retrofit that platform into a flagship product rebuild rather than starting fresh. The data structures were slightly different. The processing logic was fundamentally different. The retrofit required more time and rework than a clean build would have.
The lesson was not that acquired technology is always wrong. It was that evaluating whether something fits your actual needs — versus your imagined needs based on the vendor’s presentation — requires scrutiny that most organizations do not apply.
That problem is now running at scale in enterprise AI. Every model release arrives with benchmark scores, demo videos, and analyst quotes. Most enterprise evaluation processes are not built to filter signal from that noise.
timeline title Enterprise AI Model Evaluation Stages Release announcement : Review benchmarks — skip the demo hype Internal scoping : Map your actual production workloads Controlled testing : Run new model on real inputs vs. baseline Integration review : Check API compatibility, latency, cost structure Decision gate : Adopt, wait, or skip with explicit reasoning Post-deployment : Track performance drift after 30 and 90 days
What Benchmarks Are and Are Not
Benchmark scores measure model performance on specific datasets under specific conditions. MMLU, HumanEval, GPQA, and similar evaluations are genuinely useful for tracking progress across model generations. They tell you something real about reasoning capability, code generation quality, and knowledge breadth.
They do not tell you how the model performs on your documents, in your context, at your latency requirements, with your specific prompt patterns.
This gap matters more than it sounds. Organizations have adopted models with impressive benchmark scores and found that their actual use case — extracting structured data from scanned insurance claims, generating customer-facing emails that match a specific brand voice, summarizing legal documents in a particular jurisdiction — performed no better than the previous generation. Benchmark improvement does not guarantee task improvement.
The Evaluation Framework That Actually Works
Four steps, applied consistently, separate useful model upgrades from expensive distractions.
Step 1: Define your actual production tasks. Not what you hope to do with AI eventually — what you are doing now, or have a committed plan to deploy within 90 days. These are your evaluation workloads. Everything else is theoretical.
Step 2: Build a test set from real production data. Take 50 to 100 representative examples from your actual workflows — real documents, real prompts, real expected outputs. Label the expected outputs if they are not already labeled. This test set is what you run every new model against.
Step 3: Run side-by-side comparisons on your test set. New model versus current model, same inputs, blind evaluation if possible. Score on the dimensions that matter for your use case: accuracy, latency, formatting compliance, edge case handling. Do not score on general impressiveness.
Step 4: Apply a cost-adjusted return threshold. A new model that improves accuracy by 8% on your test set is only worth the migration cost if the migration cost is recoverable in reasonable time. Set the threshold before you run the evaluation, not after.
The Migration Cost Nobody Accounts For
Switching AI models is rarely a one-line change. Prompts that were tuned for one model frequently need retesting and revision for another. Output formats shift in ways that break downstream parsing. Latency characteristics change and affect user experience. Retrieval pipelines behave differently under different context window sizes.
The evaluation should include an honest migration cost estimate alongside the performance gain. I have seen organizations decide to hold on a model upgrade — not because the new model was not better, but because the engineering effort required was not justified by the improvement on a non-critical use case.
That is the right decision. It is also one that most organizations cannot make, because they evaluate model performance without evaluating migration cost in the same conversation.
The Retrofit Lesson From Property-Tax Processing
At LERETA, the decision to retrofit an acquired Texas platform rather than build fresh was made under time pressure and with incomplete information about the underlying differences. Six months in, those differences had surfaced as rework that cost more than a fresh build would have.
The parallel to AI model switching is direct. Organizations that adopt a new model without running it against their production workload are making the same type of decision — assuming compatibility based on surface similarity rather than testing it against actual load. The discovery that the new model handles edge cases differently, or requires prompt changes that break existing integrations, comes after the commitment has been made.
Test first. Commit after.
What the Anthropic Model Tier Structure Illustrates
The naming structure across current model tiers — Haiku, Sonnet, Opus, Fable — is useful because it maps roughly to cost and capability. Most enterprise teams should not be running the most capable model on every task. Document routing, simple classification, and response formatting tasks are not improved enough by frontier-model capability to justify frontier-model cost.
The discipline of matching model to task — rather than defaulting to the best available option for everything — is where most cost savings in enterprise AI programs actually live. New releases deserve evaluation. They do not deserve adoption by default.
The organizations getting real return from AI right now are not the ones who move fastest on every model release. They are the ones who have defined their workloads clearly enough to know when an upgrade actually matters.
