Why Most SMBs Are Measuring AI ROI Wrong

Ethan Mollick, writing in One Useful Thing, argued that AI adoption in companies is a leadership problem — not a tools problem, not a change management problem, but a leadership problem that requires answers to fundamental questions about what people should do with their time, how work is organized, and how to center people in the new working arrangement. He is right. For small and midsized businesses, the specific expression of that leadership problem is a measurement problem: most SMBs are not measuring AI return on investment correctly, and as a result, they cannot tell whether AI is working and cannot make good decisions about where to use it next.

stateDiagram-v2
direction TB
state "Ad hoc AI use<br/>(no measurement)" as Adhoc
state "Track time savings only<br/>(incomplete baseline)" as Time
state "Full ROI picture<br/>(decision + error + capacity)" as Full
state "Compounding returns<br/>(expand to next process)" as Compound
state "Stalled expansion<br/>(can't justify next step)" as Stall
[*] --> Adhoc
Adhoc --> Time: starts tracking
Time --> Full: adds quality metrics
Full --> Compound: evidence supports expansion
Time --> Stall: time savings unclear or overstated
Stall --> Full: reframe the measurement
Compound --> [*]: repeatable model

The Time Savings Trap

Time savings is the metric most SMBs reach for first when evaluating AI tools. It is visible, easy to understand, and easy to report: this task used to take four hours, now it takes forty minutes. The productivity math looks compelling.

The problem is that time savings is the least durable form of AI return. Employees adjust their work to fill available time. The hours recovered from one task often redistribute into other activities rather than compressing into measurable cost reduction or output increase. After six months, the “we save twelve hours per week per employee” claim becomes difficult to substantiate with actual business outcomes.

This is not a critique of AI productivity tools. It is a critique of a measurement framework that captures the visible metric while missing the ones that compound.

What Actually Compounds

The AI return categories that compound are the ones that improve the quality of decisions and outputs, not just the speed of their production.

I worked with the data team at Marshall & Swift, a real estate cost data firm, on a reporting system initiative that required convincing business analysts and subject matter experts to change how they structured their reporting hierarchy — not because the existing hierarchy was technically wrong, but because the existing structure made a whole category of analytical insight invisible to them. They could not see patterns in the data that mattered because the reporting model was not designed to surface them.

The return from that change was not time savings. It was decision quality: the organization started making decisions with information it previously did not have access to. That category of return is harder to measure, but it is more durable and more valuable than the hours-per-week calculation.

For SMBs implementing AI, the equivalent question is: what decisions are you now making with better information, with less variance, or with faster turnaround? Document review that was previously done inconsistently by different employees, now handled by an AI tool with a consistent standard — the return is consistency, not just speed. Customer inquiry responses that were previously dependent on which employee was available, now handled with AI-assisted drafts that maintain tone and accuracy regardless of volume — the return is reliability, not just efficiency.

The Leadership Work

The leadership decisions that determine AI ROI in an SMB are not the tool selection decisions. They are the standard-setting decisions: what does acceptable AI-assisted work look like in this business, who reviews AI outputs before they reach a customer or make an important decision, and how do we know when an AI tool is not performing to our standard.

Companies that have clear answers to those questions before deploying AI tools generate consistent returns because every employee is operating against the same standard. Companies that left those questions open — deploying tools and expecting employee judgment to fill the gap — generate inconsistent returns, because employee judgment about what AI is for varies across the team.

The measurement framework follows from those standards. If you have defined what good AI-assisted output looks like, you can measure whether you are achieving it. If you have not defined it, you are measuring time savings because it is the only thing visible.

Starting the Right Measurement

The most useful thing an SMB can do before expanding AI use is build a simple before-and-after data set for the processes already using AI: what the process cost before (time, error rate, quality consistency), what it costs now, and what the team is doing with the recovered capacity. That data set is what justifies the next expansion decision — and what makes it possible to select the next use case based on evidence rather than intuition.

The businesses that get compounding AI returns are not the ones with the most tools. They are the ones that started measuring the right things early enough to know what was working.

Frequently Asked Questions

What is the right way for a small business to measure AI return on investment?

Time savings is a valid starting point but an incomplete measure. The categories of AI return that compound over time are: decision quality improvement (are the decisions being made with AI assistance more accurate, more consistent, or faster than before?), error reduction (what was the cost of the errors that AI assistance is now preventing?), capacity unlocked (what work is the team now doing with the hours they recovered?), and customer experience improvement (what changed in output quality or response time that customers experience directly?). Most small businesses can track these categories with simple before-and-after comparisons. The discipline of tracking them, rather than just counting hours, produces better evidence of return and better decisions about where to expand AI use next.

How should an SMB decide which AI use case to pursue first?

The most productive first AI implementation is the one with the clearest before-and-after comparison available. That means choosing a process where you can measure the current state precisely — how long it takes, what the error rate is, what it costs — before deploying AI, so you can compare those metrics afterward. Processes with high frequency, low tolerance for error, and clear completion criteria (a document review that happens 40 times per week, an invoice categorization that requires consistent accuracy, a customer inquiry that always requires the same first five steps) are better first deployments than processes with fuzzy definitions of success. The goal of the first deployment is not maximum impact — it is a clear data point that informs the next decision.

What does 'AI is a leadership problem' mean for a small business owner?

It means that the limiting factor on AI ROI in most small businesses is not the tools — it is whether the business owner or the management team has decided how AI fits into how the business operates. A team with clear answers to those questions — which work AI will assist, how outputs will be reviewed, what the standard is for acceptable AI-assisted work — will generate consistent returns from AI tools. A team that has left those questions open will generate inconsistent results: some individual employees getting a lot of value from AI tools, others using them occasionally, with no shared standard for what good looks like. The leadership work is defining the standard before deploying the tools.