Our Perspective on What AI Costs to Run

A Personal Perspective

I have shipped machine learning into three businesses that were already running, which is a very different exercise from starting fresh. At Amazon Prime Video my team put content quality validation into the publishing pipeline for a catalog measured in petabytes, with rights, territories, and deadlines attached to every asset. It contributed to a 15 percent improvement in global on-time publishing. At Trax Retail, computer vision and pricing optimization drove promotions in live stores inside a business doing $57M in annual revenue. At Spotter, generative AI went to work for creators whose livelihoods depended on the output.

In all three, the model was the smallest part of the work. The hard parts were the data the model had to read, the workflow it had to fit inside, the evaluation that told us whether it was working, and the people whose jobs changed the day it shipped. None of those appear in a proof of concept, and all of them decide whether the thing survives its first quarter.

The uncomfortable version of this lesson: the demo is the part of the project with the best ratio of impressiveness to difficulty. Everything after it is the opposite.

Where the Cost Actually Lives

Companies budget for a pilot. What they sign up for is a run rate. The pilot is a one-time number that appears in a plan and then goes away. The run rate arrives every month for as long as the feature exists, and it is made of things that never came up in the demo:

• Inference at real volume, including the retries, the long contexts, and the requests that fail and get run again.
• The humans still in the loop. Most production AI ships with review attached. That review is a permanent operating cost and it is usually invisible in the business case.
• Evaluation, which is a system in its own right. If you cannot tell whether quality has regressed, you will find out from a customer.
• Data kept because the model might need it, the modern version of a habit I watched cost a fortune in the data warehouse era.
• The second system you build to watch the first, which nobody plans for and everybody eventually builds.

None of this is an argument against building. It is an argument for knowing the number before you commit, because the moment to decide is before the thing is load-bearing, not after.

Why Pilots Stall

The pattern is consistent enough to be predictable. A pilot is judged by whether it demonstrated something, and it always did. Nobody defined the operational outcome it was supposed to change, so there is no date on which anyone is wrong. The team that built it has already been assigned to the next initiative. The workflow it was meant to improve is untouched, because changing how people work requires authority that a pilot team does not have.

That last point is the one I would underline. Getting AI into production is only partly a technical problem. Mostly it is an organizational one: somebody has to be senior enough to change what people do all day, and accountable enough to still be there when the change is uncomfortable. That is not a role an advisor can fill from outside.

The Half You Should Not Build

Most AI roadmaps I am shown contain a handful of items that will pay for themselves and a longer tail that will not. The tail usually shares a few traits: it automates something rare rather than something constant, it needs data the company does not reliably have, it competes with a feature the incumbent vendor will ship within a year, or its value depends on accuracy the business has not actually defined.

Saying so out loud is straightforward for us and structurally difficult for a firm with a bench to keep busy. We are small on purpose, and one of the things that buys is the freedom to recommend building less.

Why This Matters Now

The experimentation budget of the last two years is being asked to justify itself. Boards that approved AI initiatives on principle are now asking which ones changed a number. At the same time the technology has moved far enough that the interesting question is no longer whether a model can do the task, but whether the company can absorb it: whether the data is there, whether the workflow can change, whether anyone owns the result.

That is an operating problem, not a strategy problem. It is why we work the way we do: embedded in the leadership team, accountable to the same dates as everyone else, and still there when the commitment comes due.

If any of this sounds like the conversation happening at your company right now, I would be glad to compare notes.