Field Notes

AI Stops Reasoning Once it has Produced a Plausible Answer

AI Stops Reasoning Once it has Produced a Plausible Answer

The best AI-generated financial models don't eliminate assumptions—they minimize them by deriving every possible value from operational drivers and source data instead of hardcoded numbers.

The best AI-generated financial models don't eliminate assumptions—they minimize them by deriving every possible value from operational drivers and source data instead of hardcoded numbers.

Share:

Copy link

Copied!

One pattern I've started paying much closer attention to is how AI decides where to stop reasoning.

The first few times I asked Claude to build financial models, I mostly evaluated the obvious things. Did the formulas work? Did the totals reconcile? Did the outputs look reasonable? Most of the time the answer was yes. The models looked surprisingly complete, and compared to starting from a blank workbook, they represented a meaningful improvement in speed.

What I wasn't paying enough attention to was why each number existed.

The moment this really clicked had nothing to do with a particularly sophisticated model. I was building an operating model and had asked Claude to forecast customer support costs over the next several quarters. It came back with a clean forecast holding customer support costs at $400K per quarter, which happened to be almost exactly the average spend over the previous five quarters. At first glance, that seemed entirely reasonable. In fact, if I had built a quick back-of-the-envelope forecast myself, I might have anchored on the same historical average.

The problem was that we weren't expecting the business to remain the same. We were preparing to launch a new product, and leadership expected that launch to increase support volume as new customers came online. A flat $400K forecast wasn't obviously wrong because of the number itself. It was wrong because it ignored the operational changes we already knew were coming.

Then I imagined someone asking a simple question during a model review.

"Why are customer support costs staying flat?"

I realized neither Claude nor I had a particularly good answer. The number existed because it reflected historical averages, not because it reflected how we expected the business to operate going forward. It wasn't connected to any operational assumption, and if customer growth accelerated after the product launch, nothing about that number would change with it. The model wasn't explaining the business. It was simply extrapolating the past.

So I rebuilt the line.

Instead of treating customer support costs as their own assumption, I tied them to the underlying business driver they were meant to support: Customer Count × Support Cost per Customer. Nothing about the arithmetic became more complicated, but the model immediately became more useful. As the new product drove customer growth, support costs increased automatically. If operational improvements reduced the cost to support each customer over time, the forecast reflected that as well. More importantly, the number stopped being a historical average and became the consequence of assumptions that someone could actually debate during a planning review.

The same failure mode appeared again a few days later in a completely different project.

I was working through a market-entry pricing analysis where the objective was to determine provider reimbursement rates in a new state. The underlying tradeoff was straightforward. Paying providers more should improve provider penetration, but it also reduces net revenue per appointment. To estimate that relationship, I fit a regression across existing states using provider penetration as a function of each state's reimbursement level relative to a national benchmark.

Claude ran the regression correctly.

Then it copied the slope and intercept into the workbook as static numbers.

Nothing looked broken, but those coefficients were already disconnected from the analysis that produced them. The moment someone added another state's data or updated an existing observation, the regression should have changed. Instead, the coefficients would quietly remain frozen unless someone remembered to rerun the analysis manually.

Replacing those pasted values with Excel's SLOPE() and INTERCEPT() functions solved the problem immediately. Now the regression recalculates itself whenever the underlying data changes, and the coefficients remain what they actually are: outputs of an analysis rather than assumptions entered by hand.

Although the operating model and the pricing model had almost nothing in common, they exposed exactly the same pattern. Claude consistently optimized for producing a finished answer, whereas I actually needed a model that remained connected to its underlying logic. Those are related goals, but they aren't the same thing.

I've started thinking about this as one of the default failure modes of AI-generated financial models. Left alone, Claude tends to satisfy the immediate request. Ask for customer support costs, and it gives you customer support costs. Ask for regression coefficients, and it gives you regression coefficients. From its perspective, the task is complete.

The problem is that many of those "inputs" aren't really inputs at all. They're intermediate calculations, operational relationships, or statistics that can and should be derived from existing information. When they're pasted into a workbook as static values, the model still works today, but it becomes progressively less trustworthy every time the underlying business changes.

The most useful prompt I've added to my own workflow has nothing to do with advanced prompting techniques. I simply tell Claude that no static value should exist where a logical driver already exists, and that every calculated figure should trace back to operational assumptions or source data whenever possible. It still misses things, but the quality of the first draft improves noticeably because Claude begins looking for relationships instead of filling empty cells.

More broadly, I think this reflects a useful mindset for building financial models with AI.

The objective isn't to eliminate assumptions. Every model will always require them. The objective is to make sure assumptions exist only where they genuinely belong. Everything else should be derived. Every time a hardcoded value becomes a driver-based formula, the workbook becomes easier to audit, easier to update, and easier to defend in front of someone asking, "Why is this number here?"

I've found that's becoming one of the simplest quality checks I can apply to an AI-generated model. If a number can explain itself by tracing back to operational drivers, it's usually on solid ground. If the only explanation is that the AI produced it, there's often another formula waiting to be written.

Published:

More to explore