Suppose one of your analysts brings you a number. It is a big number, it is delivered with complete confidence, and it is about to drive a real decision. You ask the most ordinary question a manager can ask — how did you get there? — and the analyst says: I can't show you. The number is right, trust me, but the workings aren't available.
You would not take that number to the board. You would not put your name beside it in a memo. And notice that your hesitation has nothing to do with whether the analyst is talented. They may be the sharpest person on the team. You hold back for a reason that sits underneath talent entirely: a figure you cannot trace is a figure you cannot defend, and in an enterprise, a decision you cannot defend is not worth more because it arrived quickly. It is worth less, because it has quietly become someone's problem to answer for later.
A lot of AI is being adopted as exactly that analyst
This is worth sitting with, because a great deal of enterprise AI is being brought in as precisely that analyst — and welcomed for the very confidence that should give you pause.
The system produces an answer. A price. A forecast. A risk rating. A decision about a specific customer. It produces it fluently and fast, in language that sounds considered and sure. And when someone asks the natural follow-up — which inputs did it use, which of our records did it lean on, what reasoning carried it from those records to this conclusion — there is no account you can actually walk through. The output arrives in full. The reasoning behind it does not arrive at all.
What makes this dangerous is that the speed is genuinely seductive. An answer in seconds feels like progress, and it photographs like progress in a status update. But speed is doing something subtle and corrosive here: it is disguising the fact that you cannot stand behind the thing it produced. The faster the answers come, the easier it is to stop noticing that not one of them could survive a serious question about how it was reached.
The reframe a CFO has to make
Here is the shift in thinking the situation demands, and it runs against the instinct to treat any fast, correct-looking output as a win.
An answer you cannot trace is not an asset. It is a liability wearing the costume of productivity. It looks like output, it gets counted like output, and it sits on the books as if it were value created — right up until the moment it is challenged. And the decisions that matter get challenged eventually, by someone entitled to ask: a regulator, an auditor, your own board, a customer who was told no and wants to know why.
In that moment, the question is never how quickly did you produce this. No one contesting a decision has ever been reassured to hear it was made fast. The question is always the older, plainer one: show me how you arrived at it. And "the model said so" is the corporate equivalent of the analyst who won't show the math. It is not an answer to that question. It is an admission that you don't have one.
Where these decisions actually land
It helps to be concrete about where automated decisions come to rest, because in the abstract this sounds like a philosophical worry, and it is not. It is an operational one.
A loan declined. A claim denied. A price set for one customer and not another. A supplier dropped. A customer flagged as a risk. Every one of those is a decision that somebody may have a legal or a commercial right to question — and your ability to answer them is only ever as good as your ability to reconstruct exactly what the system saw and how it reasoned its way to the outcome. If the honest answer, when the challenge comes, is "we can't really reconstruct that," then you have not automated a decision at all. You have manufactured an exposure at scale, one confident output at a time, and you have been recording it as efficiency the whole way.
That is the trap. The cost of an untraceable decision does not show up when the decision is made. It shows up, all at once, on the day the decision is contested — and by then you have made thousands of them.
Why speed is the wrong thing to measure
This is why speed is such a misleading way to judge these systems, and why it deserves to be demoted from the headline number it has become.
A faster answer you cannot defend is not an improvement on a slower one. It is the same liability, produced more often. The value of an enterprise decision was never only that it was correct on the day it was made — plenty of correct-looking decisions have cost companies dearly. The value was always tied to your ability to explain it afterward, to someone with the standing to demand an explanation. Strip that ability away and you have not made your decisions cheaper. You have made every single one of them impossible to stand behind, and you will not feel the price of that until one of them is put under a light.
A system that decides quickly and accounts for nothing has not lowered your costs. It has deferred them, and concentrated them, and handed them to whoever is unlucky enough to be standing there when the question finally gets asked out loud.
The requirement worth writing down
So the requirement that actually protects the business is not the one most teams write down. They write "the AI should be accurate," and accuracy, while necessary, is not the thing that saves you in a contested room.
The requirement is that every decision the system makes can be traced — which inputs, which records, which logic — well enough that you could walk a skeptical outsider through it from start to finish and have it hold up. And the timing of that requirement is the whole point. It cannot be a feature you add later, once something has already gone wrong and the lawyers are asking for files that were never kept. It has to be a condition of letting the system make the decision in the first place. If it cannot show its work, it does not get to do the work — at least not the work that ends up in front of a regulator, a board, or a customer with a grievance and a good memory.
That is not a brake on adopting AI. It is the thing that lets you adopt it without quietly accumulating a stack of decisions you would not want to explain.
The question to take into the room
So before you celebrate how much faster your AI makes a decision, ask the plainer question the analyst's refusal should have taught you long ago.
When this decision is challenged — and the ones that matter always are — can you show your work? Or have you just been taking the number on faith, and hoping no one with the standing to ask ever does?