Explain the decision, not the model.

There is a research programme, decades old and honourable, devoted to opening the black box. Interpretability asks what a model has learned, which features it leans on, what happens inside the billions of parameters no human reads. It is fascinating work, and for the frontier systems now entering credit it remains, by the admission of the people who build them, unfinished.

Meanwhile, in every bank, a compliance officer has a deadline. And here is the thing worth noticing: the regulator never asked her to open the box.

Read what the rules actually say

Start with the sharpest text. The US Consumer Financial Protection Bureau holds that a creditor cannot escape its obligations because the algorithm it used was too complex to understand. A denied applicant is owed specific reasons for the denial. Note what that demands and what it does not. It does not require you to explain the model. It requires you to explain this decision, about this person.

The pattern repeats wherever you look. The Federal Reserve’s SR 11-7 expects a model validated before you rely on it. The EU AI Act places credit scoring in its high-risk tier and requires meaningful human oversight while there is still a decision to govern. India’s Reserve Bank, in its 2026 model-risk draft, asks for explainability thresholds on AI models — and then does something unusually candid: it concedes that where full explainability is not achievable, an institution must instead wrap the model in compensating controls, among them mechanisms to verify and corroborate model outputs before they are used.

Sit with that concession. A major central bank looked at the frontier, accepted that the model’s interior may stay dark, and moved the requirement to the only place it can actually be met: the moment before the output becomes an action.

Two different objects

Conflating the model and the decision is the error underneath a hundred stalled AI projects. They are different objects with different obligations.

A model is a statistical artefact. It has weights, a training distribution, and behaviour that may never be fully characterised. Explaining it means saying something true about its interior, which is hard and may be impossible.

A decision is an event. It happened at a moment, to a named person, on a specific account, under a rule that was in force that day. Explaining it means answering a small set of concrete questions: what facts was it based on? What rules did it have to satisfy? Which of those were checked, and what did each one return? Who could have stopped it, and did anyone try?

The first is a research problem. The second is a record-keeping problem — and record-keeping problems are the kind engineers solve.

Where the record has to be made

One catch, and it is the whole design. A decision record assembled afterwards, by joining application logs and inferring what probably happened, is a reconstruction. Reconstructions are plausible. They are also exactly what degrades under dispute, when a regulator, a court or a customer asks why — and the honest answer is that nobody wrote it down at the time.

The record has to be a by-product of the decision being made. That means running the check in front of the action: before the notice is sent, before the call is placed, before the limit changes, test the proposed action against the law, the policy, the consent on file and the context of the account. What fails to clear never fires. What clears carries with it, by construction, the account of why it cleared. The explanation is not written later. It is the thing that let the action happen.

This is why the interpretability question, urgent as it is for science, is the wrong question for a bank on a deadline. You may never be able to say what the model was thinking. You can always be able to say what the institution did, to whom, on what basis, and under whose authority. That is what the rulebooks ask for. It is what an examiner reads. And it is buildable today, on models you would never dare to open.