Current neural networks struggle to provide satisfying explanations not due to a lack of information, but because their internal representations cannot be broken down into the distinct, independent facts required for human understanding, hence the "explainability paradox". The authors propose a definition of a good explanation based on counterfactuals that the recipient finds surprising (low prior probability), and suggest that achieving true explainability in AI will likely require integrating symbolic methods with neural systems to allow for decomposable information.

The concept of an explanation is one we easily use and understand - we have a clear, unconscious sense of when an explanation is satisfying.

But what is this sense? What is it that defines a good explanation? When we ask why something happened, what is it that makes some answers feel like “ah yes, now I understand” while others make us feel “no that still doesn’t explain it”?

The Paradox of Neural Explainability

This difference is at the heart of the explainability problem in AI, because what makes some systems, like neural networks, unexplainable is not a lack of information. The paradox of so-called black box systems is that we can actually look directly inside them and see all the computational steps they made, but after doing this, no human will come away feeling that the behaviour of the system has been explained. Somehow, the information just doesn’t cohere into anything like an explanation

In a recent paper, to appear in the ICML 2026 PhilML workshop, we proposed a definition of a good explanation, to investigate this explainability paradox and inform what is needed for AI systems to be explainable.

Causal and Counterfactual Explanations

As a lead up to this definition, consider a simple case of getting an explanation: Bob walks into the office with his hair wet, you ask why, and get the answer “because it’s raining” - we want to understand why this is a good explanation. You might try to say the reason it’s good is that the rain caused his hair to be wet, i.e. a good explanation tells you the cause of the thing you’re trying to explain. But in other cases it doesn’t work to equate a cause and an explanation, like when good explanations are negative statements, e.g. why is Alice hungry? because she didn’t eat breakfast, or why was the bike stolen? because it wasn’t locked - something that didn’t happen can’t cause anything. And defining what a cause is is a whole thorny problem itself, so saying an explanation is a cause may not have helped much anyway.

Another idea we could use instead is counterfactuality. A counterfactual is something which isn’t true, but which, if it was true, would have changed the result. So we can say that the reason “because it is raining” explains Bob’s hair being wet is that, if it hadn’t been raining, his hair wouldn’t be wet (assuming that’s true in our example). A counterfactual is a more general, flexible notion than a cause, and also works for the negative facts. They are often used to define explanations, especially in machine learning, and our paper takes them as a good starting point.

The Role of Prior Probability

However, there is still a problem with simply defining an explanation this way. For example, we would not want to explain Bob’s hair being wet by saying that there wasn’t a drone flying above him outside that kept him dry, or that hair is a substance water adheres to, or that the temperature/pressure on Earth mean things stay wet for a while. These are all things which, if they weren’t true, would mean Bob’s hair wasn’t wet, but if we imagine using them as explanations, they just seem weird and irrelevant.

Our main point in the paper is that the reason these are bad explanations is that we should assume they’re already known to the person looking for the explanation. Most of the counterfactuals they likely already believe, but there must be some that they don’t believe otherwise they wouldn’t be asking for an explanation, and the job of the explainer is to find those facts and tell them to the other person - that’s what should give the “ah yes, now I understand” effect. So, our definition of a good explanation is a counterfactual that the person you’re talking to had low belief in.

Explaining LLM Outputs

Under this definition, if we then try to form good explanations for LLM outputs, we run into a major problem. Suppose an LLM is used by a bank to make loan approval decisions. A user then tells this LLM their age and income, asks for a loan and gets rejected. In order to explain why, we would need to look for counterfactuals that the user placed low probability on, and that means first identifying the facts that could possibly be counterfactuals. But nowhere in the LLM can we find such facts.

You might think that you could change the digits of the age or income in the input and see if the decision changes - maybe if you increase the income and the decision changes to ‘approved’ it would seem like the explanation for the rejection could be that the user’s income was too low - but when you look at how the LLM processes the information, it is not nearly so neat as this. Humans look at this 1-2 sentence input from the applicant and see two pieces of information - age and income - which can be varied independently. These are two facts of the sort we need for explanations. For the LLM, however, the 1-2 sentences contain probably 20 or 30 tokens, and changing any one of these could potentially change the output decision. Previous work has shown that what should be irrelevant changes to the input, such as adding commas or even extra spaces, can sometimes change the output, so it might be a counterfactual that there was a trailing space at the end of the input - should we consider that as an explanation? Or sometimes there is a sequence of random characters that, if placed anywhere in the put, will make the answer always be yes or no - should the fact that the applicant didn’t include this sequence be a counterfactual we consider? We could also imagine rephrasing the input to use an entirely different set of tokens that still convey the same age and income - there are dozens of ways to do this - and there’s no guarantee that this wouldn’t change the output either.

When you look inside the LLM, at its neural connections, this mess gets even worse. There will be millions of ways you could change the network weights so that the answer would have been ‘approved’ - maybe if the connection from this neuron to that one was twice as large, and those other two were half as large etc - trying to search through the different counterfactuals for appropriate ones to include in explanations is completely hopeless.

Resolving the Paradox

The definition we provide shows how we can have full information about the computation that took place, but zero explainability. The reason is that explanations depend on being able to split the information into distinct facts, each of which mostly makes sense on its own - that is how we can identify the subset of facts that are counterfactuals, and then select the low probability counterfactual(s) - but the information for neural computation is not like this. There are so many neurons and weights, most of which can affect each other in complicated ways. We can’t even take the first step towards forming an explanation because we can’t identify the facts that could potentially form it.

Building Explainable Systems

The definition also shows us directly what we do need in a system if we want it to be explainable: we need to be able to break down its representation of information into distinct pieces, each of which can be considered independently. Systems that are known to be explainable, such as decision trees, linear models or rule-based systems, all meet this criterion. If we want good explanations from a system that uses neural networks, we must either develop a complete scientific understanding of how they work so that we actually could divide them into distinct facts, or we need to supplement them with methods that use distinct facts by definition, such as symbolic methods. Some research groups are working towards the former and it is an interesting area, but we believe it is unlikely to reach the level necessary to support good explanations. The latter offers a way to potentially combine the benefits of both symbolic methods, for the parts of the problem that require explainability, and neural methods, for the parts that do not and for the types of processing like raw natural language understanding that only neural systems can do properly.

We can’t underestimate the value of solid definitions. They can illuminate what would otherwise be vague, murky concepts and give us a better line of sight to what we’re after. Whether our particular definition is correct is for the academic community to decide - it will at least be met with some challenges and discussion - but getting clear on the nature of an explanation is a crucial step in AI explainability: it tells us what we actually seek when we ask why something happened, when it might be impossible to give a good answer, and what we need to do to change that.

For more latest research updates, sign up for our newsletter now!