There is a particular kind of confidence that only comes from looking at a heatmap.
A model classifies an X-ray as malignant. The saliency map highlights the suspicious region. The radiologist trusts the model because the highlighted area matches her own clinical intuition. Everything feels aligned.
The problem: the saliency map does not know if it’s right. It is a point estimate with no error bar. The highlighted region might correspond to the actual pathology, or it might be the model’s best guess at where to point, drawn from a distribution of equally plausible explanations that differ substantially from one.
There is no second option. No confidence interval. No “here are three alternative attribution patterns that would change the conclusion by less than 5%.” Just a single heatmap rendered with the visual authority of ground truth.
This is not a problem unique to medical imaging. It is the structural design of almost every interpretability tool in production today. SHAP values report a single attribution vector. LIME fits one local linear approximation. Integrated gradients produce one accumulation path. None of them tell you how much they would change if you perturbed the input by an amount too small for a human to notice.
When an explanation carries no uncertainty, it is not transparent. It is a suggestion wearing the clothes of measurement.
The Illusion of a Single Attribution
The core problem is mathematical, not cosmetic. Feature attribution methods compute a single explanation for a single input against a single model state. But the space of valid explanations for the same decision is often much larger than the single point these tools return.
A neural network’s decision boundary in high-dimensional space has a geometry that no 2D heatmap can faithfully project. Two inputs that differ by imperceptible noise can produce attribution maps that highlight entirely different features. This is not a bug. It is a consequence of the model’s sensitivity in directions that the attribution method does not measure or report.
The saliency map says “these pixels matter.” It does not say “these are the only pixels that could have mattered” or “here is how stable this attribution is under perturbation.” It reports one answer from a space of many, and it reports it without a measure of how tightly that answer constrains the space.
This is the difference between an explanation and an estimate. An estimate without error bars is not useless. But it should not be treated as a fact.
Why Uncertainty Is Hard to Report
There is a reason interpretability tools do not report uncertainty: it is hard.
The standard approach, perturbation-based sensitivity analysis, requires re-running the attribution method on dozens or hundreds of slightly modified inputs. For a model that already takes seconds to run, multiplying attribution cost by 50 is often a non-starter in production. And perturbation itself introduces a choice: what counts as a “small” change? Gaussian noise? Adversarial perturbation? Semantically meaningful transformations? Each choice produces a different uncertainty estimate, which means the uncertainty estimate itself is uncertain.
There is also a harder problem underneath. The uncertainty we actually care about is not “how much does the saliency map wiggle when I add noise?” It is “how confident am I that this attribution pattern corresponds to the model’s actual reasoning, rather than a superficial correlation the model learned from the training data?” That second question requires something closer to mechanistic interpretability, understanding the computation the model performs, not just the features it attends to.
Current saliency methods do not attempt mechanistic interpretation. They report statistical association between input features and output values. This is useful. It is just less useful than the heatmap format implies.
The Concrete Cost
What does this cost look like in practice?
A model trained to classify skin lesions learns to associate ruler markings with malignancy, because the training data happened to include rulers next to malignant samples more often than benign ones. A saliency map on a test image highlights the ruler. A dermatologist reading the map sees the ruler highlighted and concludes the model is using the right features, because rulers do appear near lesions in clinical photography. The saliency map has done its job: it attributed importance to the ruler pixels. The model has done its job: it used the ruler as a signal. The dermatologist has been given an explanation that confirms the model’s behavior without revealing that the behavior is a shortcut, not a diagnosis.
The saliency map is not wrong. It is exactly correct about what the model did. The problem is that “exactly correct about what the model did” is not the same as “useful for evaluating whether the model should be trusted.”
This is not a failure of the saliency method. It is a failure of the method’s scope. Saliency maps answer “what did the model attend to?” They do not answer “should I trust what the model attended to?” And they do not answer “how much would this attribution change if the input were slightly different?”
All three questions matter. Only the first one gets answered.
What Would Help
Three concrete changes would move interpretability from theater to tool.
Report attribution stability, not just attribution. Run the saliency method on a small ensemble of perturbed inputs and report the variance. If the top-3 features flip between runs, the attribution is not stable. Say so. A single heatmap with a “high variance” label is more honest than a single heatmap with no label at all.
Separate attribution from evaluation. Make it explicit that saliency methods measure feature-output association, not causal reasoning. The output should be tagged with what it measured and what it did not. “This attribution reflects gradient magnitude at this input point” is a different claim from “this attribution reveals the model’s decision logic.”
Require counterfactual pairs. For every explanation, show the input change that would flip the explanation most dramatically. If changing a single pixel changes the top attribution by 40%, the explanation is fragile. This is not expensive. It is a single optimization pass. And it tells the user something the heatmap never could: how much this explanation depends on the exact input, versus how much it reflects the model’s general behavior.
The Honest Alternative
There is an interpretation that sounds harsh but is not: saliency maps without uncertainty are a form of confidence laundering. They take a model’s opaque computation and produce a visually interpretable artifact that feels like understanding. The artifact is real. The feeling is real. The connection between them is weaker than the format suggests.
This does not mean saliency maps should be abandoned. It means they should be used with the same skepticism applied to any single-point estimate from a complex system. A regression coefficient without a standard error is not a finding. A classification accuracy without a confidence interval is not a result. A saliency map without an uncertainty bound is not an explanation.
It is a measurement with a missing instrument reading. The number is still there. You just do not know how much to trust it.