As large language models are increasingly used for consequential decisions from medical diagnoses to legal assessments, the explanations they provide alongside their answers have become an important tool for auditing their behavior. But those explanations are often unfaithful: they do not actually reflect the reasoning the model used to arrive at its answer.
A new paper from researchers at MIT and colleagues identifies two distinct dimensions of unfaithful explanations. The first is incompleteness: the explanation omits factors that actually influenced the answer. The second is unsoundness: the explanation cites factors that did not actually play a role in the reasoning. Both undermine trust in the stated reasoning of the model.
The researchers propose a removal-based approach to improve faithfulness at test time, meaning the fix does not require retraining the model. The method works by systematically removing elements from the input and observing how the output changes, allowing researchers to identify which factors actually drove the answer and which were merely mentioned in the explanation.
This matters because unfaithful explanations create a false sense of understanding. If a model says it made a decision because of factor X, but actually relied on factor Y, auditors and users are misled about the true behavior. The removal-based approach offers a practical path toward more transparent AI systems without the cost of full retraining.