acceptodds
Under review as a conference paper at ICLR 2027

Right Verdict, Wrong Reason: Counterfactual Attribution Reveals Shortcut Learning in LLM Evaluators

Abstract

As large language models take over the role of evaluator—scoring outputs, modeling rewards, and checking policy compliance—trustworthy evaluation is emerging as a problem as hard as capability itself: an evaluator's verdicts gate agent actions, human review, and training data, yet the evaluator is validated only by the labels it emits. Deployed evaluators are, moreover, continually updated: evidence pipelines are corrected, rules are amended, and the scope of a rule is widened or narrowed. When a verdict flips after an update, standard evaluation checks only whether the new label is correct—never whether it changed for the right reason. This failure mode, right verdict, wrong reason, is invisible to accuracy and is precisely what auditability requires ruling out. We make it measurable. Factoring an evaluator's inputs into typed components (evidence, rules, scope), the 2^3 old/new hybrid configurations form a counterfactual table whose inclusion-minimal component swaps define an exactly computable gold attribution for any verdict change; we prove correctness of enumeration, bound attribution multiplicity, establish a worst-case query lower bound for black-box verification, and separate predicting an attribution from verifying one. We instantiate the framework in ReasonBench, a benchmark of 19,520 cases with executable rule-based evaluators, exact gold attributions, and paired consistency controls. The findings are cautionary. Fine-tuned models reach 98.4% exact attribution accuracy in distribution, yet merely reordering semantically equivalent input sections changes their attribution about half the time; under compositional shift, verdict accuracy holds at 93.8% while attribution accuracy on changed cases collapses to 7.2%; and a pre-specified hypothesis that supervising the full counterfactual table would improve attribution is rejected—the richer target consistently hurts. Accuracy is a poor proxy for faithful attribution; reliable attribution requires executing counterfactuals rather than trusting model outputs.

Primary area:
interpretability and explainable AI
Keywords:
Large language models, LLM-as-a-judge, Counterfactual explanations, Interpretability, Shortcut learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_w9vQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether LLM evaluators can correctly attribute changes in their verdicts to the underlying changes in evidence, rules, or scope. The authors propose a counterfactual attribution framework that identifies all inclusion-minimal component changes sufficient to reproduce a new verdict. Based on this framework, they construct ReasonBench, a benchmark of 19,520 cases with executable evaluators and exact attribution labels. Experiments on 2 Qwen3 models compare direct attribution prediction with full counterfactual-table supervision. The results show that high in-distribution attribution accuracy does not necessarily translate into robustness under input reordering or compositional distribution shifts. The authors further find that supervising the full counterfactual table does not improve attribution accuracy over direct prediction.

Critical strengths

1.The paper investigates whether LLM evaluators can correctly attribute verdict changes to the underlying changes in their inputs. This distinction is meaningful, as conventional accuracy metrics cannot capture errors in attribution. 2.The paper provides a clear formulation of attribution based on inclusion-minimal component changes and constructs executable counterfactuals to obtain exact ground-truth attributions. This avoids relying on subjective judgments of explanation quality.

Critical weaknesses
  1. The experiments only evaluate Qwen3-1.7B and Qwen3-0.6B, both from the same model family. This makes it difficult to determine whether the observed attribution failures generalize to larger models or other LLM families, weakening the paper's broader conclusions about LLM evaluators.
  2. ReasonBench is constructed from two types of rule-based environments, with relatively few independent organizational policy clauses. The reliance on synthetic, deterministic decision processes raises concerns about whether the findings generalize to more diverse evaluation settings.
  3. The observed attribution failures under section reordering and compositional shifts may partly result from fixed input formatting and changes in the attribution-label distribution. Without additional controls, the evidence for shortcut learning remains incomplete.
Questions for authors
  1. Do the reported attribution failures persist across larger LLMs and different model families?
  2. Would section-order augmentation during training improve attribution consistency? How much of the compositional generalization failure can be explained by shifts in the attribution-label distribution?
  3. Could the authors provide cell-level accuracy and error analysis to explain why full-table supervision underperforms direct attribution prediction?
Reviewer_Uw3BOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether LLM evaluators can correctly attribute changes in their verdicts to changes in evidence, rules, or scope. The authors propose a counterfactual attribution framework and introduce ReasonBench, a benchmark with executable ground-truth attributions. Experiments show that models achieve high attribution accuracy in-distribution but perform poorly under semantics-preserving transformations and compositional distribution shifts. The authors also find that supervising the full counterfactual table does not improve attribution accuracy over direct prediction.

Critical strengths

The problem is meaningful: an LLM evaluator may produce a correct verdict without correctly identifying the reasons behind it. Understanding and evaluating this distinction is important for developing reliable LLM-based evaluation systems.

Critical weaknesses
  • I found the paper difficult to follow. The presentation is unnecessarily complicated in several places, making it difficult to identify the central technical contributions and distinguish them from established findings. Examples include: (1) Sections 2–3 introduce substantial terminology and mathematical formalism without providing a complete, intuitive example illustrating the eight counterfactual configurations and how the gold attribution is derived. (2) It is difficult to understand what a typical model input and output look like. The paper does not provide a complete example showing how evidence, rules, and scope are modified, how the eight counterfactual configurations are constructed, and how the ground-truth attribution is obtained. (3) The paper introduces several types of distribution shifts, but their construction is not explained concretely enough. For example, what exactly changes under the operator, prompt-template, and proof-depth shifts? (4) The relationship between the executable reference evaluator and the learned LLM evaluator is not sufficiently clarified. (5) Although different supervision targets are defined, their actual formats are not illustrated. Since comparing direct attribution and full counterfactual table supervision is a central experiment, concrete examples of both targets would help readers understand what is being compared.
  • Shortcut learning and poor OOD generalization are longstanding problems in machine learning. While the paper demonstrates these failures in LLM evaluators, I am not convinced that it provides substantial new insights beyond existing observations. In particular, the theoretical results appear relatively straightforward, largely formalizing exhaustive counterfactual enumeration and its computational properties. It remains unclear what fundamentally new understanding these results provide.
  • Limited experimental scope. The experiments focus on relatively small models (Qwen3-0.6B and 1.7B) and largely synthetic settings. It remains unclear whether these findings hold for stronger LLM evaluators or realistic evaluation tasks.
Questions for authors
  • What are the main novel theoretical or empirical insights beyond demonstrating shortcut learning and poor OOD generalization in another setting?
  • Could the authors provide a concrete end-to-end example of the benchmark construction, counterfactual attribution, and model prediction process?
Reviewer_EJnYOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper starts from the following motivation. As LLMs increasingly serve as evaluators, their verdicts may change when the evaluator is updated. Standard evaluation of the evaluator checks only whether the new verdict is correct, not why it changed, even though the same change can arise from different causes, including the wrong one. This paper targets this “right verdict, wrong reason” failure.

To this end, in Section 2, the authors sets up the problem as follows. The information present in the input provided to the evaluator when judging a fixed case xx is divided into a few components according to its nature (the paper distinguishes evidence, rules, and scope). The procedure DD that makes the judgment is itself fixed, and it is assumed that the same information always yields the same verdict. When the verdict differs between two different versions of the input for the same case, the authors aim to determine exactly which of the changed components were sufficient to produce the new verdict.

In Section 3, the authors define the ground-truth cause by running the fixed decision procedure DD on every mix of old and new components (2k2^k for kk components) and taking all minimal swaps that reproduce the new verdict. Assuming DD is deterministic and mixed inputs remain valid, the authors argue that this recovers the ground truth exactly and is cheap when kk is small. The authors suggest to grade the verdict of LLM evaluators against this ground truth, including whether it stays consistent under superficial changes to the input.

Section 4 introduces a benchmark built on this logic in Section 3, and Sections 5-6 show experimentally that small fine-tuned models, while highly accurate on inputs in the same format as their training data, fail to identify the correct cause when only the surface presentation changes or when they face combinations of changes not seen during training.

Critical strengths
  1. The motivation is compelling. as llm evaluators’ verdicts increasingly drive real decisions, it is important to check not only whether a verdict is correct but whether it changed for the right reason.
  2. The experimental and measurement design appears careful.
Critical weaknesses
  1. The paper presents, in the abstract and introduction, the setting in which LLMs take on the role of evaluators as its motivating problem, but the connection to the problem setup in Section 2 is not intuitive and is difficult to follow. The reasons the motivating problem and the problem actually solved appear to differ are as follows.
  • The identity of the evaluator changes. The abstract and introduction assume LLMs, such as LLM judges and reward models, as the evaluators, but Section 2 defines the evaluator DD as a deterministic function with a fixed decision procedure, which the experiments implement as a rule program. No setting in which an LLM takes the place of the evaluator appears in either the formalism or the experiments.
  • The role of the LLM changes. In the introduction the LLM is the evaluator, but from Section 3 onward the LLM appears as a model that predicts the cause of a rule program’s verdict change. For this reason, it is difficult to be confident that this problem setup and benchmark will be used with real impact in academia or industry. Specifically, it is unclear in which cases one would use this benchmark and what it would allow one to measure.
  1. The scope of the experiments and benchmark is narrow, and information needed for assessment is missing. As far as I understand, the experiments appear to focus on showing why the benchmark is needed. In practice, however, the benchmark is evaluated only with two small models from the same family (Qwen3-1.7B and Qwen3-0.6B). Results from only two small models are too narrow in scope to justify the need for the benchmark. Second, if the paper aims to propose a benchmark, I believe it should generally report how a variety of models perform on it. However, the paper contains no experiments examining whether the claimed “right verdict, wrong reason” failure occurs when various models are used as LLM evaluators; the phenomenon is observed only in two fine-tuned small models. Finally, no information is provided about the prompts used in the experiments or examples of cases in the dataset, which makes the assessment more difficult.
Questions for authors
  1. How would this framework apply when the evaluator itself is an LLM (e.g., an LLM judge or reward model)? In that case, what would the decision procedure DD and the evidence, rules, and scope components each correspond to?

  2. For what specific practical purposes is this benchmark suited? Could you provide examples of use cases from the perspective of academia or industry?

  3. Do you have results for other models, in particular larger models or models used through prompting without fine-tuning? Does the "right verdict, wrong reason" failure also appear in these settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.