- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether LLM evaluators can correctly attribute changes in their verdicts to the underlying changes in evidence, rules, or scope. The authors propose a counterfactual attribution framework that identifies all inclusion-minimal component changes sufficient to reproduce a new verdict. Based on this framework, they construct ReasonBench, a benchmark of 19,520 cases with executable evaluators and exact attribution labels. Experiments on 2 Qwen3 models compare direct attribution prediction with full counterfactual-table supervision. The results show that high in-distribution attribution accuracy does not necessarily translate into robustness under input reordering or compositional distribution shifts. The authors further find that supervising the full counterfactual table does not improve attribution accuracy over direct prediction.
Critical strengths
1.The paper investigates whether LLM evaluators can correctly attribute verdict changes to the underlying changes in their inputs. This distinction is meaningful, as conventional accuracy metrics cannot capture errors in attribution. 2.The paper provides a clear formulation of attribution based on inclusion-minimal component changes and constructs executable counterfactuals to obtain exact ground-truth attributions. This avoids relying on subjective judgments of explanation quality.
Critical weaknesses
- The experiments only evaluate Qwen3-1.7B and Qwen3-0.6B, both from the same model family. This makes it difficult to determine whether the observed attribution failures generalize to larger models or other LLM families, weakening the paper's broader conclusions about LLM evaluators.
- ReasonBench is constructed from two types of rule-based environments, with relatively few independent organizational policy clauses. The reliance on synthetic, deterministic decision processes raises concerns about whether the findings generalize to more diverse evaluation settings.
- The observed attribution failures under section reordering and compositional shifts may partly result from fixed input formatting and changes in the attribution-label distribution. Without additional controls, the evidence for shortcut learning remains incomplete.
Questions for authors
- Do the reported attribution failures persist across larger LLMs and different model families?
- Would section-order augmentation during training improve attribution consistency? How much of the compositional generalization failure can be explained by shifts in the attribution-label distribution?
- Could the authors provide cell-level accuracy and error analysis to explain why full-table supervision underperforms direct attribution prediction?