- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
AgentEx-Bench evaluates the quality of observable explanations separately from final-answer correctness. It collects 1,186 trajectories from 15 systems on 80 tasks drawn from BFCL, GSM8K, HotpotQA, and StrategyQA. Selected key steps are scored on validity, task-grounded faithfulness, specificity, diagnostic utility, and evidence alignment. The paper combines outcome checks, human annotation and adjudication, controlled perturbations, and uncertainty-aware comparisons. Its main claim is that trajectory explanation quality provides information that final-answer accuracy alone does not capture.
Critical strengths
-
The paper addresses a useful evaluation problem. A correct final answer does not guarantee that its explanation is specific, supported, or useful for diagnosis. Separating these properties from task success can make evaluation more informative than a single accuracy score.
-
The study includes human-reference analyses, agreement measurements, controlled perturbations, and uncertainty estimates. These are relevant components of a measurement framework. The authors also explicitly distinguish task-grounded faithfulness from causal faithfulness and acknowledge limitations in generation provenance and fine-grained model ranking.
Critical weaknesses
-
The reported results are not internally consistent, so the final data version and aggregation pipeline are unclear. Table 8 and Figure 8 both report trajectory-balanced scores on a 0–100 scale, but several entries differ substantially. For Claude Opus 4.5, validity is 98.7 in Table 8 and 97.2 in Figure 8, while faithfulness is 98.9 and 96.9. Gemma 3 27B specificity is 58.0 and 54.5. These differences exceed rounding error. Figure 7 reports 18/20, 20/20, 20/20, and 20/20 correct answers for Opus, which gives 78/80 = 97.5%, not the 96.2% in Table 8. Its four source-level explanation scores average to 93.8, rather than the 95.0 in Table 8. These may reflect different annotation or processing versions, but no such distinction is documented. Until the figures and tables are reconciled, the empirical comparisons cannot be reliably interpreted.
-
The validity of the generated data and the scoring procedure is not sufficiently established for the central evaluation claims. Section 3.3 and the appendix acknowledge that validation retries can reveal gold answers or expected function names. There are 138 trajectories with one recorded validation retry and 80 with missing retry metadata; these counts do not establish exactly which outputs received gold information. Such outputs can contain post-hoc explanations and may change both correctness and its association with explanation quality. The disclosure is useful, but a uniformly gold-blind comparison is still missing. Scoring reliability is also a concern: Table 5 reports kappa = 0.074 for specificity and 0.226 for trajectory explanation quality. Label imbalance may affect kappa, but the uncertainty should be examined rather than resolved solely by adjudication. Moreover, the matched-human and full-hybrid explanation rankings have Spearman rho = -0.046. As the authors acknowledge, this comparison mixes task composition and score provenance. It therefore leaves unresolved whether the hybrid scores support reliable model comparisons. A comparison on the same tasks, trajectories, and weights is needed to isolate this issue. Finally, the trajectories are largely one-shot structured explanations, and BFCL tool calls are not executed. The claimed diagnostic utility is not validated through actual error localization or debugging outcomes.
Overall, I recommend clear rejection in the current form. My main concern is the reliability and interpretability of the evidence supporting the benchmark, rather than the motivation. The numerical inconsistencies and unresolved generation and annotation effects require substantial clarification and validation. These observations do not, by themselves, establish data fabrication.
Questions for authors
-
Clarification of conflicting results: Which data and annotation version was used for Table 8 and Figures 7–8? Please reconcile the Opus accuracy calculation (78/80 = 97.5% in Figure 7 versus 96.2% in Table 8), the five-dimensional score differences, and the source-level versus aggregate explanation scores. Can you provide the per-trajectory values, inclusion masks, and aggregation procedure that reproduce a single consistent set of results?
-
Validation of generation provenance: How many retained trajectories actually received a gold answer or an expected function name during retry, and how are they distributed across models and task sources? How are the 80 trajectories with missing retry metadata handled? Do the main findings hold on uniformly gold-blind first-attempt outputs or a clean rerun? Simply removing failed-and-retried cases would introduce selection bias, so please distinguish that analysis from a gold-blind evaluation.
-
Validation of the scoring instrument: On the same matched tasks and trajectories, with identical weights, how do the original hybrid labels compare with the adjudicated human labels? Please report dimension-level agreement, label distributions, and model-level score changes. In particular, how do the low agreement for specificity and trajectory explanation quality affect the conclusions? Can annotation uncertainty be incorporated into the reported comparisons, rather than only task-sampling uncertainty?
-
Clarification of construct validation and scope: What numerical invariance margin and exact both-annotator decision rule were fixed before the held-out perturbation test? Please provide paired ratings to make the zero-width intervals and 16/16 pass result auditable, and identify the reported equal-weight and step-weighted sensitivity results. Given that tool calls are not executed, what existing evidence connects the diagnostic-utility scores to successful error localization or debugging? If no such evidence is available, please delimit the claims to ratings of emitted explanations.