- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to fairly evaluate models on tasks that involve multiple reasoning stages. The authors point out that evaluating a later stage only on examples where all earlier stages were correct can be misleading, because different models may be evaluated on different and much smaller subsets of examples.
To address this issue, the paper proposes a framework that compares three types of accuracy: accuracy on all examples, accuracy on examples with correct earlier stages, and accuracy when the model is given correct answers for earlier stages. It also analyzes how selecting examples affects accuracy and the appropriate random-guessing baseline.
The authors evaluate five multimodal models on 4,100 synthetic visual reasoning tasks. They find that later-stage accuracy can reach nearly 99% when only earlier-stage successes are considered, even though these examples represent just 6–15% of the applicable dataset. They also show that different evaluation methods can change model rankings.
Critical strengths
Strenghts
-
The paper discusses an important issue in multi-stage reasoning evaluation, showing how conditional accuracy can be misleading due to reduced coverage and model-specific sample selection.
-
The proposed framework is well-structured, with clear mathematically definitions of different accuracy metrics, chance baselines, and selection gaps.
-
The experiments demonstrate how prefix conditioning affects coverage, accuracy, and model rankings. The additional analysis of FinRAG-QA further supports the practical relevance of the proposed framework.
Critical weaknesses
Weakness
-
The reanalysis of FinRAG-QA is a valuable addition that demonstrates the practical relevance of the proposed framework. However, the external validation is limited to a single benchmark, while the main experiments rely on a synthetic shape-grammar dataset, where the six stages are evaluated from a single JSON output rather than an interactive reasoning process. I encourage the authors to consider one or two additional benchmarks, such as WebArena or ALFWorld (or other suitable benchmarks of their choice), to further investigate coverage and selection gaps in realistic agent tasks. This would strengthen the evidence for the framework's generalizability.
-
The chance baselines are straightforward to compute in the synthetic setting, where the answer spaces and guessing distributions are predefined. However, their applicability to open-ended tasks remains unclear. For example, in software development tasks, how should one define the probability of an agent randomly generating a correct solution? Since the space of valid solutions is difficult to enumerate, it is unclear how the proposed chance baselines would generalize to such settings. Further discussion of this limitation or potential approaches to handling open-ended tasks would strengthen the paper.
Questions for authors
See weaknesses above. If the authors can adequately address the above two weaknesses, I would be inclined to support acceptance.
3 comments
Thank you for the response. I am inclined to maintain my original score.
Thank you for considering our response and the revised manuscript. If you have a remaining concern about either the ALFWorld evaluation or the reference policy for open-ended tasks, we would appreciate knowing which aspect needs further clarification.
ALFWorld Evaluation and Chance References for Open-Ended Tasks
Dear Reviewer,
Thank you very much for your insightful comments that helped us to substantially improve the manuscript.
1. New ALFWorld pipeline
Following your suggestion, we added ALFWorld as a second external benchmark alongside FinRAG-QA. Section 5.5 and Table 3 report coverage and fixed-oracle selection gaps in an interactive agent task, with the full protocol in Appendix I.
We evaluate GPT-5.6 Terra and Kimi K3 on all 134 unseen text games using four milestones: discovery, acquisition, processing, and terminal success. Actions are executed sequentially and change the environment encountered downstream. For each later stage, an expert prefix completes the preceding applicable milestones before handing control to the model. We retain natural-run survivor membership and score the same oracle outcomes across survivors, non-survivors, and all applicable games.
This also extends the external evidence beyond coverage accounting: we measure fixed-oracle selection gaps, natural–oracle differences, and executable reference-policy success. Both models perform near ceiling. Across both models and all four milestones, survivor coverage ranged from 92% to 100%. Fixed-oracle selection-gap estimates ranged from −0.35 to 0.00 percentage points. We present these as small, unresolved contrasts, rather than evidence that selection is generally absent. The framework measures population contrasts without requiring them to be large or positive.
Executable references further distinguish evaluation regimes: terminal random-policy success is 0.33% from initial states and 11.81% from expert checkpoints, while natural survivor checkpoints yield different references. We explicitly distinguish these state/history/budget regimes.
2. Defining chance references for open-ended tasks
We agree that defining chance baselines in open-ended tasks is more challenging than in our controlled synthetic setting, particularly because the space of valid solutions cannot generally be enumerated.
To address this concern, we have generalized the definition of chance success in Section 3.2 from the probability of guessing a designated ground-truth answer to the probability that a declared reference policy produces an accepted solution. Specifically, we define the reference success probability as:
bids=EY∼qids[Aid(Y)].
Here, qids is the output distribution of the declared reference policy under information regime s, and Aid(Y) indicates whether the task accepts output Y.The original finite-answer guessing baseline is a special case of this formulation. Importantly, the information- and population-change decomposition in Equations (2)–(3) remains valid without requiring enumeration of the solution space.
We have also made the following revisions:
Appendix D.2: We added a software-engineering example. For software repair, we specify a patch-generation policy, its permitted inputs and budget, and an acceptance criterion such as passing the prescribed tests. We estimate reference success by sampling patches and reporting the fraction accepted. This requires no enumeration of valid patches.
Related sections: We revised the Abstract, Introduction, Limitation, and Conclusion to reflect the generalized formulation.
We hope the new experiment and the explicit reference-policy definition address both concerns, and we appreciate your willingness to reconsider the paper in light of these revisions.