- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies how scoring rules affect tool-fault recovery evaluation on ToolMaze. It replays agent executions and re-scores stored trajectories under different continuation rules, examining changes in pass rates and the measured benefits of recovery prompts. It also compares definitions of task dependence and separately measures independent-task execution and failure reporting.
Critical strengths
None
Critical weaknesses
-
The paper offers limited novelty and does not establish a substantial contribution. The work largely consists of incremental modifications and analyses of ToolMaze’s existing scoring rules. Higher pass rates after relaxing the set of forbidden tools are largely a direct consequence of the rule change. The contribution beyond prior work on rescoring trajectories is unclear. The paper also provides no independent evidence showing which rule better measures recovery quality in practice. The reduction in the estimated benefit of recovery prompts does not replicate on the remaining 60 tasks. This limits the generality of the result.
-
The paper is poorly written and organized, making its central argument difficult to follow. A relatively straightforward question is obscured by numerous experimental conditions, dependence definitions, metrics, and repeated qualifications. Important explanations are scattered across the main text and lengthy appendices, with insufficient separation between the central findings and supporting analyses. Even after careful reading, it takes substantial effort to identify the paper’s actual contribution and determine which conclusions are adequately supported. The presentation substantially weakens the paper’s clarity and persuasiveness.
Questions for authors
What is the substantive new contribution beyond modifying and analyzing ToolMaze’s existing scoring rules? What evidence shows that this contribution generalizes to other evaluation settings and helps assess practical recovery quality more accurately?