- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes DynSTEER, a dynamic stage-wise framework for evaluating LLM agent trajectories. Instead of evaluating entire trajectories or individual steps, DynSTEER divides agent executions into meaningful stages using automatically generated milestone graphs. It dynamically adjusts evaluation dimensions and judge levels based on stage-level performance and uncertainty. Additionally, it introduces an early-stopping mechanism to terminate potentially unrecoverable executions and reduce computational costs. Experiments on ToolSandbox and SWE-bench Pro show that DynSTEER improves model discriminability by over 85% compared to whole-trajectory evaluation while reducing execution steps.
Critical strengths
-
Novel Stage-wise Evaluation Framework. The paper proposes a stage-wise trajectory evaluation framework that addresses the limitations of both whole-trajectory and step-level evaluation. The combination of milestone graphs and adaptive multi-level judges is an interesting approach to evaluating agent execution quality.
-
Improved Evaluation Efficiency. The proposed early-stopping mechanism can identify potentially unrecoverable trajectories and reduce unnecessary execution steps. The experimental results demonstrate promising improvements in model discriminability and computational efficiency across two benchmarks.
Critical weaknesses
-
Lack of Ground-Truth Validation for Evaluation Quality. Although the authors introduce several evaluation metrics, I believe that metrics such as model discriminability, which measure how well the proposed method separates the scores of different agents, do not necessarily demonstrate that the evaluation method is better or more accurate. The authors should construct a small-scale benchmark with ground-truth annotations to directly validate the effectiveness of their evaluation method.
-
Unclear Practical Value of Trajectory Scoring. The authors do not clearly explain the practical significance of assigning scores to agent trajectories. For example, are these scores intended for data filtering or as reward signals for reinforcement learning? I believe the authors should demonstrate the effectiveness of the proposed scoring method on at least one downstream post-training task.
Questions for authors
Mentioned in Critical Weaknesses.
Response to Reviewer DdKN
Thank you for your helpful comments. Our responses to your questions are as follows.
Response to Q1
Ground-truth validation of a trajectory evaluator is difficult because the evaluation community does not yet provide an established independent meta-evaluation oracle for open-ended agent trajectories. The paper therefore uses the systematic separation criterion DS, following Benchmark²: Systematic Evaluation of LLM Benchmarks (Qian et al., arXiv:2601.03986, 2026). DS is defined at Line 339–348, Page 7, and combines score dispersion with the fraction of model pairs separated beyond a margin.
The controlled evidence is the dynamic-component ablation in Table 3 and Section 4.3, where source trajectories, milestone graphs, minefield checks, and models are held fixed (Line 659–661 and Line 693, Page 13). DS rises from 0.0228 for WHOLE-TRAJ to 0.0311 for
-w/o dynamic, 0.0439 with dynamic judge level removed, 0.0414 with dynamic weighting removed, and 0.0528 for full DynSTEER (Line 380–386 and Line 405–425, Page 8). This supports the claimed gain in systematic model discrimination rather than asserting human-quality accuracy.Response to Q2
The intended use of DynSTEER scores is evaluation-centric: auditable process diagnosis, failure localization, safety analysis, and model comparison. These uses are directly tied to the paper’s evaluation objective and do not require changing the learning objective of an agent.
Using the scores as filtering rewards or post-training objectives is a different research problem. It requires a dedicated training protocol, preference-pair construction policy, optimization analysis, and safety controls. Such a study is outside the scope of evidence for an evaluation paper. The appropriate claim is that DynSTEER supplies auditable process-level signals for evaluation and diagnosis, not that it has been validated as a downstream training reward.