- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a method for estimating LLM agent success from execution trajectories. It combines semantic step encoding, GRU-based temporal modeling, and multi-prefix supervision for early prediction. Experiments on three benchmarks show improved within-task AUROC over baselines.
Critical strengths
-
The motivation is reasonable and clear.
-
Experiments are comprehensive, the authors conducted evaluations on three different environments and provided ablation studies, experiments comparing different acting models, and controlled experiments examining the effects of task difficulty and trajectory length.
Critical weaknesses
-
STEP is introduced to provide success probabilities that can guide intervention, but the reliability in real-world settings remains unclear. The datasets only include tasks with both successful and failed attempts, which may not reflect the actual outcome distribution. In Table 8, STEP also shows worse calibration metrics than HTC on WebShop, and the additional acting-model experiments retrain STEP for each model, leaving its generalization to new agents untested.
-
Early prediction evaluation includes trajectories that have already finished before the specified prefix budget, these cases may affect the mean prefix AUROC.
Questions for authors
-
Given that the datasets only include tasks with both successful and failed attempts, how reliable would STEP's predicted probabilities be under the original task distribution?
-
What proportion of trajectories have already finished at each prefix budget? Could the authors report early prediction performance using only trajectories that are still running?
-
Since STEP is retrained separately for each acting model, does its application to a new agent always require additional labeled trajectories?
2 comments
Thanks for the authors' response. The results address most of my concerns, so I decide to keep my score.
Response to Reviewer cGfD
Thank you for the feedback on probability reliability, transfer, and ongoing evaluation.
Reply to W1 (outcome selection) & Q1 — Probabilities on unfiltered tasks
Mixed-outcome selection supports within-task comparisons. We evaluated unchanged predictors on 200 unfiltered tasks per environment: 1,000 ScienceWorld, 1,000 WebShop, and 600 InterCode Bash attempts, with success rates of 32.9%, 37.0%, and 31.3%. There was no target retraining, selection, or calibration; test tasks were excluded from their scoring model's training and validation. The table reports Full probability quality.
STEP has lower Brier, NLL, and ECE than HTC in ScienceWorld and InterCode Bash. These predictors are frozen; the separate task-prior experiment instead refits predictors. Revised Appendix H.1 (Table 23) specifies the protocol.
Reply to W1 (WebShop calibration) — Probability quality
For the main cohorts (submitted Table 8; revised Table 9, Appendix C.7), HTC has lower Brier/NLL/ECE in WebShop; STEP has lower Brier/NLL in ScienceWorld and InterCode Bash. Section 4.4 discusses these results.
Reply to W1 (new agents) & Q3 — Transfer without target labels
Not necessarily. We tested 12 cross-agent transfers in ScienceWorld with frozen source predictors and no target fitting. Test tasks were excluded from source training and validation. Each collection has 100 tasks and 500 attempts. Cells give STEP / HTC Full within-task AUROC; STEP averages five source-training seeds.
NA means unavailable token probabilities. STEP matches or exceeds HTC in all six comparable transfers (revised Section 4.2.3; Appendix F.1, Table 21).
Reply to W2 & Q2 — Predictions on ongoing executions
Using the first min(K,T) records keeps the same attempts at every budget. Restricting to T>K changes the eligible attempts and tasks as K grows, so AUROC changes can reflect population changes as well as additional execution evidence.
Using the original models, we report ongoing-only InterCode Bash results. “Finished” is the share with T≤K. AUROC uses only T>K attempts from tasks retaining both outcomes (“Tasks”); all methods use the same eligible attempts.
InterCode Bash (195 attempts)
Over K=1–9, STEP has the highest mean AUROC (.627), followed by HTC (.549). No attempts remain ongoing at K=10, so AUROC is unavailable.
See revised Section 4.2.2 and Appendix B.2 (Table 5).