- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper asks whether early agent failure prediction truly depends on the target run’s execution trace, rather than task-level information. It proposes a diagnostic that keeps the predictor and initial request fixed while replacing the process with another run from the same task and configuration. On 982 held-out τ²-Bench runs, AUROC is 0.6937 with the original process, 0.6919 with same-task replacement, and 0.6521 with cross-task replacement, with similar patterns across models and observation points. The SWE-Gym analysis further shows that process features can improve a tree model while a competitive initial-only model remains similarly strong. The main contribution is a diagnostic that separates predictive gains from genuine reliance on the target execution.
Critical strengths
The paper asks an important and under-examined evaluation question. It distinguishes between predicting whether an agent run will eventually fail and actually relying on the execution trace of that specific run. This is a meaningful distinction for agent monitoring.
Critical weaknesses
- Limited empirical coverage of the central claim. The main replacement experiment effectively evaluates only two predictor families—a tree-based model and sparse logistic regression—and does not include more explicitly trajectory-aware monitors. Moreover, the trace-replacement diagnostic is only replicated on one benchmark.
- Same-task swaps may retain outcome-related task structure. Because donor outcome is uncontrolled, near-unchanged performance after a same-task swap may partly reflect task-level outcome persistence rather than true insensitivity to execution-specific evidence.
Questions for authors
- Could the authors report the fraction of same-task donor pairs for which donor and target have the same final outcome, and, more importantly, evaluate the replacement diagnostic using only opposite-outcome same-task swaps within mixed-outcome task/configuration groups?
- The current replacement experiments cover tree-based models and sparse logistic regression. Have the authors considered applying the same diagnostic to monitors that explicitly incorporate trajectory-level risk signals or temporal dependencies?