acceptodds
Under review as a conference paper at ICLR 2027

Does Early Failure Prediction Actually Depend on Agent Execution?

Abstract

Early failure prediction aims to identify language model agent runs that are likely to fail. Yet a predictor can rank outcomes well without relying on the actions and feedback from the run being monitored. We test this distinction through task-matched process replacement. We keep the fitted predictor and the initial request fixed, then replace the visible process with one from another run of the same task and configuration. Replacements are chosen without using outcome labels. On 982 held-out-task runs from the public τ²-Bench archive, a tree predictor achieves an AUROC of 0.694 before the third action. Its mean AUROC is 0.692 when given another run's process from the same task, but falls to 0.652 when the process comes from a different task. Across three predictor settings and two early observation points, same-task replacement leaves mean AUROC close to or above the original value. Cross-task replacement lowers mean AUROC in all six comparisons. Same-task Brier score comparisons show no clear advantage for the original process. A separate analysis of 1,180 public SWE-Gym runs finds that adding process features improves a tree predictor, but an initial-only predictor achieves similar AUROC. These findings separate overall failure prediction from dependence on the current execution. Task-matched replacement provides a direct check of whether a predictor benefits from observing the particular run it is meant to monitor.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Language Model Agents, Agent Evaluation, Failure Prediction, Uncertainty and Risk Estimation, AI Reliability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1YweOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether early agent failure prediction truly depends on the target run’s execution trace, rather than task-level information. It proposes a diagnostic that keeps the predictor and initial request fixed while replacing the process with another run from the same task and configuration. On 982 held-out τ²-Bench runs, AUROC is 0.6937 with the original process, 0.6919 with same-task replacement, and 0.6521 with cross-task replacement, with similar patterns across models and observation points. The SWE-Gym analysis further shows that process features can improve a tree model while a competitive initial-only model remains similarly strong. The main contribution is a diagnostic that separates predictive gains from genuine reliance on the target execution.

Critical strengths

The paper asks an important and under-examined evaluation question. It distinguishes between predicting whether an agent run will eventually fail and actually relying on the execution trace of that specific run. This is a meaningful distinction for agent monitoring.

Critical weaknesses
  1. Limited empirical coverage of the central claim. The main replacement experiment effectively evaluates only two predictor families—a tree-based model and sparse logistic regression—and does not include more explicitly trajectory-aware monitors. Moreover, the trace-replacement diagnostic is only replicated on one benchmark.
  2. Same-task swaps may retain outcome-related task structure. Because donor outcome is uncontrolled, near-unchanged performance after a same-task swap may partly reflect task-level outcome persistence rather than true insensitivity to execution-specific evidence.
Questions for authors
  1. Could the authors report the fraction of same-task donor pairs for which donor and target have the same final outcome, and, more importantly, evaluate the replacement diagnostic using only opposite-outcome same-task swaps within mixed-outcome task/configuration groups?
  2. The current replacement experiments cover tree-based models and sparse logistic regression. Have the authors considered applying the same diagnostic to monitors that explicitly incorporate trajectory-level risk signals or temporal dependencies?
Reviewer_8hNJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tests whether agent failure predictors actually rely on the specific execution they. monitor.

To test this, they keep a trained predictor fixed and swaps in the execution record from a different run of the same task, then checks whether prediction quality drops

Their takeaway is that current predictors barely notice the swap, suggesting they mostly predict task difficulty rather than tracking how a particular run is goind

Critical strengths
  1. the question is important
  2. the diagnostic seems simple and useful
  3. the analysis is also careful and transparent.
Critical weaknesses
  1. I think my main concern is that the scope seems too narrow for the general claim. The replacement test runs on one archive with weak tfidf tree and lr predictors. the resutl may simply show that these models cannot use execution details, not that agent failure prediciton in general does not need them. Things like LLM based monitor are also not tested
  2. additionally, I think currently the observation points are very early/. before the thrid or fifth action, runs of the same task can be nearly the same, so a swap changes might be very minimal. the paper does not measure how different donor prefixes are, or test later points where execution diverges more
  3. the swap result is largely expected from Table 6: within-task concordance is near 0.5, so the predictors already cannot tell apart runs of the same task, and swapping in another run of that task should not change much.
  4. the swe gym part feel a bit disconnected/. it cannot run the replacement test, the data are very imbalanced, and most of the extra alarms come from empty patches. this reads like a separate thing rather than support for the main claim
  5. the paper currently is a bit hard to read. the main message is buried under heavy hegdging and many side comparisons. Appendix c's exploratory resuls seems to be mixed. The validation set was also reused from earlier exploratory work, so none of thi sis confirmatory?
Questions for authors

See above. Happy to discuss in rebuttal:)

Reviewer_N7DdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks at early failure predictions for agents to determine whether agent execution traces provide any improvement in prediction scores. Multiple experiments are performed for this - baselines without execution, and swapping out other execution traces to check the effect on prediction accuracy (referred to as task-matched process replacement). Same task-replacement gets comparable AUROCs, while cross-task replacement does worse, showing that failure prediction does not depend much on the actual execution of the task and survives other executions or same-task trace swaps.

Critical strengths

This paper addresses an important problem of failure prediction in agents - detecting failures early on is critical to ensure that we do not waste time and tokens on paths that will lead to undesirable outcomes. Experiments are well motivated - the paper compares baselines against swapped executions of the same task and different tasks on multiple datasets and measures AUROC for prediction.

Critical weaknesses

My main concern with this paper is the limited novelty - earlier work such as " "Automata from Agent Traces: Failure and Next-Step Prediction" - Cho et al. shows that the task harness is predictive of agent failures, which is similar to the result presented here.

The paper reads as AI generated due to the use of overly complex sentences. The different experimental configurations need to be explained more clearly to make the paper easier to read.

Questions for authors

The result on diagnosing early failure prediction is interesting but could benefit from deeper analysis of where failures occur for different types of tasks. This is also important for improving agents/systems/harnesses. Could you suggest what analysis you could perform on the experiments you have conducted to get more insights on this?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.