acceptodds
Under review as a conference paper at ICLR 2027

What Does a Failed Rollout Supervise? Separating Policy Failure from Recoverability in Tool Agents

Abstract

When a tool-using agent fails to complete a task, does that mean its earlier actions left no way to succeed? A failed continuation may reflect the limitations of the policy rather than an unrecoverable state. This distinction matters when sampled outcomes are used to label intermediate actions for training. We develop an executable audit that separates policy-relative success from recoverability within a specified continuation family and budget. Successful continuations justify positive recoverability labels, complete bounded non-reachability certificates justify negative recoverability labels, and unresolved cases remain unlabeled. Exhaustive replay over finite action inventories under fixed continuation bounds supplies exact audit labels in small deterministic environments. Our analysis distinguishes population identifiability from finite-sample certification and connects target mismatch to probability loss and representation-dependent ranking. Audits of released supervision data and six target builders show why sampled failure and native negative labels cannot uniformly be interpreted as irrecoverability. In a three-seed, four-arm Retail intervention with restricted candidates and matched training budgets, exact relabeling reduces held-out recoverability log loss by 0.150 relative to forced one-rollout labeling, while certificate-based abstention reduces it by 0.050 relative to matched deletion. Task-held-out monotone calibration of all arms, using exact labels from other tasks, narrows these gaps to 0.024 and 0.011 while preserving within-task rankings. Effects on action ranking and terminal success remain inconclusive. These findings motivate target-explicit supervision and separate evaluation of probability fit, action ranking, and terminal task success.

Primary area:
foundation or frontier models, including LLMs
Keywords:
process reward models, tool-using agents, supervision targets, recoverability, probability calibration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uN6sOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper distinguishes policy-relative rollout success from bounded recoverability in tool agents. It argues that failed rollouts only indicate failure under the current continuation policy and should not be directly interpreted as evidence that a state-action pair is irrecoverable. The paper proposes executable witness/certificate/abstain labeling, where a positive label is supported by a successful continuation and a negative label requires exhaustive bounded evidence that no successful continuation exists. Experiments on tool-agent environments show that rollout-based negative labels can contain recoverable cases and that exact relabeling improves recoverability probability estimation.

Critical strengths

The paper identifies a clear mismatch between policy-relative value and recoverability, and provides executable experiments showing that failed rollouts can still correspond to recoverable states.

Critical weaknesses

The paper does not clearly justify why bounded recoverability should be a better supervision target than policy-relative success. From the RL perspective, policy-relative success corresponds to the Q-function of the current policy, while bounded recoverability is closer to the optimal Q-function. Estimating the optimal Q-function is much harder and can be infeasible in realistic environments. The experiments show that failed rollouts can be recoverable, but this only demonstrates the difference between these two targets. They do not show that recoverability is a better target for training the policy, as the improvement is mainly in recoverability log loss, with no significant improvement in action ranking or terminal success.

Exact negative labels require exhaustive search over the continuation space. This is feasible in the main experiments because the action inventory is restricted to at most five occurrences. Moreover, the rollout-budget experiment shows that simply increasing the number of rollouts to K=4/8 can match the Top-1 performance and even achieve lower recoverability log loss than Exact supervision, weakening the practical motivation for expensive certification.

Questions for authors

The proposed recoverability supervision introduces additional certification cost, while the experiments show that increasing the rollout budget to K=4/8 can already match or outperform Exact supervision on several metrics. Could the authors provide a direct cost-performance comparison between additional rollouts and exact recoverability certification, including label-generation compute and downstream performance?

The main Retail experiments use a highly restricted candidate space that already contains the reference solution actions. How representative is this setting of realistic tool agents, where the correct continuation may not be included in a predefined candidate set?

How sensitive are the conclusions to the definition of the continuation family and budget?

Reviewer_Qp4QOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The proposed work puts forth an audit methodology which separates policy-relative success from recoverability in tool-use agentic settings. The authors turn the distinction between policy failure and bounded recoverability into an executable supervision rule based on witnesses, certificates, and abstention.

Critical strengths
  1. the experimental setup is well explained and easy to follow.
  2. the controlled four-arm learning intervention is clever, as its finite candidate menus contain all official mutating reference calls plus at most two deterministically ordered train-only donor calls
  3. distinguishing between per-attempt success, positive-edge miss rate, and negative-pool contamination is welcome and the explanation of such is descriptive.
  4. the theoretical investigation for what failed rollouts establish is sound and explains what the proposed method captures and what it does not under this framework
Critical weaknesses
  1. it would be good if the authors provided more insight into the justification for use of the retail domain in their main experimental methodology.
  2. what was the rationale behind the choice of generative models for the experiments ?
Questions for authors

See above.

Reviewer_qZFfOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

When training AI agents, a common shortcut is to judge a step by retrying the rest of the task several times from that point. This is pretty standard and often when every retry fails, the step is labeled as a mistake. This paper points out that the shortcut confuses “the agent couldn’t find a way forward” with “no way forward exists.” In small settings it checks every possibility to find the true answer.

The main problem is that training on the more accurate labels changes very little in practice. The improvement mostly comes from the model’s predictions being shifted up or down overall, and a simple correction removes nearly all of the gap. The agent picks the same actions and succeeds just as often either way. In fact, in the authors’ extra experiments, simply retrying a few more times worked as well or better.

Critical strengths

The idea that you should judge a model separately on how good its probability estimates are, how well it ranks options, and whether it finishes tasks is actually quite good.

Critical weaknesses

Training on the more accurate labels changes very little in practice. The improvement mostly comes from the model’s predictions being shifted up or down overall, and a simple correction removes nearly all of the gap. The agent picks the same actions and succeeds just as often either way. The novelty is also limited. That “all retries failed” doesn’t mean “no way forward exists” is already well understood, and the math is standard.

Questions for authors
  1. Once the overall shift in predictions is corrected, how much of the improvement remains?
  2. How much does it cost to produce the exact labels, compared with simply retrying four or eight times?
  3. Given that cheap retries did as well in your own experiments, what would you recommend people in practice actually do?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.