- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper distinguishes policy-relative rollout success from bounded recoverability in tool agents. It argues that failed rollouts only indicate failure under the current continuation policy and should not be directly interpreted as evidence that a state-action pair is irrecoverable. The paper proposes executable witness/certificate/abstain labeling, where a positive label is supported by a successful continuation and a negative label requires exhaustive bounded evidence that no successful continuation exists. Experiments on tool-agent environments show that rollout-based negative labels can contain recoverable cases and that exact relabeling improves recoverability probability estimation.
Critical strengths
The paper identifies a clear mismatch between policy-relative value and recoverability, and provides executable experiments showing that failed rollouts can still correspond to recoverable states.
Critical weaknesses
The paper does not clearly justify why bounded recoverability should be a better supervision target than policy-relative success. From the RL perspective, policy-relative success corresponds to the Q-function of the current policy, while bounded recoverability is closer to the optimal Q-function. Estimating the optimal Q-function is much harder and can be infeasible in realistic environments. The experiments show that failed rollouts can be recoverable, but this only demonstrates the difference between these two targets. They do not show that recoverability is a better target for training the policy, as the improvement is mainly in recoverability log loss, with no significant improvement in action ranking or terminal success.
Exact negative labels require exhaustive search over the continuation space. This is feasible in the main experiments because the action inventory is restricted to at most five occurrences. Moreover, the rollout-budget experiment shows that simply increasing the number of rollouts to K=4/8 can match the Top-1 performance and even achieve lower recoverability log loss than Exact supervision, weakening the practical motivation for expensive certification.
Questions for authors
The proposed recoverability supervision introduces additional certification cost, while the experiments show that increasing the rollout budget to K=4/8 can already match or outperform Exact supervision on several metrics. Could the authors provide a direct cost-performance comparison between additional rollouts and exact recoverability certification, including label-generation compute and downstream performance?
The main Retail experiments use a highly restricted candidate space that already contains the reference solution actions. How representative is this setting of realistic tool agents, where the correct continuation may not be included in a predefined candidate set?
How sensitive are the conclusions to the definition of the continuation family and budget?