- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
BLADE ie a tool that provides a boundary judge to determine the boundary to end the rollout when an agent makes a mistake. BLADE-based RL saves both compute and improve performance on selected benchmarks.
Critical strengths
BLADE appears to be a simple and practical addition to the current multi-turn RL pipeline that gates the rollout in case the agent makes a mistake: it saves compute by cutting the rollout short, and it provides cleaner signal for the GRPO blame to land: so the "rootcause" gets the blame, not all the subsequent actions that may be correct.
Critical weaknesses
- I have a concern on how general the approach could be: under this scheme, the agent would never be rewarded for self-correction. For example, if it made a mistake, and the user corrects it or asks it to try again, and then later it corrects itself, "you are right, I was wrong..." - these types of trajectories seem equally important for the agent.
- Even with training, agent can still make a mistake. At inference time when a mistake happens, the agent is never trained to recover. What would happen in this case?
- What happens if there is no objectively right/wrong signal midpoint? E.g., what if the agent didn't do something and the outcome is unknown, that will leave the judgement purely to the judge, which itself can be wrong.
Questions for authors
See the weakness section. I wonder if broader eval would dispute these concerns or if there's specific reasons why the current solution doesn't make the model lose ability to reflect or generalize.
2 comments
Response to Reviewer Hs2W (2/2)
Thank you also for raising the question of how the method applies when midpoint evidence is ambiguous.
3. Midpoint uncertainty and incorrect stopping.
A final task outcome need not be known to check some consequential errors: for example, an incorrect claim about a task rule or a database update inconsistent with the request can be assessed from the realized prefix and current state. Uncertainty about eventual success is not, by itself, evidence that an invalid action occurred. CONTINUE is not a correctness certificate and gives no intermediate reward. When the available evidence cannot support the boundary judgment, the method's applicability is correspondingly limited.
Independent human assessment of 100 STOP prefixes from different tasks found 82 supported consequential errors, 12 unsupported decisions and 6 indeterminate cases. The unsupported-STOP share is 12.0% (Wilson 95% CI: 7.0–19.8%), conditional on STOP rather than a conventional false-positive rate.
Continuing all 100 stopped branches four times with the training-time collector checkpoint that triggered STOP, frozen for continuation, and Judge disabled yielded 21.00% original-task success (84/400; prefix-cluster bootstrap 95% CI: 16.0–26.0%). Even among supported errors, 14.63% of continuations succeeded. Thus, a true error can still be recoverable: early failure is a training choice, not a claim of irreversibility. These findings motivate explicit prevention-versus-recovery qualifications, rather than a claim that stopping universally preserves reflection or generalization. Higher end-to-end pass^4 alone does not demonstrate preserved reflection or conditional recovery ability. The existing held-out-domain results concern transfer to those domains, not a general test of reflection.
Why local error evidence and terminal success must be separated.
A false claim about a current task rule or an incorrect realized tool effect can be supported by boundary-visible evidence without establishing the final outcome. For example, the address-change example concerns whether a rule was represented correctly, not a forecast of whether all possible subsequent dialogues would fail. This is why supported errors can coexist with successful continuations.
The audit separately retains indeterminate cases rather than treating every uncertainty as evidence of error. Its supported/unsupported/indeterminate categories summarize the sampled STOP boundaries. They do not measure the rate of missed errors among CONTINUE decisions or provide a calibrated probability of eventual success. For open-ended settings where comparable boundary evidence is unavailable, the applicability of this stopping criterion remains limited.
Why the collector and final-model experiments are complementary.
The continuation audit asks what the student used during training could have achieved had collection continued at the time it triggered STOP. The final-model comparison instead asks how the completed training methods behave from a common set of recoverable error states. The 60-prefix recovery set is selected from supported errors, while the audit also includes unsupported and indeterminate boundaries. Both the policy and prefix distribution therefore differ between these evaluations.
Accordingly, the 14.63% success among supported-error continuations and the final BLADE recovery rate of 36.67% cannot be subtracted to estimate training improvement. The valid final-model comparison is BLADE versus OPD or joint OPD+GRPO within the same 60-prefix experiment. Its scope and interpretation are discussed in the first reply.
Finally, the audit does not prove that early failure is the best reward design or that training is insensitive to erroneous STOP decisions. The observed successful continuations show why those should be acknowledged as modeling trade-offs, even when the overall training recipe improves repeated-run reliability.
Thank you for these thoughtful questions; they help distinguish the settings supported by our evidence from broader claims about reflection and generalization.
Response to Reviewer Hs2W (1/2)
Thank you for recognizing the practical motivation of controlling rollout cost and localizing responsibility for errors. We agree that self-correction is important, and distinguish error prevention from recovery below.
1. Does the method eliminate self-correction?
BLADE checks after a completed nonterminal interaction loop and before the next user message, rather than stopping at every token or tool call. This timing does not itself prohibit correction within a loop, although acceptance depends on the stopping criterion. Once a branch is stopped, however, a later user-correction dialogue is not collected. We acknowledge this loss of recovery coverage rather than claiming that such trajectories have no value.
The error-producing action remains in the training data. STOP ends further collection and assigns the early-failure reward; OPD supervision remains on the collected prefix, while GRPO credit is restricted to the error-onset loop. The goal is to discourage the consequential error without assigning the same task-failure signal to all preceding behavior.
Figure 1 illustrates the motivation: a false statement about address-change eligibility redirects the user toward cancellation. Subsequent actions can be locally reasonable for the changed request without fulfilling the original goal. This motivates the intervention but does not imply that all erroneous branches are irrecoverable.
2. What happens after an inference-time error?
The teacher and Boundary Judge are absent at evaluation. The student continues under the normal benchmark rules, including any opportunities for user correction or recovery. Thus, evaluation is not helped by an external gate that removes difficult states.
To test whether this comes at a cost in recovery ability, we evaluated a shared set of 60 evidence-supported, reversibly erroneous prefixes drawn from an independent audit. These came from different original tasks and were selected using tool state, rules and reversibility before observing any evaluated policy's outcome. Unsupported and indeterminate STOP cases were excluded. The final trained OPD, joint OPD+GRPO and BLADE checkpoints each received the same history, environment snapshot, simulator goal/context and original-task evaluator, with the Judge disabled and four trials per prefix.
BLADE minus joint is -1.25 percentage points. We do not interpret this as improved recovery or proof of no degradation. The set covers selected reversible errors from the collector distribution. It provides a direct but limited recovery diagnostic, alongside the main end-to-end reliability evaluation.
How to interpret the shared-prefix result.
The recovery comparison holds the erroneous history and environment state fixed across final models. It therefore avoids comparing only the different failures each model happens to make during unconstrained task starts. Selection was based on evidence of a real, reversible error before observing continuation outcomes, not on whether BLADE subsequently succeeded.
The result addresses a narrower question than general reflection: whether final models can complete the original task from these selected erroneous states. These point estimates do not establish that BLADE has learned a general repair strategy or demonstrate recovery non-inferiority. The original held-out-domain results concern transfer to After-sales and Telecom, not an independent measurement of reflection. We keep these claims separate.
Our evidence therefore supports end-to-end reliability in the reported environments and provides a direct, limited recovery diagnostic. It does not justify describing prevention as a replacement for all forms of recovery training.
The judge's applicability in ambiguous-evidence settings is discussed in the following reply. Thank you for highlighting the importance of self-correction and recovery.