acceptodds
Under review as a conference paper at ICLR 2027

Controlling Interaction Drift: Optimizing LLM Agents for Multi-Turn User Interaction

Abstract

Large language models are increasingly post-trained as multi-turn agents for tool-mediated customer service, such as e-commerce assistance. In such settings, rollouts are generated online through a branching interaction process in which agent actions, user responses, and environment updates jointly determine what happens next. An erroneous response or tool action can therefore redirect subsequent interactions onto a different branch. We call this error-driven redirection *interaction drift*. This creates a collection-time decision: whether an error-conditioned branch should continue generating interactions for policy optimization. On-policy distillation (OPD) provides fine-grained teacher supervision in student-visited contexts, whereas terminal-reward reinforcement learning provides task-level feedback only after rollout completion. Neither signal determines whether another interaction should be collected from such a branch or how task credit should be localized within the trajectory. We introduce BLADE (Boundary-Localized Advantage for Drifted Episodes), a post-training framework with a frozen, environment-grounded Boundary Judge. After each completed nonterminal interaction loop and before the next simulated-user message, the Judge either continues collection or stops the branch at the loop in which the error was introduced. Teacher supervision is retained over the full collected rollout in either case, while the stopped-rollout task signal is confined to that loop; naturally completed rollouts retain trajectory-level task credit. Experiments across two stateful interaction benchmarks and two teacher–student configurations, including held-out-domain evaluation, show consistent improvements in strict repeated-run reliability over strong post-training baselines. These results suggest that controlling interaction drift during collection can improve the consistency of successful multi-turn behavior across model scales and evaluation settings.

Primary area:
reinforcement learning
Keywords:
Large language model agents, multi-turn user interaction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Hs2WOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

BLADE ie a tool that provides a boundary judge to determine the boundary to end the rollout when an agent makes a mistake. BLADE-based RL saves both compute and improve performance on selected benchmarks.

Critical strengths

BLADE appears to be a simple and practical addition to the current multi-turn RL pipeline that gates the rollout in case the agent makes a mistake: it saves compute by cutting the rollout short, and it provides cleaner signal for the GRPO blame to land: so the "rootcause" gets the blame, not all the subsequent actions that may be correct.

Critical weaknesses
  • I have a concern on how general the approach could be: under this scheme, the agent would never be rewarded for self-correction. For example, if it made a mistake, and the user corrects it or asks it to try again, and then later it corrects itself, "you are right, I was wrong..." - these types of trajectories seem equally important for the agent.
  • Even with training, agent can still make a mistake. At inference time when a mistake happens, the agent is never trained to recover. What would happen in this case?
  • What happens if there is no objectively right/wrong signal midpoint? E.g., what if the agent didn't do something and the outcome is unknown, that will leave the judgement purely to the judge, which itself can be wrong.
Questions for authors

See the weakness section. I wonder if broader eval would dispute these concerns or if there's specific reasons why the current solution doesn't make the model lose ability to reflect or generalize.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Hs2W (1/2)

Thank you for recognizing the practical motivation of controlling rollout cost and localizing responsibility for errors. We agree that self-correction is important, and distinguish error prevention from recovery below.

1. Does the method eliminate self-correction?

BLADE checks after a completed nonterminal interaction loop and before the next user message, rather than stopping at every token or tool call. This timing does not itself prohibit correction within a loop, although acceptance depends on the stopping criterion. Once a branch is stopped, however, a later user-correction dialogue is not collected. We acknowledge this loss of recovery coverage rather than claiming that such trajectories have no value.

The error-producing action remains in the training data. STOP ends further collection and assigns the early-failure reward; OPD supervision remains on the collected prefix, while GRPO credit is restricted to the error-onset loop. The goal is to discourage the consequential error without assigning the same task-failure signal to all preceding behavior.

Figure 1 illustrates the motivation: a false statement about address-change eligibility redirects the user toward cancellation. Subsequent actions can be locally reasonable for the changed request without fulfilling the original goal. This motivates the intervention but does not imply that all erroneous branches are irrecoverable.

2. What happens after an inference-time error?

The teacher and Boundary Judge are absent at evaluation. The student continues under the normal benchmark rules, including any opportunities for user correction or recovery. Thus, evaluation is not helped by an external gate that removes difficult states.

To test whether this comes at a cost in recovery ability, we evaluated a shared set of 60 evidence-supported, reversibly erroneous prefixes drawn from an independent audit. These came from different original tasks and were selected using tool state, rules and reversibility before observing any evaluated policy's outcome. Unsupported and indeterminate STOP cases were excluded. The final trained OPD, joint OPD+GRPO and BLADE checkpoints each received the same history, environment snapshot, simulator goal/context and original-task evaluator, with the Judge disabled and four trials per prefix.

PolicyConditional original-task success
OPD83/240 = 34.58%
Joint OPD+GRPO91/240 = 37.92%
BLADE88/240 = 36.67%

BLADE minus joint is -1.25 percentage points. We do not interpret this as improved recovery or proof of no degradation. The set covers selected reversible errors from the collector distribution. It provides a direct but limited recovery diagnostic, alongside the main end-to-end reliability evaluation.

How to interpret the shared-prefix result.

The recovery comparison holds the erroneous history and environment state fixed across final models. It therefore avoids comparing only the different failures each model happens to make during unconstrained task starts. Selection was based on evidence of a real, reversible error before observing continuation outcomes, not on whether BLADE subsequently succeeded.

The result addresses a narrower question than general reflection: whether final models can complete the original task from these selected erroneous states. These point estimates do not establish that BLADE has learned a general repair strategy or demonstrate recovery non-inferiority. The original held-out-domain results concern transfer to After-sales and Telecom, not an independent measurement of reflection. We keep these claims separate.

Our evidence therefore supports end-to-end reliability in the reported environments and provides a direct, limited recovery diagnostic. It does not justify describing prevention as a replacement for all forms of recovery training.

The judge's applicability in ambiguous-evidence settings is discussed in the following reply. Thank you for highlighting the importance of self-correction and recovery.

AuthorsOpenReview ↗

Response to Reviewer Hs2W (2/2)

Thank you also for raising the question of how the method applies when midpoint evidence is ambiguous.

3. Midpoint uncertainty and incorrect stopping.

A final task outcome need not be known to check some consequential errors: for example, an incorrect claim about a task rule or a database update inconsistent with the request can be assessed from the realized prefix and current state. Uncertainty about eventual success is not, by itself, evidence that an invalid action occurred. CONTINUE is not a correctness certificate and gives no intermediate reward. When the available evidence cannot support the boundary judgment, the method's applicability is correspondingly limited.

Independent human assessment of 100 STOP prefixes from different tasks found 82 supported consequential errors, 12 unsupported decisions and 6 indeterminate cases. The unsupported-STOP share is 12.0% (Wilson 95% CI: 7.0–19.8%), conditional on STOP rather than a conventional false-positive rate.

Continuing all 100 stopped branches four times with the training-time collector checkpoint that triggered STOP, frozen for continuation, and Judge disabled yielded 21.00% original-task success (84/400; prefix-cluster bootstrap 95% CI: 16.0–26.0%). Even among supported errors, 14.63% of continuations succeeded. Thus, a true error can still be recoverable: early failure is a training choice, not a claim of irreversibility. These findings motivate explicit prevention-versus-recovery qualifications, rather than a claim that stopping universally preserves reflection or generalization. Higher end-to-end pass^4 alone does not demonstrate preserved reflection or conditional recovery ability. The existing held-out-domain results concern transfer to those domains, not a general test of reflection.

Why local error evidence and terminal success must be separated.

A false claim about a current task rule or an incorrect realized tool effect can be supported by boundary-visible evidence without establishing the final outcome. For example, the address-change example concerns whether a rule was represented correctly, not a forecast of whether all possible subsequent dialogues would fail. This is why supported errors can coexist with successful continuations.

The audit separately retains indeterminate cases rather than treating every uncertainty as evidence of error. Its supported/unsupported/indeterminate categories summarize the sampled STOP boundaries. They do not measure the rate of missed errors among CONTINUE decisions or provide a calibrated probability of eventual success. For open-ended settings where comparable boundary evidence is unavailable, the applicability of this stopping criterion remains limited.

Why the collector and final-model experiments are complementary.

The continuation audit asks what the student used during training could have achieved had collection continued at the time it triggered STOP. The final-model comparison instead asks how the completed training methods behave from a common set of recoverable error states. The 60-prefix recovery set is selected from supported errors, while the audit also includes unsupported and indeterminate boundaries. Both the policy and prefix distribution therefore differ between these evaluations.

Accordingly, the 14.63% success among supported-error continuations and the final BLADE recovery rate of 36.67% cannot be subtracted to estimate training improvement. The valid final-model comparison is BLADE versus OPD or joint OPD+GRPO within the same 60-prefix experiment. Its scope and interpretation are discussed in the first reply.

Finally, the audit does not prove that early failure is the best reward design or that training is insensitive to erroneous STOP decisions. The observed successful continuations show why those should be acknowledged as modeling trade-offs, even when the overall training recipe improves repeated-run reliability.

Thank you for these thoughtful questions; they help distinguish the settings supported by our evidence from broader claims about reflection and generalization.

Reviewer_VNgNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BLADE, a framework with a frozen environment-grounded boundary judge. This work targets the waste in training cost and drifted training signal, and proposes to avoid them by introducing a judge to call stop point for student rollout under OPD training. They also restrict the rewards and credits allocation such that subsequent rollout loops of wrong response receive no rewards and do not contribute to the training. Experiments demonstrate the effectiveness of the proposed method on 2 OPD settings and two benchmarks.

Critical strengths
  1. This paper investigates a very common situation where OPD involves drifted training signals, and proposes a method that avoids both the computational waste and misleading training signals.

  2. The results show consistent gains, and the advantage of the proposed method particularly stands out when the OPD is applied between models with substantial size differences.

Critical weaknesses
  1. My main concern is the extent of the novelty. BLADE combines online rollout truncation with the joint usage of OPD and GRPO. Closely related works include STRIDE, which stops multi-turn OPD rollouts based on teacher scores, and CM2, which uses judge-based termination before launching the next reference user query during GRPO training. Though rollout-control works are discussed in the related-work section, the relationship to closely related online stopping works could be clearer. Combining OPD and GRPO advantages is used in ATOD, which the paper acknowledges. The contribution appears to be the combination of online stopping and joint usage of OPD and GRPO. A clearer discussion of this distinction would help clarify the novelty and contribution.

  2. Another issue raising concerns is the experiment setup. The paper uses two major components, namely online stopping and joint policy optimization. Table 1 compares BLADE against GRPO-only and OPD-based methods but does not include an existing joint optimization work such as ATOD, despite acknowledging it in the related-work section. Though the ablation study covers the online stopping, comparisons with previous online stopping works are not included in Table 1.

Questions for authors
  1. Could the authors share a clearer discussion of prior work on online stopping and the joint usage of OPD and GRPO?
  2. Could the authors provide more evidence to demonstrate the advantage of the proposed method compared to other online stopping works and ATOD? Or could the authors confirm the effectiveness by extending the ablation study?
2 comments
AuthorsOpenReview ↗

Response to Reviewer VNgN (1/2)

Thank you for recognizing the practical importance of drifted training signals and the consistent improvements. We agree that the contribution should be positioned more precisely against online stopping and hybrid optimization.

1. Relationship to STRIDE, CM2 and ATOD.

We appreciate the detailed comparison to prior work. To clarify our contribution, BLADE's novelty lies in the specific coupling of boundary-level consequential-error stopping with differentiated supervision supports, rather than in online stopping or hybrid optimization in isolation. STRIDE uses teacher-score-based stopping together with prefix reuse. CM2 checks checklist strictness before releasing the next reference user query, making it a direct precedent for judge-gated continuation. ATOD combines OPD and reinforcement learning through annealing and turn-level disagreement/uncertainty reweighting.

Concretely, the error-producing loop remains in the collected prefix and receives OPD supervision; a stopped branch receives an early-failure reward with GRPO credit restricted to the error-onset loop; naturally completed trajectories retain full-trajectory task credit. Teacher and task signals are then combined before clipping. Timing alone is not the distinction from CM2, and the hybrid-advantage idea is not claimed as unique to BLADE. The contribution is this boundary-and-credit design for stateful interaction.

Why the combination has a specific optimization meaning.

The collection boundary, early outcome and credit supports are coupled rather than three interchangeable ways of reweighting a completed rollout. STOP determines that no further user-conditioned interaction is generated on the branch. The failure reward is then constructed for that stopped episode, while the task advantage is supported only on the error-onset loop. OPD retains its larger support over the collected student prefix. In naturally completed episodes, the task support instead covers the trajectory. The resulting hybrid advantage is combined before clipping.

This differs conceptually from changing the OPD weight alone: changing a weight cannot undo suffix generation, and applying a trajectory-level failure advantage need not identify the loop that introduced the error. Conversely, a stopping rule alone does not specify what supervision to retain or where task responsibility should land. BLADE supplies that joint specification.

We agree that this should be framed as a particular design and empirical result, not as the first use of stopping or hybrid OPD/RL. The overlap with CM2 is especially relevant because it already uses a judge before the next reference user query. The practical question for BLADE is whether the proposed supervision allocation is useful in the studied stateful user-interaction setting. The following reply provides the joint baseline, credit-support control and ATOD comparison for that question.

Thank you for drawing attention to these closely related methods and helping us clarify the contribution.

AuthorsOpenReview ↗

Response to Reviewer VNgN (2/2)

Thank you for recommending a direct hybrid-method comparison and more targeted ablations.

2. Additional joint-objective controls and ATOD comparison.

We implemented a comparison following ATOD in the 27B-to-4B Mix-Ecom setting, using OPD-weight annealing and turn-level disagreement/uncertainty reweighting. We also evaluated the plain joint OPD+GRPO control separately, rather than treating it as ATOD.

MethodAvg.@4 (%)pass^4 (%)
OPD49.1236.18
ATOD49.8738.69
Joint OPD+GRPO, no boundary control50.3839.20
STOP + full-prefix GRPO credit50.1338.19
BLADE51.7641.21
Loss-level joint, BLADE stopping/credit rule45.6024.62

ATOD's annealing schedule was adapted to our training budget (400 updates); we cannot rule out sensitivity to the schedule length in this setting. The 0.51-point pass^4 difference between ATOD and plain joint optimization should therefore not be read as evidence that annealing or turn-level reweighting is generally unhelpful.

The budget is 400 optimizer updates with predetermined candidate-prompt allocation. Comparisons share initialization, data splits, teacher/top-k scoring, clipping, group size, sampling rules and uniform-token reduction; the ATOD comparison includes its annealing/reweighting, while the component controls keep the joint weights fixed. Dynamic sampling can retain different groups, so this is a matched-update comparison, not an assertion of identical collected tokens or compute.

BLADE exceeds ATOD by 2.52 points in pass^4 and the plain joint control by 2.01 points in this setting. Keeping the stopping rule fixed, replacing full-prefix credit with error-loop credit gives a 3.02-point gain. With stopping and credit supports fixed, advantage-level fusion exceeds loss-level fusion by 16.59 points. These are observed point-estimate differences; we do not infer statistical significance from four evaluation trials per task or treat them as four independent training seeds.

The no-boundary contrast identifies the combined boundary package, including its reward and credit rules, rather than a pure truncation-timing effect. We distinguish these matched controls from the original Table 2 row, whose objective description does not establish that it is the same joint baseline. The new control directly addresses that ambiguity without relying on the OPD-to-BLADE difference.

These additional results support the design in the tested configuration, while the extended ablations address your alternative request to isolate its contribution more clearly.

For the plain joint, STOP/full-credit and BLADE runs, attempted groups are 6,400 in each case, with 5,120, 4,960 and 5,040 retained groups. End-to-end times are 20.5, 16.3 and 16.0 hours. These figures describe actual collection/update allocations rather than an equal-compute experiment. In particular, different outcomes can change which groups dynamic sampling retains even when the candidate-task allocation and filtering algorithm are fixed.

The ATOD row provides a hybrid-method comparison in this configuration. The joint no-boundary row tests whether the gains arise merely from introducing a hybrid objective. The full-prefix-credit row then asks whether restricting task support adds value under the same stopping rule. The fusion row answers the remaining optimization question with stopping and supports held fixed.

This ordering makes the scope of each contrast explicit and avoids drawing a broader superiority claim from comparisons that were not performed.

Thank you again for the constructive comparison requests; they make the experimental evidence more informative.

Reviewer_W1p4OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces BLADE, a post-training framework for mitigating interaction drift in stateful multi-turn LLM agents. BLADE uses a frozen Boundary Judge to detect consequential errors and terminate rollouts before subsequent interactions are generated from error-conditioned states. For terminated rollouts, it assigns a failure reward and localizes GRPO credit to the error-onset interaction loop, while retaining OPD supervision over all collected tokens. The two learning signals are combined at the advantage level.

Experiments on Mix-Ecom and ω²-bench with two teacher–student configurations report improvements in strict repeated-run success (pass4) over the evaluated baselines.

Critical strengths

S1. Relevant problem formulation. The paper addresses error propagation in stateful multi-turn agent training, where incorrect responses or tool actions can influence subsequent user interactions and environment states. This is a relevant challenge for training interactive agents.

S2. Collection-time intervention. Unlike methods that only reweight or filter trajectories after generation, BLADE makes stopping decisions before collecting subsequent user turns. This provides a distinct intervention point and may reduce unnecessary rollout computation.

Critical weaknesses

W1. The core novelty and motivation of BLADE are questionable.

The central novelty of BLADE—terminating error-conditioned interactions during training—is not convincingly justified as a desirable strategy for learning robust interactive agents. Such states naturally arise during deployment and are precisely where agents must learn to recover, adapt, and replan. By discarding these interactions, BLADE may introduce a training–deployment distribution mismatch rather than improve robustness. The paper provides no evidence that suppressing these trajectories is preferable to learning from them.

W2. The experimental design fails to isolate the contributions of the proposed components.

BLADE simultaneously introduces Judge-based early stopping, localized GRPO credit, and advantage-level OPD/GRPO integration. However, the paper lacks essential controlled ablations, particularly an OPD+GRPO baseline with identical sampling and advantage fusion but without boundary control. Consequently, it remains unclear whether the improvements arise from interaction-drift mitigation or other optimization components.

W3. The early-termination reward introduces potentially biased supervision.

BLADE assigns a failure reward of −1 to every Judge-stopped rollout, despite acknowledging that some trajectories might have recovered if continued. This conflates recoverable errors with irreversible failures and may penalize trajectories that could ultimately succeed. Without evaluating the Judge's false-positive rate or the recoverability of stopped trajectories, the validity of this reward assignment remains insufficiently supported.

Questions for authors

My major concern is: Is interaction drift a source of harmful training noise, or an essential component of the state distribution that robust agents must learn to handle?

Q1. Core Novelty and Motivation (W1)

Why is suppressing error-conditioned interactions preferable to learning from them? Can the authors demonstrate that BLADE improves agents' error-recovery and adaptation capabilities, rather than merely avoiding difficult interaction states during training?

Q2. Contribution of Individual Components (W2)

Can the authors provide controlled ablations that independently isolate the effects of early stopping, localized credit assignment, and advantage-level OPD/GRPO integration? In particular, how does BLADE compare against an OPD+GRPO baseline with identical sampling, fusion, and training budgets but without boundary control?

Q3. Validity of Early-Termination Rewards (W3)

What proportion of Judge-stopped trajectories would have eventually succeeded if allowed to continue? How sensitive is BLADE to incorrect stopping decisions, and what evidence supports assigning a failure reward of −1 to all stopped trajectories regardless of their recoverability?

2 comments
AuthorsOpenReview ↗

Response to Reviewer W1p4 (1/2)

Thank you for recognizing the importance of stateful error propagation. We agree that preventing consequential errors and recovering after them are distinct capabilities. The additional controls and diagnostics below separate these questions.

1. Why control drift if recovery is important?

We appreciate the core concern here. We want to clarify that BLADE does not treat every erroneous state as useless for learning; the design targets consequential errors whose effects propagate through the interaction history or environment state. In Figure 1, falsely stating that an address cannot be changed redirects the user toward cancellation. Later actions may be appropriate for that altered request without fulfilling the original address-change goal. This motivates limiting the budget and task credit assigned to that branch, rather than treating its entire continuation as equally informative for the original task.

BLADE does not discard the error-producing action: the collected prefix, including that action, retains OPD supervision in its original context. STOP terminates further collection, assigns the early-failure reward, and restricts GRPO credit to the error-onset loop. However, it also removes subsequent recovery dialogue from that stopped training branch. We acknowledge this coverage trade-off; learning to prevent an error does not establish improved recovery from it.

2. Matched joint-objective and credit-support controls.

The following 27B-to-4B Mix-Ecom comparisons use the same post-SFT initialization, splits, teacher/top-k scoring, OPD/GRPO weights, clipping, group size, sampling rules, dynamic sampling and uniform-token reduction. The budget is 400 optimizer updates with a predetermined candidate-prompt allocation. Retained group counts may differ; we do not claim equal tokens or wall-clock.

VariantAvg.@4 (%)pass^4 (%)
OPD (context)49.1236.18
Joint OPD+GRPO, no boundary control50.3839.20
STOP + full-prefix GRPO credit50.1338.19
BLADE, error-loop GRPO credit51.7641.21
Loss-level joint, same stopping/credit rule45.6024.62

BLADE exceeds the matched joint control by 2.01 percentage points in pass^4. This contrast measures the boundary package, including collection, early reward and credit allocation, not truncation timing alone. With the stopping rule fixed, error-loop rather than full-prefix credit improves pass^4 by 3.02 points. With stopping and supports fixed, advantage-level rather than loss-level fusion improves pass^4 by 16.59 points. These point estimates separate the joint baseline, credit support and fusion questions without attributing the entire OPD-to-BLADE difference to stopping.

Why the two signals are retained on different supports.

OPD and task reward answer different questions. The teacher evaluates student-generated actions in the contexts where they were sampled, including the action that introduced the error. The stopped-rollout task signal identifies the completed loop at which consequential error evidence appeared. Restricting that signal to the loop avoids giving all earlier retained actions any negative group-relative task advantage merely because a later loop triggered STOP. The loop remains a coarse unit of attribution: we do not claim token-level causal identification of the error.

This also explains why “discarding failed trajectories” is not an accurate description of the update. The failed prefix contributes teacher supervision, and the triggering loop can additionally receive the group-relative task signal. What is absent is the future, error-conditioned suffix. Our choice concerns whether to generate and optimize that additional suffix, rather than whether to learn from the action that caused the problem.

Thank you for suggesting these controls; they help distinguish the contributions of boundary control, credit allocation and fusion.

AuthorsOpenReview ↗

Response to Reviewer W1p4 (2/2)

Thank you also for raising the recovery and stopping-validity questions. We distinguish three measurements: whether the stopping boundary contains evidence of an error, whether the original collector could still finish, and whether final trained models can recover from the same erroneous states. These measurements should not be substituted for one another.

1. Independent boundary audit.

We independently audited 100 STOP prefixes from different original tasks using evidence available at the boundary. Of these, 82 supported a consequential error, 12 were unsupported, and 6 were indeterminate. The observed unsupported-STOP share is 12.0% (Wilson 95% CI: 7.0–19.8%). This is conditional on STOP, not a conventional false-positive rate.

2. Continue the original collector after STOP.

Restoring the same prefixes, environment states and simulator contexts, we continued each four times with the training-time collector checkpoint that triggered STOP, frozen for continuation, no Judge and no teacher takeover. Original-task success was 84/400 = 21.00% (prefix-cluster bootstrap 95% CI: 16.0–26.0%). Even within the 82 supported-error cases, 48/328 continuations succeeded (14.63%). Thus, -1 is a training preference against an evidenced consequential error, not a claim that the branch cannot recover or an unbiased terminal-outcome estimate.

3. Compare final trained models on shared recoverable states.

For a separate conditional-recovery comparison, we selected 60 supported, reversibly erroneous prefixes before observing any evaluated policy's continuation results. We evaluated the final trained OPD, joint OPD+GRPO and BLADE checkpoints, using identical restored states and four trials per prefix:

PolicyOriginal-task success
OPD83/240 = 34.58%
Joint OPD+GRPO91/240 = 37.92%
BLADE88/240 = 36.67%

The continuation audit tests the original collector; this comparison tests final models on a selected recoverable subset. Their rates cannot be subtracted to estimate training gains.

BLADE minus joint is -1.25 points. This does not establish superior recovery or non-inferiority. It provides a bounded diagnostic on these selected error states, while the main evaluation measures end-to-end reliability without teacher or Judge assistance. Training sensitivity to Judge errors remains unquantified by this audit. The supported conclusion is improved overall reliability in the evaluated setting, with recovery coverage and imperfect stopping treated as explicit limitations.

What these diagnostics do and do not justify.

Within the continuation audit, the supported-error, unsupported and indeterminate subsets yield 48/328 (14.63%), 29/48 (60.42%) and 7/24 (29.17%) original-task success, respectively. The distinction matters: detecting a real error does not establish that the original task is already irreversibly lost. Conversely, eventual success does not retroactively show that no consequential error occurred at the stopping boundary.

The reward -1 defines the outcome of the terminated training episode. It is not the counterfactual terminal reward the branch would necessarily have received. Our evidence supports an aggregate reliability benefit from the tested training recipe, while the audit quantifies successful continuations foregone by stopping. It does not establish that -1 is optimal among possible early-stop reward choices.

The recovery point estimates on 60 selected prefixes do not establish preserved recovery ability across deployment. The set is deliberately restricted to evidence-supported errors with a plausible correction path; this model comparison does not remove that distributional limitation or quantify training sensitivity to Judge mistakes.

Thank you for emphasizing this distinction between preventing errors and learning to recover; it helps sharpen the scope of our conclusions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to Reviewer G4kr (1/2)

Thank you for identifying the need to clarify the no-boundary comparison and hold the joint objective fixed.

Matched joint OPD+GRPO without boundary control.

You are correct that Table 2's no-boundary scores match the OPD baseline and that its description does not establish a joint-objective control. We therefore distinguish that reported row from the explicit matched joint OPD+GRPO comparison below, rather than attributing the entire 36.18% to 41.21% difference to boundary control. We will correct the Table 2 row description in the revision so that its objective is stated unambiguously.

In the 27B-to-4B Mix-Ecom setting, the comparisons share the post-SFT student initialization, train/test split, 27B teacher, top-k teacher scoring, fixed OPD/GRPO weights, ratio clipping, group size G=4, candidate-prompt sampling allocation, dynamic-sampling rules and uniform-token reduction. The budget is 400 optimizer updates. The original BLADE and loss-level configurations use these same settings.

VariantOnline STOPGRPO supportFusionAvg.@4 (%)pass^4 (%)
Joint OPD+GRPONoFull natural trajectoryAdv.50.3839.20
STOP + full-prefix creditYesAll retained prefix tokensAdv.50.1338.19
BLADEYesError-onset loop on STOPAdv.51.7641.21
Loss-level jointYesError-onset loop on STOPLoss45.6024.62

For naturally terminated trajectories, task credit remains trajectory-wide. For stopped trajectories, the full-prefix control and BLADE use the same stopping and early-failure rules, differing in GRPO support. OPD supervision is retained over the collected prefix used for optimization.

BLADE improves pass^4 by 2.01 percentage points over joint OPD+GRPO without boundary control. With stopping fixed, localizing credit improves pass^4 by 3.02 points; with stopping and supports fixed, advantage-level fusion improves pass^4 by 16.59 points over loss-level fusion. The first contrast measures the combined boundary mechanism, including STOP reward and credit allocation, not collection timing alone.

Interpreting the original and additional comparisons.

The original OPD-to-BLADE comparison changes the complete training recipe. It is useful for measuring overall performance, but cannot by itself assign the gain to the collection boundary. The joint OPD+GRPO row is the necessary control because it isolates the combined boundary mechanism while keeping the same advantage-level fusion.

The other controls answer distinct questions. STOP/full-prefix credit retains the early-failure assignment but broadcasts task advantage across the stopped prefix. BLADE restricts that support to the error-onset loop. The loss-level row retains the stopping and support rules but clips the OPD and GRPO channels separately before combining their losses; BLADE first combines the advantages. As stated in §4.4, opposing signs can offset before clipping in the latter case, so the two rules are not generally equivalent.

We therefore report the three contrasts separately: boundary package versus matched joint optimization, localized versus full-prefix task credit, and advantage-level versus loss-level integration. This avoids interpreting Table 3's fusion result as an answer to what happens when boundary control is disabled. The budget and dynamic-sampling details are provided in the next reply.

Thank you for identifying this missing control and the need to distinguish these experimental questions.

AuthorsOpenReview ↗

Response to Reviewer G4kr (2/2)

Thank you also for asking us to make the sampling, reduction and budget conditions explicit.

What is and is not budget-matched.

VariantUpdatesAttempted / retained groupsRollout tokensEnd-to-end time
Joint/no boundary4006,400 / 5,12017.60M20.5 h
STOP/full credit4006,400 / 4,96012.48M16.3 h
BLADE4006,400 / 5,04012.10M16.0 h

Attempted groups include all candidate groups; retained groups enter optimization after dynamic filtering. Updates use approximately 12–13 retained groups on average, so we do not claim identical retained batches. Rollout tokens include all generated candidates, and end-to-end time includes rollout, teacher scoring, boundary checking where applicable, and optimization. These results are matched in updates and candidate-task allocation, not in tokens or wall-clock. The four evaluation trials per task use the same task/protocol and seeds across methods; they are not independent training seeds.

This comparison addresses the joint-objective confound directly while keeping the separate role of credit support and fusion visible.

Sampling and token reduction.

Each attempted prompt group samples G=4 candidate trajectories. The candidate-prompt allocation is predetermined, whereas retention follows the same dynamic-sampling rule across controls. Thus, matching the rule does not imply that the resulting retained groups are identical: removing STOP can change rewards and trajectories, and therefore change retention. The table reports both attempted and retained groups so that this distinction remains visible.

With OPD and GRPO active, uniform-token reduction uses the relevant student-token support as specified by the objective. A token with zero realized task advantage is not removed solely for having a zero value; support membership determines the reduction. In stopped BLADE trajectories, earlier prefix tokens retain OPD supervision even when they are outside the GRPO support. In natural trajectories, both channels retain trajectory-wide support.

Same-prompt groups with constant rewards, including all-STOP groups, have zero group-relative GRPO advantage. Assigning -1 to a stopped trajectory therefore does not impose the same fixed penalty on every token or guarantee a nonzero task update in every group. These details matter when interpreting both dynamic retention and localized credit.

What the timing numbers establish.

The reported 16.0 hours for BLADE is approximately 22.0% lower than 20.5 hours for the joint no-boundary run, under the stated update budget and hardware setting. This is an end-to-end observation, including collection and training-side supervision, rather than a claim that Judge calls have no cost. It also does not establish performance at identical wall-clock or identical token budgets. Estimated teacher-scored-token counts are not used here to claim a precisely measured teacher-compute reduction.

The comparison remains conditional on this 27B-to-4B Mix-Ecom configuration. Four trials per task measure repeated-run reliability under the evaluation protocol; they do not quantify variability across independently trained checkpoints.

Judge inputs and Figure 2.

In our implementation, the Boundary Judge receives the successful reference trajectory shown in Figure 2, together with the policy-visible interaction prefix and read-only state view ρ(sj). The reference provides a concrete example of valid resolution for assessing consequential, uncorrected deviations; it is not used for literal trajectory matching or as a terminal-outcome shortcut. Section 4.2 omits this input and states that no hidden reference answer is used. We acknowledge this inconsistency and will revise Section 4.2 and, where necessary, Figure 2 to accurately describe the implemented inputs.

We appreciate your careful scrutiny of the experimental setup and hope these details clarify the comparison.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.