acceptodds
Under review as a conference paper at ICLR 2027

DASH-OPD: Discrepancy-Aware Switching with Hysteresis for On-Policy Distillation

15 views

Abstract

While on-policy distillation (OPD) reduces exposure bias by training student language models on their own rollouts, early student errors in long-horizon agentic scenarios can lead to contexts unfamiliar to the teacher. To improve trajectory quality, recent work on agentic OPD introduces teacher intervention into training rollouts by switching the executor between the student and the teacher. However, existing methods determine how much teacher intervention is needed—but not when. To address this limitation, we propose DASH-OPD (Discrepancy-Aware Switching with Hysteresis for OPD), the first agentic OPD method to perform adaptive, bidirectional executor switching. At each turn, DASH-OPD measures teacher–student discrepancy using a mean log-likelihood ratio over action tokens. High student-to-teacher ratios on student turns serve as drift signals, while low teacher-to-student ratios on teacher turns serve as recovery signals. These signals are accumulated over multiple turns to form drift and recovery evidence, respectively. DASH-OPD switches executors when either type of evidence exceeds its corresponding switching threshold, introducing hysteresis that prevents rapid switching triggered by transient discrepancy fluctuations. Across three benchmarks and two student model scales, DASH-OPD outperforms five baselines in all 14 task-performance comparisons, while requiring the fewest interaction turns in nine of ten efficiency comparisons. Code and trained models are available in this anonymous repository: https://anonymous.4open.science/r/DASH-OPD-7C3D.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, long-horizon agents, executor switching

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_aghVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies when a teacher should intervene and return control to a student during multi-turn agentic OPD. DASH-OPD standardizes and accumulates directional teacher–student disagreement over action tokens to select the next turn's executor, with a stagnation trigger and a minimum teacher span. Training follows Guided-OPD and covers both reasoning and action tokens; inference uses only the student. The authors report overall performance advantages over five baselines across three environments and two student scales, supported by component ablations and switching-behavior analyses.

Critical strengths
  1. DASH-OPD uses feedback from the current interaction trajectory to decide when the teacher model should take over and when control should return to the student. This makes rollout scheduling in multi-turn agent distillation more flexible than a predefined curriculum, while retaining student-only inference.
  2. The paper reports consistent performance gains over the evaluated OPD baselines across three environments and two student model sizes. The ablations also show that removing student resumption or cross-turn accumulation reduces performance, providing experimental support for these design choices.
Critical weaknesses
  1. The motivating bottleneck is insufficiently established. The introduction motivates teacher guidance through vanilla OPD's limited gains, but does not directly establish that curriculum-based intervention timing is a key limitation. Figure 1 illustrates execution schedules without showing mistimed interventions and their consequences. Given related adaptive-intervention approaches such as ThriftyDAgger and AdaSwitch, I would like clearer evidence of the problem that DASH's specific adaptation resolves.
  2. The ablations leave the gains from switching decisions insufficiently isolated. Removing teacher execution or student resumption changes both the switching behavior and teacher participation. DASH also permits teacher participation late in training, whereas Guided-OPD has a student-only tail. These comparisons do not sufficiently separate better timing from differences in supervision amount or trajectory composition. Moreover, the recovery statistics partly depend on the controller's own triggers. The action-mask ablation supports the chosen statistic empirically, but does not establish why excluding reasoning disagreement helps; signal mixing or tuning effects remain possible, unverified explanations.
Questions for authors
  1. Could the authors provide trajectory examples or aggregate evidence of mistimed curriculum interventions and their effects on task success?
  2. Could the authors clarify whether DASH retains an advantage when teacher turn/token fractions and execution spans are comparable? Is there evidence of recovery independent of the controller's subsequent triggers?
  3. Could the authors explain how the thresholds were selected, including for the no-action-mask comparison? Beyond Figure 5's local sensitivity analysis, what motivates linear scaling with H_max and the one-third return threshold?
10 comments

We thank you for recognizing DASH-OPD's flexibility and the convincing experimental results. Below, we provide detailed responses to your valuable suggestions and questions.

AuthorsOpenReview ↗

1. Limitations of curriculum-based methods (W1, Q1)

We thank you for your insightful questions. Training-progress curricula have a fundamental limitation: they allocate teacher intervention without accounting for whether the current trajectory actually needs it. We have supplemented the revised manuscript with the more detailed analysis below. Our original manuscript already identifies this problem in Section 1, noting that intervention is scheduled "even though its necessity may vary within and across trajectories." Across trajectories, the same training step can contain both productive student rollouts and rollouts trapped in repeated, unproductive actions. Within a trajectory, the need for assistance can rise sharply after an error and diminish after recovery. Neither TCOD's scheduled segment boundary nor Guided-OPD's scheduled teacher probability responds to these changes. Consequently, these schedules can replace useful student execution with unnecessary teacher turns while failing to provide timely assistance when errors begin to compound. This mismatch directly undermines the intended balance between improving trajectory quality through teacher intervention and reducing exposure bias through on-policy student learning. DASH-OPD addresses it by using accumulated feedback from the current trajectory to adaptively determine when the teacher intervenes and when the student resumes control.

Saved training trajectories illustrate how mistimed curriculum interventions can undermine task success and how DASH-OPD's adaptive switching can enable successful completion. On ALFWorld's "put a cool egg in microwave" task, TCOD-B2F hands control to the 1.7B student after a scheduled 14-action expert prefix. The student then repeats look for all 16 remaining turns and fails. On ScienceWorld's "find a plant and move it to the purple box" task, Guided-OPD's 1.7B student repeats look around in the foundry for all 30 turns without any teacher intervention, despite a teacher-selection probability of 15.20%. The task remains incomplete with a score of 8/100. A same-task ALFWorld comparison at 4B further illustrates why sustained assistance matters: on "put two candles in a drawer," both Guided-OPD and DASH-OPD encounter failed attempts to pick up the second candle while carrying the first. Guided-OPD continues interleaving teacher and student execution without resolving this obstacle and fails at turn 30. DASH-OPD maintains teacher control through turn 15: the teacher deposits the first candle at turn 13 and retrieves the second at turn 15, then returns control to the student, which completes the task at turn 17.

The existing experiments in Table 1 already provide aggregate evidence of DASH-OPD's performance advantage over curriculum-based scheduling (TCOD and Guided-OPD). Table R1 extracts the primary performance metrics from Table 1, using mean ± sample SD over seeds 0, 42, and 2026.

Table R1. Main performance of DASH-OPD and curriculum-based baselines (Table 1). Bold marks the best and underlining marks the second-best mean in each row.

Benchmark / student / metricDASH-OPDTCODGuided-OPD
WebShop / 1.7B / score43.52 ± 1.6138.05±1.62‾\underline{38.05 \pm 1.62}32.86 ± 1.68
WebShop / 4B / score43.85 ± 2.3438.53±2.00‾\underline{38.53 \pm 2.00}37.68 ± 2.05
ALFWorld / 1.7B / overall SR (%)34.18 ± 2.6429.56±2.46‾\underline{29.56 \pm 2.46}26.52 ± 1.72
ALFWorld / 4B / overall SR (%)42.82 ± 2.4335.64±2.18‾\underline{35.64 \pm 2.18}30.90 ± 2.32
ScienceWorld / 1.7B / score38.30 ± 1.5130.27 ± 1.7732.58±1.69‾\underline{32.58 \pm 1.69}
ScienceWorld / 4B / score49.29 ± 1.9846.11 ± 1.5946.17±2.13‾\underline{46.17 \pm 2.13}

DASH-OPD outperforms both curriculum-based baselines on every primary metric at both student sizes. Compared with the stronger of TCOD and Guided-OPD in each setting, DASH-OPD improves WebShop score by 5.47 / 5.32 points, ALFWorld overall SR by 4.62 / 7.18 percentage points, and ScienceWorld score by 5.72 / 3.12 points at 1.7B / 4B, respectively. These consistent gains across all six settings support the benefit of adapting teacher intervention to the current trajectory beyond scheduling it solely by training progress.

Changes in the revised manuscript (blue). Section 1 expands the explanation of assistance needs varying within and across trajectories. Figure 1 and its caption ("A research gap in agentic OPD.") are retained from the original submission. Section 5.2 summarizes the comparison with curriculum-based baselines and points to Appendix A.17, which adds the three observed training-trajectory examples and their task outcomes. Appendix A.3 cites ThriftyDAgger when contrasting the use of an auxiliary risk model.

AuthorsOpenReview ↗

2. Distinctions from prior work (W1)

We thank you for highlighting the connection between prior methods. We have cited ThriftyDAgger in the revised manuscript to acknowledge its contribution to adaptive intervention.

DASH-OPD differs from ThriftyDAgger in switching evidence and application scenarios. ThriftyDAgger (Hoque et al., 2021a) controls robot–human switching in robotic imitation learning: high state novelty or high failure risk estimated by a trained auxiliary risk model triggers takeover, while low risk and low robot–supervisor action discrepancy permit return. In contrast, DASH-OPD controls student–teacher switching during agentic OPD, accumulating standardized action-only discrepancy across interaction turns to guide both takeover and return. It reuses probabilities already available for distillation without training an auxiliary risk model.

DASH-OPD differs from AdaSwitch in switching granularity and direction. AdaSwitch (Peng et al., 2025) makes a single student-to-teacher switch at the token level within a response, after which the teacher generates all remaining tokens. In contrast, DASH-OPD supports repeated turn-level switches in both directions within an episode.

Changes in the revised manuscript (blue). Appendix A.3 cites ThriftyDAgger. Appendix A.21 adds explicit comparisons with ThriftyDAgger and AdaSwitch.

Reviewer_4RAoOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

DASH-OPD adaptively switches rollout control between a student and teacher during multi-turn distillation. It accumulates standardized action-token likelihood ratios to trigger intervention and return control, while using reverse-KL training on student turns and forward-KL training on teacher turns. Experiments on WebShop, ALFWorld, and ScienceWorld with two Qwen3 student sizes report improvements over five baselines.

Critical strengths
  1. Practical intervention mechanism. The controller reuses log-probabilities already collected for distillation and requires only constant-size statistics, making it a lightweight addition to existing training pipelines.

  2. Consistent empirical results. Improvements across three benchmarks and two student sizes, supported by component ablations, provide encouraging evidence that the overall approach is useful.

Critical weaknesses
  1. The switching signals are insufficiently validated. Eqs. 7–9 measure deviations from historical discrepancy rather than absolute disagreement. Constantly high student discrepancy produces no discrepancy-based intervention evidence; constantly low teacher discrepancy produces no recovery evidence. Moreover, action probabilities are conditioned on the executor’s reasoning, so agreement under teacher-generated reasoning does not establish autonomous student competence. These issues undermine the interpretation of the signals as drift and recovery.

  2. The experiments do not isolate the contribution of adaptive timing. Appendix A.1 states that DASH retains teacher intervention throughout training, whereas Guided-OPD ends with student-only batches. Without controls matching teacher exposure and intervention duration, gains could reflect continued teacher supervision rather than better switching decisions. Single-seed results further weaken the evidence: some WebShop improvements amount to only one additional successful task.

Questions for authors
  1. Clarification: How do teacher-generated token counts and their distribution over training compare between DASH and Guided-OPD? Were comparisons performed with matched teacher exposure, including during late training?

  2. Evidence request: Do the standardized signals predict actual student failure and successful autonomous continuation? Evidence separating this predictive ability from the controller’s own intervention decisions would materially strengthen the mechanism claim.

  3. Evidence request: Do the main gains persist across independent training seeds, with paired uncertainty estimates on evaluation tasks?

10 comments

We thank you for recognizing the controller's low overhead and the consistent gains across benchmarks and student sizes. Below, we provide detailed responses to your valuable suggestions and questions.

AuthorsOpenReview ↗

1. Switching-signal design (W1, Q2)

We thank you for the insightful review about history-relative discrepancy and student autonomy. Eq. (7) uses online standardization, a mechanism also employed in self-starting CUSUM schemes when the reference mean and variance are unknown (Hawkins, 1987). Eqs. (8–9) then accumulate these standardized deviations, a mechanism also used in classical CUSUM (Page, 1954). The revised manuscript now discusses these connections and cites both studies. These mechanisms support hysteretic switching in DASH-OPD, reducing control chattering caused by transient signal spikes.

Constantly high discrepancy and growing discrepancy play different roles. A high, stable student-to-teacher discrepancy can supply useful reverse-KL supervision on the student's own responses. Automatically replacing such turns with teacher execution would discard those on-policy learning opportunities. However, a sustained rise above the trajectory's historical discrepancy, or a sufficiently large jump, instead indicates that the student trajectory drifts away from regions familiar to the teacher and motivates intervention. Constantly low discrepancy and decreasing discrepancy also play different roles. A low, stable teacher-to-student discrepancy measures agreement under teacher-generated actions, but it does not establish that the student can continue autonomously. A decrease relative to the same trajectory’s history provides evidence of improving agreement as the teacher advances the interaction.

Constantly high student-side or low teacher-side discrepancy can also accumulate switching evidence when it remains above or below its respective historical mean, since these means are retained across switches and typically adapt more slowly than the discrepancies themselves.

We add direct ablation to test whether the historical reference matters, as shown in Table R1. We have added these new results in the revised version. In Table R1, "w/o historical reference" removes Eq. (7) and accumulates raw discrepancies using Dtraw=max⁡(Dt−1raw+dtS,0)D_t^{raw}=\max(D_{t-1}^{raw}+d_t^S,0) and Rtraw=max⁡(Rt−1raw−dtT,0)R_t^{raw}=\max(R_{t-1}^{raw}-d_t^T,0). Other controller components are kept fixed. We scale the intervention and return thresholds by 1.8 (using the same tuning budget as for full DASH-OPD).

Table R1. Ablation results for historical reference.

Method / studentWebShop score ↑WebShop SR (%) ↑WebShop turns ↓ALFWorld IID SR (%) ↑ALFWorld IID turns ↓ALFWorld OOD SR (%) ↑ALFWorld OOD turns ↓ALFWorld overall SR (%) ↑ALFWorld overall turns ↓ScienceWorld score ↑ScienceWorld SR (%) ↑ScienceWorld turns ↓
w/o historical reference / 1.7B33.95‾\underline{33.95}10.00‾\underline{10.00}6.71‾\underline{6.71}25.00‾\underline{25.00}24.3429.85‾\underline{29.85}25.26‾\underline{25.26}27.37‾\underline{27.37}24.7929.38‾\underline{29.38}11.24‾\underline{11.24}25.60‾\underline{25.60}
DASH-OPD / 1.7B (existing seed 42)43.5313.006.2531.4324.98‾\underline{24.98}34.3324.6932.8524.84‾\underline{24.84}38.4612.7723.60
w/o historical reference / 4B34.10‾\underline{34.10}11.00‾\underline{11.00}8.50‾\underline{8.50}38.57‾\underline{38.57}24.15‾\underline{24.15}35.82‾\underline{35.82}24.65‾\underline{24.65}37.23‾\underline{37.23}24.39‾\underline{24.39}45.22‾\underline{45.22}21.41‾\underline{21.41}20.42
DASH-OPD / 4B (existing seed 42)42.7716.007.1541.4322.8641.0423.4641.2423.1550.9722.4020.62‾\underline{20.62}

According to Table R1, historical reference in Eq. (7) is an essential core component of DASH-OPD's switching mechanism. Removing it lowers WebShop score by 9.58 / 8.67 points, ALFWorld overall SR by 5.48 / 4.01 percentage points, and ScienceWorld score by 9.08 / 5.75 points at 1.7B / 4B, respectively.

Changes in the revised manuscript (blue). Table 2 adds the historical-reference ablation results, with a brief analysis in Section 5.3. Section 4.3 adds the self-starting CUSUM and the CUSUM-style accumulation citations.

AuthorsOpenReview ↗

2. Student autonomy analysis (W1, Q2)

We acknowledge the insufficient clarification of autonomous student competence and add the following analysis to the revised manuscript. Teacher intervention aims to prevent trajectory collapse: compounding student errors can push the trajectory beyond regions familiar to the teacher, degrading the quality of its supervision. As action-only discrepancy decreases relative to its historical level during teacher execution, the controller accumulates evidence that the student can resume control without immediately driving the trajectory off course, at which point control should be returned promptly to reduce exposure bias. Residual reasoning discrepancies need not delay return when they do not change the actions that determine the trajectory. Because action probabilities condition on preceding reasoning, shrinking action-only discrepancy also indirectly indicates greater agreement in action-relevant reasoning.

The recovery results support autonomous student competence after teacher intervention. Across 263 teacher segments, mean discrepancy on student-generated actions drops from 1.30 before takeover to 0.32 on the first resumed turn. No takeover decision occurs in 243/263 cases (92.40%); the remaining 20 yield 18 takeovers and two decisions unexecuted due to episode termination or the turn limit. Successful task completion without further teacher execution follows 131/436 actual returns (30.05%). In 405 saved rollouts, only 18/436 actual returns have exactly one student turn before teacher re-intervention, despite no minimum student span.

The "w/o action mask" ablation in Table 2 uses full-response discrepancy, including reasoning tokens. Table R2 shows that action-only discrepancy improves the primary metric in all six settings, supporting the action-only discrepancy design.

Full-response discrepancy weights action and reasoning discrepancy by token counts. Reasoning discrepancy or length changes can thus shift it relative to its historical mean, contributing to intervention or return evidence even with unchanged action discrepancy. Action masking removes this direct contribution from non-action tokens.

The original no-action-mask ablation uses unchanged DASH-OPD thresholds. We also retune both full-response thresholds by a factor of 1.2, selected with the same tuning budget as full DASH-OPD. Action-only discrepancy still wins in all six settings (Table R2), supporting its benefit after retuning.

Table R2. Ablation results for action-only discrepancy (Table 2) and the independently retuned comparison. Bold marks the best and underlining marks the second-best values in each row.

Benchmark / student / metricAction-only discrepancyFull-response discrepancyRetuned full-response discrepancy
WebShop / 1.7B / score43.5338.4739.35‾\underline{39.35}
WebShop / 4B / score42.7737.8540.62‾\underline{40.62}
ALFWorld / 1.7B / overall SR (%)32.8529.2030.66‾\underline{30.66}
ALFWorld / 4B / overall SR (%)41.2439.4240.15‾\underline{40.15}
ScienceWorld / 1.7B / score38.4637.75‾\underline{37.75}37.03
ScienceWorld / 4B / score50.9749.0649.41‾\underline{49.41}

New results in Table R3 test predictive ability independently of subsequent controller decisions and are added to the revised manuscript. For each benchmark and model size, we uniformly sample 500 student-turn and 500 teacher-turn states from mixed-policy test rollouts without outcome filtering. After the current action and before switching resets, we record intervention evidence DtD_t or return evidence RtR_t. The frozen student then continues once from the same environment state and full history, with the remaining turn budget, standard evaluation decoding, and no further teacher assistance. All sampled states enter failure-prediction AUROC for DtD_t and success-prediction AUROC for RtR_t. Table R3 reports equal-weight averages over the 1.7B and 4B students.

Table R3. Outcome prediction independently of subsequent controller decisions (averaged over 1.7B and 4B).

BenchmarkAUROC: intervention evidence → failureAUROC: return evidence → success
WebShop0.770.75
ALFWorld0.810.77
ScienceWorld0.690.74

Across all three benchmarks, intervention evidence predicts failure (AUROC 0.69–0.81), and return evidence predicts success (0.74–0.77), supporting predictive value beyond the controller's own switching decisions.

Changes in the revised manuscript (blue). Section 5.2 adds student-recovery evidence and points to Appendix A.18 for the AUROC results, and analysis. Section 4.4 clarifies the absence of a minimum student span. Section 4.2 and Appendix A.18 expand the action-only rationale. Section 5.3 clarifies the original action-mask ablation's unchanged thresholds and points to Appendix A.20 for the independently retuned comparison.

Reviewer_xWCTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposed DASH-OPD, which perform adaptive bidirectional executor during OPD based on evidence from multiple turns. DASH-OPD is verified on several datasets and benchmarks.

Critical strengths
  1. The paper addresses an important problem in agentic distillation, as how to combine the on-policy distillation and the teachers trajectory guidance.
  2. The method is clearly described and implemented code is provided.
Critical weaknesses
  1. The paper seems to combine several existing method as discrepancy-based intervention, CUSUM-like accumulation, and asymmetric switching. I.think it should be clear that which part lead to the incremental.

  2. The method relies on several manual specified thresholds, more evidence about the thresholds choosing is needed.

  3. All experiment results use only one random seed.

  4. Longer student execution after handoff does not necessarily imply successful recovery, since re-intervention is also determined by the controller.

Questions for authors
  1. Could you please provide more evidence on how the setting of controller is selected?
  2. Does DASH-OPD retain its advantage over random or simple threshold?
  3. Does the recovery signal actually tell us when the student can continue successfully on its own?
  4. Could you run the main experiments with a few more random seeds and report the mean and standard deviation?
10 comments
AuthorsOpenReview ↗

1. Explanation of novelty (W1)

Our contribution is the first adaptive, bidirectional execution scheduling for agentic OPD. We thank you for highlighting the connections to prior work. DASH-OPD shares some ideas with CUSUM (Page, 1954), a classical sequential change-detection method, and LazyDAgger (Hoque et al., 2021b), an interactive imitation learning method. We have made these connections explicit and added the corresponding references in the revised manuscript.

DASH-OPD differs from CUSUM in application scenarios, monitored signals, and control objectives. (i) Application scenarios. CUSUM originates in statistical quality control for monitoring processes. DASH-OPD schedules teacher–student execution during multi-turn LLM agent distillation. (ii) Monitored signals. CUSUM accumulates evidence from process measurements to detect distribution shifts. DASH-OPD constructs executor-dependent action-token log-likelihood ratios: student-to-teacher on student turns and teacher-to-student on teacher turns, aligned with sampled reverse- and forward-KL distillation, respectively. It standardizes these signals against separate trajectory histories and accumulates rising student-side discrepancy for takeover and falling teacher-side discrepancy for return. (iii) Control objectives. CUSUM seeks timely change detection while controlling false alarms. DASH-OPD uses accumulated evidence to repeatedly switch the policy generating training trajectories, balancing teacher guidance with on-policy student exposure.

DASH-OPD differs from LazyDAgger in application scenarios, switching criteria, and learning objectives. (i) Application scenarios. LazyDAgger controls robot–supervisor interaction in robotic imitation learning. DASH-OPD controls student–teacher interaction in multi-turn LLM agent distillation. (ii) Switching criteria. LazyDAgger invokes the supervisor through a learned action-discrepancy classifier and returns control when directly measured action discrepancy falls below a lower threshold. DASH-OPD bases both directions on separately accumulated, history-standardized action-token likelihood evidence across turns. Both use asymmetric switching for stable control, but DASH-OPD reuses distillation log-probabilities without an auxiliary classifier or additional switching forward passes. (iii) Learning objectives. LazyDAgger learns supervisor-like robot behavior while reducing supervisor burden and context-switching costs. DASH-OPD improves student task performance by balancing teacher-guided trajectory quality with student-generated on-policy experience to limit exposure bias, retaining student-only inference after training.

Table 2 identifies contributions to the gains. Removing discrepancy standardization, evidence accumulation, student resumption, or action masking lowers the primary metric in all six settings. These results support the OPD controller beyond its classical ingredients.

Recent work offers methodological precedents for adapting classical methods to LLMs. PID Steering (ICLR 2026) and A-LQR (ICML 2026) adapt PID control and linear quadratic regulation, respectively, to activation steering; PDS (ICLR 2025) applies Pontryagin’s Maximum Principle to training-data selection; Predictive-Decoding (ICLR 2025) adapts model predictive control to LLM reasoning and agent planning; and TMPC (ICLR 2026) adapts it to inference-time preference alignment.

Changes in the revised manuscript (blue). Sections 1 and 4.3 now cite CUSUM (Page, 1954), and Section 1 cites LazyDAgger (Hoque et al., 2021b). Appendix A.21 adds explicit comparisons with CUSUM and LazyDAgger.

AuthorsOpenReview ↗

2. Switching thresholds and controller settings (W2, Q2, Q1) (Part 1/2)

We thank you for your interest in the hyperparameter settings of DASH-OPD. To account for different rollout horizons across benchmarks, we set τon=qonHmax⁡\tau_{on}=q_{on}H_{\max} and τoff=qoffHmax⁡\tau_{off}=q_{off}H_{\max}, where Hmax⁡H_{\max} is the maximum number of turns per trajectory. We intuitively set the intervention threshold ratio qon=0.10q_{on} = 0.10 and the return threshold ratio qoff=qon/3q_{off} = q_{on}/3 across all three benchmarks and both student sizes. These defaults are chosen heuristically. We already provide evidence of choosing these threshold ratios in Fig. 5. Fig. 5 shows sweeps that vary qonq_{on} over {0.08, 0.10, 0.12} at fixed qoffq_{off}, and vary qoffq_{off} independently by ±20% at fixed qonq_{on}. Table R1 details these results.

Table R1. Evidence of choosing threshold ratios (Fig. 5).

Benchmark / studentDefault SR (%)SR under 20% perturbations (%)
WebShop / 1.7B13.0012.00–14.00
WebShop / 4B16.0016.00–16.00
ALFWorld / 1.7B32.8531.02–33.21
ALFWorld / 4B41.2440.15–40.88
ScienceWorld / 1.7B12.7711.09–11.54
ScienceWorld / 4B22.4021.10–22.32

We add further evidence of choosing these threshold ratios by varying qonq_{on}/qoffq_{off} independently by ±50% at fixed qoffq_{off}/qonq_{on} in Table R2. We have added these new results in the revised manuscript.

Table R2. Further evidence of choosing threshold ratios.

Benchmark / studentDefault SR (%)SR under 50% perturbations (%)
WebShop / 1.7B13.0012.00–13.00
WebShop / 4B16.0014.00–16.00
ALFWorld / 1.7B32.8530.29–32.12
ALFWorld / 4B41.2438.32–39.42
ScienceWorld / 1.7B12.7710.86–12.16
ScienceWorld / 4B22.4020.41–22.78

According to Tables R1 and R2, among the nine tested configurations (the shared default and independent ±20% and ±50% perturbations), the shared default achieves the highest mean SR across the six settings (23.04%, versus 22.84% for the next best), supporting the default thresholds within the tested range.

The paper already includes a random-switching baseline: Guided-OPD in Table 1. It randomly selects the teacher or student at each turn according to a training-progress schedule. DASH-OPD outperforms Guided-OPD in all six settings, as shown in Table R3. The gains below are calculated from the mean values in Table R11, which summarizes the new main-experiment results across random seeds 0, 42, and 2026.

Table R3. Gains compared with using random switching.

Benchmark / metricGain at 1.7BGain at 4B
WebShop / score+10.66+6.17
ALFWorld / overall SR (pp)+7.66+11.92
ScienceWorld / score+5.72+3.12

The paper also already includes a simple threshold comparator: "w/o signal accumulation" in Table 2. It uses a single-turn signal rather than evidence accumulated from multi-turn signals, and it scales both thresholds by 0.4 (selected with the same tuning budget as full DASH-OPD). Signal accumulation improves the primary metric in all six settings, as shown in Table R4.

Table R4. Gains compared with using a simple threshold.

Benchmark / metricGain at 1.7BGain at 4B
WebShop / score+7.63+8.64
ALFWorld / overall SR (pp)+4.02+6.20
ScienceWorld / score+4.45+5.60
AuthorsOpenReview ↗

2. Switching thresholds and controller settings (W2, Q2, Q1) (Part 2/2)

Evidence for selecting controller hyperparameters (Q1). DASH-OPD has four scheduling hyperparameters: the two threshold ratios above, the minimum teacher span ℓmin⁡\ell_{\min}, and the stagnation threshold KK. We intuitively set ℓmin⁡=2\ell_{\min}=2 and K=3K=3, using the same values across all benchmarks and student sizes. These defaults are chosen heuristically. Table 2 already evaluates both mechanisms through "w/o minimum teacher span" (ℓmin⁡=1\ell_{\min}=1) and "w/o stagnation trigger" (K=+∞K=+\infty). Table R5 summarizes these results.

Table R5. Ablation evidence for the minimum teacher span and stagnation trigger (Table 2). Bold marks the best and underlining marks the second-best values in each row (ties share the same style).

Benchmark / student / metricDefault (ℓmin⁡=2\ell_{\min}=2, K=3K=3)w/o minimum teacher span (ℓmin⁡=1\ell_{\min}=1)w/o stagnation trigger (K=∞K=\infty)
WebShop / 1.7B / score43.5341.15‾\underline{41.15}39.83
WebShop / 4B / score42.7739.5840.39‾\underline{40.39}
ALFWorld / 1.7B / overall SR (%)32.8530.29‾\underline{30.29}32.85
ALFWorld / 4B / overall SR (%)41.2439.78‾\underline{39.78}39.05
ScienceWorld / 1.7B / score38.4637.59‾\underline{37.59}37.04
ScienceWorld / 4B / score50.9747.70‾\underline{47.70}46.14

Setting ℓmin⁡=1\ell_{\min}=1 lowers the reported metric in all six settings. Setting K=+∞K=+\infty lowers it in five settings and ties it in one.

We add further evidence of selecting these two controller hyperparameters by varying ℓmin⁡\ell_{\min} over {2, 3, 4} at fixed KK and varying KK over {2, 3, 4} at fixed ℓmin⁡\ell_{\min} in Table R6. We have added these new results in the revised manuscript.

Table R6. Further evidence of selecting the minimum teacher span and stagnation threshold. Bold marks the best and underlining marks the second-best values in each row (ties share the same style).

Benchmark / student / metricDefault (ℓmin⁡=2\ell_{\min}=2, K=3K=3)ℓmin⁡=3\ell_{\min}=3ℓmin⁡=4\ell_{\min}=4K=2K=2K=4K=4
WebShop / 1.7B / score43.5339.5241.4640.6441.67‾\underline{41.67}
WebShop / 4B / score42.7740.6339.6539.5042.02‾\underline{42.02}
ALFWorld / 1.7B / overall SR (%)32.8531.75‾\underline{31.75}29.2031.75‾\underline{31.75}31.02
ALFWorld / 4B / overall SR (%)41.24‾\underline{41.24}41.9738.6940.5140.88
ScienceWorld / 1.7B / score38.4637.3238.07‾\underline{38.07}37.4837.73
ScienceWorld / 4B / score50.9749.90‾\underline{49.90}46.2947.8848.67

The shared default (l_min=2, K=3) ranks first in five of six settings, and it also achieves the highest mean of the six reported metrics (41.64, versus 40.33 for the next-best configuration), supporting the default configurations within the tested range.

Changes in the revised manuscript (blue). Section 5.3 summarizes the additional evidence for selecting controller hyperparameters. In Appendix A.16, Table 8 reports the ±20% and ±50% threshold sweeps, and Table 9 reports the minimum teacher span and stagnation threshold sweeps.

Comments on the paper

2 comments

Dear Reviewers,

The authors have submitted their rebuttal. Please take a look at their responses and actively participate in the discussion, especially regarding any remaining concerns or disagreements. Please consider updating your evaluations if appropriate.

Thank you for your time and effort!

Best regards, Area Chair

AuthorsOpenReview ↗

Rebuttal summary

We thank all three reviewers for their time and constructive feedback. We appreciate Reviewer xWCT’s recognition of the importance of combining on-policy distillation with teacher guidance, Reviewer 4RAo’s recognition of the controller’s low overhead and consistent empirical gains, and Reviewer aghV’s recognition of the flexibility of trajectory-adaptive switching. In response to their feedback, we expanded the main text from 9 to 10 pages and the full manuscript from 22 to 28 pages. The revised appendix adds seven subsections (A.15–A.21) and eight tables (Tables 7–14).

DASH-OPD adaptively switches control between the teacher and student during multi-turn training rollouts. It measures directional discrepancy on action tokens, standardizes the signals against their respective trajectory histories, and accumulates evidence across turns to decide when the teacher should intervene and when the student should resume. Across WebShop, ALFWorld, and ScienceWorld, with 1.7B and 4B students and three random seeds, DASH-OPD leads five baselines in 13 of 14 task-performance comparisons and has the lowest mean turn count in nine of ten efficiency comparisons.

We added experiments and analyses in response to concerns raised by all three reviewers. For Reviewer xWCT, we clarified the connections to CUSUM and LazyDAgger, expanded the sensitivity analysis for the switching thresholds, minimum teacher span, and stagnation threshold, and reran the main comparisons with additional training seeds. For Reviewer 4RAo, we added an ablation of the historical reference used to standardize discrepancy, evaluated the switching signals using student-only continuations from sampled rollout states, and audited teacher-generated token counts and shares during training for DASH-OPD and Guided-OPD. For Reviewer aghV, we added training trajectory examples to examine when curriculum-based methods intervene or return control, explained how the switching thresholds scale with the rollout horizon and how we chose the return threshold, and compared action-only discrepancy with an independently retuned full-response variant.

Once again, we would like to express our most sincere gratitude to all Reviewers, Program Chairs, Senior Area Chairs, and Area Chairs for the valuable evaluations and assistance regarding our submission. We sincerely hope our paper can receive objective and proper evaluations. Wishing you all the best!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.