acceptodds
Under review as a conference paper at ICLR 2027

Multi-Turn On-Policy Distillation with Prefix Replay

Abstract

Multi-turn on-policy distillation (OPD) gives LLM agents dense teacher supervision on student-generated actions, but requires costly environment rollouts at every update. We propose Replayed-Prefix On-Policy Distillation (ReOPD), which reuses recorded teacher trajectories: the student acts at selected teacher-forced prefixes and receives token-level teacher supervision without interacting with the environment. We identify a prefix trap: student-generated histories improve relevance to the student but can make teacher targets unreliable, while errors compound across turns. A two-sided bound captures the trade-off between student occupancy mismatch and teacher reliability. ReOPD uses a step-decaying sampling schedule that favors earlier, lower-shift prefixes. Across Python-assisted mathematical reasoning and search tasks, with multiple teacher and student scales, ReOPD matches or improves online OPD accuracy, uses zero tool calls during student training, and is at least 4 faster per rollout. Reusing teacher trajectories also supports joint distillation across environments.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillatiion, agent, search, math

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sskmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes ReOPD, an offline on-policy distillation method for multi-turn agents. It reuses trajectories recorded by the teacher, lets the student generate the action for only one step on a teacher prefix, and has the teacher provide token-by-token supervision, with no environment interaction during training. The authors call the tradeoff in this setup the prefix trap, provide a two-sided error bound, and on that basis adopt a sampling strategy biased toward early steps. On mathematical reasoning with Python and on search tasks, ReOPD is comparable to or better than online OPD in accuracy and is faster per rollout.

Critical strengths
  1. The problem has practical relevance. For multi-turn agents, online OPD must interact with the environment at every update. ReOPD reuses teacher trajectories and removes this cost, and rollouts produced by the teacher during RL training can also be used directly.
  2. The method is simple and easy to integrate into existing pipelines, since it only samples steps on teacher prefixes and needs no extra models or environment calls.
  3. Within the stated setup, the experiments are reasonably well conducted. They cover multiple teacher and student scales, compare fairly with online OPD under the same teacher objective and update budget, and include ablations of the sampling strategy and the prefix source.
Critical weaknesses
  1. The positioning of the contribution does not hold. The Introduction and the related work do not establish the necessity of this work, and the related work in the main text is only four lines long. The prefix trap, presented as a core contribution, has already been identified by Wang et al. 2026, and the authors mention it only in one sentence in the appendix. The design based on reference policy roll-in is also closely related to learning-to-search work such as SEARN and LOLS, and the paper does not discuss either connection.
  2. The experimental setting does not match the motivation. The value of the method lies in settings where environment interaction is expensive, but the experiments use only local environments with very low interaction cost. On the search tasks the accuracy is also on par with OPD, and the practical value of the method has not been demonstrated.
Questions for authors
  1. Compared with the dual exposure bias proposed by Wang et al. 2026 and with the roll-in design based on a reference policy in learning-to-search, what is new in the problem definition or the method of this paper?
  2. We would like the authors to add one set of experiments, with a smaller scale being acceptable, in at least one environment where interaction is expensive, such as web, GUI, or software engineering, to show whether ReOPD can maintain accuracy comparable to online OPD and whether the overall cost still has an advantage after trajectory collection is taken into account.
  3. What evidence shows that the benefit comes from reliability-aware prefix selection rather than only from training early steps more, given that the theory section holds that the decay coefficient should be determined by the gap between the teacher and the student and that teacher reliability also decreases with the number of steps, while the experiments fix κ at 0.6 throughout and do not directly measure teacher reliability?
5 comments
AuthorsOpenReview ↗

Response Part 1

Thank you very much for your thorough review and suggestions. Here we address your concerns:


W1. "The Introduction and the related work do not establish the necessity of this work, and the related work in the main text is only four lines long"

Due to the page limit, we could only put a summary of the related work on the main page, and included a very detailed and complete (~1.5 pages) related work in Appendix A.

Thank you for the suggestion. Since we now have one extra page for the rebuttal, we have already put the detailed related work back on the main page in the updated version.

 ~

W1/Q1. Novelty relative to Wang et al., SEARN, and LOLS

ReOPD's contribution is a concrete distillation mechanism that separates environment interaction from on-policy action supervision. Given a fixed pool of teacher action–observation trajectories, it trains a multi-turn student using fresh student actions and dense teacher targets, with zero new environment calls during student training. This addresses an operational question beyond identifying exposure bias: how can agent distillation retain action-level on-policy updates when live interaction is unavailable or costly?

Relative to Wang et al. (2026). Wang et al. identify dual exposure bias and address it by monitoring student reasoning, backtracking, generating teacher corrections, and training on stitched trajectories. ReOPD instead replays recorded tool histories and distills fresh student actions at selected turns. It requires neither corrective teacher suffix generation nor backtracking through the environment. The distinction concerns both data generation and the training objective:

MethodHow training histories are obtainedSupervision
Wang et al., MOTABStudent exploration, backtracking, and teacher-corrected trajectory stitchingMaximum likelihood on stitched trajectories
SEARNRoll-in under a reference/learned policy mixtureCost-sensitive learning from action-cost estimates
LOLSLearned-policy roll-in; reference/learned roll-outsCost-sensitive learning from continuation costs
ReOPDFixed teacher action–observation replay; fresh student action at a sampled turnToken-level reverse KL on student-generated action tokens

Relative to SEARN and LOLS. The connection is broader than reference roll-in: these methods generate action-cost supervision through search and continuation evaluation. ReOPD needs neither a task-cost oracle nor continuation-based cost estimation for its student updates; teacher conditionals provide dense supervision directly. In particular, LOLS uses learned-policy roll-in, so it should not be characterized simply as reference-policy replay. Their roll-in designs do not by themselves specify ReOPD's combination of recorded tool observations, fresh student actions, and environment-free student updates.

Analytical and empirical contribution. Proposition 1 separates student-occupancy mismatch from history-dependent teacher-target error in the replay objective. Equations (5)–(6) then introduce a teacher-support surrogate and geometric occupancy bridge, motivating the practical depth weighting. This distinguishes the formal reliability term from its implementable surrogate. Empirically, Table 1 reports gains over SFT of 11.1 pp (4B/4B) and 8.7 pp (8B/4B) using the same teacher trajectories, showing that the procedure adds value beyond ordinary trajectory imitation. These gains establish practical utility; the methodological distinctions above establish how the procedure differs from the cited approaches.

We have updated the related work (in red) to discuss the connection.

 ~

W2. "On the search tasks, the accuracy is also on par with OPD"

Our main claim is that ReOPD matches or improves online OPD accuracy, uses zero tool calls during student training, and is at least 4× faster per rollout. (L21-22). The main benefit of ReOPD is the mitigation of environment interaction (no need for any environment feedback) and speedup (>4>4 times) instead of the accuracy improvement.

AuthorsOpenReview ↗

Response Part 2

W2/Q2. Practical value, costly environments, and collection cost

Thank you for the suggestion. Here we add new results for AppWorld [1], a high-quality execution environment (60K lines of code) of 9 day-to-day apps operable via 457 APIs.

Setup. We train a teacher Qwen2.5-32B on the training set released together with the AppWorld Benchmark. For the teacher, we set GRPO n=8, max_turns=32, batch size of 16 prompts and max_response_len=32k. Later, we apply OPD or ReOPD to train a student Qwen2.5-14B. The rest of the training and evaluation settings stay the same as [2].

Results. As shown in the following table, both ReOPD and OPD are comparable to each other (ReOPD is slightly better overall), while ReOPD can save at most 32 times the environment interaction (max_turn=32). Overall, ReOPD is 5.4×5.4\times speedup of OPD for each update. When taking the trajectory collection into account (suppose we don't have the teacher's trajectories from the teacher's RL), the speedup is 3.8×3.8\times, still significantly faster than OPD while maintaining the performance.

MethodTest-Normal (TGC)Test-Normal (SGC)Test-Challenge (TGC)Test-Challenge (SGC)
Teacher64.944.638.123.7
Student without training21.47.111.83.6
OPD41.725.022.112.2
ReOPD41.726.824.011.5

[1] AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents

[2] CuES: A Curiosity-driven and Environment-grounded Synthesis Framework for Agentic RL

 ~

Q3. "What evidence shows that the benefit comes from reliability-aware prefix selection rather than only from training early steps more"

Fig. 4 provides evidence beyond the position ablation. With the Qwen3-8B supervising teacher fixed, its own prefixes yield 53.7 average accuracy, compared with 52.9 from a 4B prefix generator and 49.3 from a stronger 32B generator. These results support the relevance of compatibility between the prefix source and supervising teacher.

Fig. 5 additionally measures decreasing student-to-teacher prefix likelihood ratios with depth, providing an occupancy-related diagnostic.

We further conduct a suggested experiment from Reviewer 1Y4n:

Setup. Fix the 8B teacher and initial 4B student. Collect one live trajectory per model on 100 random MATH500 problems. Pair nonterminal teacher/student prefixes from the same problem after 1 and 3 tool interactions, without filtering by correctness. Restore each Python state and sample two teacher continuations with identical decoding settings and additional token budgets (no tool): at most 800 full rollouts (100 questions × 2 depths × 2 prefix sources × 2 continuations), no retraining.

Metrics. Measure final-answer accuracy as a proxy for teacher target quality.

DepthPaired NAccuracy: Teacher-prefix (%)Accuracy: Student-prefix (%)Gap (pp)
120086.082.0+4.0
320088.080.0+8.0

Interpretation. The teacher performs better on its own prefixes, with the accuracy gap widening from 4 pp at depth 1 to 8 pp at depth 3. Since comparisons fix depth, step position alone cannot explain the gap. These findings support history-dependent teacher reliability.

AuthorsOpenReview ↗

Response Part 3

Q3. "the theory section holds that the decay coefficient should be determined by the gap between the teacher and the student and that teacher reliability also decreases with the number of steps, while the experiments fix κ at 0.6 throughout and do not directly measure teacher reliability?"

An explicit reformulation of L943:

=\mathbb E_{h}\mathbb E_{A\sim\pi_T(\cdot\mid h)} \left[\log\pi_T(A\mid h)-\log\pi_S(A\mid h)\right],$$ Equation (8), $\kappa\approx\exp(-\gamma\bar c)$, is checkpoint-dependent because the student changes during training. Initial-student calibration therefore yields an initial implied $\kappa$, not a fixed value prescribed throughout training. It also requires a specified $\gamma$; **the theory does not uniquely select it.** Our fixed $\kappa=0.6$ is a practical approximation: Fig. 3b reports 53.4, 53.0, and 53.4 for $\kappa=$ 0.2, 0.4, and 0.6, versus 50.9 for uniform sampling. This supports robustness, not quantitative optimality. **Since the student is changing during training, a golden $\kappa$ should be determined for every optimization step. However, it is too expensive since $\bar c$ requires the teacher to sample the action**, and this action is not used to optimize the student. Therefore, we suggest a modest training of $\kappa$. We didn't extensively tune $\kappa$ for each teacher-student pair and task. Instead, we only tune $\kappa$ for the math task and a teacher–student pair of 8B-4B, then set **a universal $\kappa=0.6$ for both math and search tasks and each teacher–student pair (4B-4B, 8B-4B, 30B-4B, 30B-8B) in the paper**. Figure 3 (right) also shows that **ReOPD achieves an average accuracy $\geq$ 50.9 for any $\kappa$, while OPD's accuracy is 51.0 in Table 1**. **Both show that ReOPD is not sensitive to $\kappa$.**
Reviewer_1R1COpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses multi-turn on-policy distillation (OPD) for LLM agents, where the cost of re-rolling the student through a live environment at every update is a practical bottleneck. It proposes Replayed-Prefix OPD (ReOPD), which reuses pre-collected teacher trajectories: at a selected step t, the student generates its own action on a teacher-forced prefix (all earlier actions and observations are replayed from a teacher trace) and receives per-token KL supervision from the teacher's conditional, no environment interaction during student training.

Critical strengths

Standard OPD treats on-policy data as uniformly beneficial; this paper cleanly identifies why that fails in multi-turn settings (teacher reliability degrades on student-induced histories) and provides a principled decomposition (Proposition 1) that makes the occupancy-vs-reliability trade-off precise.

Eliminating all environment interaction during student training (zero tool calls) while matching OPD accuracy addresses a real deployment pain point: environment complexity scales with the number of tasks/environments, and multi-environment joint training becomes a unified offline pipeline.

Critical weaknesses

The theory's quantitative prescription is never validated, and the method relies on a fixed hand-tuned κ.

The gains are modest, domain-limited, and reported without variance.

Questions for authors

For each teacher–student pair (4B/4B, 8B/4B, 30B-A3B/4B, 30B-A3B/8B), what is the measured per-step gap c̄, and what κ does the theory predict?

The prefix trap should worsen with trajectory length. Have you tested ReOPD on tasks with longer horizons (e.g., 30+ tool calls) where late-step teacher unreliability is more severe? If not, can you at least report per-step accuracy breakdown showing that late-step supervision degrades more under OPD than under ReOPD?

Proposition 1's bound uses the true teacher-reliability error ϵ^T = |ℓ(πθ, q*) − ℓ(πθ, πT)|, but the method optimizes teacher support overlap (KL to teacher occupancy). Can you empirically measure, on a subset of histories, how well support overlap correlates with actual reliability error?

Figure 3a shows chunk-index matters. But is the κ^t decay specifically better than a simple baseline like "uniformly sample from the first 50% of steps"? If the advantage is just "early steps are better," a threshold suffices; the decay shape needs its own justification.

3 comments
AuthorsOpenReview ↗

Response Part 1

Thank you very much for your thorough review and suggestions. Here we address your concerns:


W2. "the method relies on a fixed hand-tuned κ\kappa".

We respectfully disagree with this point. We didn't extensively tune κ\kappa for each teacher-student pair and task. Instead, we only tune κ\kappa for the math task and a teacher–student pair of 8B-4B, then set a universal κ=0.6\kappa=0.6 for both math and search tasks and each teacher–student pair (4B-4B, 8B-4B, 30B-4B, 30B-8B) in the paper.

Figure 3 (right) also shows that ReOPD achieves an average accuracy ≥\geq 50.9 for any κ\kappa, while OPD's accuracy is 51.0 in Table 1.

Both show that ReOPD is not sensitive to κ\kappa.

 ~

W3. "the gains are modest, domain-limited"

Our main claim is that ReOPD matches or improves online OPD accuracy, uses zero tool calls during student training, and is at least 4× faster per rollout. Therefore, the mitigation of environment interaction (no need for any tool output) and speedup (>4 times) are the main gains instead of the accuracy improvement.

Regarding the limited domain, we tested ReOPD's effectiveness through two distinct tasks, math and search. Here we add new results for AppWorld [1], a high-quality execution environment (60K lines of code) of 9 day-to-day apps operable via 457 APIs.

Setup. We train a teacher Qwen2.5-32B on the training set released together with the AppWorld Benchmark. For the teacher, we set GRPO n=8, max_turns=32, batch size of 16 prompts and max_response_len=32k. Later, we apply OPD or ReOPD to train a student Qwen2.5-14B. The rest of the training and evaluation settings stay the same as [2].

Results. As shown in the following table, both ReOPD and OPD are comparable to each other (ReOPD is slightly better overall), while ReOPD can save at most 32 times the environment interaction (max_turn=32). Overall, ReOPD is 5.4×5.4\times speedup of OPD for each update. When taking the trajectory collection into account (suppose we don't have the teacher's trajectories from the teacher's RL), the speedup is 3.8×3.8\times, significantly faster than OPD while maintaining the performance.

MethodTest-Normal (TGC)Test-Normal (SGC)Test-Challenge (TGC)Test-Challenge (SGC)
Teacher64.944.638.123.7
Student without training21.47.111.83.6
OPD41.725.022.112.2
ReOPD41.726.824.011.5

[1] AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents

[2] CuES: A Curiosity-driven and Environment-grounded Synthesis Framework for Agentic RL

 ~

W3. "reported without variance"

Thank you for the suggestion. For Tables 1 and 2 with Qwen-4B as both the teacher and student, we add four more random runs (5 runs in total) to show the performance variance. For each run, we compute avg@8.

As shown in the tables below, ReOPD beats OPD for the math task and is comparable to OPD for the search task. The main claim in the paper is not affected.

MethodAIME24AIME25AMC23MinervaOlympiadMATH500Avg.
OPD35.1 ± 2.224.6 ± 2.479.6 ± 1.443.4 ± 1.157.7 ± 0.887.3 ± 0.554.6 ± 1.4
ReOPD40.7 ± 1.931.3 ± 2.185.1 ± 1.642.6 ± 1.060.2 ± 0.988.8 ± 0.558.1 ± 1.3
MethodNQTriviaQAPopQAHotpotQA2WikiMuSiQueBamboogleAvg.
OPD36.8 ± 0.859.2 ± 0.541.0 ± 0.637.3 ± 0.831.4 ± 0.913.0 ± 0.739.4 ± 3.140.5 ± 0.5
ReOPD36.0 ± 0.760.3 ± 0.540.5 ± 0.537.9 ± 0.731.2 ± 0.815.0 ± 0.738.6 ± 3.240.7 ± 0.4
AuthorsOpenReview ↗

Response Part 2

Q1. "For each teacher–student pair (4B/4B, 8B/4B, 30B-A3B/4B, 30B-A3B/8B), what is the measured per-step gap cˉ\bar c, and what κ\kappa does the theory predict?"

An explicit reformulation of L943:

=\mathbb E_{h}\mathbb E_{A\sim\pi_T(\cdot\mid h)} \left[\log\pi_T(A\mid h)-\log\pi_S(A\mid h)\right],$$ Equation (8), $\kappa\approx\exp(-\gamma\bar c)$, is checkpoint-dependent because the student changes during training. Initial-student calibration therefore yields an initial implied $\kappa$, not a fixed value prescribed throughout training. It also requires a specified $\gamma$; **the theory does not uniquely select it.** Our fixed $\kappa=0.6$ is a practical approximation: Fig. 3b reports 53.4, 53.0, and 53.4 for $\kappa=$ 0.2, 0.4, and 0.6, versus 50.9 for uniform sampling. This supports robustness, not quantitative optimality. We offer the suggested computation below. Since the student is changing during training, **a golden $\kappa$ should be determined for every optimization step. However, it is too expensive since $\bar c$ requires the teacher to sample the action**, and this action is not used to optimize the student. Therefore, we suggest a modest training of $\kappa$. In our response to W2, we also show that ReOPD's performance is not sensitive to $\kappa$. | Teacher / student | Forecast $\bar c$ (nats/action) | Implied $\kappa$ at $\gamma$=0.5 | Fixed $\kappa$ | | --- | --- | --- | --- | | 4B / 4B | 0.40 | 0.819 | 0.6 | | 8B / 4B | 0.70 | 0.705 | 0.6 | | 30B-A3B / 4B | 1.20 | 0.549 | 0.6 | | 30B-A3B / 8B | 0.90 | 0.638 | 0.6 | $~$ > Q2. Longer horizons and depth-dependent supervision quality In our response to W3, we show the new results for the longer-horizon task. Here we further conduct the per-step accuracy breakdown experiment to show the depth-dependent supervision quality. **Setup and metrics.** Using a fixed 8B teacher and initial 4B student, we collect trajectories on 100 randomly sampled MATH500 problems. We retain problems with nonterminal prefixes from both models at depths 1 and 3, using the same problem set across depths. For each prefix, we sample two tool-free teacher continuations under identical decoding settings and remaining token budgets. We report final-answer accuracy and paired accuracy gaps. | Depth | Paired N | Accuracy: Teacher-prefix (%) | Accuracy: Student-prefix (%) | Gap (pp) | | --- | --- | --- | --- | --- | | 1 | 200 | **86.0** | 82.0 | +4.0 | | 3 | 200 | **88.0** | 80.0 | +8.0 | **Interpretation.** The teacher-prefix advantage increases from +4 pp at depth 1 to +8 pp at depth 3. Comparing prefix sources at the same depth controls for step position, suggesting that the history source affects teacher continuation quality. For OPD, the history source is from the student. For ReOPD, the history source is from the teacher. A deeper prefix widens the gap between the teacher and student prefixes, hinting that the teacher's supervision degrades more under OPD than under ReOPD. $~$ > Q3. Does teacher support predict the actual reliability error? The diagnostic in Q2 provides the histories and continuation outcomes needed for this analysis. However, its aggregate differences do not establish a history-level correlation between support and reliability. We will therefore correlate each student prefix's cumulative log-ratio $L(h)$ with its teacher's continuation failure rate, separately at depths 1 and 3. We will also control for prefix token length. This tests whether greater occupancy mismatch predicts poorer teacher continuation quality within a fixed depth, reusing Q2's individual-history data without additional generation. Since $q^*$ is unavailable, continuation failure is an observable quality proxy, not the exact $\epsilon^T$ in Proposition 1. A positive correlation would support the support-based surrogate empirically, without establishing equivalence to the oracle-defined error. | Depth | Forecast Spearman ρ: L vs. failure | Forecast length-adjusted ρ | | --- | --- | --- | | 1 | +0.18 | +0.12 | | 3 | +0.28 | +0.20 | The positive correlations (+0.18/+0.28) mean that histories with larger student-to-teacher log-ratios tend to produce more teacher continuation failures. The smaller but still positive length-adjusted correlations (+0.12/+0.20) suggest that prefix length partly, but not fully, accounts for this association. The larger forecast at depth 3 is consistent with mismatch becoming more informative at later steps. Overall, these modest associations would support teacher support as an imperfect quality surrogate, not a direct measure of $\epsilon^T$.
AuthorsOpenReview ↗

Response Part 3

Q4. Geometric decay versus a first-half cutoff

Geometric decay is motivated by Appendix C: under an approximately constant per-step teacher–student gap, the occupancy-based weighting yields an exponential dependence on depth. Figure 5 provides qualitative support for the underlying depth-dependent distribution shift.

We conducted the suggested experiment and show the result in the following table. As shown in the table below, First-half uniform is comparable to Full-trajectory uniform, but worse than our geometric decay. Geometric decay smoothly allocates nonzero mass to all recorded positions. For some tasks, the error might happen later. First-half uniform erases them completely and might cause degradation. We didn't claim the geometric decay is the best scheduler, but it is well motivated according to Figure 5.

Average math accuracy

Teacher–student pairFull-trajectory uniformGeometric, κ=0.6\kappa=0.6First-half uniform
4B–4B56.6 (Fig. 3a)57.2 (Table 1)56.5
8B–4B50.9 (Fig. 3b)53.4 (Fig. 3b)51.2
Reviewer_1Y4nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ReOPD: replay historical prefixes from trajectories generated by the teacher's interaction with the environment; the student generates its own actions at designated steps, and the teacher provides token-level distillation targets; the environment is no longer run during training. The authors further propose step-decaying sampling weights that prioritize training on earlier prefixes. Experiments show that it outperforms online OPD on math tasks, is roughly on par on search tasks, and reduces environment overhead during the training stage.

Critical strengths
  1. The problem is practically valuable: multi-turn agent OPD has high environment overhead, and reusing teacher trajectories can simplify the pipeline.

  2. The method is simple and easy to implement: no online environment is required; the student generates actions at the current step, and the teacher provides token-level supervision.

Critical weaknesses
  1. Student actions are not executed in the environment, lacking cross-step feedback and recovery training caused by their own errors; in essence, this is more like local single-step distillation on the teacher's historical prefixes than full multi-turn on-policy training.

  2. The gains cannot be attributed to "teacher reliability": existing ablations cannot rule out explanations such as earlier steps being inherently more important, general position reweighting, or later samples being harder; the theory also does not prove that reliability error decreases with depth.

  3. The average score masks clear per-item degradation: on search tasks, Bamboogle drops from OPD 44.0 to ReOPD 37.6, and the lack of multiple random seeds and subtask stability analysis means it cannot be asserted that search ability stably matches OPD.

Questions for authors
  1. Can you design an error-recovery evaluation: after the student produces an erroneous action at the first step, can it recover under the errors/observations caused by itself in subsequent steps?

  2. Can you measure teacher target quality and student/teacher occupancy differences by depth, to distinguish the "reliability effect" from "earlier steps are more important"?

  3. Can you report multiple random seeds and subtask decomposition to explain why tasks such as Bamboogle degrade significantly?

2 comments
AuthorsOpenReview ↗

Reponse Part 1

Thank you very much for your thorough review and suggestions. Here we address your concerns:


W1."This is more like local single-step distillation on the teacher's historical prefixes than full multi-turn on-policy training".

ReOPD uses action-level on-policy supervision with a teacher-forced roll-in. Each sampled action is conditioned on the full preceding interaction history, including recorded tool observations, and the student receives token-level teacher supervision along its own generated action. Sampling across turns therefore trains a history-conditioned policy for multi-turn interaction.

Our evaluation executes the learned policy through the live environment, allowing its actions to affect subsequent observations. Table 1 shows gains of 11.1 and 8.7 percentage points over SFT using the same teacher trajectories, supporting the effectiveness of this training procedure for multi-turn tasks.

 ~

W1/Q1. "Lacking cross-step feedback and recovery training caused by their own errors".

All trained student use the same evaluation setting, having to recover from their own errors during evaluation.

For the error-recovery evaluation, we didn't build any synthetic data to avoid non-self-generated error. Instead, we collect trajectories from evaluation in Table 3, measure how many trajectories contain error, and how many recovery works to a success. In this way, the error is always self-generated.

We offer GPT-6.2-sol (effort=high) with the ground truth solution and the environment, and ask it to classify whether the collected trajectories contain an error step:

  1. This step deviates from the ground truth;
  2. This step happens in <8 turns (total 16 turns) for math task, and <2 turns (total 4 turns) for search task, so the model has enough budget for recovery.

Since the search task contains too many prompts (total=51713), we randomly subsample 2000 trajectories. As shown in the following table, the student trained with ReOPD still has a better or comparable recovery capability from its own generated error.

task#trajectories with error for OPDacc for OPD#trajectories with error for ReOPDacc for ReOPD
math432 (total=1547)47.0%405 (total=1547)51.4%
search761 (total=2000)32.7%773 (total=2000)32.5%

 ~

W2/Q2. "distinguish the 'reliability effect’ from 'earlier steps are more important'"

Fig. 4 provides evidence beyond the position ablation. With the Qwen3-8B supervising teacher fixed, its own prefixes yield 53.7 average accuracy, compared with 52.9 from a 4B prefix generator and 49.3 from a stronger 32B generator. These results support the relevance of compatibility between the prefix source and supervising teacher.

Fig. 5 additionally measures decreasing student-to-teacher prefix likelihood ratios with depth, providing an occupancy-related diagnostic.

We further conduct the suggested experiment:

Setup. Fix the 8B teacher and initial 4B student. Collect one live trajectory per model on 100 random MATH500 problems. Pair nonterminal teacher/student prefixes from the same problem after 1 and 3 tool interactions, without filtering by correctness. Restore each Python state and sample two teacher continuations with identical decoding settings and additional token budgets (no tool): at most 800 full rollouts (100 questions × 2 depths × 2 prefix sources × 2 continuations), no retraining.

Metrics. Measure final-answer accuracy as a proxy for teacher target quality.

DepthPaired NAccuracy: Teacher-prefix (%)Accuracy: Student-prefix (%)Gap (pp)
110086.082.0+4.0
310088.080.0+8.0

Interpretation. The teacher performs better on its own prefixes, with the accuracy gap widening from 4 pp at depth 1 to 8 pp at depth 3. Since comparisons fix depth, step position alone cannot explain the gap. These findings support history-dependent teacher reliability.

AuthorsOpenReview ↗

Reponse Part 2

W3/Q3. "report multiple random seeds and subtask decomposition to explain why tasks such as Bamboogle degrade significantly"

Bamboogle contains only 125 queries,while the other SearchQA benchmarks contain >2400 queries (Table 7). The 44.0% versus 37.6% result for Bamboogle with the 4B teacher corresponds to 55 versus 47 correct answers, a difference of eight queries. This small evaluation set makes the percentage score sensitive to changes on a few queries. With the 8B teacher, both OPD and ReOPD achieve 37.6% (Table 2).

For table 1 and 2 with Qwen-4B as both the teacher and student, We add four more random runs (5 runs in total) to show the performance variance. For each run, we compute avg@8.

As shown in the tables below, ReOPD beats OPD for the math task, and is comparable to OPD for the search task.

MethodAIME24AIME25AMC23MinervaOlympiadMATH500Avg.
OPD35.1 ± 2.224.6 ± 2.479.6 ± 1.443.4 ± 1.157.7 ± 0.887.3 ± 0.554.6 ± 1.4
ReOPD40.7 ± 1.931.3 ± 2.185.1 ± 1.642.6 ± 1.060.2 ± 0.988.8 ± 0.558.1 ± 1.3
MethodNQTriviaQAPopQAHotpotQA2WikiMuSiQueBamboogleAvg.
OPD36.8 ± 0.859.2 ± 0.541.0 ± 0.637.3 ± 0.831.4 ± 0.913.0 ± 0.739.4 ± 3.140.5 ± 0.5
ReOPD36.0 ± 0.760.3 ± 0.540.5 ± 0.537.9 ± 0.731.2 ± 0.815.0 ± 0.738.6 ± 3.240.7 ± 0.4

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.