acceptodds
Under review as a conference paper at ICLR 2027

TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Training

Abstract

On-policy distillation (OPD) trains a student policy by matching a stronger teacher on the student's own trajectories, offering a promising framework for agent training. However, its application to long-horizon agentic tasks remains insufficiently explored. We identify two key inefficiencies in vanilla agent OPD: (1) full-horizon rollouts often waste wall-clock resources on tail turns that provide weak and noisy KL supervision, and (2) trajectory-level KL objectives concentrate most of the loss on shallow tokens, leaving deeper decision turns under-trained once initial behaviors are aligned. To address these challenges, we propose TurnOPD, a turn-level budgeting strategy for efficient on-policy distillation of long-horizon agents. TurnOPD consists of two budget controllers: adaptive rollout-depth budgeting, which uses probe-based turn statistics to determine rollout depth, and progressive turn-normalized loss budgeting, which gradually shifts KL weighting from token-level to turn-balanced supervision. Experiments on ALFWorld, WebShop, and Multi-Hop Search with task-specialized teacher models show that TurnOPD achieves superior validation accuracy under equal wall-clock training budgets and advances the accuracy–time frontier beyond vanilla OPD.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Large Language Model Agents, Long-Horizon Agen Training, Training Efficiency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bzq8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the issue of resource allocation in long-horizon agent distillation by proposing an approach that combines adaptive trajectory depth with progressive round normalization, demonstrating a favorable trade-off between performance and training time. The work’s strengths lie in its precise problem diagnosis, concise methodology, and clear practical benefits. Its primary limitations include an insufficient comparison with related methods to show the unique contributions, a weak connection between the theoretical framework and the control rules, and a lack of robust evidence for certain design choices. Overall, this is an incremental study with practical value; however, its unique contributions require further, more targeted experimental validation.

Critical strengths
  • The paper analyzes teacher supervision signals, trajectory survival counts, and loss distribution at the turn level, identifying two key issues: the inefficiency of collecting long trajectories and the over-concentration of loss in early stages. Adaptive depth control and progressive turn normalization are introduced to address these, resulting in a well-structured argument.
  • Across three tasks and four teacher-student configurations, TurnOPD consistently achieves the highest evaluation scores for a given training duration. Notably, for the ALFWorld-1.7B task, training time was reduced from 4.42 hours to 1.93 hours while simultaneously improving performance. Ablation studies further confirm that the two proposed modules play complementary roles.
Critical weaknesses
  • The boundaries of innovation and experimental advantages relative to existing work are not fully established. Concepts such as trajectory length control, dynamic truncation, and turn reweighting have been explored in prior research. The main experiments primarily compare "Vanilla OPD" against "TCOD-F2B," lacking direct comparisons with the most closely related existing methods—or simple combinations thereof—making it difficult to isolate the specific contribution of the authors' unique control rules.
  • A gap exists between the theoretical rationale and the actual control metrics employed. The "KL compression" proposition suggests that a smaller KL divergence does not necessarily imply a smaller gap in true capability; yet, the method still uses raw KL divergence to construct its supervision-value metric, a choice that warrants further justification. Furthermore, while the existence of an "optimal depth" is discussed, it is not proven that the proposed controller can effectively approach this optimum.
  • Insufficient evidence regarding the efficacy of certain key design elements and the stability of the results. "Progressive mixing" yields only a 0.1-point improvement over direct turn normalization, which is arguably insufficient to justify the necessity of a progressive scheduling approach. Additionally, the reported standard deviations are derived from four evaluation checkpoints rather than multiple independent training runs; consequently, the robustness of the observed performance differences remains to be verified.
Questions for authors
  • Compared to existing methods such as TCOD, SOD, POPD/TOPD, and ADWIN, what is the core novel mechanism introduced by TurnOPD? Can the authors demonstrate that its performance gains stem from something more than just simple trajectory truncation and loss reweighting?
  • Given that the paper notes how "raw KL" in deeper layers may be affected by context compression—thereby failing to accurately reflect policy divergence—why do the authors still use raw KL to construct HeffH_{\mathrm{eff}}? Furthermore, can they empirically validate the relationship between this metric and either task success rates or gradient utility?
  • Since "progressive round normalization" yields an improvement of only about 0.1 points over "fixed round normalization," can the authors provide evidence to justify the necessity and stability of this design?
4 comments
AuthorsOpenReview ↗

Response to Reviewer bzq8 (1/3)

We thank Reviewer bzq8 for the thoughtful and structured review, and for recognizing the clarity of our turn-level analysis, our consistent efficiency and task performance improvements across benchmarks, and the complementary roles of our two proposed modules. Below, we address each concern and question in detail.


W1 & Q1: Core Mechanisms and Distinction from TOPD, POPD, ADWIN, TCOD, and SOD

  • Versus TOPD, POPD, and ADWIN: Methods like TOPD and ADWIN were designed primarily for single-output generation, where control signals operate at the token or sequence level. In contrast, our controllers operate at turn boundaries and explicitly depend on trajectory-level success. This signal does not exist in single-output tasks, but is essential for interactive agents. Multi-turn agent environments are substantially more complex: tools exhibit sequential and topological dependencies, and OPD requires the student to learn context-dependent action decisions (e.g., tool invocations) across dynamic states. Furthermore, agent rollouts are time-consuming, consume large token budgets, and suffer from severe long-tail delays where a few struggling trajectories hold back the entire training batch. Consequently, single-output OPD techniques cannot be naively ported to multi-turn agent settings.
  • Versus TCOD: Simple scheduled truncation is insufficient and can even be detrimental. Our main paper already includes a direct comparison with TCOD. TCOD-F2B uses a predefined linear truncation schedule to speed up training. While fast on ALFWorld-1.7B (1.87 h, 2.37× speedup over vanilla), its Same-Step Avg@4 drops to 80.06, lagging behind vanilla OPD's 83.00 (-2.94). On ALFWorld-4B, evaluating on the unconstrained test set reveals that TCOD-F2B shrinks successful episode depth by 7.0% and degrades accuracy by 5.3 points. In contrast, TurnOPD matches or exceeds vanilla OPD at the same step budget while training substantially faster per step. This demonstrates that for an equivalent truncation ratio, differing control rules yield opposite outcomes: the gain comes from our adaptive control logic, not from the mechanical act of truncation itself.
  • Versus SOD: In our controlled comparison, turn reweighting alone neither meaningfully boosts accuracy nor improves efficiency. Because SOD only reweights step-level losses without adjusting rollout depth, it inherits the full rollout cost of vanilla OPD—leaving the efficiency bottleneck in multi-turn agent training completely unaddressed.

Below are empirical results comparing these approaches under identical budgets on ALFWorld-1.7B:

MethodWall-clock (100 steps)Same 100-Step Avg@4Least Time Avg@4 (~1.87h)
Vanilla OPD4.42 h83.073.5
TCOD-F2B1.87 h80.180.1
SOD4.51 h82.673.2
TurnOPD1.93 h86.385.6

Whether evaluated under equal step budgets or equal wall-clock time, TurnOPD consistently outperforms both truncation-only and reweighting-only baselines.


W2(1) & Q2: Rationale for Using Raw KL Despite Context Compression

Thanks for raising this insightful question, which gives us an opportunity to clarify our design rationale.

  1. In multi-turn OPD, KL divergence is indeed not an unconditionally faithful proxy for behavioral gaps. We explicitly noted this in the first part of the paper, and our empirical observations confirm the compression phenomenon: across all four setups, depth estimates derived from raw KL (HeffH_{\mathrm{eff}}) are systematically and substantially smaller than HcovH_{\mathrm{cov}} (Appendix G.2, Figure 8).
  2. For this exact reason, the max operator in Hctrl=max⁡(Heff,Hcov)H_{\mathrm{ctrl}} = \max(H_{\mathrm{eff}}, H_{\mathrm{cov}}) is not an arbitrary combination. Its precise role is to safeguard against compressed raw KL driving rollout depth excessively low. As noted in the paper, "Although raw KL may not directly measure behavioral error, it indicates supervision value and cost." Rather than treating raw KL as an absolute estimate of required depth, we use it as an efficiency signal to lift the rollout lower bound, while the coverage term HcovH_{\mathrm{cov}} directly counterbalances its known bias. The perceived gap between the theoretical rationale and control metrics is thus explicitly managed by this design rather than overlooked.
AuthorsOpenReview ↗

Response to Reviewer bzq8 (2/3)

W2(2): Empirical Support for Approaching Optimal Depth and Pareto Optimality

We appreciate the opportunity to clarify this. Taking ALFWorld-1.7B as a representative case:

  1. While a theoretical optimal depth H∗H^* exists, identifying it analytically or through post-hoc sweeps is practically infeasible; nonetheless, TurnOPD lies on the empirical Pareto frontier across all observed depth tiers:
    • Theoretically, H∗H^* varies dynamically with the implicit capability gap between student and teacher at each training step, task characteristics, the number of tools, tool coupling, and hardware constraints.
    • Empirically, identifying an exact optimum in noisy environments is elusive. Viewed from this perspective, the sensitivity grid in Figure 9 effectively sweeps through different late-stage rollout horizons HH across several tiers (late-stage depths around 7, 10, 14, 21–24, 38, plus unconstrained H=80H=80). The empirical landscape displays an elbow followed by a broad plateau: for depth ≥14\ge 14, performance differences are minor, with the entire spread across Avg@4 being just 1.27 points (comparable to evaluation noise).
    • TurnOPD naturally converges to the optimal tier (H≈21–24H \approx 21\text{--}24) while achieving the best wall-clock efficiency within that tier. Across the sweep, no point is simultaneously faster and more accurate than TurnOPD; TurnOPD lies squarely on the Pareto frontier.
Late-Stage Depth HH (steps 50–100)Same 100-Step Avg@4Wall-clock (100 steps)
7.081.381.464 h
7.779.941.531 h
9.881.171.709 h
14.085.081.752 h
20.886.352.037 h
23.6 (adaptive 5.4→23.65.4 \rightarrow 23.6)86.291.928 h
38.084.962.132 h
80 (unconstrained)85.082.586 h
  1. The optimal depth shifts dynamically as training progresses and cannot be adequately captured by simple linear heuristics. TCOD-F2B serves as a clear illustration: using a data-agnostic, front-to-back linear schedule over the same 100 steps results in a 6.23-point deficit under essentially identical wall-clock time:
MethodCharacteristicWall-clock (100 steps)Same-Step Avg@4
TCOD-F2BNon-adaptive linear (front-to-back) truncation1.866 h80.06
TurnOPDAdaptive coverage-aware control1.928 h86.29
  1. We do not claim that TurnOPD analytically solves for the exact theoretical H∗H^*, which is intractable to prove in practice. However, our empirical sweep demonstrates that within observable measurement resolution, TurnOPD consistently outperforms non-adaptive heuristics (both fixed truncation and linear decay schedules).

W3(1) & Q3: Value and Mechanism of Progressive Turn Normalization

Thanks for this feedback; this distinction warrants clarification. Below, we compare the step-by-step training trajectory of progressive blending against hard turn normalization on ALFWorld-1.7B:

Training StepProgressive Blending AccuracyHard Turn Normalization Accuracy
Step 100.420.17
Step 3055.5816.00
Step 5073.5874.92
Step 7081.2583.67
Step 9086.6785.08

As shown in the table, progressive blending achieves a much faster cold start (55.58% vs. 16.00% at step 30), even though both converge to a similar accuracy by step 90. The underlying mechanism directly connects to our exploratory analysis: early in training, student rollouts are short and deeper turns carry scarce supervisory signals. Hard turn normalization forces uniform weight onto uninformative deeper turns, stalling policy improvement. Progressive blending begins close to trajectory normalization, focusing gradient updates where supervision is reliable, and smoothly transitions to turn-level alignment as trajectories mature. Progressive blending thus combines early sample efficiency with late-stage turn alignment, ensuring stability throughout training.

AuthorsOpenReview ↗

Response to Reviewer bzq8 (3/3)

W3(2): Evaluation Checkpoint Variance and Multi-Seed Validation

  1. Rationale for reporting checkpoint mean and standard deviation: In individual training runs, validation scores across adjacent steps naturally fluctuate. In practical deployments, practitioners need to know whether an optimization algorithm maintains stable performance across a training window rather than producing erratic checkpoints. We therefore computed the mean and standard deviation over consecutive checkpoints to reliably reflect sustained performance.
  2. Independent multi-seed training runs: Constrained by compute budget, running multi-seed sweeps across all configurations was not feasible. However, to directly verify training stability, we conducted multi-seed evaluations on ALFWorld with Qwen3-1.7B (n=3n=3). Same-Step evaluates at 100 training steps, while Same-Time aligns to the ~1.87 h training budget from the paper:
MethodnnWall-clock (h)SpeedupSame-Step Avg@3 (multi-seed)Same-Time Avg@3 (multi-seed)
Vanilla OPD34.42 ± 0.231.00×82.5 ± 0.6973.3 ± 1.35
TurnOPD31.93 ± 0.112.29× ± 0.1386.9 ± 1.1585.7 ± 0.85

Across independent training seeds, TurnOPD consistently achieves a ~2.29× wall-clock speedup and superior accuracy under both step and time budgets, confirming that the observed gains are statistically robust.

The additional experiments support your method. However, I think the paper needs a major revision and reorganize. I will remain my sccore.

Reviewer_73JyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to allocate interaction and supervision budgets in on-policy distillation (OPD) for multi-turn agents. TurnOPD adapts rollout depth using a KL centroid weighted by surviving-trajectory coverage, together with a lower bound based on successful trajectory lengths. It also gradually shifts loss weighting from token-based allocation toward equal weighting across turns. Experiments compare TurnOPD with vanilla OPD and TCOD-F2B on ALFWorld, WebShop, and Multi-Hop Search. TurnOPD achieves the highest score in all four configurations under equal wall-clock time, with a maximum training speedup of 2.29 times over vanilla OPD.

Critical strengths
  1. Training efficiency improves clearly. On ALFWorld-1.7B, TurnOPD reduces the wall-clock time for 100 training steps from 4.42 to 1.93 hours while improving Same-Step Avg@4 from 83.00 to 86.29. It also outperforms TCOD-F2B under the same time budget. Reporting results at both equal training steps and equal wall-clock time is necessary for a method that changes the amount of computation per step. Efficiency gains are also observed on WebShop and Multi-Hop Search, so the results are not confined to one environment.

  2. The analysis of supervision across turns is useful. Table 1 reports KL, surviving-trajectory coverage, and loss share at deeper turns, helping explain why later decisions contribute relatively little to the overall objective. The ablations in Table 3 show that adjusting depth alone substantially reduces training time with little change in accuracy. Blending the loss weights alone improves accuracy. These results support studying interaction depth and supervision weighting separately, rather than focusing only on the total loss over complete trajectories.

Critical weaknesses
  1. The added value of the specific design choices is not yet established. The main table compares only against vanilla OPD and TCOD-F2B. Fixed truncation tuned on a validation set, following TOPD with any adaptation to turn boundaries stated explicitly, would help establish whether adaptive depth is necessary. A comparison with SOD without GRPO is also needed to assess the weighting strategy under comparable supervision. Appendix G.2 states that the coverage constraint dominates depth selection in ALFWorld once successful trajectories emerge, which makes an HcovH_{\mathrm{cov}}-only controller an important comparison. Table 4 reports 85.0 for uniform turn weighting and 85.1 for linear blending, leaving the additional value of the progressive schedule unclear. The efficiency gain is useful, but these missing comparisons leave it unclear whether the proposed designs are needed to achieve it.

  2. The evidence for preserving long-horizon behavior is limited. Appendix F states that WebShop typically requires only 4–12 actions, while the search evaluation mixes direct retrieval and multi-hop questions. The breakdowns by task category and source are useful, but they do not directly establish whether truncating training trajectories preserves later-stage decisions and error recovery on long tasks. This matters because those are precisely the behaviors that may receive less training when rollouts are shortened. The authors should provide evidence addressing this concern or narrow the claims about long-horizon applicability. Together with the unresolved contribution of the specific design choices, this gap is the main reason for my current weak-rejection recommendation.

Questions for authors
  1. Evidence that could change my assessment of the method. How does TurnOPD compare with tuned fixed truncation, an HcovH_{\mathrm{cov}}-only controller, and SOD without GRPO, using the same teacher and student initialization under equal time budgets? What benefit does progressive blending provide beyond uniform turn weighting, given the 85.0 versus 85.1 results in Table 4? Please also clarify whether top-50 reverse-KL renormalizes the distributions over the truncated support. These answers would help establish whether the proposed controller and weighting schedule provide gains beyond simpler or existing alternatives.

  2. Clarifications affecting the interpretation and scope of the results. Why do ALFWorld and WebShop use Qwen3 students with a Qwen3-8B-GRPO teacher, while Multi-Hop Search uses Qwen3.5-2B with a Qwen3.5-9B-GRPO teacher? Please also confirm whether the Qwen3.5-1.7B/4B names in Appendix G.1 are errors, given the Qwen3 labels in Tables 2 and 7. The configurations are aligned within each task, but their differences limit attributing cross-task patterns to task characteristics alone.

    Given the paper’s focus on long-horizon agent training, how does trajectory truncation affect performance on longer episodes, particularly later-stage decisions and recovery from errors? If this is not clear from the current experiments, would you narrow the long-horizon claims accordingly? Evidence could come from analyses of sufficiently long trajectories in the current tasks or an additional evaluation such as Terminal-Bench 2.1 or SWE-Marathon. The named benchmarks are examples of ways to address this concern, rather than the only acceptable response. Any comparison should use the same agent framework, tools, and execution budget.

Suggestions for future work. The current students are at most 4B. Experiments with Qwen3.5-9B or Qwen3.5-27B, and with a newer model such as Qwen3.8-27B, would help test whether the gains extend to larger and newer students. These are suggestions for broader validation, not independent grounds for the current rating.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 73Jy (1/2)

We thank Reviewer 73Jy for the thorough and perceptive review. We appreciate the recognition of our efficiency gains under equal time budgets and our analysis of turn-level supervision decay. Below, we address each concern with targeted ablations (fixed truncation, HcovH_{\mathrm{cov}}-only, SOD without GRPO, progressive blending dynamics) and comprehensive trajectory depth evaluations verifying long-horizon preservation.


W1 & Q1: Isolating the Added Value of Design Choices (Fixed Truncation, HcovH_{\mathrm{cov}}-only, SOD, and Progressive Blending)

To evaluate our design choices against simpler and existing alternatives under comparable supervision, we conducted additional experiments on ALFWorld-1.7B evaluated at both 100 training steps and equal wall-clock time (~1.87 h):

Method / ConfigurationInitial HHLate HHWall-clock (100 steps)Same 100-Step Avg@4Equal Time (~1.87h) Avg@4
Vanilla OPD80804.42 h83.0073.50
HcovH_{\mathrm{cov}}-only Controller5023.62.15 h85.7084.20
HeffH_{\mathrm{eff}}-only Controller44.01.41 h76.4081.00
Fixed Truncation H=8H = 8881.59 h80.2081.80
Fixed Truncation H=16H = 1616161.86 h85.1085.10
SOD (without GRPO)80804.51 h82.6073.20
TurnOPD5.423.61.93 h86.3085.60

Takeaways from the comparison:

  • Versus Fixed Truncation (H=16H=16): While tuning fixed truncation achieves 85.10, TurnOPD's adaptive schedule (5.4→23.65.4 \rightarrow 23.6) reaches 86.30 / 85.60. Adaptively expanding depth allows the policy to aggressively save compute during early exploration and capture necessary multi-step recovery once basic capabilities emerge.
  • Versus SOD (without GRPO): Applying turn-reweighting to full rollouts without depth control yields 82.60 (taking 4.51 h). Reweighting full trajectories alone cannot overcome the computational bottleneck of long-tail rollouts.
  • Versus HcovH_{\mathrm{cov}} only: Omitting HeffH_{\mathrm{eff}} fails to exploit high-KL early turns, increasing training time to 2.15 h and achieving lower performance under equal time (84.20 vs. 85.60).

Benefit of Progressive Blending over Uniform Turn Normalization:

Thank you for your feedback; we agree that this point warrants further clarification. We provide a detailed comparison at the same training step, as shown in the table below:

Training StepProgressive Blending AccuracyHard Turn Normalization Accuracy
Step 100.420.17
Step 3055.5816.00
Step 5073.5874.92
Step 7081.2583.67
Step 9086.6785.08

As shown in the table, blending takes off much faster early on (55.58% vs. 16.00% at step 30), even though both converge to a similar level by step 90. Early in training, rollouts are short and deeper turns carry little signal; hard turn normalization puts equal weight on these noisy turns, which stalls progress. Blending starts closer to trajectory-norm to spend the update budget where supervision is actually solid, then smoothly transitions to turn-level alignment as trajectories mature. In short, it gives us the best of both worlds: fast early takeoff and balanced deep-turn supervision later on.

Clarification on Top-50 Reverse-KL Support:

Top-50 reverse-KL does not renormalize over the truncated support. The teacher outputs log-probabilities restricted to its top-50 tokens, and the student's log-probabilities are evaluated directly at those token indices without computing an independent top-k normalization constant.

AuthorsOpenReview ↗

Response to Reviewer 73Jy (2/2)

W2 & Q2 (1): Empirical Evidence on Preserving Long-Horizon Decisions

We consider this to be the most significant point of critique. We agree that it is crucial—in fact, we anticipated this issue from the very beginning; it was, in truth, the original design motivation behind adaptive rollout depth. We address this directly by presenting statistics on the trajectory depth—recorded using a validation set without truncation—as follows:

Benchmark & ModelTraining MethodMean Train DepthMax Train DepthMean Success Depth (Eval)Max Eval DepthSame-Step Avg@4
ALFWorld-1.7BVanilla OPD21.926918.677683.00
TCOD-F2B14.58 (-33%)2618.61 (-0.3%)7880.06
TurnOPD13.22 (-40%)2618.14 (-2.8%)7386.29
ALFWorld-4BVanilla OPD19.626217.668091.81
TCOD-F2B13.14 (-33%)2616.43 (-7.0%)7686.50
TurnOPD15.22 (-22%)2917.84 (+1.0%)80 (=cap)92.21
WebShop-1.7BVanilla OPD8.58296.443081.65
TCOD-F2B5.43 (-37%)116.43 (-0.2%)3081.66
TurnOPD5.56 (-35%)96.12 (-5.0%)30 (=cap)82.80
Multi-Hop-2BVanilla OPD6.14346.107947.82
TCOD-F2B5.55 (-10%)236.33 (+3.8%)6747.77
TurnOPD4.17 (-32%)106.11 (+0.2%)4947.24

Three key findings:

  1. Cutting training rollouts by 22%–40% does not make the student policy shallower. On the unconstrained evaluation set, the average depth of successful episodes changes by less than ±5% across all four configurations (and is actually slightly deeper on ALFWorld-4B at +1.0% and Multi-Hop at +0.2%). The student preserves its ability to solve long tasks.
  2. What gets pruned is the unproductive long tail, not typical trajectories. Maximum training turns drop drastically (69->26, 62->29, 29->9, 34->10), while median evaluation depth remains virtually unchanged. This is exactly what the method was designed to do: eliminate compute-draining failure loops that offer almost zero useful gradient signal.
  3. However, this conclusion does not automatically hold true; TCOD-F2B provides a telling counter-example. On ALFWorld-4B, TCOD-F2B also cuts training turns by 33%, but its successful episode depth shrinks by 7.0% and accuracy drops by 5.3 points. Schedule-based truncation does indeed tend to result in shallower policies, whereas coverage-aware truncation is comparatively more moderate. Overall, this suggests that the performance gains stem from the control rules themselves rather than the act of "truncation." This insight motivates the development of an adaptive, truncation-aware scheme—one capable of deciding, at any given moment, whether to further curtail the rollout horizon or to retain more of it. This serves as a key motivation for the design of our adaptive controller.

Q2 (2) : Teacher-Student Model Family Choice and Typo Correction

  1. Why Qwen3 was used for ALFWorld/WebShop vs. Qwen3.5 for Multi-Hop Search:

    This choice was based on considerations regarding the availability and capability boundaries of the teacher models. We initially experimented with the Qwen3 series for the Multi-Hop Search task, but found that a GRPO teacher trained on Qwen3 achieved a success rate of only about 0.3. This task appears challenging for this model family; without additional effort—such as an initial SFT "cold start" phase followed by RL—using it as a teacher fails to create a meaningful distillation setup capable of demonstrating clear performance gaps. Consequently, to ensure a sufficient capability gap between the student and the teacher—thereby facilitating the measurement of our method's gains over a vanilla baseline—we shifted to using a Qwen3.5-2B student paired with a Qwen3.5-9B-GRPO teacher. This choice offers an additional advantage: by validating our approach across two distinct model families, we demonstrate that the gains provided by TurnOPD are not dependent on any specific model family.

  2. Typo in Appendix G.1:

    We confirm that the Qwen3.5 label in Appendix G.1 for ALFWorld was indeed a typographical error. Thank you for your careful review and feedback! We will correct it in the revised version.

Suggestion for Future Work

We agree that testing on larger student models is a valuable direction. While our current setups cover two model generations (Qwen3 and Qwen3.5) with students up to 4B (and teachers up to 9B), extending TurnOPD to larger models or newer releases is a natural next step, and we plan to investigate this in follow-up work.

Reviewer_SxqaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TurnOPD, a framework designed to improve the efficiency and effectiveness of on-policy distillation (OPD) for long-horizon LLM agents. The authors identify two limitations of existing OPD methods: inefficient computation on deep rollout turns that provide limited distillation signals, and imbalanced optimization caused by token-level loss aggregation that disproportionately emphasizes early turns. To address these issues, TurnOPD introduces Adaptive Rollout-Depth Budgeting, which dynamically adjusts rollout horizons based on KL divergence and successful trajectory coverage, and Progressive Turn-Normalized Loss Budgeting, which gradually shifts from token-level to turn-level loss weighting to balance supervision across different turns. Experiments on ALFWorld, WebShop, and Multi-Hop Search demonstrate improvements in both task performance and training efficiency over standard OPD baselines.

Critical strengths
  1. Important and Practical Problem. This paper addresses the important challenge of improving the training efficiency and effectiveness of OPD for long-horizon LLM agents. The analysis of diminishing KL signals and imbalanced supervision across turns provides useful insights into the limitations of existing OPD approaches.

  2. Simple and Effective Design. The proposed methods are straightforward to implement and introduce minimal additional complexity without requiring extra critic models or modifications to the underlying distillation objective. The experimental results demonstrate consistent improvements in training efficiency and task performance across multiple benchmarks, supporting the practical value of the approach.

Critical weaknesses
  1. Excessive hyperparameters. TurnOPD requires setting several hyperparameters, including Hmin⁡H_{\min}, Hmax⁡H_{\max}, KwarmK_{\mathrm{warm}}, rprober_{\mathrm{probe}}, αema\alpha_{\mathrm{ema}}, nmin⁡covn_{\min}^{\mathrm{cov}}, and (s,e)(s,e), but the paper does not systematically analyze sensitivity to these settings.

  2. Limited benchmark complexity. The experiments focus on ALFWorld, WebShop, and Multi-Hop Search, which involve relatively small action spaces and relatively few interaction turns. Evaluation on more realistic and long-horizon benchmarks, such as OSWorld and Terminal-Bench mentioned in the related work, would strengthen the evidence for the method’s effectiveness.

  3. Missing ablations of adaptive depth control. The paper lacks ablations using only HcovH_{\mathrm{cov}} or HeffH_{\mathrm{eff}}, as well as comparisons against fixed short rollout horizons.

Questions for authors

see weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer Sxqa (1/2)

We thank Reviewer Sxqa for the constructive review and for recognizing the practical importance of our problem and the simplicity and effectiveness of our design. Below, we address each concern with targeted sensitivity analyses, empirical depth distributions, and controller ablations.


W1: Hyperparameter Sensitivity Analysis and Robustness Across Tasks

We investigated the sensitivity of TurnOPD to its hyperparameters through both targeted empirical grids and cross-task verification:

  1. Sensitivity to the primary hyperparameter pp (coverage quantile threshold): As reported in Appendix H.1 and Figure 9, we already evaluated a complete 3×23 \times 2 grid on ALFWorld-1.7B comparing coverage quantiles p∈{0.4,0.6,0.8}p \in \{0.4, 0.6, 0.8\} across both success-conditioned and full-population CDF estimation methods:
Coverage Quantile ppSuccess-Conditioned CDF (Avg@4 / Time)Full-Population CDF (Avg@4 / Time)
p=0.8p = 0.886.29 ± 0.48 / 1.93 h84.96 ± 1.31 / 2.13 h
p=0.6p = 0.685.08 ± 1.21 / 1.75 h86.35 ± 2.08 / 2.04 h
p=0.4p = 0.479.94 ± 1.33 / 1.53 h81.17 ± 0.60 / 1.71 h

Performance remains stable across p∈[0.6,0.8]p \in [0.6, 0.8] (86.29 vs. 85.08 / 86.35), only noticeably degrading when truncation becomes excessively aggressive at p=0.4p = 0.4 (~80–81%). Training wall-clock time scales monotonically with pp (1.93 h -> 1.75 h -> 1.53 h). This demonstrates that pp functions as a predictable, user-controlled knob for trading compute against accuracy, rather than a brittle parameter requiring per-task tuning.

  1. Robustness of secondary control hyperparameters: Due to compute limitations, evaluating full combinatorial sensitivity grids across all parameters is challenging. However, across all four teacher-student setups (ALFWorld-1.7B, ALFWorld-4B, WebShop, and Multi-Hop Search), we reused the exact same control parameters: (Hmin⁡=4,Hmax⁡=80,Kwarm=10,rprobe=0.2,αema=0.1,nmin⁡cov=5,(s,e)=(0.2,0.8))(H_{\min}=4, H_{\max}=80, K_{\mathrm{warm}}=10, r_{\mathrm{probe}}=0.2, \alpha_{\mathrm{ema}}=0.1, n_{\min}^{\mathrm{cov}}=5, (s,e)=(0.2, 0.8)). The fact that TurnOPD achieved consistent efficiency and accuracy gains across all environments without any per-task tuning confirms that these parameters serve as robust, task-agnostic defaults.

W2: Benchmark Complexity, Empirical Trajectory Lengths, and Horizon Scope

We appreciate your perspective regarding benchmark complexity. Below, we contextualize interaction horizons using empirical trajectory distributions:

  1. Empirical trajectory length analysis on ALFWorld: While ALFWorld enforces an environment cap of 80 turns, its trajectories are far from shallow. In vanilla OPD, training rollouts average 21.92 turns and reach up to 69 turns. More importantly, on the unconstrained evaluation set, successful episodes average 18.67 turns, whereas failed episodes average 55.68 turns (reaching up to 76 turns). These uninformative, long-tail failed episodes consume the vast majority of training compute without providing useful supervision. Pruning this unproductive long tail is precisely the core objective of TurnOPD.
  2. Cross-task diversity: Our experimental evaluation covers three diverse benchmarks across four configurations with distinct action spaces and turn structures: WebShop involves e-commerce action spaces capped at 30 turns, while Multi-Hop Search requires open-ended multi-query search capped at 50 turns. The consistent gains across all four setups demonstrate that the method's value is not confined to simple environments.
  3. Future exploration on OSWorld and Terminal-Bench: We agree that OSWorld and Terminal-Bench represent valuable long-horizon environments. However, deploying on these benchmarks requires training new GRPO teacher policies and building complex sandbox execution pipelines, which was infeasible within the rebuttal timeframe. We will explore these benchmarks in future work.

AuthorsOpenReview ↗

Response to Reviewer Sxqa (2/2)

W3: Ablation of Adaptive Depth Controllers and Comparison with Fixed Rollout Lengths

To isolate the contributions of HcovH_{\mathrm{cov}} and HeffH_{\mathrm{eff}}, and to evaluate against fixed rollout horizons, we conducted controlled ablations on ALFWorld-1.7B under both equal training steps (100 steps) and equal wall-clock time (~1.87 h, based on our paper):

Method / Controller VariantInitial HHLate HHWall-clock (100 steps)Same 100-Step Avg@4Equal Time (~1.87h) Avg@4
Vanilla OPD80804.42 h83.0073.50
HcovH_{\mathrm{cov}}-only Controller5023.62.15 h85.7084.20
HeffH_{\mathrm{eff}}-only Controller44.01.41 h76.4081.00
Fixed Truncation H=8H = 8881.59 h80.2081.80
Fixed Truncation H=16H = 1616161.86 h85.1085.10
TurnOPD (Heff+HcovH_{\mathrm{eff}} + H_{\mathrm{cov}})5.423.61.93 h86.3085.60

According to these results:

  • Combining HeffH_{\mathrm{eff}} and HcovH_{\mathrm{cov}} is essential. Using HeffH_{\mathrm{eff}} alone prematurely locks rollout depth at 4 turns, dropping accuracy to 76.40%. Using HcovH_{\mathrm{cov}} alone slows down early training, taking 2.15 h and trailing TurnOPD under equal compute.
  • Fixed rollouts exhibit a clear inflection point. Shorter horizons (H=8H=8) hurt task completion, while longer horizons (H=80H=80) waste compute on uninformative tails. TurnOPD dynamically tracks the optimal horizon throughout training, achieving the highest performance at both equal steps (86.30) and equal wall-clock time (85.60).

Comments on the paper

1 comment

Dear Area Chair,

We thank the reviewers for their thoughtful feedback and recognition of the value and practical impact of our work.

Strengths of our paper. Reviewers consistently highlighted several key contributions:

  • Problem diagnosis and insightful turn-level analysis, identifying diminishing KL signals, survival trajectory sparsity, and compute wastage on unproductive deep rollouts to provide useful insights into on-policy agent distillation (Reviewers Sxqa, 73Jy, bzq8).
  • Clear practical efficiency and performance gains across benchmarks, consistently achieving the highest evaluation scores under equal training time across four teacher-student setups on ALFWorld, WebShop, and Multi-Hop Search, cutting ALFWorld-1.7B training time from 4.42 to 1.93 hours (up to 2.29× speedup) while improving task performance (Reviewers 73Jy, bzq8, Sxqa).
  • Simple, effective, and critic-free design that introduces minimal complexity without requiring extra critic networks or modifications to the underlying distillation objective, with ablations confirming the two proposed modules play complementary roles (Reviewers Sxqa, bzq8).
  • Rigorous evaluation protocol reporting results under both equal training steps and equal wall-clock time, which is essential for algorithms that alter per-step computation (Reviewer 73Jy).

Key concerns and our empirical resolutions:

  • Controlled module ablations against simpler alternatives (Reviewers Sxqa, 73Jy, bzq8): We evaluated TurnOPD against tuned fixed truncation (H∈{8,16}H \in \{8, 16\}), HcovH_{\mathrm{cov}}-only control, HeffH_{\mathrm{eff}}-only control, and SOD without GRPO under both equal steps and equal wall-clock time (~1.87 h). The results show that fixed truncation suffers from an accuracy–efficiency trade-off and HcovH_{\mathrm{cov}}-only is sluggish (2.15 h), confirming that dynamically combining HeffH_{\mathrm{eff}} and HcovH_{\mathrm{cov}} is necessary to achieve Pareto-optimal efficiency and accuracy.
  • Preserving long-horizon behavior under trajectory truncation (Reviewer 73Jy): We tracked full trajectory depth statistics on the unconstrained evaluation set across all four teacher-student configurations. The evidence demonstrates that pruning 22%–40% of training rollouts specifically removes unproductive failure loops (maximum depth drops drastically, e.g., 69->26), while the average depth of successful episodes remains virtually unchanged (within ±5%), preserving full long-horizon problem-solving capability.
  • Value and mechanism of progressive turn normalization (Reviewers 73Jy, bzq8): We provided step-by-step training trajectories comparing progressive blending against hard turn normalization. This demonstrates that progressive blending prevents optimization stalls on uninformative deep turns during early exploration (accelerating early takeoff from 16.0% to 55.6% at step 30) while ensuring balanced supervision alignment across deep turns in later stages.
  • Distinction from existing methods and Pareto optimality (Reviewer bzq8): We clarified fundamental differences from single-output distillation methods (TOPD, POPD, ADWIN) that lack turn-boundary and trajectory-success signals, and conducted an empirical sweep across multiple late-stage rollout tiers (H∈[7,80]H \in [7, 80]). This sweep confirms that TurnOPD naturally converges to the optimal depth tier (H≈21–24H \approx 21\text{--}24) on the empirical Pareto frontier, substantially outperforming heuristic linear truncation (TCOD-F2B) and SOD.
  • Hyperparameter robustness and multi-seed stability (Reviewers Sxqa, bzq8): We conducted a 3×23 \times 2 sensitivity grid on coverage quantile pp and independent multi-seed training runs (n=3n=3). Results show performance is remarkably stable across p∈[0.6,0.8]p \in [0.6, 0.8] without per-task tuning across all four environments, and TurnOPD achieves a statistically significant 2.29±0.13× speedup and superior accuracy across random seeds.

Following our rebuttal, one reviewer decided to maintain their score, while we did not receive further responses from the other two reviewers during the discussion period. We hope our detailed responses demonstrate that all core technical concerns have been rigorously and thoroughly addressed. We are committed to incorporating these constructive suggestions into the final version, and we hope you will take these developments into account in your final assessment.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.