- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the issue of resource allocation in long-horizon agent distillation by proposing an approach that combines adaptive trajectory depth with progressive round normalization, demonstrating a favorable trade-off between performance and training time. The work’s strengths lie in its precise problem diagnosis, concise methodology, and clear practical benefits. Its primary limitations include an insufficient comparison with related methods to show the unique contributions, a weak connection between the theoretical framework and the control rules, and a lack of robust evidence for certain design choices. Overall, this is an incremental study with practical value; however, its unique contributions require further, more targeted experimental validation.
Critical strengths
- The paper analyzes teacher supervision signals, trajectory survival counts, and loss distribution at the turn level, identifying two key issues: the inefficiency of collecting long trajectories and the over-concentration of loss in early stages. Adaptive depth control and progressive turn normalization are introduced to address these, resulting in a well-structured argument.
- Across three tasks and four teacher-student configurations, TurnOPD consistently achieves the highest evaluation scores for a given training duration. Notably, for the ALFWorld-1.7B task, training time was reduced from 4.42 hours to 1.93 hours while simultaneously improving performance. Ablation studies further confirm that the two proposed modules play complementary roles.
Critical weaknesses
- The boundaries of innovation and experimental advantages relative to existing work are not fully established. Concepts such as trajectory length control, dynamic truncation, and turn reweighting have been explored in prior research. The main experiments primarily compare "Vanilla OPD" against "TCOD-F2B," lacking direct comparisons with the most closely related existing methods—or simple combinations thereof—making it difficult to isolate the specific contribution of the authors' unique control rules.
- A gap exists between the theoretical rationale and the actual control metrics employed. The "KL compression" proposition suggests that a smaller KL divergence does not necessarily imply a smaller gap in true capability; yet, the method still uses raw KL divergence to construct its supervision-value metric, a choice that warrants further justification. Furthermore, while the existence of an "optimal depth" is discussed, it is not proven that the proposed controller can effectively approach this optimum.
- Insufficient evidence regarding the efficacy of certain key design elements and the stability of the results. "Progressive mixing" yields only a 0.1-point improvement over direct turn normalization, which is arguably insufficient to justify the necessity of a progressive scheduling approach. Additionally, the reported standard deviations are derived from four evaluation checkpoints rather than multiple independent training runs; consequently, the robustness of the observed performance differences remains to be verified.
Questions for authors
- Compared to existing methods such as TCOD, SOD, POPD/TOPD, and ADWIN, what is the core novel mechanism introduced by TurnOPD? Can the authors demonstrate that its performance gains stem from something more than just simple trajectory truncation and loss reweighting?
- Given that the paper notes how "raw KL" in deeper layers may be affected by context compression—thereby failing to accurately reflect policy divergence—why do the authors still use raw KL to construct ? Furthermore, can they empirically validate the relationship between this metric and either task success rates or gradient utility?
- Since "progressive round normalization" yields an improvement of only about 0.1 points over "fixed round normalization," can the authors provide evidence to justify the necessity and stability of this design?
4 comments
Response to Reviewer bzq8 (2/3)
W2(2): Empirical Support for Approaching Optimal Depth and Pareto Optimality
We appreciate the opportunity to clarify this. Taking ALFWorld-1.7B as a representative case:
- While a theoretical optimal depth exists, identifying it analytically or through post-hoc sweeps is practically infeasible; nonetheless, TurnOPD lies on the empirical Pareto frontier across all observed depth tiers:
- Theoretically, varies dynamically with the implicit capability gap between student and teacher at each training step, task characteristics, the number of tools, tool coupling, and hardware constraints.
- Empirically, identifying an exact optimum in noisy environments is elusive. Viewed from this perspective, the sensitivity grid in Figure 9 effectively sweeps through different late-stage rollout horizons across several tiers (late-stage depths around 7, 10, 14, 21–24, 38, plus unconstrained ). The empirical landscape displays an elbow followed by a broad plateau: for depth , performance differences are minor, with the entire spread across Avg@4 being just 1.27 points (comparable to evaluation noise).
- TurnOPD naturally converges to the optimal tier () while achieving the best wall-clock efficiency within that tier. Across the sweep, no point is simultaneously faster and more accurate than TurnOPD; TurnOPD lies squarely on the Pareto frontier.
| Late-Stage Depth (steps 50–100) | Same 100-Step Avg@4 | Wall-clock (100 steps) |
|---|---|---|
| 7.0 | 81.38 | 1.464 h |
| 7.7 | 79.94 | 1.531 h |
| 9.8 | 81.17 | 1.709 h |
| 14.0 | 85.08 | 1.752 h |
| 20.8 | 86.35 | 2.037 h |
| 23.6 (adaptive ) | 86.29 | 1.928 h |
| 38.0 | 84.96 | 2.132 h |
| 80 (unconstrained) | 85.08 | 2.586 h |
- The optimal depth shifts dynamically as training progresses and cannot be adequately captured by simple linear heuristics. TCOD-F2B serves as a clear illustration: using a data-agnostic, front-to-back linear schedule over the same 100 steps results in a 6.23-point deficit under essentially identical wall-clock time:
| Method | Characteristic | Wall-clock (100 steps) | Same-Step Avg@4 |
|---|---|---|---|
| TCOD-F2B | Non-adaptive linear (front-to-back) truncation | 1.866 h | 80.06 |
| TurnOPD | Adaptive coverage-aware control | 1.928 h | 86.29 |
- We do not claim that TurnOPD analytically solves for the exact theoretical , which is intractable to prove in practice. However, our empirical sweep demonstrates that within observable measurement resolution, TurnOPD consistently outperforms non-adaptive heuristics (both fixed truncation and linear decay schedules).
W3(1) & Q3: Value and Mechanism of Progressive Turn Normalization
Thanks for this feedback; this distinction warrants clarification. Below, we compare the step-by-step training trajectory of progressive blending against hard turn normalization on ALFWorld-1.7B:
| Training Step | Progressive Blending Accuracy | Hard Turn Normalization Accuracy |
|---|---|---|
| Step 10 | 0.42 | 0.17 |
| Step 30 | 55.58 | 16.00 |
| Step 50 | 73.58 | 74.92 |
| Step 70 | 81.25 | 83.67 |
| Step 90 | 86.67 | 85.08 |
As shown in the table, progressive blending achieves a much faster cold start (55.58% vs. 16.00% at step 30), even though both converge to a similar accuracy by step 90. The underlying mechanism directly connects to our exploratory analysis: early in training, student rollouts are short and deeper turns carry scarce supervisory signals. Hard turn normalization forces uniform weight onto uninformative deeper turns, stalling policy improvement. Progressive blending begins close to trajectory normalization, focusing gradient updates where supervision is reliable, and smoothly transitions to turn-level alignment as trajectories mature. Progressive blending thus combines early sample efficiency with late-stage turn alignment, ensuring stability throughout training.
Response to Reviewer bzq8 (3/3)
W3(2): Evaluation Checkpoint Variance and Multi-Seed Validation
- Rationale for reporting checkpoint mean and standard deviation: In individual training runs, validation scores across adjacent steps naturally fluctuate. In practical deployments, practitioners need to know whether an optimization algorithm maintains stable performance across a training window rather than producing erratic checkpoints. We therefore computed the mean and standard deviation over consecutive checkpoints to reliably reflect sustained performance.
- Independent multi-seed training runs: Constrained by compute budget, running multi-seed sweeps across all configurations was not feasible. However, to directly verify training stability, we conducted multi-seed evaluations on ALFWorld with Qwen3-1.7B (). Same-Step evaluates at 100 training steps, while Same-Time aligns to the ~1.87 h training budget from the paper:
| Method | Wall-clock (h) | Speedup | Same-Step Avg@3 (multi-seed) | Same-Time Avg@3 (multi-seed) | |
|---|---|---|---|---|---|
| Vanilla OPD | 3 | 4.42 ± 0.23 | 1.00× | 82.5 ± 0.69 | 73.3 ± 1.35 |
| TurnOPD | 3 | 1.93 ± 0.11 | 2.29× ± 0.13 | 86.9 ± 1.15 | 85.7 ± 0.85 |
Across independent training seeds, TurnOPD consistently achieves a ~2.29× wall-clock speedup and superior accuracy under both step and time budgets, confirming that the observed gains are statistically robust.
The additional experiments support your method. However, I think the paper needs a major revision and reorganize. I will remain my sccore.
Response to Reviewer bzq8 (1/3)
We thank Reviewer bzq8 for the thoughtful and structured review, and for recognizing the clarity of our turn-level analysis, our consistent efficiency and task performance improvements across benchmarks, and the complementary roles of our two proposed modules. Below, we address each concern and question in detail.
W1 & Q1: Core Mechanisms and Distinction from TOPD, POPD, ADWIN, TCOD, and SOD
Below are empirical results comparing these approaches under identical budgets on ALFWorld-1.7B:
Whether evaluated under equal step budgets or equal wall-clock time, TurnOPD consistently outperforms both truncation-only and reweighting-only baselines.
W2(1) & Q2: Rationale for Using Raw KL Despite Context Compression
Thanks for raising this insightful question, which gives us an opportunity to clarify our design rationale.
maxoperator in Hctrl=max(Heff,Hcov) is not an arbitrary combination. Its precise role is to safeguard against compressed raw KL driving rollout depth excessively low. As noted in the paper, "Although raw KL may not directly measure behavioral error, it indicates supervision value and cost." Rather than treating raw KL as an absolute estimate of required depth, we use it as an efficiency signal to lift the rollout lower bound, while the coverage term Hcov directly counterbalances its known bias. The perceived gap between the theoretical rationale and control metrics is thus explicitly managed by this design rather than overlooked.