- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper formulates long-horizon on-policy distillation (OPD) as hierarchical supervision allocation and proposes LENS-OPD, a coarse-to-fine framework organized around three nested stages: LOCATE, VALIDATE, REFINE. The central principle is that productive guidance lies at the intersection of future utility (does an intervention improve the student's subsequent behavior?) and current learnability (can the student realize and internalize that improvement?), and that this intersection evolves as the student learns.
Critical strengths
The "future utility × current learnability" framing and the diagnostic evidence for it is a conceptual contribution. The hierarchical view: trajectory exposure determines which states are visitable, turn validation determines which intervention is worth learning, token focusing determines where the learning pressure goes, is the right abstraction.
Consistent, well-seeded gains across benchmarks, student scales, and model families with proper baselines. Unlike much distillation work that reports single-run gains over only OPD/SFT, this paper runs 5 seeds, evaluates on 3 benchmarks, tests 2 student scales and 2 model families, and compares against 4 adaptive baselines under matched budgets in a common codebase.
Critical weaknesses
The future-validation evaluator is teacher-centric and its circularity is neither addressed nor tested. The teacher is both the source of the hint and the arbiter of whether the hint helped. This has two consequences the paper does not address: (a) a teacher hint that makes the student more teacher-like (higher teacher likelihood) will be accepted even if it moves the student away from the task-successful action, (b) when the teacher is wrong, the evaluator reinforces the wrong behavior.
The framework's complexity is high, its computational cost is unreported, and its many hyperparameters are not sensitivity-tested. LENS-OPD introduces 3 stages with ≥8 method-specific hyperparameters (η, K_max, c_tok, β, λ_GI, H, plus the F2B base). Each trajectory requires: a full student rollout, teacher scoring of all turns, state restoration, one teacher hint, two 3-turn student continuations, and teacher scoring of both continuations, plus state-restoration infrastructure.
Questions for authors
What is the per-step wall-clock time and total token generation cost of LENS-OPD relative to vanilla OPD (and to TCOD, which also has curriculum overhead)? Does the 2–3× per-trajectory generation overhead (two continuations + hint) hold in practice, and how does it affect the per-compute-step gain?
How sensitive is LENS-OPD to c_tok, β, λ_GI, and H? Were these tuned on a validation split, or set by default? If the method is robust to them, a sensitivity table would strengthen the paper considerably.
Guided-OPD and FTB underperform vanilla OPD by 5–18 points on several configs. Can you confirm these baselines are faithfully implemented (e.g., do they improve over OPD in the original papers' reported settings)? If they reliably degrade here, is there a confound in the shared codebase?
5 comments
Response to Reviewer xFqX (2/3)
W2-1 and Q1 — What is the actual computational overhead of paired validation, does the estimated 2–3× generation overhead hold, and how does cost affect the gains?
We would like to clarify two implementation details relevant to the overhead estimate. First, the main student rollout stops at the current curriculum horizon , or earlier if the episode terminates, rather than always running to the full environment limit. Second, paired validation does not generally require generating two additional student continuations. When the original rollout contains a usable continuation, we reuse it as the no-hint branch. This occurred in 2,698 of 2,900 completed ALFWorld validations (93.0%), so most completed validations require only one additional student continuation and a short teacher hint.
Student rollout and teacher scoring of the original responses are already part of vanilla OPD. LENS-OPD selects one candidate turn per trajectory and performs the additional hint generation, state restoration, and continuation evaluation at that turn. We measure the resulting cost over 250 optimizer updates:
| Training metric | Vanilla OPD | LENS-OPD |
|---|---|---|
| Optimizer updates | 250 | 250 |
| Wall-clock time (h) | 16.5 | 22.5 |
| Mean update interval (s) | 238.0 | 325.6 |
| Total environment interactions | 69,815 | 57,650 |
Wall-clock time is measured from the first to the 250th logged update and includes waiting for rollout data. Interaction counts include the main rollouts, validation branches, and state-restoration replay.
LENS-OPD uses 17.4% fewer total environment interactions. Its adaptive curriculum limits rollout depth during early and middle training, and this reduction more than offsets the additional interactions required for validation. Its wall-clock cost is approximately 1.37× that of vanilla OPD, reflecting the additional generation, scoring, and restoration work. These measurements quantify the actual training overhead, while the branch-reuse statistics clarify why two validation branches do not imply two additional rollouts per trajectory.
We will distinguish the matched optimizer-update budget from computational cost and report these implementation details and measurements explicitly.
W2-2 and Q2 — How sensitive is LENS-OPD to its additional hyperparameters, and were their values tuned on a validation set or set by default?
The reported hyperparameters were set without validation-set tuning. follows the environment interaction limit, and follows the F2B reference clock. The remaining parameters control token focusing ( and ), guidance internalization (), and validation length (). To assess sensitivity, we varied each parameter individually in the Qwen3-32B → Qwen3-1.7B ALFWorld setting while keeping the others at their defaults.
| Configuration | Overall SR (%) | Seen SR (%) | Unseen SR (%) |
|---|---|---|---|
| Default | 47.9 ± 2.2 | 44.6 ± 2.9 | 51.3 ± 3.0 |
| 48.9 ± 2.7 | 47.6 ± 7.1 | 50.3 ± 6.3 | |
| 52.6 ± 2.7 | 44.7 ± 6.5 | 60.8 ± 4.4 | |
| 44.0 ± 0.9 | 43.1 ± 1.6 | 44.9 ± 2.9 | |
| 47.7 ± 1.9 | 43.9 ± 1.6 | 51.6 ± 3.3 | |
| 47.2 ± 4.2 | 45.6 ± 6.9 | 48.8 ± 4.0 |
The gains persist across all tested settings: overall SR ranges from 44.0% to 52.6%, compared with 36.4% for vanilla OPD. Doubling the GI coefficient, halving the token-focus scale, or shortening validation to one turn maintains performance close to the default. Reducing the token-weight cap further improves performance, while weakening GI produces a larger decrease but still retains a substantial advantage over OPD. These results show that the method remains effective across the tested parameter choices and does not rely on a narrowly tuned configuration.
Response to Reviewer xFqX (3/3)
Q3 — Are Guided-OPD and FTB faithfully implemented, and could their performance drops reflect a confound in the shared framework?
We use the authors’ implementations of Guided-OPD and FTB, integrated into a shared training framework. Within each comparison, all methods share the teacher checkpoint, student initialization, prompts, environment interface, action parser, optimizer-update budget, and evaluation protocol. Both baselines achieve gains over vanilla OPD in our Qwen3 experiments. For the Qwen3-1.7B student on WebShop, SR increases from 9.6% with OPD to 10.2% with Guided-OPD and 18.2% with FTB. For Qwen3-4B on ALFWorld, they improve SR from 42.8% to 46.1% and 45.2%, respectively. These results show that the shared framework supports improvements from both methods.
The largest ALFWorld regressions occur with the Qwen2.5-7B-RL teacher and Qwen2.5-3B student, a pairing evaluated in neither original paper. Guided-OPD falls below OPD by 18.1 points on both Seen and Unseen, while FTB falls below it by 14.8 and 13.9 points. These are therefore results in an additional model-family setting, rather than failed reproductions of results reported for that pairing.
LENS-OPD improves ALFWorld performance across both tested model families. With Qwen3, it increases overall SR from 36.4% to 47.9% for the 1.7B student and from 42.8% to 52.4% for the 4B student. With Qwen2.5, it improves Seen/Unseen SR from 40.4%/36.3% to 46.3%/41.9%. This consistency supports the effectiveness of our supervision-allocation approach across the evaluated teacher–student configurations. We will document the baseline implementation settings and checkpoint differences explicitly.
Response to Reviewer xFqX (1/3)
Thank you for recognizing the conceptual framing and the scope of the evaluation. We address the teacher-based evaluator with an external progress check and acceptance controls, then report measured overhead and parameter sensitivity.
We use teacher likelihood as a dense surrogate for local guidance utility. Evaluating terminal success would require both branches to continue until termination or the interaction limit, substantially increasing validation cost. For a small student, both branches may ultimately fail even when a hint improves the current decision, making binary success too sparse to distinguish useful guidance. The teacher’s length-normalized likelihood provides a graded comparison of the continuations that the same frozen student generates, with and without a hint, from the same restored state.
Because the teacher both supplies the hint and scores its consequences, we specifically test whether the gate accepts guidance that increases teacher agreement while worsening task progress. At the Top-KL positions selected during training, an ALFWorld symbolic planner independent of the teacher compares the remaining steps to task completion after up to three matched continuation turns. Among 80 valid Qwen3 paired replays, 3 cases (3.75% of all pairs) show erroneous acceptance: the likelihood gate accepts the hinted branch even though the planner judges it farther from task completion. As a complementary check, judgments from GPT-5.6 sol agree with the likelihood-based comparison in 70 of 80 cases (87.5%).
We also added two controls to examine whether the acceptance rule improves learning beyond simply providing more guidance. Always accepts every valid candidate, while Random accepts candidates independently with a fixed probability. The observed acceptance rates are 71.0% for LENS-OPD and 73.7% for Random, providing an approximately rate-matched comparison. We evaluate all three methods at 250 optimizer updates on the same ALFWorld tasks and report mean success rate ± standard deviation over five evaluation seeds.
LENS-OPD improves overall SR by 3.3 percentage points over Random and 8.3 points over Always, with gains on both Seen and Unseen tasks. The advantage over Random supports the value of selecting useful interventions at a comparable acceptance rate, while the comparison with Always shows that accepting more guidance does not necessarily improve learning. Together, the external checks and training controls provide evidence that the gate selects useful guidance, while explicitly measuring cases where teacher preference conflicts with task progress. The gate controls the additional GI loss and token focusing. Rejected or incomplete validations retain ordinary OPD supervision on the original student trajectory.