acceptodds
Under review as a conference paper at ICLR 2027

Dense Is Not Enough: Hierarchical Supervision Allocation for Long-Horizon On-Policy Distillation

19 views

Abstract

On-policy distillation (OPD) transfers the capabilities of a large language model to a smaller student by providing teacher supervision on the student's own rollouts. In long-horizon agentic tasks, however, uniform token-level matching can allocate supervision poorly: a large local discrepancy need not improve future behavior, while consequential guidance may be beyond the current student's reach or fail to persist without privileged input. We formulate long-horizon OPD as hierarchical supervision allocation and argue that productive guidance lies at the intersection of future utility and current learnability. Crucially, this intersection evolves as the student learns. Based on this principle, we propose LENS-OPD, a coarse-to-fine framework that organizes supervision through Locate, Validate, and Refine. Locate adapts trajectory exposure to the student's evolving competence and proposes a candidate decision for intervention. Validate tests whether teacher guidance at that decision improves the same student's subsequent behavior. Refine internalizes the beneficial guided behavior into the deployable policy and concentrates token-level supervision on decisive teacher-student conflicts within the validated turn. These stages are nested: each finer allocation is conditioned on the coarser decision, rather than being optimized as an independent importance score. Experiments across multiple long-horizon agent benchmarks and student-teacher configurations show that LENS-OPD consistently improves task performance over vanilla OPD and strong curriculum- and selection-based baselines. Our results suggest that effective long-horizon distillation requires teaching at the right depth, the right decision, and the right token.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, LLM Agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xFqXOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper formulates long-horizon on-policy distillation (OPD) as hierarchical supervision allocation and proposes LENS-OPD, a coarse-to-fine framework organized around three nested stages: LOCATE, VALIDATE, REFINE. The central principle is that productive guidance lies at the intersection of future utility (does an intervention improve the student's subsequent behavior?) and current learnability (can the student realize and internalize that improvement?), and that this intersection evolves as the student learns.

Critical strengths

The "future utility × current learnability" framing and the diagnostic evidence for it is a conceptual contribution. The hierarchical view: trajectory exposure determines which states are visitable, turn validation determines which intervention is worth learning, token focusing determines where the learning pressure goes, is the right abstraction.

Consistent, well-seeded gains across benchmarks, student scales, and model families with proper baselines. Unlike much distillation work that reports single-run gains over only OPD/SFT, this paper runs 5 seeds, evaluates on 3 benchmarks, tests 2 student scales and 2 model families, and compares against 4 adaptive baselines under matched budgets in a common codebase.

Critical weaknesses

The future-validation evaluator is teacher-centric and its circularity is neither addressed nor tested. The teacher is both the source of the hint and the arbiter of whether the hint helped. This has two consequences the paper does not address: (a) a teacher hint that makes the student more teacher-like (higher teacher likelihood) will be accepted even if it moves the student away from the task-successful action, (b) when the teacher is wrong, the evaluator reinforces the wrong behavior.

The framework's complexity is high, its computational cost is unreported, and its many hyperparameters are not sensitivity-tested. LENS-OPD introduces 3 stages with ≥8 method-specific hyperparameters (η, K_max, c_tok, β, λ_GI, H, plus the F2B base). Each trajectory requires: a full student rollout, teacher scoring of all turns, state restoration, one teacher hint, two 3-turn student continuations, and teacher scoring of both continuations, plus state-restoration infrastructure.

Questions for authors

What is the per-step wall-clock time and total token generation cost of LENS-OPD relative to vanilla OPD (and to TCOD, which also has curriculum overhead)? Does the 2–3× per-trajectory generation overhead (two continuations + hint) hold in practice, and how does it affect the per-compute-step gain?

How sensitive is LENS-OPD to c_tok, β, λ_GI, and H? Were these tuned on a validation split, or set by default? If the method is robust to them, a sensitivity table would strengthen the paper considerably.

Guided-OPD and FTB underperform vanilla OPD by 5–18 points on several configs. Can you confirm these baselines are faithfully implemented (e.g., do they improve over OPD in the original papers' reported settings)? If they reliably degrade here, is there a confound in the shared codebase?

5 comments
AuthorsOpenReview ↗

Response to Reviewer xFqX (1/3)

Thank you for recognizing the conceptual framing and the scope of the evaluation. We address the teacher-based evaluator with an external progress check and acceptance controls, then report measured overhead and parameter sensitivity.

W1 — Can a teacher that both generates and evaluates hints accept incorrect guidance, and how is this risk assessed independently?

We use teacher likelihood as a dense surrogate for local guidance utility. Evaluating terminal success would require both branches to continue until termination or the interaction limit, substantially increasing validation cost. For a small student, both branches may ultimately fail even when a hint improves the current decision, making binary success too sparse to distinguish useful guidance. The teacher’s length-normalized likelihood provides a graded comparison of the continuations that the same frozen student generates, with and without a hint, from the same restored state.

Because the teacher both supplies the hint and scores its consequences, we specifically test whether the gate accepts guidance that increases teacher agreement while worsening task progress. At the Top-KL positions selected during training, an ALFWorld symbolic planner independent of the teacher compares the remaining steps to task completion after up to three matched continuation turns. Among 80 valid Qwen3 paired replays, 3 cases (3.75% of all pairs) show erroneous acceptance: the likelihood gate accepts the hinted branch even though the planner judges it farther from task completion. As a complementary check, judgments from GPT-5.6 sol agree with the likelihood-based comparison in 70 of 80 cases (87.5%).

We also added two controls to examine whether the acceptance rule improves learning beyond simply providing more guidance. Always accepts every valid candidate, while Random accepts candidates independently with a fixed probability. The observed acceptance rates are 71.0% for LENS-OPD and 73.7% for Random, providing an approximately rate-matched comparison. We evaluate all three methods at 250 optimizer updates on the same ALFWorld tasks and report mean success rate ± standard deviation over five evaluation seeds.

Acceptance ruleOverall SR (%)Seen SR (%)Unseen SR (%)
LENS-OPD47.9 ± 2.244.6 ± 2.951.3 ± 3.0
Random44.6 ± 2.041.1 ± 2.948.2 ± 1.6
Always39.6 ± 2.137.0 ± 3.842.2 ± 1.9

LENS-OPD improves overall SR by 3.3 percentage points over Random and 8.3 points over Always, with gains on both Seen and Unseen tasks. The advantage over Random supports the value of selecting useful interventions at a comparable acceptance rate, while the comparison with Always shows that accepting more guidance does not necessarily improve learning. Together, the external checks and training controls provide evidence that the gate selects useful guidance, while explicitly measuring cases where teacher preference conflicts with task progress. The gate controls the additional GI loss and token focusing. Rejected or incomplete validations retain ordinary OPD supervision on the original student trajectory.

AuthorsOpenReview ↗

Response to Reviewer xFqX (2/3)

W2-1 and Q1 — What is the actual computational overhead of paired validation, does the estimated 2–3× generation overhead hold, and how does cost affect the gains?

We would like to clarify two implementation details relevant to the overhead estimate. First, the main student rollout stops at the current curriculum horizon KK, or earlier if the episode terminates, rather than always running to the full environment limit. Second, paired validation does not generally require generating two additional student continuations. When the original rollout contains a usable continuation, we reuse it as the no-hint branch. This occurred in 2,698 of 2,900 completed ALFWorld validations (93.0%), so most completed validations require only one additional student continuation and a short teacher hint.

Student rollout and teacher scoring of the original responses are already part of vanilla OPD. LENS-OPD selects one candidate turn per trajectory and performs the additional hint generation, state restoration, and continuation evaluation at that turn. We measure the resulting cost over 250 optimizer updates:

Training metricVanilla OPDLENS-OPD
Optimizer updates250250
Wall-clock time (h)16.522.5
Mean update interval (s)238.0325.6
Total environment interactions69,81557,650

Wall-clock time is measured from the first to the 250th logged update and includes waiting for rollout data. Interaction counts include the main rollouts, validation branches, and state-restoration replay.

LENS-OPD uses 17.4% fewer total environment interactions. Its adaptive curriculum limits rollout depth during early and middle training, and this reduction more than offsets the additional interactions required for validation. Its wall-clock cost is approximately 1.37× that of vanilla OPD, reflecting the additional generation, scoring, and restoration work. These measurements quantify the actual training overhead, while the branch-reuse statistics clarify why two validation branches do not imply two additional rollouts per trajectory.

We will distinguish the matched optimizer-update budget from computational cost and report these implementation details and measurements explicitly.

W2-2 and Q2 — How sensitive is LENS-OPD to its additional hyperparameters, and were their values tuned on a validation set or set by default?

The reported hyperparameters were set without validation-set tuning. Kmax⁡K_{\max} follows the environment interaction limit, and η=5\eta = 5 follows the F2B reference clock. The remaining parameters control token focusing (ctokc_{\mathrm{tok}} and β\beta), guidance internalization (λGI\lambda_{\mathrm{GI}}), and validation length (HH). To assess sensitivity, we varied each parameter individually in the Qwen3-32B → Qwen3-1.7B ALFWorld setting while keeping the others at their defaults.

ConfigurationOverall SR (%)Seen SR (%)Unseen SR (%)
Default47.9 ± 2.244.6 ± 2.951.3 ± 3.0
β=0.5\beta = 0.548.9 ± 2.747.6 ± 7.150.3 ± 6.3
ctok=2.5c_{\mathrm{tok}} = 2.552.6 ± 2.744.7 ± 6.560.8 ± 4.4
λGI=0.5\lambda_{\mathrm{GI}} = 0.544.0 ± 0.943.1 ± 1.644.9 ± 2.9
λGI=2.0\lambda_{\mathrm{GI}} = 2.047.7 ± 1.943.9 ± 1.651.6 ± 3.3
H=1H = 147.2 ± 4.245.6 ± 6.948.8 ± 4.0

The gains persist across all tested settings: overall SR ranges from 44.0% to 52.6%, compared with 36.4% for vanilla OPD. Doubling the GI coefficient, halving the token-focus scale, or shortening validation to one turn maintains performance close to the default. Reducing the token-weight cap further improves performance, while weakening GI produces a larger decrease but still retains a substantial advantage over OPD. These results show that the method remains effective across the tested parameter choices and does not rely on a narrowly tuned configuration.

AuthorsOpenReview ↗

Response to Reviewer xFqX (3/3)

Q3 — Are Guided-OPD and FTB faithfully implemented, and could their performance drops reflect a confound in the shared framework?

We use the authors’ implementations of Guided-OPD and FTB, integrated into a shared training framework. Within each comparison, all methods share the teacher checkpoint, student initialization, prompts, environment interface, action parser, optimizer-update budget, and evaluation protocol. Both baselines achieve gains over vanilla OPD in our Qwen3 experiments. For the Qwen3-1.7B student on WebShop, SR increases from 9.6% with OPD to 10.2% with Guided-OPD and 18.2% with FTB. For Qwen3-4B on ALFWorld, they improve SR from 42.8% to 46.1% and 45.2%, respectively. These results show that the shared framework supports improvements from both methods.

The largest ALFWorld regressions occur with the Qwen2.5-7B-RL teacher and Qwen2.5-3B student, a pairing evaluated in neither original paper. Guided-OPD falls below OPD by 18.1 points on both Seen and Unseen, while FTB falls below it by 14.8 and 13.9 points. These are therefore results in an additional model-family setting, rather than failed reproductions of results reported for that pairing.

LENS-OPD improves ALFWorld performance across both tested model families. With Qwen3, it increases overall SR from 36.4% to 47.9% for the 1.7B student and from 42.8% to 52.4% for the 4B student. With Qwen2.5, it improves Seen/Unseen SR from 40.4%/36.3% to 46.3%/41.9%. This consistency supports the effectiveness of our supervision-allocation approach across the evaluated teacher–student configurations. We will document the baseline implementation settings and checkpoint differences explicitly.

Reviewer_emhHOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LENS-OPD, a coarse-to-fine framework that combines different types of OPD losses. Specifically, it combines a self-distillation like loss with another loss that controls the strength of distillation. Although the method achieves certain performance improvements on ALFWorld, WebShop, and ScienceWorld, the overall contribution appears relatively incremental.

Critical strengths
  1. The paper proposes LENS-OPD, a coarse-to-fine framework that combines different OPD losses,and achieves some performance improvements on ALFWorld, WebShop, and ScienceWorld.
Critical weaknesses
  1. For the final loss, the paper only validates the effectiveness of Guidance Internalization through an ablation study, without analyzing its sensitivity to the weighting coefficient. Therefore, it remains unclear how sensitive the model is to the relative weighting of the two OPD loss terms and whether the reported gains depend on a specific loss balance.
  2. The evaluation of teacher hints is not objective. A hint is considered better mainly based on whether it aligns with the teacher policy. However, agreement with the teacher policy does not mean the hint is genuinely useful.
  3. It's biased to use the disagreement between the teacher and student to determine the intervention turn. As the trajectory becomes longer, the mismatch between the teacher and student tends to accumulate, making the teacher–student mismatch/disagreement naturally increase over the trajectory position.
Questions for authors

See above

6 comments
AuthorsOpenReview ↗

Response to Reviewer emhH (1/2)

Thank you for the assessment. Our contribution is a hierarchical mechanism for deciding where teacher supervision is useful and learnable. The curriculum controls how far the student explores; paired validation tests whether a hint improves the student’s own continuation; accepted hints then guide internalization and token-level focusing. These stages are coupled: teacher–student disagreement proposes an intervention, but only evidence of improved student behavior activates the additional supervision. Table 3 and the added acceptance controls provide empirical support for this design.

W1 — How sensitive are the gains to the GI coefficient, and do they depend on a specific balance between the loss terms?

The default λGI=1.0\lambda_{\mathrm{GI}} = 1.0 was set without validation-set tuning. To assess sensitivity to the loss balance, we varied only λGI\lambda_{\mathrm{GI}} in the Qwen3-32B → Qwen3-1.7B ALFWorld setting, keeping training at 250 optimizer updates and evaluating on the same 140 Valid Seen and 134 Valid Unseen tasks.

λGI\lambda_{\mathrm{GI}}Overall SR (%)Seen SR (%)Unseen SR (%)
0.544.0 ± 0.943.1 ± 1.644.9 ± 2.9
1.0 (default)47.9 ± 2.244.6 ± 2.951.3 ± 3.0
2.047.7 ± 1.943.9 ± 1.651.6 ± 3.3

Doubling λGI\lambda_{\mathrm{GI}} leaves overall SR nearly unchanged (47.9% versus 47.7%), while halving it reduces SR to 44.0%. Across all three values, performance remains above vanilla OPD’s 36.4% overall SR in Table 1. These results show that the gains persist across the tested range of GI weights and do not depend on a narrowly chosen loss balance.

W2 — Does greater agreement with the teacher indicate useful guidance, and what task-based evidence supports the acceptance rule?

We use teacher likelihood to obtain a dense measure of local improvement within a short validation horizon. Evaluating terminal success would require both branches to continue until termination or the interaction limit, substantially increasing the cost of each validation. Moreover, for a small student, both branches may ultimately fail even when a hint improves the current decision, making binary success too sparse to distinguish useful guidance. The teacher’s length-normalized likelihood provides a graded comparison of the continuations that the same frozen student generates, with and without a hint, from the same restored state.

We further examine whether this signal agrees with task progress using an ALFWorld symbolic planner independent of the teacher. At the Top-KL positions selected during training, we compare the remaining steps to task completion after up to three matched continuation turns. Among 80 valid Qwen3 paired replays, 3 cases (3.75% of all pairs) show erroneous acceptance: the likelihood gate accepts the hinted branch even though the planner judges it farther from task completion. This directly tests whether greater teacher agreement can come at the expense of task progress. As a complementary check, judgments from GPT-5.6 sol agree with the likelihood-based comparison in 70 of 80 cases (87.5%).

We also added two controls to examine whether the acceptance rule improves learning beyond simply providing more guidance. Always accepts every valid candidate, while Random accepts candidates independently with a fixed probability. The observed acceptance rates are 71.0% for LENS-OPD and 73.7% for Random, providing an approximately rate-matched comparison. We evaluate all three methods at 250 optimizer updates on the same ALFWorld tasks and report mean success rate ± standard deviation over five evaluation seeds.

Acceptance ruleOverall SR (%)Seen SR (%)Unseen SR (%)
LENS-OPD47.9 ± 2.244.6 ± 2.951.3 ± 3.0
Random44.6 ± 2.041.1 ± 2.948.2 ± 1.6
Always39.6 ± 2.137.0 ± 3.842.2 ± 1.9

LENS-OPD improves overall SR by 3.3 percentage points over Random and 8.3 points over Always, with gains on both Seen and Unseen tasks. The advantage over Random supports the value of selecting useful interventions at a comparable acceptance rate, while the comparison with Always shows that accepting more guidance does not necessarily improve learning. These results directly support the acceptance mechanism and complement the component ablations in Table 3.The gate activates GI and token focusing only when the hinted continuation receives a higher score. Rejected or incomplete validations retain ordinary OPD supervision on the original student trajectory.

AuthorsOpenReview ↗

Response to Reviewer emhH (2/2)

W3 — Does selecting the highest-disagreement turn favor later positions because teacher–student mismatch accumulates along the trajectory?

We would like to clarify a distinction between accumulated trajectory mismatch and the turn-level disagreement score used in our method. Our proposal score is the mean content-token KL at each individual turn, with the teacher and student conditioned on the same student-generated context. It does not accumulate KL across preceding turns, so a later position does not automatically receive a higher score. Earlier actions can still affect the current context, and we therefore checked whether a bias toward later turns occurs in practice.

The observed pattern is the opposite. In the Qwen3-1.7B ALFWorld run, among 1,907 trajectories with at least ten valid per-turn KL measurements, 77.5% of Top-KL proposals fall within the first fifth of the trajectory. Within each trajectory, the last quarter’s mean KL is lower than the first quarter’s by 0.328 on average. These measurements do not support the hypothesized preference for later turns in this setting.

Moreover, Top-KL only nominates a candidate for validation; the paired continuation test determines whether its guidance activates GI and token focusing. Table 3 further shows that replacing Top-KL with the last valid turn, while retaining the same validation and refinement mechanisms, reduces SR from 44.6% to 37.4% on Valid Seen and from 51.3% to 45.1% on Valid Unseen. This supports the effectiveness of the disagreement-based proposal over a fixed last-turn rule.

Thank you for your response. The ALFWorld analysis is helpful, but it remains unclear whether the observed KL pattern generalizes. In a different ALFWorld setting, prior work [1-2] found that per-turn teacher–student KL increased with turn index despite conditioning on the same student-generated history. I will therefore maintain my original score.

[1] Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents. COLM 2026. [2] Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. arxiv:2604.13016

Reviewer_fe13OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LENS-OPD, a framework for long-horizon on-policy distillation (OPD) built on the idea that teacher supervision is productive only where it improves the student’s future behavior and is learnable by the current student. The authors formulate long-horizon OPD as hierarchical supervision allocation and augment TCOD’s forward-to-backward horizon curriculum with a frontier-gap pace controller, a comparison of three-turn student continuations with and without a one-sentence teacher hint scored by teacher log-likelihood, a loss that trains the hint-free student to reproduce its own hinted response, and a within-turn reweighting of OPD toward tokens where the student resists a confident teacher; the validation gate controls only the two refinement terms, and rejected interventions fall back to ordinary OPD. Experiments distill Qwen3-32B into Qwen3-1.7B/4B on ALFWorld, WebShop and ScienceWorld and Qwen2.5-7B-RL into Qwen2.5-3B on ALFWorld, against vanilla OPD, TCOD, Guided-OPD and FutureBridge-OPD, with component ablations and training diagnostics. Compared with vanilla OPD, the reported advantages are most pronounced on ALFWorld success rate and on WebShop success rate for the 4B student, while on ScienceWorld the advantage over the strongest baseline is small or absent.

Critical strengths

S1. Coherent framework. The paper clearly motivates why supervision allocation should consider exploration depth, intervention selection, and token-level learning jointly, and the diagnostics in Sec. 3.2 make the case concrete. The conditional, coarse-to-fine organization is a useful perspective beyond uniform token matching.

S2. Substantial empirical evaluation. The experiments cover three environments, two student scales, a second teacher–student family, four recent baselines, and five random seeds with standard deviations, with full prompt templates and hyperparameter tables. The ALFWorld improvements are encouraging, although gains over the strongest baseline vary across settings.

S3. Useful component and dynamics evidence. Within Table 3, each level is ablated and every delta quoted in Sec. 5.4 reproduces from the table; removing guidance internalization reduces Valid Unseen SR by 6.5 points, providing evidence that this component benefits the deployable student beyond the training tasks. The training-dynamics analyses show what the method actually does during training, which is rare in this literature.

Critical weaknesses

W1. Baseline results and evaluation protocols need clarification. TCOD-F2B performs below vanilla OPD in Table 2, reversing the substantial gains reported in TCOD. Given the stated reuse of TCOD configurations, the authors should explain relevant changes in checkpoints, implementation, and evaluation settings. The ALFWorld reporting is also unclear: Table 1 gives 47.9 SR, whereas Table 3 reports 44.6/51.3 on Seen/Unseen, and B.2 specifies unseen evaluation. Please clarify whether Table 1 aggregates splits. Finally, the claim of consistent superiority should acknowledge exceptions, including the 1.7B ScienceWorld score dropping from 42.1 with OPD to 35.8 with LENS-OPD.

W2. The validation mechanism needs more direct evidence. The gate selects guidance using teacher log-likelihood differences between sampled three-turn continuations, but its relationship to actual task improvement is not established. Existing ablations support several components without isolating the acceptance rule. Always accepting guidance and randomly accepting the same proportion would provide useful controls. The methodological differences from FTB’s paired-continuation validation also deserve clearer discussion. In addition, only about 24% of recorded WebShop episodes in the two illustrated runs have complete paired validations; the authors should explain how incomplete validations affect training.

W3. Important reproducibility and cost details are missing. Hard is not a standard ALFWorld split. The paper should specify its construction, size, selection teacher, and whether its tasks were excluded from student training, particularly if it reuses TCOD’s training-derived subset. The claimed budget matching also needs explicit accounting for teacher hints, paired rollouts, and additional scoring, including total environment interactions and training time. Finally, the guidance-internalization coefficient λ_GI, the role of the guidance-score temperature τ, and the exact ALFWorld and ScienceWorld evaluation sets should be specified.

Questions for authors

Please see the Critical Weaknesses section.

I look forward to the authors’ response and will carefully consider their clarifications when updating my review.

5 comments
AuthorsOpenReview ↗

Response to Reviewer fe13 (1/3)

Thank you for recognizing the hierarchical formulation, the breadth of the evaluation, and the component analyses. We address the baseline settings, validation evidence, and reproducibility details below.

W1-1 — Why does TCOD-F2B underperform vanilla OPD, and how do our checkpoints and experimental settings differ from TCOD’s?

We directly use TCOD’s GitHub repository as our experimental codebase and closely follow its evaluation protocol. Two differences from the original TCOD experiments are relevant to the results in Table 2.

First, the teacher checkpoint differs. TCOD’s original GRPO-trained teacher was unavailable, so we used siaosiao/grpo-qwen2.5-7b-alfworld-global-step-150. Since the teacher supplies the distillation targets, this substitution changes the supervision received by the student and may affect the relative benefit of TCOD’s curriculum over vanilla OPD.

Second, the hardware and training precision differ. TCOD used H20 GPUs with FP32 training, whereas our experiments used H100 GPUs with BF16 training. The difference in numerical precision may affect optimization and contribute to variation in final performance.

These differences may contribute to the discrepancy with TCOD’s originally reported results. Within our experiments, all methods share the same teacher checkpoint, hardware, training precision, and evaluation protocol, ensuring a controlled comparison under our experimental setup. We will explicitly document these differences in the revised manuscript.

W1-2 — Does Table 1 aggregate the ALFWorld Seen and Unseen splits, and how does it reconcile with Table 3 and Appendix B.2?

Table 1 pools 140 Valid Seen and 134 Valid Unseen tasks, whereas Table 3 reports the splits separately. Weighting the unrounded split success rates by their task counts gives 47.88%, which rounds to the 47.9% reported in Table 1.

\frac{140\times44.57\\%+134\times51.34\\%}{274}\approx47.88\%.

The two tables are therefore consistent. We will state the aggregation explicitly in the Table 1 caption and update Appendix B.2 to describe both evaluation splits. Train Hard is reported separately and is excluded from this aggregate.

W1-3 — How should the lower ScienceWorld score be interpreted alongside higher success rate, and how should the superiority claims be qualified?

On 1.7B ScienceWorld, LENS-OPD increases task success from 14.3% to 16.9%, while the average final score decreases from 42.1 to 35.8. The distinction lies in how the two metrics treat unfinished tasks. Under the 30-interaction budget, SR measures whether a task is successfully completed. Score also credits partial progress retained when an unfinished trajectory exhausts its interaction budget, whereas negative scores from terminal failures are mapped to zero, as specified in Appendix B.2.

Our trajectory analysis makes this distinction concrete. Only 34.4% of OPD episodes reach an environment terminal state, compared with 55.25% for LENS-OPD. Thus, many OPD trajectories exhaust the budget before reaching a terminal outcome and retain their intermediate rewards despite not completing the task. LENS-OPD brings more trajectories to terminal outcomes: some succeed, while others fail and lose previously accumulated partial credit. Consistent with this pattern, the LENS-OPD trajectories have a mean within-episode peak score of 50.30 but a mean final score of 35.81 after negative scores are mapped to zero. The higher average score of OPD therefore should not be interpreted as a higher task-completion capability. For the objective of successfully completing tasks within a fixed interaction budget, SR is the more direct metric, and LENS-OPD improves it by 2.6 percentage points.

We will refine the manuscript’s performance claims to distinguish these metrics and accurately reflect the reported results.

AuthorsOpenReview ↗

Response to Reviewer fe13 (2/3)

W2-1 — Why use teacher likelihood to validate guidance, and what evidence connects this signal to actual task progress?

We use teacher likelihood to obtain a dense measure of local improvement within a short validation horizon. Evaluating terminal success would require both branches to continue until termination or the interaction limit, substantially increasing the cost of each validation. Moreover, for a small student, both branches may ultimately fail even when a hint improves the current decision, making binary success too sparse to distinguish useful guidance. The teacher’s length-normalized likelihood provides a graded comparison of the continuations that the same frozen student generates, with and without a hint, from the same restored state.

We further examine whether this signal agrees with task progress using an ALFWorld symbolic planner independent of the teacher. At the Top-KL positions selected during training, we compare the remaining steps to task completion after up to three matched continuation turns. Among 80 valid Qwen3 paired replays, only 3 cases (3.75% of all pairs) show erroneous acceptance: the likelihood gate accepts the hinted branch even though the planner judges it farther from task completion. This directly tests the concern that greater teacher agreement could come at the expense of actual progress. A complementary evaluation with GPT-5.6 sol agrees with the likelihood-based comparison in 70 of 80 cases (87.5%). These external checks support teacher likelihood as a practical proxy for local guidance utility. The Always and Random acceptance controls reported below (W2-2) further test whether this selection translates into improved task success after training. The gate activates GI and token focusing only when the hinted continuation receives a higher score. Rejected interventions retain ordinary OPD supervision on the original student trajectory.

W2-2 — Does positive-gain selection improve learning compared with accepting all hints or randomly accepting a similar proportion?

We added two controls to examine whether the acceptance rule improves learning beyond simply providing more guidance. Always accepts every valid candidate, while Random accepts candidates independently with a fixed probability. The observed acceptance rates are 71.0% for LENS-OPD and 73.7% for Random, providing an approximately rate-matched comparison. We evaluate all three methods at 250 optimizer updates on the same ALFWorld tasks and report mean success rate ± standard deviation over five evaluation seeds.

Acceptance ruleOverall SR (%)Seen SR (%)Unseen SR (%)
LENS-OPD47.9 ± 2.244.6 ± 2.951.3 ± 3.0
Random44.6 ± 2.041.1 ± 2.948.2 ± 1.6
Always39.6 ± 2.137.0 ± 3.842.2 ± 1.9

LENS-OPD improves overall SR by 3.3 percentage points over Random and 8.3 points over Always, with gains on both Seen and Unseen tasks. The advantage over Random supports the value of selecting useful interventions at a comparable acceptance rate, while the comparison with Always shows that accepting more guidance does not necessarily improve learning. These results directly support the acceptance mechanism and complement the component ablations in Table 3.

W2-3 — How does LENS-OPD differ from FTB’s paired-continuation validation in its intervention and subsequent learning?

FTB’s paired-continuation validation informed our design. Both methods compare short continuations from a restored state, but they differ in what the validation tests and what the student learns from an accepted intervention.

FTB executes a teacher-generated bridge action at the intervention turn before returning control to the student. LENS-OPD instead provides a short textual hint, with the same frozen student generating both the intervention action and all subsequent actions. Our validation therefore tests whether the current student can translate guidance into an action it can execute and continue from the resulting state. For a small model, continuing from a state reached by the teacher and reaching that state through its own action require different capabilities. This distinction implements the paper’s emphasis on current learnability alongside future utility.

After acceptance, guidance internalization transfers the hinted student’s response distribution into the policy conditioned on the same decision context without the hint. Token focusing separately reweights the original on-policy response. Together, these mechanisms connect guidance that the student can execute during validation to behavior it can reproduce without hints at deployment. We will clarify this relationship to FTB in the method and related-work discussion.

AuthorsOpenReview ↗

Response to Reviewer fe13 (3/3)

W2-4 — With only about 24% of WebShop validations completed, how do incomplete pairs affect training?

When a validation pair is incomplete, the original student trajectory still contributes to ordinary OPD training; only GI and the associated token reweighting are skipped. Thus, the approximately 24% validation completion rate limits the availability of additional refinement, not the use of the original trajectories for training. Most incomplete pairs arise from timeouts during the additional validation rollout, which requires restoring the shopping session by replaying earlier actions and generating a short student continuation. We will clarify this fallback behavior in Appendix D.

W3-1 — How is the ALFWorld Hard subset constructed, which teacher selects it, and are its tasks included in student training?

Hard is TCOD’s train_hard subset of 121 ALFWorld training tasks. TCOD constructed it by selecting tasks on which its GRPO-fine-tuned Qwen2.5-7B teacher failed in all ten sampled attempts. All 121 tasks are included in our student training pool. We will label this subset Train Hard and explicitly document its construction and selection teacher. Train Hard measures performance on difficult training tasks, while Valid Seen and Valid Unseen contain held-out tasks in seen and unseen environment layouts, respectively. We report these splits separately to distinguish performance on difficult training tasks from evaluation on held-out tasks.

W3-2 — What budget is matched across methods, and what additional training time and environment interactions does validation require?

All methods are trained for 250 optimizer updates. We report wall-clock time and total environment interactions in the ALFWorld runs below. Wall-clock time is measured from the first to the 250th logged update and includes waiting for rollout data. Interaction counts include the main rollouts, validation branches, and state-restoration replay.

Training metricVanilla OPDLENS-OPD
Optimizer updates250250
Wall-clock time (h)16.522.5
Mean update interval (s)238.0325.6
Total environment interactions69,81557,650

LENS-OPD uses 17.4% fewer environment interactions, including all validation and replay interactions. Its adaptive horizon curriculum limits rollout depth during the early and middle stages of training and progressively expands it as the student learns. The resulting reduction in rollout interactions more than offsets the additional interactions required for validation.

LENS-OPD takes approximately 1.37× the wall-clock time of vanilla OPD, reflecting the additional computation for hint generation, continuation generation and scoring, and state restoration. We reduce this overhead by reusing the original student continuation as the no-hint branch whenever available. This occurred in 93.0% of completed validation pairs, so most validations require only one additional student continuation and a short teacher hint.

We will clarify that the matched budget refers to optimizer updates and explicitly report the total interaction counts and measured wall-clock overhead.

W3-3 — What are the GI coefficient and guidance-score temperature used for, and which ALFWorld and ScienceWorld evaluation sets are used?

The default GI coefficient is λGI\lambda_{\mathrm{GI}} = 1.0; it scales the cross-context GI loss before addition to the OPD loss. The guidance-score temperature τ = 1.0 belongs to an optional turn-level weighting path. In the archived ALFWorld configuration, this weight is capped at 1.0 and remains 1.0 for every turn. It therefore does not change the reported ALFWorld loss weights. The acceptance rule depends only on whether Δfuture\Delta_{\mathrm{future}} > 0 and is independent of τ.

Table 1 evaluates ALFWorld on all 140 Valid Seen and 134 Valid Unseen tasks and reports their task-count-weighted aggregate. Train Hard is separate. ScienceWorld uses 1,308 evaluation instances across 13 task types from the official dataset, with at most 30 interactions per episode. We will include these exact evaluation definitions in Appendix B.2.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

We thank all three reviewers for their careful evaluation and constructive feedback. We are encouraged by the recognition of our hierarchical formulation, broad empirical evaluation, and component analyses. Our responses clarify the methodological contribution of LENS-OPD and provide additional evidence on guidance validation, hyperparameter sensitivity, and computational cost.

1. The contribution is the coordinated allocation of supervision across trajectory, turn, and token levels. The curriculum controls which states the student encounters; paired validation tests whether guidance improves the student’s own continuation; accepted interventions activate guidance internalization and token focusing. This connects the paper’s two central criteria—future utility and current learnability—to the behavior of the deployable, hint-free student.

2. Additional experiments support the selective validation mechanism. On ALFWorld, positive-gain acceptance achieves 47.9% overall SR, compared with 44.6% for Random acceptance and 39.6% for Always acceptance. A teacher-independent symbolic planner identifies erroneous acceptance in 3 of 80 valid paired replays, while a complementary external-model evaluation agrees with the likelihood comparison in 70 of 80 cases. These checks provide evidence beyond teacher likelihood itself while identifying cases where the proxy diverges from task progress.

3. The gains persist across the tested hyperparameter settings. The default values were set without validation-set tuning. Across the additional sensitivity experiments, overall ALFWorld SR ranges from 44.0% to 52.6%, compared with 36.4% for vanilla OPD. The results support effectiveness beyond a single choice of loss balance or refinement parameters.

4. We also clarify the experimental protocol and measured cost. The matched training budget is 250 optimizer updates. In the measured ALFWorld runs, LENS-OPD uses 17.4% fewer total environment interactions, including validation and replay, while requiring approximately 1.37× the wall-clock time of vanilla OPD. Reusing the original continuation in 93.0% of completed validations reduces the additional generation required. We explain the TCOD checkpoint and execution differences, reconcile the ALFWorld split reporting, and specify the treatment of incomplete validations and the Train Hard subset.

We will incorporate these clarifications and additional results into the revision. Together, the existing results and new analyses strengthen the evidence for allocating supervision according to what helps the student progress and what it can learn.

AuthorsOpenReview ↗

Summary of Manuscript Revisions

We have incorporated the key clarifications and additional results from our rebuttal into the revised manuscript. All additions and revised passages are marked in blue:

  • Guidance validation and acceptance controls — Section 5.5, Table 4, and Appendix E.1. We added the Always and approximately rate-matched Random controls to the main text. LENS-OPD achieves 47.9% overall ALFWorld SR, compared with 39.6% and 44.6%, respectively. Appendix E.1 reports the independent symbolic-planner and external-model checks of guidance utility.

  • Methodological distinctions and intervention selection — Sections 2.2 and 4.3, and Appendix E.2. We clarify the distinction between FTB’s teacher-executed bridge action and LENS-OPD’s student-executed guidance. We also clarify the per-turn KL proposal score and provide the trajectory-position analysis in the appendix.

  • Hyperparameter sensitivity and computational cost — Section 5.5 and Appendices E.3–E.4. Table 7 reports sensitivity to GI strength, token focusing, and validation length. Table 8 reports wall-clock time and total environment interactions, including validation and replay.

  • Evaluation and reproducibility — Table 1, Appendices A–B, and Appendix D.1. We specify the Seen/Unseen aggregation for ALFWorld, the construction and training inclusion of Train Hard, the evaluation sets, and the relevant baseline settings. We also document continuation reuse and the ordinary-OPD fallback for incomplete validations.

  • ScienceWorld outcomes — Section 5.2 and Appendix E.5. We distinguish task success from final partial-progress score and include the termination and peak-versus-final-score analysis supporting this interpretation.

Area_Chair_yAbeOpenReview ↗

Discussion

Dear Reviewers,

We are reaching the end of the discussion period! If the authors have responded to your review, please read their response, leave a comment, and update your rating if your assessment has changed.

Thank you,

AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.