- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper tests whether student–teacher disagreement indicates where on-policy distillation (OPD) supervision helps. For each sampled state, it executes every candidate action with a fixed continuation controller and labels the state beneficial, unnecessary or harmful by the teacher-minus-student expected utility N(s). With a Qwen3-1.7B student and a Qwen3-32B teacher on ALFWorld (120 states) and WebShop (100 states), only 17.44% and 35.07% of states are beneficial, and disagreement-based proxies are at or below chance. A 16-step prefix estimate (H16) identifies beneficial states (AUROC 0.86–0.87; 0.80/0.79 on new tasks), and HSNG, which applies teacher targets only to H16-selected states, reduces off-selection drift in a single-seed micro-training run.
Critical strengths
-
A sharp, counterfactual question. The audit directly tests a common but unchecked assumption of selective OPD, with frozen candidate sets, paired seeds and a preregistered margin.
-
Careful protocol hygiene. Populations are frozen before outcomes are joined, ScienceWorld is excluded by a preregistered gate and reported (Appendix G), and the frozen H16 rule is re-checked on disjoint new tasks with independent rollouts (Table 3).
Critical weaknesses
W1. It is unclear whether the labels reliably measure benefits to the student.
The undefined “progress” term has a large effect: removing it reduces the beneficial share on WebShop from 35% to about 9–10% (Appendix B). All candidate actions are also evaluated using Qwen3-32B to continue the task, so the labels may reward actions that the teacher can follow up on but the 1.7B student cannot. Finally, each candidate uses only four rollouts, while the labeling margin is 0.01, and label stability is not checked. The paper should define progress and test whether the labels remain consistent with student continuations and independent rollout samples.
W2. The training experiment does not establish that HSNG’s state selection improves performance.
HSNG explicitly anchors unselected states to the initial student distribution, which is also the reference used to measure drift. The reported 90.60% drift reduction therefore does not, by itself, show that H16 selects useful states. The experiment uses one seed and 85 states, only six of which are selected. It includes no trained random-gate baseline with the same anchoring and reports no task-success results. The utility point estimate also favors Exact OPD over HSNG (0.0173 versus 0.0030; Table 6). A comparison of held-out task success across Exact OPD, HSNG, and a matched random gate, over multiple seeds and including the added rollout cost, would provide stronger evidence.
Questions for authors
Q1: How is “progress” defined and computed in each environment? On ALFWorld, how do the label proportions and agreement with the original labels change under success-only utility?
Q2: Keeping the states, candidate actions, and utility fixed, how often do the labels change when the student replaces the teacher as the continuation controller?
Q3: How well do the labels agree between disjoint 2/2 splits of the four rollouts per candidate?
Q4: Could you compare held-out task success for Exact OPD, HSNG, and a trained random-gate baseline with the same number of selected states and the same anchoring as HSNG? Please report results across multiple seeds and the additional rollout cost.
Thank you for your comments.
W1. What BNA measures
BNA estimates how expected environmental utility changes when the current action is drawn from the teacher's distribution instead of the student's. At each state, we evaluate both distributions over the same candidate actions and use the same policy to continue from each action. This gives a comparison of the two action distributions under a specified continuation policy. We will add the environment-specific definitions of
progressto the main text.W2. State selection and parameter updates
Table 4 evaluates state selection at a fixed number of teacher targets. H16's target-level utility exceeds the analytic expectation for uniform random selection by 0.0359 on ALFWorld and 0.0359 on WebShop. Thus, at the same selection count, H16 assigns teacher targets to states with higher terminal gains on average.
Table 5 evaluates the combined effect of H16 selection and anchoring during training. Relative to Exact OPD, HSNG reduces mean KL drift on unselected states by 90.60%, and all six selected states move toward their teacher targets. As you point out, Table 6 shows a larger mean conditional utility shift for Exact OPD (0.0173) than for HSNG (0.0030) across all 85 states. Both 95% intervals include zero.
Q1. How is progress computed, and how do the ALFWorld labels change under success-only utility?
On ALFWorld,
progressis the environment's binarywonflag and equalssuccess. Giving success and progress equal weight therefore yields the same utility as using success alone. At the original threshold, all state labels agree (100% agreement), and the weighted shares in Table 1 remain 17.44% beneficial, 65.29% unnecessary, and 17.27% harmful.On WebShop,
progressis the environment's purchase reward on a scale from 0 to 1. It is zero before purchase. At purchase, the environment awards partial credit according to how well the item meets the requested attributes, options, and price constraint, taking product type into account. A purchase counts as a success only if it earns full credit. Removing partial credit reduces the beneficial share from 35.07% under the primary utility to about 9–10% under success-only utility (Appendix B).Q2. How often do labels change under student continuation?
Table 3 evaluates H16 on new tasks under the same continuation policy. The selected states have mean terminal gains of +0.10 on ALFWorld and +0.07 on WebShop.
Q3. How well do labels agree across 2/2 rollout splits?
Appendix C.1 uses two of the four rollouts per candidate to score the prefix and the other two to evaluate terminal gain. Across all six directed splits, the state-pair-weighted mean terminal gain of the selected states is positive in both environments. The lowest ALFWorld estimate is +0.0586. To report per-state label agreement, we will use the same four rollouts and compute a terminal label from each half of each of the three unique 2/2 splits. For each split, we will report overall agreement and the proportion of states for which one half gives a beneficial label and the other a harmful label.
Q4. Training controls, held-out success, and cost
We will compare Exact OPD, HSNG, and a trained random gate under matched update settings. The random gate will select the same number of teacher targets as HSNG and apply the same anchoring to unselected states. Across multiple training seeds, we will report task success on held-out tasks executed entirely by the student. We will also report the additional environment steps and model calls required to compute H16.