acceptodds
Under review as a conference paper at ICLR 2027

Disagreement Is Not Guidance: Auditing and Selecting Supervision in Agentic On-Policy Distillation

Abstract

On-policy distillation (OPD) uses teacher supervision on states visited by the student. Action- distribution disagreement alone gives little information about whether teacher guidance improves environmental utility. We propose Behavioral Necessity Audit (BNA), which evaluates student and teacher distributions on a fixed executable candidate set and shared continuation policy. BNA jointly measures action-distribution distance, executable outcome distance, and the conditional utility gain from shifting probability toward the teacher. On frozen ALFWorld and official full WebShop populations, outcome-aware transport costs are 93.16% and 93.40% lower than identity-based costs, while beneficial guidance occurs on 17.44% and 35.07% of states. We then propose Horizon-Split Necessity Gating (HSNG), which uses a short executable prefix to select teacher targets. H16 identifies terminally beneficial guidance with AUROCs of 0.8595 and 0.8693 on the original populations. On disjoint follow-up populations of 80 new states from 40 new task clusters per environment, the frozen rule reaches AUROCs of 0.80 and 0.79, with positive accepted terminal gains of and . In a matched, single-seed ALFWorld micro-update, HSNG reduces mean off-selection KL drift by 90.60% while all six selected states move toward their teacher targets. These results ground supervision allocation in conditional environmental utility rather than surface action disagreement.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, language agents, knowledge distillation, selective supervision, behavioral evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5oxROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper tests whether student–teacher disagreement indicates where on-policy distillation (OPD) supervision helps. For each sampled state, it executes every candidate action with a fixed continuation controller and labels the state beneficial, unnecessary or harmful by the teacher-minus-student expected utility N(s). With a Qwen3-1.7B student and a Qwen3-32B teacher on ALFWorld (120 states) and WebShop (100 states), only 17.44% and 35.07% of states are beneficial, and disagreement-based proxies are at or below chance. A 16-step prefix estimate (H16) identifies beneficial states (AUROC 0.86–0.87; 0.80/0.79 on new tasks), and HSNG, which applies teacher targets only to H16-selected states, reduces off-selection drift in a single-seed micro-training run.

Critical strengths
  1. A sharp, counterfactual question. The audit directly tests a common but unchecked assumption of selective OPD, with frozen candidate sets, paired seeds and a preregistered margin.

  2. Careful protocol hygiene. Populations are frozen before outcomes are joined, ScienceWorld is excluded by a preregistered gate and reported (Appendix G), and the frozen H16 rule is re-checked on disjoint new tasks with independent rollouts (Table 3).

Critical weaknesses

W1. It is unclear whether the labels reliably measure benefits to the student.

The undefined “progress” term has a large effect: removing it reduces the beneficial share on WebShop from 35% to about 9–10% (Appendix B). All candidate actions are also evaluated using Qwen3-32B to continue the task, so the labels may reward actions that the teacher can follow up on but the 1.7B student cannot. Finally, each candidate uses only four rollouts, while the labeling margin is 0.01, and label stability is not checked. The paper should define progress and test whether the labels remain consistent with student continuations and independent rollout samples.

W2. The training experiment does not establish that HSNG’s state selection improves performance.

HSNG explicitly anchors unselected states to the initial student distribution, which is also the reference used to measure drift. The reported 90.60% drift reduction therefore does not, by itself, show that H16 selects useful states. The experiment uses one seed and 85 states, only six of which are selected. It includes no trained random-gate baseline with the same anchoring and reports no task-success results. The utility point estimate also favors Exact OPD over HSNG (0.0173 versus 0.0030; Table 6). A comparison of held-out task success across Exact OPD, HSNG, and a matched random gate, over multiple seeds and including the added rollout cost, would provide stronger evidence.

Questions for authors

Q1: How is “progress” defined and computed in each environment? On ALFWorld, how do the label proportions and agreement with the original labels change under success-only utility?

Q2: Keeping the states, candidate actions, and utility fixed, how often do the labels change when the student replaces the teacher as the continuation controller?

Q3: How well do the labels agree between disjoint 2/2 splits of the four rollouts per candidate?

Q4: Could you compare held-out task success for Exact OPD, HSNG, and a trained random-gate baseline with the same number of selected states and the same anchoring as HSNG? Please report results across multiple seeds and the additional rollout cost.

1 comment

Thank you for your comments.

W1. What BNA measures

BNA estimates how expected environmental utility changes when the current action is drawn from the teacher's distribution instead of the student's. At each state, we evaluate both distributions over the same candidate actions and use the same policy to continue from each action. This gives a comparison of the two action distributions under a specified continuation policy. We will add the environment-specific definitions of progress to the main text.

W2. State selection and parameter updates

Table 4 evaluates state selection at a fixed number of teacher targets. H16's target-level utility exceeds the analytic expectation for uniform random selection by 0.0359 on ALFWorld and 0.0359 on WebShop. Thus, at the same selection count, H16 assigns teacher targets to states with higher terminal gains on average.

Table 5 evaluates the combined effect of H16 selection and anchoring during training. Relative to Exact OPD, HSNG reduces mean KL drift on unselected states by 90.60%, and all six selected states move toward their teacher targets. As you point out, Table 6 shows a larger mean conditional utility shift for Exact OPD (0.0173) than for HSNG (0.0030) across all 85 states. Both 95% intervals include zero.

Q1. How is progress computed, and how do the ALFWorld labels change under success-only utility?

On ALFWorld, progress is the environment's binary won flag and equals success. Giving success and progress equal weight therefore yields the same utility as using success alone. At the original threshold, all state labels agree (100% agreement), and the weighted shares in Table 1 remain 17.44% beneficial, 65.29% unnecessary, and 17.27% harmful.

On WebShop, progress is the environment's purchase reward on a scale from 0 to 1. It is zero before purchase. At purchase, the environment awards partial credit according to how well the item meets the requested attributes, options, and price constraint, taking product type into account. A purchase counts as a success only if it earns full credit. Removing partial credit reduces the beneficial share from 35.07% under the primary utility to about 9–10% under success-only utility (Appendix B).

Q2. How often do labels change under student continuation?

Table 3 evaluates H16 on new tasks under the same continuation policy. The selected states have mean terminal gains of +0.10 on ALFWorld and +0.07 on WebShop.

Q3. How well do labels agree across 2/2 rollout splits?

Appendix C.1 uses two of the four rollouts per candidate to score the prefix and the other two to evaluate terminal gain. Across all six directed splits, the state-pair-weighted mean terminal gain of the selected states is positive in both environments. The lowest ALFWorld estimate is +0.0586. To report per-state label agreement, we will use the same four rollouts and compute a terminal label from each half of each of the three unique 2/2 splits. For each split, we will report overall agreement and the proportion of states for which one half gives a beneficial label and the other a harmful label.

Q4. Training controls, held-out success, and cost

We will compare Exact OPD, HSNG, and a trained random gate under matched update settings. The random gate will select the same number of teacher targets as HSNG and apply the same anchoring to unselected states. Across multiple training seeds, we will report task success on held-out tasks executed entirely by the student. We will also report the additional environment steps and model calls required to compute H16.

Reviewer_RZD9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether disagreement between a teacher and student model is a reliable indicator of when teacher supervision is useful in agentic on-policy distillation. The authors introduce Behavioral Necessity Audit (BNA), which compares action distributions, downstream environmental outcomes, and expected utility gains from following the teacher.

Using ALFWorld and WebShop, they show that differences in actions do not necessarily translate into differences in outcomes, and that teacher guidance is beneficial in only a subset of evaluated states.

The paper then introduces Horizon-Split Necessity Gating (HSNG), which uses short executable continuations to identify states where teacher guidance is likely to help. H16 achieves promising AUROC results, including on held-out tasks. A small-scale training experiment also shows that HSNG substantially reduces policy drift on unselected states while still learning from selected teacher targets.

Critical strengths
  1. Interesting and important research question. The paper challenges the assumption that teacher-student disagreement indicates useful supervision. I think distinguishing action disagreement from actual downstream utility is an important direction for improving on-policy distillation. The BNA framework provides a systematic way to study this, and the results across ALFWorld and WebShop are interesting.

  2. Good experimental analysis of supervision selection. The paper goes beyond measuring disagreement and evaluates whether teacher guidance actually improves outcomes. The H16 results are promising, particularly the evaluation on independent new tasks. I also appreciate the analysis of guidance necessity, utility sensitivity, and the attempt to separate useful supervision from unnecessary changes to the student policy.

Critical weaknesses
  1. Limited evidence that HSNG improves actual agent performance. While H16 identifies states where teacher guidance appears beneficial, the paper does not demonstrate that HSNG improves end-to-end task success compared to standard OPD. The training experiment uses only one seed and six selected states. HSNG also achieves smaller conditional utility improvements on selected states despite substantially reducing policy drift. It is therefore unclear whether the proposed method produces a better student or primarily prevents the student from changing.

  2. The main conclusions depend on several evaluation choices. The behavioral necessity labels depend on the frozen Qwen3-32B continuation policy, restricted candidate actions, and the utility function. Appendix B already shows substantial sensitivity to the success/progress weighting. Similarly, the reported 93% transport relief could be influenced by how environmental distances are defined and normalized. These choices make it difficult to establish how broadly the findings generalize to other models and environments.

Questions for authors
  1. How much do the results depend on the frozen Qwen3-32B continuation policy? Would the conclusions change if continuations were generated by the student or another model?

  2. The paper reports around 93% transport relief. How much of this comes from the choice and normalization of the environmental distance metric rather than genuine behavioral similarity?

  3. Appendix B shows substantial sensitivity to the utility function. How should we interpret the guidance labels when using actual task success alone, and would the main conclusions still hold?

  4. H16 achieves promising AUROC, but does selecting these states actually improve end-to-end task success after distillation compared to standard OPD?

  5. The training experiment uses only one seed and six selected states. Can the authors provide larger-scale, multi-seed results to establish whether the improvements are consistent?

  6. HSNG substantially reduces policy drift but also achieves smaller improvements on selected states than standard OPD. Is lower drift necessarily desirable without corresponding improvements in overall task performance?

  7. How much of HSNG's benefit comes from selecting the right states versus anchoring unselected states to the initial student? Could these effects be separated through additional training ablations?

  8. What is the actual computational cost of H16, including candidate-wise rollouts and model calls? Is it practical for large-scale distillation compared to simpler disagreement-based approaches?

  9. Given that existing methods are compared using reconstructed selection signals rather than complete training pipelines, how confident can we be about HSNG's advantage over these methods?

1 comment

Thank you for your comments.

Scope of the audit. At a state visited by the student, BNA estimates how much expected environmental utility changes if the current action is drawn from the teacher's distribution instead of the student's. We execute each candidate action from the same snapshot and use the same frozen Qwen3-32B controller for all subsequent steps. This lets us evaluate both distributions using the same action outcomes. The resulting gain is conditional on that continuation policy.

Interpretation of the training results. Table 4 asks which states benefit most from supervision when the number of teacher targets is fixed. Tables 5 and 6 examine how the student's distribution changes during training when H16 selects the states that receive teacher targets and the remaining states are anchored to the initial student. We will clarify in the main text that the observed reduction in drift on unselected states reflects the combined training procedure.

Q1. Do guidance gains transfer to student-controlled continuations?

Table 3 evaluates the fixed H16 rule on tasks disjoint from the original audit, with independently sampled prefix and terminal rollouts. Each environment contributes 80 states from 40 new task clusters. H16 achieves AUROCs of 0.80 on ALFWorld and 0.79 on WebShop. The selected states have mean terminal gains of +0.10 and +0.07, respectively. These results show that the rule continues to select states with positive average guidance gains on new tasks under the same continuation policy.

Q2. How much does informed selection contribute beyond the number of teacher targets and the anchoring of other states?

Table 4 compares H16 with the analytic expectation for uniform random selection at the same teacher-target count. H16's target-level utility exceeds that expectation by 0.035926 on ALFWorld and 0.035900 on WebShop. Its point estimate also remains higher when random selection matches the H16 count within each stratum of initial teacher–student KL divergence. Under the fixed candidate sets and continuation utilities, the choice of states therefore affects the expected gain from supervision.

We will train two additional controls. An equal-count random gate will use the same anchoring and update settings as HSNG, allowing us to compare selection rules. A second control will retain the H16 gate and remove anchoring, allowing us to assess anchoring for the same selected states. We will evaluate all arms across multiple training seeds using task success on the same held-out tasks, executed entirely by each trained student.

Q3. How sensitive are necessity labels to rollout sampling and candidate-set truncation?

Appendix C.1 splits the four recorded rollouts per candidate into two for prefix scoring and two for terminal evaluation. Across all six directed splits, the state-pair-weighted mean terminal gain of the selected states is positive in both environments. The lowest ALFWorld estimate is +0.0586. This measures the average gain across selected states for each split.

Appendix E reports that all 120 ALFWorld states meet the 0.80 candidate probability-mass coverage threshold for both policies. The minimum captured mass is 0.9937 for the student and 0.8416 for the teacher. We will add independent continuations for representative states near the decision thresholds and report state-level gain uncertainty, changes in individual labels, the beneficial share, and H16 AUROC. For states with lower coverage, we will also broaden the candidate set and measure how the estimated gains and labels change.

Reviewer_o3XBOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether teacher–student action disagreement identifies useful supervision in agentic on-policy distillation. It introduces Behavioral Necessity Audit (BNA), which separates action disagreement, outcome-aware transport, and conditional teacher utility, together with Horizon-Split Necessity Gating (HSNG), which selects teacher targets using 16-step executable continuations. Experiments on ALFWorld and WebShop report a mismatch between disagreement and utility, positive accepted-gain estimates on disjoint follow-up tasks, and reduced off-selection KL drift in a small matched-update study.

Critical strengths

Separating action similarity from directional guidance value is well motivated and gives the audit a clear conceptual purpose. Outcome-independent state selection, paired continuations, and task-cluster confidence intervals strengthen the empirical protocol. The follow-up evaluation on disjoint tasks with independent prefix and terminal rollouts addresses dependence in the exploratory analysis, while the sensitivity analyses and explicit failure accounting help clarify the scope of the findings.

Critical weaknesses

The main limitation is the gap between the conditional audit and the practical supervision-allocation conclusions. BNA explicitly defines utility under a shared Qwen3-32B continuation controller, which is a valid controlled estimand. However, the paper does not establish whether its beneficial/harmful labels remain informative when the student controls subsequent execution. The disjoint-task H16 validation strengthens generalization across tasks under the specified evaluator, but does not resolve this dependence on the continuation policy.

The training evidence does not isolate the contribution of informed selection or establish a benefit for student task performance. Equation 9 combines H16 selection with explicit anchoring of unselected states to the initial student, while Exact OPD applies teacher targets everywhere. The equal-count random comparison in Table 4 is an analytic target-level reference, not a trained control. Consequently, the substantial reduction in off-selection KL drift cannot be attributed specifically to informative selection, and its value for preserving useful behavior remains untested. The single-seed micro-study contains only six selected states from three task clusters. Moreover, Table 6 reports larger selected-state conditional-utility point estimates for Exact OPD (0.1444 versus 0.0253), while both full-cohort intervals include zero. These results document a trade-off, but do not establish a practical advantage for HSNG.

The reported utility/margin sensitivity analyses and complementary rollout-slot checks are useful, but uncertainty in individual necessity labels remains insufficiently characterized. The primary margin is 0.01, and candidate utilities are estimated from a small number of continuations. Task-cluster intervals summarize variation across the observed tasks without fully characterizing how within-state rollout uncertainty affects classification near the threshold. Candidate-support truncation is an additional conditionality, although the high reported ALFWorld captured mass makes it a secondary concern.

Questions for authors

1. Does the shared continuation controller measure guidance value for the student’s own execution?

BNA appropriately states that its labels are conditional on the continuation policy. The unresolved issue is whether these labels support the broader interpretation of where student supervision helps. A Qwen3-32B controller may recover from candidate actions whose consequences the student could not handle, potentially changing the magnitude or sign of the teacher advantage. Could the authors repeat a representative subset with frozen-student and mixed continuation policies, keeping states, candidate actions, horizons, and utility definitions fixed? Reporting label agreement, beneficial/harmful shares, and whether H16-selected states retain positive terminal gains under student-controlled continuation would directly test this transfer. The independent new-task evaluation is valuable, but task transfer under the specified evaluator does not establish continuation-policy transfer.

2. Does HSNG improve learning beyond selecting fewer teacher targets and anchoring the remaining states?

Equation 9 directly trains unselected states toward their initial-student distributions, so reduced KL drift on those states is partly aligned with an explicit training objective. Table 4 supports selection information at the target level, but Appendix C.3 confirms that the random comparison is not a third trained arm. Could the authors add a trained equal-count random gate with identical anchoring, together with an H16-gated variant without anchoring, and evaluate held-out student-only task performance across training seeds? The random gate should match the teacher-target count and use the same anchoring and update settings as HSNG. Table 6 measures probability changes on fixed candidates under fixed continuation utilities, and Exact OPD has larger selected-state utility point estimates. Thus, neither teacher-target movement nor reduced off-selection drift alone establishes that HSNG preserves useful behavior or improves learned task performance. These controls would isolate the learning contribution of informed selection and test the practical value of the reported trade-off.

3. How stable are the necessity labels to rollout uncertainty and candidate-support selection?

Appendix B already varies the utility weights and classification margin, and Appendix C.1 examines complementary rollout slots. These analyses address important sensitivities, but do not directly quantify uncertainty in each estimated N(s)N(s) near the primary decision boundaries of ±0.01\pm 0.01. Could the authors report state-level uncertainty, repeat representative near-threshold cases with more independent continuations, and quantify the resulting label changes, necessity shares, and H16 AUROC? A targeted broader-support check on lower-coverage states would additionally assess truncation sensitivity; the generally high ALFWorld captured mass should be taken into account when interpreting this check. These checks would distinguish uncertainty in individual labels from the across-task variation summarized by the cluster-bootstrap intervals.

1 comment

Thank you for your comments.

Scope of the audit. At a state visited by the student, BNA estimates how much expected environmental utility changes if the current action is drawn from the teacher's distribution instead of the student's. We execute each candidate action from the same snapshot and use the same frozen Qwen3-32B controller for all subsequent steps. This lets us evaluate both distributions using the same action outcomes. The resulting gain is conditional on that continuation policy.

Interpretation of the training results. Table 4 asks which states benefit most from supervision when the number of teacher targets is fixed. Tables 5 and 6 examine how the student's distribution changes during training when H16 selects the states that receive teacher targets and the remaining states are anchored to the initial student. We will clarify in the main text that the observed reduction in drift on unselected states reflects the combined training procedure.

Q1. Do guidance gains transfer to student-controlled continuations?

Table 3 evaluates the fixed H16 rule on tasks disjoint from the original audit, with independently sampled prefix and terminal rollouts. Each environment contributes 80 states from 40 new task clusters. H16 achieves AUROCs of 0.80 on ALFWorld and 0.79 on WebShop. The selected states have mean terminal gains of +0.10 and +0.07, respectively. These results show that the rule continues to select states with positive average guidance gains on new tasks under the same continuation policy.

Q2. How much does informed selection contribute beyond the number of teacher targets and the anchoring of other states?

Table 4 compares H16 with the analytic expectation for uniform random selection at the same teacher-target count. H16's target-level utility exceeds that expectation by 0.035926 on ALFWorld and 0.035900 on WebShop. Its point estimate also remains higher when random selection matches the H16 count within each stratum of initial teacher–student KL divergence. Under the fixed candidate sets and continuation utilities, the choice of states therefore affects the expected gain from supervision.

We will train two additional controls. An equal-count random gate will use the same anchoring and update settings as HSNG, allowing us to compare selection rules. A second control will retain the H16 gate and remove anchoring, allowing us to assess anchoring for the same selected states. We will evaluate all arms across multiple training seeds using task success on the same held-out tasks, executed entirely by each trained student.

Q3. How sensitive are necessity labels to rollout sampling and candidate-set truncation?

Appendix C.1 splits the four recorded rollouts per candidate into two for prefix scoring and two for terminal evaluation. Across all six directed splits, the state-pair-weighted mean terminal gain of the selected states is positive in both environments. The lowest ALFWorld estimate is +0.0586. This measures the average gain across selected states for each split.

Appendix E reports that all 120 ALFWorld states meet the 0.80 candidate probability-mass coverage threshold for both policies. The minimum captured mass is 0.9937 for the student and 0.8416 for the teacher. We will add independent continuations for representative states near the decision thresholds and report state-level gain uncertainty, changes in individual labels, the beneficial share, and H16 AUROC. For states with lower coverage, we will also broaden the candidate set and measure how the estimated gains and labels change.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.