acceptodds
Under review as a conference paper at ICLR 2027

IG-OPD: Information-Geometric Control for On-Policy Distillation

Abstract

On-policy distillation (OPD) trains student language models using teacher feedback on self-generated trajectories, but such feedback can become unreliable when student-generated contexts deviate from teacher-preferred trajectories. In particular, the teacher distribution conditioned on a deviated student prefix may induce a correction direction that conflicts with the target associated with a selected task-valid reference prefix. We study this phenomenon from an information-geometric perspective by modeling next-token distributions on the probability simplex equipped with the Fisher metric. Using the square-root embedding of the categorical Fisher simplex, we derive an exact finite-step identity and a necessary-and-sufficient condition under which an idealized Fisher–Rao correction toward the teacher target conditioned on a deviated prefix moves away from a selected reference target. We further exploit the relationship between Fisher–Rao distance and the Bhattacharyya coefficient to derive a tractable teacher–student discrepancy signal. Guided by these geometry-motivated heuristics, we propose IG-OPD, which combines geometry-aware token weighting, discrepancy-guided prompt routing between off-policy supervised fine-tuning and OPD, and token-wise adaptive interpolation between forward and reverse KL objectives. Experiments across textual and multimodal tasks, teacher configurations, and student scales consistently improve over strong OPD baselines. Controlled-prefix diagnostics further demonstrate that larger teacher–student discrepancy is associated with more frequent reference-direction conflict.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
On-policy distillation, Knowledge distillation, Information geometry, Large language models, Fisher–Rao geometry

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4QX3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces IG-OPD, an information-geometric framework designed to improve On-Policy Distillation (OPD) for language models. The authors identify a "reference-direction conflict": when the context (prefix) generated by the student model deviates, the supervision signal provided by the teacher model may steer the student away from the teacher distribution associated with an "effective reference prefix" for the task. Leveraging Fisher-Rao geometry, the authors derive a formal characterization of this phenomenon under finite-step updates and introduce a computationally efficient divergence metric based on the Bhattacharyya coefficient.

Building on this analysis, IG-OPD integrates three mechanisms: (1) geometry-aware token weighting to down-weight supervision signals with high divergence; (2) divergence-guided prompt routing, which dynamically allocates between Off-Policy Supervised Fine-Tuning (SFT) and On-Policy Distillation (OPD); and (3) adaptive interpolation between forward and reverse KL-divergence objectives. The method is evaluated on benchmarks for mathematical reasoning, code generation, and multimodal tasks, covering scenarios such as single-teacher, multi-teacher, and "strong-to-weak" distillation. Experimental results demonstrate robust performance improvements over existing OPD baselines, while ablation studies and diagnostic analyses using "controlled prefixes" further validate the method's effectiveness.

Critical strengths

1. Provides a unique geometric perspective on "unreliable on-policy supervision."

This paper identifies a critical limitation of On-Policy Distillation (OPD): when the context generated by a student model deviates, the feedback provided by a teacher model may misalign with the optimal guidance derived from a valid reference trajectory. Through Fisher-Rao geometric analysis, the authors provide a mathematically precise characterization of this phenomenon under idealized update conditions. This introduces a novel perspective on token selection, distinct from conventional approaches based on entropy or divergence.

2. A unified framework with strong practical utility and broad empirical coverage.

This method integrates token weighting, prompt routing, and adaptive KL interpolation through a unified "divergence signal," all without requiring any modifications to the model architecture. The experiments encompass a diverse range of task types, student models of varying scales, and multiple teacher configurations. Furthermore, the paper’s diagnostic analysis of "controlled prefixes" and its ablation studies for individual components go beyond standard benchmarking, providing robust empirical support for the effectiveness of the proposed mechanism.

Critical weaknesses

1. The fairness and statistical reliability of the primary experimental comparisons have not been sufficiently substantiated.

A major issue is that the results for several core baseline model (including the strongest competitor, Uni-OPD) are taken directly from prior literature, whereas the evaluation of IG-OPD was conducted using the authors' own experimental pipeline. Consequently, in the absence of a fully unified experimental setup, the claimed performance gains over Uni-OPD are not adequately validated. Even minor discrepancies in model configuration, evaluation protocols, or training implementation details could significantly influence the reported performance improvements.

More importantly, the paper asserts that the 95% confidence intervals, calculated via a paired hierarchical bootstrap method, for the performance gains over the strongest baseline lie entirely above zero. However, for those baselines cited from previous studies, the paper fails to explain how the "per-example predictions" were obtained, nor does it specify the bootstrap procedure used or provide the actual confidence intervals. Furthermore, while the paper reports averages based on three random seeds, it omits variability estimates in the main tables. Given that the performance gains over the strongest baseline are typically only around 1–2 percentage points, and even smaller differences are observed in some ablation studies. I believe these missing details significantly undermine confidence in the alleged "superiority" of IG-OPD.

2. The connection between the theoretical analysis and the proposed optimization rules lacks sufficient justification.

The theoretical analysis characterizes a "reference-direction conflict" through the alignment between a "deviated-prefix" and a "reference correction direction." However, this alignment cannot be directly observed during the training process; consequently, IG-OPD relies instead on the "magnitude of the teacher–student discrepancy." Yet, a large discrepancy does not necessarily imply that the supervisory signal is harmful or that the directional alignment is negative; it could just as well reflect a constructive correction provided by a more capable teacher model.

Although experiments using "controlled-prefixes" reveal a moderate correlation (Spearman correlation coefficient of 0.38) between the magnitude of the discrepancy and negative alignment, this does not prove that the discrepancy magnitude can reliably identify harmful updates. Furthermore, the particular choices of exponentially decaying token weights, discrepancy-based SFT routing, and KL interpolation are primarily heuristic rather than directly implied by the theoretical results. Moreover, comparisons with other discrepancy metrics show only marginal performance gains, which leaves the concrete advantages offered by the "information-geometric" theoretical framework somewhat unclear.

Questions for authors

Same as weaknesses.

Reviewer_xFKQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes IG-OPD, an information-geometric framework that improves on-policy distillation through adaptive token weighting, prompt routing, and KL divergence control. Experiments across math, code, and multimodal tasks show consistent improvements over existing OPD baselines.

Critical strengths
  1. A coherent framework for addressing heterogeneous OPD supervision. The paper identifies an important limitation of OPD: teacher feedback conditioned on student-generated prefixes may not always provide useful corrections.

  2. Broad empirical evaluation. The method is evaluated across mathematical reasoning, code generation, multimodal tasks, different student scales, and strong-to-weak distillation.

Critical weaknesses

1. The theoretical motivation does not fully establish the effectiveness of the practical optimization strategy. The finite-step Fisher–Rao analysis characterizes an idealized correction toward a teacher distribution conditioned on a deviated prefix. However, the practical method uses stop-gradient discrepancy-based weights and adaptive KL objectives, rather than explicitly following the analyzed Fisher–Rao geodesic. Moreover, the Bhattacharyya discrepancy measures the magnitude of the teacher–student correction, not whether its direction conflicts with the reference target. The reported correlation between discrepancy and negative reference alignment is moderate. It remains unclear how reliably these signals identify harmful supervision in general training settings.

2. The empirical gains are not sufficiently disentangled from the effects of additional training strategies and computational cost. IG-OPD combines three mechanisms, including routing high-discrepancy prompts to teacher-forced SFT and retaining additional rollouts for prompts assigned to OPD. These choices may change the effective training data, optimization dynamics, and computational budget relative to baseline methods. Although the paper includes extensive ablations, it is not sufficiently clear whether the overall gains persist under strictly matched compute and sample budgets, or how much improvement comes specifically from the information-geometric signal rather than from adaptive SFT–OPD scheduling and KL mixing. A more controlled comparison would strengthen the empirical claims.

Questions for authors

How strong is the connection between the geometric analysis and practical training? Since the Bhattacharyya discrepancy measures correction magnitude rather than directional conflict, can the authors demonstrate that it predicts harmful updates and reference-direction displacement across different models, tasks, and training stages? How should the moderate observed correlation be interpreted?

Are the improvements robust under compute-matched comparisons? Can the authors report training FLOPs or GPU hours, teacher queries, total generated tokens, and the number of optimization steps for IG-OPD and the strongest baselines? In particular, how much of the gain comes from prompt-level SFT routing compared with the two token-level controllers?

How sensitive is the method to the routing threshold and the choice of discrepancy signal? The routing threshold is calibrated using the initial student's 80th-percentile discrepancy and then held fixed. Would alternative calibration strategies or distribution shifts substantially change routing behavior and final performance? How much does the geometric signal outperform appropriately scale-matched alternatives such as symmetric KL or Jensen–Shannon divergence?

Reviewer_8EdaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes IG-OPD to improve the reliability of teacher feedback on student-generated prefixes. It analyzes reference-direction conflict using Fisher–Rao geometry and uses a Bhattacharyya discrepancy to control token weighting, SFT/OPD routing, and forward/reverse KL interpolation. Experiments cover textual and multimodal tasks, student scales, and teacher configurations, reporting consistent gains over strong baselines, including improvements of 1.5 and 1.4 percentage points over Uni-OPD in single-teacher mathematics and code generation, respectively.

Critical strengths
  1. Addresses a practical limitation of OPD with a clear definition of reference-direction conflict and geometric derivations that hold under the stated assumptions.
  2. Uses a shared discrepancy signal across three simple controllers, requiring no architectural changes or additional online correctness labels.
  3. Provides broad evaluations, component and controller/signal substitution ablations, and diagnostics involving actual optimizer updates.
Critical weaknesses
  1. Statistical support for key comparisons is insufficiently documented. Table 1 imports strong baseline results from prior work, while the text reports three-seed means and positive 95% paired hierarchical bootstrap intervals. Interval endpoints, pairing/resampling units, and the source of per-example and per-seed baseline outputs are unclear. The small differences in Tables 8–9 also lack uncertainty estimates, leaving the robustness of the reported gains difficult to assess.
  2. Geometric conflict is not yet firmly linked to harmful supervision. Proposition 3.1 measures displacement from a selected reference distribution, but valid correction after an erroneous prefix may itself differ from the original continuation. Thus, ρt<0\rho_t<0 does not establish reduced task accuracy. The correlation diagnostics need support from repair success, task correctness, and robustness across valid references. Proposition 3.3 also does not determine the specific rule λt=1−BCt\lambda_t=1-BC_t.
  3. Attribution and overall cost require stronger controls. Under the default setting, when the weight floor is inactive, wt=BCt=1−λtw_t=BC_t=1-\lambda_t, coupling KL orientation with gradient attenuation. Routing gains may also reflect task difficulty or SFT data effects. Controls that match loss scale, and random routing with matched SFT rates and budgets, would clarify the contributions. Table 11 times only the OPD branch; calibration, teacher-trajectory preparation, and total training cost should also be reported.
Questions for authors
  1. Which strong baselines were reproduced in the same pipeline? Please provide per-seed results, confidence-interval endpoints, and the bootstrap pairing/resampling protocol.
  2. Does negative Fisher alignment predict failed teacher correction or lower task accuracy? Are the conclusions stable across alternative valid references?
  3. Can you add controls that match loss scale and random-routing controls with matched SFT rates and budgets, and report total GPU-hours, student training steps, and teacher-trajectory preparation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.