acceptodds
Under review as a conference paper at ICLR 2027

On the Geometry of On-Policy Distillation

Abstract

On-policy distillation (OPD) improves language-model reasoning, but the geometry of its attention weight updates remains poorly understood. We compare OPD with supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR). Endpoint diagnostics place OPD in a relaxed off-principal regime: more selective and spectrum-preserving than SFT, but less constrained than RLVR, with qualitative replication on Llama-3.2-3B. Our central trajectory-level finding is subspace locking: cumulative OPD updates align early with a persistent low-dimensional channel. Constraining subsequent training to an early rank-16 subspace leaves OPD performance essentially unchanged, whereas SFT degrades by a much larger margin. Token sparsification and off-policy rollouts preserve the rank trajectory, whereas changes to the objective signal affect its geometry. Mean-broadcast and objective-mixing controls probe this sensitivity further, while exposing confounding by training collapse and unequal gradient scales. Together, these findings characterize OPD's update trajectory and its functional role without positing a universal locking mechanism or claiming efficiency gains.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Natural Language Processing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dQQAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates the geometric properties of parameter updates in LLMs under RLVR, SFT, and OPD by analyzing various metrics of parameter changes before and after training, as well as conducting experiments that constrain the parameter update space during subsequent training. The experiments show that the geometric properties of OPD updates generally lie between those of SFT and RLVR, and that OPD exhibits more stable parameter update directions.

Critical strengths
  1. The methodology is relatively rigorous, with few exaggerated claims or overly idealized assumptions.
  2. The experimental findings provide useful insights into the differences in update geometry among RLVR, SFT, and OPD.
  3. The experiments are relatively comprehensive, and most of the paper's conclusions are supported by experimental evidence.
Critical weaknesses
  1. The experimental settings are not sufficiently comprehensive. For example, in Section 4.2, the experiments validating the narrow update subspace of OPD are entirely based on Qwen3 models. However, models with different parameter counts within the same family may still exhibit similar logit distributions. Such similarity between the student and teacher could be one of the reasons behind the narrow update subspace observed in OPD. Therefore, experiments conducted solely on Qwen3 models are insufficient to attribute this phenomenon to OPD itself.
  2. The experimental comparisons are not sufficiently fair. For example, in Section 3, SFT is compared against the base model, whereas RLVR and OPD are initialized from SFT models. These differences in initialization may lead to different geometric behaviors. Furthermore, their batch sizes, training steps, and other hyperparameters also differ.
  3. The defensive writing is too prominent, and the language often feels unnatural. First, dashes are used excessively, which is not a conventional writing style. Second, many sections contain defensive statements, such as "so it is not a clean isolation of credit variation" and "and denser sweeps and independent seeds remain necessary". While such statements can help prevent readers from misunderstanding the results, they also make the conclusions less convincing. Why not address these limitations through better experimental design in the first place?
Questions for authors
  1. Which experiments and conclusions do you consider relatively more rigorous and more likely to generalize, and why?
  2. What advantages do you think the stability of parameter update directions in OPD could bring?
Reviewer_FWZPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper compares on-policy distillation (OPD) with supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR), and explores their differences in the geometry of the learning process. The authors follow the weight-space diagnostics of [Zhu et al., 2025] and extend their analysis framework. They find that the learning process of OPD lies between SFT and RLVR in parameter space, and OPD's updates enter a fixed subspace in the early stage. They also conduct experiments and find that the subspace depends on the training objective but remains unchanged with token selection.

Critical strengths
  • The main strength of the paper is a thorough comparison of OPD against SFT and RLVR on the parameter space. The experiments are well-conducted, and all experimental details are included in the paper. This makes the results trustworthy and reproducible.
  • The paper also extends the analysis framework of [Zhu et al., 2025] and provides further researchers with a more integral analysis process.
Critical weaknesses
  • My main concern is that the paper stops at phenomenon description. However, the paper never asks the next question: what should future research do differently because of this? I find the phenomenon hard to value without at least one experiment showing how these insights can be embedded into a better algorithm.
  • The paper does not explain why the phenomenon happens. If there's no direct explanation for the cause, the finding is a well-measured effect with no clear reason.
Questions for authors
  1. Can the authors give one or two concrete design choices that follow from the geometry and lead to better results in the training process? It will make the results more valuable if a changed algorithm can yield better results given the phenomenon.
  2. Is there any theoretical or experimental explanation for what causes the early subspace locking?
Reviewer_cuYAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the geometry and training dynamics of on-policy distillation (OPD) for large language models, comparing it with supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR). The authors analyze attention-weight updates using geometric diagnostics, including update sparsity, spectral structure, effective dimensionality, and subspace alignment. Experiments with Qwen and Llama models show that OPD exhibits an intermediate update geometry between SFT and RLVR, with greater selectivity and spectral preservation than SFT but less constrained updates than RLVR. The paper further identifies an early subspace-locking phenomenon, in which OPD's cumulative updates align with a persistent low-dimensional subspace during training. By projecting subsequent gradients onto an early rank-16 subspace, the authors observe that OPD largely maintains its performance, whereas SFT experiences a more substantial degradation. Additional controlled experiments involving token sparsification, rollout-policy changes, and objective mixing investigate the factors associated with this behavior and suggest that the update geometry is more sensitive to objective composition than to token coverage or rollout-policy changes.

Critical strengths
  1. The paper investigates an important and relatively underexplored aspect of on-policy distillation (OPD), focusing on the geometry and evolution of model parameter updates during training. Understanding how OPD update directions evolve, particularly in comparison with SFT and RLVR, provides useful insights beyond final benchmark performance and contributes to a better understanding of LLM post-training behavior.

  2. The paper provides extensive experimental evaluations using multiple geometric metrics, reasoning benchmarks, and two model families (Qwen and Llama). Beyond endpoint diagnostics, the authors investigate the evolution of effective dimensionality and subspace alignment throughout training. The rank-16 projected continuation experiment further examines whether the early identified subspace can support subsequent learning, showing that OPD largely maintains its performance while SFT experiences greater degradation. These experiments strengthen the empirical findings by connecting the observed update geometry with its functional implications.

  3. The paper conducts controlled experiments involving token sparsification, rollout-policy changes, and objective mixing to investigate the factors associated with OPD's update geometry. These analyses provide additional evidence about the robustness of the observed geometric behavior under different training conditions and help examine the role of objective composition in shaping update trajectories.

Critical weaknesses
  1. The comparison between SFT, OPD, and RLVR does not sufficiently isolate the effects of their training objectives from other experimental factors. The methods differ in training budgets, batch sizes, learning rates, and initialization. In particular, SFT starts from a pretrained base model, whereas OPD and RLVR start from an SFT anchor. Although the authors explain the different training budgets based on performance convergence, these differences may still confound the observed geometric patterns and make it difficult to attribute them specifically to the training objectives.
  2. The geometric analysis focuses on attention-weight matrices, without examining MLP, embedding, and output-head parameters. This limits the generality of the conclusions about model update geometry. In addition, the main results rely on single training runs, and the fluctuations observed in Figures 6 and 7 raise concerns about experimental reliability. Independent runs and a clearer treatment of evaluation uncertainty would strengthen the findings. A more systematic layer-by-layer analysis could also provide additional insights into the observed geometric behavior.
  3. Although the paper provides trajectory-level analysis and rank-constrained continuation experiments, its additional contributions beyond closely related studies are not sufficiently established. Previous research, including Cai et al. (2026), has investigated low-rank concentration and stabilization of OPD updates. A clearer comparison is needed to explain what new understanding the trajectory analysis, projected continuation experiments, and objective-level controls provide beyond existing observations.
  4. The rank-16 projected continuation experiments require more careful interpretation. In Figure 6, OPD and SFT have substantially different amounts of remaining performance improvement at the projection starting point (approximately 4% versus 20%). Therefore, the smaller performance degradation observed for OPD may partly reflect its smaller remaining adaptation requirement rather than the particular effectiveness of the identified early subspace. Reporting performance gains relative to the shared projection starting point, together with uncertainty estimates and a matched-rank random-subspace control, would provide stronger evidence.
Questions for authors
  1. Section 4.2 projects gradients onto the early subspace, while the training setup uses Adam. Since Adam's coordinate-wise rescaling does not generally preserve this constraint, is the projection applied to the raw gradient or to the final parameter update? How are optimizer states and weight decay handled? Could the authors report the relative magnitude of the actual updates outside the specified subspace and clarify which parameter groups are constrained?

  2. Appendix C.3 specifies a default reference-mask fraction of 50%, while Figure 3 reports an SFT update density of 92.73%. If these values refer to the same matrix and use the stated set definitions, the conditional overlap with any 50% reference mask must be at least approximately 46.08%. However, the reported overlaps are 29.66% and 31.88%. Does Figure 3 use a different mask fraction, denominator, or aggregation procedure?

  3. Could the authors make the operational definition of "locking" more precise? Section 4.1 describes stable rank as remaining within a band rather than trending, but Figure 4(a) shows OPD's stable rank increasing from approximately 19 to 27. This increase does not itself rule out stable directions, but the distinction between low dimensionality, stable dimensionality, and stable orientation needs clarification. How are the bandwidth, acceptable drift, and similarity threshold of approximately 0.7 chosen, and how sensitive is the early-locking conclusion to these choices?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.