acceptodds
Desk rejected at ICLR 2027

On the Orthogonal Geometry of On-Policy Distillation

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

On-policy distillation (OPD) trains a student on its own sampled trajectories using dense token-level teacher feedback. Standard low-rank adaptation (LoRA) makes OPD more parameter-efficient, but restricts learning to an initialization-dependent channel whose suitability for OPD is unclear. We introduce OP-OPD and OM-OPD, two orthonormal spectral initialization methods for low-rank OPD. They initialize the LoRA row space with, respectively, the principal and minor right-singular directions of pretrained weights, while a zero output factor preserves the initial model function. A first-step analysis shows that this choice projects the OPD gradient onto a selected subspace, making the initialization channel explicit. On three mathematical reasoning benchmarks, both methods improve over standard LoRA-OPD on aggregate mean@16 and maj@16 in two teacher–student settings. Update-trajectory and spectral diagnostics show that the two variants follow distinct pretrained channels, consistent with the proposed routing interpretation. The results identify initialization geometry as a design dimension for OPD, with gains observed across both evaluated teacher–student pairs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Adapter Row Space, Spectral Geometry, Geometry-Preserving LoRA, Low-Rank Post-Training

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_E8qMOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates three initialization impacts of low-rank adaptation on on-policy distillation. The key contributions are analyses of training trajectory, accuracy on math benchmarks, subspace overlap, entropy, stable rank, gradient magnitude, and Hill tails. Several math reasoning tasks are provided to show minor singular vectors can outperform other initialization.

Critical strengths

The paper has relatively rich analyses for low-rank on-policy distillation in training trajectory, accuracy on math benchmarks, subspace overlap, entropy, stable rank, gradient magnitude, and Hill tails. It shows minor singular vectors relatively outperforms principal singular vectors or random initialization on several different math reasoning benchmarks. The introduced method is clear, practical, and easy to understand.

Critical weaknesses

The critical weakness of the paper is its lack in novelty. As the paper noted, there are many prior literature on LoRA initialization. The use of principal and minor singular vectors are known in such papers already. Simply applying them for on-policy distillation does not provide substantially new findings. Even considering several follow-up analyses, new discovery is very limited, and they are not convincingly useful. Especially, the paper does not provide any useful justification, consequences or hypotheses from the numerical analyses. There are some restrictions in the paper's scope: the benchmark is only for math reasoning tasks; only a specific rank of 16 is considered; nothing new methodologies are provided.

Questions for authors

What happens on different ranks? What happens on different tasks other than math reasoning? What happens if we exhaustively search for the best subspace initialization? Minor singular vectors are best? Why not mixing Principal and minor in some ratios? How useful are the provided analyses? How are they related to downstream task performance, and any justification?

Reviewer_n7A6OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies LoRA initialization for on-policy distillation (OPD). It carries over, unchanged, an orthonormal initialization previously proposed for RLVR [1]. The adapter's input-side factor is set to the top (OP-OPD) or bottom (OM-OPD) right-singular directions of the pretrained weight. The output-side factor is initialized to zero, so the model starts exactly at its original state. Based on an analysis of the first SGD step, the authors interpret this initialization as selecting which components of the OPD gradient enter the update.

Experiments use two 1.5B model pairs, where each teacher is the corresponding student further trained with RL. On three math benchmarks, both variants outperform standard LoRA-OPD. However, the gain is large for one model and very small for the other. The paper then presents a spectral diagnosis, arguing that the updates remain within the initialized subspace.

[1] R. Zhang et al. Geometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in RLVR. ICML, 2026.

Critical strengths
  • A test of whether spectral initialization matters for OPD. The paper tests whether spectral initialization matters for OPD. Both variants share the same rank and scale and start from the unmodified model, so the only difference is which spectral directions the adapter starts from. The initialization scheme is taken directly from prior work on RLVR [1], but the paper evaluates it in the OPD setting. Since OPD behaves differently from both SFT and RLVR, it is not obvious that an initialization designed for RLVR carries over unchanged. However, the analyses lack a random-orthonormal control, a tuned LoRA learning rate, and repeated runs. Without these, they do not establish how much of the gain comes from the spectral selection itself, or how robust the results are.

[1] R. Zhang et al. Geometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in RLVR. ICML, 2026.

Critical weaknesses
  1. Limited novelty and missing baselines. OP-OPD and OM-OPD reuse the idea from prior work without modification [1]. The paper also does not argue why these particular subspaces are especially suited to OPD. Since gradient routing is presented as the motivation, a comparison against other gradient-based initializations [2, 3] is particularly needed, but it is missing.
  2. Insufficient experiments and unclear robustness. Without a random-orthonormal baseline and a learning-rate sweep, the experiments cannot separate the effect of spectral selection from that of orthonormality, initialization scaling, or optimization factors. Moreover, the benchmark performance of vanilla LoRA does not improve between steps 50 and 150, which raises the question of whether the baseline was sufficiently tuned. The evaluation is limited to small math-reasoning models, and each teacher is an RL-trained version of the corresponding student base model. Whether the results generalize to broader settings therefore remains untested.

[1] R. Zhang et al. Geometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in RLVR. ICML, 2026.
[2] S. Wang et al. LoRA-GA: Low-Rank Adaptation with Gradient Approximation. NeurIPS, 2024.
[3] Y. Zhang et al. LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently. ICML, 2025.

Questions for authors
  1. What are the step-0 scores of the student and the teacher? Since all low-rank runs start from the original student, these scores would indicate whether the spectral variants actually improve the student.
  2. How are maj@16 and best@16 computed? For some methods, maj@16 is lower than mean@16, and the reported values are not multiples of 1/30.
  3. Do the gains hold against a random-orthonormal initialization and against LoRA-OPD with a tuned learning rate?
  4. Do the results hold when the teacher is not an RL-trained copy of the student, and when compared against gradient-based initializations [2, 3]?
Reviewer_1KkjOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how low-rank initialization geometry can affect on-policy distillation (OPD). To do so, the authors developed two methods called OP-OPD and OM-OPD. Both methods use the principal and minor right-singular vectors, respectively, of pretrained weight matrices to initialize LoRA adapters. Both of these methods also use row-orthonormal initialization and set their output factor to zero. The zero output factor preserves the original model function, while row-orthonormal initialization controls the subspace through which the initial OPD gradient is propagated.

The authors begin with a first-step gradient analysis to demonstrate that the effective change in weights under SGD is equivalent to projecting the OPD gradient onto the chosen adapter subspace. The proposed approaches were tested on two pairs of teacher-student models, each being evaluated on three mathematical reasoning benchmarks (AIME25, AMC23, and AIME24). The results indicate that the new approaches outperform standard LoRA-OPD on average for mean@16 and maj@16 performance. Additionally, the authors examine other aspects of behavior, such as update trajectories, spectral alignment, and weight-space changes, in order to further describe the effects of varying initialization strategies.

Critical strengths

A clear formulation of initialization-dependent gradient geometry in OPD.

The authors address a relevant question of how the initial subspace of a low-rank adapter influences the transmission of teacher supervision during on-policy distillation. A short mathematical analysis is provided by the authors which demonstrates that when the output factor is initialized to zero and the input factor is row-orthonormal, the first effective SGD update projects the OPD gradient onto the chosen adapter subspace.

Although the projection property is a well-known consequence of the parameterization and is not unique to this work, the way in which the authors utilize it to provide motivation for different types of spectral initialization within the framework of OPD creates a consistent view of the relationship between low-rank parameterization and optimization guided by a teacher.

Encouraging experimental results with complementary update-space analyses.

The authors evaluate the proposed methods using two teacher-student model pairs and three mathematical reasoning benchmarks. Both OP-OPD and OM-OPD achieve improved average mean@16 and maj@16 results compared to standard LoRA-OPD in the reported settings. These improvements were larger in the DeepSeek-based setting, in which OM-OPD showed an increase in average mean@16 from 0.2986 to 0.4625 and maj@16 from 0.3072 to 0.5806.

The further analyses of update magnitude, stable rank, and spectral alignment provide insight into how each of the various initialization techniques produces different patterns in the resulting weight updates. It appears that the initialization subspace is an important factor when designing low-rank OPD.

Critical weaknesses

A clear formulation of initialization-dependent gradient geometry in OPD.

The authors address a relevant question of how the initial subspace of a low-rank adapter influences the transmission of teacher supervision during on-policy distillation. A short mathematical analysis is provided by the authors which demonstrates that when the output factor is initialized to zero and the input factor is row-orthonormal, the first effective SGD update projects the OPD gradient onto the chosen adapter subspace.

Although the projection property is a well-known consequence of the parameterization and is not unique to this work, the way in which the authors utilize it to provide motivation for different types of spectral initialization within the framework of OPD creates a consistent view of the relationship between low-rank parameterization and optimization guided by a teacher.

Encouraging experimental results with complementary update-space analyses.

The authors evaluate the proposed methods using two teacher-student model pairs and three mathematical reasoning benchmarks. Both OP-OPD and OM-OPD achieve improved average mean@16 and maj@16 results compared to standard LoRA-OPD in the reported settings. These improvements were larger in the DeepSeek-based setting, in which OM-OPD showed an increase in average mean@16 from 0.2986 to 0.4625 and maj@16 from 0.3072 to 0.5806.

The further analyses of update magnitude, stable rank, and spectral alignment provide insight into how each of the various initialization techniques produces different patterns in the resulting weight updates. It appears that the initialization subspace is an important factor when designing low-rank OPD.

Questions for authors

Contribution of spectral direction selection (experimental validation).

Can the authors provide a baseline with random orthonormal initialization that has the same rank, scaling, initialization norm, training objective, and optimizer configuration as OP-OPD and OM-OPD? Also, can the authors compare the proposed methods against existing spectral initialization methods in the same OPD setting to show whether using the principal or minor pretrained directions provides any benefit over random orthonormal initialization?

Relationship between spectral initialization and the OPD gradient (technical and experimental clarification).

Could the authors directly quantify the amount of the initial OPD gradient retained by each of the principal, minor, and random orthonormal subspaces (e.g., using the normalized projection magnitude ∥G0P0∥F/∥G0∥F\|G_0P_0\|_F/\|G_0\|_F)? What relationship exists between these quantities and subsequent performance improvements?

Since the projection property also applies to other differentiable losses under SGD, could the authors further clarify what makes the proposed initialization particularly suitable for on-policy distillation?

Generalization and robustness of the reported improvements (experimental validation).

Would it be possible for the authors to show results from multiple independent training runs across different random seeds and LoRA ranks? Also, it would be beneficial to know whether the benefits observed in these experiments continue to exist after training for a longer period of time and when tested on other reasoning benchmarks. These types of tests will enable us to assess whether the benefits found here are consistent effects of spectral initialization or simply artifacts of the specific training parameters used.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.