- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies gradient fields induced by different KL divergences; FKL is sign aligned but RKL is provably misaligned, which widens teacher-student distribution gap. They further show that this pitfall cannot be overcome by just linearly interpolating FKL and RKL and propose m-RKL which corrects the gradient fields.
Critical strengths
- Going beyond mode-averaging vs mode-seeking, this paper studies gradient-based optimization, and fixed RKL objectives, which is the de facto standard in practical OPD methods. m-RKL is plug-and-play and a clear contribution.
- The custom autograd operator code is provided and the experimental results validates m-RKL on math (ID) and IF/Code (OOD) domains with probability error decreasing during training.
Critical weaknesses
- I find no major weaknesses in this paper, but I would like the authors to clarify the following points.
Questions for authors
- Could the authors provide the statistics of the logit gradients when the signs of the teacher and student logits are opposite? (Fig.1) I’m interested in the norm of the gradients in the sign-reversal case.
- Is there any computational overhead in m-RKL compared to RKL? If so, could the authors discuss FLOPs/complexity or report wall-clock training times?
3 comments
Thank you very much for additional experiments on logit analysis and clarifications on computation cost. I am leaning slightly toward accept. But, I am not an expert in this area and I will maintain my positive rating (3).
We sincerely thank the reviewer for maintaining the positive assessment. We greatly appreciate the reviewer's time and constructive comments, which have helped improved our paper.
We sincerely appreciate your recognition and positive assessment of our work. We have added quantitative analyses for both questions. In particular, we separate statistics at the generation-step level from those at the token-coordinate level, since a generation step occurring sign reversal can have only a subset of sign-reversal coordinates in its vocabulary distribution.
Answer to Q1: Following the definition in Theorem 2, we measure sign reversal on over-allocated tokens for which qk>pk but the RKL logit gradient is negative. We report statistics at training checkpoints 5, 10, 15, 20, and 25, using the sampled on-policy trajectories.
Generation-step-level statistics. Table 4 compares normal generation steps with steps containing at least one sign-reversal coordinate. At sign-reversal steps, the average full-vocabulary gradient norm ranges from 0.422 to 0.518, compared with only 0.013∼0.015 at normal steps — approximately 28∼37 times larger. The average loss is also substantially higher at sign-reversal steps (0.845∼1.084, versus 0.011∼0.017 at normal steps).
The teacher distribution is considerably more uncertain at these steps: average teacher entropy ranges from 0.785 to 0.904 at sign-reversal steps, compared with 0.041∼0.071 at normal steps. Thus, sign reversal is empirically associated with high-uncertainty generation contexts and larger gradient responses, rather than occurring predominantly in nearly deterministic contexts.
Table 4: Generation-step level statistics (normal positions vs. sign-reversal positions)
Table 5: Token-coordinate-level statistics
Token-coordinate-level statistics. Table 5 further examines the vocabulary coordinates that actually violate SLSM within the sign-reversal steps. The gradient norm over these coordinates ranges from 0.048 to 0.065, while the corresponding full-vocabulary gradient norm ranges from 0.422 to 0.518. The reported sign-reversal-coordinate norm is therefore approximately 11∼13 of the full-vocabulary norm and remains larger than the average full-vocabulary norm at normal generation steps.
Importantly, the sign-reversal coordinates collectively carry 26.7∼27.4 of the student probability mass, and this proportion remains stable across the five checkpoints. Although the gradient norms decrease moderately from Step 5 to Step 25, the probability mass assigned to the affected coordinates does not vanish. These results indicate that sign reversal is not confined to negligible-probability tail tokens or coordinates with vanishingly small gradients.
Answer to Q2: m-RKL adds only two per-position vocabulary reductions to compute S+ and S−, requiring O(V) additional work. It requires no additional forward or backward pass and no teacher-side modification. In our implementation, total training time changes from 1973 s for RKL to 1997 s for m-RKL, corresponding to approximately 1.22 overhead. Throughput changes from 7.15 to 7.08 samples/s, a decrease of approximately 0.98, while both methods report 2.175×1013 total FLOPs. We have included these measurements in the appendix to make the practical cost explicit.
We thank the reviewer again for these suggestions, which helped us provide a more complete empirical characterization of both the sign-reversal phenomenon and the computational cost of m-RKL.