- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates the geometric properties of parameter updates in LLMs under RLVR, SFT, and OPD by analyzing various metrics of parameter changes before and after training, as well as conducting experiments that constrain the parameter update space during subsequent training. The experiments show that the geometric properties of OPD updates generally lie between those of SFT and RLVR, and that OPD exhibits more stable parameter update directions.
Critical strengths
- The methodology is relatively rigorous, with few exaggerated claims or overly idealized assumptions.
- The experimental findings provide useful insights into the differences in update geometry among RLVR, SFT, and OPD.
- The experiments are relatively comprehensive, and most of the paper's conclusions are supported by experimental evidence.
Critical weaknesses
- The experimental settings are not sufficiently comprehensive. For example, in Section 4.2, the experiments validating the narrow update subspace of OPD are entirely based on Qwen3 models. However, models with different parameter counts within the same family may still exhibit similar logit distributions. Such similarity between the student and teacher could be one of the reasons behind the narrow update subspace observed in OPD. Therefore, experiments conducted solely on Qwen3 models are insufficient to attribute this phenomenon to OPD itself.
- The experimental comparisons are not sufficiently fair. For example, in Section 3, SFT is compared against the base model, whereas RLVR and OPD are initialized from SFT models. These differences in initialization may lead to different geometric behaviors. Furthermore, their batch sizes, training steps, and other hyperparameters also differ.
- The defensive writing is too prominent, and the language often feels unnatural. First, dashes are used excessively, which is not a conventional writing style. Second, many sections contain defensive statements, such as "so it is not a clean isolation of credit variation" and "and denser sweeps and independent seeds remain necessary". While such statements can help prevent readers from misunderstanding the results, they also make the conclusions less convincing. Why not address these limitations through better experimental design in the first place?
Questions for authors
- Which experiments and conclusions do you consider relatively more rigorous and more likely to generalize, and why?
- What advantages do you think the stability of parameter update directions in OPD could bring?