- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates three initialization impacts of low-rank adaptation on on-policy distillation. The key contributions are analyses of training trajectory, accuracy on math benchmarks, subspace overlap, entropy, stable rank, gradient magnitude, and Hill tails. Several math reasoning tasks are provided to show minor singular vectors can outperform other initialization.
Critical strengths
The paper has relatively rich analyses for low-rank on-policy distillation in training trajectory, accuracy on math benchmarks, subspace overlap, entropy, stable rank, gradient magnitude, and Hill tails. It shows minor singular vectors relatively outperforms principal singular vectors or random initialization on several different math reasoning benchmarks. The introduced method is clear, practical, and easy to understand.
Critical weaknesses
The critical weakness of the paper is its lack in novelty. As the paper noted, there are many prior literature on LoRA initialization. The use of principal and minor singular vectors are known in such papers already. Simply applying them for on-policy distillation does not provide substantially new findings. Even considering several follow-up analyses, new discovery is very limited, and they are not convincingly useful. Especially, the paper does not provide any useful justification, consequences or hypotheses from the numerical analyses. There are some restrictions in the paper's scope: the benchmark is only for math reasoning tasks; only a specific rank of 16 is considered; nothing new methodologies are provided.
Questions for authors
What happens on different ranks? What happens on different tasks other than math reasoning? What happens if we exhaustively search for the best subspace initialization? Minor singular vectors are best? Why not mixing Principal and minor in some ratios? How useful are the provided analyses? How are they related to downstream task performance, and any justification?