- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors develop a Muon-like method for LoRA finetuning. Instead of the normal spectral-norm constraint in Muon, they adopt outergradient set to ensure smoothness in each per-sample loss. Combined with curvature estimate and update magnitude balance, the proposed method empirically performs better than the baseline.
Critical strengths
- The proposed method is interesting and backed by theory.
Critical weaknesses
- For the writing part, a large number of practical details are only slightly mentioned in the main paper in the Optimizer step paragraph in Sec 3 without sufficient discussion and ablation.
- For the experiments, the setup seems to be less common for LoRA, the models are small yet the data set sizes are larger, and only a single lora rank of 256 is tested, which is also larger than usual. Reporting loss as the main comparison target instead of accuracy on math and code also seem unusual.
Questions for authors
Some of the main component developed in Sec 2 seem to not only applicable to LoRA. Are there any limitations for their usage for the normal full finetuning case? If not, did the authors have any exploration to see if those construct is useful for the full finetuning case, or somehow it only improvess LoRA finetuning?
3 comments
The authors address most of my concern, I am raising my score to 3.
However, I do think the writing can still be improved. It appear to me the long list of details in the Optimizer step paragraph can be either mentioned in other more suitable places or elaborate more in detail. (Even moving some items entirely to the appendix might be better if they are less important. Otherwise, it just leaves chunks without much purpose and explanation in the main text)
Thank you for raising your score. We revised the “Optimizer step” paragraph of Section 3. Please see the revised draft.
Thank you for the review. We answer the three points in turn.
“A large number of practical details are only slightly mentioned in the main paper in the Optimizer step paragraph in Sec 3.”
The main text summarizes the step; the details are in the appendix:
The "Optimizer step" paragraph in Sec. 3 already points to the appendix for the preconditioners, spectral-norm estimation and Newton–Schulz. The revised paper adds pointers there to the initialization and damping.
“The setup seems to be less common for LoRA.”
Our settings are standard for LoRA finetuning:
“Only a single lora rank of 256 is tested.”
Our submission tests r = 32, 64, 128 and 256 (Table 2, Figure 3), and PoLoRA's speedup over AdamW holds at every rank.
“Reporting loss as the main comparison target instead of accuracy on math and code also seem unusual.”
See item 1 of the general response.
“Are there any limitations for their usage for the normal full finetuning case?”
Yes. The preconditioned Muon step of Section 2 applies to full finetuning or pretraining, which is outside this paper's scope and left to future work. PoLoRA's product-aware step is specific to LoRA.