- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MPO-PRO temporarily factorizes training updates into matrix product operators, then contracts them into dense weights or fixed-rank LoRA adapters. The fixed-rank branch approximately matches a LoRA-Pro update through a damped solver. Experiments on Qwen3.5-0.8B and Llama-2-7B report math/code improvements and decompose updates into a reference direction, first-order deviation, and higher-order remainder.
Critical strengths
MPO-PRO temporarily factorizes training updates into matrix product operators, then contracts them into dense weights or fixed-rank LoRA adapters. The fixed-rank branch approximately matches a LoRA-Pro update through a damped solver. Experiments on Qwen3.5-0.8B and Llama-2-7B report math/code improvements and decompose updates into a reference direction, first-order deviation, and higher-order remainder.
Critical weaknesses
-
The MPO-specific contribution and mechanism remain unclear. Temporary overparameterization is already central to OPF and LoRA-Over. The decomposition associates gains with , but does not establish why MPO structure provides a distinct advantage over generic reparameterization or implicit preconditioning.
-
The practical gains are modest relative to training cost. Improvements are task-dependent and require approximately 1.9× training time, without corresponding dialogue gains. This leaves the benefit over simpler fine-tuning alternatives uncertain under equal resource budgets.
Questions for authors
- What benefit is specific to MPO structure relative to OPF, LoRA-Over, or another temporary factorization?
- Why should the direction of improve generalization, and how does this explanation differ from implicit preconditioning?
- How do the measured magnitude and direction of vary with damping, and how do these measurements reconcile the zero-damping result with the decomposition study?
- Do the gains over LoRA-Pro and LoRA-Over persist at matched total training time and tuning budget?