acceptodds
Under review as a conference paper at ICLR 2027

MPO-PRO: TRAJECTORY SHAPING THROUGH TEMPO- RARY MATRIX FACTORIZATION

Abstract

Post-training adapts pretrained language models to tasks such as mathematical reasoning and code generation, but local deployment often limits model size and inference cost. One way to improve post-training is to use overparameterization during training, allowing a richer parameterization than the final deployed model. We propose MPO-PRO, an overparameterization-based approach that enriches post-training by using matrix product operators (MPOs) as temporary training coordinates for dense updates and LoRA factors, and contracts them before deployment. This keeps the original model architecture or adapter rank unchanged while providing a richer training parameterization. For the fixed-rank branch, we use the first-order LoRA-Pro update as a reference and approximately match it in the temporary MPO representation.On GSM8K and HumanEval, the dense and fixed-rank branches outperform full fine-tuning and LoRA-Pro, respectively. When trained on SciInstruct, MPO-PRO also outperforms LoRA-Pro on SciBench Physics. We further analyze the update mechanism through decomposition and find that the performance gain is mainly associated with the first-order implementation deviation introduced by the temporary representation relative to LoRA-Pro.

Primary area:
optimization
Keywords:
post-training, deployment-equivalent training, train-time parameterization, low-rank adaptation, matrix product operator, temporary overparameterization, optimization geometry, large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_eNTyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MPO-PRO temporarily factorizes training updates into matrix product operators, then contracts them into dense weights or fixed-rank LoRA adapters. The fixed-rank branch approximately matches a LoRA-Pro update through a damped solver. Experiments on Qwen3.5-0.8B and Llama-2-7B report math/code improvements and decompose updates into a reference direction, first-order deviation, and higher-order remainder.

Critical strengths

MPO-PRO temporarily factorizes training updates into matrix product operators, then contracts them into dense weights or fixed-rank LoRA adapters. The fixed-rank branch approximately matches a LoRA-Pro update through a damped solver. Experiments on Qwen3.5-0.8B and Llama-2-7B report math/code improvements and decompose updates into a reference direction, first-order deviation, and higher-order remainder.

Critical weaknesses
  1. The MPO-specific contribution and mechanism remain unclear. Temporary overparameterization is already central to OPF and LoRA-Over. The decomposition associates gains with EE, but does not establish why MPO structure provides a distinct advantage over generic reparameterization or implicit preconditioning.

  2. The practical gains are modest relative to training cost. Improvements are task-dependent and require approximately 1.9× training time, without corresponding dialogue gains. This leaves the benefit over simpler fine-tuning alternatives uncertain under equal resource budgets.

Questions for authors
  1. What benefit is specific to MPO structure relative to OPF, LoRA-Over, or another temporary factorization?
  2. Why should the direction of EE improve generalization, and how does this explanation differ from implicit preconditioning?
  3. How do the measured magnitude and direction of EE vary with damping, and how do these measurements reconcile the zero-damping result with the decomposition study?
  4. Do the gains over LoRA-Pro and LoRA-Over persist at matched total training time and tuning budget?
Reviewer_UhEhOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

MPO-PRO uses temporary MPO factors to train dense updates or LoRA factors, then contracts them before deployment. For LoRA, a regularized Gram-CG solve approximately matches the LoRA-Pro update. The paper reports math and code gains at unchanged deployment size, and uses update-decomposition experiments to investigate where those gains come from.

Critical strengths
  • Keeping the deployed architecture and adapter rank unchanged makes the approach practically appealing.
  • The matched LoRA-Pro comparisons, three-run results, and update-decomposition experiments provide more useful evidence than benchmark scores alone. Reporting the weaker MT-Bench results is also helpful.
  • The decomposition ΔA=P+E+H\Delta A=P+E+H and corresponding interventions provide useful diagnostics beyond aggregate benchmark scores.
Critical weaknesses
  • The dense comparison changes more than the factorization. Eq. (3) adds a trainable gate and initializes the additive update from the pretrained weight. Scrambled MPO keeps both choices, so it does not tell us whether a much simpler gated dense update would work equally well.
  • Table 4 is interesting, but I would hesitate to treat it as an explanation of the gain. Appendix B does not explain how the edited A-space updates are converted into the next-step cores. That procedure and BF16 rounding could affect the comparison between retaining and removing EE.
  • The fixed-rank gains are fairly small for roughly 1.51.5–2.1×2.1\times the training time. Rank-16 HumanEval pass@1 improves by only 0.329 points, while MT-Bench falls and matched GLUE barely changes. A longer or better-tuned LoRA-Pro run seems an important missing comparison.
Questions for authors
  • For Eq. (3), what happens with the unfactorized control W=W0+γMW=W_0+\gamma M, initialized with M0=W0M_0=W_0 and γ0=10−8\gamma_0=10^{-8}, training both variables with Adam?
  • In Table 4, how are the next-step cores obtained after applying P+EP+E or P+HP+H? Does the same ordering hold with FP32 master weights and otherwise identical interventions?
  • Does MPO-PRO still outperform LoRA-Pro when the latter receives the same GPU-hour budget, with its learning rate and training duration retuned?
Reviewer_UtHqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Within this work, authors proposed a methodology, re-express a weight update (or a LoRA factor) as a chain of matrix product operator (MPO) cores, during fine-tuning. The approach trains those cores, then contracts them back into an ordinary matrix before deployment. The deployed model keeps exactly the same architecture or LoRA rank. The authors argue that what changes is the optimization trajectory, not the set of reachable solutions.

Critical strengths

Per this work, I do consider the analysis regarding decomposition helped and provided certain insight for other researchers, which, as described within the manuscript, decomposes each contracted step into three parts: P, the LoRA-Pro target. E, the first-order implementation deviation. H, the higher-order remainder.

Within this paper, the proposed method exports exactly the same artifact as its baselines or prior arts. For the dense branch, this means an unchanged Qwen3.5-0.8B architecture. For the fixed-rank branch, it means the same 19.99M-parameter rank-8 adapter as LoRA-Pro, as well as other LoRA formula, even though, I think most of the temporal overparameterization work shall have similar benefit.

I do think this research practice could provide other researchers some extra options/inspiration to further explore this domain, as well as down-stream application domains.

Critical weaknesses

The major concern regarding this work comes from the gains, which is not that much significant from certain perspective. For instance, Rank-16 HumanEval pass@1 improves by +0.33 and Rank-8 GSM8K improves by +0.81. The decomposition result separates branches by about 1.

Furthermore, MPO-PRO looks like use more training resource compared with LoRA-Pro, where I didn't find the comparison against LoRA-Pro on equal compute, like steps or ranks, as well.

Questions for authors

Since temporary overparameterization will generally improve the post-training compared with other parameter efficiency methods is also part of the claim, I do think it's worth to compare with certain some of other advanced LoRA settings, like Riemannian-preconditioned LoRA, LoRA-RITE, AdaLoRA, HiRA and MoSLoRA. Comparing with all different kind of settings might be challenging within this manuscript. But adding some settings does help the community to follow-up, compare and re-produce.

I also think it's worth to clarify the impact of scrambled MPO on MPO structure itself, which felt a little bit confusing, at least to me.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.