acceptodds
Under review as a conference paper at ICLR 2027

PoLoRA: A Preconditioned Orthogonalized LoRA Optimizer

Abstract

Low-rank adaptation (LoRA) reduces the cost of finetuning large language models by training low-rank updates instead of the full weight matrices. Each update is the product of two trainable matrix factors. These factors are usually trained with Adam, which effectively treats them as a single flat vector of parameters and ignores the matrix and product structure of LoRA. Applying a matrix-aware optimizer such as Muon to each factor does not consistently improve over Adam, and neither do the recently proposed product-aware variants of Muon. We introduce PoLoRA (Preconditioned Orthogonalized LoRA), a product-aware spectral optimizer that adds curvature preconditioning and controls the size of each factor update. We evaluate PoLoRA on instruction-tuning datasets for code and math across models from 1B to 8B parameters, and find that it reaches the final held-out loss achieved by tuned Adam with a – speedup in training steps and at most 3% per-step overhead. Compared to Adam, PoLoRA is also less sensitive to the learning rate, and its optimal learning rate is stable across ranks.

Primary area:
optimization
Keywords:
optimization, LoRA

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_EjowOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors develop a Muon-like method for LoRA finetuning. Instead of the normal spectral-norm constraint in Muon, they adopt outergradient set to ensure smoothness in each per-sample loss. Combined with curvature estimate and update magnitude balance, the proposed method empirically performs better than the baseline.

Critical strengths
  1. The proposed method is interesting and backed by theory.
Critical weaknesses
  1. For the writing part, a large number of practical details are only slightly mentioned in the main paper in the Optimizer step paragraph in Sec 3 without sufficient discussion and ablation.
  2. For the experiments, the setup seems to be less common for LoRA, the models are small yet the data set sizes are larger, and only a single lora rank of 256 is tested, which is also larger than usual. Reporting loss as the main comparison target instead of accuracy on math and code also seem unusual.
Questions for authors

Some of the main component developed in Sec 2 seem to not only applicable to LoRA. Are there any limitations for their usage for the normal full finetuning case? If not, did the authors have any exploration to see if those construct is useful for the full finetuning case, or somehow it only improvess LoRA finetuning?

3 comments

Thank you for the review. We answer the three points in turn.

“A large number of practical details are only slightly mentioned in the main paper in the Optimizer step paragraph in Sec 3.”

The main text summarizes the step; the details are in the appendix:

  • the B = 0 initialization (Appendix E, Initialization);
  • relative damping δ (Appendix E, Relative damping);
  • spectral-norm estimation (Appendix E.1);
  • Gram Newton–Schulz for the matrix sign and inverse square root (Appendix E.2);
  • the estimation of P and Q (Appendix D).

The "Optimizer step" paragraph in Sec. 3 already points to the appendix for the preconditioners, spectral-norm estimation and Newton–Schulz. The revised paper adds pointers there to the initialization and damping.

“The setup seems to be less common for LoRA.”

Our settings are standard for LoRA finetuning:

  • “Only a single lora rank of 256 is tested.”

    Our submission tests r = 32, 64, 128 and 256 (Table 2, Figure 3), and PoLoRA's speedup over AdamW holds at every rank.

  • “Reporting loss as the main comparison target instead of accuracy on math and code also seem unusual.”

    See item 1 of the general response.

“Are there any limitations for their usage for the normal full finetuning case?”

Yes. The preconditioned Muon step of Section 2 applies to full finetuning or pretraining, which is outside this paper's scope and left to future work. PoLoRA's product-aware step is specific to LoRA.

The authors address most of my concern, I am raising my score to 3.

However, I do think the writing can still be improved. It appear to me the long list of details in the Optimizer step paragraph can be either mentioned in other more suitable places or elaborate more in detail. (Even moving some items entirely to the appendix might be better if they are less important. Otherwise, it just leaves chunks without much purpose and explanation in the main text)

Thank you for raising your score. We revised the “Optimizer step” paragraph of Section 3. Please see the revised draft.

Reviewer_dMtjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PoLoRA is a LoRA optimizer that combines product-aware orthogonalization, curvature preconditioning, and control of factor-update magnitudes. Its derivation starts from controlling linearized per-sample loss changes and leads to a preconditioned spectral constraint. On four models from 1B to 8B parameters and code/math instruction-tuning datasets, it reportedly reaches the final held-out loss of tuned Adam under a fixed training budget with a 1.2–1.7× speedup in steps and at most 3% per-step overhead. The paper also studies learning-rate sensitivity, ranks, component ablations, and distribution shift.

Critical strengths
  1. A structured and practically relevant optimizer design. PoLoRA accounts for the matrix-product structure of LoRA instead of treating its factors as unrelated parameter vectors. The local analysis provides a principled rationale for the update design, and the discussion connecting related product-aware optimizers is useful. The supplied core implementation is broadly consistent with the proposed algorithm.

  2. Consistent improvements over Adam with modest overhead. The main comparisons cover four models and two datasets, with improvements in all eight model–dataset settings. Learning-rate and rank analyses, component ablations, and the four-seed comparison in Figure 1 provide useful supporting evidence. The small per-step overhead makes the reported step savings potentially meaningful.

Critical weaknesses
  1. The connection between the local analysis and training acceleration is not sufficiently established. Controlling linearized per-sample loss changes motivates the spectral constraint, but does not by itself explain why fewer steps are needed to reach a target loss. I would like a clearer distinction between the properties supported by the theory and the acceleration observed empirically; a general convergence-rate theorem is not required. The component ablation also has an attribution issue: the variant without magnitude control changes the factor initialization (Appendix E, p. 22). Adjusting the base weights preserves the initial model output, but not the factor Jacobians or optimization geometry. Consequently, the claim that magnitude control accounts for about half of the speedup is not cleanly isolated by this comparison.

  2. The empirical validation is too limited to establish a robust advantage over related optimizers. Muon, iMuon, and LoRA-RITE are compared only on Llama-3.2-1B/math at rank 256, whereas the broader comparisons are mainly against Adam. Representative alternatives such as LoRA-Pro and Riemannian Preconditioned LoRA are not evaluated. The same held-out split is used for learning-rate selection and reported loss, and uncertainty estimates do not cover the broader evaluation despite the four-seed main comparison. Loss is a reasonable optimizer metric, but reaching fixed-budget Adam loss sooner does not establish faster attainment of downstream code or math quality. The released package also lacks complete configurations and reproduction support for the other models, optimizer baselines, and ablations; its ready-to-run configurations concentrate on Llama-1B, although the implementation can accept other models. These limitations leave the scope and robustness of the gains insufficiently supported. I therefore lean toward rejection while recognizing the method's promise.

Questions for authors
  1. What evidence most directly connects the proposed loss-change control to the observed step savings? Can you provide an ablation with matched factor initialization, or otherwise separate the effect of magnitude control from the initialization change described in Appendix E?

  2. Do the gains persist against closely related LoRA optimizers across additional model–dataset settings under comparable tuning budgets? Do the step savings translate into less training time to reach a fixed downstream code or math score?

  3. Can you report performance on a final evaluation split separate from learning-rate selection and provide uncertainty estimates for the broader comparisons? How sensitive are the conclusions to learning-rate schedules and clipping choices? I recognize that the constant schedule is shared and that the paper explains why LoRA-RITE disables global gradient clipping.

  4. Can you release the configurations and evaluation scripts for the remaining models, baselines, and ablations, including the initialization details needed to reproduce Figure 2?

3 comments

Thank you for the detailed questions. We answer them in order.

“What evidence most directly connects the proposed loss-change control to the observed step savings?”

We have to concede this one: we don't yet have a direct link between controlling the per-sample loss change and PoLoRA's speedup. But here we are in good company, since there is also no convergence proof that explains Muon's speedup either.

Instead the best explanation we have for Muon is robustness: it bounds the spectral norm of ΔW, and so bounds the change ‖ΔW x‖ in a linear layer's output for every input x. PoLoRA extends this robustness from the output of a linear layer to the loss on every input.

The one experimental link between this robustness and the speedup is that robust methods tolerate much larger learning rates, and thus can move further through parameter space. Both Muon and PoLoRA can use learning rates that are orders of magnitude larger than AdamW. On our Llama-3.2-1B finetuning task (Figure 3), PoLoRA trains at learning rates up to almost 1e-1, while AdamW clearly diverges near 1e-3.

“Can you provide an ablation with matched factor initialization?”

Yes we now have such an ablation. See item 2 of the general response.

“Do the gains persist against closely related LoRA optimizers across additional model–dataset settings?”

We have now added more LoRA optimizers (LoRA-RITE and Riemannian Preconditioned LoRA) and an additional task OLMo-2-1B code, see item 3 of the general response. Again we find that PoloRA achieved a better loss as compared to all related LoRA optimizers, with AdamW/Muon being roughly tied for second place.

“Do the step savings translate into less training time to reach a fixed downstream code or math score?”

Item 1 shows that PoLoRA's lower validation loss comes with higher accuracy on all four benchmarks, so we expect the step savings in loss to carry over to downstream scores.

“Can you report performance on a final evaluation split separate from learning-rate selection…”

Item 1 reports test loss on a fresh split never used to select learning rates. On it, the speedup over AdamW is the same as on the validation split.

“…and provide uncertainty estimates for the broader comparisons?”

The 4-seed AdamW spread (σ = 0.0017) is typically smaller than the relevant gaps between methods.

“How sensitive are the conclusions to learning-rate schedules…”

We reran Llama-3.2-1B math with linear decay to zero, re-tuning both learning rates. PoLoRA still reaches the final loss of AdamW in fewer steps (Appendix I.3 of the revised paper).

ScheduleAdamWPoLoRASpeedup
Constant0.38230.36501.63×
Linear decay to zero0.35310.3477≈1.2×

Interestingly the speedup is smaller than under a constant schedule, but is still non-trivial.

“…and clipping choices?”

All methods except LoRA-RITE clip the gradient norm at 1.0 (Table 3 of our submission). LoRA-RITE runs without clipping, since a global clip would break its invariance.

“Can you release the configurations and evaluation scripts for the remaining models, baselines, and ablations?”

The revised supplementary code includes the PoLoRA and AdamW configurations for every setting in our submission. We will release the evaluation scripts and the configurations for the baselines and ablations.

Thank you for the thorough response; the new baselines, downstream results, and held-out evaluation resolve most of my concerns, so I am raising my score to 3. Remaining issues: magnitude control contributes little in the matched ablation (1.48× vs. 1.43×), and the ~1.2× gain under linear decay needs multi-seed support. Most importantly, please report wall-clock time to reach AdamW's final loss, including per-step overhead.

Thank you for raising your score.

“Please report wall-clock time to reach AdamW's final loss, including per-step overhead.”

With step speedup SS and per-step overhead oo (Table 1), the wall-clock speedup is Swall=S1+oS_{wall} = \frac{S}{1+o}.

ModelSwallS_{wall}, codeSwallS_{wall}, math
OLMo-2-1B1.56×1.69×
Llama-3.2-1B1.50×1.58×
Qwen2.5-1.5B1.26×1.59×
Llama-3-8B1.17×1.55×

Values are on RTX PRO 6000. On H100 each differs by at most 0.01×.

“The ~1.2× gain under linear decay needs multi-seed support.”

Over three AdamW seeds under linear decay, the seed standard deviation is σ=0.0002\sigma = 0.0002, and the final loss of PoLoRA is 23σ23\sigma below the AdamW mean (Appendix I.3).

“Magnitude control contributes little in the matched ablation.”

We agree that for the initialization of Li et al. (2025) magnitude control contributes little. However we note that the B = 0 initialization is more common in practice and gives both PoLoRA and AdamW a lower loss (Appendix I.4).

Reviewer_2BNwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PoLoRA, an optimizer designed specifically for training LoRA. It builds the update direction from the product BABA of the two matrices, uses second-order gradient statistics as a preconditioner, and then applies a Muon-style orthogonalization step. It also controls the update magnitude of each matrix separately to avoid overly large steps. In code and math fine-tuning experiments on 1B–8B models, PoLoRA reaches Adam’s final validation loss with about 1.2–1.7× fewer training steps, adds no more than 3% overhead per step, and makes learning rates easier to reuse across ranks. However, the comparison is mainly in terms of validation loss, and the paper does not yet show clearly whether these gains carry over to better accuracy on downstream tasks.

Critical strengths

(1) The motivation is very clear for me. It respects LoRA’s product structure, and both the preconditioning and the update control are from limiting the per-sample loss change.

(2) The proposed method is evaluated on various datasets and models.

(3) Practical learning rate behavior. In the reported rank experiments, the best learning rate for PoLoRA stays stable across ranks, and performance is less sensitive to the learning rate than with Adam.

(4) Ablations help explain the gains. Removing the curvature preconditioning and then the update magnitude control steadily reduces the speedup, which shows that both components contribute to the final result.

Critical weaknesses

(1) Downstream gains are unclear. The experiments mainly focus on validation loss. Adding code generation and math reasoning benchmarks would help show whether faster loss reduction actually leads to better task performance.

(2) Limited comparison with other LoRA based Optimizers. The main cross-model results in this paper mainly compare with Adam. However, Muon, iMuon, and RPLoRA are only compared on two experiment settings (Llama-3.2-1B and OLMo-2-1B) on coding tasks.

(3) The contribution of the update magnitude is not very clear. In the main ablation study results (Figure 2), the variant without te control also uses a different initialization method, which make the contribution unclear.

(4) The theoretical motivation relies on several approximations. These include linearizing the loss change, dropping the ΔBΔA term in the product update, and using a diagonal curvature estimate. The paper does not quantify how much these approximations affect its loss-control explanation in actual training.

(5) The low overhead is only shown for specific hardware and batch sizes. It would help to test smaller batches and other GPUs, since the extra matrix operations may take up a larger share of training time in those settings.

Questions for authors

N/A

3 comments

Thank you for the careful summary. Weaknesses (1)–(4) correspond to items 1, 3, 2 and 4 of the general response, and we answer (5) here.

  • (1) “Downstream gains are unclear.”

    See item 1 of the general response.

  • (2) “Limited comparison with other LoRA optimizers.”

    See item 3 of the general response.

  • (3) “The ablation on update magnitude control also changes the initialization.”

    See item 2 of the general response.

  • (4) “The theoretical motivation relies on several approximations.”

    See item 4 of the general response.

  • (5) “The low overhead is only shown for specific hardware and batch sizes.”

    We have now run our experiments on different hardware. We found that, PoLoRA's extra time per step as a share of AdamW's step time, at our batch of 16 sequences (32k tokens) was stable on RTX PRO 6000 and H100 GPUs (Table 1 of the revised paper), see the table below:

    ModelRTX PRO 6000H100
    Llama-3.2-1B3.2%3.6%
    OLMo-2-1B3.0%3.6%
    Qwen2.5-1.5B3.3%4.0%
    Llama-3-8B2.5%2.8%
    • Our 32k tokens per batch is within the range of common LoRA setups: QLoRA (Dettmers et al., 2023, Table 9) uses up to 49k.

Thank you for the detailed response and the additional experiments. I still have one concern: Most experiments are based on held-out loss, but a lower loss does not usually mean better downstream performance. Appendix I.2 reports downstream accuracy only for AdamW and PoLoRA. Could the authors also report it for more baselines (Muon, RPLoRA) ?

Thank you for engaging with our response and for the additional experiments you suggested. We scored Muon and RPLoRA with the harness of Appendix I.2.

Accuracy (%) of Llama-3.2-1B fine-tuned on math at r = 256 (step 9000):

OptimizerGSM8KMATH-500
AdamW43.118.6
Muon43.617.2
RPLoRA45.517.6
PoLoRA45.323.0

If these results address your remaining concern, we would be grateful if you would consider raising your score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers. We address the common concerns in the general response, and we have uploaded a revised paper with every change highlighted in red.

1. Downstream accuracy (2BNw, dMtj, Ejow). As the reviewers requested, we now report downstream accuracy on math and code benchmarks. We trained AdamW and PoLoRA on Llama-3.2-1B (r = 256) on math and on code. We scored the trained adapters zero-shot with greedy decoding. Test loss is measured on a fresh split, disjoint from the validation split that our submission uses both to select learning rates and to report loss. See Appendix I.2 of the revised paper.

Math (OpenMathInstruct-2)

MethodValidation loss ↓Test loss ↓GSM8KMATH-500
AdamW0.38220.394743.118.6
PoLoRA0.36440.376345.323.0

Code (OpenCoder)

MethodValidation loss ↓Test loss ↓HumanEvalMBPP
AdamW0.69490.704928.731.0
PoLoRA0.68490.695032.934.6

We find that the gains in validation loss reported in our submission are tightly correlated with gains in downstream accuracy.

2. Ablation with matched initialization (2BNw, dMtj). The reviewers rightly point out that our component ablation used a different initialization with and without magnitude control. Only the variant without magnitude control used the initialization of Li et al. (2025), because its step diverges at B₀ = 0, while the other runs used B₀ = 0. To address this, we reran the ablation with the Li et al. initialization for every run (Appendix E, Initialization), which initializes B₀ ~ N(0, σ²) instead of B₀ = 0. See Section 4.3 and Appendix I.4 of the revised paper.

Final lossSpeedup over AdamW (same init)
PoLoRA0.37001.48×
without magnitude control0.37181.43×
without curvature (P = Q = I)0.37601.24×
Product Muon (both removed)0.38141.07×

With the same initialization for every variant, including AdamW, both curvature and magnitude control still improve the speedup, though their gains overlap (Section 4.3 and Appendix I.4 of the revised paper). This initialization slightly worsens every method, so the paper keeps the standard B = 0.

3. Baselines beyond one setting (2BNw, dMtj). We have now added Riemannian Preconditioned LoRA on both settings and LoRA-RITE on OLMo-2-1B code, and report the final validation loss at r = 256 below (new runs marked *). Every baseline uses the same learning-rate sweep protocol as PoLoRA and AdamW, where we report the best rate inside its grid. See Figure 1 and Appendix I.1 of the revised paper.

MethodLlama-3.2-1B mathOLMo-2-1B code
AdamW0.38230.7524
Muon0.38310.7500
iMuon0.41350.7521
LoRA-RITE0.38690.7696*
Riemannian Preconditioned LoRA0.3866*0.7605*
PoLoRA0.36500.7356

We did not run LoRA-Pro since its AdamW version keeps AdamW moments for the full d_out × d_in gradient, that is, the optimizer memory of full finetuning. Its factor gradients use the preconditioners (BᵀB)⁻¹ and (AAᵀ)⁻¹, so the factor preconditioning of LoRA-Pro is the same as Riemannian Preconditioned LoRA's, which we compare against.

4. Approximations in the analysis (2BNw, dMtj). Dropping the second-order term ΔBΔA and linearizing the loss are what make the step tractable enough to be analytically defined. As a sanity check, we have now monitored the effect of dropping ΔBΔA on our math and code runs. That is, every 80 steps, we checked that

  • The second-order term is negligible: ‖ΔBΔA‖₂ is on average 0.1–0.2% of ‖BΔA + ΔBA‖₂, and at most 1.2%.
  • The linearized loss change ⟨∇ℓᵢ(W), ΔW⟩ matches the actual change ℓᵢ(W + ΔW) − ℓᵢ(W) across held-out examples i (Appendix I.5 of the revised paper).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.