- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how optimizer choice affects the effective rank of LoRA updates at a fixed adapter rank. ISO-LoRA reconstructs the joint tangent-projected weight gradient from factor gradients, orthogonalizes this signal, projects it back into the feasible tangent space, and reconstructs factor updates through small-core computations. First-step theory and conditional finite-horizon analyses characterize spectral differences between optimizers. Experiments across 0.1B–7B language models report improved rank utilization and downstream performance, particularly at moderate-to-large adapter ranks.
Critical strengths
-
Coherent geometric construction with appropriately scoped theory. The joint tangent formulation accounts for interactions between factor updates, and the small-core implementation avoids dense optimizer state. Appendix B explicitly distinguishes the ideal construction from damping and factor-space momentum. The theoretical claims are limited to the stated first-step and conditional gradient models; they do not assert a general training-trajectory ordering.
-
Substantial empirical support and relevant resource controls. The experiments include matched-rank geometric baselines, hyperparameter sweeps, three-seed downstream results, learned-adapter spectra, and fixed-time comparisons. At rank 128, LLaMA-2-7B achieves GSM8K accuracy of , compared with for LoRA-Muon. The separate fixed-time study also favors ISO-LoRA, although those endpoints are single-seed (Table 1; Appendix G.6).
Critical weaknesses
- The central update diagnostic is not validated against actual finite factor movement. With applied increments and , including weight decay, the exact change is:
where is the tangent update and is the finite cross-term. The reported update-rank diagnostics concern . Learned-adapter spectra do not establish that its spectrum describes at the learning rates used. Factor reconstruction involves inverse Gram matrices, making this distinction particularly relevant when factors are poorly conditioned; damping may control it, but the reported measurements do not resolve that question. This leaves the link between tangent rank utilization and the network's actual steps insufficiently established (Sections 3–4; Appendices B and G.7).
- The performance contribution of spectral spreading remains incompletely isolated. Appendix G.4 appropriately describes its 45-run correlation analysis as descriptive. However, these comparisons vary optimizer and nominal rank, jointly changing step scale, preconditioning, and spectral shape. The comparison with LoRA-Muon strengthens the evidence for the overall method, but does not isolate useful spectral spreading from these other changes. Since rank utilization is the paper's central explanatory contribution, a controlled comparison within the same tangent-reconstruction pipeline is needed to establish what the added spectral operation contributes beyond the combined optimizer design.
Questions for authors
-
Do the exact updates preserve the reported effective-rank ordering? Please measure , exact-update spectra, and gradient alignment at representative early and late checkpoints, including poorly conditioned layers. Evidence that the finite updates retain the claimed ordering under the reported settings would resolve the first concern.
-
At a fixed model and adapter rank, does polar equalization improve downstream performance when the tangent signal, reconstruction, momentum, and step-norm convention are held fixed? A focused comparison with the unflattened signal, with variability across runs, would materially strengthen the claimed mechanism. Together with the finite-update check, a persistent benefit would support raising the rating to weak acceptance.
3 comments
Response to Reviewer dQJS (2/2)
2. "The performance contribution of spectral spreading remains incompletely isolated..." (W2 and Q2)
We agree that the 45-run correlation in Appendix G.4 cannot isolate the spectral operation, since it varies the optimizer and the rank at the same time. Following the reviewer's suggestion, we built a controlled comparison inside the Iso-LoRA pipeline.
All variants share the tangent-signal construction, the damped reconstruction, momentum 0.9, damping 1e-8 and weight decay 0.01; the only difference is the spectral step. A0 is Iso-LoRA, which applies polar to the joint tangent signal; A1 uses the unflattened tangent signal. To hold the step-norm convention fixed, at every step and in every layer we rescale A1's tangent update by a scalar. This makes its Frobenius norm match the joint reference computed from the same state (relative tolerance 1e-3). The first-order step size is therefore identical, and only its spectral shape differs.
We use , and FP32 factors. We select each variant's LR on seed 42 by validation loss and report mean ± sample SD over seeds 43/44/45.
| Comparison | A0 (polar) val. loss | A1 (no polar) val. loss | Paired A1−A0 loss | Paired A1−A0 BLEU |
|---|---|---|---|---|
| Each at its best LR (A0: 2e-3, A1: 1.25e-4) | 1.1516±0.0015 | 1.2910±0.0012 | +0.1394±0.0008 | −1.03±0.20 |
| Shared LR 1.25e-4 | 1.2645±0.0003 | 1.2910±0.0012 | +0.0265±0.0013 | −0.19±0.27 |
| Shared LR 2.5e-4, FP64 factors | 1.2073±0.0004 | 1.3919±0.0580 | +0.1846±0.0577 | −3.27±1.16 |
The answer to Q2 is yes. When each variant uses its best learning rate, removing the polar step increases validation loss by 0.139 and decreases BLEU by 1.03 points, consistently across all three seeds. A0 is also better on NIST, METEOR, ROUGE-L and CIDEr. The benefit persists when both variants use the same learning rate (+0.026 in loss), and it is larger in a separate FP64-factor control (+0.185 in loss, −3.27 BLEU). In the seed-42 sweeps, A0 has a lower validation loss than A1 at every learning rate where both were run.
The comparison also shows why the polar step matters in this pipeline. Without it, the exact update collapses to a stable rank of about 2, versus about 150 for A0. At the same Frobenius norm, the step is therefore concentrated in a few directions. As a result, A1's factors become extremely ill-conditioned even at the shared learning rate: the maximum Gram condition number is for A1, versus 39 for A0. At learning rates of 2.5e-4 and above in FP32, A1 can no longer satisfy the norm-matching check, so its best stable learning rate is 16× smaller than A0's. In this sense, spreading the step across directions is what allows a larger and still stable step at the same norm.
We also compared joint polar with polar applied separately to the two tangent components (A2), which is the factor-wise analogue. Joint polar is better on every seed at the two learning rates where performance is competitive, but slightly worse at the smallest learning rate; details are in our response to Reviewer uF3g. We could not complete the same ablation at 7B, because A1 failed the norm-matching check after 113 and 121 steps on two seeds. We therefore do not claim a 7B mechanism result.
We hope that the exact-update measurements and the controlled polar comparison address both of the reviewer's concerns, and we would be happy to provide any further details.
Thank you for running both comparisons. I appreciated seeing the early steps where LoRA-Muon has the higher stable rank, and from step 50 onward the sampled GPT-2 checkpoints show that Iso-LoRA's advantage carries over to the actual weight updates. The polar ablation on GPT-2 is also informative, though the downstream BLEU gain is modest at a shared learning rate and the 7B check could not be completed. While this adds support for the mechanism on GPT-2, the evidence explaining the headline downstream gains still feels somewhat limited.
I also have a question about the 59.12% GSM8K exact match reported for LoRA-Muon in Table 15 (and Table 1(a)). If each of the three seeds evaluated all 1,319 test questions, the nearest possible means round to 59.11% and 59.14%. Could you give the number correct and the number evaluated for each seed, and clarify how you computed the mean? In addition, I could not locate the new diagnostic and ablation tables in the PDF available to me; if they appear in a revised version, a page reference would help. I have taken the results in your response into account and am keeping my rating at 2.
Response to Reviewer dQJS (1/2)
We thank the reviewer for the careful reading and the two very concrete questions. Since Q1 and Q2 are exactly the tests requested in W1 and W2, we answer each weakness together with its question.
1. "The central update diagnostic is not validated against actual finite factor movement..." (W1 and Q1)
We agree that the tangent update Tt alone does not show what the network actually receives. To answer this directly, we re-ran the reported GPT-2/E2E setting: r=128, s=4, seed 42, and the paper's learning rates (2e-3 / 1e-3 / 3e-3 / 1e-2 for Iso-LoRA / AdamW / Muon / LoRA-Muon). During these runs we recorded the factors immediately before and after every optimizer step, so weight decay is included. From these we compute the exact increment ΔWt=Tt+Et, with Tt=s(ΔBtAt+BtΔAt) and Et=sΔBtΔAt. For each step we report its stable rank, ∥Et∥F/∥Tt∥F, the regularized Gram condition number of the factors, and cos(−Gt,ΔWt), averaged over all 48 adapted matrices. Step 1 has zero learning rate because of warmup.
Stable rank of the exact update ΔWt (for Iso-LoRA we also show the tangent part Tt):
The answer to Q1 is yes. Against Muon and AdamW, Iso-LoRA's exact update has a higher stable rank at every sampled step, from the first steps to the end of training. Against LoRA-Muon, the ordering holds at every sampled step from step 50 until the end of training; steps 370, 1110 and 1480 (not shown) behave like steps 740 and 1406. Only in the first few steps (steps 5 and 10) is LoRA-Muon's rank higher, and this is exactly where the finite cross-term is largest.
Et/Tt is 0.30 on average (0.36 at most) at step 5, but it quickly becomes negligible: 0.035 by step 100 and below 0.01 from step 740 onward. From then on, the exact rank and the tangent rank coincide (188.6 vs. 188.9 at step 740). Thus, Tt faithfully describes the actual step for essentially all of training, and the rank ordering it predicts carries over to the exact update. We will include the full per-step table in the appendix.
On poorly conditioned layers. We agree that, because the reconstruction uses inverse Gram matrices, poorly conditioned factors are the critical case. In our measurements, Iso-LoRA keeps its own factors well conditioned. Over all layers and sampled steps, the largest regularized Gram condition number is 118 for Iso-LoRA, compared with 188 for LoRA-Muon, 245 for Muon and 1.1×106 for AdamW.
The worst-conditioned layers also behave normally (separate run, seed 43): layers in the top condition decile have a mean Et/Tt of 0.050, essentially the same as the 0.048 over all layers. For example, the worst-conditioned layer at step 100 (h9.mlp.c_proj, κ=53.6) still has exact-update stable rank 164.7 and Et/Tt=0.033. Consistent with this, changing the damping from 1e-8 to 1e-10 or 1e-6 changes the final validation loss by less than 0.001 over three seeds (see our response to Reviewer uF3g).
On gradient alignment. We tracked cos(−Gt,ΔWt) on the same three fixed layers for all four methods; it is positive at every recorded nonzero step.
Spreading the update over more directions therefore does not cost descent alignment: Iso-LoRA is the most aligned method at step 100 and close to Muon late in training.
At 7B. The LLaMA-2-7B diagnostic runs (seed 0, 224 adapted matrices) show the same picture. At step 5, Iso-LoRA's exact-update rank is slightly below LoRA-Muon's (122.0 vs. 123.9). By the middle of training (step 1563), it is 203.2 vs. 127.8, with Et/Tt of 0.004 and 0.002, respectively.
Taken together, the exact steps keep the claimed rank ordering, the finite cross-term becomes negligible early in training, and the reconstruction does not operate in an ill-conditioned regime. We will add these results to the appendix.