acceptodds
Under review as a conference paper at ICLR 2027

Rank-Efficient LoRA via Joint Tangent-Space Optimization under Isotropic Curvature

22 views

Abstract

Low-Rank Adaptation (LoRA) is an effective approach for adapting large pretrained models by learning low-rank weight updates. In practice, the LoRA rank is used to control an adapter's parameter budget and representational capacity. We show that this view is incomplete: while the nominal rank determines the representational capacity, the optimizer shapes how much of that capacity is used in the induced weight-space updates. In a case study of GPT-2 adaptation with LoRA, we observe a strong rank-dependent optimizer effect. Despite using the same nominal rank, AdamW often produces per-step updates with concentrated singular spectra and low effective rank, whereas Muon uses a richer set of directions and benefits more consistently from increasing LoRA rank. These observations motivate Iso-LoRA, an optimizer that couples the LoRA factor updates through spectral descent on the induced tangent perturbation in weight space. Iso-LoRA promotes updates that distribute energy more evenly across singular directions, improving rank utilization while preserving compatibility with the LoRA parameterization. We complement this design with theoretical guarantees showing that Iso-LoRA can achieve higher effective rank than standard factor-wise optimizers through a one-step analysis under a stylized spiked-gradient model. We validate this design on language-model adaptation across 0.1B–7B-parameter models, where Iso-LoRA improves effective rank and downstream performance, with the strongest gains at moderate-to-large LoRA ranks. Our results highlight rank utilization as a key factor in LoRA optimization and suggest that optimizer design offers an important path toward stronger parameter-efficient adaptation.

Primary area:
optimization
Keywords:
Low-Rank Adaptation (LoRA), Parameter-Efficient Fine-Tuning (PEFT), Tangent-Space Optimization, Spectral Optimization, Effective Rank, Deep Learning Optimization, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dQJSOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how optimizer choice affects the effective rank of LoRA updates at a fixed adapter rank. ISO-LoRA reconstructs the joint tangent-projected weight gradient from factor gradients, orthogonalizes this signal, projects it back into the feasible tangent space, and reconstructs factor updates through small-core computations. First-step theory and conditional finite-horizon analyses characterize spectral differences between optimizers. Experiments across 0.1B–7B language models report improved rank utilization and downstream performance, particularly at moderate-to-large adapter ranks.

Critical strengths
  1. Coherent geometric construction with appropriately scoped theory. The joint tangent formulation accounts for interactions between factor updates, and the small-core implementation avoids dense optimizer state. Appendix B explicitly distinguishes the ideal construction from damping and factor-space momentum. The theoretical claims are limited to the stated first-step and conditional gradient models; they do not assert a general training-trajectory ordering.

  2. Substantial empirical support and relevant resource controls. The experiments include matched-rank geometric baselines, hyperparameter sweeps, three-seed downstream results, learned-adapter spectra, and fixed-time comparisons. At rank 128, LLaMA-2-7B achieves GSM8K accuracy of 61.87±0.64%61.87\pm0.64\%, compared with 59.12±0.37%59.12\pm0.37\% for LoRA-Muon. The separate fixed-time study also favors ISO-LoRA, although those endpoints are single-seed (Table 1; Appendix G.6).

Critical weaknesses
  1. The central update diagnostic is not validated against actual finite factor movement. With applied increments dA=At+1−AtdA = A_{t+1} - A_t and dB=Bt+1−BtdB = B_{t+1} - B_t, including weight decay, the exact change is:

Wt+1−Wt=Tt+EtW_{t+1} - W_t = T_t + E_t

where Tt=s(dABt⊤+AtdB⊤)T_t = s(dA B_t^\top + A_t dB^\top) is the tangent update and Et=s⋅dA dB⊤E_t = s \cdot dA \, dB^\top is the finite cross-term. The reported update-rank diagnostics concern TtT_t. Learned-adapter spectra do not establish that its spectrum describes Tt+EtT_t + E_t at the learning rates used. Factor reconstruction involves inverse Gram matrices, making this distinction particularly relevant when factors are poorly conditioned; damping may control it, but the reported measurements do not resolve that question. This leaves the link between tangent rank utilization and the network's actual steps insufficiently established (Sections 3–4; Appendices B and G.7).

  1. The performance contribution of spectral spreading remains incompletely isolated. Appendix G.4 appropriately describes its 45-run correlation analysis as descriptive. However, these comparisons vary optimizer and nominal rank, jointly changing step scale, preconditioning, and spectral shape. The comparison with LoRA-Muon strengthens the evidence for the overall method, but does not isolate useful spectral spreading from these other changes. Since rank utilization is the paper's central explanatory contribution, a controlled comparison within the same tangent-reconstruction pipeline is needed to establish what the added spectral operation contributes beyond the combined optimizer design.
Questions for authors
  1. Do the exact updates preserve the reported effective-rank ordering? Please measure ∥∣Et∥∣F/∥∣Tt∥∣F\||E_t\||_F/\||T_t\||_F, exact-update spectra, and gradient alignment at representative early and late checkpoints, including poorly conditioned layers. Evidence that the finite updates retain the claimed ordering under the reported settings would resolve the first concern.

  2. At a fixed model and adapter rank, does polar equalization improve downstream performance when the tangent signal, reconstruction, momentum, and step-norm convention are held fixed? A focused comparison with the unflattened signal, with variability across runs, would materially strengthen the claimed mechanism. Together with the finite-update check, a persistent benefit would support raising the rating to weak acceptance.

3 comments

Response to Reviewer dQJS (1/2)

We thank the reviewer for the careful reading and the two very concrete questions. Since Q1 and Q2 are exactly the tests requested in W1 and W2, we answer each weakness together with its question.

1. "The central update diagnostic is not validated against actual finite factor movement..." (W1 and Q1)

We agree that the tangent update TtT_t alone does not show what the network actually receives. To answer this directly, we re-ran the reported GPT-2/E2E setting: r=128r=128, s=4s=4, seed 42, and the paper's learning rates (2e-3 / 1e-3 / 3e-3 / 1e-2 for Iso-LoRA / AdamW / Muon / LoRA-Muon). During these runs we recorded the factors immediately before and after every optimizer step, so weight decay is included. From these we compute the exact increment ΔWt=Tt+Et\Delta W_t = T_t + E_t, with Tt=s(ΔBtAt+BtΔAt)T_t = s(\Delta B_t A_t + B_t \Delta A_t) and Et=s ΔBtΔAtE_t = s\,\Delta B_t \Delta A_t. For each step we report its stable rank, ∥Et∥F/∥Tt∥F\|E_t\|_F/\|T_t\|_F, the regularized Gram condition number of the factors, and cos⁡(−Gt,ΔWt)\cos(-G_t, \Delta W_t), averaged over all 48 adapted matrices. Step 1 has zero learning rate because of warmup.

Stable rank of the exact update ΔWt\Delta W_t (for Iso-LoRA we also show the tangent part TtT_t):

StepIso-LoRA (TtT_t / ΔWt\Delta W_t)LoRA-MuonMuonAdamWIso-LoRA Et/TtE_t/T_t
2128.0 / 128.0127.855.12.10.000
5136.8 / 115.5122.052.91.90.304
10116.7 / 103.7121.851.51.70.094
50152.9 / 147.395.254.22.30.046
100173.8 / 168.593.456.83.20.035
740188.9 / 188.6104.062.16.00.007
1406188.9 / 188.9104.760.27.10.001

The answer to Q1 is yes. Against Muon and AdamW, Iso-LoRA's exact update has a higher stable rank at every sampled step, from the first steps to the end of training. Against LoRA-Muon, the ordering holds at every sampled step from step 50 until the end of training; steps 370, 1110 and 1480 (not shown) behave like steps 740 and 1406. Only in the first few steps (steps 5 and 10) is LoRA-Muon's rank higher, and this is exactly where the finite cross-term is largest.

Et/TtE_t/T_t is 0.30 on average (0.36 at most) at step 5, but it quickly becomes negligible: 0.035 by step 100 and below 0.01 from step 740 onward. From then on, the exact rank and the tangent rank coincide (188.6 vs. 188.9 at step 740). Thus, TtT_t faithfully describes the actual step for essentially all of training, and the rank ordering it predicts carries over to the exact update. We will include the full per-step table in the appendix.

On poorly conditioned layers. We agree that, because the reconstruction uses inverse Gram matrices, poorly conditioned factors are the critical case. In our measurements, Iso-LoRA keeps its own factors well conditioned. Over all layers and sampled steps, the largest regularized Gram condition number is 118 for Iso-LoRA, compared with 188 for LoRA-Muon, 245 for Muon and 1.1×1061.1\times10^{6} for AdamW.

The worst-conditioned layers also behave normally (separate run, seed 43): layers in the top condition decile have a mean Et/TtE_t/T_t of 0.050, essentially the same as the 0.048 over all layers. For example, the worst-conditioned layer at step 100 (h9.mlp.c_proj, κ=53.6\kappa = 53.6) still has exact-update stable rank 164.7 and Et/Tt=0.033E_t/T_t = 0.033. Consistent with this, changing the damping from 1e-8 to 1e-10 or 1e-6 changes the final validation loss by less than 0.001 over three seeds (see our response to Reviewer uF3g).

On gradient alignment. We tracked cos⁡(−Gt,ΔWt)\cos(-G_t, \Delta W_t) on the same three fixed layers for all four methods; it is positive at every recorded nonzero step.

StepIso-LoRAMuonLoRA-MuonAdamW
1000.1410.0950.1030.106
14060.1960.2080.1420.103

Spreading the update over more directions therefore does not cost descent alignment: Iso-LoRA is the most aligned method at step 100 and close to Muon late in training.

At 7B. The LLaMA-2-7B diagnostic runs (seed 0, 224 adapted matrices) show the same picture. At step 5, Iso-LoRA's exact-update rank is slightly below LoRA-Muon's (122.0 vs. 123.9). By the middle of training (step 1563), it is 203.2 vs. 127.8, with Et/TtE_t/T_t of 0.004 and 0.002, respectively.

Taken together, the exact steps keep the claimed rank ordering, the finite cross-term becomes negligible early in training, and the reconstruction does not operate in an ill-conditioned regime. We will add these results to the appendix.

Response to Reviewer dQJS (2/2)

2. "The performance contribution of spectral spreading remains incompletely isolated..." (W2 and Q2)

We agree that the 45-run correlation in Appendix G.4 cannot isolate the spectral operation, since it varies the optimizer and the rank at the same time. Following the reviewer's suggestion, we built a controlled comparison inside the Iso-LoRA pipeline.

All variants share the tangent-signal construction, the damped reconstruction, momentum 0.9, damping 1e-8 and weight decay 0.01; the only difference is the spectral step. A0 is Iso-LoRA, which applies polar to the joint tangent signal; A1 uses the unflattened tangent signal. To hold the step-norm convention fixed, at every step and in every layer we rescale A1's tangent update by a scalar. This makes its Frobenius norm match the joint reference computed from the same state (relative tolerance 1e-3). The first-order step size is therefore identical, and only its spectral shape differs.

We use r=128r=128, s=4s=4 and FP32 factors. We select each variant's LR on seed 42 by validation loss and report mean ± sample SD over seeds 43/44/45.

ComparisonA0 (polar) val. lossA1 (no polar) val. lossPaired A1−A0 lossPaired A1−A0 BLEU
Each at its best LR (A0: 2e-3, A1: 1.25e-4)1.1516±0.00151.2910±0.0012+0.1394±0.0008−1.03±0.20
Shared LR 1.25e-41.2645±0.00031.2910±0.0012+0.0265±0.0013−0.19±0.27
Shared LR 2.5e-4, FP64 factors1.2073±0.00041.3919±0.0580+0.1846±0.0577−3.27±1.16

The answer to Q2 is yes. When each variant uses its best learning rate, removing the polar step increases validation loss by 0.139 and decreases BLEU by 1.03 points, consistently across all three seeds. A0 is also better on NIST, METEOR, ROUGE-L and CIDEr. The benefit persists when both variants use the same learning rate (+0.026 in loss), and it is larger in a separate FP64-factor control (+0.185 in loss, −3.27 BLEU). In the seed-42 sweeps, A0 has a lower validation loss than A1 at every learning rate where both were run.

The comparison also shows why the polar step matters in this pipeline. Without it, the exact update collapses to a stable rank of about 2, versus about 150 for A0. At the same Frobenius norm, the step is therefore concentrated in a few directions. As a result, A1's factors become extremely ill-conditioned even at the shared learning rate: the maximum Gram condition number is 4.9×10114.9\times10^{11} for A1, versus 39 for A0. At learning rates of 2.5e-4 and above in FP32, A1 can no longer satisfy the norm-matching check, so its best stable learning rate is 16× smaller than A0's. In this sense, spreading the step across directions is what allows a larger and still stable step at the same norm.

We also compared joint polar with polar applied separately to the two tangent components (A2), which is the factor-wise analogue. Joint polar is better on every seed at the two learning rates where performance is competitive, but slightly worse at the smallest learning rate; details are in our response to Reviewer uF3g. We could not complete the same ablation at 7B, because A1 failed the norm-matching check after 113 and 121 steps on two seeds. We therefore do not claim a 7B mechanism result.

We hope that the exact-update measurements and the controlled polar comparison address both of the reviewer's concerns, and we would be happy to provide any further details.

Thank you for running both comparisons. I appreciated seeing the early steps where LoRA-Muon has the higher stable rank, and from step 50 onward the sampled GPT-2 checkpoints show that Iso-LoRA's advantage carries over to the actual weight updates. The polar ablation on GPT-2 is also informative, though the downstream BLEU gain is modest at a shared learning rate and the 7B check could not be completed. While this adds support for the mechanism on GPT-2, the evidence explaining the headline downstream gains still feels somewhat limited.

I also have a question about the 59.12% GSM8K exact match reported for LoRA-Muon in Table 15 (and Table 1(a)). If each of the three seeds evaluated all 1,319 test questions, the nearest possible means round to 59.11% and 59.14%. Could you give the number correct and the number evaluated for each seed, and clarify how you computed the mean? In addition, I could not locate the new diagnostic and ablation tables in the PDF available to me; if they appear in a revised version, a page reference would help. I have taken the results in your response into account and am keeping my rating at 2.

Reviewer_nQzVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how optimizer choice affects the effective rank of weight updates in LoRA. Motivated by observations that AdamW produces more spectrally concentrated updates than Muon, the authors propose ISO-LoRA, which jointly optimizes the weight-space tangent update induced by both LoRA factors. The paper provides a first-step stable-rank comparison of GD, Muon, and ISO-LoRA under a spiked-gradient model, together with conditional multi-step results for GD and Muon. Experiments across language models report higher effective rank and downstream improvements, particularly at moderate-to-large LoRA ranks.

Critical strengths
  1. The method accounts for how both factor updates jointly affect the adapted weight, rather than applying spectral optimization independently to each factor. The tangent-space formulation explicitly identifies feasible updates and provides a neat reconstruction of factor updates that preserves the standard LoRA parameterization.

  2. ISO-LoRA recovers the tangent-gradient signal from factor gradients and performs SVD through a small core matrix, avoiding large computation, which makes it practical for larger models.

  3. ISO-LoRA is reported to be effective at moderate-to-large ranks.

Critical weaknesses
  1. Spectral concentration of per-step updates does not necessarily imply underutilization of LoRA’s representational capacity: useful task information may lie in a few directions. Besides, successive low-effective-rank updates may explore different subspaces.

  2. The first-step analysis uses a restrictive spiked-gradient model, while the multi-step results cover GD and Muon rather than ISO-LoRA. These results do not establish sustained benefits for the proposed optimizer.

  3. Spectral flattening may amplify noise-dominated directions. The batch-size ablation provides indirect evidence but does not isolate gradient noise or establish robustness at low signal-to-noise ratios.

Questions for authors
  1. Given the close connection to concurrent LoRA-TSD, could the authors briefly clarify the differences in factor reconstruction and practical implementation? This would help contextualize the contribution.

  2. If the method improves rank utilization, can a lower-rank ISO-LoRA adapter match a higher-rank baseline while reducing parameter count, memory, or training time?

  3. The experiments appear to fix the LoRA scaling factor while tuning other optimizer hyperparameters. How sensitive are the conclusions to this choice, and does ISO-LoRA retain its advantage when both the scaling factor and learning rate are tuned for the baselines?

4 comments

We sincerely thank the reviewer for the thoughtful and precise critique. We address each weakness and question below. Unless stated otherwise, all methods are re-run in the GPT-2/E2E setting of Table 1 under a common protocol, each with its tuned configuration, and we report mean ± SD over three seeds.

Regarding weaknesses

1. "Spectral concentration of per-step updates does not necessarily imply underutilization... successive low-effective-rank updates may explore different subspaces."

We thank the reviewer for this important point. It contains two separate claims, and we tested both directly.

Do successive updates explore different subspaces? For each method, we accumulated the second moments of the exact updates over 50-step windows: ∑tΔWtΔWt⊤\sum_t \Delta W_t \Delta W_t^\top on the output side and ∑tΔWt⊤ΔWt\sum_t \Delta W_t^\top \Delta W_t on the input side. We used three windows, at steps 148–197, 740–789 and 1332–1381. For each second moment we computed its entropy rank, and we also computed the overlap of its top-32 subspace with that of the next window. The table averages over three fixed layers (h0.attn.c_attn, h4.attn.c_attn and h9.mlp.c_proj) for seed 43.

MethodWindow rank (output / input side)Overlap between successive windows (output / input)
Iso-LoRA480.6 / 454.90.34 / 0.45
Muon398.8 / 195.40.33 / 0.62
LoRA-Muon295.5 / 291.40.34 / 0.41
AdamW259.6 / 51.20.23 / 0.76

The window-accumulated rank directly measures how many directions a method covers over time, so it is exactly the quantity that would reveal such compensation. Iso-LoRA covers by far the most directions on both sides (480.6 / 454.9). Even after accumulating 50 steps, AdamW covers only about half as many on the output side (259.6) and about one ninth on the input side (51.2). On the input side, AdamW also keeps returning to the same subspace (overlap 0.76). On the output side, its subspace does change more between windows (overlap 0.23), but this rotation does not make up the gap: within each window, its accumulated updates still span far fewer directions than Iso-LoRA's. Thus, changing subspaces over time does not compensate for low per-step rank; Iso-LoRA's per-step advantage carries over to the accumulated updates.

Could the useful information lie in a few directions? We tested this by intervention, checking whether the additional directions actually help. In a controlled ablation, we keep the tangent signal, reconstruction, momentum and per-layer step norm of Iso-LoRA fixed and only remove the polar step. The exact update then collapses from a stable rank of about 150 to about 2. Validation loss becomes worse on every seed: by 0.139 when each variant uses its best learning rate, and by 0.026 at a shared learning rate (details in our responses to Reviewers dQJS and uF3g). So the additional directions carry useful information, and the concentrated update does leave useful capacity unused.

We will add both analyses to the paper. Together, they support the underuse claim directly, by measuring accumulated subspace coverage and by intervention, rather than inferring it from the per-step spectrum alone.

2. "The first-step analysis uses a restrictive spiked-gradient model, while the multi-step results cover GD and Muon rather than ISO-LoRA..."

We thank the reviewer for the precise summary of the theory's scope. The theory is designed to explain the mechanism, not to give a sustained guarantee for Iso-LoRA. The first-step spiked-gradient result isolates why spectral equalization raises the rank of an update when the gradient is dominated by a few directions, and the conditional multi-step results characterize how the update spectrum evolves under GD and Muon, the baselines whose behavior motivates our method. The isotropic-curvature model is the assumption from which the update rule is derived, as is common for preconditioned optimizers; we do not claim that real Hessians are isotropic. In the revision, we will make this scope explicit in Section 4.

The sustained behavior of Iso-LoRA itself is supported by direct measurement. We tracked the exact finite update ΔWt=Tt+Et\Delta W_t = T_t + E_t throughout training at the reported settings. Iso-LoRA's exact-update stable rank stays above Muon's and AdamW's at every sampled step until the end of training (e.g., 188.6 vs. 62.1 and 6.0 at step 740), and from step 50 onward it is also above LoRA-Muon's. The finite cross-term EtE_t is below 1% of the tangent step from step 740 onward, so the tangent-space picture used in the analysis accurately describes the actual updates. The same ordering holds in our LLaMA-2-7B diagnostic runs (203.2 vs. 127.8 for LoRA-Muon at mid-training).

Response to Reviewer nQzV (2/3)

3. "Spectral flattening may amplify noise-dominated directions..."

We thank the reviewer for raising this point. We agree that the batch-size ablation is only indirect evidence, so we tested the concern directly in two ways: by measuring gradient SNR along the update directions in normal training, and by a controlled noise-injection stress test.

Gradient SNR along the update directions. If flattening amplified noise-dominated directions, a large share of Iso-LoRA's update energy should lie in directions where the gradient signal is weaker than its noise. We estimated a per-direction gradient SNR along the singular directions of the actual update in clean training. We used steps 148/740/1406, the same three layers as above and seed 43. With K=16K=16 resampled minibatches, we define SNRj=max⁡(μ^j2−sj2/K, 0)/sj2\mathrm{SNR}_j = \max(\hat\mu_j^2 - s_j^2/K,\,0)/s_j^2. The share of update energy lying in directions with SNR < 1 is:

Iso-LoRAAdamWLoRA-MuonMuon
69.8%73.4%66.8%58.9%

All four methods place a similar share of their update energy in low-SNR directions (59–73%), which reflects the minibatch noise level of the task rather than the optimizer. Iso-LoRA lies within this range and puts less energy into low-SNR directions than AdamW, so in normal training, equalizing the spectrum does not make the update chase noise. A plausible reason is that the polar step equalizes the singular values but keeps the singular directions of the momentum-averaged tangent signal, in which much of the minibatch noise is already averaged out.

Noise-injection stress test. We then injected noise explicitly. At each step, we draw a Gaussian perturbation NN in weight space and pull it back to the factors as nA=sB⊤Nn_A = sB^\top N and nB=sNA⊤n_B = sNA^\top. We scale it to ρ\rho times the factor-gradient norm and inject it before gradient clipping. The seed fixes the Gaussian draws, so all methods see the same noise sequence. At ρ=1\rho=1, the injected noise is as large as the gradient itself, which is a deliberately extreme setting.

MethodClean val. lossVal. loss at ρ\rho=1Paired increaseBLEU at ρ\rho=1 (clean)
Iso-LoRA1.15161.1921+0.0405±0.002468.91 (68.88)
AdamW1.18391.2018+0.0180±0.001767.78 (67.81)
Muon1.16161.2005+0.0388±0.002868.14 (68.06)
LoRA-Muon1.16081.2299+0.0691±0.001368.02 (68.64)

Even under this extreme noise, Iso-LoRA's BLEU (68.91) is the highest of all methods. Compared with LoRA-Muon, the other method that applies a spectral operation to LoRA updates, Iso-LoRA has both a smaller loss increase and a lower noisy validation loss. Validation loss rises more than for AdamW and Muon at ρ=1\rho=1, but this does not carry over to generation quality. At a more moderate noise level, ρ=0.25\rho=0.25, Iso-LoRA's validation loss is 1.1697±0.0011, still lower than the clean loss of tuned AdamW (1.1839).

Taken together, the SNR analysis reflects the realistic regime, whereas ρ=1\rho=1 is a deliberately extreme stress test in which the noise is as large as the gradient itself. In realistic training, Iso-LoRA does not amplify noise-dominated directions, and even under extreme injected noise it still produces the best generations by BLEU. We will add both analyses to the appendix.

Response to Reviewer nQzV (3/3)

Regarding questions

Q1. "Given the close connection to concurrent LoRA-TSD, could the authors briefly clarify the differences in factor reconstruction and practical implementation?"

Thank you for the question. We cite and discuss LoRA-TSD as concurrent work in the paper, and we are happy to clarify the differences in more detail here. Following its Eqs. (12), (16) and (18) and Algorithms 1–2, and writing both methods in W=BAW = BA notation, the main differences are as follows.

  • Factor reconstruction. LoRA-TSD reconstructs ΔB=ZA†\Delta B = Z A^{\dagger} and ΔA=B†(Z−ΔBA)\Delta A = B^{\dagger}(Z - \Delta B A). Our ideal (undamped) reconstruction assigns the column-space component first: ΔA=B†Z\Delta A = B^{\dagger} Z and ΔB=(I−PB)ZA†\Delta B = (I - P_B) Z A^{\dagger}. Both realize the same tangent direction, but they split it differently between the two factors, which changes the finite cross-term EtE_t.
  • Spectral step. LoRA-TSD alternates tangent projection and polar refinement τ\tau times. Iso-LoRA applies a single polar operation to the joint tangent signal, followed by a damped inverse-Gram reconstruction.
  • Implementation. Both methods use factor momentum and work only with small 2r2r-dimensional cores.

The two papers also ask different questions. LoRA-TSD focuses on convergence. Our focus is rank utilization, i.e., how the optimizer shapes the effective rank of LoRA updates, together with the corresponding analysis and controlled experiments. We will expand the comparison in the related-work discussion accordingly.

Q3. "...does ISO-LoRA retain its advantage when both the scaling factor and learning rate are tuned for the baselines?" (We answer Q3 before Q2, since Q2 relies on these tuned baselines.)

Yes. For each baseline, we tuned the learning rate jointly with s=α/rs = \alpha/r on seed 42: s∈{1,4,16,64,256}s \in \{1, 4, 16, 64, 256\} for AdamW, and s∈{1,4,16}s \in \{1, 4, 16\} for Muon and LoRA-Muon. Each selected learning rate has tested neighbors on both sides.

Method (selected ss, LR), r=128r=128Val. lossPaired Iso-LoRA − baselineBLEU
Iso-LoRA (4, 2e-3)1.1516±0.0015–68.88±0.31
AdamW (256, 3.125e-5)1.1839±0.0005−0.0323±0.001167.81±0.36
Muon (4, 3e-3)1.1616±0.0010−0.0101±0.000868.06±0.48
LoRA-Muon (16, 1.25e-3)1.1608±0.0009−0.0093±0.001568.64±0.40
LoRA-Muon (1, 2e-2)1.1616±0.0003−0.0100±0.001868.57±0.24

Iso-LoRA keeps the lowest validation loss and highest BLEU score against every tuned baseline, and every Iso-LoRA seed is below every completed baseline seed.

AdamW is the baseline that benefits most from a larger scaling factor, but the gain saturates. Its best seed-42 loss goes 1.2156 / 1.1968 / 1.1863 / 1.1828 / 1.1812 for ss = 1 / 4 / 16 / 64 / 256. The last two 4× increases gain only 0.0035 and 0.0016, far less than the 0.032 gap to Iso-LoRA.

At the selected s=256s=256, AdamW's learning rate is bracketed by stable neighbors on both sides (the doubled learning rate gives 1.2207 vs. 1.1812), so this is a genuine optimum of the extended search.

Q2. "...can a lower-rank ISO-LoRA adapter match a higher-rank baseline while reducing parameter count, memory, or training time?"

Yes in terms of parameter count. Iso-LoRA at r=64r=64 (9.4M adapter parameters) beats tuned AdamW at r=128r=128 (18.9M), with a paired validation-loss difference of −0.0193±0.0012, and comes within 0.004 of Muon and LoRA-Muon at r=128r=128 with half the parameters (+0.0030±0.0014 and +0.0038±0.0001). With a quarter of the parameters, Iso-LoRA at r=32r=32 (4.7M) nearly matches AdamW at r=128r=128 (+0.0034±0.0010). BLEU differences in all of these comparisons are within seed noise.

For memory and time, the picture at GPT-2 scale is shaped by activations. In single-H100 measurements, peak memory is dominated by activations rather than by the adapter or optimizer state, so Iso-LoRA at r=32r=32 and AdamW at r=128r=128 have similar peak memory (22.70 vs. 23.21 GiB), while the optimizer state itself shrinks by 4× (36 vs. 144 MiB). The per-step time of Iso-LoRA at r=32r=32 is slightly higher (0.381 vs. 0.332 s).

Thanks for the rebuttal, which alleviates some of my original concerns, particularly through the controlled polar ablation and the update diagnostics. My remaining reservation is primarily about the overall significance and scope of the contribution. The evidence appears to be concentrated on two task pipelines, and most new mechanistic analyses are confined to GPT-2/E2E. I therefore remain slightly below the acceptance threshold.

Reviewer_uF3gOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a novel optimizer, ISO-LoRA, for large language models with LoRA. This optimizer is designed to improve rank utilization in LoRA fine-tuning. The authors provide theoretical analyses and conduct comprehensive experiments across language models ranging from 0.1B to 7B parameters, demonstrating improved rank utilization and downstream performance compared with existing optimizers (AdamW and Muon). The idea is novel. The workload for this paper is huge, but most content is well-organized. Thus, I gave this paper a relatively high rating.

Critical strengths
  1. The motivation is inspiring and clear. Through comparing ISO-LoRA with AdamW and Muon, the authors unveil the impact of optimizers on LoRA rank utilization.
  2. The proposed method is well-founded and reasonable.
  3. Comprehensive experiments were included in the paper. More comparisons can be found in Appendix.
Critical weaknesses
  1. Insufficient component ablation studies. Though Table 15 compares ISO-LoRA and LoRA-Muon, the paper lacks ablation studies to isolate the contributions of its key components, particularly joint tangent-space optimization.

  2. Limited diversity of downstream tasks. The experiments only cover two types of NLP tasks: data-to-text generation and mathematical reasoning. Evaluations on other NLP tasks, such as text classification or summarization, would help demonstrate ISO-LoRA's generalizability.

  3. Insufficient hyperparameter sensitivity analysis. Although learning-rate sweeps are provided in Appendix G, the reported results are selected over other hyperparameters, making it difficult to assess ISO-LoRA's robustness compared with existing optimizers under different hyperparameter settings.

  4. Insufficient evidence linking higher stable rank to performance improvements. Though Figure 7 shows a strong correlation between stable rank and validation loss, the analysis combines different LoRA ranks and optimizers. It remains unclear to what extent the performance improvements can be attributed specifically to increased rank utilization.

Questions for authors

Please see Weaknesses.

2 comments

We sincerely thank the reviewer for the positive assessment and the constructive suggestions. We address each weakness below. Unless stated otherwise, all methods are re-run in the GPT-2/E2E setting of Table 1 under a common protocol, and we report mean ± SD over three seeds.

1. "Insufficient component ablation studies... particularly joint tangent-space optimization."

We agree, and we added a controlled ablation within the Iso-LoRA pipeline. All variants share the same tangent signal, damped reconstruction, momentum 0.9, damping 1e-8 and weight decay 0.01. In addition, at every step each layer's tangent step norm is matched to that of the joint variant, so that the variants differ only in the spectral operation:

  • A0 (Iso-LoRA) applies polar to the joint tangent signal;
  • A1 removes the polar step and uses the unflattened signal;
  • A2 applies polar separately to the two tangent components. This is the factor-wise analogue, close in spirit to running Muon on each factor.

Because the variants prefer different learning rates, we compare them at three shared learning rates as well as at each variant's own best one. In the table, "rank" is the stable rank of the exact update ΔWt\Delta W_t, including the finite cross-term. It is averaged over 48 layers and 11 sampled steps of seed 43.

Variant, LRVal. loss ↓BLEU ↑Rank
A1 (no polar), 1.25e-4 (A1's best)1.2910±0.001267.85±0.362.0
A0 (joint), 1.25e-41.2645±0.000368.04±0.45150.0
A2 (separate), 1.25e-41.2609±0.000768.30±0.1695.4
A0 (joint), 1e-31.1568±0.000268.65±0.84159.9
A2 (separate), 1e-3 (A2's best)1.1613±0.000268.84±0.43102.1
A0 (joint), 2e-3 (A0's best)1.1516±0.001568.88±0.31161.6
A2 (separate), 2e-31.1609±0.001368.83±0.27104.1

Both components contribute.

The polar step. Removing the polar step is clearly harmful. Without it, the exact update collapses from a stable rank of about 150 to about 2, so the step concentrates in a few directions. At their respective best learning rates, A1's validation loss is 1.2910 versus 1.1516 for A0, a paired increase of 0.139±0.001 that holds on every seed, with BLEU 1.03 points lower. The gap is not just a matter of learning rate: at the shared learning rate 1.25e-4, where both variants take exactly the same step size, A1 is still worse (+0.026 in loss). A1 also cannot use larger learning rates, because its factors become severely ill-conditioned (Gram condition number up to 4.9×10114.9\times10^{11}, versus 39 for A0).

The joint coupling. Applying polar jointly gives a clear further improvement over applying it separately. At learning rates 1e-3 and 2e-3, which contain the best setting of both variants, A0 has a lower validation loss than A2 on every seed, with paired differences of 0.0045±0.0003 and 0.0093±0.0024. At their respective best learning rates, A0 reaches 1.1516 versus 1.1613 for A2. A0's update is also substantially richer: its stable rank is about 160, versus about 102 for A2. The reason is that separate polar flattens the two tangent components individually, but not their sum, which is the update the network actually receives. Joint polar flattens this sum directly. (At the smallest learning rate, both variants are far from converged, with losses around 1.26, and A2 is marginally lower.)

This also indicates where Iso-LoRA's gain over factor-wise spectral methods comes from. The factor-wise variant A2 reaches a best validation loss of 1.1613, essentially the same as tuned Muon (1.1616) and LoRA-Muon (1.1608). Only the joint formulation (A0) improves further, to 1.1516. In other words, Iso-LoRA's advantage of about 0.01 over these baselines matches the gain from joint coupling.

2. "Limited diversity of downstream tasks..."

We agree that two task types are limited. Following the reviewer's suggestion, we added AG News topic classification. This task differs in kind from data-to-text generation and mathematical reasoning: instead of generating long sequences, the model predicts a single label, so it tests whether Iso-LoRA's benefit carries over to a discriminative objective.

Setup. We fine-tune GPT-2 with a smaller adapter, r=32r=32 and s=4s=4. The model is trained with the cross-entropy on the answer label token, for one epoch over 108,000 training examples. We hold out a stratified validation set of 12,000 examples, used to select each method's learning rate, and evaluate on the full test set of 7,600 examples. Results are over three seeds; all runs completed.

MethodVal. label CE ↓Test label CE ↓Test accuracy (%) ↑Macro-F1 (%) ↑
Iso-LoRA0.1707±0.00160.1817±0.002294.03±0.2294.02±0.22
AdamW0.1716±0.00040.1826±0.003493.71±0.1593.71±0.15
LoRA-Muon0.1746±0.00170.1834±0.002893.57±0.1393.56±0.12

2. (continued)

Iso-LoRA achieves the best result on all four metrics, on both the validation and the test set. The improvement over LoRA-Muon, the closest spectral baseline, is consistent across seeds: +0.46±0.10 accuracy points in paired comparison, with lower cross-entropy on both splits. Iso-LoRA also improves on AdamW in every metric (+0.32 accuracy points on average). Since accuracy on AG News is already close to saturation for GPT-2, these margins are in line with what one would expect on this benchmark. Together with the existing experiments, this shows that Iso-LoRA's advantage holds across data-to-text generation, mathematical reasoning and classification, and also at a smaller adapter rank. We will add this task to the paper.

3. "Insufficient hyperparameter sensitivity analysis..."

We agree. The learning-rate sweeps in Appendix G plot, for each learning rate, the best result over momentum and weight decay, so they do not show how each method behaves when the other hyperparameters are held fixed. We now report three sensitivity analyses in which only one hyperparameter changes at a time.

(a) Learning rate, compared across optimizers. The table shows seed-42 validation loss as the learning rate moves around each method's selected value, with all other hyperparameters fixed (r=128r=128, each method at its selected scaling factor). "Diverged" means the loss became non-finite, and "–" means not run, since the loss already rises on that side.

Method (scaling ss)¼×½×1× (selected)2×4×
Iso-LoRA (ss=4)1.17331.15511.15261.1740diverged
AdamW (ss=256)1.22101.19181.18121.2207–
Muon (ss=4)–1.17521.15981.1890diverged
LoRA-Muon (ss=16)–1.17551.15921.16061.2208

At half of its selected learning rate, Iso-LoRA (1.1551) is still better than every baseline at that baseline's own best learning rate. Over the 4× range from ½× to 2×, Iso-LoRA's loss varies by at most 0.021. This is comparable to Muon (0.029) and LoRA-Muon (0.016), and clearly smaller than AdamW (0.040). In other words, Iso-LoRA's advantage does not rely on hitting a narrow learning-rate window. (Baseline ss was tuned jointly with the learning rate; AdamW's large s=256s=256 is intentional: its loss kept improving with ss, so we extended the grid until the gain saturated; see our response to Reviewer nQzV, Q3.)

(b) LoRA scaling factor. Iso-LoRA's weight-space step scales as LR⋅s2\mathrm{LR}\cdot s^2, so we scale the learning rate as 1/s21/s^2 when changing ss. Under this rule, Iso-LoRA is essentially insensitive to ss: the seed-42 validation losses are 1.1489 / 1.1526 / 1.1494 at ss = 1 / 4 / 16. Over three seeds, s=1s=1 gives 1.1511±0.0009, versus 1.1516±0.0015 at s=4s=4.

(c) Damping. Changing only the damping ϵ\epsilon (r=128r=128, s=4s=4, LR 2e-3, three seeds):

ϵ\epsilon1e-101e-8 (default)1e-6
Val. loss1.1507±0.00171.1516±0.00151.1511±0.0009
BLEU68.94±0.4168.88±0.3168.84±0.34

Across four orders of magnitude, loss changes by less than 0.001 and BLEU by less than 0.1.

Overall, Iso-LoRA is robust along all three axes: it beats every baseline even at half its selected learning rate, it is insensitive to the scaling factor once the learning rate follows the LR⋅s2\mathrm{LR}\cdot s^2 rule, and its damping can be left at the default. We will add the full learning-rate × scaling grids, including diverged runs, to Appendix G.

4. "Insufficient evidence linking higher stable rank to performance improvements..."

We agree that Figure 7 pools different optimizers and ranks, so on its own it cannot attribute the gains to rank utilization. The ablation in point 1 provides exactly this controlled test: it keeps the nominal rank, tangent signal, reconstruction and step norm fixed and changes only the spectral operation, so differences in update rank are no longer confounded by the optimizer family or the adapter size.

In this controlled setting, higher rank utilization consistently comes with better performance. Removing the polar step lowers the stable rank from about 150 to about 2 and increases validation loss by 0.026 at a shared learning rate and by 0.139 at the best learning rates. Replacing joint with separate polar lowers the rank from about 160 to about 102 and again increases the loss on every seed at learning rates 1e-3 and 2e-3. The three variants are ordered identically by rank (A0 > A2 > A1) and by best validation loss (1.1516 < 1.1613 < 1.2910).

We will add this controlled table to the paper next to Figure 7, as direct evidence that the gains are tied to increased rank utilization.

We thank the reviewer again for the suggestions, which made the empirical support for the mechanism considerably more direct.

Comments on the paper

1 comment

General response to all reviewers

We sincerely thank all three reviewers for their careful and constructive reviews. Here we briefly summarize the main additions; full details are in the individual responses.

  1. Exact finite updates (dQJS). The rank analysis in the paper is based on the first-order tangent step. The weight update the network actually receives is the exact increment ΔWt=s(Bt+ΔBt)(At+ΔAt)−sBtAt=Tt+Et\Delta W_t = s(B_t+\Delta B_t)(A_t+\Delta A_t) - sB_tA_t = T_t + E_t. Here Tt=s(ΔBtAt+BtΔAt)T_t = s(\Delta B_t A_t + B_t \Delta A_t) is the tangent step and Et=s ΔBtΔAtE_t = s\,\Delta B_t \Delta A_t is the finite second-order cross-term. We re-ran the paper's GPT-2/E2E setting (r=128r=128, s=4s=4, the paper's learning rates) with Iso-LoRA, LoRA-Muon, Muon and AdamW. During training we recorded the factors before and after every optimizer step and computed the stable rank of ΔWt\Delta W_t, averaged over all 48 adapted matrices.
    • From step 50 until the end of training, Iso-LoRA's exact-update stable rank is the highest of the four methods. It is above Muon's and AdamW's at every sampled step; at step 740, for example, it is 188.6, versus 62.1 for Muon and 6.0 for AdamW.
    • EtE_t falls below 1% of TtT_t from step 740 onward, so the tangent step accurately describes the actual update for almost all of training.
    • Iso-LoRA's factors are the best conditioned of all methods: the maximum Gram condition number is 117.5 for Iso-LoRA, versus 1.1×1061.1\times10^6 for AdamW.
    • In diagnostic runs on LLaMA-2-7B, the ordering against LoRA-Muon is the same. At mid-training, Iso-LoRA's exact-update rank is 203.2 versus 127.8.
  2. Controlled ablation of the spectral step (dQJS, uF3g). To isolate the role of the polar step, we compared variants inside the Iso-LoRA pipeline (GPT-2/E2E, r=128r=128). All variants share the tangent signal, damped reconstruction and momentum. At every step, each layer's update norm is rescaled to match Iso-LoRA's, so only the spectral shape differs.
    • Removing the polar step collapses the stable rank of the exact update from about 150 to about 2. Validation loss rises by 0.139 at each variant's best learning rate, on all three seeds.
    • The gap remains when the factors are kept in FP64 at a shared learning rate (1.207 vs. 1.392), so it is not a numerical-precision artifact.
    • Applying polar to the joint tangent signal also beats applying it separately to the AA and BB components, on every seed, at learning rates 1e-3 and 2e-3.
  3. Baselines with tuned scaling factor and learning rate (nQzV). For each baseline, we tuned the learning rate jointly with the LoRA scaling factor s=α/rs=\alpha/r on seed 42, with ss up to 256 for AdamW and up to 16 for Muon and LoRA-Muon. We then evaluated on three new seeds. Iso-LoRA keeps the lowest validation loss: 1.1516, versus 1.1839 (AdamW), 1.1616 (Muon) and 1.1608 (LoRA-Muon). The paired improvements are 0.009–0.032, and every Iso-LoRA seed is below every baseline seed.
  4. Robustness to hyperparameters (uF3g).
    • Learning rate. At half of its selected learning rate, Iso-LoRA (1.1551) still beats every baseline at that baseline's own best learning rate (at best 1.1592).
    • Scaling factor. Iso-LoRA's weight-space step scales as LR⋅s2\mathrm{LR}\cdot s^2. With the learning rate adjusted accordingly, the scaling factor barely matters: 1.1511 at s=1s=1 vs. 1.1516 at s=4s=4.
    • Damping. Varying the damping from 1e-10 to 1e-6 changes the loss by less than 0.001.
  5. Rank efficiency and a new task (nQzV, uF3g).
    • Rank efficiency. On GPT-2/E2E, Iso-LoRA at r=64r=64 (9.4M adapter parameters) reaches 1.1646 validation loss, beating tuned AdamW at r=128r=128 (18.9M parameters, 1.1839).
    • New task. On AG News topic classification (GPT-2, r=32r=32), Iso-LoRA has the lowest validation cross-entropy and the highest test accuracy (94.03%) among Iso-LoRA, AdamW (93.71%) and LoRA-Muon (93.57%).
  6. Subspace exploration over time (nQzV). We checked whether low per-step rank is compensated by updates that move through different subspaces over time. To do this, we summed the second moments of the exact updates over 50-step windows. Iso-LoRA's accumulated updates still span the most directions of all methods: the entropy rank is 480.6 / 454.9 on the output / input side, versus 259.6 / 51.2 for AdamW.

We will add all of these results to the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.