acceptodds
Under review as a conference paper at ICLR 2027

Beyond Non-Zero Initialization: Investigating Update Balance in Structured LoRA Initialization

Abstract

Low-rank adaptation (LoRA) fine-tunes pretrained models through two small factor matrices. Structured initializers choose a nonzero adapter using spectral or contextual information, but even the same initialized adapter can be represented by factor pairs that learn differently. We identify their relative scale as an overlooked degree of freedom: reciprocal rescaling leaves the initial adapter and frozen residual unchanged while redistributing the factors’ contributions to an update. Our analysis shows how the selected structure determines this allocation and distinguishes correctable imbalance from limits that scalar rescaling cannot overcome. For orthogonal spectral factors, we derive an exact population scaling rule under isotropic Gaussian inputs and single-sample SignSGD. This leads to spectral and activation-based scale choices; InitCal complements them with reversible AdamW probes that measure the optimizer’s response. Across Qwen3 and Llama models, calibration improves the four-task average in all nine structured model–initializer settings, with gains in 35 of 36 task-level comparisons. Paired controls show that update magnitude alone does not explain these improvements. Factor-scale allocation thus provides a practical way to improve structured initialization without changing the adapter it selects. The code is available [here](https://anonymous.4open.science/r/initcal-7406).

Primary area:
optimization
Keywords:
Low-rank adaptation, LLM Fine-tuning, Parameter-efficient fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mtWiOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors study the issues of scalings for the two factors in LoRA, specifically for the non-zero structured initializer case. They propose SignSGD and other variant to improve the optimization for structured initializers without changing the actual initialization.

Critical strengths

The work seemingly unify or summarize a way to view the many previous works in the literature.

Critical weaknesses
  1. One the main issue is writing. The equations, definition, and descriptions are scatter around the places. It is hard to track what are the proposals and the reasoning behind them. For example. in Sec 4, the readers are pointed to equation 7 and appendix N respectively to see the definition of the method and some of the property it has without any explanation.

  2. The empirical evaluations seem incomplete and the relationship with some related work remains unclear. Methods like LoRA-RITE and LoRA-Muon are mentioned, but seemingly never compared in the experiments, while these method seem to be applicable.

Questions for authors

Could the authors compare with LoRA-RITE and LoRA-Muon, or explainable why they are not comparable?

4 comments
AuthorsOpenReview ↗

Response to Reviewer mtWi [Part I]

Hi, Reviewer mtWi:

Thank you for your careful reading and detailed suggestions. We sincerely appreciate the time you spent following the method. Your comments pinpoint where the presentation needs a clearer explanation. We have revised the method and expanded the optimizer comparisons.

Q1: "The equations, definition, and descriptions are scatter around the places."

R1: Thank you for this careful feedback and the specific example in Section 4. We apologize for making the method harder to follow than it needs to be. We agree that the presentation needs a broader revision. We have placed each scale rule beside its purpose and conditions before referring readers elsewhere.

Beyond this example, we have reviewed the equations, definitions, and cross-references throughout the paper. We have clarified symbol definitions, placed rules beside their explanations, and removed unnecessary jumps between sections.

For Equation 7, Section 4.1 now explains that Spectrum-full uses the selected singular values to choose the factor scale. Section 4.2 now explains that Forward-moment uses measured activations instead of a Gaussian input model. Its rule requires mutually orthogonal nonzero columns of BB. This condition is now beside the rule; its derivation remains in Appendix N.

Section 4.3 now introduces Direct with a short explanation. Measure the output-update magnitudes DAD_A and DBD_B from one temporary optimizer step, restore the temporary changes, and choose c=DB/DAc=\sqrt{D_B/D_A}. Then apply B←cBB\leftarrow cB, A←c−1AA\leftarrow c^{-1}A. This preserves the initial adapter while changing the two factors' update contributions. Ordinary AdamW training starts with a fresh optimizer.

Section 4 now explains the method without requiring an appendix search. It also clarifies that SignSGD is the analysis model, while language-model training uses AdamW.

Q2: "It is hard to track what are the proposals and the reasoning behind them."

R2: We derive what ratio to use and compute it automatically. The new result is that different selected singular values need different scales, even at the same width and rank. The Gaussian checks fix input/output sizes at 48/96 and rank at 8. Predicted/observed scales are 0.7708/0.7715 for a flat spectrum. They are 0.5789/0.5799 for a concentrated spectrum. Across 60 configurations, median/maximum prediction errors are 0.11%/0.51%.

The analysis separates the ratio of the two update contributions from their combined size. It also gives the condition that keeps their interaction bounded. This motivates the control in Table 1 that matches the initial combined update size. It scores 33.75/45.30 on PiSSA/MiLoRA, compared with Warm's 34.49/45.76. Choosing the factor ratio brings a benefit beyond matching the initial update size. Section 3.3 now connects this theoretical distinction directly to the matched-size control in Section 5.2.

Q3: "Could the authors compare with LoRA-RITE and LoRA-Muon, or explainable why they are not comparable?"

R3: Thank you for suggesting these comparisons. Yes. Warm with AdamW gives 34.49/45.76, close to LoRA-Muon's 34.41/45.86 and LoRA-RITE's 34.37/45.74. We find this comparison useful. Users can reach competitive task scores with a small adjustment before training and keep their familiar AdamW setup.

We compare PiSSA/Qwen3-0.6B and MiLoRA/Qwen3-1.7B under matched training setups. Each optimizer keeps its own update rule. Mean validation negative log-likelihood (NLL) selects one of five learning rates. The table reports its five-seed task means in percent.

ModelMethodGSM8KMATH-500MMLU-ProBBHAvg.
0.6BRaw + AdamW55.8822.5222.6132.9433.49
0.6BRaw + LoRA-RITE56.5724.2023.2633.4634.37
0.6BRaw + LoRA-Muon57.0123.9222.9733.7334.41
0.6BForward-moment + AdamW56.7423.4023.0433.3034.12
0.6BDirect + AdamW56.7123.3623.2433.4634.19
0.6BWarm + AdamW56.8223.8023.4433.9034.49
1.7BRaw + AdamW67.6634.5635.3342.6945.06
1.7BRaw + LoRA-RITE67.7636.4435.7443.0245.74
1.7BRaw + LoRA-Muon68.9534.9635.8843.6745.86
1.7BForward-moment + AdamW68.2335.3235.9743.2745.70
1.7BDirect + AdamW68.2835.0835.8243.2845.62
1.7BWarm + AdamW68.3235.8035.8543.0845.76

A one-time scale adjustment brings AdamW close to these specialized optimizers in the two tested settings.

Revision locations We have revised Sections 4.1–4.3 and added optimizer comparisons in Section 5.4, Table 4.

Please see Part II for the remaining responses.

AuthorsOpenReview ↗

Response to Reviewer mtWi [Part II]

W2: "The empirical evaluations seem incomplete and the relationship with some related work remains unclear."

R4: Our method changes the factor scales once, before training. The training optimizer stays unchanged. LoRA-RITE and LoRA-Muon address factor-aware updates throughout training. We choose initial scales automatically, then use AdamW. Part I shows the practical value of this change before training.

The controls in Table 1 help explain this benefit. Configurations match Part I. The five candidates are five preset combinations of the common learning rate and the factor learning-rate ratio, as specified in Appendix G. Scores are percentages averaged over five seeds.

ModelMethodGSM8KMATH-500MMLU-ProBBHAvg.
0.6BSpectrum-full56.7423.5623.1033.7334.28
0.6BTuned dual learning rates, 5 candidates56.5123.4423.0233.6134.15
0.6BLayer-norm match56.2922.6822.9033.1333.75
0.6BWarm56.8223.8023.4433.9034.49
0.6BWarm-mapped learning rates56.9223.8023.4633.7834.49
1.7BSpectrum-full68.3235.3235.6043.1645.60
1.7BTuned dual learning rates, 5 candidates67.9935.2435.6043.0445.47
1.7BLayer-norm match67.9535.0435.5642.6445.30
1.7BWarm68.3235.8035.8543.0845.76
1.7BWarm-mapped learning rates68.3135.8835.7943.1045.77

We determine the ratio automatically. Matching update size alone gives less benefit, while learning rates mapped from Warm recover its scores. Section 5.2 now explains that the chosen allocation matters; Section 1 identifies automatic selection as the shared benefit of both implementations.

We also tested how the methods handle changes in the initial factor scales. Direct makes AdamW much less sensitive to the initial scale in this test. On Qwen3-1.7B/MiLoRA, rank 64, we apply B←cBB\leftarrow cB, A←c−1AA\leftarrow c^{-1}A with c∈{1/4,1,4}c\in\{1/4,1,4\}. Each representation has the same initial adapter, frozen residual, and output.

Each method keeps the rate selected at c=1c=1. Direct recalibrates once at each starting scale, without retuning. Scores are five-seed means. The c=1c=1 rows reproduce Part I.

MethodInitial scaleGSM8KMATH-500MMLU-ProBBHAvg.
Raw + AdamWc=1/4c=1/467.2833.5634.9742.4644.57
Raw + AdamWc=1c=167.6634.5635.3342.6945.06
Raw + AdamWc=4c=467.7334.1635.1542.4544.87
Direct + AdamWc=1/4c=1/468.2935.2035.5843.6445.68
Direct + AdamWc=1c=168.2835.0835.8243.2845.62
Direct + AdamWc=4c=468.2634.8035.6543.3045.50
Raw + LoRA-RITEc=1/4c=1/467.4136.1635.8542.9745.60
Raw + LoRA-RITEc=1c=167.7636.4435.7443.0245.74
Raw + LoRA-RITEc=4c=467.5435.7635.3742.7245.35
Raw + LoRA-Muonc=1/4c=1/468.7635.0435.4543.1945.61
Raw + LoRA-Muonc=1c=168.9534.9635.8843.6745.86
Raw + LoRA-Muonc=4c=468.2834.5236.1143.1645.52

Direct reduces AdamW's sensitivity to the starting scale without changing the training optimizer. Its average-score range is 0.18 points, compared with 0.49 for Raw AdamW, 0.39 for LoRA-RITE, and 0.34 for LoRA-Muon. Section 5.4 now highlights this benefit.

The original study improves all nine model-initializer averages across Qwen and Llama. Complete Warm calibration adds 0.42% time in the reported configuration, with no change to inference or trainable parameters.

Revision locations Section 5.4 and Table 4 present the optimizer comparison; Appendix P.6, Table 39, gives the scale test. Section 4 explains the procedure before its derivation.

Thank you again for your valuable feedback, which has greatly improved our work. We hope these responses address your concerns. Please let us know of any further questions.

The revision seems to improve the writing. Currently, I am more or less neutral about the acceptance (which is slightly more positive than original).

Dear Reviewer mtWi,

Thank you very much for taking another look at our revision and for your encouraging feedback. We sincerely appreciate your recognition of the improvements.

We are glad that the revised presentation is clearer. We also hope that the newly added comparisons with LoRA-RITE and LoRA-Muon, together with the scale-sensitivity experiments, help address your original concerns about empirical completeness and the relationship to prior work.

We are grateful that these changes have moved your assessment in a more positive direction, and we hope they can be reflected in your final evaluation.

Thank you again for your time and constructive feedback.

Reviewer_QYPtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the problem of imbalance between the output changes produced by updating the two LoRA factors in structured, nonzero, output-preserving initializations. It analyzes how reciprocal rescaling of the factors changes their relative update contributions while preserving their product and the model’s initial output.

It distinguishes balancing the two contributions from controlling their overall magnitude through the learning rate. It shows that the selected singular values help predict a balancing scale beyond what width and rank alone capture. It proposes several calibration methods using spectral information from the initial weights, activation statistics from a forward pass, or measured optimizer responses from one or two virtual steps.

Experiments across Qwen3 and Llama models test whether the analytical scales predict measured optimizer responses and evaluate downstream performance across several initializers. The results support the value of spectral information and show modest improvements over uncalibrated methods such as PiSSA and MiLoRA.

Critical strengths
  1. The paper identifies a clear problem in structured LoRA initialization and provides analysis explaining how factor scaling, spectral structure, and learning rates affect update contributions. This analysis motivates relatively simple calibration methods that preserve the initial model output. The connection between the analysis and a practical intervention makes the contribution potentially useful for LoRA initialization. The methods are also straightforward to apply at initialization, which strengthens their practical appeal.

  2. The experiments include careful controls, validation-based hyperparameter selection, and explicit uncertainty reporting. Comparisons against tuned factor learning rates and matched update magnitudes help isolate the effects of calibration, while paired runs and additional seeds help assess robustness.

Critical weaknesses
  1. Although I find the simplicity and plug-and-play nature compelling, my main concern is that the performance improvements are limited. The experiments lack a comparison with an initialization method that explicitly addresses scaling, such as LoRA-GA, which combines gradient-informed initialization with an explicit rule for scaling factor magnitudes. Although its scale-stability objective differs from the proposed relative-factor balancing, it provides a relevant practical comparison. Given the modest downstream improvements, these comparisons matter for judging whether the method offers a compelling practical advance. The related-work discussion should also cover [1], which addresses factor-update imbalance through gradient preconditioning, and [2], which addresses inter-factor scale conflicts through a single-matrix parameterization.

  2. The paper provides limited practical guidance on choosing among Spectrum-full, Forward-moment, Direct, and Warm. Their procedures and information requirements are described, and the paper acknowledges that the more elaborate variants are not consistently better. This raises the question of what distinct practical benefit justifies including these additional variants. Clearer guidance on a default choice and on when additional optimizer probes or corrections provide enough benefit to justify their use would strengthen the practical contribution.

[1] Fangzhao Zhang and Mert Pilanci. Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models. International Conference on Machine Learning (ICML), 2024. [2] David Bensaïd, Noam Rotstein, Roy Velich, Daniel Bensaïd, and Ron Kimmel. SingLoRA: Low Rank Adaptation Using a Single Matrix. arXiv preprint arXiv:2507.05566, 2025.

Questions for authors
  1. Practical significance and comparison evidence: Given the modest downstream gains, what evidence demonstrates that the proposed calibration offers a sufficiently meaningful benefit to justify its adoption over existing initialization choices? Results under comparable training and tuning budgets against methods with explicit scale selection, such as LoRA-GA, would be particularly helpful. My current assessment is weak rejection, but convincing evidence of the contribution’s practical or scientific impact could change my recommendation.

  2. Clarification of practical guidance: Which of Spectrum-full, Forward-moment, Direct, and Warm should users choose by default, and under what conditions should they choose another variant? What distinct practical benefit does each variant provide, and is there sufficient evidence to justify retaining all four?

6 comments
AuthorsOpenReview ↗

Response to Reviewer QYPt [Part I]

Hi, Reviewer QYPt:

Thank you for recognizing the clear problem and practical appeal. Your valuable suggestions motivated the LoRA-GA comparison and broader tests below. Both are now included in the revision.

Q1: "Although I find the simplicity and plug-and-play nature compelling, my main concern is that the performance improvements are limited."

R1: It improves several initializers at a small one-time cost. Users need no factor-ratio search or changes to inference. The original structured study improves 35 of 36 task cells by an average of 0.82 points. Complete Warm calibration costs 14.3 seconds beside a 57.3-minute run, or 0.42% additional time, in the reported configuration. It adds no trainable parameters and retains AdamW.

The adjustment is small, but it addresses an overlooked choice in structured initialization: how to scale the two factors after selecting their product. We have taken care to explain why this choice matters, with theory that connects the selected spectrum and factor scales to their update contributions. This analysis leads to an automatic scale rule without an extra ratio search. We have made this connection clearer in the revised introduction.

Q2: "The experiments lack a comparison with an initialization method that explicitly addresses scaling, such as LoRA-GA, which combines gradient-informed initialization with an explicit rule for scaling factor magnitudes."

R2: Thank you for this valuable suggestion. Warm improves on LoRA-GA in both focused settings, and Direct also improves LoRA-GA itself. The second comparison tests calibration on an initializer that already has a scale rule.

All methods use the same data, 2,500 training steps, five seeds, and five learning-rate candidates. LoRA-GA keeps its own scale rule, unchanged in the LoRA-GA + Direct pair. No extra scale search is used. Mean validation negative log-likelihood (NLL) selects one setting per method. We report its test scores below. Calibration is an additional initialization cost, separate from these training runs. Appendix P.1 now states both budgets.

ModelMethodGSM8KMATH-500MMLU-ProBBHAvg.
0.6BPiSSA Raw55.8822.5222.6132.9433.49
0.6BLoRA-GA56.6922.5223.0433.6933.99
0.6BLoRA-GA + Direct57.3024.0023.5334.1034.73
0.6BPiSSA + Spectrum-full56.7423.5623.1033.7334.28
0.6BPiSSA + Forward-moment56.7423.4023.0433.3034.12
0.6BPiSSA + Direct56.7123.3623.2433.4634.19
0.6BPiSSA + Warm56.8223.8023.4433.9034.49
1.7BMiLoRA Raw67.6634.5635.3342.6945.06
1.7BLoRA-GA68.1735.3635.5443.2945.59
1.7BLoRA-GA + Direct68.6436.3236.3643.7946.28
1.7BMiLoRA + Spectrum-full68.3235.3235.6043.1645.60
1.7BMiLoRA + Forward-moment68.2335.3235.9743.2745.70
1.7BMiLoRA + Direct68.2835.0835.8243.2845.62
1.7BMiLoRA + Warm68.3235.8035.8543.0845.76

Direct raises LoRA-GA from 33.99 to 34.73 and from 45.59 to 46.28 while preserving its adapter. Calibration also helps an initializer that already has a scale rule. We have highlighted this finding beside the comparison in Appendix P.4.

To test how broadly this benefit carries over, we applied the same Direct correction to the other initializers below. We use four training examples and one temporary step throughout. No initializer-specific ratio or correction count is tuned.

ModelMethodGSM8KMATH-500MMLU-ProBBHAvg.
0.6BMiLoRA55.0622.2422.4332.1832.98
0.6BMiLoRA + Direct55.9422.9622.7332.8033.61
0.6BCorDA-IPM55.2823.2022.6632.9633.53
0.6BCorDA-IPM + Direct55.7224.1223.1433.5134.12
0.6BOLoRA56.0622.7222.6133.4433.71
0.6BOLoRA + Direct56.6323.7223.1833.7134.31
1.7BPiSSA68.2935.1235.3341.8745.15
1.7BPiSSA + Direct68.8435.4035.6242.1645.51
1.7BCorDA-IPM71.7731.5634.6841.1144.78
1.7BCorDA-IPM + Direct72.6232.6435.3841.7145.59
1.7BOLoRA68.0435.1235.6142.9645.43
1.7BOLoRA + Direct68.3435.7236.1343.7845.99

Users can keep their initializer and apply the same correction. It helps factors chosen from large or small singular values, context data, gradients, and QR decomposition.

Revision locations We have clarified the practical contribution in Section 1. Section 5.4 and Appendix P.4, Table 38, present the initializer comparisons.

Please see Part II for the remaining responses.

AuthorsOpenReview ↗

Response to Reviewer QYPt [Part II]

Q3: "Which of Spectrum-full, Forward-moment, Direct, and Warm should users choose by default, and under what conditions should they choose another variant?"

R3: We agree that readers need a clear recommendation. Use Forward-moment when calibration examples are available and the nonzero columns of BB are mutually orthogonal. If this factor condition fails, use Direct, which measures the actual optimizer response. We have put this guide before the equations in Section 4. The results below show what the low-cost forward pass provides.

For convenience, we reproduce the relevant task results from the paper's information comparison below. Scores are five-seed means in percent.

ConfigurationMethodGSM8KMATH-500MMLU-ProBBHAvg.
PiSSA/0.6BRaw55.8822.5222.6132.9433.49
PiSSA/0.6BSpectrum-full56.7423.5623.1033.7334.28
PiSSA/0.6BForward-moment56.7423.4023.0433.3034.12
PiSSA/0.6BDirect56.7123.3623.2433.4634.19
MiLoRA/1.7BRaw67.6634.5635.3342.6945.06
MiLoRA/1.7BSpectrum-full68.3235.3235.6043.1645.60
MiLoRA/1.7BForward-moment68.2335.3235.9743.2745.70
MiLoRA/1.7BDirect68.2835.0835.8243.2845.62
CorDA/3BRaw31.379.1622.2334.2924.26
CorDA/3BCovariance-aware31.529.6822.4234.4124.51
CorDA/3BForward-moment31.9810.2823.0034.9225.04
CorDA/3BDirect32.3710.6822.9334.7525.18

Forward-moment gives 34.12/45.70/25.04, compared with Raw's 33.49/45.06/24.26. On CorDA it also improves on the covariance reference's 24.51. Measured activation statistics therefore give useful scales without a backward pass.

ChoiceInformation usedRecommended use
Spectrum-fullSelected singular valuesUse with spectral initializers such as PiSSA and MiLoRA without a calibration pass.
Forward-momentCalibration activationsDefault with mutually orthogonal nonzero columns of BB.
DirectOne temporary optimizer stepMeasure the response to the actual training loss and optimizer.
WarmInitial scale estimate and two correctionsOptional refinement when task accuracy is the priority.

Warm is an optional accuracy-focused choice when two temporary optimizer steps are affordable. In Table 1, it raises Direct's averages from 34.19/45.62 to 34.49/45.76. The reported complete Warm calibration takes 14.3 seconds beside a 57.3-minute run, or 0.42% extra time. This gives a concrete reason to retain it: users can spend a small one-time cost on a further measured correction. The recommendation is based on this practical tradeoff. Under the ideal scaling law, one correction already suffices.

We have organized Section 4 around two decisions. For spectral initializers such as PiSSA and MiLoRA, use Spectrum-full when calibration examples are unavailable. With examples, start from Forward-moment when its factor condition holds, or Direct otherwise. Warm remains an optional refinement for users who prioritize task accuracy. This makes the choices useful without asking users to search over four variants.

Q4: "The related-work discussion should also cover [1], which addresses factor-update imbalance through gradient preconditioning, and [2], which addresses inter-factor scale conflicts through a single-matrix parameterization."

R4: Thank you for recommending these papers. We have included both and explained their differences. Riemannian Preconditioned LoRA changes gradient updates during training. SingLoRA represents the adapter with one matrix. We choose initial factor ratios from singular values or measured responses. Users keep their initializer and AdamW training.

The spectrum and activation statistics change the ratio even at fixed shape and rank. Our theory gives this relationship and uses it to compute the scale. Section 6 now distinguishes changes to training updates, the adapter representation, and initial factor scales.

Revision locations We have added the selection guide and Warm's practical benefit in Section 4, and both related-work comparisons in Section 6.

Thank you again for your valuable feedback, which has greatly improved our work. We hope these responses address your concerns. Please let us know of any further questions.

Dear Reviewer QYPt,

Thank you again for your thoughtful and constructive review. We wanted to kindly follow up before the discussion period concludes.

Your concerns about practical significance and comparisons with scale-aware initialization motivated substantial additional experiments. In particular, we added matched-budget comparisons with LoRA-GA and showed that our calibration also improves LoRA-GA itself. We have also clarified the recommended default method, its computational cost, and the distinct roles of the calibration variants.

We would greatly appreciate your thoughts on whether these new results and clarifications address your main concerns. We would be happy to clarify any remaining issues.

Thank you again for your time and consideration.

Reviewer_UP3QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes that structured initialization determines the type of adapter selected by LoRA at the beginning of training, but does not determine how much of the subsequent update is undertaken by the two low-rank factors. This may lead to an imbalance in the updates. To address this issue, the authors propose three factor-scale calibration methods: Spectrum-full, Forward-moment, and InitCal. The experiments verify whether the singular value spectrum can predict the scale, compare different scale selection methods, and evaluate their performance on downstream tasks.

Critical strengths

1.The authors identify a meaningful problem by distinguishing between the selection of the initial adapter and the allocation of scales between the two low rank factors. 2.They propose three factor scale calibration methods based on different sources of information. These methods form a gradual progression and provide a relatively complete set of solutions 3. The authors propose three factor scale calibration methods based on different sources of information. These methods form a gradual progression and provide a relatively complete set of solutions.

Critical weaknesses

1.The authors state that reciprocal rescaling and factor specific learning rates can be equivalent under idealized conditions. If InitCal ultimately only obtains a factor specific learning rate, why is it necessary to modify the initialization factors? 2.The paper does not clearly explain whether the proposed method is a new parameterization theory, or a combination of existing ideas applied to structured LoRA. 3.The scale of the experiments is limited to models smaller than 4B parameters. No experiments are conducted on larger models, so it is not possible to verify whether the method remains effective for models with a larger number of parameters.

Questions for authors
  1. If reciprocal rescaling and factor-specific learning rates can be equivalent under idealized conditions, what specific update benefit does InitCal provide?
  2. Can the improvement observed after calibration also be achieved on larger models, such as 8B parameter models? This should be verified. 3.It would be better to include additional nonzero initialization methods in the experiments for comparison.
3 comments
AuthorsOpenReview ↗

Response to Reviewer UP3Q [Part I]

Hi, Reviewer UP3Q:

Thank you for your thoughtful questions and recognition of the distinction between adapter selection and factor scaling.

Q1: "If InitCal ultimately only obtains a factor specific learning rate, why is it necessary to modify the initialization factors?"

R1: We automatically determine each layer's factor ratio, while LoRA+ leaves the learning-rate multiplier to tune. We use singular values, activations, or an optimizer response. Users tune the common learning rate without a ratio search.

We search the global learning-rate ratio ρ=ηB/ηA\rho=\eta_B/\eta_A, shared across layers, using ηA=η/ρ\eta_A=\eta/\sqrt{\rho} and ηB=ηρ\eta_B=\eta\sqrt{\rho}. The three ratio sets are {16}\{16\}, {1/4,1,16}\{1/4,1,16\}, and {1/16,1/4,1,4,16}\{1/16,1/4,1,4,16\}. Each ratio is paired with five common rates η∈{3,5,8,13,20}×10−5\eta\in\{3,5,8,13,20\}\times10^{-5}, giving 5, 15, or 25 training candidates per seed. Appendix P.2 specifies these grids; Appendix G defines the separate five-pair control.

Mean validation negative log-likelihood (NLL) over five seeds selects one setting. We report its test scores for PiSSA/Qwen3-0.6B and MiLoRA/Qwen3-1.7B below.

ModelRatio choiceCandidatesGSM8KMATH-500MMLU-ProBBHAvg.
0.6BFixed ratio 16556.3823.1223.1033.3133.98
0.6B3 ratios1556.4123.3623.2933.4034.11
0.6B5 ratios2556.5723.4423.0833.6734.19
0.6BDirect556.7123.3623.2433.4634.19
0.6BWarm556.8223.8023.4433.9034.49
1.7BFixed ratio 16567.6034.6435.2843.1845.17
1.7B3 ratios1568.2035.4435.0243.1845.46
1.7B5 ratios2568.0735.2835.6643.1045.53
1.7BDirect568.2835.0835.8243.2845.62
1.7BWarm568.3235.8035.8543.0845.76

Direct matches or exceeds the full grid's average with 5 rather than 25 training candidates. It removes 80% of the search runs, and Warm further improves accuracy.

Users can apply the chosen ratios in two ways. Table 1's Warm-mapped learning rates apply the chosen ratios without changing the factors or searching again. They give 34.49/45.77, close to Warm's 34.49/45.76. Appendix L gives the cumulative-scale mapping. The contribution is to choose the ratios automatically. Rescaling is one way to apply them.

Q2: "The paper does not clearly explain whether the proposed method is a new parameterization theory, or a combination of existing ideas applied to structured LoRA."

R2: We derive a new rule that uses the selected singular values to choose the factor scale. Different selected singular values can require different scales, even at the same layer shape and rank. For input/output sizes din=48d_{\mathrm{in}}=48, dout=96d_{\mathrm{out}}=96, and rank r=8r=8, the scale is about 0.771 for a flat spectrum. A concentrated spectrum gives about 0.579. That is a difference of about 25%. A flat spectrum has equal values. A concentrated one has dominant values.

Appendix I, Table 11, gives the predictions and observations below. Observations are mean ±\pm SD across three seeds, each using 16,384 Gaussian samples.

Input/output sizeSpectrumPredicted scaleObserved scale
48/96Flat0.77080.7715±0.00110.7715\pm 0.0011
48/96Concentrated0.57890.5799±0.00100.5799\pm 0.0010
96/48Flat0.45900.4591±0.00040.4591\pm 0.0004
96/48Concentrated0.34470.3449±0.00100.3449\pm 0.0010

The Gaussian single-sample SignSGD rule depends on qσ=(∑iσi)2/∑iσiq_{\sigma}=(\sum_i\sqrt{\sigma_i})^2/\sum_i\sigma_i. Here, σi\sigma_i are the selected singular values. Across 60 square and rectangular configurations, the median error is 0.11% and the maximum is 0.51%. We do not fit the predictions to the observations.

Figure 2 and Table 21 test AdamW on Qwen3-0.6B. Below is the five-seed mean of median absolute log⁡2\log_2 scale error across 28 layers. Lower is better.

InitializerWidth-rank ruleSpectrum-aware rule
PiSSA0.2180.156
MiLoRA0.2420.190

We wanted to understand why such a small adjustment helps. Our analysis connects the selected spectrum and factor scales to their update contributions. The analysis explains why selected spectra matter even at the same layer shape and rank. The numerical checks agree with the prediction, and the spectrum-aware rule reduces error under AdamW. The theory also separates the factor ratio, total update size, and interaction between updates. This supports the simple operation B←cBB\leftarrow cB, A←c−1AA\leftarrow c^{-1}A, which preserves the adapter.

Revision locations We have clarified the contribution in Sections 1 and 3.2 and added the search comparison in Section 5.3, Table 2; Appendix P.2 gives the grids.

Please see Part II for the remaining responses.

AuthorsOpenReview ↗

Response to Reviewer UP3Q [Part II]

Q3: "Can the improvement observed after calibration also be achieved on larger models, such as 8B parameter models?"

R3: Yes. Warm raises PiSSA's average from 66.30 to 67.52 and MiLoRA's from 66.13 to 67.31 on Qwen3-8B. Forward-moment and Direct improve both initializers as well.

Each Raw/calibrated pair uses the same initial adapter and training data, at rank 64. We select one of five learning rates by mean validation NLL, then report its five-seed task scores in percent.

Qwen3-8B methodGSM8KMATH-500MMLU-ProBBHAvg.
PiSSA Raw88.2351.6059.3366.0366.30
PiSSA Forward-moment89.1652.6860.0866.7067.15
PiSSA Direct89.9652.1259.8567.0167.24
PiSSA Warm89.8052.5660.4067.3467.52
MiLoRA Raw87.8251.7659.1265.8266.13
MiLoRA Forward-moment89.0252.8859.4566.8167.04
MiLoRA Direct88.4852.7659.4166.3766.75
MiLoRA Warm88.8653.1260.0767.2067.31

The improvement extends to both initializers at 8B, across all four tasks. Forward-moment already improves the averages to 67.15/67.04 with one forward pass. Warm reaches 67.52/67.31. We have added this table in Appendix P.3 and its main finding in Section 5.4.

Q4: "It would be better to include additional nonzero initialization methods in the experiments for comparison."

R4: We especially appreciate this suggestion. The same Direct correction improves LoRA-GA and OLoRA. The added comparisons show that it also helps initializers built from gradients or QR decomposition. The method therefore extends beyond the spectral initializers that motivated our analysis.

We use the same Direct rule for both initializers: four training examples and one temporary AdamW step, with no initializer-specific ratio search. Each pair shares its initial adapter and training setup. As above, validation NLL selects the learning rate, and we report five-seed task means.

ModelMethodGSM8KMATH-500MMLU-ProBBHAvg.
0.6BLoRA-GA56.6922.5223.0433.6933.99
0.6BLoRA-GA + Direct57.3024.0023.5334.1034.73
0.6BOLoRA56.0622.7222.6133.4433.71
0.6BOLoRA + Direct56.6323.7223.1833.7134.31
1.7BLoRA-GA68.1735.3635.5443.2945.59
1.7BLoRA-GA + Direct68.6436.3236.3643.7946.28
1.7BOLoRA68.0435.1235.6142.9645.43
1.7BOLoRA + Direct68.3435.7236.1343.7845.99

LoRA-GA already includes a scale rule. Direct raises its averages from 33.99 to 34.73 and from 45.59 to 46.28. It also raises OLoRA from 33.71 to 34.31 and from 45.43 to 45.99. The correction preserves each initializer's adapter product, selected directions, and frozen residual.

The same correction therefore works with PiSSA, MiLoRA, CorDA-IPM, LoRA-GA, and OLoRA. The original study uses three seeds on Qwen3-1.7B/4B and Llama-3.2-3B. All nine model-initializer averages and 35 of 36 task cells improve. The average gain is 0.82 points. The added experiments test an 8B model and two further initializers.

Calibration therefore complements how the adapter is chosen, including when the initializer already uses a scale rule. Revision locations We have added the results in Section 5.4 and Appendix P.3–P.4, Tables 37–38. Section 4.3 explains the common correction.

Thank you again for your valuable feedback, which has greatly improved our work. We hope these responses address your concerns. Please let us know of any further questions.

Dear Reviewer UP3Q,

Thank you again for your constructive feedback. We wanted to kindly follow up as the discussion period approaches its end.

In response to your concerns, we have added Qwen3-8B experiments, comparisons with LoRA-GA and OLoRA, and a direct comparison with factor-specific learning-rate tuning. We have also clarified our theoretical contribution and provided additional evidence showing how the selected singular-value spectrum determines the balancing scale.

We would be very grateful if you could take another look at our responses and revised manuscript. We would especially appreciate knowing whether these additions have addressed your main concerns, or if any important questions remain.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.