- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SPECTRA, a method for predicting a stable, near-optimal learning rate for LoRA fine-tuning from the initialization-time gradient spectrum and the adapter rank. It fits shared predictor coefficients on a reference model and uses one anchor configuration to calibrate a new model. The paper also develops a local stability analysis based on a gradient-aligned, preconditioned weight-space surrogate.
Critical strengths
- The paper addresses a practical and well-motivated problem. Learning-rate sweeps can substantially erode the efficiency benefit of LoRA, and predicting a rate for a particular task and rank is a useful goal.
- The work is thorough. It combines a concrete predictor, theoretical analysis, experiments across several model families and datasets, ablations of the rank and spectral terms (Table 8), and training runs around the predicted rates (Figure 3).
- I appreciate that the authors test several LoRA scaling conventions (Table 3 and Appendices J–L) and examine a setting in which prediction accuracy degrades. This makes the empirical account more informative than a result under a single default configuration.
Critical weaknesses
-
Relationship to TML and practical guidance. LoRA Without Regret proposes the fitted rule . It also reports that, under , LoRA’s optimal learning rate is roughly the full-fine-tuning rate. When introducing SPECTRA’s predictor after Algorithm 1, the authors should cite and compare this formula directly with . Table 2 shows improved prediction accuracy, but what practical guidance does the spectral term add? In particular, when and by how much does it adjust the TML recommendation across tasks and ranks?
-
Connection to existing Hessian analyses. The Hidden Power of Scaling Factor in LoRA Optimization (NeurIPS 2026) analyzes the LoRA parameter-space Hessian, separating task-signal curvature from the term induced by the bilinear adapter parameterization. It also studies how affects curvature and the stable learning-rate regime. This is directly relevant to SPECTRA’s Section 4. SPECTRA derives a stability boundary for a gradient-aligned, preconditioned weight-space surrogate and transfers it to the LoRA factor learning rate under Assumption 4.1. The authors should explain when that surrogate captures the curvature governing actual LoRA updates, and when the bilinear term could alter the predicted boundary.
-
Comparison with LoRA-α. The same prior work proposes LoRA-α, a simple method intended to improve LoRA optimization while reducing hyperparameter search. Although Table 3 evaluates several scaling conventions, it does not evaluate LoRA-α’s proposed default scaling and learning-rate prescription. I encourage a comparison using matched models, datasets, ranks, training budgets, and evaluation protocols. Because the methods address different parts of hyperparameter selection, the most informative outcomes would be final performance and total calibration or tuning cost, rather than learning-rate prediction accuracy alone. It would also be useful to test whether SPECTRA provides an additional benefit when applied on top of LoRA-α, stating any recalibration this requires.
-
Related work and cost accounting. The Primacy of Magnitude in Low-Rank Adaptation (NeurIPS 2025 Spotlight) connects learning rate, scaling, and initialization through weight-update magnitude. The authors should distinguish this from Proposition 4.5’s idealized invariance to global gradient scaling; the two statements concern different quantities and need not conflict. On the Convergence Rate of LoRA Gradient Descent (ICML 2026 Oral) provides complementary context for the step-size analysis, while studying a different optimizer and theoretical setting. Finally, the reported search-time comparison would be easier to interpret with a clear accounting of reference-model sweeps, target-model anchor calibration, and the number of later tasks over which those costs are amortized.
Questions for authors
Overall assessment. I lean toward weak acceptance. The problem is useful, the predictor is concrete, and the experiments provide substantial evidence of its effectiveness in the tested settings.
Attribution of the theoretical results. I encourage the authors to discuss the closely related work at the relevant propositions. The curvature-based stability argument underlying Proposition 4.3 should be discussed alongside Proposition 3 and Appendix Lemma 1 of LoRA-α. Proposition 4.5 should likewise cite and distinguish the scaling-equivalence analyses in The Primacy of Magnitude in Low-Rank Adaptation and TML’s LoRA Without Regret. These works analyze broader transformations involving LoRA scaling, initialization, and factor learning rates. Clarifying what follows from prior results and what SPECTRA adds would strengthen the paper’s theoretical positioning.
Additional experiments. I would be interested in a comparison with LoRA-α under matched settings, including final performance and tuning cost, and in whether SPECTRA provides an additional benefit when combined with LoRA-α.
6 comments
Reviewer's Comment: Comparison with LoRA-α.// Additional experiments.
Response: We thank the reviewer for this valuable suggestion. LoRA-α and SPECTRA address complementary aspects of LoRA optimization. LoRA-α adjusts the scaling factor to make LoRA compatible with the smaller learning rates commonly used in full-parameter fine-tuning. However, two practical challenges remain: (i) obtaining a suitable full-fine-tuning learning rate through empirical search can be computationally expensive, and (ii) even within this smaller learning-rate regime, selecting an appropriate rate for a specific model and dataset remains necessary. Moreover, practitioners often prefer user-defined scaling factors rather than adopting a particular scaling prescription. SPECTRA addresses this remaining learning-rate selection problem by predicting a stable, high-performing learning rate under the chosen scaling convention.
Regarding final performance, we agree that this is a useful complementary evaluation, but it does not directly measure learning-rate prediction quality. SPECTRA aims to identify stable, near-optimal learning rates without exhaustive per-configuration search, rather than guarantee the exact global optimum. Therefore, final performance should be considered alongside learning-rate prediction accuracy and total tuning cost.
Combining SPECTRA with LoRA-α offers a promising direction for further reducing calibration cost. If LoRA-α provides a reliable initial learning-rate range, SPECTRA could exploit this narrower range to accelerate model-specific anchor calibration while maintaining comparable prediction accuracy. However, this reduction is not automatic: SPECTRA currently calibrates against empirically near-optimal learning rates, and a narrower search range only reduces cost if fewer training evaluations are required.
Reviewer's Comment: Related work and cost accounting.
Response: We thank the reviewer for highlighting these complementary studies and the need for clearer cost accounting.
Relationship to prior work. The Primacy of Magnitude in Low-Rank Adaptation studies how learning rate, scaling, and initialization influence effective weight-update magnitudes. SPECTRA also investigates the interaction between learning rate η and scaling factor α (Table 3 and Appendix L), but further leverages initialization-time gradient spectral information to predict task- and rank-specific learning rates under Adam. Proposition 4.5 establishes Adam’s invariance to global gradient rescaling in the idealized limit (ε→0), motivating our focus on spectral structure rather than gradient magnitude alone.
Importantly, On the Convergence Rate of LoRA Gradient Descent also provides theoretical guidance for learning-rate selection, deriving convergence conditions dependent on adapter parameter and gradient norms and proposing adaptive learning rates that improve convergence and stability. However, its analysis concerns non-asymptotic convergence under gradient descent and dynamically adjusts learning rates during training, whereas SPECTRA characterizes a local stability boundary under Adam-preconditioned dynamics and predicts learning rates across tasks and ranks before fine-tuning. The GD convergence guarantees do not directly extend to Adam. Our contributions are therefore complementary to these studies, and we have revised the Related Work section to clarify these distinctions and appropriately acknowledge their contributions.
Cost accounting and amortization. We clarify that the reported 4-hour cost includes the model-specific calibration sweeps for all evaluated models but excludes the one-time reference-model calibration sweeps. Each new model requires calibration across multiple LoRA ranks on only one dataset, after which SPECTRA predicts learning rates for all remaining datasets and ranks without additional training sweeps, achieving 88.5% [email protected]× and recovering 89% of the 90-hour grid-search baseline's coverage. Importantly, the reference-model calibration is a shared, one-time expense that can be amortized across subsequent models and datasets, making SPECTRA particularly valuable for large-scale fine-tuning infrastructure. We revise the manuscript to explicitly distinguish the one-time reference-model calibration cost from the reported 4-hour target-model calibration cost, and clarify how the total cost is amortized as the number of models and datasets increases.
Reviewer's Comment: Attribution of the theoretical results
Response: We thank the reviewer for identifying these important theoretical connections. We agree that the curvature-based stability argument and LoRA scaling equivalences have been investigated in prior work, and we will explicitly acknowledge these foundations at the relevant propositions.
Regarding Proposition 4.3, LoRA-α analyzes curvature suppression under low-rank parameterization and its implications for the stable learning-rate range. Our analysis builds on this established curvature–stability connection rather than claiming it as a new result. SPECTRA's contribution is to connect task-specific gradient spectral decay to an empirically calibrated learning-rate predictor, enabling predictions that adapt to gradient characteristics and LoRA rank under the stated local curvature assumptions.
Regarding Proposition 4.5, The Primacy of Magnitude in Low-Rank Adaptation and TML's LoRA Without Regret establish broader parameterization equivalences involving scaling factors, initialization, and factor-specific learning rates. We acknowledge that the underlying scaling transformations are not unique to SPECTRA. Our contribution is to incorporate a restricted scaling-transfer relationship into the spectral learning-rate predictor, rather than to establish a new general equivalence of LoRA optimization dynamics. Consistent with this distinction, our Appendix M experiments recover rank-coefficient shifts of 0.23 and 0.47, close to the predicted 0.25 and 0.50 under the balanced two-factor regime. These observations support the approximation in the evaluated settings without establishing equivalence of the full optimization dynamics.
We cite and distinguish these works directly alongside Propositions 4.3 and 4.5, clarify which arguments follow from established results, and emphasize that SPECTRA's novelty lies in the spectral predictor and its empirical calibration rather than the underlying curvature or reparameterization identities.
Response: We thank the reviewer for this suggestion and explicitly compare SPECTRA with TML's fitted formula after Algorithm 1. While TML provides valuable guidance based on model family and hidden size, its formula does not explicitly account for task-specific gradient characteristics, LoRA rank, or alternative scaling conventions. Moreover, applying TML's empirical 10× FullFT learning-rate heuristic requires knowing the optimal FullFT learning rate, which may itself demand prohibitively expensive full-parameter sweeps; although TML's separately fitted formula avoids this requirement, its empirical calibration primarily covers Llama and Qwen models with α=32.
SPECTRA instead provides task- and configuration-specific predictions without FullFT sweeps: at fixed rank, a change in spectral decay Δβ adjusts the predicted learning rate by a factor of 10−bβΔβ, while rank dependence contributes a factor of (r2/r1)−c. Thus, faster spectral decay leads to a lower predicted learning rate, with the adjustment magnitude determined by calibrated coefficients. Empirically, SPECTRA achieves 88.5% [email protected]× versus 65.3% for the TML formula (Table 2). Importantly, α is not necessarily fixed at 32 in practical LoRA applications; across four scaling conventions, SPECTRA achieves 84.7% versus 39.5% for the evaluated TML formula (Table 3), demonstrating the importance of adapting learning-rate predictions to task characteristics and LoRA configurations.
Response: We thank the reviewer for highlighting this important connection. We agree that The Hidden Power of Scaling Factor in LoRA Optimization provides a complementary parameter-space analysis, demonstrating that the scaling factor α and learning rate η are not interchangeable. Specifically, its Signal–Drift decomposition distinguishes task-aligned curvature from bilinear parameterization-induced curvature, showing that increasing α can enhance the task signal without proportionally increasing drift, whereas increasing η amplifies both signal and second-order drift.
SPECTRA addresses a different but complementary question: predicting task- and rank-specific learning rates through a gradient-aligned, preconditioned weight-space surrogate. We acknowledge that this surrogate does not explicitly capture bilinear drift curvature, which may alter the actual stability boundary, particularly when nonlinear factor interactions become significant. Accordingly, Assumption 4.1 provides an approximate scaling transfer rather than an exact equivalence.
Importantly, Appendix M provides independent empirical support for this transfer assumption. Our β-free paired experiments recover rank-coefficient shifts of 0.23 and 0.47, closely matching the theoretical predictions of 0.25 and 0.50 under the balanced two-factor regime. These results support the proposed scaling relationship in the evaluated settings, although they do not establish equivalence of the full Hessians. We incorporate the cited Signal–Drift analysis and clarify the conditions and limitations of our stability approximation in Appendix P.