acceptodds
Under review as a conference paper at ICLR 2027

SPECTRA: Spectral Stable Learning Rate Prediction for Low-rank Adaptation

15 views

Abstract

We propose **SPECTRA**, a gradient-spectral framework for predicting stable learning rates for LoRA fine-tuning. Although LoRA substantially reduces the cost of full-parameter SFT, its performance remains highly sensitive to learning-rate selection. Existing studies offer valuable scaling laws, heuristics, and empirical insights, but they do not directly predict an appropriate learning rate for a specific task and LoRA configuration, leaving practitioners reliant on costly hyperparameter sweeps. **Methodologically**, SPECTRA uses a calibrated log-linear predictor to map the initialization-time gradient spectral decay and LoRA rank to a stable, near-optimal learning rate. **Theoretically**, we derive the local stability threshold for rank-constrained optimization, identify the most fragile module as the shared-rate bottleneck, and establish Adam’s gradient-scale invariance in the idealized limit. Under a local monotonic relationship between spectral decay and curvature, faster spectral decay reduces the stability threshold. **Experimentally**, SPECTRA achieves an **88.5%** success rate, outperforming all sweep-free baselines by over **9.7** points and recovering **89%** of the coverage of an exhaustive per-dataset grid search at **4%** of its time cost.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LoRA, Learning Rate, Stable Training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_w3a6OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SPECTRA, a method for predicting a stable, near-optimal learning rate for LoRA fine-tuning from the initialization-time gradient spectrum and the adapter rank. It fits shared predictor coefficients on a reference model and uses one anchor configuration to calibrate a new model. The paper also develops a local stability analysis based on a gradient-aligned, preconditioned weight-space surrogate.

Critical strengths
  1. The paper addresses a practical and well-motivated problem. Learning-rate sweeps can substantially erode the efficiency benefit of LoRA, and predicting a rate for a particular task and rank is a useful goal.
  2. The work is thorough. It combines a concrete predictor, theoretical analysis, experiments across several model families and datasets, ablations of the rank and spectral terms (Table 8), and training runs around the predicted rates (Figure 3).
  3. I appreciate that the authors test several LoRA scaling conventions (Table 3 and Appendices J–L) and examine a setting in which prediction accuracy degrades. This makes the empirical account more informative than a result under a single default configuration.
Critical weaknesses
  1. Relationship to TML and practical guidance. LoRA Without Regret proposes the fitted rule ηTML=MLoRA(2000/dhidden)pmodel+pLoRA\eta_{\mathrm{TML}}=M_{\mathrm{LoRA}}(2000/d_{\mathrm{hidden}})^{p_{\mathrm{model}}+p_{\mathrm{LoRA}}}. It also reports that, under α=32\alpha=32, LoRA’s optimal learning rate is roughly 10×10\times the full-fine-tuning rate. When introducing SPECTRA’s predictor after Algorithm 1, the authors should cite and compare this formula directly with log⁡10η^=a0(M)−clog⁡10r−bββ(D;r)\log_{10}\hat{\eta}=a_0(M)-c\log_{10}r-b_\beta\beta(D;r). Table 2 shows improved prediction accuracy, but what practical guidance does the spectral term add? In particular, when and by how much does it adjust the TML recommendation across tasks and ranks?

  2. Connection to existing Hessian analyses. The Hidden Power of Scaling Factor in LoRA Optimization (NeurIPS 2026) analyzes the LoRA parameter-space Hessian, separating task-signal curvature from the term induced by the bilinear adapter parameterization. It also studies how α\alpha affects curvature and the stable learning-rate regime. This is directly relevant to SPECTRA’s Section 4. SPECTRA derives a stability boundary for a gradient-aligned, preconditioned weight-space surrogate and transfers it to the LoRA factor learning rate under Assumption 4.1. The authors should explain when that surrogate captures the curvature governing actual LoRA updates, and when the bilinear term could alter the predicted boundary.

  3. Comparison with LoRA-α. The same prior work proposes LoRA-α, a simple method intended to improve LoRA optimization while reducing hyperparameter search. Although Table 3 evaluates several scaling conventions, it does not evaluate LoRA-α’s proposed default scaling and learning-rate prescription. I encourage a comparison using matched models, datasets, ranks, training budgets, and evaluation protocols. Because the methods address different parts of hyperparameter selection, the most informative outcomes would be final performance and total calibration or tuning cost, rather than learning-rate prediction accuracy alone. It would also be useful to test whether SPECTRA provides an additional benefit when applied on top of LoRA-α, stating any recalibration this requires.

  4. Related work and cost accounting. The Primacy of Magnitude in Low-Rank Adaptation (NeurIPS 2025 Spotlight) connects learning rate, scaling, and initialization through weight-update magnitude. The authors should distinguish this from Proposition 4.5’s idealized invariance to global gradient scaling; the two statements concern different quantities and need not conflict. On the Convergence Rate of LoRA Gradient Descent (ICML 2026 Oral) provides complementary context for the step-size analysis, while studying a different optimizer and theoretical setting. Finally, the reported search-time comparison would be easier to interpret with a clear accounting of reference-model sweeps, target-model anchor calibration, and the number of later tasks over which those costs are amortized.

Questions for authors

Overall assessment. I lean toward weak acceptance. The problem is useful, the predictor is concrete, and the experiments provide substantial evidence of its effectiveness in the tested settings.

Attribution of the theoretical results. I encourage the authors to discuss the closely related work at the relevant propositions. The curvature-based stability argument underlying Proposition 4.3 should be discussed alongside Proposition 3 and Appendix Lemma 1 of LoRA-α. Proposition 4.5 should likewise cite and distinguish the scaling-equivalence analyses in The Primacy of Magnitude in Low-Rank Adaptation and TML’s LoRA Without Regret. These works analyze broader transformations involving LoRA scaling, initialization, and factor learning rates. Clarifying what follows from prior results and what SPECTRA adds would strengthen the paper’s theoretical positioning.

Additional experiments. I would be interested in a comparison with LoRA-α under matched settings, including final performance and tuning cost, and in whether SPECTRA provides an additional benefit when combined with LoRA-α.

6 comments

Reviewer's Comment: Relationship to TML and practical guidance.[]

Response: We thank the reviewer for this suggestion and explicitly compare SPECTRA with TML's fitted formula after Algorithm 1. While TML provides valuable guidance based on model family and hidden size, its formula does not explicitly account for task-specific gradient characteristics, LoRA rank, or alternative scaling conventions. Moreover, applying TML's empirical 10× FullFT learning-rate heuristic requires knowing the optimal FullFT learning rate, which may itself demand prohibitively expensive full-parameter sweeps; although TML's separately fitted formula avoids this requirement, its empirical calibration primarily covers Llama and Qwen models with α=32.

SPECTRA instead provides task- and configuration-specific predictions without FullFT sweeps: at fixed rank, a change in spectral decay ΔβΔβ adjusts the predicted learning rate by a factor of 10−bβΔβ10^{-b_\beta\Delta\beta}, while rank dependence contributes a factor of (r2/r1)−c(r_2/r_1)^{-c}. Thus, faster spectral decay leads to a lower predicted learning rate, with the adjustment magnitude determined by calibrated coefficients. Empirically, SPECTRA achieves 88.5% [email protected]× versus 65.3% for the TML formula (Table 2). Importantly, α is not necessarily fixed at 32 in practical LoRA applications; across four scaling conventions, SPECTRA achieves 84.7% versus 39.5% for the evaluated TML formula (Table 3), demonstrating the importance of adapting learning-rate predictions to task characteristics and LoRA configurations.

Reviewer's Comment: Connection to existing Hessian analyses.

Response: We thank the reviewer for highlighting this important connection. We agree that The Hidden Power of Scaling Factor in LoRA Optimization provides a complementary parameter-space analysis, demonstrating that the scaling factor α and learning rate η are not interchangeable. Specifically, its Signal–Drift decomposition distinguishes task-aligned curvature from bilinear parameterization-induced curvature, showing that increasing α can enhance the task signal without proportionally increasing drift, whereas increasing η amplifies both signal and second-order drift.

SPECTRA addresses a different but complementary question: predicting task- and rank-specific learning rates through a gradient-aligned, preconditioned weight-space surrogate. We acknowledge that this surrogate does not explicitly capture bilinear drift curvature, which may alter the actual stability boundary, particularly when nonlinear factor interactions become significant. Accordingly, Assumption 4.1 provides an approximate scaling transfer rather than an exact equivalence.

Importantly, Appendix M provides independent empirical support for this transfer assumption. Our β-free paired experiments recover rank-coefficient shifts of 0.23 and 0.47, closely matching the theoretical predictions of 0.25 and 0.50 under the balanced two-factor regime. These results support the proposed scaling relationship in the evaluated settings, although they do not establish equivalence of the full Hessians. We incorporate the cited Signal–Drift analysis and clarify the conditions and limitations of our stability approximation in Appendix P.

Reviewer's Comment: Comparison with LoRA-α.// Additional experiments.

Response: We thank the reviewer for this valuable suggestion. LoRA-α and SPECTRA address complementary aspects of LoRA optimization. LoRA-α adjusts the scaling factor to make LoRA compatible with the smaller learning rates commonly used in full-parameter fine-tuning. However, two practical challenges remain: (i) obtaining a suitable full-fine-tuning learning rate through empirical search can be computationally expensive, and (ii) even within this smaller learning-rate regime, selecting an appropriate rate for a specific model and dataset remains necessary. Moreover, practitioners often prefer user-defined scaling factors rather than adopting a particular scaling prescription. SPECTRA addresses this remaining learning-rate selection problem by predicting a stable, high-performing learning rate under the chosen scaling convention.

Regarding final performance, we agree that this is a useful complementary evaluation, but it does not directly measure learning-rate prediction quality. SPECTRA aims to identify stable, near-optimal learning rates without exhaustive per-configuration search, rather than guarantee the exact global optimum. Therefore, final performance should be considered alongside learning-rate prediction accuracy and total tuning cost.

Combining SPECTRA with LoRA-α offers a promising direction for further reducing calibration cost. If LoRA-α provides a reliable initial learning-rate range, SPECTRA could exploit this narrower range to accelerate model-specific anchor calibration while maintaining comparable prediction accuracy. However, this reduction is not automatic: SPECTRA currently calibrates against empirically near-optimal learning rates, and a narrower search range only reduces cost if fewer training evaluations are required.

Reviewer's Comment: Related work and cost accounting.

Response: We thank the reviewer for highlighting these complementary studies and the need for clearer cost accounting.

Relationship to prior work. The Primacy of Magnitude in Low-Rank Adaptation studies how learning rate, scaling, and initialization influence effective weight-update magnitudes. SPECTRA also investigates the interaction between learning rate η and scaling factor α (Table 3 and Appendix L), but further leverages initialization-time gradient spectral information to predict task- and rank-specific learning rates under Adam. Proposition 4.5 establishes Adam’s invariance to global gradient rescaling in the idealized limit (ε→0), motivating our focus on spectral structure rather than gradient magnitude alone.

Importantly, On the Convergence Rate of LoRA Gradient Descent also provides theoretical guidance for learning-rate selection, deriving convergence conditions dependent on adapter parameter and gradient norms and proposing adaptive learning rates that improve convergence and stability. However, its analysis concerns non-asymptotic convergence under gradient descent and dynamically adjusts learning rates during training, whereas SPECTRA characterizes a local stability boundary under Adam-preconditioned dynamics and predicts learning rates across tasks and ranks before fine-tuning. The GD convergence guarantees do not directly extend to Adam. Our contributions are therefore complementary to these studies, and we have revised the Related Work section to clarify these distinctions and appropriately acknowledge their contributions.

Cost accounting and amortization. We clarify that the reported 4-hour cost includes the model-specific calibration sweeps for all evaluated models but excludes the one-time reference-model calibration sweeps. Each new model requires calibration across multiple LoRA ranks on only one dataset, after which SPECTRA predicts learning rates for all remaining datasets and ranks without additional training sweeps, achieving 88.5% [email protected]× and recovering 89% of the 90-hour grid-search baseline's coverage. Importantly, the reference-model calibration is a shared, one-time expense that can be amortized across subsequent models and datasets, making SPECTRA particularly valuable for large-scale fine-tuning infrastructure. We revise the manuscript to explicitly distinguish the one-time reference-model calibration cost from the reported 4-hour target-model calibration cost, and clarify how the total cost is amortized as the number of models and datasets increases.

Reviewer's Comment: Attribution of the theoretical results

Response: We thank the reviewer for identifying these important theoretical connections. We agree that the curvature-based stability argument and LoRA scaling equivalences have been investigated in prior work, and we will explicitly acknowledge these foundations at the relevant propositions.

Regarding Proposition 4.3, LoRA-α analyzes curvature suppression under low-rank parameterization and its implications for the stable learning-rate range. Our analysis builds on this established curvature–stability connection rather than claiming it as a new result. SPECTRA's contribution is to connect task-specific gradient spectral decay to an empirically calibrated learning-rate predictor, enabling predictions that adapt to gradient characteristics and LoRA rank under the stated local curvature assumptions.

Regarding Proposition 4.5, The Primacy of Magnitude in Low-Rank Adaptation and TML's LoRA Without Regret establish broader parameterization equivalences involving scaling factors, initialization, and factor-specific learning rates. We acknowledge that the underlying scaling transformations are not unique to SPECTRA. Our contribution is to incorporate a restricted scaling-transfer relationship into the spectral learning-rate predictor, rather than to establish a new general equivalence of LoRA optimization dynamics. Consistent with this distinction, our Appendix M experiments recover rank-coefficient shifts of 0.23 and 0.47, close to the predicted 0.25 and 0.50 under the balanced two-factor regime. These observations support the approximation in the evaluated settings without establishing equivalence of the full optimization dynamics.

We cite and distinguish these works directly alongside Propositions 4.3 and 4.5, clarify which arguments follow from established results, and emphasize that SPECTRA's novelty lies in the spectral predictor and its empirical calibration rather than the underlying curvature or reparameterization identities.

Reviewer_jL4VOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SPECTRA, which uses initialization-time gradient spectral features and LoRA rank to predict a learning rate shared across LoRA modules. A predictor is fitted on a reference model and calibrated for the target model to reduce learning-rate search on subsequent tasks. Experiments cover five models and five datasets, reporting an average [email protected]× of 88.5%. The approach has practical potential, but the necessity of the spectral feature, the reusability of calibration, and the actual cost–benefit trade-off need stronger evidence.

Critical strengths
  1. The paper addresses the practical cost of repeated LoRA tuning with a simple prediction formula. If calibration can be reliably reused, the method could reduce search costs across multiple tasks.
  2. The experiments cover multiple models and datasets. The substantial performance drop after removing β in Table 8 supports the spectral feature’s contribution to the full predictor.
Critical weaknesses
  1. The added value of the central idea remains insufficiently established. Prior work has studied how LoRA learning rates vary with rank, model size, and scaling; the main novelty here is the task-dependent gradient spectral feature. Although the β ablation supports its usefulness, it does not establish superiority over simpler prediction rules with the same calibration budget. The theoretical connection between β and the stability boundary also relies on additional assumptions without direct validation. The paper should better establish why this feature justifies its computational and implementation cost.

  2. The calibration protocol underlying the headline result needs clarification. Figure 2 and Algorithm 1 describe calibration using a single dataset–rank configuration, whereas Appendix I.1 and Table 6 associate the 88.5% result with calibration across ranks on one dataset; single-point calibration achieves 83.9%. Please clarify which protocol the reported four-hour cost covers.

Questions for authors
  1. With the same calibration data and budget, how much does the spectral predictor improve over simple rank/scaling rules or other inexpensive task statistics? Is there direct evidence linking β to the actual stable learning rate?
  2. Please clarify the calibration procedure and total cost underlying the headline result.
4 comments

Reviewer's Comment: The added value of the central idea .... // With the same calibration data and budget,....

Response: We thank the reviewer for this important comment and agree that the key contribution of SPECTRA is the task-dependent gradient spectral feature, rather than rank-dependent learning-rate scaling alone. Under the same calibration protocol, removing the spectral feature β reduces [email protected]× from 88.5% to 54.2% (Table 8), demonstrating its substantial contribution beyond rank-based prediction.

To directly address comparisons with simpler alternatives, we additionally evaluate a three-point learning-rate grid search over {2×10⁻⁵, 2×10⁻⁴, 2×10⁻³} on Sec 5, which achieves 79.3% success at 11.5 hours, compared with 88.5% at 4 hours (model-specific anchor sweeps for all evaluated models) for SPECTRA. Importantly, even a small grid search requires prior knowledge of an appropriate search range: the effectiveness of selecting learning rates around 10e-4 relies on empirical guidance from previous studies, such as TML's recommendations based primarily on Llama and Qwen configurations. Without such prior knowledge, practitioners cannot be expected to know in advance that 10e-4 is the appropriate learning-rate scale, particularly for new model families, datasets, or LoRA scaling conventions. SPECTRA instead uses initialization-time gradient statistics to predict configuration-specific learning rates after lightweight calibration, reducing dependence on manually chosen search ranges.

Moreover, SPECTRA requires extensive calibration only once on a reference model across datasets and ranks; each subsequent model requires only a single-dataset anchor sweep, after which learning rates can be predicted across its remaining datasets and ranks without additional training sweeps. This amortization is particularly valuable for large-scale fine-tuning infrastructure, where repeated sweeps across hundreds of models and datasets incur substantial computational costs.

Regarding the theoretical connection, we acknowledge that the relationship between spectral decay and the stability boundary is conditional on the local curvature assumptions, rather than a universal guarantee for nonlinear LoRA optimization. Our theoretical analysis motivates the predictor's functional form, while the ablation results and comparisons against small-grid search provide empirical evidence of its practical utility.

Reviewer's Comment: The calibration protocol ….// Please clarify the calibration procedure and total cost underlying the headline result.

Response: We thank the reviewer for identifying this ambiguity. We clarify that the 88.5% [email protected]× result and the reported four-hour cost correspond to model-specific calibration across multiple LoRA ranks on a single dataset, rather than calibration using a single dataset–rank point. As shown in Appendix I.1 and Table 6, single-point calibration achieves 83.9%, whereas calibration across ranks on one dataset improves performance to 88.5%. Importantly, this protocol requires calibration on only one dataset per new model, after which learning rates can be predicted for all remaining datasets and ranks without additional sweeps. The reported four-hour cost includes the model-specific calibration sweeps for all evaluated models but excludes the one-time reference-model calibration, whose shared coefficients can be reused across subsequent models and tasks. We revise the experimental setup to clarify the calibration protocol and distinguish the one-time offline calibration cost from the reported model-specific calibration cost.

Dear reviewers,

As the rebuttal deadline is approaching, I would like to kindly check whether our responses have addressed your concerns and questions. If there is anything that is still unclear or could benefit from further clarification, please let us know, and we will be happy to elaborate.

Thank you very much for your time and effort in reviewing our work.

Thanks for your detailed response, which resolve most of my concerns. I would like to maintain my score.

thanks for your positive score and if you still have any problems, we also welcome to discuss.

Reviewer_jwsLOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SPECTRA, a framework for predicting the learning rate for LoRA fine-tuning. It uses a calibrated log-linear predictor that maps the initialization-time gradient spectral decay β\beta and the LoRA rank to a learning rate. It also gives a local stability analysis to motivate this form. Experiments cover 5 models and 5 datasets.

Critical strengths

The paper gives a local stability analysis to motivate the log-linear form of the predictor, and it offers some useful intuition, for example why a scale-invariant feature such as β\beta is suitable under Adam.

Critical weaknesses
  • The practical motivation is not convincing. SPECTRA is presented as a way to avoid learning-rate sweeps, but it still relies on sweeps for calibration:

    • sweeps on a reference model across several datasets and ranks
    • a sweep on one dataset of each new model

    The gain over simple defaults is also modest. The 2.4× tolerance accepts anything within a roughly 6× window around the optimum, and under it a fixed learning rate of 2e-4 already reaches 78.8%, compared with 88.5% for SPECTRA and 99.0% for grid search. Despite needing two rounds of learning-rate sweeps (one on a reference model and one on each new model), SPECTRA still misses the 2.4× window in 11.5% of configurations. Given this, I am not convinced that a fitted predictor is a better choice than simply running a small sweep of 3–5 learning rates, as the methods in Table 1 do.

  • The experiments are incomplete, and the code is not released. All runs use a single training configuration on tested models. Without source codes, key details are hard to verify. For example, Section 3 fits β\beta over the full range 1≤k≤2561 \leq k \leq 256, while Appendix E.1 uses a local window around rr. Finally, it is unclear whether the predictor works for LoRA with different initializations, such as PiSSA, MiLoRA, LoRA-GA, or EVA.

Questions for authors
  • Does the 4-hour cost in Table 2 include both calibration sweeps (on the reference model and on each new model)?
  • How does SPECTRA compare with a small sweep of 3–5 learning rates under a similar compute budget?
  • How do the results in Table 2 change under a stricter tolerance, such as 2×?
  • Does the predicted learning rate still work with a different training configuration (e.g., more training steps or a larger batch size)?
  • Can SPECTRA be applied to LoRA with different initializations, such as PiSSA, MiLoRA, LoRA-GA, or EVA? Would it need to be recalibrated?
5 comments

Reviewer's Comment: The practical motivation is not convincing….// How does SPECTRA compare ….

Response: We thank the reviewer for this important observation. We agree that SPECTRA should be positioned as amortizing learning-rate search rather than eliminating calibration entirely. Specifically, the shared coefficients are calibrated once on a reference model, while each new model requires only a single anchor sweep on one dataset. Subsequently, learning rates for all other datasets and LoRA ranks can be predicted without additional training sweeps. The primary use case envisioned for SPECTRA is for commercial training-as-a-service providers. For platforms such as TML's Tinker, Fireworks, and AWS SageMaker, where thousands of users may bring their own custom datasets, the cost of a one-time calibration sweep is negligible when amortized across customer requests. Providers only need to perform one anchor sweep for each newly supported model, after which SPECTRA enables learning-rate prediction for arbitrary customer datasets and LoRA ranks in (O(1)) time, without further sweeps.

Regarding the fixed-learning-rate baseline, we acknowledge that 2e-4 achieves 78.8% [email protected]× under the evaluated constant-α setting, but this favorable result does not reliably generalize across LoRA scaling conventions: the defaults were adopted from empirical guidance primarily involving Llama and Qwen configurations, whereas our Table 3 demonstrates that across four scaling conventions, SPECTRA achieves 84.7% success compared with only 58.1% for the strongest fixed-rate baseline.

Furthermore, the apparent effectiveness of the 2e-4 baseline relies on prior empirical knowledge from TML's study. Without consulting such guidance or conducting preliminary sweeps, practitioners would not necessarily know that the appropriate learning-rate scale is around e-4, particularly for new model families, datasets, or LoRA scaling conventions. SPECTRA addresses this limitation by predicting configuration-specific learning rates from initialization-time gradient statistics after lightweight calibration, rather than relying on manually selected defaults.

To directly address the suggestion of using a small sweep, we additionally evaluated a three-point grid search over {2×10⁻⁵, 2×10⁻⁴, 2×10⁻³} in Sec 5 , which achieves only 79.3% success at 11.5 hours of search cost, compared with 88.5% at 4 hours for SPECTRA; although we acknowledge that the 2.4× criterion permits some prediction error and SPECTRA does not match exhaustive search in every configuration, these results support its practical advantage in reducing aggregate search cost while providing more reliable task- and configuration-specific learning-rate predictions.

Reviewer's Comment: Does the 4-hour cost ...

Response: We thank the reviewer for raising this question. We clarify that the 4-hour cost reported in Table 2 includes the model-specific anchor sweeps for all evaluated models, but excludes the one-time reference-model calibration sweeps. The reference-model calibration is performed offline and its resulting coefficients are reused across subsequent models, datasets, and LoRA ranks. Once the model-specific anchor calibration is completed, SPECTRA predicts learning rates without further per-configuration sweeps. We explicitly clarify this cost accounting and distinguish the one-time offline calibration cost from the reported 4-hour cost in the revised manuscript.

Reviewer's Comment: Does the predicted ....

Response: We thank the reviewer for this question. SPECTRA predicts learning rates based on local optimization stability, whose theoretical threshold does not explicitly depend on the number of training steps. Increasing the training horizon therefore does not inherently invalidate the predicted learning rate; however, when training is terminated before sufficient convergence, the observed loss may primarily reflect undertraining rather than the quality of learning-rate selection, making such comparisons potentially misleading. Regarding batch size, we acknowledge that substantial changes may alter gradient statistics and optimization dynamics; our current experiments use a fixed training configuration, and we clarify this limitation rather than claim guaranteed transfer across batch sizes.

Reviewer's Comment: The experiments are incomplete, and the code is not released….// Can SPECTRA be applied to LoRA with different initializations, …..

Response: We thank the reviewer for highlighting these limitations. We acknowledge that our experiments use a controlled training configuration and recognize the importance of reproducibility. We release our implementation, calibration procedures, and training configurations in the revision. Regarding the spectral estimation, SPECTRA computes singular values over the full evaluation range (up to Kmax⁡=256K_{\max}=256), but estimates the spectral-decay exponent β\beta within a rank-dependent local window around rr, as described in Appendix E.1. We clarify this distinction in Section 3.

Regarding different LoRA initializations, SPECTRA can be extended through an initialization-specific learning-rate correction without retraining the entire spectral predictor. Since SPECTRA extracts gradient spectra with respect to the effective model weights, its spectral features remain applicable across initialization strategies, although different factor initializations can alter the optimizer-level learning-rate scale.

Our additional experiments (Sec 5) on Gemma-3-1B and Qwen3-1.7B support this generalization. SPECTRA achieves 83.3% [email protected]× for MiLoRA and 91.7% for PiSSA, substantially outperforming fixed learning rates and the TM formula. Specifically, for MiLoRA, SPECTRA improves over the strongest fixed learning rate (75.0%) and TM formula (8.3%). For PiSSA, SPECTRA achieves 91.7%, compared with 50.0% for the strongest fixed learning rate and 0.0% for TM. These results demonstrate that conventional learning-rate prescriptions can transfer poorly across initialization strategies, whereas SPECTRA maintains strong prediction accuracy with lightweight initialization-specific calibration.

We incorporate these results, clarify the correction procedure and its calibration cost.

LoRA VariantMethodGemma-3-1BQwen3-1.7BAverage
MiLoRA1e-483.366.775.0
2e-483.350.066.7
4e-466.70.033.3
TM formula16.70.08.3
SPECTRA100.066.783.3
Grid, per (model, dataset)100.0100.0100.0
PiSSA1e-483.316.750.0
2e-433.30.016.7
4e-40.00.00.0
TM formula0.00.00.0
SPECTRA100.083.391.7
Grid, per (model, dataset)100.0100.0100.0

Reviewer's Comment: How do the results in Table 2 ....

Response: We thank the reviewer for this suggestion and have additionally evaluated all methods under the stricter Hit@2× criterion. As shown in the updated results and updated Table 14, SPECTRA achieves 78.2% success, outperforming the strongest fixed-learning-rate baseline (71.1%) by 7.1 percentage points and the TML formula (58.6%) by 19.6 points, while requiring only 4 hours compared with 90 hours for exhaustive grid search.

We clarify that the original 2.4× tolerance is theoretically motivated rather than arbitrarily selected: Appendix E.7 derives it as 2×1.2, accounting for the multiplicative uncertainty introduced by the coarse-to-fine learning-rate sweep under a local log-symmetry assumption. Importantly, Table 2 and Table 14 considers only the constant α=32 setting, where conventional learning-rate defaults are already relatively effective. When α varies with rank, the optimal learning rate can shift substantially, making fixed defaults less transferable. As demonstrated in Table 3, SPECTRA achieves 84.7% [email protected]× across four scaling conventions, compared with 58.1% for the strongest fixed-rate baseline. We include the stricter-tolerance results and clarify the motivation for both evaluation criteria in the revised manuscript.

MethodGemmaLlama2Llama3.1Qwen3-1.7BQwen3-8BAverageTime
1e-435.030.050.038.643.239.40
2e-465.070.072.777.370.571.10
4e-455.070.050.061.452.357.70
TM formula55.070.054.561.452.358.6—
SPECTRA (ours)78.872.580.791.267.978.24h
Grid, per (model, dataset)95.0100.0100.0100.0100.099.090h

Thank you for the clarifications and the additional experiments. I will raise my score accordingly. However, I am still not convinced that predicting the learning rate is meaningfully better than running a small learning-rate sweep, especially in the service scenarios you describe. The method has its own hyperparameters and requires calibration, so it is unclear how much tuning effort it actually saves overall. In addition, the paper does not show a way to detect when a prediction is unreliable or to choose a fallback. If a prediction is wrong, recovering may require extra training runs or a search, which could cancel out the claimed savings.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Rebuttal/Revision Response

We sincerely thank the reviewers for their insightful feedback and constructive suggestions, which have substantially improved the technical rigor, empirical validation, and clarity of our work.

This paper introduces SPECTRA, a gradient-spectral framework for predicting stable, near-optimal learning rates for LoRA fine-tuning without repeated configuration-specific searches. Below, we summarize our key contributions and the major revisions incorporated during the rebuttal.

Key Contributions

  • Task-dependent spectral prediction: We introduce the initialization-time gradient spectral-decay feature (\beta) to predict task- and rank-specific LoRA learning rates, extending beyond existing rank- and scaling-based prescriptions (Sec. 3–4).

  • Amortized learning-rate calibration: SPECTRA calibrates shared coefficients once on a reference model and requires only one dataset for each new model's anchor calibration across ranks, enabling learning-rate prediction for subsequent datasets and ranks without additional training sweeps (Sec. 3, Algorithm 1).

  • Theoretical foundation: We characterize a local stability boundary under Adam-preconditioned, gradient-aligned weight-space dynamics and connect spectral decay, LoRA rank, and module-level stability bottlenecks under explicit assumptions (Sec. 4).

  • Accuracy–cost efficiency: Across five language models and five datasets, SPECTRA achieves 88.5% [email protected]×, outperforming the strongest fixed-learning-rate baseline by 9.7 percentage points, while reducing reported model-specific search cost from 90 to 4 hours relative to exhaustive grid search (Table 2).


Major Revisions

1. Clarified Theoretical Contributions and Related Work

  • Revised Sec. 4: Added direct discussion alongside Propositions 4.3 and 4.5, distinguishing SPECTRA's restricted stability analysis and Adam gradient-scale invariance from existing curvature and LoRA reparameterization results. reviwer w3a6

  • Revised Appendix B: Expanded related work to include LoRA-α, LoRA Without Regret, The Primacy of Magnitude in Low-Rank Adaptation, and On the Convergence Rate of LoRA Gradient Descent.reviwer w3a6

  • New Appendix P: Clarified the limitations of transferring weight-space stability results to nonlinear LoRA factor optimization, including the role of bilinear parameterization-induced curvature.reviwer w3a6

2. Strengthened Empirical Evaluation

  • Revised Sec. 5, Table 2: Added a three-point grid-search baseline, achieving 79.3% [email protected]× at 11.5 hours, compared with 88.5% at 4 hours for SPECTRA. reviwer jL4V reviwer jwsL

  • New Sec. 5.1, Table 4: Extended evaluation to MiLoRA and PiSSA initialization strategies, achieving 83.3% and 91.7% [email protected]×, respectively. all reviewers

  • New Appendix O, Table 14: Introduced the stricter [email protected]× criterion, under which SPECTRA achieves 78.2%, outperforming the strongest fixed-learning-rate baseline by 7.1 percentage points. reviwer jwsL

3. Clarified Practical Applicability and Computational Cost

  • Revised Sec. 3: Distinguished full-range singular-value computation (Kmax⁡=256K_{\max}=256) from rank-dependent local estimation of the spectral-decay exponent. reviwer jwsL

  • Revised Sec. 5: Clarified that the reported 4-hour cost includes model-specific anchor calibration across evaluated models but excludes one-time reference-model calibration. all reviewers

  • New Appendix N: Discussed practical deployment in large-scale fine-tuning services, where shared calibration coefficients can be reused across numerous datasets and configurations. reviwer jwsL


We once again thank the Area Chair and reviewers for their valuable time and constructive feedback. These revisions strengthen SPECTRA's empirical evidence, clarify its relationship to established LoRA optimization theory, and more precisely characterize its accuracy–cost trade-off. All revisions have been incorporated into the updated manuscript, with changes highlighted in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.