acceptodds
Under review as a conference paper at ICLR 2027

Characterizing Local Loss Gaps in Low-Rank Adaptation

Abstract

Low-rank adaptation (LoRA) makes fine-tuning efficient, but a suitable rank is not known in advance. Informed rank adjustment requires diagnosing whether the current rank limits attainable improvement and how much additional improvement a specific increase could unlock. Performance gaps alone cannot answer these questions, because optimization and update scale confound the comparison. We propose a local diagnostic framework that compares the best dense and low-rank updates to the same module within a fixed Frobenius ball. A local quadratic approximation makes this comparison tractable. With separable curvature and feasible unconstrained optima, its gap equals the spectral tail of a curvature-whitened gradient. We control the discrepancy from the actual loss and correct for optima outside the radius. Under explicit approximation-error bounds, the resulting conditional intervals distinguish rank limitation, local sufficiency, and inconclusive cases relative to a prescribed tolerance. The reduction in the gap to the same dense comparator quantifies the optimal benefit of increasing rank; newly captured spectral energy predicts this benefit in the interior regime. Controlled objectives evaluate interval coverage and width, while RoBERTa/SST-2 experiments assess fixed-rank prediction and comparison sensitivity. Newly captured energy ranks observed upgrade gains with Spearman correlations of 0.903 on SST-2 and 0.734 on MRPC, compared with 0.333 and 0.414 for the current tail. The framework informs local rank adjustment while distinguishing optimal capacity gains from finite-budget training outcomes.

Primary area:
optimization
Keywords:
Low-Rank Adaptation, Local Loss Gap, Rank Constraints, Curvature-Aware Analysis, Generalized Gauss–Newton

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.60 over 5 reviews, above 70% of the others.
Reviewer_SYGTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce a local rank gap with actual-loss error bounds that compares the optimal dense and rank-constrained updates. They also establish approximation-error bounds connecting this quadratic proxy to the actual loss, and develop boundary corrections when the optimal updates exceed the prescribed radius. They further separate evaluations of interval validity and predictive usefulness. The authors conduct experiments on controlled quadratic objectives validate the theoretical intervals, and conduct RoBERTa experiments on SST-2 and MRPC evaluate the predictive usefulness of the proposed spectral measures.

Critical strengths

Following are the strengths of this paper:

  • The paper focuses on the an important question in parameter-efficient fine-tuning, and constructs a meaningful mathematical framework. Under a shared norm constraint, the authors define the local rank gap that can provide a meaningful mathematical framework. Under this setting, the author can isolate the representational limitation imposed by rank from the influence from learning rates, initialization, and update magnitudes.
  • The authors provide a rigorous theoretical analysis. The authors establish bounds that account for errors in gradients, curvature approximation, local quadratic modeling, and trust-region constraints. The authors present their paper with mathematical rigor and clearly defined conditions.
  • The authors present careful numerical validation and have a transparent reporting of limitations. The paper includes many controlled experiments with numerical-reference audits, sensitivity analyses, and comparisons between different curvature approximations. The authors also clearly show the limitations.
Critical weaknesses

Following are the weaknesses of this paper:

  • The paper's theoretical guarantees rely on strong assumptions whose practical validity is not established. The core spectral identity relies on separable positive-definite curvature and the feasibility of unconstrained optimal updates. While the authors introduce approximation-error terms to absorb deviations, the resulting guarantees require uniform bounds on Hessian variation, GGN-to-Hessian mismatch, and curvature-proxy error. Those quantities are notoriously difficult to estimate in modern networks. The empirical diagnostics suggest directional alignment but do not establish these uniform bounds on real models. The paper stops short of showing that the certification framework yields informative, non-vacuous bounds in realistic fine-tuning regimes, weakening the central claim that it can rigorously distinguish rank limitation from local sufficiency.
  • In this paper, the paper has limited empirical evaluation and insufficient evidence of generalization. The main neural-network evaluation is restricted to RoBERTa-base on SST-2 and MRPC across only a handful of modules (10–20 per task). While the appendix includes an exploratory Llama-3.1-8B run, it evaluates only a single module and omits the calibrated loss-gap and rank-upgrade analyses present in the main text. Without testing across broader model families, autoregressive generation tasks, or larger module sets, and without error bars or dispersion metrics across independent layer groups, it is difficult to gauge whether the observed rank correlations hold generally.
  • The connection between theoretically optimal rank gains and actual training outcomes is not sufficiently validated. The theoretical guarantees characterize the gap between local optimal losses, whereas the experiments track empirical, finite-budget training runs that are inherently confounded by rank-dependent optimization dynamics. While the empirical correlations are promising, they demonstrate rank ordering rather than quantitative estimation of achievable loss reductions. There are not enough demonstration on contrasting the estimated spectral gains against both numerical approximations of the true local optima and the empirical training trajectories.
Questions for authors

Following are the questions:

  • How can the uniform bounds on gradient error, curvature approximation, and Hessian variation required by Theorem 4.1 be computed or reasonably bounded in realistic architectures? Can the authors provide an empirical case on RoBERTa (or another model) where the framework yields a non-trivial, certified rank-limitation or sufficiency bound?
  • Can this diagnostic drive an active rank-allocation schedule? How does such a policy compare empirically to baselines like AdaLoRA under identical total parameter and FLOP budgets?
  • Can the authors provide fixed-rank and rank-upgrade results across multiple modules on larger autoregressive models (e.g., full Llama-3.1 evaluations)? Does the advantage of curvature-whitened spectral energy hold on generation benchmarks?
  • How reliably does the captured spectral energy predict the actual numerical magnitude of the optimal rank-upgrade gain, as opposed to merely ranking module candidates? Can the authors quantify the discrepancy attributable to finite-budget optimization?
5 comments

Thank you for highlighting how the common update radius isolates the effect of rank, and for asking how the guarantees and scores apply to neural networks and training.

W1 and Q1 · A certified loss gap on RoBERTa

On RoBERTa-base/SST-2 module L04-Q at R=0.05R=0.05, Theorem 5.1 certifies rank 4 as rank-limited and rank 8 as locally sufficient. The certified gaps between the best rank-constrained and dense losses in the same ball are [3.046,3.289]×10−4[3.046,3.289]\times10^{-4} for rank 4 and [2.446,2.689]×10−4[2.446,2.689]\times10^{-4} for rank 8. The intervals lie on opposite sides of the same tolerance, ε=2.9×10−4\varepsilon=2.9\times10^{-4}. The original interior-feasibility screen required radius 0.0590.059 for this module, so the certificate uses the boundary corrections.

We keep the original expansion center and the mean cross-entropy over the same fixed 1,024 SST-2 examples, with all other parameters frozen and dropout disabled. The center gradient uses all 1,024 examples, giving zero subsampling error. The shared-position proxy is separable and positive definite after damping each factor by 10−3I10^{-3}I.

Center curvature. We bound the full Hessian–proxy difference directly, covering GGN mismatch, separable approximation and damping. With the objective, proxy and audit settings fixed before sampling, we apply the 16th power of the scaled residual operator to 20 independent Gaussian probes. Hessian-vector products and proxy multiplication with validated numerical enclosures yield ∥∇2L(0)−S^∥op≤ϵH=0.002\|\nabla^2L(0)-\widehat S\|_{\mathrm{op}}\le\epsilon_H=0.002 with failure probability at most 2−202^{-20}.

Uniform Hessian variation. We propagate value ranges and analytic derivative bounds through the frozen network with outward rounding. This bounds the third derivative norm at the center by C3=0.128C_3=0.128 and the fourth derivative norm over the entire ball of radius 0.100.10 by M4=0.400M_4=0.400. Integrating along line segments gives

\le \rho\|\Delta\|_F, \qquad \rho=C_3+0.10M_4=0.168, \qquad \|\Delta\|_F\le 0.10.$$ At $R=0.05$, the curvature term is $5.0\times10^{-6}$ and the Taylor term is $7.0\times10^{-6}$. Allowing up to $10^{-7}$ for numerical gradient and endpoint errors gives $E_{\mathrm{tot}}\le1.21\times10^{-5}$. Both certified intervals hold on the same audit event. Revised Section 6.4 summarizes the result. **Appendix G.1** details the verification and reports the results at four radii in **Table 22**. ### W2 and Q3 · Prediction across Llama layers on GSM8K On Llama-3.1-8B/GSM8K, the full-factor tail gives the lowest calibrated mean absolute error (MAE) for fixed-rank loss gaps, and the newly captured full-factor energy gives the highest Spearman correlation with observed upgrade gains among the predictors tested. Q2 below reports the effect of using the upgrade score to allocate rank on generation accuracy. **Fixed-rank loss gaps.** We evaluate one query projection per layer. For each of ranks 4 and 8, we fit a scale and offset on 16 calibration layers and test on 16 disjoint layers. The dense and low-rank references use a common radius $R=0.02$ and the same 512-example objective as the scores, with references averaged across three seeds. All MAEs below use $10^{-3}$ loss units. | Rank | Full-factor Spearman correlation | Raw full-factor MAE | Calibrated full-factor MAE (95% CI) | Calibrated diagonal-factor MAE | | ---: | -------------------------------: | ------------------: | ----------------------------------: | -----------------------------: | | 4 | 0.976 | 0.463 | **0.197** $[0.131,0.295]$ | 0.410 | | 8 | 0.918 | 0.402 | **0.193** $[0.124,0.309]$ | 0.526 | The intervals use 4,000 bootstrap draws, resampling calibration and evaluation layers separately and refitting the calibration on each draw. See **Appendix G.2, Table 24** for all baselines. **Rank upgrades during joint training.** We train all 32 query adapters without a norm cap. After 200 steps at rank 4, we fork a rank-4 continuation and 16 branches that each upgrade one prespecified adapter to rank 8, preserving its effective update at the fork. All adapters remain trainable, and each branch runs for 400 more steps with matched minibatch order and optimizer-state handling. The observed gain is the final answer-only cross-entropy of the rank-4 continuation minus that of the upgrade branch, averaged over three seeds per candidate. The score $t_{4,8}=s_4-s_8$, computed at the pretrained weights, has Spearman correlation **0.68** with these gains. The diagonal-factor, current-tail and first-order scores reach 0.45, 0.30 and 0.24, respectively. The 95% bootstrap interval across candidate layers is $[0.272,0.911]$, based on 4,000 draws using the three-seed mean gains. Leaving out one layer at a time gives correlations between 0.625 and 0.775. See **Appendix G.2, Table 25**.

Q2 · Active rank allocation against AdaLoRA

On Llama-3.1-8B/GSM8K, at matched total parameter and cumulative FLOP budgets, score-guided allocation reaches 51.10% mean test accuracy, 0.51 percentage points above AdaLoRA (50.59%) and 2.10 above uniform rank 5 (49.00%).

We jointly train all 32 query adapters from rank 4 without a norm cap. At each checkpoint, we refresh t4,8t_{4,8} for rank-4 adapters and upgrade the highest-scoring one to rank 8. Eight upgrades give 8×8+24×4=1608\times8+24\times4=160 rank units. Our allocation and uniform rank 5 each use 1,310,720 LoRA factor parameters. Parameter counts include AdaLoRA's extra parameters. FLOPs reflect executed ranks, score refreshes and allocation overhead.

Allocation policyValidation lossGSM8K accuracy (%)
Uniform rank 50.842±0.0070.842\pm0.00749.00±0.7249.00\pm0.72
Random selection0.847±0.0080.847\pm0.00848.50±0.8448.50\pm0.84
First-order gradient gain0.840±0.0070.840\pm0.00749.20±0.7249.20\pm0.72
Current spectral tail0.838±0.0070.838\pm0.00749.41±0.7649.41\pm0.76
AdaLoRA0.829±0.0060.829\pm0.00650.59±0.6550.59\pm0.65
Newly captured spectral energy0.824±0.0060.824\pm0.00651.10±0.7251.10\pm0.72

Results use three seeds and greedy decoding on all 1,319 test examples (Section 6.5 and Appendix G.2, Table 26 and Eq. (G.1)).

W3 and Q4 · Gain magnitude and finite-budget optimization

We quantify upgrade gains using a RoBERTa certificate, numerical network references and exact quadratic optima. Each comparison fixes the objective, module, center and radius.

Optimal gain on RoBERTa. On L04-Q at R=0.05R=0.05, the raw score is t4,8=0.600×10−4t_{4,8}=0.600\times10^{-4}. Applying Corollary 5.2 on the same audit event as Q1 gives

B∗=m4−m8∈[0.47899,0.72100]×10−4B^*=m_4-m_8\in[0.47899,0.72100]\times10^{-4}

∣B∗−t4,8∣≤0.12101×10−4.\qquad |B^*-t_{4,8}|\le 0.12101\times10^{-4}.

The dense correction cancels, so the transfer budget enters once. The interval includes low-rank boundary and numerical allowances (Section 6.4 and Appendix G.1).

Errors against network references. The reference gain BrefB_{\mathrm{ref}} is rank-4 minus rank-8 reference loss. When the ranks share a dense reference, it equals the difference in fixed-rank gaps. Summing both fixed-rank MAEs bounds the upgrade MAE.

EvaluationRaw spectral differenceDifference of rank-wise calibrated predictionsLoss units
RoBERTa/SST-2, 10 held-out modules0.44270.398210−410^{-4}
Llama/GSM8K, 16 held-out layers0.8660.39110−310^{-3}

The RoBERTa reference gain varies by at most 0.02407×10−40.02407\times10^{-4} from 400 to 800 steps and 0.08517×10−40.08517\times10^{-4} across paired restarts. Residuals to the true optima remain unknown. See Appendix F.2.1, Table 11 and Figure 5, and Appendix G.3, Table 27 and Eq. (G.2).

Direct comparison with exact optima. We use quadratic objectives with exact SVD solutions at R=0.05R=0.05 and a perturbed separable curvature proxy. We fit a scale and offset directly to upgrade gains on 10 cases and evaluate on 10 disjoint cases. Against the exact gains, MAE is 0.1014×10−40.1014\times10^{-4} for the raw score and 0.0649×10−40.0649\times10^{-4} after calibration, compared with 0.1454×10−40.1454\times10^{-4} for the calibration-mean constant. These are 16.90%, 10.82% and 24.23% of the mean exact gain of 0.6000×10−40.6000\times10^{-4}, respectively.

For a feasible endpoint at rank kk, let ek(T)=L(Δk(T))−mke_k(T)=L(\Delta_k(T))-m_k be the loss above the true minimum after TT steps. The observed gain Btrain(T)=L(Δ4(T))−L(Δ8(T))B_{\mathrm{train}}(T)=L(\Delta_4(T))-L(\Delta_8(T)) satisfies

Btrain(T)−B∗=e4(T)−e8(T).B_{\mathrm{train}}(T)-B^*=e_4(T)-e_8(T).

The exact optima let us measure this optimization contribution separately from prediction error. All errors below use 10−410^{-4} loss units.

StepsTraining gain vs. exact gain MAERaw score vs. training gain MAECalibrated score vs. training gain MAE
2000.07190.17330.1273
4000.03740.13880.0958
8000.02020.12160.0819

The MAE between training and exact gains falls from 11.98% to 3.37% of the mean exact gain between 200 and 800 steps. See Appendix D.2 for the residual decomposition and Appendix G.3, Tables 28–29 and Eq. (G.3) for the controlled objectives and full results.

Dear Reviewer,

Thank you for taking the time to review our paper and share your feedback. We’d be happy to clarify anything in our response if helpful.

Best regards, The Authors

Reviewer_d5bsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a local diagnostic framework to evaluate how much a chosen rank r restricts loss improvement in LoRA. To isolate the effect of rank from optimization dynamics, the authors study a single module in isolation while freezing all other network parameters. They compare the best dense update and the best rank-r update within a fixed Frobenius ball. To approximate this gap, the authors use a second-order Taylor model with a Kronecker-factored (K-FAC style) curvature proxy.

In the theoretical analyses, the authors derive deterministic error bounds and boundary corrections.

Finally, they evaluate the framework on synthetic 24-dimensional quadratic/quartic objectives and single-module fine-tuning experiments on RoBERTa (SST-2, MRPC).

Critical strengths

The paper makes a clean and useful observation regarding rank upgrades. When increasing rank, the relevant predictor is not the total discarded spectral tail, but the newly captured energy. These results are examined on SST-2 and MRPC as well.

Moreover, the authors rightly point out that comparing raw LoRA and dense training runs is confounded by different update norms. Controlling the Frobenius radius and accounting for curvature whitening provides a cleaner local comparator than raw gradient norms.

Critical weaknesses

In the theoretical analyses, the paper focuses on deriving certified intervals for the true loss gap using a Taylor transfer budget where the curvature error is decomposed into GGN mismatch, sampling error, and K-FAC proxy error. However, these certified intervals are never instantiated on any neural network. In the experimental analyses, the interval bounds are only tested on synthetic 24-dimensional toy functions where analytic bounds are plugged in by hand. On RoBERTa and Llama, uniform operator-norm bounds and the Hessian Lipschitz constant are either intractable to compute or completely vacuous. Consequently, the interval bounds are not employed, and the heuristic score are utilized.

Moreover, there is a major contradiction in regimes where the Taylor error E(R)E(R) is small:

  • For the cubic Taylor error to be small on a nonlinear network, the radius RR must be very small.
  • However, when RR is small relative to the unconstrained step, the dense boundary penalty is large and quickly exceeds the tail srs_r.
  • As soon as dense boundary penalty exceeds the tail, the lower bound collapses to zero, while the upper bound penalty grows large. Thus, outside a narrow Goldilocks radius, the boundary interval is vacuous.

The novelty of the theoretical results is also limited. For instance, Theorem 3.1 and Corollary 3.2 utilize the standard in Kronecker-factored low-rank approximation for whitening. Moreover, Proposition B.3.1 is a direct application of the Matrix Bernstein inequality.

In the experimental analyses, only a single weight matrix is updated at a time while the rest of RoBERTa is frozen, and the optimizer is explicitly constrained to a tiny Frobenius ball. In practice, all adapters are trained jointly without a small Frobenius ball constraint. Moreover, standard unconstrained endpoints have norms up to 0.50.5 and do not follow the local quadratic gap. Furthermore, 4 out of 24 candidate modules on RoBERTa are excluded from the main results.

Questions for authors

Can you compute the certified interval from Theorem 5.1 for any module in RoBERTa/SST-2?

Otherwise, is E(R)E(R) orders of magnitude larger than sr∼10−4s_r \sim 10^{-4} on real networks?

How well does tr,r′t_{r,r'} computed at initialization (or a checkpoint) predict rank-upgrade gains when all LoRA modules in RoBERTa are trained simultaneously under standard unconstrained AdamW, rather than single-module ball-constrained training?

3 comments

Thank you for highlighting the value of a common update radius and of using the newly captured spectral energy to assess a rank increase.

W1 and Q1–Q2 · A certificate on RoBERTa/SST-2

Theorem 5.1 certifies rank 4 as rank-limited and rank 8 as locally sufficient for RoBERTa/SST-2 module L04-Q at R=0.05R=0.05. The intervals for the gap between the best rank-constrained and dense losses in the same ball are [3.046,3.289]×10−4[3.046,3.289]\times10^{-4} for rank 4 and [2.446,2.689]×10−4[2.446,2.689]\times10^{-4} for rank 8. The first lies entirely above the tolerance 2.9×10−42.9\times10^{-4}, and the second lies entirely below it. L04-Q required radius 0.0590.059 to pass the interior-feasibility screen (Table 10), so this certificate uses the boundary corrections.

We use the original expansion center and mean cross-entropy over the same 1,024 SST-2 examples, with all other parameters frozen and dropout off. Evaluating the gradient on all 1,024 examples gives zero subsampling error. We bound the total operator-norm discrepancy between the center Hessian and the proxy directly, covering GGN mismatch, proxy approximation and damping. With the objective, proxy and audit settings fixed in advance, we apply the 16th power of the scaled residual operator to 20 independent Gaussian probes. Hessian-vector products with validated numerical enclosures give an upper bound of ϵH=0.002\epsilon_H=0.002 with failure probability at most 2−202^{-20}.

For the Taylor remainder, outward-rounded propagation bounds the third derivative norm at the center by 0.1280.128 and the fourth derivative norm over the entire ball of radius 0.100.10 by 0.4000.400. Integrating along line segments gives the uniform Hessian-variation bound ∥∇2L(Δ)−∇2L(0)∥op≤0.168∥Δ∥F\|\nabla^2L(\Delta)-\nabla^2L(0)\|_{\mathrm{op}}\le 0.168\|\Delta\|_F. All four radii share the same audit event.

At R=0.05R=0.05, the curvature term R2ϵH=5.0×10−6R^2\epsilon_H=5.0\times10^{-6} and the Taylor term ρR3/3=7.0×10−6\rho R^3/3=7.0\times10^{-6} give E(R)=1.20×10−5E(R)=1.20\times10^{-5} before numerical allowances. Adding at most 10−710^{-7} for numerical gradient and endpoint errors gives Etot≤1.21×10−5E_{\mathrm{tot}}\le1.21\times10^{-5}. This upper allowance is about 3.56% and 4.32% of the rank-4 and rank-8 spectral scores (3.40×10−43.40\times10^{-4} and 2.80×10−42.80\times10^{-4}). Both diagnoses become inconclusive at R=0.10R=0.10 as the budget grows. Revised Section 6.4 and Appendix G.1 (Table 22) give the audit and all four radii.

W2 · The upper bound at small radii

When the dense boundary penalty exceeds the spectral tail, the lower bound is zero and the upper bound can still be tight. Equation (18) subtracts the dense penalty from the upper bound.

=\widehat q(\alpha_r D_r)-v_d(R)+E(R).$$ Here $\widehat q(\alpha_r D_r)$ is the proxy loss at the feasible scaled rank-$r$ step, and $v_d(R)$ is the dense proxy minimum in the same ball. The scaled step has proxy loss at most $\widehat q(0)=0$, and positive curvature gives $v_d(R)\ge -R\|\widehat g_0\|_F$. Thus $UB_r\le R\|\widehat g_0\|_F+E(R)\to0$ as $R\to0$ under the fixed error bounds of Theorem 4.1. An upper bound at or below the tolerance certifies local sufficiency even when the lower bound is zero. **A $2\times2$ example.** Take the exact quadratic $q(\Delta)=\tfrac12\|\Delta\|_F^2-\langle\operatorname{diag}(4,3),\Delta\rangle$ and set the rank cap to $1$. For $0<R<2$, both optima use the full radius. The dense step achieves linear improvement $5R$ and the rank-one step achieves $4R$, with the same quadratic cost $R^2/2$. The exact gap is therefore $R$. In Eq. (18), $E(R)=0$, $s_1=4.5$, $d_d(R)=\tfrac12(5-R)^2>s_1$, and $d_1(R)=\tfrac12(4-R)^2$. At $R=0.10$, the lower bound is zero and the upper bound is $4.5+7.605-12.005=0.100$, exactly the true gap. **RoBERTa shows the same behavior.** On L04-Q at $R=0.01$ and $0.02$, the dense penalties ($6.90$ and $4.37$, in units of $10^{-4}$) exceed both spectral tails, so the lower bounds are zero. The largest upper bounds across ranks 4 and 8 are $0.640$ and $1.286$ in the same units. Both are below the tolerance of $2.9$, so both ranks are certified locally sufficient at both radii (revised **Appendix G.1, Table 22**).

W3 · Actual-loss bounds and rank upgrades

We bound the actual-loss gap caused by a rank restriction and the gain from a specified rank upgrade within a common Frobenius ball. Theorem 3.1 uses completion of the square, Corollary 3.2 uses rank-preserving whitening and the Eckart–Young–Mirsky theorem, and Proposition B.3.1 applies Matrix Bernstein to sampling error.

When the radius constrains the update, the same whitened spectrum can yield different gaps (Appendix B.4). Theorems 4.1 and 5.1 handle this by combining an exact dense boundary correction from a convex trust-region problem with a uniform transfer budget for the actual loss. A feasible scaled update suffices to bound the low-rank boundary correction, so both gap endpoints can be computed without solving the rank-constrained problem to global optimality.

For a proposed upgrade r→r′r\to r', Corollary 5.2 directly compares the two rank-constrained minima. The dense term cancels, so the comparison pays E(R)E(R) once. If both unconstrained low-rank proxy optima fit in the ball, the optimal gain differs from the newly captured energy tr,r′=sr−sr′t_{r,r'}=s_r-s_{r'} by at most E(R)E(R), regardless of dense feasibility. For L04-Q at R=0.05R=0.05, we use the boundary-corrected form of Corollary 5.2. The score t4,8=0.600×10−4t_{4,8}=0.600\times10^{-4} gives a certified optimal gain of [0.47899,0.72100]×10−4[0.47899,0.72100]\times10^{-4} after including the boundary and numerical allowances (revised Section 6.4 and Appendix G.1).

W4 and Q3 · Joint training and candidate exclusions

Under joint, unconstrained AdamW on RoBERTa/SST-2, t4,8t_{4,8} computed at initialization has near-zero Spearman correlation (−0.067-0.067) with observed upgrade gains across all 24 Q/V candidates. For each of three seeds, we trained all adapters at rank 4 for 400 steps, then forked a rank-4 continuation and one upgrade branch per candidate from the same checkpoint. Each upgrade changed one adapter to rank 8, preserving its effective update at the fork. All adapters remained trainable with no norm projection. Each branch reset optimizer state and ran for 400 more steps on matched minibatches. The gain is the drop in final cross-entropy relative to the rank-4 continuation, averaged across seeds before computing the correlation. The current-tail and first-order baselines give −0.035-0.035 and −0.142-0.142, respectively (revised Appendix G.1.2, Table 23).

The shared radius controls the update budget in the local rank comparison. In the layer-10 experiment, the raw dense update norm was about 0.5, versus 0.1395 to 0.0449 for LoRA across ranks 1 to 16. Raw evaluation gaps were nearly flat (0.01294–0.01298). The gaps decreased with rank when projecting the endpoints to a common radius and when optimizing both updates directly under that radius (Appendix F.4.1, Table 14).

In a separate joint, unconstrained Llama-3.1-8B/GSM8K experiment with 32 trainable query adapters, initialization t4,8t_{4,8} achieved Spearman 0.680.68 across 16 upgrade candidates and three seeds (95% layer-group bootstrap interval [0.272,0.911][0.272,0.911]). A separate active allocation experiment using checkpoint scores achieved 51.10% test accuracy versus 49.00% at uniform rank 5, with matched parameter and cumulative FLOP budgets (revised Appendix G.2, Tables 25–26).

The original local evaluation screened out four of 24 modules. Thresholds were fixed before evaluating predictive accuracy. They were applied independently on SST-2 and MRPC, yielding the same four exclusions (Table 10 and revised Appendix G.1.1).

ModuleExclusion and recorded diagnostic
L03-QMaximum relative directional residual of 0.0540 exceeded the locality threshold of 0.05.
L04-QThe smallest radius containing all three unconstrained proxy optima (dense, rank 4 and rank 8) was 0.0590, above the evaluation radius of 0.05.
L07-VThe step-800 paired-restart gap spread was 0.2244×10−40.2244\times10^{-4}, above the 0.10×10−40.10\times10^{-4} threshold.
L10-VThe scoring job ended before saving the full-factor score. Training references were retained and the score was left missing.

Re-including L04-Q with the rank-specific SST-2 calibration held fixed changes rank-4/rank-8 MAE from 0.221/0.1770.221/0.177 to 0.241/0.1710.241/0.171 on held-out SST-2 modules and from 0.204/0.2270.204/0.227 to 0.230/0.2560.230/0.256 on MRPC (all in 10−410^{-4} loss units, Appendix F.2.2, Table 12). The joint RoBERTa experiment obtained scores for all 24 candidates, including L10-V.

Dear Reviewer,

Thank you for taking the time to review our paper and share your feedback. We’d be happy to clarify anything in our response if helpful.

Best regards, The Authors

Reviewer_izZrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how to evaluate the role of rank in LoRA, focusing on whether rank limits potential improvements and how much additional improvement can be achieved by increasing it. The authors propose a local analysis framework based on a quadratic approximation to study these questions. Experiments are conducted using RoBERTa on the SST-2 and MRPC classification tasks.

Critical strengths

The research question is interesting and worth exploring.

Critical weaknesses
  1. The overall presentation lacks clarity, making the main ideas and technical details difficult to follow.
  2. The experiments are limited to relatively small language models and simple classification tasks. This limited evaluation raises concerns about the practical relevance and effectiveness of the proposed method.
Questions for authors
  1. The proposed method relies on a quadratic approximation. What is its computational cost? Can it scale to larger language models and more practical tasks?
  2. Could the authors provide more intuitive explanations of the main theorems to help readers better understand their meaning and implications?
10 comments

W1: Paper outline and theorem roadmap

Thank you for this suggestion. We have added Outline of the Paper (Appendix A.2) and Roadmap of Main Theorem (Appendix A.3) to explain the paper’s structure and how the main results are connected. The outline follows the main argument: compare dense and low-rank updates under the same conditions, study their best losses in a quadratic model, and then account for approximation error and the update-size limit. Appendix A.3.2 explains how these steps lead to the main loss-gap bounds and where the supporting results are used. It also clarifies that the singular-value formula and the approximation-error bound are separate parts of the argument.

Q2: Clarification of the main results

Thank you for your feedback. Following your suggestion, we have added Clarification of the Main Results (Appendix A.3.1), which explains the meaning of each result and its role in the argument. Theorem 3.1 identifies the extra cost of rank in the quadratic model. Under its curvature and size-limit conditions, Corollary 3.2 expresses this cost using the discarded singular values. Theorem 4.1 bounds the difference between the quadratic-model gap and the actual-loss gap. Theorem 5.1 combines this error bound with the size-limit corrections to judge whether rank is limiting, sufficient, or still uncertain. Corollary 5.2 then bounds the benefit of a specific rank increase. Appendix A.3.3 explains their practical use. In the RoBERTa L04-Q example, rank 4 is rank-limited while rank 8 is sufficient at the same stated radius and tolerance. For a rank-4-to-8 upgrade, the score counts only the added directions 5 through 8. We also explain that these bounds concern the best attainable losses; the gains reached by training depend on how closely each run approaches those losses.

W2 & Q1: Computational cost and evaluation on a larger model

Scoring took 6.6 seconds on RoBERTa. On Llama-3.1-8B/GSM8K, it took 18.4 of 141.7 minutes (13.0%) and peaked at 34.2 GiB on a 48 GB GPU (Appendix G.2, Runtime and memory). We rescale the gradient with two separable curvature factors, avoiding the full Hessian. One SVD gives s4s_4, s8s_8, and t4,8=s4−s8t_{4,8}=s_4-s_8.

On RoBERTa, gradient and factor collection took 6.3 of 6.6 seconds. Scoring took 7.3% of the 400-step continuation time (Section 6.3 and Appendix F.3, Table 13).

StageTime
Gradient and full-factor collection6.3 s
Eigendecomposition and whitening0.25 s
SVD and score evaluation0.05 s
Diagnostic total6.6 s
400-step continuation90 s

We extend the exploratory single-module Llama/SST-2 study (Appendix F.4.7, Table 21) to Llama-3.1-8B/GSM8K. We use the base checkpoint, 6,961 training and 512 validation examples, answer-only cross-entropy, AdamW at 2×10−42\times10^{-4}, effective batch size 16 and three seeds. Scores use 512 training examples and full separable factors with relative damping 10−310^{-3}. We decode greedily on all 1,319 test examples (Section 6.5 and Appendix G.2).

At ranks 4 and 8, we compare dense and low-rank updates to one query projection at a time with other parameters frozen and a shared radius R=0.02R=0.02. We score at pretrained weights and average numerical references over three seeds on the same 512-example objective. We fit each predictor's scale and offset on 16 calibration groups and test on 16 disjoint layer groups. MAEs are in 10−310^{-3} units. We resample the groups separately and refit each map in 4,000 bootstrap draws (Table 24).

RankFull-factor Spearman ρ\rhoFull-factor calibrated MAE (95% CI)Diagonal-factor calibrated MAEGradient-norm calibrated MAERank-only calibrated MAE
40.9760.197 [0.131, 0.295]0.4100.6000.752
80.9180.193 [0.124, 0.309]0.5260.7640.769

We score pretrained weights and jointly train query adapters in all 32 layers without a norm cap. After 200 rank-4 steps, we fork a rank-4 continuation and 16 runs, each upgrading one candidate to rank 8. Other adapters remain trainable at rank 4. Branches run 400 more steps with matched minibatch order and optimizer-state handling. Upgrades preserve the effective update and α/r=1\alpha/r=1. We average final loss reductions relative to the rank-4 continuation over three seeds per candidate (Table 25).

PredictorUpgrade-gain Spearman ρ\rho
First-order gradient gain0.24
Current spectral tail s4s_40.30
Diagonal-factor energy0.45
New spectral energy t4,8t_{4,8}0.68

Resampling the 16 candidate layers 4,000 times gives a 95% interval of [0.272, 0.911] for the correlation of t4,8t_{4,8}. Omitting any one layer gives correlations of 0.625–0.775.

To test whether the score improves allocation in practice, we start all 32 adapters at rank 4 in a separate run without a norm cap. We reserve 32 rank units, refresh the score at training checkpoints, and move the highest-scoring eligible adapter to rank 8 at each of eight upgrades. Our schedule and uniform rank 5 each use 1,310,720 LoRA factor parameters. All policies have matched total parameter and cumulative FLOP budgets. We count AdaLoRA's extra parameters and include score refreshes and allocation in the FLOP budget (Appendix G.2, Table 26).

Allocation policyValidation lossGSM8K test accuracy (%)
Uniform rank 50.842 ± 0.00749.00 ± 0.72
Random selection0.847 ± 0.00848.50 ± 0.84
First-order gradient gain0.840 ± 0.00749.20 ± 0.72
Current spectral tail s4s_40.838 ± 0.00749.41 ± 0.76
AdaLoRA0.829 ± 0.00650.59 ± 0.65
Score-guided allocation0.824 ± 0.00651.10 ± 0.72

Across three seeds, score-guided allocation improved mean test accuracy by 2.10 percentage points over uniform rank 5 and 0.51 points over AdaLoRA.

Dear Reviewer,

Thank you for taking the time to review our paper and share your feedback. We’d be happy to clarify anything in our response if helpful.

Best regards, The Authors

Reviewer_LDV3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper defines a local rank gap by comparing the best dense and rank-constrained updates to one module under the same loss, expansion point, and Frobenius-norm budget. Under separable positive-definite curvature and feasible unconstrained optima, the quadratic gap equals half the sum of squared discarded singular values of a curvature-whitened gradient. The authors derive approximation-error and boundary corrections, and characterize the benefit of increasing rank through newly captured spectral energy. Controlled objectives test fixed-rank prediction and interval validity, while RoBERTa experiments on SST-2 and MRPC test prediction of numerical loss gaps and finite-budget rank-upgrade gains.

Critical strengths
  1. A precise comparison with careful error accounting. Holding the module, objective, center, and radius fixed separates rank capacity from update magnitude. The boundary corrections address a real failure of the uncorrected spectral tail, while the framework distinguishes curvature approximation from finite-budget optimization error (Sections 3–5). This makes the framework a useful contribution beyond the interior SVD identity alone.

  2. Evidence that the score predicts more than rank itself. Fixed-rank comparisons use disjoint calibration and evaluation layers and include gradient-only and diagonal-curvature baselines. For upgrades from rank 4 to 8, newly captured energy gives Spearman correlations of 0.903 on SST-2 and 0.734 on MRPC, versus 0.333 and 0.414 for the current tail (Table 3). These results support the practical value of distinguishing upgrade-specific energy from total remaining capacity.

Critical weaknesses
  1. Practical relevance beyond the local regime is unclear. The main RoBERTa experiments, including rank upgrades, restrict total updates to R=0.05R=0.05. The paper does not establish whether predictions within this small region support useful rank decisions at larger update scales.

  2. Certified rank diagnosis is not demonstrated on neural models. The neural results support predictive ordering, but the approximation bounds needed to certify rank sufficiency are not established. Such certification is validated only on controlled objectives, a limitation the authors acknowledge.

  3. Missing experimental details limit reproducibility. Key data and optimization settings, curvature damping, and rank-upgrade initialization are unspecified for the main neural experiments. These choices affect both the predictor and measured gains, making the results difficult to assess and reproduce.

Questions for authors
  1. Experimental clarification: Please specify or point to the data/objective, optimizer, curvature-factor estimation and damping, norm-constraint enforcement, and rank-upgrade initialization/budget settings underlying Tables 2–3. Are the two continuation arms otherwise matched?
  2. Screening clarification: What thresholds determined the exclusions in Table 10, were they fixed before evaluating predictive accuracy, and why is the L10-V score unavailable? Were the same exclusions independently justified on MRPC?
  3. Radius clarification: How was R=0.05R=0.05 selected for Tables 2–3, and was that choice fixed before examining prediction accuracy? What existing evidence connects the diagnostic to practical rank decisions beyond this evaluated radius?
5 comments

Thank you for your careful review and questions about rank decisions in joint training.

W1 & Q3: Radius choice and rank decisions beyond the local setting

We fixed R=0.05R=0.05 for Tables 2–3 before checking prediction correlations on the evaluation layers. A shared radius gives ranks the same update budget. On the calibration layers, we compared R∈{0.02,0.05,0.10}R\in\{0.02,0.05,0.10\} by checking quadratic fit and whether the unconstrained proxy optima fit within each ball (Appendix E.1). At the chosen radius, the upgrade score reached Spearman correlations of 0.903 on SST-2 and 0.734 on MRPC (Table 3).

In RoBERTa layer 10, the dense endpoint had Frobenius norm about 0.5 and LoRA endpoints ranged from 0.0449 to 0.1395. Evaluation loss gaps were nearly flat (0.01294–0.01298). Optimizing both under the same cap reduced the gap from 4.1×10−44.1\times10^{-4} at rank 1 to 1.3×10−41.3\times10^{-4} at rank 16 (Appendix F.4.1, Table 14).

In joint training without a norm cap, we tested initialization scores on all RoBERTa Q/V adapters (including L10-V) and 16 of Llama's 32 trainable query adapters. From a shared rank-4 checkpoint, we compared upgrading one adapter to rank 8 with keeping all at rank 4. Other adapters remained trainable at rank 4 in both arms. Each arm ran 400 more steps with matched minibatch order. We report Spearman correlations with final cross-entropy reductions averaged over three seeds per candidate (Appendices G.1.2 and G.2, Tables 23 and 25).

Initialization scoreRoBERTa/SST-2 (24 candidates)Llama-3.1-8B/GSM8K (16 candidates)
New spectral energy s4−s8s_4-s_8−0.0670.68
Current spectral tail s4s_4−0.0350.30
First-order gradient gain−0.1420.24
Diagonal-factor energy—0.45

The initialization score tracked upgrade gains on Llama (95% bootstrap interval [0.272, 0.911] across candidate layers) and showed little association on RoBERTa.

In a separate unconstrained Llama experiment, we refreshed scores at checkpoints and upgraded the highest-scoring rank-4 adapter to rank 8, eight times in total. At matched parameter and cumulative FLOP budgets, mean test accuracy was 51.10 ± 0.72%, 2.10 percentage points above uniform rank 5 (49.00 ± 0.72%). AdaLoRA reached 50.59 ± 0.65%. Our allocation and uniform rank 5 each use 1,310,720 LoRA factor parameters and yield validation losses of 0.824 ± 0.006 and 0.842 ± 0.007, respectively. FLOPs include score refreshes and allocation. Results summarize three seeds (Section 6.5 and Appendix G.2, Table 26).

The guarantees apply at a specified radius (Section 8). For the RoBERTa module in W2, increasing the radius to R=0.10R=0.10 gives inconclusive bounds for both ranks at the same tolerance (Appendix G.1, Table 22).

W2: A certified rank diagnosis on RoBERTa

For RoBERTa-base/SST-2 module L04-Q, Theorem 5.1 certifies that rank 4 limits improvement and rank 8 is locally sufficient at R=0.05R=0.05 and tolerance ε=2.9×10−4\varepsilon=2.9\times10^{-4}.

We kept the original expansion center and mean cross-entropy objective on the same 1,024 SST-2 examples, with other parameters frozen and dropout off. Both ranks used a shared separable curvature proxy with 10−3I10^{-3}I damping per factor. The smallest radius that contains all the unconstrained proxy optima is 0.059, so we apply the boundary correction at R=0.05R=0.05.

Using Hessian-vector products and 20 independent Gaussian probes, we bound the center Hessian–proxy difference by 0.0020.002 in operator norm. We propagate derivative bounds through the frozen network. The third-derivative bound at the center is 0.1280.128, and the fourth-derivative bound over the radius-0.100.10 ball is 0.4000.400. Together they bound Hessian variation by 0.168∥Δ∥F0.168\|\Delta\|_F throughout that ball.

Evaluating all 1,024 examples removes subsampling error for the fixed objective. With at most 10−710^{-7} of numerical gradient and endpoint error, the total allowance at R=0.05R=0.05 is at most 1.21×10−51.21\times10^{-5}. Both intervals hold on the same audit event, with failure probability at most 2−202^{-20}.

RankActual-loss gap interval (10−410^{-4} units)Diagnosis at ε=2.9×10−4\varepsilon=2.9\times10^{-4}
4[3.046, 3.289]Rank-limited (LB4>εLB_4>\varepsilon)
8[2.446, 2.689]Locally sufficient (UB8<εUB_8<\varepsilon)

Section 6.4 summarizes the certificate. Appendix G.1 gives the audit, endpoint formulas and outward-rounded intervals at four radii (Table 22). Section 8 discusses scope.

W3 & Q1: Experimental settings and matched continuation arms

Yes. Both continuation arms in Table 3 start from the same checkpoint after 400 steps at rank 4. To isolate the effect of the rank increase, we reset the optimizer state in both arms and run each for 400 more steps with the same objective, minibatch order, AdamW settings, norm limit, and evaluation rule. The rank-8 arm uses more FLOPs.

Table 2 compares dense and low-rank updates on RoBERTa-base/SST-2. Table 3 compares rank-4 and rank-8 continuations on SST-2 and MRPC. In both experiments, we update one Q or V matrix and freeze the rest of the model. We use mean cross-entropy on a fixed subset of 1,024 training examples per task. AdamW uses a constant learning rate of 10−310^{-3}, zero weight decay, batch size 16, and no warmup.

The score and reference losses use the same examples. At the original expansion center, we compute the gradient and average input activation outer products and loss-weighted curvature terms to form the two shared-position curvature factors. We exclude padding and add 10−3I10^{-3}I to each factor (Appendices B.3 and G.1.1).

After each step, we rescale any update whose Frobenius norm exceeds R=0.05R=0.05 back to that radius, scaling one factor for LoRA. This cap applies to the total effective update from the original center, including after the rank increase (Appendix G.1.1).

For the fixed-rank comparison in Table 2, we use ten repeats per module and rank. At 400 and 800 steps, we take the lowest loss among three restarts separately for the dense and low-rank updates (Appendices E.3 and F.2.1).

For the upgrade in Table 3, we keep the existing rank-4 factors, initialize the added input factors randomly and the added output factors to zero, and keep α/r=1\alpha/r=1. This preserves the effective update and model output at the point where the runs branch. We compute t4,8=s4−s8t_{4,8}=s_4-s_8 from a shared proxy for both ranks, without separate calibration. For each module, we average the final rank-4 loss minus the final rank-8 loss over ten paired repeats. Table 3 reports Spearman correlations between the scores and these module-level gains (Appendices G.1.1 and F.3).

Q2: Screening thresholds, the missing score and MRPC

We fixed the thresholds before measuring predictive accuracy on the evaluation layers and kept them fixed. The three checks determine whether the quadratic model fits the loss, whether its unconstrained optima fit within the radius, and whether the numerical reference gap is stable across restarts. The table below gives the thresholds from Appendix G.1.1 and the corresponding SST-2 exclusions from Table 10 (Appendix F.2.1).

CheckAcceptance criterionExcluded candidate and diagnostic
Local quadratic fitMaximum relative directional residual ≤0.05\leq0.05L03-Q: 0.05400.0540
Interior feasibilityThe unconstrained dense, rank-4 and rank-8 proxy updates must all fit within R=0.05R=0.05L04-Q: required radius 0.05900.0590
Restart stabilityPaired-restart gap spread at step 800 ≤0.10×10−4\leq0.10\times10^{-4}L07-V: maximum spread 0.2244×10−40.2244\times10^{-4}

Restart spread is the range of low-rank minus dense loss differences across paired restarts at step 800 (Appendix F.2.1).

L10-V was excluded from comparisons requiring a full-factor score because its scoring job ended before saving that score. We kept the training-reference records and left the score missing without imputation. The separate joint RoBERTa experiment has scores for all 24 candidates, including L10-V. We report its results in W1 & Q3 above and discuss both the joint experiment and the original exclusions in our response to Reviewer d5bs (W4 & Q3).

We screened MRPC independently by applying the same thresholds and score availability check to MRPC's own records. L03-Q failed the local fit check, L04-Q required a radius of 0.059 for interior feasibility, L07-V failed the restart stability check, and L10-V lacked the full-factor score (Appendices G.1.1 and F.2.2).

Dear Reviewer,

Thank you for taking the time to review our paper and share your feedback. We’d be happy to clarify anything in our response if helpful.

Best regards, The Authors

Reviewer_WQqWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the loss gap between full-rank and low-rank updates under a shared norm constraint. Under a quadratic model with separable curvature, the authors derive a spectral characterization of this gap. Experiments on RoBERTa fine-tuning suggest that a score based on newly captured spectral energy can help predict which modules benefit most from increasing their rank.

Critical strengths

The paper asks an interesting question and the spectral characterization gives a clean interpretation of the local loss gap. The analysis of hadnling the case where the unconstrained optima lie outside the norm constraint uses a convex problem and a feasible low-rank update to bound the gap. I found this argument interesting. The experiments provide some nice evidence that the proposed score can help identify which modules benefit from additional rank.

Critical weaknesses

The main theoretical result relies on standard low-rank approximation ideas, so the mathematical novelty is somewhat limited. The experiments are limited and do not show that the proposed score leads to better rank allocation in practice.

I found the writing difficult to follow. The writing is excessively LLM assisted.

Questions for authors

.

2 comments
AuthorsOpenReview ↗

Response to Reviewer WQqW

Theoretical contribution

Thank you for your feedback. We bound the loss due to a rank cap and the gain from a specific rank increase, even when the update-size limit binds. The rank gap compares the best losses with and without the cap, holding the loss function, weight matrix, starting point and update-size limit fixed.

With positive-definite, separable curvature, classical low-rank approximation gives the quadratic-model gap from the singular values (Corollary 3.2) if both unconstrained optima fit within the size limit. When the limit binds, identical spectra after curvature scaling can yield different gaps (Appendix B.4).

Given a positive-definite separable proxy and the uniform error bound from Theorem 4.1, Theorem 5.1 bounds the actual-loss gap even when the size limit binds. A convex problem gives the full-rank correction, and a feasible update bounds the low-rank correction. Rank is limiting if the lower bound exceeds the tolerance and sufficient if the upper bound is within it. Otherwise, the bounds leave the question open.

Corollary 5.2 bounds the gain from raising rank from rr to r′r'. The score tr,r′=sr−sr′t_{r,r'}=s_r-s_{r'} counts the energy in the added directions (5 through 8 for a move from rank 4 to 8). Comparing the two ranks directly cancels the full-rank reference, giving a model-error allowance of E(R)E(R) versus 2E(R)2E(R) from subtracting separate gap intervals. If the unconstrained quadratic optima at both ranks fit the limit, the score is within E(R)E(R) of the optimal gain in the actual loss, even if the full-rank optimum lies outside the limit.

Does the score improve rank allocation in practice?

In a new Llama-3.1-8B/GSM8K experiment, we jointly train all 32 query adapters from rank 4 without a norm cap. At each of eight checkpoints, we refresh the scores and upgrade the highest-scoring rank-4 adapter to rank 8. Under matched parameter and total FLOP budgets, our allocation improves mean test accuracy by 2.10 percentage points over uniform rank 5. We count AdaLoRA's extra parameters and include score refreshes and allocation in the compute budget. The table summarizes three seeds.

Allocation policyValidation lossGSM8K test accuracy (%)
Uniform rank 50.842 ± 0.00749.00 ± 0.72
Random selection0.847 ± 0.00848.50 ± 0.84
First-order gradient gain0.840 ± 0.00749.20 ± 0.72
Current spectral tail s4s_40.838 ± 0.00749.41 ± 0.76
AdaLoRA0.829 ± 0.00650.59 ± 0.65
New spectral energy t4,8t_{4,8}0.824 ± 0.00651.10 ± 0.72

In a separate paired experiment, we computed the scores at the pretrained weights. For each of 16 candidate adapters, we branched from the same training checkpoint and compared upgrading that adapter to rank 8 with keeping all adapters at rank 4 for the same number of steps. We averaged each candidate's loss reduction over three seeds and computed the bootstrap intervals by resampling candidate layers.

PredictorSpearman correlation95% bootstrap CILeave-one-layer-out range
First-order gradient gain0.24[−0.388, 0.713][0.082, 0.325]
Current spectral tail s4s_40.30[−0.171, 0.649][0.232, 0.425]
Diagonal-factor energy0.45[−0.024, 0.753][0.375, 0.536]
New spectral energy t4,8t_{4,8}0.68[0.272, 0.911][0.625, 0.775]

Section 6.5 summarizes both experiments. Appendix G.2 gives the protocols and results (Tables 25–26).

Clearer writing and a simpler explanation

Sections 2–5 now open by stating the question each section addresses. Section 2 states what we hold fixed when comparing dense and low-rank updates. Before Theorem 3.1, we explain what the unconstrained update represents. We now use quadratic-model gap and actual-loss gap consistently.

After Corollary 3.2, we explain how keeping the leading singular directions gives the spectral formula. After Theorem 4.1, we explain why the approximation error bound applies to both optimal losses. Section 5 shows how to read the interval when deciding whether a rank is sufficient and explains that the upgrade score counts the energy in the added directions.

Section 6 opens by explaining what each group of experiments tests. A new RoBERTa example shows how to use the bounds. For the same matrix, update-size limit, and tolerance, rank 4 limits improvement and rank 8 is sufficient.

Dear Reviewer,

Thank you for taking the time to review our paper and share your feedback. We’d be happy to clarify anything in our response if helpful.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.