- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors introduce a local rank gap with actual-loss error bounds that compares the optimal dense and rank-constrained updates. They also establish approximation-error bounds connecting this quadratic proxy to the actual loss, and develop boundary corrections when the optimal updates exceed the prescribed radius. They further separate evaluations of interval validity and predictive usefulness. The authors conduct experiments on controlled quadratic objectives validate the theoretical intervals, and conduct RoBERTa experiments on SST-2 and MRPC evaluate the predictive usefulness of the proposed spectral measures.
Critical strengths
Following are the strengths of this paper:
- The paper focuses on the an important question in parameter-efficient fine-tuning, and constructs a meaningful mathematical framework. Under a shared norm constraint, the authors define the local rank gap that can provide a meaningful mathematical framework. Under this setting, the author can isolate the representational limitation imposed by rank from the influence from learning rates, initialization, and update magnitudes.
- The authors provide a rigorous theoretical analysis. The authors establish bounds that account for errors in gradients, curvature approximation, local quadratic modeling, and trust-region constraints. The authors present their paper with mathematical rigor and clearly defined conditions.
- The authors present careful numerical validation and have a transparent reporting of limitations. The paper includes many controlled experiments with numerical-reference audits, sensitivity analyses, and comparisons between different curvature approximations. The authors also clearly show the limitations.
Critical weaknesses
Following are the weaknesses of this paper:
- The paper's theoretical guarantees rely on strong assumptions whose practical validity is not established. The core spectral identity relies on separable positive-definite curvature and the feasibility of unconstrained optimal updates. While the authors introduce approximation-error terms to absorb deviations, the resulting guarantees require uniform bounds on Hessian variation, GGN-to-Hessian mismatch, and curvature-proxy error. Those quantities are notoriously difficult to estimate in modern networks. The empirical diagnostics suggest directional alignment but do not establish these uniform bounds on real models. The paper stops short of showing that the certification framework yields informative, non-vacuous bounds in realistic fine-tuning regimes, weakening the central claim that it can rigorously distinguish rank limitation from local sufficiency.
- In this paper, the paper has limited empirical evaluation and insufficient evidence of generalization. The main neural-network evaluation is restricted to RoBERTa-base on SST-2 and MRPC across only a handful of modules (10–20 per task). While the appendix includes an exploratory Llama-3.1-8B run, it evaluates only a single module and omits the calibrated loss-gap and rank-upgrade analyses present in the main text. Without testing across broader model families, autoregressive generation tasks, or larger module sets, and without error bars or dispersion metrics across independent layer groups, it is difficult to gauge whether the observed rank correlations hold generally.
- The connection between theoretically optimal rank gains and actual training outcomes is not sufficiently validated. The theoretical guarantees characterize the gap between local optimal losses, whereas the experiments track empirical, finite-budget training runs that are inherently confounded by rank-dependent optimization dynamics. While the empirical correlations are promising, they demonstrate rank ordering rather than quantitative estimation of achievable loss reductions. There are not enough demonstration on contrasting the estimated spectral gains against both numerical approximations of the true local optima and the empirical training trajectories.
Questions for authors
Following are the questions:
- How can the uniform bounds on gradient error, curvature approximation, and Hessian variation required by Theorem 4.1 be computed or reasonably bounded in realistic architectures? Can the authors provide an empirical case on RoBERTa (or another model) where the framework yields a non-trivial, certified rank-limitation or sufficiency bound?
- Can this diagnostic drive an active rank-allocation schedule? How does such a policy compare empirically to baselines like AdaLoRA under identical total parameter and FLOP budgets?
- Can the authors provide fixed-rank and rank-upgrade results across multiple modules on larger autoregressive models (e.g., full Llama-3.1 evaluations)? Does the advantage of curvature-whitened spectral energy hold on generation benchmarks?
- How reliably does the captured spectral energy predict the actual numerical magnitude of the optimal rank-upgrade gain, as opposed to merely ranking module candidates? Can the authors quantify the discrepancy attributable to finite-budget optimization?
5 comments
[2/2]
Q2 · Active rank allocation against AdaLoRA
On Llama-3.1-8B/GSM8K, at matched total parameter and cumulative FLOP budgets, score-guided allocation reaches 51.10% mean test accuracy, 0.51 percentage points above AdaLoRA (50.59%) and 2.10 above uniform rank 5 (49.00%).
We jointly train all 32 query adapters from rank 4 without a norm cap. At each checkpoint, we refresh for rank-4 adapters and upgrade the highest-scoring one to rank 8. Eight upgrades give rank units. Our allocation and uniform rank 5 each use 1,310,720 LoRA factor parameters. Parameter counts include AdaLoRA's extra parameters. FLOPs reflect executed ranks, score refreshes and allocation overhead.
| Allocation policy | Validation loss | GSM8K accuracy (%) |
|---|---|---|
| Uniform rank 5 | ||
| Random selection | ||
| First-order gradient gain | ||
| Current spectral tail | ||
| AdaLoRA | ||
| Newly captured spectral energy |
Results use three seeds and greedy decoding on all 1,319 test examples (Section 6.5 and Appendix G.2, Table 26 and Eq. (G.1)).
W3 and Q4 · Gain magnitude and finite-budget optimization
We quantify upgrade gains using a RoBERTa certificate, numerical network references and exact quadratic optima. Each comparison fixes the objective, module, center and radius.
Optimal gain on RoBERTa. On L04-Q at , the raw score is . Applying Corollary 5.2 on the same audit event as Q1 gives
The dense correction cancels, so the transfer budget enters once. The interval includes low-rank boundary and numerical allowances (Section 6.4 and Appendix G.1).
Errors against network references. The reference gain is rank-4 minus rank-8 reference loss. When the ranks share a dense reference, it equals the difference in fixed-rank gaps. Summing both fixed-rank MAEs bounds the upgrade MAE.
| Evaluation | Raw spectral difference | Difference of rank-wise calibrated predictions | Loss units |
|---|---|---|---|
| RoBERTa/SST-2, 10 held-out modules | 0.4427 | 0.3982 | |
| Llama/GSM8K, 16 held-out layers | 0.866 | 0.391 |
The RoBERTa reference gain varies by at most from 400 to 800 steps and across paired restarts. Residuals to the true optima remain unknown. See Appendix F.2.1, Table 11 and Figure 5, and Appendix G.3, Table 27 and Eq. (G.2).
Direct comparison with exact optima. We use quadratic objectives with exact SVD solutions at and a perturbed separable curvature proxy. We fit a scale and offset directly to upgrade gains on 10 cases and evaluate on 10 disjoint cases. Against the exact gains, MAE is for the raw score and after calibration, compared with for the calibration-mean constant. These are 16.90%, 10.82% and 24.23% of the mean exact gain of , respectively.
For a feasible endpoint at rank , let be the loss above the true minimum after steps. The observed gain satisfies
The exact optima let us measure this optimization contribution separately from prediction error. All errors below use loss units.
| Steps | Training gain vs. exact gain MAE | Raw score vs. training gain MAE | Calibrated score vs. training gain MAE |
|---|---|---|---|
| 200 | 0.0719 | 0.1733 | 0.1273 |
| 400 | 0.0374 | 0.1388 | 0.0958 |
| 800 | 0.0202 | 0.1216 | 0.0819 |
The MAE between training and exact gains falls from 11.98% to 3.37% of the mean exact gain between 200 and 800 steps. See Appendix D.2 for the residual decomposition and Appendix G.3, Tables 28–29 and Eq. (G.3) for the controlled objectives and full results.
Dear Reviewer,
Thank you for taking the time to review our paper and share your feedback. We’d be happy to clarify anything in our response if helpful.
Best regards, The Authors
[1/2]
Thank you for highlighting how the common update radius isolates the effect of rank, and for asking how the guarantees and scores apply to neural networks and training.
W1 and Q1 · A certified loss gap on RoBERTa
On RoBERTa-base/SST-2 module L04-Q at R=0.05, Theorem 5.1 certifies rank 4 as rank-limited and rank 8 as locally sufficient. The certified gaps between the best rank-constrained and dense losses in the same ball are [3.046,3.289]×10−4 for rank 4 and [2.446,2.689]×10−4 for rank 8. The intervals lie on opposite sides of the same tolerance, ε=2.9×10−4. The original interior-feasibility screen required radius 0.059 for this module, so the certificate uses the boundary corrections.
We keep the original expansion center and the mean cross-entropy over the same fixed 1,024 SST-2 examples, with all other parameters frozen and dropout disabled. The center gradient uses all 1,024 examples, giving zero subsampling error. The shared-position proxy is separable and positive definite after damping each factor by 10−3I.
Center curvature. We bound the full Hessian–proxy difference directly, covering GGN mismatch, separable approximation and damping. With the objective, proxy and audit settings fixed before sampling, we apply the 16th power of the scaled residual operator to 20 independent Gaussian probes. Hessian-vector products and proxy multiplication with validated numerical enclosures yield ∥∇2L(0)−S∥op≤ϵH=0.002 with failure probability at most 2−20.
Uniform Hessian variation. We propagate value ranges and analytic derivative bounds through the frozen network with outward rounding. This bounds the third derivative norm at the center by C3=0.128 and the fourth derivative norm over the entire ball of radius 0.10 by M4=0.400. Integrating along line segments gives
\le \rho\|\Delta\|_F, \qquad \rho=C_3+0.10M_4=0.168, \qquad \|\Delta\|_F\le 0.10.$$ At $R=0.05$, the curvature term is $5.0\times10^{-6}$ and the Taylor term is $7.0\times10^{-6}$. Allowing up to $10^{-7}$ for numerical gradient and endpoint errors gives $E_{\mathrm{tot}}\le1.21\times10^{-5}$. Both certified intervals hold on the same audit event. Revised Section 6.4 summarizes the result. **Appendix G.1** details the verification and reports the results at four radii in **Table 22**. ### W2 and Q3 · Prediction across Llama layers on GSM8K On Llama-3.1-8B/GSM8K, the full-factor tail gives the lowest calibrated mean absolute error (MAE) for fixed-rank loss gaps, and the newly captured full-factor energy gives the highest Spearman correlation with observed upgrade gains among the predictors tested. Q2 below reports the effect of using the upgrade score to allocate rank on generation accuracy. **Fixed-rank loss gaps.** We evaluate one query projection per layer. For each of ranks 4 and 8, we fit a scale and offset on 16 calibration layers and test on 16 disjoint layers. The dense and low-rank references use a common radius $R=0.02$ and the same 512-example objective as the scores, with references averaged across three seeds. All MAEs below use $10^{-3}$ loss units. | Rank | Full-factor Spearman correlation | Raw full-factor MAE | Calibrated full-factor MAE (95% CI) | Calibrated diagonal-factor MAE | | ---: | -------------------------------: | ------------------: | ----------------------------------: | -----------------------------: | | 4 | 0.976 | 0.463 | **0.197** $[0.131,0.295]$ | 0.410 | | 8 | 0.918 | 0.402 | **0.193** $[0.124,0.309]$ | 0.526 | The intervals use 4,000 bootstrap draws, resampling calibration and evaluation layers separately and refitting the calibration on each draw. See **Appendix G.2, Table 24** for all baselines. **Rank upgrades during joint training.** We train all 32 query adapters without a norm cap. After 200 steps at rank 4, we fork a rank-4 continuation and 16 branches that each upgrade one prespecified adapter to rank 8, preserving its effective update at the fork. All adapters remain trainable, and each branch runs for 400 more steps with matched minibatch order and optimizer-state handling. The observed gain is the final answer-only cross-entropy of the rank-4 continuation minus that of the upgrade branch, averaged over three seeds per candidate. The score $t_{4,8}=s_4-s_8$, computed at the pretrained weights, has Spearman correlation **0.68** with these gains. The diagonal-factor, current-tail and first-order scores reach 0.45, 0.30 and 0.24, respectively. The 95% bootstrap interval across candidate layers is $[0.272,0.911]$, based on 4,000 draws using the three-seed mean gains. Leaving out one layer at a time gives correlations between 0.625 and 0.775. See **Appendix G.2, Table 25**.