acceptodds
Withdrawn from ICLR 2027

Singular Energy-Aware Layer-Wise Rank Allocation for Accurate Low-Rank Fine-Tuning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Fine-tuning large-scale neural networks in a low-rank space has become a popular strategy in the era of large language models (LLMs). However, most existing methods, such as low-rank reparameterization and low-rank adaptation (LoRA), assume a uniform rank across all layers. We provide practical insights into exploiting the learning capacity of low-rank models through layer-wise rank allocation. In particular, we find that the singular energy of gradients plays a key role in determining layer-wise ranks for strong downstream performance. Based on this observation, we propose a singular energy-aware (SEA) rank allocation scheme that can be applied to a broad class of low-rank fine-tuning methods. We further provide a theoretical analysis of the optimal rank in terms of retained energy after pruning. Extensive experiments demonstrate consistent improvements in downstream performance with negligible additional computational and memory overhead.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Rank Allocation, Parameter-Efficient Fine-Tuning, Low-Rank Reparameterization, Singular Value Decomposition, Large Language Models, Low-Rank Adaptation, Model Compression

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_tLNLOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes SEA, a static layer-wise rank allocation scheme for low-rank fine-tuning. Ranks are assigned by greedily retaining singular modes with the highest gradient-energy density under a parameter budget, where the gradient spectrum is estimated once via SVD of the averaged gradient over high-loss calibration samples. Experiments on five models and several benchmarks show improvements over uniform and adaptive rank-allocation baselines.

Critical strengths
  1. The method is simple, static, and adds no per-step training overhead; it is compatible with both SVD reparameterization and LoRA.
  2. The weight-vs-gradient spectral analysis is nicely presented, and the empirical coverage across models and benchmarks is broad.
Critical weaknesses
  1. The core objective is a heuristic rather than a justified criterion. The paper optimizes retained normalized gradient singular energy, but it never shows that this surrogate actually tracks downstream loss, either theoretically or through targeted ablations (such as comparing against a magnitude-weighted variant). In particular, normalizing the energy per layer is an unexamined design choice that discards gradient magnitudes and could easily introduce bias rather than remove it. Meanwhile, the theoretical results (convexity, threshold rule, and near-optimality) are just standard fractional-knapsack derivations and do not bridge the gap between the surrogate objective and the true loss. Overall, the central premise of the method relies on assertion rather than solid evidence.

  2. The gradient calibration step is methodologically questionable. SEA relies on applying SVD to the averaged gradient over high-loss samples. However, averaging gradients across samples allows individual directions to cancel out, which can severely distort the singular spectrum being measured. In addition, the claimed norm relationship between the full-batch and calibration gradients does not hold as stated, and selecting only high-loss samples tends to over-represent outliers. A more principled alternative, such as using the uncentered second moment of the gradients (as done in GaLore), is neither discussed nor compared against. Because the entire rank allocation depends on this calibration signal, these methodological flaws are a major concern.

  3. The empirical gains cannot be clearly attributed to the proposed score. Because the experiments lack a random non-uniform allocation baseline, it is impossible to tell whether the improvements come from SEA's specific ranking or simply from using a non-uniform rank distribution. The inverted-order ablation only shows that the method is better than an adversarial worst-case assignment. Given that the main contribution of the paper is the importance score itself, failing to isolate its effect from generic non-uniformity is a critical gap.

Questions for authors
  1. How does the spectrum of the averaged calibration gradient compare to the full-batch gradient spectrum (e.g., subspace alignment), and to the spectrum of E[GG⊤]?
  2. Can you add a random non-uniform allocation baseline at matched budget to isolate the value of SEA's ordering?
  3. Can you ablate the per-layer normalization against a magnitude-weighted objective?
Reviewer_kTS2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Singular Energy-Aware rank allocation for low-rank fine-tuning. SEA uses calibrated gradient singular spectra to allocate ranks across matrices under a parameter budget, ranking singular modes by their layer-normalized energy per parameter. The paper analyzes a continuous relaxation of the normalized spectral-energy objective and provides an approximation bound for discrete threshold selection under stated assumptions. Experiments evaluate SEA with both low-rank weight reparameterization and LoRA across several language models and downstream tasks.

Critical strengths

1.The paper addresses a practical allocation problem: a uniform rank need not provide an effective use of a fixed parameter budget. Using downstream gradients provides a task-dependent alternative to allocating ranks from pretrained weights alone. 2.The evaluation covers both weight reparameterization and LoRA, with models ranging from OPT-350M to LLaMA-2-7B. The reported improvements over uniform and weight-based allocation are encouraging, particularly at tighter compression budgets. 3.The continuous formulation provides a useful explanation of the energy-density threshold rule. The calibration ablations and some useful experiments, e.g., Table 10, help characterize the method.

Critical weaknesses
  1. Layer-wise normalization makes each matrix’s singular-value energies sum to one, removing information about absolute gradient magnitude. 2.Algorithm 1 does not guarantee feasibility of the returned ranks as written. 3.Appendix B describes shared training settings and reference parameter budgets, but the handling of LoRA scaling when SEA assigns different ranks is not clear. 4.A correlation between individual sample losses and gradient magnitudes does not establish that high-loss calibration preserves the normalized singular-value spectra, or the resulting rank allocation, obtained from the full-data gradient. Table 10 shows high-loss selection in one OPT-350M compression setting, but does not establish its robustness across larger models or noisy and atypical samples.
Questions for authors

1.How is the minimum-rank constraint handled in the implementation? Do all reported parameter counts include the adjustment in Algorithm 1, line 14? 2.Which matrices are included in the global allocation? How are LoRA scaling factors set when ranks vary across matrices? 3.Can you provide per-setting dispersion and peak calibration/SVD memory for representative experiments?

Reviewer_813HOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to split a fixed parameter budget across layers in low-rank fine-tuning, for both SVD reparameterization (compress-then-tune) and LoRA. It proposes SEA: estimate each layer’s gradient from a small set of high-loss calibration samples, SVD it, normalize the singular energies within each layer, and greedily keep modes in descending order of energy-per-parameter until the budget is exhausted. The allocation is static (done once before training).

The authors do not explicitly mention the main contributions, but I interpret them as:

  • A characterization of optimal allocation for the chosen spectral-energy objective, together with an additive bound for the discrete threshold solution (Theorem 2, Proposition 4).
  • The choice of the downstream gradient spectrum, normalized per layer and estimated from high-loss samples, as the importance signal.
  • A plug-in procedure (Algorithm 2) applicable to both reparameterization and LoRA.
  • An empirical study across several model families and task types.
Critical strengths
  • The main text is well written and easy to follow. Figure 1, Table 1, and Algorithms 1-2 do a good job conveying the main ideas.
  • The problem has a good motivation and the method is simple, static, and cheap: one calibration pass, one SVD per matrix, zero training-time overhead.
  • Comparing weight/gradient-spectrum allocation under the same allocator (Tables 2-3) is the right controlled experiment: it holds Algorithm 1, the budget, and the training recipe fixed and changes only which matrix supplies the spectrum, so any difference between the “Weight-based” and “SEA” rows can be attributed to the choice of signal, which is the paper’s central claim. In the settings where the models are clearly functioning (e.g., Table 2 GLUE at 75%), SEA is ahead.
  • Table 5 (LLaMA-2-7B) is in a non-degenerate regime and compares against current SOTA reparameterization and adaptation baselines. It is the strongest evidence in the paper (e.g., PTB perplexity 13.98 vs. 18.02 for DobiSVD, with the shortest epoch time among adapters).
  • The calibration ablations (Figure 3, Table 10) answer practical questions a user would have.
Critical weaknesses

Recommendation: reject. Key reasons: (1) The central design choice, per-layer normalization of gradient energy, is unjustified, is in tension with the paper’s own first-order motivation, and is never ablated. (2) A substantial part of the empirical evidence comes from collapsed models and under-tuned or misconfigured baselines with implausibly small reported variance, so the claims are not yet supported. The idea is coherent and the Table 5 results are promising; with those two issues resolved and SEA still winning, this would be a reasonable paper.

Details:

  • Per-layer normalization is not justified. This normalization removes differences in total gradient magnitude, and the paper’s own justification in section 3.3 (discarding leading gradient directions causes the largest first-order loss increase) is an argument about absolute energy. No experiment compares normalized vs. unnormalized densities. This is the central design choice separating SEA from a plain global energy threshold and should be ablated. Have you compared normalized and unnormalized energy using the same allocator and parameter budget? Does per-layer normalization help, hurt, or not matter?
  • Comparisons among collapsed models. Some Table 2 comparisons are made between models that have degenerated to trivial predictors, so the reported gaps do not reflect allocation quality. E.g., on GLUE, three configurations score exactly 69.13 on CoLA, which is the dev-set majority-class rate (721/1043), indicating a constant-label predictor. In these settings a few-point advantage means one method collapsed slightly less, not that it allocates rank better. Please (a) report CoLA as Matthews correlation, under which these failures are visible, (b) indicate which Table 2 rows you consider functioning models, and (c) state whether the 25% claim in line 420 holds once such degenerate settings are excluded.
  • Several Table 4 baselines appear misconfigured rather than outperformed. On GSM8K, MoDeGPT (0.50% on Qwen2-1.5B, 2.35% on Phi-1.5), Layer-wise Independent selection (2.43%) and WeLore (5.61%) score 15–20 points below plain uniform SVD at the same 50% budget, and IGU-LoRA (35.23) scores 8 points below vanilla LoRA (43.37). Given that published methods are losing that badly to the simplest baseline, the fault could be in the adaptation rather than the methods. Were the authors able to reproduce each baseline’s published numbers in its native setting before adapting it?
  • Variance is implausibly small and seeds are unfixed. “Standard deviations mostly ≤ 0.1” across three unseeded runs is not credible. E.g., for RTE (277 examples), one example = 0.36 points. Several LoRA-table gains (e.g., RTE +2.9, MRPC +2.5) that carry the averages are within typical seed noise. Please report per-task mean ± std with fixed seeds.
  • Related work. EVA (Explained Variance Adaptation) and GoRA (Gradient-driven Adaptive Low Rank Adaptation), both accepted at NeurIPS 2025, are the closest prior work but neither are cited nor compared. EVA’s rank-redistribution step is essentially SEA’s Algorithm 1 applied to the activation spectrum (global sort of per-layer-normalized spectral energy, keep top-k, count per layer), and GoRA likewise allocates static ranks from pre-training gradients. Both bundle allocation with adapter initialization. Can the authors position SEA relative to both and provide a controlled comparison that separates SEA from their contributions?
Questions for authors
  • Have the authors compared normalized and unnormalized energy using the same allocator and parameter budget? Does per-layer normalization help, hurt, or not matter?
  • Can the authors report CoLA as Matthews correlation, under which these failures are visible, (b) indicate which Table 2 rows you consider functioning models, and (c) state whether the 25% claim in line 420 holds once such degenerate settings are excluded.
  • Were the authors able to reproduce each baseline’s published numbers in its native setting before adapting it?
  • For each table, can the authors report per-task mean ± std with fixed seeds?
  • Can the authors position SEA relative to EVA and GoRA and provide a controlled comparison that separates SEA from their contributions?
Reviewer_dW77OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper gives different ranks to different layers for low-rank fine-tuning. The total parameter budget is fixed. The method is called SEA. It computes gradients on a small set of high-loss samples (Eq. 4). For each layer, it takes the SVD of the gradient. It normalizes the squared singular values in each layer (Eq. 1). Then it picks singular modes one by one, from high to low energy per parameter, until the budget is used (Alg. 1). These ranks are used for SVD reparameterization or for LoRA (Alg. 2). The theory has three parts. The best allocation uses one shared threshold (Thm 2). Uniform ranks are not the best for this objective (Cor 3). The greedy solution has a bounded gap (Prop 4). The experiments use five models, from OPT-350M to LLaMA-2-7B. The tasks are GLUE, Commonsense, MedQA, GSM8K, HumanEval, C4 and PTB.

Critical strengths

S1. LoRA gains on Commonsense. In Table 3, SEA gets 72.53 vs 69.87 for uniform LoRA and 71.12 for Weight-based (r=32 budget). It gets 71.94 vs 69.30 and 69.72 (r=8 budget). I think this is the clearest evidence that the method helps.

S2. The method is simple and fast. In Table 4, the time per epoch is close to Uniform SVD (for example 280 s vs 273 s). The authors also show the gap of their solution in an honest way (§3.3, l.256-260, Prop 4).

Critical weaknesses

W1. Lack of novelty. Also, the theory is about a proxy, not about accuracy.

  • The theory is standard. The proof of Prop 4 says "The relaxed problem is a fractional knapsack problem" (App A, l.749). Alg. 1 is the usual greedy rule for this problem. Thm 2 is a Lagrangian condition for a convex problem with one budget (l.681-686). The paper does not cite any source for these.
  • The paper says WeLore allocates rank "from a min-max normalized threshold search over the pretrained weight spectrum" (§2, l.89-90). SEA is close to this idea. It also uses one global threshold over a per-layer normalized spectrum. The main changes are the gradient signal and the cost per parameter. Table 4 compares the two, but Table 1 and §2 do not discuss this link.
  • The theory is for M=G^M = \hat G (l.228). But Alg. 2 (line 3) only passes the ranks to the next step. In the reparameterization path, this step uses truncated SVD of the selected matrices (App B.2, l.898). So the kept directions are not the gradient modes that the objective counts. The only link to the loss is one first-order sentence (l.204-205), with no proof. The paper does not report how kept gradient energy relates to accuracy.
  • §1 says the problem is "a convex optimization problem" (l.39). This is true only for the relaxed problem with continuous ranks (App A, l.686). The problem that is solved, (5), is binary. §1 also says SEA "identifies optimal layer-wise ranks for efficient and accurate fine-tuning" (l.52). But l.258 says the solution is not guaranteed to be exact.

So the theory supports the rule for kept gradient energy. It does not support the accuracy gains. The accuracy claims rest on the experiments.

W2. The claim "gradients are better than weights" is not tested alone. Some numbers do not match.

  • SEA and Weight-based (Tables 2-3) differ in two things. One is the spectrum source. The other is the choice of high-loss samples (l.411-414). Only one ablation changes which samples are used (Table 10). Figure 3 only changes the amount of data. No ablation compares gradient and weight spectra with the same samples. In Table 10, gradients from random samples give 70.09. This is close to Weight-based, which gets 69.76 (Table 2).
  • Some learning rates in Table 7 are not in the grid of App B.3 (l.905). SEA and AdaLoRA use 5e-4 on CodeFeedBack. LoRA uses 5e-4 on MetaMathQA. IGU-LoRA uses 1e-2 on MetaMathQA. On CodeFeedBack, LoRA uses 5e-5 and SEA uses 5e-4. This is ten times larger. So the HumanEval gap (25.00 vs 35.36, Table 4) may partly come from the learning rate.
  • In Figure 3 (left; Phi-1.5, MedQA, 50%), the top value is 0.326. Table 4 gives 33.81 for SEA and 32.70 for EigenDamage in the same setting.
  • The paper says "Up to a 7% improvement" (l.53-54). This is 74.57 vs 67.68 (Table 2, GLUE 25%). Most of it comes from STS-B: 67.42 vs 21.45. Without STS-B, the averages are 75.59 vs 74.28.
  • The paper says the results are averaged over 3 runs and that the standard deviations were mostly at most 0.1 (l.402-403). But no standard deviation or per-run result is shown. In the reparameterization part of Table 4, three of the four SEA margins over the best baseline are only 0.60 to 1.18 points. Q3 explains why I cannot check the 3-run claim.

So the main claim has no clean test. Also, some settings, text and figures do not match the tables.

Questions for authors

Clarification requests (answers could change my assessment):

  1. Can you test the spectrum source and the sample choice separately? For example: weight spectrum, random gradients, high-loss gradients, and weight spectrum with high-loss samples. Please also run this on LLaMA-3.2-3B with LoRA.
  2. Which settings gave Table 10 and Figure 3? Please make them match Table 6 and Table 4.
  3. Please give the result of each run and the standard deviations. All 12 MedQA numbers in Table 4 are whole counts over 1,272 (for example 33.81 = 430/1272). This looks like one single run. Please also say which metrics you use for CoLA and STS-B. CoLA is 69.13 in three different rows of Table 2.
  4. Is the budget kept? The SEA ratio is 0.36 and 0.35 for 50% retention in Table 8, but 0.50 for 50% in Table 9. l.862-865 says this ratio is chosen to meet the budget. What does it measure? Alg. 1 line 14 also sets rl=max⁡(1,∑ixl,i)r_{l} = \max(1, \sum_{i} x_{l,i}) after the budget loop. Please give the real parameter counts for all methods.
  5. Why are some baselines far below Uniform SVD in Table 4? On GSM8K, MoDeGPT gets 2.35 and 0.50, and Layer-wise Indep. gets 13.80 and 2.43, while Uniform SVD gets 24.72 and 17.89. How did you set up these baselines?

Suggestions (would not change my assessment):

  • Please add Uniform and Weight-based rows to Table 5. The ascending-order test (App C, l.1007-1015) shows that a bad order fails. It does not show how much SEA gains over uniform or weight-based ranks at 7B.
  • Some settings and text do not match the tables and figures. Table 10 says LR 1e-4, but Table 6 says 1e-5 for seven GLUE tasks. The Large Loss row in Table 10 is the same as the SEA 25% row in Table 2. §3.1 (l.133-137) says layer 8 is darker red in the weight map. But in Figure 2 (left), row 8 is one of the bright rows.
  • Please report the peak memory and the calibration time for the 3B and 7B models. The abstract says "negligible additional computational and memory overhead". But the paper gives no memory numbers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.