acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Mean Gradient: Spectral Allocation for Conditional Low-Rank Adaptation

Abstract

Low-rank adaptation reduces fine-tuning cost by constraining weight updates to low-rank matrices, but standard LoRA still applies the same learned update to every input. Gradient-aware methods improve this shared update using an aggregate calibration gradient, which can suppress systematic differences in the update directions preferred by individual samples. We propose Spectral Gradient Basis LoRA (SGB-LoRA), which turns this within-task gradient variation into an input-conditioned low-rank update under a fixed per-matrix rank budget. SGB-LoRA decomposes per-sample gradients into a shared mean and centered residuals, extracts reusable residual modes, and allocates rank across mean and residual directions according to both gradient energy and low-rank compressibility. The retained bases initialize LoRA branches, while coordinates refitted on the same bases supervise a lightweight prompt-side router that predicts signed branch weights. After warm-up, the router and branches are jointly optimized, and inference requires neither targets nor gradients. We show that the proposed allocation is optimal for calibration-gradient reconstruction within a fixed recovered spectral family. Across instruction following, mathematical reasoning, code generation, and commonsense reasoning, SGB-LoRA consistently outperforms standard LoRA and strong gradient-aware baselines. Further analyses show that within-task gradient variation is compressible, stable across calibration subsets, and predictable from the prompt, supporting the mechanism behind the observed gains.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Parameter-Efficient Fine-Tuning, Low-Rank Adaptation, Per-Sample Gradients, Input-Dependent Routing.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_taxLOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Spectral Gradient Basis LoRA (SGB-LoRA), an input-conditioned low-rank adaptation method built from per-example calibration gradients. It decomposes descent directions into a shared mean and centered residuals, extracts residual PCA modes, and allocates a fixed per-matrix factor-rank budget across singular directions of the mean and residual modes using energy-aware rank allocation (EARA). Coordinates refitted on the retained low-rank bases supervise a prompt-side router during warm-up, followed by joint optimization of the router and adapter factors. At inference, the router predicts signed branch coefficients from the prompt without targets or gradients. The paper provides a fixed-coordinate calibration-reconstruction optimality result and reports results on instruction following, mathematical reasoning, code generation, and commonsense reasoning.

Critical strengths
  1. The construction connects per-example gradient structure to both adapter initialization and routing supervision under a single per-matrix factor-rank budget. The distinction between shared descent directions and predictable residual variation provides a concrete motivation for conditional adaptation rather than relying only on a generic mixture of adapters.

  2. The paper precisely scopes the EARA optimality result to reconstruction with fixed recovered modes, fixed PCA coordinates, and exact spectra, and discusses the limits of approximate calibration and subsequent joint training. The reported component ablations and diagnostics provide useful evidence that centered gradient structure, rank allocation, and coordinate supervision contribute within the evaluated controlled setting.

Critical weaknesses

W1. Missing comparisons with the closest conditional-adaptation and gradient-routing methods

The central contribution is not merely a better initialization for a shared LoRA update, but the construction of gradient-derived branches and supervision of input-dependent routing. However, the main experiments compare primarily with static or shared-update PEFT methods and omit the most relevant conditional alternatives discussed in Section 2, such as MoLE, HydraLoRA, MixLoRA, and MiLoRA, as well as per-example-gradient-based methods such as ELREA and GradientSpace. The architecture-matched random-basis and task-only-router ablations show that the proposed components help within the authors' system, but do not substitute for comparisons with existing conditional adaptation approaches. Without representative comparisons matched for total trainable parameters, factor-rank budget, and training budget, it remains unclear whether continuous signed spectral combinations offer a meaningful advantage over established expert routing or gradient-based grouping.

W2. Published-result comparisons do not isolate the source of the reported performance gains

The baseline task-quality results in Tables 1 and 2 are taken from prior papers rather than rerun in a common experimental pipeline, as explicitly acknowledged in Appendix F. Consequently, differences in implementation, training, data processing, and evaluation remain potential confounders in the reported gains. Moreover, the main task-quality experiments adapt all linear maps, whereas the component ablations and systems measurements use an attention-only configuration. Local runtime measurements of LoRA and LoRA-GA in Table 5 therefore do not establish a matched task-quality comparison in the main setting. These limitations do not demonstrate that the reported results are incorrect, but they weaken the attribution of the main-table improvements to SGB-LoRA itself. Common-pipeline reruns of LoRA, strong gradient-aware baselines such as LoRA-GA and LoRA-Pro, and representative conditional adapters are needed to substantiate the claimed performance advantage.

Questions for authors
  1. Comparison affecting the contribution assessment: Under matched total trainable parameters, factor-rank budgets, and training budgets, how does SGB-LoRA compare with representative routed LoRA methods and the closest gradient-based alternatives, especially ELREA and GradientSpace?

  2. Evidence affecting the performance assessment: Do the gains in Tables 1 and 2 persist when LoRA, LoRA-GA, LoRA-Pro, and representative conditional adapters are rerun with the same target modules, training data and preprocessing, optimization and model-selection protocol, and evaluator as SGB-LoRA?

Reviewer_NXpMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SGB LoRA, which is a method that uses gradient changes per sample to improve low rank adaptation. It extracts shared and residual gradient directions, assigns ranks based on spectral information, and combines different LoRA branches using hint based routers.

Critical strengths
  1. This paper proposes an interesting idea of using gradient variations beyond the average gradient. This method has sufficient motivation and clear explanation.
  2. The experiment covers several tasks, including mathematical reasoning, code generation, and commonsense reasoning. The results of the report show sustained improvement compared to standard LoRA and several strong baselines.
  3. Research on ablation is useful. They studied the effects of gradient decomposition, rank allocation, and router supervision.
Critical weaknesses
  1. The novelty is somewhat limited. This method combines existing techniques such as PCA, spectral rank allocation, and conditional LoRA. It can better demonstrate the differences between closely related methods.
  2. Many of the baseline results in Tables 1 and 2 were taken from previous papers rather than replicated under the same experimental setup. This makes the comparison less convincing.
  3. These experiments lack direct comparison with gradient based routing methods such as GradientSpace and ELLEA.
  4. Theoretical guarantees only apply to gradient reconstruction under fixed spectral bases. It does not directly explain the improvement in downstream performance. The evaluation is also limited to the 7B-8B model.
Questions for authors
  1. Can the author directly compare with GradientSpace or other LoRA methods under the same experimental settings?
  2. How sensitive is this method to errors introduced by TensorSketch and random SVD? Will there be significant changes in the results through precise gradient decomposition?
  3. Does SGB LoRA still provide significant improvements to larger language models?
Reviewer_6hGBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Spectral Gradient Basis LoRA (SGB-LoRA), a conditional low-rank adaptation method motivated by the observation that averaging per-example gradients can obscure systematic differences in the update directions preferred by different inputs. Rather than relying on a single shared update, the method decomposes calibration gradients into a mean component and centered residuals, applies spectral analysis to identify reusable residual directions, and allocates a fixed rank budget between the mean and residual components using an energy-aware rank allocation strategy (EARA). It then fits the residual coordinates on the retained low-rank bases and trains a prompt-side router to predict these coordinates without requiring target tokens or gradients at inference time.

The paper provides both theoretical analysis of gradient reconstruction and empirical evidence supporting the proposed mechanism. Experiments on instruction following, mathematical reasoning, code generation, and commonsense reasoning show improvements over several LoRA-based baselines. Additional analyses examine residual-gradient compressibility, basis stability, routing predictability, component ablations, and computational overhead.

Critical strengths
  • A mean gradient captures the shared component of task-level adaptation but can suppress directions that are important for particular examples. Modeling these differences explicitly is a reasonable way to improve the use of a limited rank budget.
  • SGB-LoRA combines per-example gradient decomposition, spectral residual modeling, rank allocation, low-rank initialization, and prompt-conditioned routing.
  • The paper explicitly measures calibration time, training memory, fine-tuning time, and inference performance.
Critical weaknesses
  • The novelty relative to existing gradient-based conditional adaptation methods is not sufficiently established. The paper's main conceptual contribution is to derive both reusable update directions and routing targets from per-example gradients.
  • I would find the theory more compelling if it established a connection between calibration-gradient reconstruction error, router prediction error, and downstream adaptation quality under explicit assumptions.
  • The paper would benefit from evaluation on held-out examples with a clear separation between calibration, router-supervision, and final evaluation data. It should also report full-vector coordinate prediction metrics and investigate whether routing remains useful under distribution shift, different prompt formulations, or changes in task composition.
Questions for authors
  • How much of the improvement comes specifically from spectral gradient allocation?
  • Can the theoretical result be connected more directly to downstream performance?
  • Does the router learn genuinely example-specific adaptation, or does it mainly exploit broad task or prompt categories?
  • How sensitive are the results to the calibration and training protocol?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.