- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes Spectral Gradient Basis LoRA (SGB-LoRA), an input-conditioned low-rank adaptation method built from per-example calibration gradients. It decomposes descent directions into a shared mean and centered residuals, extracts residual PCA modes, and allocates a fixed per-matrix factor-rank budget across singular directions of the mean and residual modes using energy-aware rank allocation (EARA). Coordinates refitted on the retained low-rank bases supervise a prompt-side router during warm-up, followed by joint optimization of the router and adapter factors. At inference, the router predicts signed branch coefficients from the prompt without targets or gradients. The paper provides a fixed-coordinate calibration-reconstruction optimality result and reports results on instruction following, mathematical reasoning, code generation, and commonsense reasoning.
Critical strengths
-
The construction connects per-example gradient structure to both adapter initialization and routing supervision under a single per-matrix factor-rank budget. The distinction between shared descent directions and predictable residual variation provides a concrete motivation for conditional adaptation rather than relying only on a generic mixture of adapters.
-
The paper precisely scopes the EARA optimality result to reconstruction with fixed recovered modes, fixed PCA coordinates, and exact spectra, and discusses the limits of approximate calibration and subsequent joint training. The reported component ablations and diagnostics provide useful evidence that centered gradient structure, rank allocation, and coordinate supervision contribute within the evaluated controlled setting.
Critical weaknesses
W1. Missing comparisons with the closest conditional-adaptation and gradient-routing methods
The central contribution is not merely a better initialization for a shared LoRA update, but the construction of gradient-derived branches and supervision of input-dependent routing. However, the main experiments compare primarily with static or shared-update PEFT methods and omit the most relevant conditional alternatives discussed in Section 2, such as MoLE, HydraLoRA, MixLoRA, and MiLoRA, as well as per-example-gradient-based methods such as ELREA and GradientSpace. The architecture-matched random-basis and task-only-router ablations show that the proposed components help within the authors' system, but do not substitute for comparisons with existing conditional adaptation approaches. Without representative comparisons matched for total trainable parameters, factor-rank budget, and training budget, it remains unclear whether continuous signed spectral combinations offer a meaningful advantage over established expert routing or gradient-based grouping.
W2. Published-result comparisons do not isolate the source of the reported performance gains
The baseline task-quality results in Tables 1 and 2 are taken from prior papers rather than rerun in a common experimental pipeline, as explicitly acknowledged in Appendix F. Consequently, differences in implementation, training, data processing, and evaluation remain potential confounders in the reported gains. Moreover, the main task-quality experiments adapt all linear maps, whereas the component ablations and systems measurements use an attention-only configuration. Local runtime measurements of LoRA and LoRA-GA in Table 5 therefore do not establish a matched task-quality comparison in the main setting. These limitations do not demonstrate that the reported results are incorrect, but they weaken the attribution of the main-table improvements to SGB-LoRA itself. Common-pipeline reruns of LoRA, strong gradient-aware baselines such as LoRA-GA and LoRA-Pro, and representative conditional adapters are needed to substantiate the claimed performance advantage.
Questions for authors
-
Comparison affecting the contribution assessment: Under matched total trainable parameters, factor-rank budgets, and training budgets, how does SGB-LoRA compare with representative routed LoRA methods and the closest gradient-based alternatives, especially ELREA and GradientSpace?
-
Evidence affecting the performance assessment: Do the gains in Tables 1 and 2 persist when LoRA, LoRA-GA, LoRA-Pro, and representative conditional adapters are rerun with the same target modules, training data and preprocessing, optimization and model-selection protocol, and evaluator as SGB-LoRA?