acceptodds
Under review as a conference paper at ICLR 2027

SPARK: Sparse Spectral Adaptation Via Smooth Top- Selection

Abstract

Parameter-efficient fine-tuning (PEFT) methods such as LoRA, DoRA and, more recently, HiRA have become the standard way of adapting large language models to downstream tasks, as they train only a small fraction of the parameters and thereby make adaptation considerably faster and cheaper than full fine-tuning. However, every adapted task still produces an adapter that has to be stored and loaded at inference time, and in settings that require many adapters, for instance one per user, the accumulated storage and memory cost becomes a significant bottleneck. LoRA-XS addresses this problem by training only a small matrix placed between frozen factors obtained from the singular value decomposition (SVD) of the pretrained weight. Its adapters are extremely compact, but they are restricted to the leading singular directions, which are determined by the pretrained weight alone and are not necessarily the most relevant for the downstream task. In this paper, we propose SPARK, which removes this restriction. SPARK trains in a much larger subspace spanned by singular directions and simultaneously learns which of them to retain, using a differentiable top- selection that is gradually hardened during training. After training, only a matrix and the indices of the selected directions are stored, so the adapter is as small as that of LoRA-XS while being considerably more expressive. SPARK is compatible with any SVD-based update, and we apply it to both LoRA-XS and an SVD-based variant of HiRA. Experiments with Llama-3-8B on eight commonsense reasoning benchmarks show that SPARK consistently outperforms LoRA-XS-style adapters of the same size and surpasses HiRA while storing up to 88 times fewer parameters.

Primary area:
optimization
Keywords:
Parameter-efficient fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_emukOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

SPARK trains an over-parameterized spectral adapter of rank r, learns which spectral directions to retain through a differentiable top-k selection, and stores only the selected k directions after hardening. The method aims to improve the quality-storage trade-off of spectral PEFT methods by learning the retained subspace instead of fixing it a priori. Experiments show promising improvements over existing spectral baselines at comparable storage budgets.

Critical strengths
  • The paper is clearly written and easy to follow.
  • The differentiable top-k formulation provides a principled way to learn sparse spectral selection rather than relying on a fixed truncation rule.
  • The method is simple and integrates naturally with the spectral adaptation framework.
  • Experimental results are promising across the evaluated benchmarks and storage budgets.
Critical weaknesses
  • The paper focuses primarily on stored adapter size, whereas for PEFT methods the more important concern is training-time memory, followed by runtime, with storage being the least restrictive resource. SPARK optimizes an r×rr \times r spectral core and associated selection variables, which increases memory consumption relative to LoRA-XS.
  • The retained structure is restricted to selecting kk rows and kk columns of the spectral core, yielding a Cartesian k×k block. It is unclear how expressive this structured sparsity pattern is compared to less constrained alternatives.
  • The use of training rank r≥kr \ge k improves storage efficiency but increases training parameters, optimizer state, and memory footprint compared to LoRA-XS. The additional soft top-k relaxation and annealing also add overhead considering the selection is not sparse during training.
  • The experiments do not fully isolate the source of the gains. It remains unclear how much improvement comes from learned sparse selection versus simply training a larger spectral model before compression.
  • Proposition 1 provides an approximation-error result but its role in motivating the specific design choices of SPARK is not entirely clear. There is also a minor inconsistency in the labeling, where Appendix D refers to a proof of Theorem 1 while the main result is presented as Proposition 1.
Questions for authors
  • What are the actual peak training-memory and runtime overheads of SPARK compared to LoRA-XS at matched storage budgets?
  • How much expressive power is lost due to the restriction to Cartesian row/column selection compared with more general sparse spectral structures?
  • Could you provide an ablation that controls for training rank and trainable parameter count to isolate the contribution of the learned top-k selection mechanism?
  • How does Proposition 1 guide the specific smooth top-k formulation and hardening procedure used in SPARK?
Reviewer_n1fLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes SPARK, a PEFT method to reduce the storage cost of saving multiple adapters. The main idea is to a wider spectral basis of length r, and then select only k of them during training. The selection is done by using a soft differentiable top-k operator. Experiments are performed on Llama3-8B to compare the proposed method with LoRA-XS and HiRA.

Critical strengths
  1. The paper resolves the storage problem of saving multiple adapters for various tasks. The idea itself is clean and sound.
  2. The paper is well-written and easy to follow.
  3. The proposed method has good performance on single-task evaluations.
Critical weaknesses
  1. The proposed algorithm is most useful only when lots of adapters have to be saved. The single-task performance is already demonstrated in the paper. However, it would be better to include at least one experiment that tests the real setting motivating SPARK. A real test is to train one model on various tasks using SPARK and train it on the same set of tasks using other PEFT methods. The comparison should be done under the same storage costs for all methods. Here, the storage cost of SPARK should include the shared A and B.

  2. A follow-up to the above concern: the comparison in Tables 1 and 2 may not be fair either. In the single-task setting, the A and B matrices in SPARK need to be stored. But only the R matrix and indices are reported as the storage cost. A fairer way is to consider the multi-task setting and count the total storage, including A and B.

  3. In the real setting, how many adapters are commonly stored for one model during inference? The number decides the pratical impact of the proposed method.

  4. The evaluations are only done for the single Llama-8B model and on two tasks. The evaluations on more models and tasks are essential, specifically for an algorithm designed for a multi-task scenario.

Questions for authors

See weakness.

Reviewer_K7FbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SPARK, a parameter-efficient fine-tuning (PEFT) method that decouples the rank of an SVD-based adapter from its stored rank and learns the retained directions by gradient descent under a cardinality constraint. The experiments across commonsense reasoning benchmarks show that SPARK matches or outperforms strong baselines of the same size at every storage budget.

Critical strengths

This paper addresses the practical issue of accumulated storage and memory costs of PEFT in language model serving scenarios. The distinction between training capacity and deployment storage is well motivated. Selecting task-dependent directions while retaining a compact core addresses a meaningful limitation of fixed spectral truncation.

Critical weaknesses

The experiments are limited to Meta-Llama-3-8B, which is somewhat outdated, and an important baseline, PiSSA [1], is missing. Please extend the experiments to more recent model families (e.g., Qwen) and scales.

The task types are limited to commonsense and math reasoning. Does SPARK generalize to more diversified downstream tasks such as coding?

SPARK trains a wider core and repeatedly applies smooth selection across adapted layers, but the paper does not report the resulting runtime, memory use, or preprocessing overhead.

The robustness is limited because the main results are based on single runs. Variation across seeds may affect the comparative conclusions.

[1] PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models

Questions for authors

Please see the questions and comments in the Critical Weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.