- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
SPARK trains an over-parameterized spectral adapter of rank r, learns which spectral directions to retain through a differentiable top-k selection, and stores only the selected k directions after hardening. The method aims to improve the quality-storage trade-off of spectral PEFT methods by learning the retained subspace instead of fixing it a priori. Experiments show promising improvements over existing spectral baselines at comparable storage budgets.
Critical strengths
- The paper is clearly written and easy to follow.
- The differentiable top-k formulation provides a principled way to learn sparse spectral selection rather than relying on a fixed truncation rule.
- The method is simple and integrates naturally with the spectral adaptation framework.
- Experimental results are promising across the evaluated benchmarks and storage budgets.
Critical weaknesses
- The paper focuses primarily on stored adapter size, whereas for PEFT methods the more important concern is training-time memory, followed by runtime, with storage being the least restrictive resource. SPARK optimizes an spectral core and associated selection variables, which increases memory consumption relative to LoRA-XS.
- The retained structure is restricted to selecting rows and columns of the spectral core, yielding a Cartesian k×k block. It is unclear how expressive this structured sparsity pattern is compared to less constrained alternatives.
- The use of training rank improves storage efficiency but increases training parameters, optimizer state, and memory footprint compared to LoRA-XS. The additional soft top-k relaxation and annealing also add overhead considering the selection is not sparse during training.
- The experiments do not fully isolate the source of the gains. It remains unclear how much improvement comes from learned sparse selection versus simply training a larger spectral model before compression.
- Proposition 1 provides an approximation-error result but its role in motivating the specific design choices of SPARK is not entirely clear. There is also a minor inconsistency in the labeling, where Appendix D refers to a proof of Theorem 1 while the main result is presented as Proposition 1.
Questions for authors
- What are the actual peak training-memory and runtime overheads of SPARK compared to LoRA-XS at matched storage budgets?
- How much expressive power is lost due to the restriction to Cartesian row/column selection compared with more general sparse spectral structures?
- Could you provide an ablation that controls for training rank and trainable parameter count to isolate the contribution of the learned top-k selection mechanism?
- How does Proposition 1 guide the specific smooth top-k formulation and hardening procedure used in SPARK?