- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes SPSO, which formulates sparse fine-tuning as an optimizer-level dynamic sparse optimization problem. SPSO introduces a Safe-Update Score (SUS) that combines a lightweight pre-trained weight-based preservation prior with online gradient-based task relevance. Experiments on several models and tasks show promising results, but some details and comparisons remain unclear.
Critical strengths
-
The method addresses a practical problem and jointly considers parameter selection, update size, and optimizer memory.
-
The experiments cover several models and tasks, with useful ablations and additional results in the appendix.
Critical weaknesses
-
Alg.1 prunes moment states before updating all active parameters, but it is unclear how parameters whose states were removed are updated. The paper also describes both periodic and per-step selection. Eq.10 uses gradient magnitudes, while Alg.1 uses signed gradients.
-
The early SPSO coverage in Fig.1(b) is below 0.1 despite the stated 10% active budget. SPIDER’s optimizer memory differs between Fig.1(a) and Tab.5 and exceeds its reported peak GPU memory. FISH Mask updates about 10% of parameters cumulatively, yet only 2.14% remain nearly unchanged.
-
Fig.1(b–d) omit Adam without explaining why. SPIDER and SPSO have substantially different update budgets in Tab.5, so a matched-budget comparison is needed.
-
SUS also reduces update sizes, and the paper should better separate the benefit of parameter scoring from that of smaller updates. Training-time results are also missing, despite the extra sampling, top-K selection, and state-management operations.
-
Fig.2 does not clearly explain the method. Parameter selection scope, sparsity terminology, and metrics for missing target scores need consistent definitions. The writing also needs careful revision.
Questions for authors
-
How are active parameters updated if their moment states have been removed? Is selection periodic or per-step, and does Eq.10 or Alg.1 reflect the actual implementation?
-
Can you explain the early coverage below 0.1 in Fig.1(b), SPIDER’s memory discrepancies, and FISH Mask’s cumulative updates versus final sparsity?
-
Why is Adam omitted from Fig.1(b–d)? Does SPSO still outperform SPIDER under a matched update budget?
-
Do the gains remain when update magnitudes are matched? What is the training-time overhead of sampling, top-K selection, and state management?
-
Could you revise Fig.2 and clarify the parameter selection scope, sparsity definitions, and calculation of metrics when target scores are missing?
General Response to Area Chair and Reviewers
We thank the Area Chair and Reviewers for their careful evaluation and constructive feedback. During the rebuttal, we conducted additional controlled experiments and uploaded a revised manuscript incorporating the new results and methodological clarifications. The key findings are summarized below.
Although these results strengthen the empirical support for SPSO, we believe the paper would benefit from clearer methodological descriptions and more systematic integration of the new analyses. We have therefore decided to withdraw the submission to further improve its completeness and rigor.
Responses to Key Concerns
1. Methodological Clarification.
We clarified SPSO's optimization procedure and distinctions from existing sparse methods:
2. Fair Comparisons with SPIDER.
The official SPIDER implementation uses importance-based gating without a fixed update budget. We conducted complementary controls on LLaVA-1.5-7B:
These controls suggest that update-budget differences alone do not explain SPSO's gains.
3. SUS Scaling and Training Controls.
We evaluated whether smaller updates or shorter training explain the improvements on LLaVA-1.5-7B and NVILA-Lite-8B:
These findings support benefits beyond global learning-rate reduction or shorter training; realized update magnitudes were not strictly matched.
4. Early Commitment and Update-Space Reallocation.
We examined fixed-support variants and the role of stochastic sampling:
These results support continued reallocation of the update space beyond stochastic exploration.
5. Efficiency and Robustness.
Efficiency was evaluated with 12-layer fine-tuning on four H20 GPUs using ZeRO-2: