- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
- StreamSelect proposes a fine-tuning method that uses sparse residual updates.
- The paper views the Top-K score selection from the perspective of selection based on quantile boundaries, where memership to top-k can be decided by comparing the score with the upper-tail quantile's boundary.
- The training is organized into multiple rounds. During each round, the selected weights from the previous round are trained, and the the importance for candidate weights is estimated using the empirical mean and standard deviation of the gradients as the score. The candidate weights are sampled from the full set of unused weights as
- Once the sparsity budget is full, the delta weights who's value deviates by a small magnitude are reset to 0, it's optimizer state is discarded.
Critical strengths
- The ablation experiments that show sampling can perform better than of using rull matrix for the ranking. This shows the strength of the sampling mechanism proposed by the method.
- StreamSelect obtains higher performance than baseline sparse tuning methods and standard PEFT methods like LoRA/DoRA with a lower parameter count. This might be useful for applications such as interpretability, or serving personalized models. However, the methods must be trained and evaluated under controlled settings, not by citing numbers from prior papers as currently done.
Critical weaknesses
- The core method is written in a manner that's hard to parse and as a result understand in depth.
- The method does not compare training throughput/wall time and memory with LoRA. LoRA is use precisely because it is memory and time efficient, even though it can perform lower than full fine-tuning [1]. Any method comparing with LoRA must compare along these aspects as well, not just the numebr of trainable parameters.
- The baseline scores are reported directly from prior papers. It's unclear if the training for StreamSelect is done under the same settings (e.g., same framework for training/evaluation, precision, etc.)
- The three stage algorithm introduces a lot of hyperparameters (sparsity, replacement threshold, reservoir size), and also presents a lot of overhead and complexity, and without the training wall time/memory comparison with LoRA, it seems impractical.
--
References
[1] Biderman et al., "LoRA Learns Less and Forgets Less", TMLR 24
Questions for authors
None other than the weaknesses.
3 comments
Thank you for your rebuttal. I will take your response into consideration for the final score.
Thanks again for your thoughtful review. If you have any remaining concerns, please let us know and we would be happy to clarify further.
Thank you for recognizing the sampling results and the effectiveness of sparse updates with fewer trainable parameters. Your comments helped us clarify the procedure and make its training cost explicit.
Method and training procedure (W1):
We have clarified the objective and purpose of the three stages in the revised Introduction: identify promising locations under a limited budget, refine the trainable set as importance changes, and finally fix it to let the retained updates converge. Section 3.2 now explicitly states the ideal score-guided selection criterion before explaining its online approximation. The revised presentation emphasizes the online selection framework as the main contribution, with EMA and normalization serving as an implementation choice. We hope this makes the method easier to follow.
Training cost versus LoRA (W2):
We run StreamSelect, LoRA and DoRA on Math10K with three seeds using LLaMA2-7B (7B) and LLaMA3-8B (8B) on 2$ \times $RTX Pro 5000 72G. The table reports mean accuracy (Avg., %), training time including selection and validation (Time, minutes), and per-run peak allocated memory averaged across seeds (Peak, MiB per GPU). StreamSelect achieves higher accuracy and lower peak memory on both models, with training times longer than LoRA's and shorter than DoRA's. These results and measurement details are provided in Appendix B.5 of the revised manuscript.
Reliability of the gains (W3):
We ensure that common training and evaluation settings are aligned across methods, including precision, training epochs, sequence length and decoding. The Math10K baselines are run under a shared protocol, and commonsense and code experiments follow the public settings of SMT and the settings reported by Jiang et al. [1], respectively. To further assess the reliability and significance of the gains, we reran StreamSelect's main experiments across three seeds. Updated results are reported in Sections 5.1–5.3, Tables 1–3 of the revised manuscript, with configurations and significance tests in Appendices B.3–B.4.
Practicality of our method (W4):
Although StreamSelect has many hyperparameters, most are set empirically rather than tuned independently. We set selection parameters in advance based on the expected training length, without searching over them. As with the baselines, only the learning rate is tuned, using the same grid. Longer training allows fewer candidates per round (smaller $ m $), less frequent selection (larger $ H $), or a higher grow quantile admitting fewer candidates. For example, $ m $ is 10k/40k/80k for commonsense/Math10K/code as training lengths decrease. We use a small fixed integer for the floor and set the cap to 2mρtsel. Longer runs can devote a larger fraction of training to selection and replacement, because even a smaller remaining fraction still provides enough steps to train the final fixed set. Other shared settings remain fixed across tasks. Appendix B.3 explains these choices alongside the configurations. Our response to W2 above reports training time and peak memory compared with LoRA and DoRA.
Thank you again for the helpful feedback. We would be happy to clarify any remaining points.
[1] Jiang, Shuyang, Yusheng Liao, Ya Zhang, Yanfeng Wang, and Yu Wang. "Fine-tuning with Reserved Majority for Noise Reduction." The Thirteenth International Conference on Learning Representations. 2025.