acceptodds
Under review as a conference paper at ICLR 2027

StreamSelect: Sparse Fine-Tuning via Online Quantile Estimation

Abstract

Sparse fine-tuning efficiently adapts a pretrained large model to downstream tasks by updating a small number of existing weights. Choosing which weights to update, however, typically requires importance scores over the entire weight matrix, and a static choice can also become outdated as training proceeds. We introduce StreamSelect, which identifies and dynamically updates trainable weights online without full-matrix ranking. Specifically, StreamSelect derives importance scores for a candidate set sampled from the frozen weights and uses an online estimate of an upper-tail quantile to filter high-score candidates. We further design a replacement mechanism that maintains a fixed trainable budget by removing small, stabilized updates according to an online lower-tail quantile. The quantile-based selection and replacement then form a three-stage training framework. Theoretically, we characterize how score and quantile estimation errors affect the quality of selected weights, and bound the risk incurred by resetting a single update under local smoothness. Experiments on commonsense reasoning, mathematical reasoning, and code generation show that StreamSelect achieves the best average performance across all evaluated settings while updating only of model weights.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Sparse fine-tuning, parameter-efficient fine-tuning, online quantile estimation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aNhHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary
  • StreamSelect proposes a fine-tuning method that uses sparse residual updates.
  • The paper views the Top-K score selection from the perspective of selection based on quantile boundaries, where memership to top-k can be decided by comparing the score with the upper-tail quantile's boundary.
  • The training is organized into multiple rounds. During each round, the selected weights St−1\mathcal{S}_{t-1} from the previous round are trained, and the the importance for candidate weights is estimated using the empirical mean and standard deviation of the gradients as the score. The candidate weights are sampled from the full set of unused weights as Ct⊆It=I∖St−1\mathcal{C}_t \subseteq \mathcal{I}_t = \mathcal{I} \setminus \mathcal{S}_{t-1}
  • Once the sparsity budget is full, the delta weights who's value deviates by a small magnitude are reset to 0, it's optimizer state is discarded.
Critical strengths
  • The ablation experiments that show sampling can perform better than of using rull matrix for the ranking. This shows the strength of the sampling mechanism proposed by the method.
  • StreamSelect obtains higher performance than baseline sparse tuning methods and standard PEFT methods like LoRA/DoRA with a lower parameter count. This might be useful for applications such as interpretability, or serving personalized models. However, the methods must be trained and evaluated under controlled settings, not by citing numbers from prior papers as currently done.
Critical weaknesses
  • The core method is written in a manner that's hard to parse and as a result understand in depth.
  • The method does not compare training throughput/wall time and memory with LoRA. LoRA is use precisely because it is memory and time efficient, even though it can perform lower than full fine-tuning [1]. Any method comparing with LoRA must compare along these aspects as well, not just the numebr of trainable parameters.
  • The baseline scores are reported directly from prior papers. It's unclear if the training for StreamSelect is done under the same settings (e.g., same framework for training/evaluation, precision, etc.)
  • The three stage algorithm introduces a lot of hyperparameters (sparsity, replacement threshold, reservoir size), and also presents a lot of overhead and complexity, and without the training wall time/memory comparison with LoRA, it seems impractical.

--

References

[1] Biderman et al., "LoRA Learns Less and Forgets Less", TMLR 24

Questions for authors

None other than the weaknesses.

3 comments

Thank you for recognizing the sampling results and the effectiveness of sparse updates with fewer trainable parameters. Your comments helped us clarify the procedure and make its training cost explicit.

Method and training procedure (W1):

We have clarified the objective and purpose of the three stages in the revised Introduction: identify promising locations under a limited budget, refine the trainable set as importance changes, and finally fix it to let the retained updates converge. Section 3.2 now explicitly states the ideal score-guided selection criterion before explaining its online approximation. The revised presentation emphasizes the online selection framework as the main contribution, with EMA and normalization serving as an implementation choice. We hope this makes the method easier to follow.

Training cost versus LoRA (W2):

We run StreamSelect, LoRA and DoRA on Math10K with three seeds using LLaMA2-7B (7B) and LLaMA3-8B (8B) on 2$ \times $RTX Pro 5000 72G. The table reports mean accuracy (Avg., %), training time including selection and validation (Time, minutes), and per-run peak allocated memory averaged across seeds (Peak, MiB per GPU). StreamSelect achieves higher accuracy and lower peak memory on both models, with training times longer than LoRA's and shorter than DoRA's. These results and measurement details are provided in Appendix B.5 of the revised manuscript.

Method7B Avg.7B Time7B Peak8B Avg.8B Time8B Peak
LoRA53.7823.8829,04565.2122.0434,685
DoRA53.9451.4245,03966.0049.2249,506
StreamSelect56.6633.8326,73273.1332.9332,158

Reliability of the gains (W3):

We ensure that common training and evaluation settings are aligned across methods, including precision, training epochs, sequence length and decoding. The Math10K baselines are run under a shared protocol, and commonsense and code experiments follow the public settings of SMT and the settings reported by Jiang et al. [1], respectively. To further assess the reliability and significance of the gains, we reran StreamSelect's main experiments across three seeds. Updated results are reported in Sections 5.1–5.3, Tables 1–3 of the revised manuscript, with configurations and significance tests in Appendices B.3–B.4.

Practicality of our method (W4):

Although StreamSelect has many hyperparameters, most are set empirically rather than tuned independently. We set selection parameters in advance based on the expected training length, without searching over them. As with the baselines, only the learning rate is tuned, using the same grid. Longer training allows fewer candidates per round (smaller $ m $), less frequent selection (larger $ H $), or a higher grow quantile admitting fewer candidates. For example, $ m $ is 10k/40k/80k for commonsense/Math10K/code as training lengths decrease. We use a small fixed integer for the floor and set the cap to 2mρtsel2m\rho_t^{\mathrm{sel}}. Longer runs can devote a larger fraction of training to selection and replacement, because even a smaller remaining fraction still provides enough steps to train the final fixed set. Other shared settings remain fixed across tasks. Appendix B.3 explains these choices alongside the configurations. Our response to W2 above reports training time and peak memory compared with LoRA and DoRA.

Thank you again for the helpful feedback. We would be happy to clarify any remaining points.

[1] Jiang, Shuyang, Yusheng Liao, Ya Zhang, Yanfeng Wang, and Yu Wang. "Fine-tuning with Reserved Majority for Noise Reduction." The Thirteenth International Conference on Learning Representations. 2025.

Thank you for your rebuttal. I will take your response into consideration for the final score.

Thanks again for your thoughtful review. If you have any remaining concerns, please let us know and we would be happy to clarify further.

Reviewer_h86kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

StreamSelect is a dynamic sparse fine-tuning method that chooses which weights to train without scoring the full weight matrix. In each round, it samples mm candidate locations, accumulates their gradient statistics over HH steps, and admits candidates whose gradient SNR exceeds an EMA-smoothed upper-tail quantile. Once the budget BB is full, it removes updates that are stable and small, using a lower-tail quantile estimated from a reservoir of past stabilization magnitudes. Training has three stages: fill, replace, and fixed-support optimization. The theory bounds the screening error under quantile-tracking assumptions (Thm. 4.3) and the loss increase from resetting one update (Thm. 4.4). On commonsense, math, and code tasks with LLaMA2-7B and LLaMA3-8B, StreamSelect reports the best average scores (+0.5–1.8 points over the next best method) with 0.36–0.67% trainable parameters.

Critical strengths

A simple and practical selection mechanism. Replacing full-matrix ranking with a sampled, quantile-based threshold is easy to implement. It keeps the selection state at O(B+m+R)O(B+m+R) per matrix and avoids the memory cost of full-matrix statistics (26.7 GiB vs. 43.7 GiB, Fig. 3b). The ablations support both parts of the design: continued replacement beats a fixed support built from the same fill trajectory by 3.75 points (Fig. 3a), and the gain over SpIEL holds when SpIEL also uses SNR.

Consistent results with fewer trainable parameters. StreamSelect gives the best average score on all five model–task settings while using about half (reasoning) or one third (code) of the baselines' trainable parameters.

Critical weaknesses

The empirical evidence is weak relative to the claimed gains. All runs use a single seed (42), and the margins over the next-best method are small (0.5–1.8 points). Many baseline results are taken from four different papers that use different protocols; for example, the math results use 4-beam decoding and select checkpoints by validation loss. StreamSelect also has many task-specific hyperparameters (BB, mm, HH, fill/grow quantiles, floors and caps, ρrep\rho^{rep}, τ\tau, PP, λ\lambda, fixed-support start, and learning rate), but the paper does not report the baselines' tuning budgets. Given this setup, it is difficult to assess whether the gains are reliable.

A random-selection baseline is missing, and the full-ranking result makes it important. Exact full-matrix SNR ranking performs 2.15 points worse than sampling 40k candidates (Fig. 3a). The authors explain that five-step SNR is noisy, causing the extreme tail to favor noise. If the score is this noisy, much of the benefit may come from turnover rather than from the score itself. Random support with the same replacement rule and random replacement with the same fill are needed to show that quantile-based selection matters.

Limited novelty and theoretical contribution. The method is close to SpIEL: both use dynamic grow-and-drop with cheaper candidate screening. The main difference is that StreamSelect replaces gradient-magnitude prescreening with uniform sampling and a quantile threshold. Thm. 4.3 largely follows from Assumption 4.1, which assumes that the estimated quantile tracks the true quantile. Thm. 4.4 is a standard second-order Taylor bound. Neither result provides much insight into final fine-tuning quality.

Questions for authors
  1. Can you report the mean and standard deviation across at least three seeds for the main tables and rerun the copied baselines under your protocol?
  2. How do random fill and random replacement (same BB, same schedule) perform compared with StreamSelect?
  3. How do memory use and throughput compare with LoRA at the same accuracy? Sparse updates often require scatter kernels, so wall-clock cost matters.
  4. In Fig. 1, how much of the Jaccard drop is due to estimation noise in five-batch SNR rather than actual changes in importance? What is the overlap between two independent SNR estimates at the same step?
2 comments

Thank you for recognizing the selection mechanism and its effectiveness with fewer trainable parameters. Your questions helped us strengthen the comparisons and clarify the role of importance-guided selection.

Reliability of the gains (W1, Q1):

To address these concerns as well as the LoRA wall time comparison(Q3), we repeated StreamSelect's main experiments across three seeds and the Math10K baselines under shared training and evaluation settings. The updated main results are reported in Sections 5.1–5.3, Tables 1–3 of the revised manuscript. StreamSelect's Avg. mean ± sample SD over 3 seeds is 83.56 ± 0.30 and 87.06 ± 0.17 on commonsense (7B/8B), 56.66 ± 0.26 and 73.13 ± 0.63 on Math10K (7B/8B), and 59.13 ± 0.93 on code, all in percent.

We also clarify the tuning budgets for these comparisons. StreamSelect’s selection hyperparameters were set in advance according to the number of expected training steps, without a hyperparameter search. Appendix B.3 reports the configurations and their rationale. For StreamSelect and the baselines, learning rate was the only hyperparameter searched, using the same grid and checkpoint-selection criterion. Thus, these rerun comparisons use a matched learning-rate tuning budget, without an additional search over StreamSelect’s hyperparameters.

To assess the gains of StreamSelect over the second-best baseline in each setting, we report the following one-sided tests on Avg. accuracy. Each entry is StreamSelect's Avg. accuracy in a repeat minus the second-best baseline score (SMT for commonsense and SpIEL for math and code), in percentage points. Given the available computational resources, we use four seeds for Math10K and three seeds for commonsense and code. The last column reports one-sided tt-test pp-values for a positive mean difference. Details are given in Appendix B.4 of the revised manuscript. All five comparisons are significant at 0.1 level, with three also meeting the 0.05 level. Smaller pp-values indicate stronger statistical evidence for positive Avg. gains.

Task / modelRepeat 1Repeat 2Repeat 3Repeat 4pp-value
Commonsense / LLaMA2-7B+1.70+2.08+1.49--0.0047
Commonsense / LLaMA3-8B+0.34+0.37+0.07--0.0576
Math10K / LLaMA2-7B+0.42+2.19+1.48+0.600.0325
Math10K / LLaMA3-8B+1.15+1.38-0.03+0.920.0352
Code / LLaMA3-8B-Instruct+3.13+3.29-0.12--0.0999

Random-selection baselines (W2, Q2):

These controls keep other settings unchanged. Random-Grow shares the entire fill trajectory, then randomizes incoming locations within each candidate window; Random-Fill/Grow also randomizes fill; and Random-Fill/Grow/Retire additionally randomizes outgoing locations within the stability-qualified pool, removing the preference for small update magnitudes.
Each additional randomization lowers both averages in these runs, supporting the selection and retirement criteria. The scores retain useful signal despite sampling noise, and turnover alone does not explain the gains.

LLaMA2-7B/Math10KAvg.Weighted Avg.
StreamSelect56.8554.96
Random-Grow56.0553.93
Random-Fill/Grow55.9153.75
Random-Fill/Grow/Retire52.5250.05

Novelty and theoretical role (W3):

StreamSelect builds on the objective shared by SHiRA and SpIEL: identify locations most worth training under a limited budget and give their updates time to converge. SHiRA selects once, while SpIEL revises the trainable set. Our focus is to approximate this importance-guided oracle accurately at lower cost through online quantile boundaries from random candidates and historical stabilization records.

The theory serves this objective. Theorem 4.3 quantifies how score and boundary errors affect acceptance of important candidates and rejection of unimportant ones. Appendix A.4 supports the EMA tracking assumption with sampling-error and drift-lag bounds under stated conditions. Theorem 4.4 relates the retirement threshold to immediate removal cost. We clarify the shared objective and our contribution in the revised Introduction and Section 3.2.

Training cost versus LoRA (Q3):

We compare StreamSelect with LoRA and DoRA on Math10K using LLaMA2-7B (7B) and LLaMA3-8B (8B), with all results averaged over three runs on 2$ \times $RTX Pro 5000 72G. Avg. denotes average accuracy (%), Time denotes training time in minutes including selection and validation, and Peak denotes the mean per-run peak allocated memory in MiB per GPU. The results below show higher accuracy and lower peak allocated memory for StreamSelect on both models, with training times longer than LoRA's and shorter than DoRA's.

Method7B Avg.7B Time7B Peak8B Avg.8B Time8B Peak
LoRA53.7823.8829,04565.2122.0434,685
DoRA53.9451.4245,03966.0049.2249,506
StreamSelect56.6633.8326,73273.1332.9332,158

Motivation figure (Q4):

Thanks for pointing this out. The original probe used the same 5 fixed calibration batches at each probed model state, with identical examples and dropout masks, not 5 random batches. Also, to reduce possible small-sample estimation error, we expanded the probe to 100 fixed batches and obtained the same qualitative result. The updated results and protocol are provided in Fig. 1 and Appendix B.7 of the revised manuscript.

We hope these additions clarify the reliability and scope of our results. Please let us know if any questions remain.

Reviewer_5S3tOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a dynamic sparse fine-tuning method that leverages online upper-tail and lower-tail quantile estimation, and demonstrates through experiments the effectiveness of this online weight selection approach.

Critical strengths
  1. The motivation is clear. The observations presented in the introduction directly lead to the rationale for online selection.
  2. The methodological design is straightforward and sound. By employing upper-tail and lower-tail quantile estimation, the approach enables online selection while mitigating the computational and memory overheads associated with ranking the importance of an entire matrix.
  3. Experiments demonstrate the effectiveness of the proposed method, and ablation studies further validate the efficacy of the proposed mechanism.
Critical weaknesses
  1. Previous research has already explored threshold-based sparse selection and dynamic weight replacement strategies. Although StreamSelect introduces online quantile estimation, its specific advantages over simpler strategies (such as sampling-based Top-k selection) remain unclear. Could the authors further clarify the methodological differences and advantages?
  2. Normalized EMA appears to be a core component of the proposed selection mechanism, however, the paper directly adopts a fixed value of β = 0.2. Could the authors compare the current approach against a version where β = 1 (i.e., using only the quantiles from the current iteration without historical smoothing) and a version using unnormalized EMA?
  3. The paper estimates the lower-tail quantile using the first-stabilization magnitude from the past. It is not clear why this historical distribution is superior to a distribution derived from ongoing, stabilized updates. Could the authors provide further clarification or experimental comparisons?
Questions for authors

See Weaknesses.

3 comments

Thank you for recognizing the motivation and effectiveness of online selection. Your suggestions helped us clarify the contribution and test the estimator choices directly.

Contribution and sampled Top-k (W1):

Our contribution is a framework for discovering and revising sparse trainable locations through online quantile estimation. It uses random candidate observations to estimate the current upper-tail selection boundary, and historical stabilization records to estimate the lower-tail retirement boundary. To our knowledge, prior sparse FT has not used this joint formulation. Sampled Top-k fits directly within this framework: each arriving candidate batch supplies an empirical upper-tail cutoff. This is a simple batchwise online quantile procedure, recovered by setting $ \beta=1 $ in our implementation. Thus the comparison with sampled Top-k is a specific case of the online estimator within StreamSelect.

The benefit of $ \beta<1 $ depends on the candidate-window size. With a large window, $ \beta=1 $ can already estimate the quantile accurately; with smaller windows, averaging recent estimates can reduce sampling variance [1, Sec. 2.1]. We test this on LLaMA2-7B/Math10K with 40k, 8k and 2.5k candidates, and Section 5.4 “Online Quantile Smoothing” of our revised manuscript gives the implementation details.

CandidatesAvg., $ \beta=0.2 $Avg., $ \beta=1 $
40k56.8556.94
8k55.8454.72
2.5k55.9055.13

The results follow this pattern: $ \beta=1 $ and $ 0.2 $ perform similarly at 40k, while smoothing improves Avg. by 1.12 and 0.77 points at 8k and 2.5k, respectively.

Normalization within the framework (W2):

Candidate scores share training batches, and their scale can change between windows. Smoothing raw quantiles mixes these scales. Normalization expresses each quantile relative to its window's mean score before smoothing, then restores the current scale. Both normalized and unnormalized EMA fit our framework: normalization is an optional implementation choice for handling scale changes. The ablation below shows improvements in both Avg. and Weighted Avg. with normalization in this setting.

Retirement reference and current-set Bottom-k (W3):

Retirement aims to identify admitted locations with low learned importance. We use accumulated update magnitude after stabilization as a proxy, comparing locations that have been admitted and stabilized, including those since removed. Each stay contributes one record when the update first stabilizes, giving the historical first-stabilization distribution. Using only the current stable set instead makes importance relative to the survivors: deleting small updates can raise the next cutoff even when no magnitudes change. First stabilized records avoid this effect. The ablation below supports this reference choice.

Under the same 40k configuration, the ablations give accuracy (%):

VariantAvg.Weighted Avg.
StreamSelect56.8554.96
Unnormalized EMA56.4254.39
Current-stable retirement reference53.9352.05

We hope these clarifications and additional results address your concerns, and we welcome any further questions.

[1] Chen, Fei, Diane Lambert, and José C. Pinheiro. "Incremental quantile estimation for massive tracking." Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data mining. 2000.

I appreciate the authors for adding the relevant experiments, which address my concerns to some extent. However, an issue mentioned in W1 remains, preventing me from raising my score.

I do agree with the authors' point that "sampled Top-k" can be viewed as a special case of the online estimator where β = 1. This implies that, in terms of the selection rule, the advantage of this method over sampled Top-k is essentially the advantage of using β = 0.2 instead of β = 1. Notably, this advantage is only significant in the 2.5k and 8k scenarios. When m = 40k (the setting used for the main experiments in the paper), there is virtually no difference between the two. I will therefore maintain my current score.

Thank you for your valuable feedback and for taking the time to review our additional experiments.

We would like to further clarify that the online mechanism in StreamSelect applies not only to selection but also to retirement. As shown in our previous rebuttal (W3), replacing our online quantile-based retirement algorithm with Top-k selection for retirement reduces the average accuracy from 56.85% to 53.93%, a drop of 2.92 percentage points.

MethodAvg.Weighted Avg.
StreamSelect56.8554.96
Top-k selection for retirement53.9352.05

These results indicate that the online retirement mechanism also plays an important role in the overall performance of StreamSelect.

We would also like to clarify that our commonsense reasoning experiments use 10k candidates. Due to the limited time remaining in the discussion period, we are unable to complete the corresponding comparison with sampled Top-k at this stage. We will include this experiment in a future revision of the paper.

Thank you again for your time and consideration. If you have any further questions, please feel free to let us know.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.