StreamSelect: Sparse Fine-Tuning via Online Quantile Estimation

Sparse fine-tuning efficiently adapts a pretrained large model to downstream tasks by updating a small number of existing weights. Choosing which weights to update, however, typically requires importance scores over the entire weight matrix, and a static choice can also become outdated as training proceeds. We introduce StreamSelect, which identifies and dynamically updates trainable weights online without full-matrix ranking. Specifically, StreamSelect derives importance scores for a candidate set sampled from the frozen weights and uses an online estimate of an upper-tail quantile to filter high-score candidates. We further design a replacement mechanism that maintains a fixed trainable budget by removing small, stabilized updates according to an online lower-tail quantile. The quantile-based selection and replacement then form a three-stage training framework. Theoretically, we characterize how score and quantile estimation errors affect the quality of selected weights, and bound the risk incurred by resetting a single update under local smoothness. Experiments on commonsense reasoning, mathematical reasoning, and code generation show that StreamSelect achieves the best average performance across all evaluated settings while updating only $<1\%$ of model weights.