- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates whether high-performing random perturbations in the neighborhood of a pretrained model continue to exist within sparse coordinate subspaces. SuPES selects a subset of the lowest-magnitude parameters in each layer, applies Gaussian perturbations to them, and then selects and ensembles candidate models based on their training-set performance. It also compares perturbations over randomly selected, highest-magnitude, and full-space parameters, and proposes a norm-matching heuristic that scales the noise inversely with the square root of the perturbed parameter fraction. The main finding is that perturbing only about 1% of the parameters can still produce useful candidates and competitive ensemble performance.
Critical strengths
The research question investigated in this paper is interesting and potentially meaningful.
Critical weaknesses
-
Please include models from a wider range of model families and consider evaluating larger models, such as 70B-parameter models.
-
At the algorithmic level, SuPES mainly adds a fixed parameter mask before applying RandOpt’s random perturbations and adjusts the noise scale according to the fraction of perturbed parameters. Therefore, the algorithmic contribution appears rather limited.
-
Since the proposed method is primarily motivated by empirical observations, more extensive experiments are needed to provide convincing support for the conclusions.
-
The experimental evaluation lacks comparisons with other relevant baseline methods (only RandOpt for current version).
Questions for authors
-
Do Sparse Neural Thickets still exist when the fraction of perturbed parameters is reduced to substantially less than 1% or on larger models?
-
How many computational resources were used for the noise-scale search? Is this additional cost included in the sampling budgets reported in Figure 4?
-
Compared with RandOpt, what are the actual runtime and peak GPU memory consumption of SuPES?
Q2 – Computational resources for the noise-scale search; included in Figure 4? We thank the reviewer for this question. For the newly added models, we select noise scales with a coarse grid search on the training data. Each order of magnitude is split into a coarse grid of four points, {1, 3, 5, 7} × 10⁻ᵏ, for Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct, and five points, {1, 2, 3, 5, 7} × 10⁻ᵏ, for OLMo3-7B-Instruct. Guided by the norm-preserving heuristic (Eq. 4), we start the search in the 10⁻⁴ decade for RandOpt and in the 10⁻³ decade for SuPES (1%), since perturbing 1% of the parameters predicts a 10× larger scale. For OLMo2-1B-Instruct we used a grid of four points, {1,3,5,7}×10⁻ᵏ, choosing the same k as the larger models, while also including k+1. For each grid point, we evaluate the mean reward of 200 randomly sampled perturbations on 200 training examples. We select the scale at which this mean train reward peaks, i.e., both neighboring grid points yield a lower mean reward, and then add one neighboring grid point on each side. Across these models and all tasks, the peak lies in the 10⁻⁴ decade for RandOpt and in the 10⁻³ decade for SuPES (1%).
The search therefore requires 800 (Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct), 1,000 (OLMo3-7B-Instruct) or 1,600 (OLMo2-1B-Instruct) perturbation evaluations on 200 training examples, compared with roughly 5,000 candidates evaluated in the main run, i.e., roughly 14–24% of the total evaluation cost. For Qwen2.5-1.5B-Instruct on GSM8K, the noise scale was instead taken from the more extensive analysis in Section 4.1. These sweeps, which characterize how solution density depends on the noise scale, should be considered as analysis. More than 30 noise scales with 1,000 samples each on 200 training questions were run for each of RandOpt and SuPES. For using SuPES in practice, the coarse grid search described above is sufficient.
This cost is not included in the sampling budgets in Figure 4, whose exact values are given in Table A.10:
Table A.10: Top-50 test accuracy (%) for OLMo3-7B-Instruct as a function of the number of sampled perturbations in the candidate pool (columns).
The search procedure is the same for SuPES and RandOpt (same grid and number of evaluations), so including its cost would shift the curves of both methods equally and leave their comparison unchanged. Moreover, the norm-preserving heuristic transfers a scale calibrated for one method or perturbation fraction to another, which allows the search to be narrowed further around the predicted value. The scales selected for the 7–8B models are:
Selected noise scales for the 7–8B models: the scale with the peak mean training reward on the coarse grid (bold) and its two grid neighbors.