acceptodds
Under review as a conference paper at ICLR 2027

Sparse Neural Thickets: Sampling the Small Weights Subspace

Abstract

The loss landscape around pretrained models contains dense neighborhoods of task-improving solutions, known as _Neural Thickets_. We show that this structure persists even when perturbations are restricted to low-dimensional parameter subspaces, revealing _Sparse Neural Thickets_. Strikingly, subspaces of low-magnitude parameters are consistently enriched with high-performing solutions compared to subspaces of large-magnitude parameters and dense subspaces. This suggests that favorable regions of the local loss landscape are concentrated in an easily identifiable subset of parameters. Leveraging this structure, we introduce **Su**bspace **P**arameter-**E**fficient **S**ampling (SuPES). The higher density of good solutions enables more efficient exploration: SuPES achieves higher accuracy at comparable sampling budgets or comparable accuracy with fewer samples, while perturbing substantially fewer parameters than full-space sampling. Experiments across language models, parameter scales, and datasets establish Sparse Neural Thickets and demonstrate their potential for parameter- and sample-efficient sampling around pretrained models.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Neural Thickets, Parameter Efficient Fine-Tuning, Ensembles, Model Sampling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YXmDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether high-performing random perturbations in the neighborhood of a pretrained model continue to exist within sparse coordinate subspaces. SuPES selects a subset of the lowest-magnitude parameters in each layer, applies Gaussian perturbations to them, and then selects and ensembles candidate models based on their training-set performance. It also compares perturbations over randomly selected, highest-magnitude, and full-space parameters, and proposes a norm-matching heuristic that scales the noise inversely with the square root of the perturbed parameter fraction. The main finding is that perturbing only about 1% of the parameters can still produce useful candidates and competitive ensemble performance.

Critical strengths

The research question investigated in this paper is interesting and potentially meaningful.

Critical weaknesses
  1. Please include models from a wider range of model families and consider evaluating larger models, such as 70B-parameter models.

  2. At the algorithmic level, SuPES mainly adds a fixed parameter mask before applying RandOpt’s random perturbations and adjusts the noise scale according to the fraction of perturbed parameters. Therefore, the algorithmic contribution appears rather limited.

  3. Since the proposed method is primarily motivated by empirical observations, more extensive experiments are needed to provide convincing support for the conclusions.

  4. The experimental evaluation lacks comparisons with other relevant baseline methods (only RandOpt for current version).

Questions for authors
  1. Do Sparse Neural Thickets still exist when the fraction of perturbed parameters is reduced to substantially less than 1% or on larger models?

  2. How many computational resources were used for the noise-scale search? Is this additional cost included in the sampling budgets reported in Figure 4?

  3. Compared with RandOpt, what are the actual runtime and peak GPU memory consumption of SuPES?

1 comment

Q2 – Computational resources for the noise-scale search; included in Figure 4? We thank the reviewer for this question. For the newly added models, we select noise scales with a coarse grid search on the training data. Each order of magnitude is split into a coarse grid of four points, {1, 3, 5, 7} × 10⁻ᵏ, for Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct, and five points, {1, 2, 3, 5, 7} × 10⁻ᵏ, for OLMo3-7B-Instruct. Guided by the norm-preserving heuristic (Eq. 4), we start the search in the 10⁻⁴ decade for RandOpt and in the 10⁻³ decade for SuPES (1%), since perturbing 1% of the parameters predicts a 10× larger scale. For OLMo2-1B-Instruct we used a grid of four points, {1,3,5,7}×10⁻ᵏ, choosing the same k as the larger models, while also including k+1. For each grid point, we evaluate the mean reward of 200 randomly sampled perturbations on 200 training examples. We select the scale at which this mean train reward peaks, i.e., both neighboring grid points yield a lower mean reward, and then add one neighboring grid point on each side. Across these models and all tasks, the peak lies in the 10⁻⁴ decade for RandOpt and in the 10⁻³ decade for SuPES (1%).

The search therefore requires 800 (Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct), 1,000 (OLMo3-7B-Instruct) or 1,600 (OLMo2-1B-Instruct) perturbation evaluations on 200 training examples, compared with roughly 5,000 candidates evaluated in the main run, i.e., roughly 14–24% of the total evaluation cost. For Qwen2.5-1.5B-Instruct on GSM8K, the noise scale was instead taken from the more extensive analysis in Section 4.1. These sweeps, which characterize how solution density depends on the noise scale, should be considered as analysis. More than 30 noise scales with 1,000 samples each on 200 training questions were run for each of RandOpt and SuPES. For using SuPES in practice, the coarse grid search described above is sufficient.

This cost is not included in the sampling budgets in Figure 4, whose exact values are given in Table A.10:

DatasetMethod7021002150030005000
GSM8KRandOpt88.7888.7888.5588.5588.62
GSM8KSuPES (1%)88.8688.9388.8689.0888.93
CountdownRandOpt90.6590.5090.4590.3090.70
CountdownSuPES (1%)89.4589.6589.8590.3590.35
OlympiadBenchRandOpt58.2358.2358.2358.2359.494
OlympiadBenchSuPES (1%)60.5561.6061.6060.5562.025

Table A.10: Top-50 test accuracy (%) for OLMo3-7B-Instruct as a function of the number of sampled perturbations in the candidate pool (columns).

The search procedure is the same for SuPES and RandOpt (same grid and number of evaluations), so including its cost would shift the curves of both methods equally and leave their comparison unchanged. Moreover, the norm-preserving heuristic transfers a scale calibrated for one method or perturbation fraction to another, which allows the search to be narrowed further around the predicted value. The scales selected for the 7–8B models are:

ModelTaskRandOptSuPES (1%)
Qwen2.5-7B-InstructGSM8K3×10⁻⁴, 5×10⁻⁴, 7×10⁻⁴3×10⁻³, 5×10⁻³, 7×10⁻³
Qwen2.5-7B-InstructCountdown5×10⁻⁴, 7×10⁻⁴, 1×10⁻³5×10⁻³, 7×10⁻³, 1×10⁻²
Qwen2.5-7B-InstructOlympiadBench7×10⁻⁵, 1×10⁻⁴, 3×10⁻⁴7×10⁻⁴, 1×10⁻³, 3×10⁻³
Llama-3.1-8B-InstructGSM8K3×10⁻⁴, 5×10⁻⁴, 7×10⁻⁴7×10⁻⁴, 1×10⁻³, 3×10⁻³
Llama-3.1-8B-InstructCountdown5×10⁻⁴, 7×10⁻⁴, 1×10⁻³1×10⁻³, 3×10⁻³, 5×10⁻³
Llama-3.1-8B-InstructOlympiadBench5×10⁻⁴, 7×10⁻⁴, 1×10⁻³1×10⁻³, 3×10⁻³, 5×10⁻³
OLMo3-7B-InstructGSM8K1×10⁻⁴, 2×10⁻⁴, 3×10⁻⁴7×10⁻⁴, 1×10⁻³, 2×10⁻³
OLMo3-7B-InstructCountdown3×10⁻⁴, 5×10⁻⁴, 7×10⁻⁴1×10⁻³, 2×10⁻³, 3×10⁻³
OLMo3-7B-InstructOlympiadBench5×10⁻⁴, 7×10⁻⁴, 1×10⁻³5×10⁻³, 7×10⁻³, 1×10⁻²

Selected noise scales for the 7–8B models: the scale with the peak mean training reward on the coarse grid (bold) and its two grid neighbors.

Reviewer_J9PEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether the Neural Thicket hypothesis requires perturbing the entire parameter space of a pretrained model, or whether dense neighborhoods of task-improving solutions also exist within sparse parameter subspaces. The authors introduce Subspace Parameter-Efficient Sampling (SuPES), which restricts random weight perturbations to select parameter subspaces. They also propose a norm-preserving heuristic that adjusts the perturbation scale according to the fraction of perturbed parameters.

The authors demonstrate empirically that perturbing only 1% of the parameters can achieve performance competitive with RandOpt, while requiring fewer sampled perturbations or fewer sampled ensemble members.

Critical strengths
  1. The paper addresses a relevant problem by investigating whether task-specific experts can be discovered more efficiently within sparse parameter subspaces.
  2. The proposed SuPES method is simple, intuitive, and clearly explained. Despite its simplicity, it achieves competitive performance while perturbing substantially fewer parameters.
Critical weaknesses
  1. Several key experiments are conducted on different models without consistently replicating the analyses across architectures. For example, the perturbation-scale (Section 4.1) and solution densities (Section 4.2) experiments are primarily conducted on Qwen2.5-1.5B-Instruct, whereas the robustness-to-ensemble-size analysis is restricted to OLMo3-7B-Instruct. This makes it difficult to assess whether the reported properties of SuPES generalize consistently across models.

  2. The experiments throughout the paper do not appear to be repeated across multiple independent random seeds. Given the inherently stochastic nature of the proposed method, reporting mean performance and standard deviations across independent runs would strengthen the reliability of the reported results.

Questions for authors
  1. The introduction motivates selecting a subset of low magnitude parameters partly through their potential of introducing a lower risk of catastrophic forgetting. Have you investigated whether SuPES preserves the pretrained model's original capabilities better than RandOpt?
Reviewer_sAxnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors explore whether model performance can be improved through randomly perturbing only a small set of model parameters. This paper proposes SuPES, which perturbs low-magnitude weights and ensembles in the selected models. Experiments are conducted on Qwen2.5-1.5B-Instruct and OLMo3-7B-Instruct. The results show that perturbation using around 1% of parameters can perform comparably to full-space RandOpt.

Critical strengths
  • The paper is well organized and presented.
  • The experiment covers multiple benchmarks on different model families.
Critical weaknesses
  • The large-magnitude ratio in Table 1 seems to be 0.60/0.64 = 0.94, rather than 0.56.
  • The downstream performance validation lacks independent repeated trials and error bars. It is difficult to evaluate how reliable the reported result is.
  • It would be beneficial to report the computational cost, such as runtime and memory cost, which helps establish how practical the proposed method is.
Questions for authors

None

Comments on the paper

1 comment

Dear Area Chair,

Thank you for your time. We understand reviewers are busy, and Reviewers sAxn and YXmD have not yet had a chance to respond. Here is a short summary of the main concerns and how we addressed them.

  1. Robustness of the results (sAxn, J9PE, YXmD). All main comparisons now report confidence intervals over multiple seeds. We also added three models of different scales. SuPES remains competitive compared to RandOpt. Low-magnitude subspaces remain enriched with good solutions on most tasks. After these additions, Reviewer J9PE wrote: "These results address my main concerns about the paper", and raised their score. Sparse Neural Thickets also persist when perturbing only 0.01% of the weights. A 70B model will be added for the camera-ready version.
  2. Computational cost (sAxn, YXmD). SuPES has the same runtime and memory cost as RandOpt. The noise-scale search is modest and identical for both methods. It therefore does not affect the comparison.
  3. Novelty and baselines (YXmD). SuPES is deliberately a minimal change to RandOpt. Reviewer J9PE named this simplicity a strength: "Despite its simplicity, it achieves competitive performance while perturbing substantially fewer parameters." Our main contributions are two findings. Neural Thickets persist in sparse subspaces, and low-magnitude subspaces are consistently enriched with good solutions. We also contribute a norm-preserving heuristic that transfers noise scales across subspace sizes. Finally, SuPES is a drop-in restriction of RandOpt and is evaluated in the same setting. It therefore inherits RandOpt's comparisons against RL, evolution strategies and sampling baselines.
  4. Catastrophic forgetting (J9PE). In a new cross-task experiment, we evaluated ensembles selected on one task on the other tasks. Neither method shows forgetting, and both perform similarly. J9PE found this experiment "very interesting".

In summary, the new results strengthen our central finding. Neural Thickets persist when perturbations are restricted to sparse subspaces, revealing Sparse Neural Thickets. Easily identifiable low-magnitude subspaces are consistently enriched with high-performing solutions compared to large-magnitude and dense subspaces. This holds across model families and scales, and even at extreme sparsity. SuPES exploits this structure with a deliberately simple design. It remains competitive with RandOpt while perturbing substantially fewer parameters, at no additional cost. Together, these results establish Sparse Neural Thickets and demonstrate their potential for parameter- and sample-efficient sampling around pretrained models.

Full details are in the individual responses. Thank you for your consideration.

Kind regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.