- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper aims to critique the claim from Gan & Isola (2026) that pretrained models are surrounded by a dense population of task experts that can be discovered through random perturbations to the weights. The authors propose a simple statistical model of selection effects to study how the apparent advantage of selected perturbations may arise from more simple models than the “neural thickets” as presented in the original paper. They study how the advantage changes with the number of selection questions as well as transfer to held out data, and compare the roles of ensembling in achieving the results from Gan and Isola.
The authors argue that much of the apparent expert advantage can arise from selection noise, and can be recovered from sampling noise and pure ensembling, weakening the interpretation that random search can easily discover task specialists.
I consider this an important question and a valuable direction of work. The proposed statistical explanations are sensible, and I find the simple models useful for understanding how what looks like neural thickets could arise from empirical observations. However, I am not yet fully convinced that the empirical evidence supports the full breadth of the conclusions. In particular, unresolved reproduction differences between the papers, a change in the definition of expert density, and limited analysis of perturbation scale make it difficult to determine which original claims have actually been contradicted. I believe addressing these issues would make the paper substantially more convincing.
Critical strengths
- The paper asks whether striking gains from random weight search actually justify claims about the structure of pretrained models, which is clearly an important question that requires careful testing and understanding.
- The model presented makes clear why a large best-of- advantage can appear even when candidates have no persistent differences in expected performance. Its prediction that this advantage should shrink with the number of selection questions is a meaningful contribution.
- Held-out evaluation, matched answer-budget comparisons with anchor-model majority voting, repeated anchor evaluations, and controls for weight-restoration artifacts help distinguish the value of selecting weight perturbations from other potential sources of apparent improvement.
Critical weaknesses
- The paper reports that it cannot reproduce the approximately eight-percentage-point single-expert improvement originally reported for OLMo on Countdown by Gan and Isola. This is important, but it is not clear why this is the case, and I worry that this may also impact some of the other results presented. The original paper also includes held-out ‘rank-1’ results, including Figures 9 and 11. These results may need a more careful matching of model, scoring, decoding, data splits, and perturbation settings. The authors need not reproduce every original experiment, but their conclusions should reflect which results have been reproduced. E.g. if the gains on OLMO are not reproduced but the gains on Qwen are, wouldn’t it make more sense to have Figure 2 be the equivalent result but on the Qwen models? Instead it focuses on OLMo settings with little or modest transfer. Since the paper reports substantial held-out gains for Qwen-3B and Qwen-7B on Countdown, it might be a comparison worth adding. This would help establish where the null model ceases to explain the observations and how broadly the interpretation applies.
- As I understand, Figures 6 and 7 measure the fraction of perturbations exceeding a threshold relative to the perturbation population’s own mean. The original density analysis in Gan and Isola instead uses the unperturbed model as the reference. The new statistic is useful for studying variation among perturbations, but its relevance as a test of the original density claim is insufficiently motivated. Perturbations could become increasingly concentrated around their own mean while an increasing fraction outperform the base model. Consequently, the trend in Figure 7 can coexist with the original claim.
- Perturbation scale is not sufficiently separated from model size or selection effects. I would like to see Figure 3-style held-out results separately by scale, using comparable candidate budgets and including the original reported range. It seems possible that these neural thickets do exist but we aren’t seeing them on these plots because the scale chosen is too small.Figure 7 could also benefit from an analysis varying both model size and perturbation scale: larger models might require different, potentially larger, perturbations to reveal useful variation.
- The relationship to controls and qualifications already in the original paper needs sharper treatment The original paper explicitly discusses formatting effects and includes a test-time majority-voting baseline. The critique should clarify what its new evidence adds to those analyses and explain the large differences in baseline performance where possible.
Questions for authors
If the following questions are resolved, I am happy to increase my score, since I believe they will improve the paper.
- Why were you unable to reproduce the approximately eight-percentage-point single-expert improvement for OLMo on Countdown, and could this discrepancy also affect other results presented in the paper? Have the model, scoring, decoding, data splits, and perturbation settings been carefully matched when comparing against the original held-out results?
- Given that the OLMo gains are not reproduced but substantial held-out gains are observed for Qwen-3B and Qwen-7B on Countdown, would it make sense to add the equivalent of Figure 2 for these Qwen settings? Would this help establish where the null model explains the observations and how broadly your interpretation applies?
- Why is density relative to the perturbation population’s own mean the appropriate statistic for testing the original density claim, which uses the unperturbed model as its reference? Could the trend in Figure 7 coexist with the original claim, since perturbations could become more concentrated around their own mean while an increasing fraction outperform the base model? Maybe there is a way to make this more clear in the main text.
- Could you show Figure 3-style held-out results separately by perturbation scale, using comparable candidate budgets and including the original reported range? Is it possible that useful neural thickets are not visible because the chosen scale is too small? For Figure 7, could larger models require different, potentially larger, perturbations to reveal useful variation?
- Since the original paper already discusses formatting effects and includes a test-time majority-voting baseline, can you make clear in the paper what the new evidence adds to those analyses?
4 comments
Author response (pt 2)
W2 and Q3. Density relative to the anchor
We agree with the reviewer’s criticism; alteration of the de-meaning process should be more adequately justified. Aside from our belief that population-based de-meaning is statistically cleaner than anchor-based de-meaning, a crucial component in our decision is that we observe little qualitative difference between the two methods. In particular, under either choice of reference, we find no consistent monotonic increase in density with model size.
Below we report results that demean using the mean of 5 greedy anchor evaluations, i.e., using Gan & Isola’s method. The tables report density levels for perturbations at \sigma = 1e-3, on m=200 questions, agreeing with the settings of Gan & Isola. (Note that the value of \sigma = 5e-3 stated in the original Neural Thicket paper happens to be a typo, as mentioned by the authors in their GitHub repo (https://github.com/sunrainyg/RandOpt/issues/5#issuecomment-4781029924).)
Countdown
| Threshold | Qwen 3B | Qwen 7B | Qwen 32B |
|---|---|---|---|
| 90% | 60.36 | 89.84 | 72.08 |
| 95% | 51.95 | 81.51 | 48.08 |
| 100% | 43.71 | 69.87 | 25.56 |
| 105% | 35.82 | 56.21 | 10.68 |
| 110% | 28.96 | 41.89 | 3.10 |
GSM8K
| Threshold (τ) | Qwen 3B | Qwen 7B | Qwen 32B |
|---|---|---|---|
| 90% | 90.86 | 99.85 | 99.83 |
| 95% | 71.59 | 94.84 | 97.89 |
| 100% | 27.91 | 33.57 | 63.64 |
| 105% | 2.28 | 0.26 | 2.02 |
| 110% | 0.02 | 0.00 | 0.00 |
MBPP
| Threshold (τ) | Qwen 3B | Qwen 7B | Qwen 32B |
|---|---|---|---|
| 90% | 93.53 | 99.59 | 99.96 |
| 95% | 77.24 | 86.22 | 86.71 |
| 100% | 44.74 | 34.50 | 13.19 |
| 105% | 14.63 | 3.99 | 0.06 |
| 110% | 2.40 | 0.15 | 0.00 |
We find that on Countdown, density peaks at Qwen-7B and falls at 32B at every threshold from 90% through 110%. GSM8K and MBPP show no consistent increase either: the pattern changes with the threshold and task. In the next version of the paper, we will add the complete anchor-relative analysis to the appendix and mention this comparison in the main text.
W3 and Q4. Perturbation scale
We agree with the reviewer that perturbation scale is a subtle and important point. As the reviewer correctly notes, most results in the paper aggregate over a population of N = 5000 (or N = 1000) perturbations for which each scale \sigma is drawn uniformly at random from {0.5, 1, 2}e-3. Though we made this decision intentionally to match the released RandOpt method of Gan & Isola, we acknowledge that it can somewhat obscure the precise role of sigma. (For this reason, we were mindful to present density-based results for the fixed choice of sigma=1e-3 in Figures 6 and 7.)
The table below reports the \sigma-stratified version of Figure 3; each cell records the mean held-out accuracy gain of the five highest-ranked individual experts over the anchor, in percentage points. The ± value is the standard deviation across 200 candidate subsets. Within each scale, we draw 1,000 candidates for the N = 5000 searches and 300 for the N = 1000 Qwen-32B search, then rank them on the same 200 selection questions. (There is one missing cell for which we lack the necessary held-out evaluations.)
Though distinct sigma scales can perform considerably differently for a given model–task pair, overall we find that the stratified results tell much the same story. Outside Qwen-3B and 7B on Countdown, the top five experts gain at most about 2 accuracy points on average at any evaluated scale, with several groups performing worse than the anchor.
| Model–task | σ=0.5×10^-3 | σ=1×10^-3 | σ=2×10^-3 |
|---|---|---|---|
| OLMo–Countdown | +0.61 ± 0.25 | +1.07 ± 0.25 | -0.02 ± 0.29 |
| OLMo–GSM8K | +0.03 ± 0.15 | -0.49 ± 0.15 | -0.60 ± 0.22 |
| OLMo–MBPP | +1.43 ± 0.35 | +1.53 ± 0.68 | +1.66 ± 0.60 |
| Qwen 3B–Countdown | +1.33 ± 0.79 | +4.43 ± 1.41 | +7.54 ± 2.11 |
| Qwen 3B–GSM8K | +0.56 ± 0.31 | +0.36 ± 0.32 | -1.52 ± 0.32 |
| Qwen 3B–MBPP | +0.75 ± 0.34 | +1.07 ± 0.29 | -2.39 ± 0.44 |
| Qwen 7B–Countdown | +5.55 ± 0.47 | +6.98 ± 0.39 | +6.74 ± 1.11 |
| Qwen 7B–GSM8K | +0.17 ± 0.17 | -0.35 ± 0.13 | -1.21 ± 0.27 |
| Qwen 7B–MBPP | -0.46 ± 0.36 | -1.35 ± 0.38 | -2.29 ± 0.63 |
| Qwen 32B–Countdown | +0.22 ± 0.37 | +1.68 ± 0.19 | — |
| Qwen 32B–GSM8K | +1.86 ± 0.08 | +2.01 ± 0.16 | +1.81 ± 0.06 |
| Qwen 32B–MBPP | -1.63 ± 0.15 | -1.63 ± 0.17 | -2.83 ± 0.11 |
Regarding Figure 7, we appreciate the suggestion to vary perturbation scale with model size. We note, however, that Gan and Isola use the fixed value sigma=1e-3 throughout their Section 2, including in their discovery of the “Scaling Law of the Solution Density” (their Figure 3). The original claim of Gan & Isola is therefore a fixed-sigma claim, which would itself be meaningful to challenge in its own right.
Author response (pt 3)
Nevertheless, it is worth carefully considering the relationship between perturbation scale and model size. From the previous table, we find that there is little task-independent relationship between model size and favorable perturbation scale. For instance, for Qwen-3B, raising sigma from 0.5e-3 to 2e-3 increases the top-five Countdown gain from +1.33 to +7.54, but consequently the GSM8K and MBPP gains turn negative. Qwen-32B’s GSM8K gain, meanwhile, stays near +2 points at all three scales. We thus see little evidence of a task-independent relationship between model size and a favorable sigma.
Regarding larger sigma values, we also tested the paper’s printed sigma=5e-3 before learning that it was a typo, and found it to be far too destructive for Qwen models. In particular, with 500 perturbations for each of Qwen-0.5B, 1.5B, 3B, and 32B on Countdown, GSM8K, and MBPP, we found that not a single perturbation beat the anchor on held-out data in 11 of the 12 settings, consistent with the reproduction concern raised on GitHub.
W4 and Q5. Relationship with Gan & Isola’s TT-MV Baseline
We thank the reviewer for raising this point. We begin by noting that our paper examines the broader question of how much random weight search adds to held-out performance, and what accounts for such improvement (when it exists).
In particular, our individual-expert evaluations provide a considerably more pessimistic picture than that of Gan & Isola (i.e., their figure 9). On Countdown, we find that the held-out gains of the top-ranking experts for Qwen 3B and 7B are driven primarily formatting. That is, 94% and 83%, respectively, of the performance improvement over the anchor model arises from changes in which questions receive valid expressions. The correctness difference on questions where both models produce valid expressions contributes only 0.74 and 1.91 accuracy points to their total gains of 12.16 and 11.41 points. Qwen-7B retains a smaller, statistically significant shared-question advantage.
Furthermore, our analysis of USPTO reveals a separate axis along which expert improvement can be deceptive. As USPTO has significant label imbalance, we find that top experts simply exploit such imbalance in order to outperform the anchor. Upon randomly reassigning category numbers independently for each question, the top-20 experts’ mean advantage fell from 5.1 to merely 1.1 accuracy points. (Meanwhile, anchor accuracy remains virtually unchanged under such shuffling.)
Together, we believe these experiments provide the community with a more thorough understanding of the mechanisms that can underlie expert improvements, showing how random search can favor benchmark-specific output tendencies rather than (considerably) improving problem-solving ability.
We thank the reviewer again for the careful and constructive review. We believe these changes will substantially improve the paper.
Dear Reviewer TNiM,
As the discussion period is ending in a few hours, could you please let us know if our response has addressed your concerns? We would be happy to provide further clarifications before the discussion period ends.
Best regards,
Authors of Submission 47017
Author response (pt 1)
We thank the reviewer for carefully reading the paper and for their detailed review. We are encouraged that the reviewer considers this an important question and finds the statistical models usefu, and we agree that the paper should more clearly justify changes in evaluation metrics, e.g., de-meaning using expert population rather than anchor.
W1 and Q1–Q2. Reproduction scope and the Qwen exceptions
We thank the reviewer for raising this important point. We made an extensive effort to match Gan & Isola’s experimental setup, using their official implementation, yet could not recover their headline OLMo Countdown expert gain. (Further detail below.) Furthermore, Gan & Isola’s released materials unfortunately do not include evaluation artifacts underlying their reported results. Given our efforts and the magnitude of the discrepancy, we regard this as a reproducibility issue with the original paper. Our findings, meanwhile, are directly inspectable: we provide experiment code, analysis code, and evaluation artifacts needed to regenerate every figure and table in our paper, through our official repository. We have just updated our submission with an anonymized version of the repository (https://anonymous.4open.science/r/RevisitingNeuralThickets-7C36/). We believe that our fully documented results serve as a more reliable reference point for the research community to study neural thickets than the original headline findings that we failed to reproduce.
For the direct OLMo Countdown reproduction, we matched virtually every setting that can be clearly recovered from the official repository:
allenai/Olmo-3-7B-Instruct, pinned to revision6e5971d9. This matches the named model in the original paper and released scripts, though the exact revision used for the published results is unavailable.536df0a, modified only to add logging. (The exact environment used for the original runs is unavailable.){0.5, 1, 2} × 10^-3. The original paper’s Table 3 instead reports{1, 2, 3} × 10^-3; we note this discrepancy in footnote 4 on page 14. We have also performed separate analysis for each sigma-value population, as we describe below. In a nutshell, the same behavior we report in the paper appears separately at each sigma: a general absence of “true experts”, aside from on Qwen-3B and 7B Countdown.We also corresponded directly with Gan & Isola, but were unable to identify the cause of the discrepancy, despite their assistance.
The reviewer is correct in noting that the Qwen Countdown results deserve greater prominence. However, we note that OLMo currently takes center stage for two reasons: it is Gan & Isola’s headline setting, and it represents the typical pattern in our experiments. At m=200, the selection-noise null predicts 76+% of the observed lift in 10 of 12 model–task settings. The two exceptions are precisely Qwen-3B and Qwen-7B on Countdown, with 31% and 53%.
We agree that the Qwen equivalent of Figure 2 displays distinct behavior, with a persistent expert advantage across selection set size (see below). We mention this Qwen advantage in the final paragraph of Section 3.1, but agree that the contrast deserves clearer emphasis in the revised manuscript.