Does Random Search Discover Task Experts? Revisiting Neural Thickets

Pretrained models are typically viewed as generalists, requiring bespoke post-training to achieve task-specific expertise. Gan & Isola (2026) recently proposed a striking alternative: that pretrained models lie in a dense “neural thicket” of task specialists, which can be surfaced with simple Gaussian weight perturbations. We examine this neural thicket of task-specific experts and identify several key findings. First, we find that the apparent lift of the best-of-$N$ selected perturbation decays quickly with the size $m$ of the selection set, strongly suggesting overfitting to the selection questions. In fact, for $10$ of the $12$ model–task pairs we study, a simple null model of pure selection noise predicts more than $75\%$ of the top expert's reward lift above the perturbation-population mean. Second, the top experts' advantage often vanishes on held-out questions, or remains within the run-to-run variability of the unperturbed "anchor" model. Third, we find that majority voting over the top-$K$ selected perturbations—coined RandOpt($N,K$) by Gan & Isola (2026)—is often matched or outperformed by majority voting over $K$ samples from the base model. Overall, our results suggest that random weight perturbations may be ineffective at uncovering task experts around pretrained models, and that adapting pre-trained models to downstream tasks may require more sophistication than random search.