acceptodds
Under review as a conference paper at ICLR 2027

Scaling Zero-Order Pretraining Through Model Sharding

Abstract

Zero-order optimization (ZO) enables training without backpropagation, or retention of activations, making it relevant to forward-only hardware and non-differentiable loss, but its gradient error grows with perturbed dimension. This inhibits large model training. Sharded Optimization Mixture of Assemblies (SOMA) is an architecture designed with ZO in mind. SOMA is an ensemble of LSTM experts that train independently on clusters of data using simultaneous perturbation stochastic approximation (SPSA). Its separable loss function removes cross-expert perturbation noise at the cost of jointly learned representations across domains. Experts train independently, without exchanging gradients, activations or optimizer state. We use 80,000 estimated RTX 5090 GPU-hours to study SOMA compared to baseline methods. Modest sharding improves training compute efficiency over all tested monolithic ZO controls. We study a 8.44M model at 150 aggregate GPU-hour budget and show SOMA with 64 perturbations reaches 1.76 test nats/byte, versus 2.00–2.11 for monolithic SPSA at 64, 256 or 1,024 perturbations and 2.21 for EGGROLL. On WikiText-103, these frozen checkpoints reach 2.07, 2.25–2.36 and 2.49, respectively. We isolate the mechanism and prove that independent losses reduce relative gradient error to approximately of a shared-loss estimator's. Holding starting weights, data, perturbations and compute fixed, local rather than summed losses lower SOMA test loss by 0.035 nats/byte after 1,000 updates across three seeds. Finally, we show larger ensembles offer a separate inference benefit. At similar model size with top- routing (), SOMA achieves 2.36M tokens/s versus 257k for SOMA (, including routing), at lower test loss (1.68 versus 1.71), albeit with SOMA using as much aggregate training compute. We release all training and evaluation code and checkpoints for reproduction.

Primary area:
optimization
Keywords:
zero-order optimization, ensemble learning, language modeling

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_VYY8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes SOMA, a zeroth order (ZO) pretraining recipe that splits a byte level language model into small LSTM experts. Each of experts is trained independently with a ZO optimizer on its own tf-idf cluster of FineWeb-Edu. At inference time, the predictions of the top-k routed experts are averaged. The motivation is that the variance of ZO gradient estimates grows with the number of parameters being optimized.

Critical strengths
  • The idea is simple and well motivated. It addresses the core weakness of ZO (variance grows with dimension) through the architecture rather than the estimator.

  • The variance analysis is clean and closely matches the measurements.

  • Reproducibility is strong (code, checkpoints, plotted data), and the limitations are reported honestly, including a large gap to backpropagation.

Critical weaknesses

Scale and architecture. The largest model has 8M parameters, and at N=256 each expert has only 32k. This is much smaller than earlier ZO pretraining work that the paper itself cites: a 20M model (Allaire et al., 2025), GPT-2 Small trained with KronZO, and recurrent models scaled to a billion parameters (Chaubard & Kochenderfer, 2025). At this scale the resulting language models are far from useful. So it is hard to see what a practitioner could do with the method today, or whether the advantage survives as models grow. The choice of LSTM experts is justified by the transformer KV cache. LSTMs are rarely used for language modeling today, and the paper tests neither transformers nor modern recurrent models such as Mamba, RWKV or xLSTM. It is unclear whether the gains carry over to these architectures or to larger models. A transformer experiment and at least one larger scale run would greatly strengthen the generality claim.

I may be missing something, and the paper may well contain a valuable contribution, but it is hard for me to see how the method could be used for current tasks and challenges. I would appreciate it if the authors could clarify the intended use case.

Questions for authors

Line 72: please add "our method" before introducing SOMA, so the reader knows it is the proposed method and not related work.

Line 245: the reference to Table 1 is not clickable.

Reviewer_BtZ6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Sharded Optimization Mixture of Assemblies (SOMA), an ensemble of independently trained LSTM models on semantically clustered data for zeroth-order language-model pretraining. SOMA outperforms the tested monolithic SPSA baselines at approximately matched model size and training budget.

Critical strengths
  1. The paper demonstrates improved training efficiency over the tested monolithic SPSA baselines at approximately matched model size and compute, providing practical evidence for architectural decomposition in zeroth-order pretraining.
  2. The proposed mechanism is supported by theory and carefully controlled ablations, including paired independent-versus-summed-loss experiments and selected multi-seed comparisons. These strengthen confidence that independent losses contribute to the observed gains.
Critical weaknesses
  1. Limited evidence of generalizability. Experiments focus on small LSTM models trained on one corpus with a byte-level vocabulary and substantial data reuse. Prediction loss alone does not establish whether independent specialists preserve capabilities requiring shared representations across domains. The finite-budget comparisons also leave unclear whether SOMA’s advantage persists with longer training.

  2. Incomplete validation against alternative optimization strategies. The independent-versus-summed-loss ablation demonstrates a benefit within an already separated ensemble, but does not establish whether similar gains are achievable through blockwise perturbations in a connected model. Nor does the paper demonstrate that sharding remains beneficial with stronger estimators. These omissions limit confidence that sacrificing shared learning is a broadly useful trade-off.

Questions for authors
  1. Clarification: What determined the stopping budgets for SOMA and monolithic SPSA? Do the latest checkpoints show signs of convergence, and is there evidence that their relative ranking persists at larger budgets?

  2. Clarification: Were blockwise perturbations within a connected model evaluated? If so, how do they compare at matched compute? This would help assess whether architectural independence is needed to obtain the optimization gains.

  3. Request for additional evidence: Are there results showing that sharding remains beneficial with a stronger perturbation estimator, or beyond the current small byte-level LSTM setting? Such evidence would materially strengthen the case for generalizability.

Reviewer_HXLhOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies zeroth-order pretraining, where the relative variance of simultaneous perturbation gradient estimates grows with the number of perturbed parameters, making reliable updates increasingly expensive as models grow. The proposed system, SOMA, clusters a corpus by text similarity and trains an independent recurrent expert on each shard using simultaneous perturbation updates under its own loss, with no communication between experts. For separable objectives, independent per-shard losses reduce this relative variance roughly in proportion to the number of shards, and paired continuation experiments measured a reduction closely matching that prediction. At a fixed aggregate compute budget, the sharded system reaches clearly lower test loss than monolithic simultaneous perturbation training and than a reproduced earlier zeroth-order method.

Critical strengths
  • The main mechanism is tested directly rather than only claimed. In paired runs starting from the same weights and data, independent per-shard losses beat a summed loss on every seed, and the measured variance reduction matches the theoretical prediction closely.

  • The experimental effort is large and the work is reproducible. The authors run extensive ablations and a multi-seed comparison with equal tuning budgets, and they state that all code and checkpoints are released.

  • The authors are honest about scope. They openly concede that gradient-based training is far ahead and position the method for forward-only or non-differentiable settings.

Critical weaknesses
  • The evidence is only at a small, byte-level scale, which does not support the broad pretraining-scaling framing. The gradient-based comparison is appendix-level context rather than a matched comparison, so readers cannot tell how far this approach lags gradient training or whether the gap changes with scale.

  • The headline comparison changes several things at once: the architecture, the clustered data, and the loss design. The cleanly isolated effect of the loss design alone is much smaller, and clustering also helps under gradient training, so the source of the gain is not well attributed.

  • The monolithic zeroth-order baselines are not tuned as carefully as the proposed method. The larger-perturbation controls reuse settings from the smaller control, and the reproduced comparison method differs in precision and architecture, so the baseline comparison is not fully fair.

Questions for authors
  • How much of the headline advantage comes from the independent-loss variance mechanism, rather than from cluster specialization and the routed-ensemble architecture? What does the gap look like under IID, unclustered partitioning, where the gradient-based controls suggest part of the gain already exists without zeroth-order estimation?

  • If the larger-perturbation monolithic controls received the same search budget over learning rate and perturbation scale as the proposed method, would their loss at the main budget stay where it is reported? What happens if the learning-rate grid is extended below its current minimum, since both architectures selected the lowest value in the grid?

Reviewer_19YXOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Sharded Optimization Mixture of Assemblies (SOMA), a modular architecture and training recipe designed to overcome the dimensional gradient-variance bottleneck of zeroth-order (ZO) pretraining. It does so by by adapting a c-BTM paradigm to an ensemble of byte-level recurrent (LSTM) experts training them independently on TF-IDF corpus clusters via Simultaneous Perturbation Stochastic Approximation (SPSA). SOMA makes the global training objective separable and eliminates cross-expert perturbation noise. As the main result, authors prove theoretically and verify empirically that independent local losses reduce relative gradient estimation variance by ~1/N; they also demonstrate that modest sharding (N=2N=2) improves training-compute efficiency over monolithic SPSA and EGGROLL baselines at an 8.44M parameter scale, while heavier sharding (N=256N=256) with top-kk routing provides substantial inference throughput speedups in exchange for higher aggregate training compute.

Critical strengths

The paper presents an original synthesis of unsupervised modular sharding and zeroth-order optimization, demonstrating both theoretically and through tightly controlled continuation experiments (4.02×4.02\times measured vs. 4.00×4.00\times predicted error reduction at N=4N=4) that decomposing a model into NN independent experts with separable losses eliminates cross-expert perturbation noise and reduces relative gradient variance by approximately 1/N1/N.

Pairing constant-state byte-level LSTM experts with a weight-tied embedding/decoder head updated via an exact local delta rule avoids both the O(Tdt)O(T d_t) multi-expert KV-cache memory growth of transformers at inference and the search-space inflation of perturbing large vocabulary projections during SPSA training.

Using roughly 80,000 GPU-hours of experiments, the authors establish a clear operational trade-off between training-compute efficiency at modest sharding (N=2N=2 outperforming monolithic SPSA and EGGROLL on FineWeb-Edu and WikiText-103 at 150 GPU-hours) and active-parameter inference throughput at heavy sharding (N=256N=256 achieving 2.36M2.36\text{M} tokens/s via custom grouped Triton kernels).

Critical weaknesses

The fixed-parameter comparisons (≃8.44M\simeq 8.44\text{M} neural parameters) between the wider models (N∈1,2,8N \in {1, 2, 8}) and N=256N=256 are confounded because the wider controls include 2dEdt2 d_E d_t bottleneck projections and continuously update expert-local heads via a first-order delta rule at every step, whereas N=256N=256 uses no projections and freezes a shared head after the seed run; similarly, the EGGROLL baseline uses an external 6-layer GRU recipe with untied heads and 100-byte stream updates rather than the matched 2-layer LSTM architecture (Reference Paper: Evolution strategies at the hyperscale (EGGROLL), Bidipta Sarkar et al., 2025).

The theoretical derivations in Section 5 and Appendix F assume dense Rademacher probes, whereas the implemented algorithm uses unrescaled sparse probes (qj=1/2q_j = 1/2 on ordinary weights and qj=1q_j = 1 on LayerNorm gains) that introduce an O(1)O(1) coordinate-wise scaling bias (diag(qe)ge\text{diag}(q_e)g_e), double the relative strength of coupled weight decay on ordinary weights, and rely on first-order BPTT proxies for key architectural ablations.

The reported ≃8.44M\simeq 8.44\text{M} model sizes omit up to 6.4M6.4\text{M} fitted TF-IDF/SVD router coefficients needed for N∈8,256N \in {8, 256}, the 150 GPU-hour monolithic SPSA baseline incurs 4-GPU DDPP synchronization overhead despite fitting on 1 GPU, and the headline 9.19×9.19\times inference speedup of N=256N=256 over N=8N=8 evaluates teacher-forced scoring across different batch sizes (B=4,096B=4,096 vs. B=1,024B=1,024) where N=8N=8 is actually more than twice as slow as the N=1N=1 monolith.

The study is restricted to small 8.44M-parameter byte-level models where heavy sharding (N=256N=256) requires ~700 passes over each data shard, and as shown in Appendix Table 5, SOMA still trails exact first-order Backpropagation Through Time (BPTT) by a wide margin at matched compute budgets (1.83−2.121.83-2.12 vs. 1.01−1.181.01-1.18 validation nats/byte), a limitation that deserves greater visibility in the main text.

Questions for authors

Address the weakness points above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.