- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes SOMA, a zeroth order (ZO) pretraining recipe that splits a byte level language model into small LSTM experts. Each of experts is trained independently with a ZO optimizer on its own tf-idf cluster of FineWeb-Edu. At inference time, the predictions of the top-k routed experts are averaged. The motivation is that the variance of ZO gradient estimates grows with the number of parameters being optimized.
Critical strengths
-
The idea is simple and well motivated. It addresses the core weakness of ZO (variance grows with dimension) through the architecture rather than the estimator.
-
The variance analysis is clean and closely matches the measurements.
-
Reproducibility is strong (code, checkpoints, plotted data), and the limitations are reported honestly, including a large gap to backpropagation.
Critical weaknesses
Scale and architecture. The largest model has 8M parameters, and at N=256 each expert has only 32k. This is much smaller than earlier ZO pretraining work that the paper itself cites: a 20M model (Allaire et al., 2025), GPT-2 Small trained with KronZO, and recurrent models scaled to a billion parameters (Chaubard & Kochenderfer, 2025). At this scale the resulting language models are far from useful. So it is hard to see what a practitioner could do with the method today, or whether the advantage survives as models grow. The choice of LSTM experts is justified by the transformer KV cache. LSTMs are rarely used for language modeling today, and the paper tests neither transformers nor modern recurrent models such as Mamba, RWKV or xLSTM. It is unclear whether the gains carry over to these architectures or to larger models. A transformer experiment and at least one larger scale run would greatly strengthen the generality claim.
I may be missing something, and the paper may well contain a valuable contribution, but it is hard for me to see how the method could be used for current tasks and challenges. I would appreciate it if the authors could clarify the intended use case.
Questions for authors
Line 72: please add "our method" before introducing SOMA, so the reader knows it is the proposed method and not related work.
Line 245: the reference to Table 1 is not clickable.