acceptodds
Under review as a conference paper at ICLR 2027

Momentum Streams for Optimizer-Inspired Transformers

Abstract

The residual update of a pre-norm Transformer layer admits an interpretation as one step of a first-order optimizer acting on a surrogate token energy, wherein the attention and MLP sublayers function as gradient oracles. Based on this observation, we build a family of optimizer-inspired Transformers (triple-momentum, Adam/AdamW, Muon, SOAP) and compare them under matched compute. In our main pretraining experiment, the triple-momentum TMMFormer achieves the lowest validation loss, outperforming the vanilla Transformer and prior architectural variants. A controlled ablation and supporting theory show that momentum, not preconditioning, is the main source of the gain. We further show that TMMFormer and other momentum-based designs reach flatter minima than the vanilla Transformer, which leads to less forgetting and better generalization

Primary area:
foundation or frontier models, including LLMs
Keywords:
model architectures, pre-training, optimization methods

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yPJ7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Prior work has shown that a transformer block can be viewed as an implementations of one step of gradient descent on a surrogate loss, inspiring the proposal of alternative "optimiser-inspired" architectures derived from modelling transformer blocks on other algorithms. In this paper, the authors conduct a systematic empirical study of optimizer-inspired transformers, demonstrating that the implementation of momentum-inspired architectures robustly leads to lower validation loss, flatter minima and less forgetting after fine tuning than both vanilla and preconditioning-inspired architectures.

Critical strengths
  1. The result that momentum-inspired architectures account for the majority of gains (with preconditioner components seemingly doing very little) is intriguing and will be of interest to the community.
  2. The theoretical support is intuitive and easy to follow.
Critical weaknesses

None

Questions for authors

None of these questions will affect my assessment.

  1. Could the authors comment on how essential it is that transformers be used as the underlying architecture here? E.g. what's stopping an interpretation of residual MLP or convnets in this same optimiser-inspired fashion? Might there be a relationship between the Densenet architecture and these optimiser-inspired architectures?

  2. Spelling error on line 466: "faltter" should be "flatter".

Reviewer_cjVJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper interprets Transformer layer updates through the lens of optimization algorithms and investigates architectures that incorporate momentum and preconditioning through auxiliary streams. It compares the proposed TMMFormer with existing momentum-based architectures and a vanilla Transformer using language-model validation loss and downstream task performance. I t further analyzes Jacobians, loss curvature, and forgetting during continued training to explain the mechanisms underlying the reported improvements.

Critical strengths

The idea of incorporating optimizer state updates into Transformer architecture design is interesting. The multi-seed comparison against the vanilla baseline on TinyStories provides preliminary evidence that the proposed architecture may be promising. However, the evidence establishing the novelty and effectiveness of the specific proposal remains insufficient, and I do not currently identify a strong contribution that would support acceptance.

Critical weaknesses

In its current form, the manuscript is not sufficiently complete or validated for publication. Several passages read as an internal progress report, while important comparisons remain unresolved and conclusions extend beyond the available evidence. The following statements support this assessment:

  1. Unperformed experiments and job-management history (p.20, Appendix E.7)

The sweep is partial: only the ×0.5 Muon-LR cell ran for each architecture; the ×2 Muon-LR cells and the four pure-AdamW LR cells of the originally designed 2×2×2 grid were submitted and then cancelled, and seed count is N =1.

This reads like an internal progress report rather than a completed experimental account. The relevant information is which conditions were actually evaluated and what conclusions those results support. Describing a planned grid and cancelled jobs adds no empirical evidence, and the scientific relevance of this history is unclear without an explanation of why the experiments were cancelled.

  1. An important baseline is treated as provisional for an unspecified operational reason (p.20, Appendix E.7)

A secondary depth-18 run (18L, dmodel =768, 166.85M) was disrupted by partition-walltime requeues and is reported only as suggestive.

The statement identifies a cluster-management issue but does not explain what makes the result unreliable. Did the run fail to complete the intended training budget, was checkpoint restoration incorrect, or did the evaluation conditions differ? Interruption and resubmission alone do not establish that a result should be treated as provisional.

  1. The conclusion is maintained while a potentially consequential comparison remains unresolved (p.21, Appendix E.7)

The depth-18 Vanilla appears to close a larger fraction of the gap (best validation ≈1.130); pending a clean rerun we do not draw any conclusion from the width–depth contrast. The headline conclusion stands either way: the bulk of TMMFormer’s advantage on TinyStories is not explained by parameter count—giving Vanilla TMMFormer’s parameter budget, spent as extra width, recovers under half the gap.

The combination of “pending a clean rerun” and “The headline conclusion stands either way” is difficult to justify. The deeper Vanilla baseline approaches TMMFormer’s performance, yet this comparison is deferred while the weaker result from widening Vanilla is used to maintain the broader conclusion about parameter count. Until the depth comparison is resolved, the conclusion should be restricted to the tested width expansion; “clean rerun” also needs a concrete definition.

  1. Unmeasured reproducibility is inferred from a different dataset (p.6, Section 4.2)

We run this on TinyStories because the smaller corpus has both a tighter architectural gap and less data, so seed noise could plausibly be larger; since the noise on TS is already very small, OWT (with a larger gap) should be at least as stable.

This substitutes speculation for an unperformed reproducibility check. Small seed variation on TinyStories does not establish small seed variation on OpenWebText, and a larger observed performance gap does not imply lower variance. The appropriate statement is that robustness across seeds has not been evaluated on OpenWebText.

  1. A general claim about noise extends beyond the reported measurements (p.6, Section 4.2)

Across all four ablations the noise is small (a few thousandths of a nat) and the architectural effect is the dominant signal; full per-ablation numbers are in Appendix E.7.

Appendix E.7 explicitly states that the learning-rate sweep uses a single seed. It is therefore unclear what measurements support the claim that noise is small “across all four ablations.” The paper should distinguish conditions with measured seed variation from those evaluated only once and restrict its summary accordingly.

  1. A seed-noise threshold is asserted without supporting measurements (p.18, Appendix E.5)

All runs use a single seed (42); differences within ∼0.01 acc_norm on these benchmarks at this model scale are within seed noise and are not interpreted as signal.

The paper acknowledges using a single seed and then assigns a quantitative threshold to seed noise without providing supporting measurements or a cited basis. Treating small differences cautiously is reasonable, but an unverified difference is not the same as a difference demonstrated to fall within seed noise. This caution should also be applied consistently when discussing TMMFormer’s small advantages over YuriiFormer.

  1. Runtime conclusions exceed the comparability of the timing measurements (pp.18–19, Appendix E.6)

Per-step wall-clock for the OpenWebText runs, as printed by the training loop (hardware varies across SLURM jobs, so values are indicative).

The TMMFormer auxiliary velocity stream and per-layer scalars add negligible cost relative to attention and the MLP, so TMMFormer does not materially increase runtime over the vanilla baseline.

The first statement explicitly limits the comparability of the measurements, whereas the second draws a firm conclusion about runtime overhead. Measurements obtained on different hardware cannot isolate the runtime effect of the architectural change. A controlled comparison on the same hardware and under the same batch conditions is needed to support this claim.

  1. Undefined internal terminology appears in a figure caption (p.19, Figure 4)

OWT validation-loss training curves (real per-step values from the canonical SLURM runs).

The phrases “real per-step values” and “canonical SLURM runs” sound like internal experiment-management terminology and do not provide a reproducible description of the plotted results. What makes a run “canonical,” and how were the displayed runs selected? The caption should instead identify the configurations, seeds, evaluation intervals, and run-selection criteria.

These concerns require substantial additional validation and revision, rather than proofreading alone, and motivate my Strong Reject recommendation. I recommend completing the consequential experiments, clearly establishing the technical contribution relative to existing momentum-based architectures, and thoroughly revising and proofreading the manuscript before resubmission.

Questions for authors
  1. Do the reported improvements over YuriiFormer persist when the main comparisons are repeated across multiple seeds? For the depth-18 vanilla baseline, please clarify whether the intended training budget was completed and what uncertainty arose from interruption and resumption. Before this comparison is finalized, what supports the conclusion that parameter count does not explain most of the improvement?

  2. What performance difference remains in a controlled comparison that preserves the auxiliary embeddings, LayerNorm components, and learnable coefficients while removing only the dependence on previous states? Please also explain why a Jacobian computed with the auxiliary stream held fixed adequately characterizes perturbation propagation across layers when the full state includes momentum. Can the proposed mechanism be validated using both the full-state Jacobian and this controlled ablation?

  3. Please explain why the learning-rate sweep was partially cancelled and the SOAPFormer experiment remained incomplete, and clarify which conclusions these experiments can support. What empirical evidence supports claims of small noise in conditions evaluated with a single seed, or the extrapolation of OpenWebText stability from TinyStories results? Finally, can you provide a runtime comparison on identical hardware and under matched conditions to substantiate the claim of negligible overhead?

Reviewer_8LEZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies optimizer-inspired Transformers by interpreting residual updates as steps of optimization algorithms. The authors explore several optimizer-inspired architectures and propose TMMFormer. Experiments on TinyStories and OpenWebText show that TMMFormer achieves the lowest validation loss among the evaluated models. Through controlled ablations and theoretical analysis, the authors show that momentum, rather than preconditioning, is the main source of improvement. They further analyze residual dynamics and show that momentum-based architectures tend to reach flatter minima, exhibit less forgetting, and achieve better out-of-distribution generalization than Vanilla Transformer.

Critical strengths
  • The paper provides a comprehensive comparison of optimizer-inspired Transformer architectures. In particular, the controlled momentum–preconditioning ablation offers useful insights into which optimizer components contribute most to performance improvements, showing that momentum accounts for most of the observed gains.

  • The paper provides a theoretical interpretation of momentum as second-order filtering across Transformer depth, complemented by Jacobian spectral analysis. This offers useful insights into how momentum changes information propagation through the network.

Critical weaknesses
  • The central framework and momentum-stream design have already been established by YuriiFormer, which also studies Heavy-Ball momentum and several alternative update schemes. TMMFormer mainly introduces an additional learned velocity-reinjection coefficient, making the architectural contribution relatively incremental.

  • The observed validation-loss improvements over YuriiFormer are relatively small, and the reported multi-seed experiment only compares TMMFormer against Vanilla on TinyStories. The statistical reliability and specific contribution of the additional coefficient remain unclear.

Questions for authors
  • The theoretical analysis mainly establishes the benefit of second-order momentum dynamics over first-order residual updates, while the advantage of TMMFormer over YuriiFormer is supported primarily by an expressivity containment argument. Could the authors provide a more specific theoretical explanation of how the additional learnable coefficient (ν\nu) improves the residual dynamics beyond the fixed (ν=1\nu=1) setting of YuriiFormer?

  • While the paper reports multi-seed validation for TMMFormer versus Vanilla on TinyStories, the improvements over YuriiFormer are relatively small (0.0033 on TinyStories and 0.0071 on OpenWebText) and are based on single-seed results. Could the authors provide multi-seed comparisons between TMMFormer and YuriiFormer, reporting the mean, standard deviation, and confidence intervals to establish whether these improvements are statistically significant?

  • Appendix D.7 reports a preliminary result for 18-layer, but the experiment was not completed. Could the authors extend the experiments to deeper models (e.g., 18 layers or beyond) for Vanilla, YuriiFormer, TMMFormer, and other baselines? This would help assess whether TMMFormer's improvements on large-scale.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.