In its current form, the manuscript is not sufficiently complete or validated for publication.
Several passages read as an internal progress report, while important comparisons remain unresolved and conclusions extend beyond the available evidence.
The following statements support this assessment:
- Unperformed experiments and job-management history (p.20, Appendix E.7)
The sweep is partial: only the ×0.5 Muon-LR cell ran for each architecture; the ×2 Muon-LR cells and the four pure-AdamW LR cells of the originally designed 2×2×2 grid were submitted and then cancelled, and seed count is N =1.
This reads like an internal progress report rather than a completed experimental account. The relevant information is which conditions were actually evaluated and what conclusions those results support. Describing a planned grid and cancelled jobs adds no empirical evidence, and the scientific relevance of this history is unclear without an explanation of why the experiments were cancelled.
- An important baseline is treated as provisional for an unspecified operational reason (p.20, Appendix E.7)
A secondary depth-18 run (18L, dmodel =768, 166.85M) was disrupted by partition-walltime requeues and is reported only as suggestive.
The statement identifies a cluster-management issue but does not explain what makes the result unreliable. Did the run fail to complete the intended training budget, was checkpoint restoration incorrect, or did the evaluation conditions differ? Interruption and resubmission alone do not establish that a result should be treated as provisional.
- The conclusion is maintained while a potentially consequential comparison remains unresolved (p.21, Appendix E.7)
The depth-18 Vanilla appears to close a larger fraction of the gap (best validation ≈1.130); pending a clean rerun we do not draw any conclusion from the width–depth contrast. The headline conclusion stands either way: the bulk of TMMFormer’s advantage on TinyStories is not explained by parameter count—giving Vanilla TMMFormer’s parameter budget, spent as extra width, recovers under half the gap.
The combination of “pending a clean rerun” and “The headline conclusion stands either way” is difficult to justify. The deeper Vanilla baseline approaches TMMFormer’s performance, yet this comparison is deferred while the weaker result from widening Vanilla is used to maintain the broader conclusion about parameter count. Until the depth comparison is resolved, the conclusion should be restricted to the tested width expansion; “clean rerun” also needs a concrete definition.
- Unmeasured reproducibility is inferred from a different dataset (p.6, Section 4.2)
We run this on TinyStories because the smaller corpus has both a tighter architectural gap and less data, so seed noise could plausibly be larger; since the noise on TS is already very small, OWT (with a larger gap) should be at least as stable.
This substitutes speculation for an unperformed reproducibility check. Small seed variation on TinyStories does not establish small seed variation on OpenWebText, and a larger observed performance gap does not imply lower variance. The appropriate statement is that robustness across seeds has not been evaluated on OpenWebText.
- A general claim about noise extends beyond the reported measurements (p.6, Section 4.2)
Across all four ablations the noise is small (a few thousandths of a nat) and the architectural effect is the dominant signal; full per-ablation numbers are in Appendix E.7.
Appendix E.7 explicitly states that the learning-rate sweep uses a single seed. It is therefore unclear what measurements support the claim that noise is small “across all four ablations.” The paper should distinguish conditions with measured seed variation from those evaluated only once and restrict its summary accordingly.
- A seed-noise threshold is asserted without supporting measurements (p.18, Appendix E.5)
All runs use a single seed (42); differences within ∼0.01 acc_norm on these benchmarks at this model scale are within seed noise and are not interpreted as signal.
The paper acknowledges using a single seed and then assigns a quantitative threshold to seed noise without providing supporting measurements or a cited basis. Treating small differences cautiously is reasonable, but an unverified difference is not the same as a difference demonstrated to fall within seed noise. This caution should also be applied consistently when discussing TMMFormer’s small advantages over YuriiFormer.
- Runtime conclusions exceed the comparability of the timing measurements (pp.18–19, Appendix E.6)
Per-step wall-clock for the OpenWebText runs, as printed by the training loop (hardware varies across SLURM jobs, so values are indicative).
The TMMFormer auxiliary velocity stream and per-layer scalars add negligible cost relative to attention and the MLP, so TMMFormer does not materially increase runtime over the vanilla baseline.
The first statement explicitly limits the comparability of the measurements, whereas the second draws a firm conclusion about runtime overhead. Measurements obtained on different hardware cannot isolate the runtime effect of the architectural change. A controlled comparison on the same hardware and under the same batch conditions is needed to support this claim.
- Undefined internal terminology appears in a figure caption (p.19, Figure 4)
OWT validation-loss training curves (real per-step values from the canonical SLURM runs).
The phrases “real per-step values” and “canonical SLURM runs” sound like internal experiment-management terminology and do not provide a reproducible description of the plotted results. What makes a run “canonical,” and how were the displayed runs selected? The caption should instead identify the configurations, seeds, evaluation intervals, and run-selection criteria.
These concerns require substantial additional validation and revision, rather than proofreading alone, and motivate my Strong Reject recommendation. I recommend completing the consequential experiments, clearly establishing the technical contribution relative to existing momentum-based architectures, and thoroughly revising and proofreading the manuscript before resubmission.