- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper builds a multi-agent system that searches for Transformer pretraining optimizers. Candidate update rules are written in a typed DSL, screened for scale risk and learning-rate bounds before any training, then evaluated in a three-stage proxy cascade from 50M to 200M parameters; selected candidates are trained at 1.1B parameters against Muon. The motivating observation is that candidates which look better on the small proxies often lose that advantage at 1.1B, which the authors call proxy hacking. The paper attributes these reversals to parameter-norm-dependent amplitude scaling, supports that attribution with controlled edits and a dose-response sweep, and turns the diagnoses into admission rules. It reports eighteen designs that beat Muon on validation bpb at 1.1B under a single training seed, one of which (AnnealSink) is validated over five seeds, and shows transfer to a CIFAR-100 ViT and two other language-model configurations. The framing is recursive self-improvement: the research process is said to improve the knowledge, search space and constraints that guide the next round.
Critical strengths
- The failure analysis is causal rather than correlational, and that is the part of the paper I find convincing.
Figure 4 is the strongest evidence in the submission. Replacing AnnealSink's amplitude-correction numerator with the parameter norm degrades it severely (off-scale). Removing the additional parameter term or lowering the learning rate from 0.045 to 0.020 improves the affected candidates. Sweeping the parameter-term coefficient over {0, 0.5, 1, 2} at a fixed rate produces a monotone degradation. This connects a specific program dependency to a specific failure, and it is the kind of evidence that a correlation between program features and proxy scores could never supply. If the paper were only this section plus the 1.1B confirmation, I would argue for acceptance.
- The full-scale evaluation is real work and is reported with appropriate caution.
Training at 1.1B parameters for 21k steps against matched-seed Muon, rather than stopping at the proxy, is more than the surrounding literature usually does. The paper also keeps a rejected candidate that succeeds at the target scale (AnnealSink-S, rejected for declaring a rate of 0.025, yet improving by 1.0e-3), and reports that AnnealSink is slightly worse on a faithful GPT-2 configuration. Both are negative results that the authors could have omitted, and the limitations section is honest about the gate's conservatism.
Critical weaknesses
- The phenomenon the paper is organized around is never measured, and the one experiment that could test its thesis measures something else.
The paper claims that proxy-to-target reversal is the central obstacle in optimizer autoresearch. The evidence offered is four representative candidates in Figure 2, three of which reverse at 1.1B, plus qualitative discussion of candidate histories. No base rate is reported: among all candidates that passed the proxy cascade, how many lost their advantage at the target scale? What is that rate for Generation 1 and Generation 2 separately? Without a denominator, the paper establishes that reversals occur and can be diagnosed, not that they are the dominant failure mode that motivates the architecture.
The only cohort with target-scale labels runs against the framing. Of 28 fully evaluated designs, 17 of the 18 admitted designs beat Muon and 1 of the 10 rejected designs does, which is a 93% accurate gate and a 5.6% reversal rate among admitted candidates. The authors describe this cohort as a catalog of successful designs rather than an evaluation cohort, which is precisely the point: the denominator has been removed by curation.
The ablation is misaligned with the thesis in the same way. Table 2 reports three quantities, all of them proxy measurements (best proxy score, number of proxy wins, gate pass fraction). But the claim under test is that shared experience amplifies proxy-specific bias. To test that, one should ask whether removing shared knowledge changes the target-scale transfer rate of the candidates it produces. As reported, the ablation shows that knowledge raises the proxy score, which is the direction the paper itself identifies as suspicious. The differences involved (1.6274, 1.6262, 1.6255) also sit inside the run-to-run standard deviation of 0.0011 to 0.0023, with no test reported.
- The headline result does not survive the statistics the paper itself supplies, and the search baselines are never evaluated where it matters.
All eighteen designs in Table 4 are single-seed: the paper states that every result there uses training seed 0. The improvements are 1.0e-3 to 2.7e-3 in validation bpb. From Table 9 (paired difference 0.00260, t = 3.44, n = 5) the standard deviation of the paired difference is about 1.7e-3, which is close to the between-seed standard deviation of Muon (1.99e-3) reported in the same table. Pairing on the training seed therefore buys little variance reduction here, and a single-seed comparison at this effect size is roughly a 1.5-sigma test. The eighteen designs were then selected as winners from a much larger pool, with no multiple-comparison control, so a substantial fraction of the individual claims are expected to be noise. Only four designs have repeated-seed support at all: AnnealSink over five seeds (paired t-test, p = 0.026) and three others over three seeds each.
The one design with proper support is also the one that does not improve downstream. AnnealSink reaches a lower validation bpb than Muon but a CORE score of 26.186 against 27.349, and an identical Avg(9) of 46.09, while its BoolQ result drops by 14.8 points relative to a 9.6-point gain on CSQA. The abstract's hedge ("selected designs also improving downstream performance") is fair, but the paper's own best-supported optimizer is a validation-loss result rather than a capability result.
Separately, the count of eighteen is inflated by near-duplicate variants. E11 and E12 differ only in the weight-decay coefficient (0.25 against 0.30). E6 and E8 differ only in the constants of the momentum schedule b(t; a, h), with a = 300 against a = 400. The KronGate family contributes four entries that differ in coefficient and schedule. E1 to E4 are four decompositions of one skeleton: momentum annealing plus alternating row/column normalization. The number of independent mechanism families is closer to seven than to eighteen.
Finally, the search baselines are compared only on proxy score. Table 2 gives 1.6273 for the full system against 1.6207 and 1.6215 for OpenEvolve over the DSL and over code, a difference of about 0.4%. The most informative experiment in the paper is missing: take OpenEvolve's best candidate to 1.1B. If it reverses, the gate is justified by direct evidence; if it does not, the motivation weakens considerably. Either outcome is worth more than the comparison that is reported, and the comparison that is reported uses the metric the paper spends its introduction arguing against.
Questions for authors
1.Among all candidates that were trained at 1.1B, what fraction passed the proxy cascade and then failed to beat Muon? Please give the count and the rate by generation, including candidates that are not in the Table 4 catalog. If the target-scale labels do not exist for Generations 1 and 2, please say so explicitly.
2.How many of the eighteen designs have been evaluated on five or more seeds? For those that have, what is the result after correcting for the eighteen comparisons?
3.Which component of the admission rule carries the discriminative power? The learning-rate bound is (0, 0.02], the default declared rate is 0.02, and Muon's own best grid point is 0.014 (Table 8), so I would like to know whether the rule does more than enforce the calibrated default. Can it be tested prospectively on a pool of candidates that was not used to derive it?
4.In the ablation without shared knowledge, what is the target-scale transfer rate of the candidates produced? The current ablation reports only proxy metrics.
5.Appendix B.2 notes that the proxy and target differ in model size, training horizon, effective batch size and corpus. Can a single-axis comparison isolate scale, for instance 1.1B at a short horizon, or the largest proxy at the full 21k steps, so that the scale attribution in Section 3.3 is not confounded with the corpus and batch changes?
6.Which parts of the loop are agent-authored and which are author-authored? In particular, are the admission rules in Eq. 7 written by the specialists from diagnosed failures, or written by the authors and then versioned, and who decided the transitions from Generation 1 to 2 and from 2 to 3? I ask because the recursive self-improvement framing depends on the answer, and Section 4.3 and Appendix B.5 read differently on this point.