acceptodds
Under review as a conference paper at ICLR 2027

MOMO: Mixture of Model Optimization

Abstract

Reinforcement learning with verifiable rewards (RLVR) can train a language model to improve its solutions on a single problem. But each model refines toward its own set of behaviors, and once it stalls, further training rarely changes its approach. We propose Mixture of Model Optimization (MOMO): instead of training one model longer, we use the solutions it has found as the starting point for training other models. Each recipient keeps its own weights, so it brings different learned behaviors to the same solutions. Across mathematical constructions, circuit placement, and qubit routing with three models of roughly 30B parameters, MOMO consistently improves on single-model training, making progress after the previous model had stalled. MOMO sets a new state of the art on circuit placement, qubit routing, and the Erdos minimum overlap problem, where its gains consistently hold across seeds. Code is available at https://anonymous.4open.science/r/momo-anon-release-38B5.

Primary area:
reinforcement learning
Keywords:
RLVR, discovery, ensembling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uPXuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MOMO for finding a single best solution to a verifiable code-optimization problem. Several language models first perform independent RLVR-based refinement. At each round boundary, the best states are transferred to the other models, which retain their own trained weights while resetting search visitation statistics and optimizer state. The motivation is that different model families favor different optimization techniques and can improve each other's solutions after individual runs stall. Experiments cover two mathematical construction problems, circuit placement, and qubit routing. MOMO reports new best results on three tasks, while AC1 improves without reaching the previous best. The paper also studies state transfer versus prompt summaries and retained versus fresh weights.

Critical strengths
  1. The method is simple and the resulting discoveries are concrete. In particular, the new Erdős minimum-overlap construction is a substantive output even though the numerical improvement is small. MOMO improves over the better individual model in all three reported seeds on this problem. The results on mathematical constructions and hardware optimization make the approach more interesting than a demonstration on only one objective.

  2. The paper provides useful evidence beyond final scores. The AC1 trajectory shows Muse moving Gemma's construction to finer grids and Qwen subsequently polishing it; the routing analysis traces changes to decay and front-layer attraction. These examples make model complementarity plausible. Table 2 also directly tests two handoff choices, and the appendices explain the evaluators, budgets, generated programs, and baseline protocols in considerable detail.

Critical weaknesses
  1. The central fixed-budget comparison remains incomplete. Figure 5 aligns MOMO and individual runs at 25 steps along a lineage, but MOMO uses 37 aggregate model-steps, versus 25 for each individual run. Taking the best of both independent 25-step runs would instead cost 50 steps, so this is not evidence that MOMO necessarily uses more total compute than an independent ensemble. Rather, the plot does not resolve which strategy is better at a common aggregate budget. A continued single-model run and an independent ensemble under the same total generation/evaluation budget would materially strengthen the main claim. A continuation with the same state pruning and search/optimizer resets would also help distinguish changing models from changing the search conditions at a round boundary. This matters because the proposed goal is the best solution within a fixed search budget.

  2. The experiments do not sufficiently isolate the contribution of heterogeneous, continuously trained models. TTT-Discover already combines online RL with a search buffer, and AlphaEvolve already exploits complementary models within evaluated program search (both are acknowledged). MOMO's particular combination is distinct, but its value should be demonstrated against a same-family population with separate seeds/adapters and the same state exchanges. Table 2 shows that retaining weights helps relative to reinitializing them; however, the fresh-weight recipients also undergo RL, so this does not isolate the benefit of continued weight updates over a frozen recipient. The retained-weight gains are relatively modest and have no reported uncertainty. The technique counts and detailed trajectories are informative, but do not substitute for these controls. I therefore remain uncertain whether the proposed combination contributes enough beyond strong search and handoff to support acceptance.

Questions for authors
  1. Can you report best-result curves against aggregate candidate evaluations and generated tokens, including every branch and round, together with total training and evaluator compute? At that budget, how does MOMO compare with longer single-model refinement and an independent ensemble? Please clarify which branches in Figure 6 are included in the search that produced the reported results.

  2. Can you compare with a same-family population that exchanges states identically, and with heterogeneous recipients whose weights are frozen at their round-1 checkpoints during subsequent rounds? Keeping the evaluator, refinement objective, handoff schedule, and search resets matched would distinguish model-family diversity from ordinary population search and continued learning.

  3. For Table 2, are the starting donor states, recipient budgets, and seeds matched between the fresh and retained conditions, and how variable are these gains? A self-continuation control applying the same top-state selection and resets would also be useful. The three-seed Erdős result is encouraging; how stable are the handoff gains on the other tasks, especially routing, where the final reported total is only 21 SWAPs below Hyra? These are requests to assess the current claims rather than suggestions to broaden the task scope.

4 comments

Thank you for the constructive feedback and interest in our paper! We address your questions below:

Compute comparison. At a matched budget of 37 model steps on the Erdos problem (~19k candidates), MOMO falls between standalone Qwen and Muse training in generated tokens, grader-hours, and training FLOPs.

ConfigurationStepsCandidatesTokens (M; k/cand)Valid evals (rate)Grader-hoursFLOPs (×10¹⁹)
R1: Qwen126,08095 (15.6)3,494 (57%)1,4221.8
R1: Muse126,14477 (12.5)5,306 (86%)1,9371.6
R2: Muse on Qwen's tree84,09650 (12.2)3,732 (91%)1,4011.0
R3: Qwen52,56052 (20.3)2,238 (87%)9631.0
MOMO total3718,88027314,770 (78%)5,7245.4
Qwen-only3718,73633113,924 (74%)5,6236.1
Muse-only3718,94421717,314 (91%)5,9444.7

Qwen's generation length increases over training, leading to higher token consumption per candidate. Grader-hours also increase as generated programs increasingly exhaust the 1,000-second evaluation time limit. Qwen's validity rate improves substantially (57% → 87%), whereas Muse starts with a much higher validity rate (86% → 91%). Here, validity is the fraction of generated programs that pass compilation and timeout checks. We expect an independent ensemble matched for the number of rollouts to exhibit similar aggregate compute characteristics to MOMO.

At this matched budget, we find the MOMO exceeds the performance of single-model training across seeds (represented as distance to a target C_5 bound of 0.38085, in units of 1e-6, lower is better):

SeedQwen@25Qwen@37Muse@25Muse@37MOMO@37 (12/8/5)
17.067.0513.5313.526.93†
218.3418.3420.4419.2018.26†
312.5612.5320.3719.2912.32†

Same-family population vs heterogenous recipients. We ran the same-family control. On a Qwen tree at step 12, we sampled one step with the tree’s own adapter, another Qwen seed’s adapter at step 12, Muse’s adapter at step 12, and the base model (same 16 parents, 512 children each, same grader). All trained adapters far exceed the base model (improvement rate 0.29–0.42 vs 0.10). We found that the average improvement rate of using another Qwen seed's adapter versus Muse's adapter was statistically indistinguishable (Qwen was 0.36e-6 while Muse was 0.32e-6), though each was larger than that of the tree's own adapter. This indicates that mixing seeds for a model can be a useful strategy for introducing variance.

Self Continuation Control. We tested this by running a variant of our experiment, whereby for Qwen at seeds 1-3 on the Erdos problem at step 12, we created two clones of the adapter in two forked threads. Then we reset the optimizer state, visitation count, and set the seed of the fork 1 to (source seed)*3+1 and fork 2 to (source seed)*3+2. We compared the performance at step 20 across three seeds as shown below. In 2/3 seeds, simply resetting the optimizer state of a run is worse than letting it continue.

SourceSource @20Its 2 clones @20Clones vs. leader
seed 17.097.59, 7.61Both worse
seed 28.539.62, 10.19Both worse
seed 39.929.11, 8.83Both better

Handoff procedure detail For Table 2, we drew these results by allowing the same donor states, recipient budget (8 steps of training), and all training conditions between all arms done on Muse and Gemma. We only ran one seed per model and used cross-model agreement to justify our design choice.

Stability in Qubit Routing For qubit routing, we tested two additional seeds as seen below. As also mentioned in Appendix D, we used a larger compute budget, because a) qubit routing is a more challenging problem than Erdos, as the model has to optimize a singly policy across three quantum topologies b) the models are still rapidly improving at step 12. We use 20 steps for round 1, 9 for round 2, and 6 for round 3 (totaling 55 between Gemma and Muse). After step 20, we see that Gemma and Muse models taper off performance, and running them to step 55 only reduces SWAPS by an average of 0.2% and 4.3% for Gemma and Muse, respectively. However running MOMO, reduces SWAPS from the R1 model, Gemma, by 0.6% on average, which is 3x that of running Gemma longer.

Our world record run came from seed 1, and our claim is not that we consistently achieve it, but that MOMO's procedure consistently gives better gains than single-model refinement.

SeedGemma@20Gemma@55Muse@20Muse@55MOMO@55 (20/9/6)
186,66486,54198,13995,01386,102
287,19486,98996,34891,23486,632
386,79286,59796,54392,15986,243

I did the review myself and received feedback from the author.
First, thanks for the additional experiments. The matched-step comparisons and the two additional routing seeds address an important part of my original concern, which show that MOMO can improve on extending a single-model run under the tested budgets.

I still share the concern raised by another reviewer about how best to spend a fixed search budget. In both Erdős seeds reported for the restart comparison, the best of three short Qwen runs slightly outperforms MOMO. The one-step adapter comparison is also useful, but it does not establish whether cross-family handoff improves on same-family exchange over multiple rounds, or whether recipients benefit from continuing to update their weights after handoff.

Some updates:

  1. "Whether recipients benefit from continuing to update their weights after handoff" - We have two experiments for this.

First a proxy experiment. We tested running historical adapters on the step 12 state-tree, and measuring the improvement rate relative to the step 12 adapter. Values represent percentage changes relative to the step-12 adapter (0% baseline).

ModelMetricBaseStep 4Step 8Step 12
QwenImprovement Rate-59%-50%-46%0%
QwenValidity-43%-31%-11%0%
MuseImprovement Rate-60%-53%-29%0%
MuseValidity-3%+4%-9%0%

We see that for Qwen, training helps improve the improvement rate and crucially its validity rate, because most programs starting out are invalid (code doesn't compile, syntax errors, timeout errors, etc.). In contrast, Muse starts out with a very good validity rate. Over the course of training from step 0 -> 12, we see that both models' improvement rate increases monotonically.

Second, at the hand-off position we compared on two seeds whether pure inference from step 12 - 20 could outweigh continued weight updates on the recipient model.

SeedDonor @12Recipient @20 (with training)Recipient @20 (without training)
Seed 17.596.977.04
Seed 218.9618.9018.91
Seed 313.9512.6813.20

In 2/3 seeds we see that training enables a larger gain to the step 12 split point, especially on seed 3.

  1. "Same-family exchange over multiple rounds vs cross-family": We provided the 1-step adapter experiment as a proxy. We are still running seeded experiments for this and will add them as they come.

  2. "Best of 3 short-runs conclusion": We agree, which is why we have continued MOMO experiments, pairing Qwen with stronger models that we can train on Tinker, like Inkling2Small a large MoE model with 17B activated, to understand whether using stronger models, even with pure inference, might be able to better demonstrate the effects of MOMO on the Erdos problem. We trained Inkling2small with 5 steps of training on R1, and also tested a no-train version that runs MOMO R2 from step 12 -> 20. Both versions outperform best of 3 draws.

SeedBest of 3 Qwen (12/12/13)MOMO@37 (12/8/5) (Qwen/Muse)New MOMO@32 (12/8) (Qwen/Inkling2small, no-train)New MOMO@32 (12/8) (Qwen/Inkling2small, 5-step R1 train)
16.896.936.8046.682
312.2812.3212.2412.08

Thank you for the follow-up. The trained-versus-frozen recipient comparison directly addresses part of my concern about continued learning after handoff. The trained recipients perform better in all three reported seeds, although the gains vary substantially.

The Inkling2Small results are encouraging, but this comparison changes both the recipient model and the search strategy. Without an independent-search or no-handoff baseline using the same models and a comparable total budget, it remains unclear how much of the improvement comes from handoff rather than access to a stronger recipient. Model-step counts alone do not establish comparable cost across these configurations.

Reviewer_nuGoOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces Mixture of Model Optimization, which transfers promising solutions between language models while retaining each model’s own trained weights. This allows models to build on one another’s approaches after individual training stalls. Experiments cover two mathematical construction problems, circuit placement, and qubit routing, with reported state-of-the-art results on three tasks.

Critical strengths

The paper is very well written, with a clear abstract and an accessible explanation of the method. Although my familiarity with this area is limited, using model diversity through solution transfer appears promising and novel to the best of my own. The ablations support transferring solution buffers and retaining model weights, while concrete examples help explain how different models contribute improvements.

Critical weaknesses

The abstract would be stronger with quantitative results, such as the placement cost reduction from 0.9533 to 0.9214, alongside the smaller routing improvement of 21 SWAPs over the previous best. A short contribution list at the end of Section 1 would also clarify the paper’s main advances. Additionally, broader repeated-seed evaluation and comparisons under equal total compute would strengthen the evidence.

Questions for authors
  • Do MOMO’s gains persist when single-model baselines receive the same total compute budget, given that the seeded comparison uses 37 model-steps for MOMO versus 25 for each baseline?

  • How consistent are the improvements across seeds on the other tasks, particularly given the small routing gain?

  • Finally, how sensitive is performance to model selection and the timing of solution transfers?

1 comment

Thank you for the constructive feedback and interest in our paper! We will adjust the abstract with concrete numbers in the final version. We address your questions below:

MOMO's consistency at matched compute We ran a matched compute experiment of MOMO, where each single model received 37 steps of training. At this matched budget, we find the MOMO exceeds the performance of single-model training across seeds (represented as distance to a target C_5 bound of 0.38085, in units of 1e-6, lower is better):

SeedQwen@25Qwen@37Muse@25Muse@37MOMO@37 (12/8/5)
17.067.0513.5313.526.93†
218.3418.3420.4419.2018.26†
312.5612.5320.3719.2912.32†

For qubit routing, we tested two additional seeds as seen below. As also mentioned in Appendix D, we used a larger compute budget, because a) qubit routing is a more challenging problem than Erdos, as the model has to optimize a singly policy across three quantum topologies b) the models are still rapidly improving at step 12. We use 20 steps for round 1, 9 for round 2, and 6 for round 3 (totaling 55 between Gemma and Muse). After step 20, we see that Gemma and Muse models taper off performance, and running them to step 55 only reduces SWAPS by an average of 0.2% and 4.3% for Gemma and Muse, respectively. However running MOMO, reduces SWAPS from the R1 model, Gemma, by 0.6% on average, which is 3x that of running Gemma longer.

SeedGemma@20Gemma@55Muse@20Muse@55MOMO@55 (20/9/6)
186,66486,54198,13995,01386,102
287,19486,98996,34891,23486,632
386,79286,59796,54392,15986,243

Our world record run came from seed 1, and our claim is not that we consistently achieve it, but that MOMO's procedure consistently gives better gains than single-model refinement.

MOMO's sensitivity to model selection and timing of solution transfer The choice of model and branching step can have a large effect. For instance, on the qubit routing problem, when using a Qwen model on top of a Gemma model, resulted in nearly zero gain during R2. Additionally, the schedule of the solution transfer was determined post-hoc after observing the behavior of the models up to step 25. For Erdos, we found that models stopped improving (beyond the noise floor of 1e-7) at step 12. For qubit, we found that the models continued improving past 12 and tapered off at step 20. If you branch at Gemma's step 12 to Muse to step 20 (following the Erdos schedule), you get a larger reduction in SWAP count, compared to running Muse at the same point. The schedule is itself a hyper-parameter choice determined by a cross product of the model x problem, largely invariant to seed.

Reviewer_uSrkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies RLVR applied to single-problem discovery tasks, where a model iteratively improves programs in a state buffer. The motivating observation is that different models trained on the same problem converge to different, sometimes mutually exclusive, technique profiles. MOMO exploits this by organizing training into rounds: in each round, every model except the previous round's winner continues training from a copy of the winner's best-state buffer (visitation counts reset) while retaining its own weights (optimizer state reset). Across four tasks with three LLM models, MOMO reports new state-of-the-art results on Erdős minimum overlap , circuit placement , and qubit routing, and improves over single-model RLVR on all four tasks. Ablations support transferring the state buffer (vs. a prompt summary) and retaining weights (vs. fresh weights).

Critical strengths
  1. The new Erdős minimum overlap bound (C5 ≤ 0.3808567) improves on EinsteinArena (0.3808586), Hyra (0.3808590), SimpleTES (0.3808680), and TTT-Discover (0.3808760), and the winning construction is qualitatively different from the previous SOTA (801 pieces / more blocks vs. 3,584 pieces / 7 blocks) — i.e., this is a genuinely new construction, not a re-polish. The routing result (86,102 vs. Hyra's 86,123) and the placement result (0.9214 vs. re-run winners 0.9533/0.9611) are on standardized public benchmarks.
  2. The per-model "what changed" narratives (Sec. 4.1, App. B, D) and Figure 14 (Muse lowering Gemma's decay penalty from ~0.67 to ~0.44 while weighted SWAPs fall from 32,166 to 32,017, with 63% vs. 21% parent-beating rates for decay-weakening vs. decay-preserving edits) go beyond score reporting and give evidence for why cross-model continuation helps.
  3. The progression "observe technique divergence → restart across models instead of within one→ test design choices for what crosses a round boundary" is exemplary empirical methodology. The method is deliberately simple (only the state buffer is exchanged), which makes the contribution easy to understand, adopt, and extend.
Critical weaknesses
  1. The central "better than single-model training" claim lacks a compute-matched control (main concern). In the seeded study (Sec. 3.3, Fig. 5), MOMO consumes 37 model-steps (2×12 + 8 + 5 with K=2) while each single-model control gets 25 steps; the paper even states this explicitly. Comparisons in Fig. 4b/5 are therefore against a weaker compute budget, and the natural controls are missing: (i) a single model run for 37 steps, and (ii) three independent parallel runs (no handoff) whose total model-steps equal MOMO's, taking the best buffer — i.e., "parallel restarts without transfer." Without (ii), it is not established that the buffer handoff adds anything beyond what independent parallel search with the same total budget already yields; the restart discussion in Sec. 3.1 argues this verbally but never tests it. The same issue applies to SOTA comparisons: MOMO's candidate budget (3 models × T_r × 512 candidates/step) dwarfs single-run baselines, which is fine for a SOTA claim but not for the "diversity helps" claim.
  2. Seed studies use n=3 (Fig. 4b, 5) with a one-sided paired t-test at p=0.027 — with 3 pairs this test has almost no power to detect anything but huge effects, and p=0.027 here should not be read as strong evidence. The placement and routing SOTA claims (Tables 4–5) appear to come from single runs with no seed variance at all; given that the routing margin over Hyra is 21 SWAPs (0.02%) and Round 3 on placement is an exact tie with Round 2, run-to-run variability matters and is unreported.
  3. Table 2 does not readable encode its 2×2 design. The design crosses state continuation {prompt summary, state buffer} with weight continuation {fresh, retained}, yielding four cells, but the table prints four columns ("Prompt summary | State buffer | Fresh | Retained") whose mapping to the four cells is undefined. Under any reading, two columns denote the same configuration (buffer + retained) yet disagree (Muse: +0.81 vs. +0.64; Gemma: +0.41 vs. +0.57), and under the reading suggested by the prose the "prompt summary" cell (+0.64 for Muse) would contradict "both recipients end well above Qwen's solution." The ablation that justifies MOMO's two core design choices is currently unverifiable.
  4. No compute/cost accounting, and reproducibility is effectively restricted to a TPU-pod lab. Every step evaluates 512 candidate programs (1,000 s each on 2 CPUs for Erdős/AC1; 17×300 s for placement; ~1,900 s for routing). Total candidate evaluations, TPU/CPU hours, and wall-clock per task are never reported, so readers cannot judge cost-effectiveness vs. search-only baselines (SimpleTES) or reproduce at any scale. The code release helps, but the paper's own infrastructure description (App. A.3: v4-64/v5p-32 slices) signals that the reported protocol is out of reach for most groups; ICLR reviewers need at least aggregate cost numbers to weigh the contribution.
  5. Scalability claims exceed the evidence. The abstract and conclusion state that MOMO "scales naturally to more model families and more rounds," but everything reported uses K=3 (and K=2 in the seeded study) and R=3. There is no experiment varying K or R, so the paper cannot distinguish "MOMO benefits from diversity" from "one lucky handoff per task carries the result" — note that on AC1 "most of the gain is Muse's re-gridding" (Sec. 4.1) and on routing most of the gain is one Muse edit phase (App. D), i.e., single-handoff effects.
  6. Missing connection to island-model / migration literature. MOMO is structurally an island-model evolutionary algorithm: islands = models with private weights, migration = copying best individuals (buffers) between islands after each epoch, with the donor island paused. This decades-old line (and more generally population-based / cooperative coevolutionary transfer) is the closest prior art for the mechanism and is absent from Related Work, which only covers tree search and ensembling. Discussing it would sharpen the novelty claim (weight retention + learned improvement operators are indeed new ingredients) rather than weaken it.
Questions for authors
  1. Can you add, for at least Erdős and routing: (a) a single-model run with 37 steps, and (b) three independent single-model runs with no handoff whose summed model-steps equal MOMO's, reporting best-buffer outcomes per seed? If MOMO still wins at equal total budget, the paper's core claim becomes substantially stronger. 2 Please print the full 2×2 grid (prompt/buffer × fresh/retained, four cells per recipient model) and state which configuration each current column corresponds to. Which cell is +0.81 vs. +0.64 for Muse?
  2. What are total candidate evaluations, core/TPU-hours, and wall-clock per task and per round? How does MOMO's cost compare to SimpleTES-style search-only scaling at equal candidate budget?
  3. How many seeds underlie Tables 3–5 (other than the Erdős seed study)? Can you report mean±std over ≥5 seeds for placement and routing, given the 21-SWAP and 0.0-cost margins?
  4. Any preliminary evidence for K>3 or R>3? If not, would you soften the "scales naturally" wording to a conjecture?
  5. Resetting visitation counts at handoff is a design choice stated but not ablated; does keeping donor counts change recipient behavior materially?
  6. For the 11 circuits Hyra did not release, you add SABRE counts that "add one SWAP" in total. Please confirm this is one SWAP across all 11 circuits (not per circuit), since the SOTA margin is 21.
3 comments
AuthorsOpenReview ↗

Rebuttal Part 1

Thank you for the constructive feedback and interest in our paper! We address your questions below:

MOMO Compute-Matched Comparison on Seeded Trials

a) We ran a matched compute experiment of MOMO, where each single model received 37 steps of training. At this matched budget, we find the MOMO exceeds the performance of single-model training across seeds (represented as distance to a target C_5 bound of 0.38085, in units of 1e-6, lower is better):

SeedQwen@25Qwen@37Muse@25Muse@37MOMO@37 (12/8/5)
17.067.0513.5313.526.93†
218.3418.3420.4419.2018.26†
312.5612.5320.3719.2912.32†

For qubit routing, we tested two additional seeds as seen below. As also mentioned in Appendix D, we used a larger compute budget, because a) qubit routing is a more challenging problem than Erdos, as the model has to optimize a single policy across three quantum topologies b) the models are still rapidly improving at step 12. We use 20 steps for round 1, 9 for round 2, and 6 for round 3 (totaling 55 between Gemma and Muse). After step 20, we see that Gemma and Muse models taper off performance, and running them to step 55 only reduces SWAPS by an average of 0.2% and 4.3% for Gemma and Muse, respectively. However running MOMO, reduces SWAPS from the R1 model, Gemma, by 0.6% on average, which is 3x that of running Gemma longer.

SeedGemma@20Gemma@55Muse@20Muse@55MOMO@55 (20/9/6)
186,66486,54198,13995,01386,102
287,19486,98996,34891,23486,632
386,79286,59796,54392,15986,243

Our world record run came from seed 1, and our claim is not that we consistently achieve it, but that MOMO's procedure consistently gives better gains than single-model refinement.

b) You pointed out another important baseline, which is testing whether the best of three parallel runs (run shorter) can exceed the performance of MOMO. Firstly, we re-ran all the Erdos experiment for 9 seeds for 25 steps, and found that the std-dev across many seeds is ~2.74e-6. However, the gain from running MOMO is ~7.37e-7 as shown in the compute matched comparison above. This indicates a compute should be spent, at least initially with parallel sampling across seeds. Additionally, there is variance even within a seed that can be exploited. In seeds 1 and 3, we found that 3 short runs (12-13 step) of Qwen produced slightly better results than MOMO, and even a compute-matched best of Qwen and Muse. Interestingly, seed 1 contains two "lottery ticket" programs that are always generated on the first step where LORA initialization has zero effect, and the model exploits these programs consistently over training.

SeedBest of 3 Qwen (12/12/13)Best of 3 Muse (12/12/13)Best(Qwen@19, Muse@18)MOMO@37 (12/8/5)
16.8914.667.076.93
312.2814.7312.6412.32

However, two things can be true. You should prioritize parallel draws across seeds, but MOMO's benefit is whether you should sequentially refine across models or continue the same model once convergence is reached. Our compute-matched comparison for qubit and erdos indicate that mixing helps more than training longer.

AuthorsOpenReview ↗

Rebuttal Part 2

MOMO Hardware and Experiment Details At a matched budget of 37 model steps on the Erdos problem (~19k candidates), MOMO falls between standalone Qwen and Muse training in generated tokens, grader-hours, and training FLOPs.

ConfigurationStepsCandidatesTokens (M; k/cand)Valid evals (rate)Grader-hoursFLOPs (×10¹⁹)
R1: Qwen126,08095 (15.6)3,494 (57%)1,4221.8
R1: Muse126,14477 (12.5)5,306 (86%)1,9371.6
R2: Muse on Qwen's tree84,09650 (12.2)3,732 (91%)1,4011.0
R3: Qwen52,56052 (20.3)2,238 (87%)9631.0
MOMO total3718,88027314,770 (78%)5,7245.4
Qwen-only3718,73633113,924 (74%)5,6236.1
Muse-only3718,94421717,314 (91%)5,9444.7

Qwen's generation length increases over training, leading to higher token consumption per candidate. Grader-hours also increase as generated programs increasingly exhaust the 1,000-second evaluation time limit. Qwen's validity rate improves substantially (57% → 87%), whereas Muse starts with a much higher validity rate (86% → 91%). Here, validity is the fraction of generated programs that pass compilation and timeout checks. These factors all affect the wall-clock time. Increasing generation length will increase rollout generation latency. Increasing validity rate means more samples get graded, and increased program run-time over the course of training makes grading latency even higher. So reporting wall-clock time, even per round, isn't specific enough, because it is so variable over the course of a run. Adding to that variability is our TPU hardware availability. We use Spot TPUs provided by the TPU Research Cloud, where at some points during the run there is plentiful capacity, allowing us to fully parallelize generation and rollouts. However, there are other points where capacity is limited and generation and grading take much longer. The strength of our open-code release is that our framework allows for efficient usage and load-balancing across Spot TPUs, which are significantly cheaper and more accessible to academia.

In terms of comparison to SimpleTES, they used GPT-OSS-120B and sampled 16 states and performed 32 completions per step, but ran for 100 steps. In terms of number of rollouts generated and therefore the number of grading hours used, MOMO uses 63x and 45x less, for Erdos and qubit routing. However, if you combine the training cost of MOMO, since SimpleTES is inference-only, this may exceed the number of FLOPs used by SimpleTES.

Any preliminary evidence for K>3 or R>3? For the erdos problem, we ran three seeds with R=5. This involves switching the model 5 times during the sequential run. As shown below, the R=3 schedule wins 3/3 times, and the R=5 schedule wins 2/3 times. Therefore, with more rounds we have some preliminary evidence that it is still beneficial when compared to training longer, but when compared to R=2, it is worse. However, when we stated that the method "scales naturally", we meant that using more models or additional rounds are simply hyperparameters of the algorithm and should be tuned per problem.

SeedQwen@37MOMO@37 (12/8/5)MOMO@37 (12/3/3/3/4)
17.056.936.71†
218.3418.2618.96
312.5312.3212.49

Resetting visitation counts We initially chose this because we wanted the model to explore different parts of the state-tree during the handoff process. However, we found that the performing a reset of visitation count hardly matters in changing the result. We re-analyzed, what "would have" happened at each step, had we used the existing visitation count, and since at each step, there is on average some child that outperforms the parent, the visit on the child is always 0. The child get's reselected because we also use the PUCT-tail state-selection method as described in App. A.2 in TTT-Discover [1]. Therefore resetting visitation count has no effect.

SABRE Swap Count It is one SWAP in total across all 11 circuits, not per circuit. Ten of the 11 unreleased circuits need zero SWAPs under SABRE, and the eleventh (alu-v0_27 on Q20) needs one. The 61 released circuits, all of which pass the SimpleTES checker, account for 86,122 SWAPs, and adding the 11 SABRE-routed circuits gives exactly Hyra's published total of 86,123.

[1] Mert Yuksekgonul, Daniel Koceja, Xinhao Li, Federico Bianchi, Jed McCaleb, Xiaolong Wang, Jan Kautz, Yejin Choi, James Zou, Carlos Guestrin, and Yu Sun. Learning to discover at test time. In ICML, 2026.

AuthorsOpenReview ↗

Update to the Best of 3

Best of 3 short-runs Update: We have an updated comparison to the best of 3 comparison per seed. We paired Qwen with stronger models that we can train on Tinker, like Inkling2Small a large MoE model with 17B activated, to understand whether using stronger models, even with pure inference, might be able to better demonstrate the effects of MOMO on the Erdos problem. We trained Inkling2small with 5 steps of training on R1, and also tested a no-train version that runs MOMO R2 from step 12 -> 20. Both versions outperform best of 3 draws.

SeedBest of 3 Qwen (12/12/13)MOMO@37 (12/8/5) (Qwen/Muse)New MOMO@32 (12/8) (Qwen/Inkling2small, no-train)New MOMO@32 (12/8) (Qwen/Inkling2small, 5-step R1 train)
16.896.936.8046.682
312.2812.3212.2412.08

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.