acceptodds
Under review as a conference paper at ICLR 2027

Selection Is Not Learning: Choosing Model Futures

20 views

Abstract

Self-evolving language models train on tasks they generate, so every round must decide which updated model the next round inherits. Current loops such as R-Zero, Agent0, and Skill Self-Play make this decision through the task, admitting tasks that look learnable and inheriting every resulting update. We identify a selection-to-update gap in these loops, where a task that looks learnable need not produce a useful update. Most updates on verified tasks fail to make correct answers more likely, yet these loops inherit them all. HEIRCOMMIT freezes the updated model and uses fresh evaluation questions to decide whether to inherit it or retain the current model. EVOHEIR separates parameter inheritance from experience inheritance, using current and archived experience to repair failures and preserve successes before final confirmation. Our analysis characterizes selection-induced optimism and decomposes regret into selection, absorption, and confirmation errors. HEIRCOMMIT cuts the harmful share of accepted updates by 68–83% relative to random inheritance at the same acceptance rate. Across four backbones, EVOHEIR achieves the best mathematics and general-reasoning averages, raising the math average by 5.3% relative to the strongest baseline, Agent0. The framework lets useful experience persist even when the model that produced it is not inherited.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, self-improvement, reinforcement learning, reasoning, update selection, knowledge distillation, experience reuse

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rW67OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

In this paper, the authors identify an important problem: a task that looks learnable need not produce a useful update. Based on this observation, they propose HEIRCOMMIT to select data, and further integrate this into a pipeline called EVOHEIR. EVOHEIR is better than baselines after the same rounds of learning, while consuming more resources than baselines.

Critical strengths
  1. The overall workflow makes a lot of sense to me. The novelty of the proposed HEIRCOMMIT and EVOHEIR is significant enough. By reading Figure 1 and 3, one can quickly get the main idea of the paper.

  2. The evaluation is conducted on a variety of benchmarks and models, and SOTA baselines are included in the comparison. Also, the authors test their method on 10 rounds, making it much more compelling than the results of merely 2~3 rounds.

Critical weaknesses
  1. Despite the strength, I identified one fairness issue in the comparison. In all main experiments, the comparison is based on the same amount of iteration rounds, instead of same FLOPs/compute. This is unfair for other baselines. While the authors add a brief compute matched experiment in Section 6.6, the setting is a bit uncommon. Specifically, the authors should not alter the setup / hyperparameters of baselines to align compute. Instead, faster baselines should be run for more rounds. I would suggest that the authors plot this figure:

x-axis: Training FLOPs (including everything such as sampling and learning)

y-axis: Performance

The authors can (and ideally should) just reuse the runs in figure 6. but change the x-axis from round to FLOPs. If the strength of EVOHEIR still holds, then the evaluation would be much stronger.

  1. Currently, the authors are adding a theory in the main paper. However, the theory has gap with the actual case. The authors also claim this: The reported zero-threshold gate trades this positive-margin guarantee for greater acceptance sensitivity. In this case, it's better to alter the framing to Motivation or Proposition, instead of stating it as a formal theorem.
Questions for authors

I would like to maintain my positive score if the authors can add the requested curve.

Minor: In Figure 1, NLL gain overlaps with other parts of the figure.

2 comments
AuthorsOpenReview ↗

Response to Reviewer rW67

Thank you for the positive evaluation and for proposing a clean compute-matching protocol. We agree: rather than changing baseline hyperparameters to fit a budget (as in Fig. 9a), faster baselines should simply run more rounds with their own recipes, and performance should be plotted against cumulative training FLOPs. We followed this protocol. The revision adds the curves (new Fig. 12 and Tables 8–9 in App. G.1, summarized in Sec. 6.3), reframes Theorem 1 as Proposition 3 (design rationale), and fixes the label overlap in Fig. 1.

1. Compute-matched comparison (App. G.1, Fig. 12, Tables 8–9)

FLOPs accounting. Reusing the Figure 6 runs (4 backbones ×\times 3 seeds), we counted all training-side FLOPs per round from logged token counts (2N2N per token for forward passes and generation, 6N6N for full-parameter training, ≈4N\approx 4N for LoRA):

  • task generation, admission rollouts and native training rollouts, including Agent0's tool-call turns;
  • Scout and learner policy updates, including reference and old-policy passes;
  • for EvoHeir, also screening, recipient attempts, frozen-teacher passes, LoRA absorption and confirmation.

Benchmark evaluation is excluded. Per round, EvoHeir (Full) uses 1.09–1.17×1.09\text{--}1.17\times the training FLOPs of Agent0 and 1.19–1.24×1.19\text{--}1.24\times those of R-Zero, matching measured GPU-hours on 2×962 \times 96 GB GPUs (Table 8). We resumed R-Zero and Agent0 from round 10 with unchanged hyperparameters and ran them to rounds 11–12, until their cumulative FLOPs reached EvoHeir's 10-round total.

BackboneEvoHeir 10-round FLOPs (×1018\times 10^{18})Per-round ratio vs. R-Zero / Agent0R-Zero best (round) / interp.Agent0 best (round) / interp.EvoHeir (10 rounds)Lead over Agent0 (best / interp.)
Qwen3-4B-Base1.801.23× / 1.13×29.77 (r7) / 28.3030.28 (r12) / 30.2231.82+1.54 / +1.60 pp
Qwen3-8B-Base3.911.24× / 1.09×32.65 (r10) / 32.0834.39 (r11) / 34.3736.51+2.12 / +2.14 pp
OLMo-3-7B3.321.19× / 1.16×22.67 (r6) / 21.3223.71 (r12) / 23.6424.94+1.23 / +1.30 pp
Gemma-4-E4B-it1.961.21× / 1.17×46.14 (r8) / 44.6146.72 (r11) / 46.6247.21+0.49 / +0.59 pp

"Best" is each baseline's best round up to the first round reaching EvoHeir's budget (which may slightly exceed the budget, favoring the baselines). "Interp." interpolates at the exact budget.

  • Math Avg: at matched FLOPs, EvoHeir outperforms Agent0 on all four backbones—by +1.23–2.12 pp on the three base models (+1.30–2.14 pp interpolated) and by +0.49 pp (+0.59 pp interpolated) on instruction-tuned Gemma-4-E4B-it.
  • General Avg: the same pattern holds on all four backbones: +0.78 to +1.09 pp over Agent0's best extended round on the base models and +0.49 pp on Gemma (30.52 vs. 29.74, 36.48 vs. 35.46, 21.40 vs. 20.31, 40.93 vs. 40.44; Table 9).
  • Extra rounds help the baselines little: they add only 0.06–0.36 pp to Agent0, and R-Zero's best round falls at or before round 10 on every backbone.

Reverse check (no extra baseline training). Stopping EvoHeir at the FLOPs each baseline spends in 10 rounds (Fig. 12):

  • At Agent0's budget (EvoHeir round 8.56–9.14), EvoHeir reaches 31.63, 36.23, 24.44 and 46.94, above 10-round Agent0 (30.01, 34.11, 23.35, 46.66) on all four backbones.
  • At R-Zero's budget (round 8.05–8.38, 0.81–0.84×0.81\text{--}0.84\times Full's budget), it reaches 31.34, 35.24, 24.41 and 46.85, above 10-round R-Zero (28.79, 32.65, 21.73, 45.11) by +1.74–2.68 pp.

Where the curves cross. In Fig. 12, because each EvoHeir round costs more, on Qwen3-4B EvoHeir trails along the FLOPs axis until 54% of the ten-round budget (by at most 0.57 pp); beyond 54% of the budget it leads on every backbone. Whereas squeezing 10 rounds into 0.8×0.8\times budget by trimming per-round steps hurt Full in Fig. 9a, scaling compute by round count keeps EvoHeir ahead at R-Zero's 10-round budget (0.81–0.84×0.81\text{--}0.84\times FLOPs) and Agent0's (0.86–0.91×0.86\text{--}0.91\times FLOPs).

2. Theory framing and Fig. 1

  • Theorem 1 is now Proposition 3 (design rationale: search–inheritance separation) in Sec. 5. A new paragraph, "Protocol satisfaction and operating modes", explains that the protocol meets its conditions (frozen proposal, independent held-out probes, paired scores in [−1,1][-1, 1], giving σcon2=1\sigma_{\mathrm{con}}^2 = 1 by Hoeffding's lemma). It separates the certified mode (τt>0\tau_t > 0, on consensus-labeled probes) from our τt=0\tau_t = 0 gate, whose decisions agree with ground truth in 85.0% of rounds (App. D.1).
  • Fig. 1: the "NLL gain" axis label no longer overlaps the tick labels.

Thanks! I appreciate the new Figure 12, which solves my main concern. Therefore, as promised, I maintain my positive rating of 3. However, I do agree with Reviewer tRTg that the authors should have used less AI to write the paper (and the rebuttal).

Reviewer_Jq1LOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a key problem in self-evolving language models: selecting useful training tasks does not always lead to better models. The authors propose HeirCommit, which accepts a model update only if it improves performance on fresh evaluation questions, and EvoHeir, which also reuses useful experience from previous updates, even rejected ones. Experiments on four models show consistent improvements over existing methods.

Critical strengths

The paper identifies an important problem and provides clear experimental evidence that useful training tasks do not always produce useful model updates.

The proposed method is intuitive and practical. The experiments and ablation studies support the benefits of its main components.

Critical weaknesses

Evaluating model updates before accepting them is not a new idea. The paper could better explain what is fundamentally new compared with existing approaches.

The experiments mainly focus on mathematical reasoning with relatively small models. It is unclear whether the method generalizes to more complex agent tasks.

The confirmation step uses only 32 questions, which may lead to noisy decisions about whether an update is actually beneficial.

The theoretical analysis has too strong assumptions

Questions for authors

Q1. Does the confirmation step measure actual improvement? The confirmation labels are generated from the current model's consensus rather than ground-truth answers. If the current model is confidently wrong, a better model could be rejected simply because it disagrees with the old model. How does the method distinguish genuine improvement from agreement with the current model?

Q2. Is greedy model selection optimal for long-term self-evolution? HeirCommit accepts an update based on its immediate performance gain. However, an update that temporarily reduces performance might enable greater improvements in later rounds. Could greedy acceptance prevent the model from exploring more promising learning paths?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Jq1L (1/3)

Thank you for the constructive review and for recognizing the importance of the problem, the clarity of the empirical evidence, and the practicality of the framework.

W1. What is fundamentally new compared with prior update gates. While evaluating a candidate before accepting it appears in AlphaGo Zero, APE, PACE and agent-side skill gates, which we cite (Sec. 1–2), all of those acceptance tests are whole-state: when a candidate is rejected, everything it learned is discarded with its weights. Our work introduces three advances that prior update-gating methods do not provide.

(1) Decoupling parameter inheritance from experience inheritance (EvoHeir and Recovery). An update that regresses overall should not supply the next round's parameters, yet it often finds verified solutions to problems the current model cannot solve. EvoHeir separates which weights persist from which solutions are absorbed:

  • Non-destructive rejection. Verified solutions from unselected branches and rejected rounds stay in the archive MtM_t. Across the 77 committed rounds of Table 1, at least 44.5% of the 4,443 absorbed records came from states whose parameters were never inherited, and 68.8% (53/77) of committed rounds absorbed solutions from rejected rounds (App. F).
  • Weaker models are useful donors. 8 of 10 donors weaker than the current root (mean donor gain −1.09-1.09 pp) still improve the recipient (mean absorption gain +0.71+0.71 pp; Fig. 7a). Admitting history from rejected rounds raises the absorption gain from +0.73+0.73 to +1.39+1.39 pp at a fixed 64-reference budget (Fig. 7c), and matched source replacement gives +0.34+0.34 to +0.73+0.73 pp over same-age inherited sources, with 95% CIs above zero (Fig. 7d).
  • Recipient-conditioned absorption (Eq. 4). Fine-tuning directly on the same 64 references (Cache SFT) loses 2.61 pp of existing successes, for a net −0.54-0.54 pp (Fig. 7b). Recovery instead uses a frozen copy of the recipient as teacher along the recipient's own prefixes: shown the verified solution on failed problems (DtrepairD_t^{\mathrm{repair}}), and without it on solved ones (DtkeepD_t^{\mathrm{keep}}). It cuts forgetting by more than half and nets +1.39 pp (+1.93+1.93 pp over Cache SFT; +2.46+2.46 pp on the 2,400-problem diagnostic in Fig. 11a).

(2) A diagnosis of the selection-to-update gap in LLM self-evolution. R-Zero, Agent0 and Skill-SP gate learning only at task admission and inherit every resulting update. Proposition 1 proves that pre-update behavior cannot identify update value, and the measurements agree:

  • 42–52% of admitted tasks leave the frontier under fresh responses, with 8.60 and 4.10 pp selection optimism (Fig. 4a);
  • 60 of 96 updates on exact-valid tasks fail to reduce correct-target NLL (Fig. 4b);
  • 47–77% of native updates across R-Zero, Skill-SP and Agent0 are harmful (Table 4).

Against a matched-random control with the same acceptance rate, HeirCommit cuts harmful commits by 68–83% and turns negative pool gain positive on R-Zero and Skill-SP (Fig. 4c,d). This proves that the gain comes from the transition test rather than from committing less.

(3) Search–inheritance separation without ground-truth labels. Reusing screening scores to accept a winner inflates its expected value (Proposition 2) and pays a KK-branch union bound (Proposition 3). Separating screening (PtscrP_t^{\mathrm{scr}}) from fresh confirmation (PtconP_t^{\mathrm{con}}) removes this bias; removing independent confirmation (NoConfirm) causes the largest drop, −2.02-2.02 Math pp (Table 2).

Revision. Sec. 1, Sec. 2 and App. F now highlight this contrast between whole-state acceptance and decoupled inheritance, together with the 44.5% / 68.8% provenance statistics above.

AuthorsOpenReview ↗

Response to Reviewer Jq1L (2/3)

Q1. Does confirmation measure real improvement rather than agreement with a confidently wrong current model?

Short answer: Yes—across all N=120N = 120 commit decisions of the Table 1 runs (4 backbones ×\times 3 seeds ×\times 10 rounds), we audited every frozen (root, proposal) pair against 512 held-out MATH problems with ground-truth answers and relabeled all 3,840 confirmation probes with Qwen3-235B-A22B-Thinking. The 32-probe consensus gate agrees with the ground-truth decision in 85.0% of rounds (Spearman ρ=0.73\rho = 0.73), retains 90.4% of truly beneficial updates, and rejects a better proposal because of a consensus label error in only 3 of 120 decisions (2.5%) (new App. D.1, Tables 5–6). Five observations explain this.

(a) Probes with ≥70%\ge 70\% agreement are highly accurate. Training tasks are chosen near the uncertainty frontier, whereas confirmation probes require ≥70%\ge 70\% agreement across 10 admission rollouts (App. A). Relabeled by Qwen3-235B-A22B-Thinking (majority of 4 samples at T=0.6T = 0.6), these consensus labels match the 235B reference solver on 92.3% of probes (3,408 of 3,691 probes).

(b) Direct ground-truth audit (App. D.1, Table 6). We froze every root StS_t and proposal St+S_t^+ from the 120 rounds and scored each pair on 512 held-out MATH problems (levels 3–5, deduplicated against all 7 math benchmarks). 73 of the 120 proposals truly improve on ground truth:

GateCommitsHarmful share of commitsUseful retainedAgrees with ground truthMean GT pool gain
Commit every update12039.2%100.0%60.8%+0.54 pp
Matched random (same rate)7739.2%64.2%53.1%+0.35 pp
HeirCommit (32 probes, root consensus)7714.3%90.4%85.0%+0.89 pp
32 probes, 235B reference labels758.0%94.5%91.7%+0.94 pp
512 probes, root consensus726.9%91.8%90.8%+0.93 pp

Accepted proposals gain +1.38 pp on ground truth and rejected ones lose −0.97-0.97 pp; the Spearman correlation between the 32-probe gain and the 512-problem ground-truth gain is 0.73. Swapping in the 235B reference labels removes 5 false commits and 3 false rejections, so label errors account for only 8 of the gate's 18 disagreements with ground truth. The remaining 10 come from sampling noise on near-neutral updates and from the domain gap between self-generated probes and the ground-truth pool.

(c) The anchor moves. Probes are regenerated and relabeled every round by the latest inherited model StS_t, so the gate is never tied to the base model's initial errors.

(d) A false rejection does not lose the experience. Even in the 3 rounds where a better proposal was rejected due to a label error, its verified solutions still entered the archive MtM_t and remained available to later recipients; history from rejected rounds adds +0.66+0.66 pp beyond committed-round history (Fig. 7c).

(e) Ground-truth evaluations confirm the end-to-end gain. In the host audits (Table 4), HeirCommit's decisions are graded by correct-target NLL on a disjoint pool, cutting harmful commits from 47–77% to 13–16%. In full self-evolution, removing the gate (NoConfirm, Table 2) lowers external ground-truth Math Avg by 2.02 pp and General Avg by 1.19 pp. Where an external verifier or unit test exists, it can replace consensus labels directly (Sec. 4.3, App. D.1).

AuthorsOpenReview ↗

Response to Reviewer Jq1L (3/3)

W3. Why 32 confirmation questions are enough.

  • 128 paired responses. Each decision compares St+S_t^+ and StS_t on the exact same 32 questions ×\times 4 responses with shared sampling seeds, so question difficulty cancels in the paired difference.
  • Agreement with a 16×16\times larger pool. With 512 consensus probes per round instead of 32, the decision is identical in 89.2% of rounds (107/120; App. D.1). Drawing 32-probe subsets from the 512 flips the decision with probability 9.1% on average; the 14 borderline rounds with flip rate above 20% have a mean absolute ground-truth change of only 0.27 pp (vs. 1.31 pp for the other rounds), showing that sampling noise mainly affects nearly neutral updates.
  • Compute trade-off. In the equal-budget sweep (Fig. 9c), 8, 16, 32, 64 and 128 confirmation questions achieve 30.97, 31.64, 31.82, 31.82 and 31.32 Math Avg: 32 and 64 tie for the peak, while 128 takes budget away from native training.

W2. Beyond math, and model scale.

  • Transfer to general reasoning. EvoHeir achieves the highest General Avg on all four backbones across SuperGPQA, MMLU-Pro, BBEH, LiveBench and KOR-Bench (+0.56+0.56 to +1.35+1.35 pp over Agent0; Table 1, Fig. 5). The backbones span three families (Qwen3-4B/8B-Base, OLMo-3-7B and instruction-tuned Gemma-4-E4B-it) at the 4–8B scale used by R-Zero and Agent0.
  • Tool use and a new Python code-synthesis domain (App. D.1, Table 7). Table 4 already audits Agent0, a tool-integrated curriculum agent. We also ran a new 64-update audit in an executable Python code-synthesis loop: Qwen3-4B-Base writes synthesis tasks with five unit tests, and updates are graded by unit-test pass@1 on 256 held-out synthesis problems. HeirCommit cuts the harmful share of commits by 73% (from 54.7% to 14.8%, matching the 68–83% cut in Table 4), retains 79.3% of useful updates, and turns mean pool gain from −0.37-0.37 to +0.79+0.79 pp. The new scope paragraph in Sec. 7 states that multi-turn agent environments remain untested.

W4. Theoretical assumptions. None of our propositions rely on strong distributional assumptions:

  • Proposition 1 is a constructive two-world lower bound with no distributional assumption.
  • Proposition 2 requires only unbiased per-candidate estimators (E[εk]=0\mathbb{E}[\varepsilon_k]=0).
  • Proposition 3 (formerly Theorem 1, matching Reviewer rW67's suggestion) requires only a conditionally sub-Gaussian error on confirmation probes. Because the protocol freezes St+S_t^+ before confirmation, draws held-out probes independently, and bounds every per-question paired score in [−1,1][-1, 1], this condition holds with σcon2=1\sigma^2_{\mathrm{con}}=1 by Hoeffding's lemma (Sec. 5, App. C.2). Sec. 5 ("Protocol satisfaction and operating modes") also distinguishes the certified positive-margin mode (τt>0\tau_t > 0, certifying gains on consensus-labeled probes rather than ground truth) from our zero-threshold gate (τt=0\tau_t = 0, which agrees with ground-truth decisions in 85.0% of rounds; App. D.1).

Q2. Does step-wise acceptance block promising long-term paths? EvoHeir is step-wise in parameters but non-greedy in experience: a rejected candidate's verified solutions enter the archive MtM_t and are absorbed by later recipients (occurring in 68.8% of committed rounds). Three results confirm the gate does not stall learning:

  1. Steady progress. EvoHeir commits in 6.42 of 10 rounds on average and never rejects more than 3 rounds in a row (Table 5). Its mean Math Avg rises in every round through round 10, while R-Zero peaks before round 10 on three of four backbones (Fig. 6).
  2. Accepting every proposal hurts. NoConfirm lowers Math Avg from 31.82±0.5531.82 \pm 0.55 to 29.80±1.1329.80 \pm 1.13, doubling the seed SD (Table 2).
  3. One-step lookahead (new, App. D.1). On Qwen3-4B-Base (3 seeds), we continued each of the 9 proposals rejected before round 10 for one more native round. The rejected path won once, by just 0.19 pp Math Avg; in the other 8 of 9 cases (88.9%) it still trailed the inherited trajectory by 1.27 pp on average (1.41 pp on the ground-truth pool), confirming that rejected proposals are predominantly genuine regressions rather than temporary dips.

Non-greedy parameter variants, such as a negative threshold or carrying a rejected proposal forward as an extra candidate, also fit the same rule.

Thank you for the detailed rebuttal. The additional experiments address most of my concerns. While some questions remain regarding novelty and generalization to more complex agent settings, I find the empirical evidence convincing. I have increased my score to 3.

Reviewer_tRTgOpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

Authors propose two new methods for identifying which updates to retain for self-evolving language models. Self-evolving models are those that generate both tasks and training data to continually improve themselves. Authors identify a gap in current methods: in some cases, models degrade instead of improving over time. Authors claim this is an issue of not evaluating transitions between model phases/epochs of training. Their HeirCommit method evaluates the new model before transitioning to it to generate the training data for the next phase, thereby avoiding cases where the model would transition to a worse model and therefore degrade over time. Their EvoHeir method tries to specifically identify which training data is useful in transitioning to a better model, and which is not. To be honest I was not able to understand how EvoHeir works. The paper is extremely dense and difficult to understand.

Critical strengths

None

Critical weaknesses

Paper is very dense and hard to read due to AI writing. I am unable to parse a large majority of the content due to this. I don't really understand how EvoHeir works.

HeirCommit makes some sense to me, however questions remain. What happens after a transition is rejected? Does the system just keep trying until it gets a model that improves against its previous iteration? Does the new version of the model have to improve on every single held out test set? How are the held out test sets/probes even generated, since the model is generating its own training data?

Questions for authors

In HeirCommit, what happens when a transition is rejected? Like does it just try again until the model has a transition that actually improves it? How many retries does it usually need?

How do the probes, on which the next version of the model is evaluated in HierCommit, actually work? Where does the data from the probe come from? Is the model not generating its own training data in the self-evolving regime?

6 comments
AuthorsOpenReview ↗

Response to Reviewer tRTg (1/2)

Thank you for your feedback and questions. In the revised manuscript (changes in blue), Section 3 now defines every term in plain language, and Section 4 opens with a step-by-step, notation-free walk-through of EvoHeir. Below is that walk-through; Part 2 answers each of your questions directly.

How EvoHeir works, in plain words (Fig. 3, Algorithm 1)

The goal. In self-evolution, a current model MM generates its own math problems and trains on them round after round. Because self-generated updates are noisy, 47–77% of the updates in existing loops (R-Zero, Agent0, Skill-SP) make the model worse (Fig. 4c). Yet even an update that gets slightly worse overall often learns to solve some hard problems that MM could not.

  • Existing methods either accept the whole update (inheriting its regressions) or reject the whole update (throwing away the new solutions it found).
  • EvoHeir separates which model weights to keep from which solved examples to learn from.

Each round has the following steps:

  1. Generate tasks and a held-out validation split (same data source as R-Zero). The model writes candidate problems and answers each one 10 times. The majority answer becomes the problem's pseudo-label; no human labels or external datasets are used. Before any training, questions with strong consensus (≥70%\ge 70\% agreement) are split into disjoint held-out sets: 16 screening questions and 32 confirmation questions ("probes"). The remaining frontier questions form the training batches.

  2. Explore (train two branches). Two copies of MM are trained on two different batches of 64 training problems, giving two candidates, Branch 1 and Branch 2. Verified solutions they produce are saved to an archive.

  3. Screen (pick the starting weights). MM, Branch 1 and Branch 2 answer the 16 screening questions. The highest-scoring model becomes the recipient (ties favor MM).

  4. Recover experience (repair failures, keep successes). We gather verified problem–solution pairs from this round's branches and from the archive of earlier rounds. The recipient attempts these problems without seeing the reference solutions:

    • If it fails (repair): a frozen copy of the recipient is given the verified solution as a hint and acts as teacher. The trainable recipient learns to match the hinted teacher's token predictions along its own attempt, so it learns to solve the problem without the hint.
    • If it succeeds (keep): it is trained to preserve its own predictions on that problem, so it does not forget skills it already has.

    This step is lightweight: at most 16 LoRA updates.

  5. Confirm with HeirCommit (accept, or keep MM). The model from Step 3 and the current MM each answer the 32 confirmation probes 4 times. If the new model scores higher than MM, it becomes the next MM; otherwise the old MM is kept.

  6. The archive persists either way. Verified solutions from Step 1 stay in the archive even if Step 4 rejects the new weights, so a rejected round still teaches later rounds.

Concrete example (Fig. 8) and measured impact (Fig. 7)

  • Example: at round 5 on Qwen3-4B-Base, the recipient drops the cross term 2x2\sqrt{x} when squaring x+9=x+1\sqrt{x+9}=\sqrt{x}+1 and answers x=0x=0. The archive holds a correct solution (x=16x=16) found in round 4 by a candidate whose weights were rejected. Recovery uses that archived solution to teach the recipient, which then derives 2x2\sqrt{x} and x=16x=16 on its own, without seeing the reference.
  • Impact: across our main runs, at least 44.5% of all absorbed solutions came from models whose weights were never inherited, and 68.8% of committed rounds learned from previously rejected rounds (App. F). Even donors that are overall weaker than the current model improve the recipient in 8 of 10 cases (Fig. 7a).
AuthorsOpenReview ↗

Response to Reviewer tRTg (2/2)

Q1. What happens when a transition is rejected in HeirCommit? Does the system keep retrying until it gets a model that improves? How many retries does it need?

  • There are no retries. When a proposal does not beat the current model MM on the 32 confirmation probes, HeirCommit restores the exact state of MM (weights, optimizer, scheduler and RNG state; Sec. 4.3, App. B.4) and the round ends.
  • We do not fall back to another branch, and we do not loop or retry within the same round. The pipeline moves to round t+1t+1, where MM generates a fresh batch of tasks and confirmation probes. Every round therefore has a fixed compute budget.
  • A rejected round is not wasted: all verified solutions from that round's exploration are saved in the archive and used by Recovery in later rounds.
  • How often updates are accepted (new App. D.1, Table 5):
    • Across the 120 rounds of our main ten-round EvoHeir runs (4 backbones ×\times 3 seeds ×\times 10 rounds), 6.42 of 10 rounds are accepted on average (77 of 120), and no run rejects more than 3 rounds in a row.
    • When HeirCommit is applied standalone to the native updates of R-Zero, Skill-SP and Agent0, it accepts 23–52% of them and cuts harmful commits by 68–83% relative to random acceptance at the same rate (Table 4).

Q2. Does the new model have to improve on every held-out test set?

  • No. The decision uses a single number on one small, self-generated confirmation set PtconP_t^{\mathrm{con}} (32 questions ×\times 4 sampled responses = 128 paired responses): the average accuracy gain of the new proposal over the current model MM on those questions (Eq. 5). If this paired difference is above 0, the update is accepted.
  • External benchmarks are never used in decisions. The 12 evaluation benchmarks of Table 1 (MATH-500, AIME, MMLU-Pro, etc.) are used only for final reporting and never enter training, screening or HeirCommit (Sec. 3, App. H.4).

Q3. How do the probes work, and where does the probe data come from if the model generates its own training data?

  • Probes are a held-out validation split carved out of the model's own self-generated tasks. At the start of round tt, the problem writer (Curriculum Scout) drafts candidate problems, and the current solver MM samples 10 answers per problem.
    1. Labeling by high-confidence consensus: if at least 70% of MM's valid answers agree on the same final answer, the problem qualifies as a probe, with the majority answer as its frozen pseudo-label (App. A).
    2. Strict separation before training: before any branch is trained, the problems are partitioned into disjoint pools: 64-problem training batches, 16 screening probes (PtscrP_t^{\mathrm{scr}}) and 32 confirmation probes (PtconP_t^{\mathrm{con}}).
    3. Fresh evaluation: in Step 4 (HeirCommit), both MM and the new proposal generate 4 fresh responses per confirmation probe (separate from the 10 responses that formed the label), which are scored against the frozen labels.
  • Do self-generated consensus probes track real improvement? Yes (new App. D.1, Table 6):
    • We relabeled all 3,840 confirmation probes with a much stronger reference solver (Qwen3-235B-A22B-Thinking). The ≥70%\ge 70\% consensus labels match the 235B solver on 92.3% of probes (3,408 of 3,691 probes).
    • We also rescored all 120 commit decisions on 512 held-out MATH problems with ground-truth answers. The 32-probe gate agrees with the ground-truth decision in 85.0% of rounds (102 of 120), keeps 90.4% of the truly beneficial updates, and lowers the harmful share of commits from 39.2% to 14.3%.

Revisions made for clarity

  1. Plain-language definitions of all terms (root, branch, recipient, donor, proposal, probe) in Section 3 (p. 3).
  2. A step-by-step walk-through of one round at the opening of Section 4 (p. 4), right below Figure 3.
  3. Explicit statements of the no-retry rule, the commit rate (6.4 of 10 rounds) and ground-truth probe fidelity in Section 4.3 (p. 5), Section 6.2 (p. 7) and Appendix D.1 (p. 22).

We hope this walk-through and the direct answers resolve your concerns regarding the clarity and soundness of our framework, and we would be grateful if you would consider them in your updated assessment.

Please clarify how "verified solutions" are produced for model-generated queries (which is, if I understand correctly, the case for all of the training data). Is it simply the majority-voted solution?

Comments on the paper

1 comment

We thank the three reviewers for careful and concrete reviews. The reviewers found the problem important and the evidence clear that useful tasks need not produce useful updates (Jq1L). They found the method intuitive and practical, with ablations that support each component (Jq1L), and the novelty "significant enough" (rW67). rW67 also judged the ten-round, four-backbone evaluation against SOTA baselines "much more compelling than the results of merely 2~3 rounds". The concerns fall into four themes. We answer each below and in the individual replies. The revised PDF shows all changes in blue.

The paper in two sentences. Self-evolving loops such as R-Zero and Agent0 pick tasks and then keep every resulting model update; in the native updates of R-Zero, Agent0 and Skill-SP, 47–77% of committed updates make the model worse (Table 4). HeirCommit keeps an update only if the updated model beats the current one on fresh held-out probes, and EvoHeir decouples parameter inheritance from experience inheritance: across the 77 committed rounds of Table 1, at least 44.5% of absorbed experience came from states whose parameters were never inherited, and 68.8% of committed rounds harvested verified solutions from rejected rounds (App. F).

1. Equal compute (rW67). As rW67 suggested, we counted total training FLOPs (generation, sampling, optimization, screening, Recovery and confirmation) and ran R-Zero and Agent0 for more rounds, with unchanged recipes, until they matched EvoHeir's 10-round FLOPs (App. G.1, Fig. 12, Tables 8–9).

  • At matched FLOPs, EvoHeir leads the best extended Agent0 round on all four backbones: by +1.23–2.12 pp Math Avg on the three base models (+1.30–2.14 pp interpolated) and by +0.49 pp (+0.59 pp interpolated) on instruction-tuned Gemma-4-E4B-it, while leading R-Zero by +1.07–3.86 pp.
  • Given only Agent0's or R-Zero's 10-round compute (EvoHeir rounds 8.1–9.1), EvoHeir already exceeds every 10-round baseline on all four backbones. Per round, EvoHeir costs 1.09–1.17×1.09\text{--}1.17\times Agent0's FLOPs and 1.19–1.24×1.19\text{--}1.24\times R-Zero's.

2. Does the gate measure real improvement? (Jq1L; tRTg) Every benchmark number we report is on 12 external benchmarks with ground-truth answers that never enter training or any decision (App. H.4). On Qwen3-4B, removing the gate (NoConfirm) lowers Math Avg by 2.02 pp and General Avg by 1.19 pp and doubles the seed SD (Table 2). We also re-checked all N=120N = 120 commit decisions of Table 1 against 512 held-out ground-truth MATH problems and relabeled all 3,840 confirmation probes with Qwen3-235B-A22B-Thinking (App. D.1, Tables 5–6):

  • The 32-probe gate agrees with the ground-truth decision in 85.0% of rounds (vs. 60.8% for keeping every update and 53.1% for matched random), retains 90.4% of useful updates, and cuts harmful commits from 39.2% to 14.3% (Spearman ρ=0.73\rho = 0.73).
  • Root consensus labels match the 235B solver on 92.3% of probes, and a better proposal is rejected due to a label error in only 3 of 120 decisions (2.5%).
  • In a unit-tested Python code-synthesis domain (Table 7), HeirCommit cuts harmful commits by 73% (54.7% to 14.8%) and turns mean pool gain from −0.37-0.37 to +0.79+0.79 pp.

3. Clarity (tRTg; Jq1L). Section 3 now defines each term (root, branch, recipient, donor, proposal, probe) in plain words and says where probes come from. Section 4 opens with a plain-language walk-through of one round (also in our reply to tRTg).

4. Theory framing (Jq1L; rW67). Theorem 1 is now Proposition 3 (design rationale). Sec. 5 explains that our protocol satisfies its conditions (frozen proposal, independent held-out probes, paired scores in [−1,1][-1, 1], giving σcon2=1\sigma_{\mathrm{con}}^2 = 1 by Hoeffding's lemma), and separates the certified mode (τt>0\tau_t > 0, on consensus-labeled probes) from our τt=0\tau_t = 0 gate (validated against ground truth in App. D.1).

Other changes.

  • A one-sentence scope statement in Sec. 7: training on verifiable math at the 4–8B scale, transfer to general reasoning, and a gate audit in a code domain; multi-turn agent environments remain untested.
  • Commit statistics for the ten-round runs: on average 6.4 of 10 rounds commit (Sec. 4.3, App. D.1).
  • New appendix material: gate-fidelity, lookahead, and code-domain audits (App. D.1, Tables 5–7), experience provenance (App. F), and compute-matched curves (App. G.1, Tables 8–9, Fig. 12).
  • Related Work contrasts whole-state acceptance with decoupled inheritance and adds four references; Figs. 1, 3 and 8 are cleaned up.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.