acceptodds
Under review as a conference paper at ICLR 2027

The Red Queen Gödel Machine: Co-Evolving Agents and Their Evaluators

Abstract

Self-improving agents are state-of-the-art on agentic coding benchmarks, yet their search methods assume a stationary evaluation criterion. This ignores a central feature of evolution: species adapt as their environments change with them. We introduce the Red Queen Godel Machine (RQGM), an evolutionary framework for recursive self-improvement under non-stationary utilities. This allows learned evaluators to improve alongside the agents they guide. On DeepSWE, the RQGM improves over its fixed-evaluator baseline by adding a complementary agent-as-a-judge code-review signal: a co-evolved reviewer grades coder patches to guide search. At low reasoning effort, the RQGM coder passes 82.1% of held-out tasks against the baseline's 75.0%, and nearly matches the GPT-6 Astra model at high effort. In scientific paper writing and reviewing, and Olympiad-level proof writing and grading, co-evolved evaluators provide an evaluation criterion. Anchored to human IMO grades, a co-evolved grader writes its own milestone rubric and exceeds static baselines at a 3x lower search cost, driving the prover to the best mean score. Since the RQGM can modify the search objective across epochs, it can regularize the search. For example, the RQGM reduces self-preference bias via an additional adversarial objective to discover reviewers equally stringent on AI and human work. Guided by these calibrated reviewers, co-evolved writers reach 1.78x-1.86x higher acceptance rates than the baseline under an agent-as-a-judge panel. The RQGM enables self-improving systems where agents and evaluators recursively bootstrap each other beyond the limits of static evaluation.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
self-improving agents, learned evaluation, multi-agent systems, auto-mated scientific discovery, controlled utility evolution, co-evolutionary search, autoresearch

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_4rVoOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces the Red Queen Gödel Machine (RQGM), a framework in which task agents and their evaluators co-evolve. Evaluation criteria remain fixed within each epoch, while evaluators can be replaced at checkpoints using externally labeled evidence. Records dependent on displaced evaluators are then removed from the search statistics. The paper studies this approach across coding, paper writing, mathematical proofs, and reward design.

Critical strengths

The research question is important, and the method directly addresses it. The paper brings evaluators into the self-improvement loop and carefully handles evidence that becomes outdated when evaluation criteria change. External anchors, frozen evaluators within epochs, and selective erasure form a clear and well-motivated framework.

The empirical study is broad and includes concrete mechanism analysis. Beyond comparisons with fixed-evaluator baselines, the paper provides component ablations, cross-reviewer evaluation, and search-trajectory analysis. In particular, the DeepSWE analysis connects reviewer decisions to actual patch test outcomes, examining both the ability to reject failing patches and the cost of rejecting correct ones.

Critical weaknesses
  1. The DeepSWE case study does not yet clearly connect reviewer improvements to coder improvements. Appendix E.4 shows that the reviewer that best distinguishes passing from failing patches emerged after the winning coder had already been generated. Earlier reviewers were generally permissive even toward failing patches. Consequently, the stronger final reviewer does not directly explain how the winning coder was discovered. While I acknowledge the overall performance gains, it remains unclear which evaluator updates provided useful search signals before the winning coder emerged and how those signals contributed to its improvement.
Questions for authors
  1. Could the authors trace the winning coder’s lineage and explain how preceding reviewer updates affected candidate selection or code modifications? This would clarify the connection between evaluator co-evolution and the final performance gains.
  2. Could the authors clarify the discrepancies between the paper and the supplied implementation? The paper-writing experiments specify GPT-5.5 with low reasoning effort, whereas the supplied configuration uses GPT-5.6 Luna with high reasoning effort. The paper also specifies fixed weights for the adversarial evaluator objective, whereas the implementation pools successes and trials across objectives before computing the score. Which configuration and selection rule produced the reported results? Please provide the corresponding code revision, configurations, and run records.
6 comments
AuthorsOpenReview ↗

Rebuttal Comment 1

Dear Reviewer 4rVo

Thank you for reviewing both the paper and the provided code, and for recognising that "external anchors, frozen evaluators within epochs, and selective erasure form a clear and well-motivated framework". We address your points below.


W1 and Q1. Reviewer influence on the search

The reviewer improved the search through its anchor utility and through its code, which runs inside the coder. To show this effect more clearly, we replayed the 45 parent selections the DeepSWE search made before the winning coder appeared, removed each reviewer signal in turn, and recorded which parent the search would have chosen instead. The table below traces the lineage of the winning coder; Tab. 23 (App. E.4) adds the replayed selection probability of each step, and the new App. A.3, in blue, defines every quantity below.

StepWhat changedSeated reviewer
root to e28cf32fTwo-call reviewer that reads the repository, and a contract checklist for the coderroot
to 1bf78e50, then f076c908No code edits. The meta-agent's context grew instead, with its ancestors' latest train and validation results after a reviewer replacementbeec9b7, then 5f52214
to 337755f2The coder treats a review as a fallible code-review hypothesis5f52214
to 904d5a2a, the winnerCoder prompt only, which adds regression tests and keeps existing tests5f52214
  • At the 45 parent selections before the winner, removing the reviewer's anchor utility changes 25 decisions, including 2 of the 5 steps in the winning lineage, and lowers the replayed probability of the recorded winning chain 125×. With the same scores shuffled across reviewer trials, the chain is still 69× less likely than with the real scores. The replay holds the recorded archive fixed; the reruns in Tab. 4 and 5 test the whole search.
  • Each coder episode calls its own node's reviewer through a self_review tool and revises after a rejection. A change to the reviewer is therefore a change to the coder, and a better reviewer gives the coder better feedback. Overall, 2 of the 5 nodes in the lineage edited this functionality.
  • While the early reviewers tended to over-accept, they decided 5 of the 45 parent selections before the winner. In 4 of the decided 5, they chose the parent whose branch later produced the better coder, although that branch received less search.

Proof writing shows the same behaviour, across 4 runs on two models. We repeated the replay on Olympiad proof writing: the Nemotron 3 Ultra run of the proof ablations (Tab. 20), and the 3 Claude Haiku 5.5 seeds of Tab. 5. See Tab. 10.

  • The grader's anchor score steers the search. Without it, the search would have picked a different parent in most expansions of every run. It decided 2 of the 4 steps that led to the Nemotron winning prover, and 29 of 38 on the Haiku winning chains.
  • The prover uses the grader. As in DeepSWE, each prover sends its proof to its own node's grader and revises it, and the search keeps editing that grader, in 58 of 114 Nemotron expansions and 56 of 204 Haiku ones.
  • Its verdicts help. In the Nemotron run, the parents the grader's verdicts selected led to the better prover 12 times out of 17.

Continued in part 2.

AuthorsOpenReview ↗

Rebuttal Comment 2

Dear Reviewer 4rVo (continued)

Q2. Differences between the paper and the supplied implementation

Initially, we released the codebase to provide the reference code that was used to produce the results, where all settings were stated in the paper. Nevertheless, we have revised the supplementary material to explicitly include these configurations as runnable code. App. C.4 lists the settings of every run, the table below maps each result to its configuration. The supplement also includes the winning lineage of each reported run, as the per-node code changes from the root agent to each winner, with a script that rebuilds every winner from them. Runs that predate the released code also ship their original run configurations.

  • Adversarial objective. The paper defines the score a reviewer ee is promoted on as QP(e)=p+(e)/2+p−(e)/4+rP(e)/4Q_P(e) = p_+(e)/2 + p_-(e)/4 + r_P(e)/4. Here p+p_+ is its accuracy on anchor papers labelled accept, p−p_- its accuracy on anchor papers labelled reject, and rPr_P the share of the adversarial pool PP it rejects, where the pool holds AI-written papers that the previous reviewer accepted. The code does not apply these weights as coefficients. The scheduler samples the three kinds of item in the proportions of these weights, and composite_belief in rqgm/src/rqgm/search/belief.py adds up their successes and trials, so the pooled score converges in expectation to QPQ_P. To make this more explicit in our work, the new paragraph “Pooled implementation” in App. F.6, in blue, states this and extends Prop. 1 to this setting. App. C.5 now also states which promotion rule each run used.
ResultSupplementary config
DeepSWE RQGM, Fig. 1 and App. D.1deepswe-rqgm
DeepSWE HGM-Hdeepswe-hgm, coder only
Paper writer and reviewer, Tab. 2 and Fig. 7paper-rqgm and paper-hgm
Proof prover and grader, Tab. 3 and Fig. 6, and ablations, App. D.2.2imo-rqgm and imo-hgm
Reward design, Tab. 1reward-design-curriculum and reward-design-static
Polyglot, Fig. 3, and paper ablations, App. D.2original run configurations; lineages for Polyglot
Proof writing on Claude Haiku 5.5, Tab. 5 and Figs. 8 to 9campaign-imo-haiku; two-stage: campaign-imo-haiku-hgm then tools/two_stage/stage2.py
DeepSWE on Claude Haiku 5.5, Tab. 4 and 7 and Fig. 10campaign-deepswe-haiku, campaign-deepswe-haiku-s0 and campaign-deepswe-haiku-hgm
Evaluator influence and anchor perturbations, Tab. 10 and 11scripts in analysis/

Thank you for the careful reading of both the paper and the code. If you find our responses satisfactory in addressing your concerns, we kindly ask you to consider raising your score.

The Authors

Thanks for the additional comments. And looking forward to a clearer repository with settings.

Reviewer_cyomOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Summary:

The Red Queen Gödel Machine tackles a longstanding problem in self-improving agents such as DGM, HGM, and HyperAgents: the agent keeps improving, but the evaluator that decides whether it has made progress remains outside the loop. Once benchmarks are missing, saturated, or exploited, the system loses direction. The paper’s solution is to let the evaluator evolve alongside the agent through “controlled utility evolution.” The evaluator stays frozen within each epoch to preserve HGM’s theoretical guarantees and can be replaced only at sparse checkpoints, where a candidate must achieve a higher ϵ\epsilon-best-belief on fixed, human-annotated anchors. Records that depend on the previous evaluator are then selectively cleared. An optional adversarial pool also turns AI-generated outputs accepted by discarded evaluators into negative examples, targeting loopholes that a static objective would keep rewarding.

The most interesting part is how the anchors and adversarial pool divide the work. The anchors keep the evaluator from regressing on human-defined criteria, while the adversarial pool helps it keep up with the generator’s changing outputs and address blind spots beyond anchor coverage. The paper does not say this explicitly, but this is a key reason why continued co-evolution could offer more than training a better evaluator once and freezing it.

The reported performance looks very strong: 82.1% versus 75.0% on DeepSWE, and acceptance-rate gains of 1.78–1.86× in academic writing and IMO problem solving, where the method provides search signals without formal benchmarks. The adversarial objective brings acceptance rates for AI-generated and human-authored papers into alignment, and the authors openly report the cost in anchor accuracy. Ablations, released rubrics, and evaluator data also make the process relatively easy to inspect.

The main issue is that the experiments do not yet separate the sources of these gains. The writer and prover baselines use weak frozen judges—Sakana has 63% anchor accuracy, compared with 88% for the reviewer evolved by HGM-H—so the results do not rule out the possibility that most of the benefit comes from a better judge rather than co-evolution itself. DeepSWE has only 28 problems, random seeds are not reported, and successive reviewers perform near chance on the anchors, leaving their contribution unclear. There is also no sensitivity analysis of anchor noise, contamination, or distribution mismatch.

Overall, the paper offers a clear mechanism and transparent experiments, but its performance claims need further validation. It turns the idea that “evaluation criteria should evolve too” into an algorithm that can be analyzed and tested. A two-stage baseline, together with sensitivity analyses of the anchors and adversarial-pool weighting, is needed to put its central claims on firmer ground.

Critical strengths
  1. The problem matters beyond Gödel Machines. The paper questions a basic assumption in self-improving agents: the agent improves, but its evaluation criteria stay fixed outside the optimization loop. This becomes a problem when benchmarks are missing, saturated, or exploited. The issue observed frequently in general LLM-as-judge systems and AI research agents---anywhere a learned judge scores a generator. These systems need ways to improve the judge without letting it drift or leaving its blind spots unchecked. The paper gives this shared problem a concrete, formal treatment.

  2. The mechanism makes the idea actionable. Rather than stopping at “evaluation criteria should evolve too,” the paper spells out how this could work, with explicit invariants: freezing evaluators within epochs, using separate positive and negative anchor constraints to rule out trivial solutions, selectively removing old records, and applying the best-belief rule in two roles. These choices provide useful design principles for other systems built around learned judges.

  3. Adversarial pools help cover what the anchors miss. This is a particularly interesting reason to keep the judge evolving instead of training it once and freezing it. Outputs that exploit earlier judges become negative examples for later ones, helping the evaluator keep up with the generator and close gaps in anchor coverage. The adversarial debiasing experiments make this concrete: the authors acknowledge that the anchors cannot detect self-preference and openly report the trade-off, with anchor accuracy falling from 88% to 80%.

Critical weaknesses
  1. It is hard to tell how much co-evolution itself helps. The baselines use relatively weak, off-the-shelf evaluators such as Sakana or ProofAutoGrader. A stronger comparison would take an evaluator evolved by HGM-H, freeze it, and let the generator keep improving. Without that baseline, the reported 1.78–1.86× gains could reflect a better evaluator, continued co-evolution, or both. The coding comparison has a fair baseline, but the reviewers perform near chance and their replacements appear noisy, so it is also unclear whether they provide useful supplementary signals.

  2. The anchors need closer scrutiny. Anchors determine which evaluators get promoted, but the paper does not sufficiently test their quality or how sensitive the results are to their construction. Open questions include label noise and possible contamination in APReS, where reviewer anchor accuracy reaches 84–88% despite a reported human consensus ceiling of about 75%; the distribution mismatch between CRAVE and Polyglot; the synthetic writing tasks; and the effect of anchor sample size on promotion decisions. DeepSWE illustrates the concern: reviewers changed eight times across 11 replacements, while anchor scores moved from 0.22 to 0.19 and back to 0.22. This raises the question of whether the selection process is tracking meaningful improvements or mostly reacting to noise.

Questions for authors

I would be happy to raise my scores if the questions below are satisfactorily addressed:

  1. How much of the gain comes from simply having a better fixed evaluator? The writer and prover baselines use frozen, off-the-shelf evaluators, while HGM-H evolves a reviewer with much higher anchor accuracy (88% vs. Sakana’s 63%). This makes it hard to separate the benefit of a better evaluator from that of continued co-evolution. Could the authors add a two-stage baseline: evolve a strong reviewer/grader against the anchors, freeze it, and then evolve the writer/prover using the same model, budget, and external evaluation panel? This would help show how much of the reported 1.78–1.86× acceptance-rate gain actually needs continued co-evolution. A similar question comes up in coding. In Table 15, reviewers preceding the winning agent have low best-belief scores and accept 82–100% of failing patches. Could the authors keep the two-role structure and shared code, but freeze the reviewer or randomize its judgments? This would help distinguish useful evaluation signals from gains due to additional shared-code edits. Both comparisons would benefit from multiple seeds and uncertainty estimates, especially given DeepSWE’s 28-problem holdout set.

  2. What happens when the anchors are unreliable? Evaluator promotion depends entirely on anchor-based best-belief, but the paper does not test how sensitive it is to anchor noise, sample size, distribution mismatch, or contamination. Several results make this worth checking: reviewers repeatedly switch on DeepSWE as anchor scores fluctuate; APReS accuracy exceeds reported human agreement, with possible training-data contamination; and Polyglot reviewers are selected on multi-file pull requests but used to judge single-file exercises. Could the authors perturb the anchors in at least one domain and measure both the stability of promotion decisions at fixed checkpoints and the generator’s final performance? Useful tests include bootstrap resampling, changing anchor size, adding realistic label noise, using review decisions made after the model’s training cutoff, and trying task-matched anchors. Varying the adversarial-pool weight would also clarify the trade-off between anchor accuracy and the AI–human acceptance-rate gap. The key question is whether controlled utility evolution degrades gracefully when its reference becomes unreliable, or simply shifts the uncertainty from the evaluator to the anchors.

  3. Minor: Could the authors make Figure 2’s caption more helpful? It is currently hard to tell how to read the figure. A brief explanation of the main elements and how to follow them would make it easier to understand.

5 comments
AuthorsOpenReview ↗

Rebuttal Comment 1

Dear Reviewer cyom

Thank you for your detailed review and for recognising that "the mechanism makes the idea actionable" with "explicit invariants" and that "adversarial pools help cover what the anchors miss".


Q1 and W1. Two-stage baseline

We ran the two-stage baseline you suggest (Tab. 5, Fig. 9). On proof writing, co-evolution beats freezing an evolved grader on the prover and matches it on the grader. We test it on proof writing since the hand-crafted baseline is more informative than Sakana is for paper writing.

  • Setup. We conduct two stages: stage 1 evolves only the grader on the IMO-GradingBench and freezes its best-belief winner. Stage 2 evolves only the prover. Both stop at 1,024M blended token, inclusive of Stage 1. We use Claude Haiku 5.5, with no adversarial pool since it cannot apply directly to this two stage baseline. Experiments were repeated over 3 seeds.
  • Prover. On the IMO-hard held-out problems, scored by ProofAutoGrader and the two graders of the proof ablations (Tab. 20), the median final prover scores 3.72/7 for RQGM (min 3.17, max 3.78) against 2.27 (min 2.23, max 3.27). The panel's pass@7 is 50% of completed proofs (min 42, max 50) against 30% (min 30, max 36) (Tab. 9). This is consistent when we isolate for static and learnt graders: when each method is scored only by ProofAutoGrader, it is 3.58 against 2.20. When we use the other method's grader, it is 3.75 against 2.40. RQGM leads on 6 of the 7 problems and ties the seventh (sign-flip p = 0.031), and completes 36 of 42 attempts against 31, scoring 3.56 against 2.61 on completed proofs.
  • Grader. On the 548 held-out IMO-GradingBench responses, median exact agreement with the human mark is 61.1% for RQGM (min 60.0, max 62.6) against 60.9% (min 60.0, max 66.6).
  • Frozen Graders May Cause Overfitting. Two-stage provers reach the highest search belief, 0.95 to 0.97, against their frozen grader, yet the lowest held-out scores (Fig. 9). This shows that the provers learn how to overfit to the particularities of that frozen grader.

In addition, while the evaluator utility is noisy, the joint evolution provides benefit in guiding the RQGM search process. Upon replaying the 45 DeepSWE Astra parent selections before the winner, removing the reviewer's anchor utility changes 25 of them and makes the winning lineage 125× less likely (Tab. 23). This is evidenced by the coder which evolves to call its own node's reviewer in every episode in two of the five lineages. Our reply to 4rVo, W1 and Q1, discusses in depth how co-evolution changes the search's choices, in DeepSWE, paper writing and proof writing.


DeepSWE beyond 28 tasks, and multiple seeds

We now evaluate both DeepSWE winners on all 113 tasks, in addition to the 28 held-out ones, at low, medium and high effort. RQGM leads at every effort (Fig. 1, left).

GPT-6 Astra effort28 held-out, RQGM28 held-out, HGM-HAll 113, RQGMAll 113, HGM-H
Low (search effort)23, 82.1%21, 75.0%85, 75.2%80, 70.8%
Medium24, 85.7%22, 78.6%86, 76.1%83, 73.5%
High24, 85.7%23, 82.1%86, 76.1%84, 74.3%

Multiple seeds. We ran 3 RQGM and 3 HGM-H seeds on Claude Haiku 5.5, where HGM-H keeps the reviewer frozen at the root. Pooled over seeds, RQGM passes 55.6% of 252 held-out attempts against 50.8%, with mixed-model odds ratio over all efforts 1.41 (95% CI 1.01 to 1.97) (Tab. 4 and 7, Fig. 10). In every rerun we ran in coding and proofs, removing evaluator co-evolution at matched budget gave lower aggregate held-out scores (Tab. 4 and 5). Our reply to JHVg, Q3, gives every seed.


Q3. The caption of Figure 2

We rewrote the caption of Fig. 2 to add a more explicit narrative. Fig. 11 (App. B) expands it step by step, Fig. 12 (App. C.2) shows which evidence survives an evaluator replacement in each domain, and Fig. 13 (App. C.2) gives the call flow of every role per domain.

Continued in part 2.

AuthorsOpenReview ↗

Rebuttal Comment 2

Dear Reviewer cyom (continued)


Q2 and W2. What happens when the anchors are unreliable?

Offline replay shows promotion is robust to moderate anchor perturbation: perturbed anchors still promote evaluators about as good as the recorded one, while shuffled labels do not.

Setup. We re-run every evaluator promotion offline from the stored anchor verdicts, on 1,000 altered versions of the anchor per setting: resampled, with 20% of labels flipped, with only 25% of items kept, and with all labels shuffled as a control. Each altered anchor set produces a different final evaluator whose quality we use to determine the impact of the shift. Since all candidates may be of high quality in some searches due to random chance alone, we report its percentile rank: the share of the run's candidate evaluators it beats on the full, unaltered anchor. A high percentile means the information within the anchor picked a better evaluator, not just that the search produced good ones (Tab. 6 and 11, App. A.4).

  • Bootstrap: anchor items resampled within each label.
  • Label noise: 20% of labels flipped.
  • Smaller anchor: a random 25% of items.
  • Shuffled: all labels permuted, so the anchor carries no information.
Percentile rank of the selected evaluatorDeepSWE, GPT-6 AstraDeepSWE, Haiku (3 seeds)Proof, Haiku (3 seeds)
Root evaluator69.564.16.3
Recorded final95.793.292.0
Bootstrap93.886.387.7
20% noise92.380.785.7
25% of items92.977.485.6
Shuffled62.550.558.5
  • Result. In every setting, bootstrap, noise and a quarter of the anchor select evaluators ranked above both the root and shuffled labels. Shuffled labels select near the 50th percentile, like a random pick. The anchor's information drives promotion.
  • Spread. The selected DeepSWE reviewer beats the root in 99% of bootstraps and 96% of noisy and reduced anchors, against 49% under shuffled labels.
  • Scope. This replays promotion offline from stored verdicts; end-to-end robustness of the search to anchor shift is future work (App. G). Reviewers with near-equal scores can swap, which is why we score the rank of the selected evaluator, not its identity. At 9 of the 11 Astra promotions the recorded reviewer is the most frequent bootstrap pick; at the other 2 it is keeping the incumbent. App. A.4 gives results for every seed.

Pool weight. Raising the pool weight trades anchor accuracy for a smaller gap between AI and human acceptance. We re-ran the paper reviewer's last promotion with score (1 − w)·anchor + w·pool, on validation papers:

wAnchor accuracyAccepts humanAccepts AI
0 to 0.15100%47%94%
0.2 to 0.55, and QPQ_P90%29%48%
0.6 and above64%36%24%

Every weight from 0.2 to 0.55 selects the original adversarial reviewer, showing the robustness of the adversarial pool against exact coefficients.

Other points in W2.

  • 0.22, 0.19 and 0.22. These are the end-of-tenure scores of different reviewers (App. E.4), not one score moving back and forth. The seated reviewer's score rises at each tenure, from 0.03 to 0.48 (App. E.4).
  • Task-matched anchors. Polyglot is the only dataset with significant miss-match, the other tasks uses either use the same or overlapping datasets for both tasks,
  • APReS. The recorded paper-reviewer winners are within 0.017 best-belief of three other reviewers. Bootstraps most often select the RQGM specialist, 84% accurate on held-out papers (App. D.1). Our paper does not report a human-consensus ceiling of 75% for this dataset.

If you find that these additional results address your concerns, we kindly ask you to consider raising your score.

The Authors

AuthorsOpenReview ↗

Rebuttal Comment 3

Dear Reviewer cyom (follow-up)

As the discussion period comes to an end, we would greatly appreciate your feedback on whether our response addresses your concerns. We have also extended the experiments reported in our initial rebuttal comments.


Q1 and W1. Two-stage baseline on paper writing

We ran the two-stage baseline on paper writing. Co-evolution beats freezing an evolved reviewer on the writer and matches it on the reviewer.

  • Setup. We conduct two stages: stage 1 evolves only the reviewer on the APReS anchors and freezes its best-belief winner while stage 2 evolves only the writer. To finish within the discussion period, each run is limited to 256M blended tokens, inclusive of Stage 1. We use Claude Haiku 5.5, with no adversarial pool since the two-stage baseline does not permit an equivalent two-stage mechanism. Both methods' writers are scored on the same 100 held-out prompts by the other method's best reviewer and by a fixed Haiku Reviewer. Over 6 runs, RQGM leads in all 6 pairs under both reviewers (exact sign test p = 0.03, Tab. 29).
Metric (median of 6 runs, min to max)Two-stageRQGM
Reviewer accuracy on 100 held-out APReS papers (%)80.5 (76 to 84)80.0 (69 to 86)
Writer acceptance rate, other method's best reviewer (%)1.5 (0 to 27)16.2 (5 to 49)
Writer acceptance rate, Haiku Reviewer (% of judgments)4.2 (1.7 to 14.3)10.1 (3.8 to 28.3)

We hope that these additional results help address your concerns and we look forward to responding to any further requests you may have

The Authors

Reviewer_JHVgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a new framework for self-improving agentic search, called the Red Queen Gödel Machine (RQGM). In this framework, both task agents and learned evaluators are allowed to improve over time, by co-evolving. The search process is grouped into epochs, during which a fixed evaluator is used. Every few epochs an evaluator challenger can be promoted, using ground truth from evaluations that are independent of the evaluator used. The authors provide an argument that the RQGM remains consistent within each epoch. They then demonstrate the RQGM on several tasks: coding, designing rewards, reviewing and writing papers, and grading and writing Olympiad solutions. They compare results to a fixed evaluator (HGM-H) and discuss improvements in performance.

Critical strengths

The authors identify a real-world problem with the use of evolved evaluators in iterative agent optimization: when agents optimize over a large distribution, the evaluators can saturate, become unhelpful, biased, or even exploitable.

The proposed solution is to freeze the evaluator during each epoch and replace it if needed based on anchor evidence derived from the evaluation process (which does not depend on the frozen evaluator). This is a reasonable design choice, especially since the records used to make such decisions should not contain evidence that depends on the frozen evaluator. The distinction between evaluator independent anchor evidence and evaluator dependent search records is well motivated.

The paper provides extensive details about implementation, experiments, and analyses (pseudocodes, descriptions of data splits, cost reports, lineage and mechanisms analysis, ablation studies). The coding experiments also report held out execution results, which are more convincing than just using evolved judge evaluations. The reviewer discrimination analysis is also useful because it shows that later reviewers seem to discriminate better between good and bad patch submissions.

This is a very relevant issue, and the authors propose a concrete way of intervening by introducing an adversarial reviewer objective and replay pool.

Critical weaknesses

The paper reports on limited evidence for the efficacy of the proposed RQGM approach (e.g., statistical evidence for the claimed 82.1% vs 75.0% result is based on just 28 tasks, with about 23 vs 21 solved, not even with uncertainty intervals or paired task-level analysis). Polyglot gains are also modest.

The evaluation of the papers written by the different search systems seems to be circular. These papers are evaluated by a panel of reviewers, including reviewers from the search systems being compared. Thus, it is not clear that higher acceptance rates indicate higher scientific quality, factuality, novelty, or usefulness. It might simply be prompt injection.

When a new agent is generated, maybe the evaluator feedback was explicitly included. Also, if agents are selected based on their parents, then the evaluator might have influenced the parent. And if the parent was selected based on the evaluator's feedback, then this might be a dependency chain. There are a lot of ways that the evaluator could cheat and affect what gets passed to the meta-agent, which then passes to the training examples, etc. So, in addition to the evaluator's feedback, we need to consider whether the evaluator affects the prompt itself, or the parent, or the prompt's parent, etc.

Questions for authors

It would have been helpful to have more details on the RQGM setup: e.g., what are the different reviewer roles? How are reviewer calls made? What are the training/evaluation procedures, and how are the replay/challenge pools constructed? What are the interactions between the shared workspaces? Is there a difference in the effective amount of computing power used?

How was the budget for the DeepSWE evaluations matched to those for RQGM? As far as I can tell, the paper states that the selected RQGM winner is the one with the lowest average number of blended tokens per solved task, whereas the HGM-H winner has the smallest ratio of blended tokens to number of solved tasks. That is, the RQGM winner has more blended tokens than the HGM-H winner, though not necessarily more total tokens. This is not sufficient to ensure that the two searches were conducted at comparable budgets. More generally, I would like to see a more detailed discussion of budget matching in the paper, including a description of the selection criterion used to pick the winner in each case, and ideally, more than one run.


Please provide multiple independent seeds for all headline RQGM versus HGM-H comparisons, especially DeepSWE and Polyglot. Report per-seed outcomes, confidence intervals, and cost-performance curves.

5 comments
AuthorsOpenReview ↗

Rebuttal Comment 1

Dear Reviewer JHVg

Thank you for your careful review and for recognising that "the authors identify a real-world problem" and that freezing the evaluator within each epoch "is a reasonable design choice". We address your points below.


Q2. How the DeepSWE budget was matched

Neither winner is selected by “tokens per solved task”. Both are selected by the same validation best-belief; 587M and 429M are where each winner first appeared, and both searches ran for the same total tokens.

  • Selection score, Sec. 3.1. The search returns the node with the highest ε-best-belief once the budget B is spent, where BBϵ(a)BB_\epsilon(a) is the ε-quantile of Beta(1+Sa,1+Fa)\mathrm{Beta}(1+S_a,1+F_a) and SaS_a and FaF_a count ground-truth successes and failures. Tokens do not enter it.
  • We use the same rule in both searches, App. C.5. Replacement and final selection share the ε-best-belief rule (Alg. 1, line 38), inherited from HGM.
  • Token counts, App. C.3. Following HGM, a winner's cost is counted until it first attains its highest ε-best-belief; App. D.3.1 reports the full run up to the entire token horizon, showing how the final best-belief agent does not change anymore.

Budget. The two searches are compared at 2,532.7M blended tokens, the full HGM-H run (App. D.3.1). The RQGM winner was found later, at 587M against 429M. As mentioned in Sec. 5.1 of the original submission, the HGM-H finds no better coder over the rest of its run, so the RQGM leads to a better final coder over the full horizon.

  • Revision. App. C.5 now states the rule and the matched budget together, that neither leader changes up to 2,532.7M, and the evaluations and tokens of every run.
  • For multiple seeds see our response to Q3.

W1. The 82.1% against 75.0% result rests on 28 tasks

We now report a full evaluation on all 113 DeepSWE tasks (Fig. 1, left), and 3 seeds per method on Claude Haiku 5.5 (Tab. 4, Q3).

GPT-6 Astra effort28 held-out, RQGM28 held-out, HGM-HAll 113, RQGMAll 113, HGM-H
Low (search effort)23, 82.1% (64.4 to 92.1)21, 75.0% (56.6 to 87.3)85, 75.2% (66.5 to 82.3)80, 70.8% (61.8 to 78.4)
Medium24, 85.7% (68.5 to 94.3)22, 78.6% (60.5 to 89.8)86, 76.1% (67.5 to 83.0)83, 73.5% (64.6 to 80.7)
High24, 85.7% (68.5 to 94.3)23, 82.1% (64.4 to 92.1)86, 76.1% (67.5 to 83.0)84, 74.3% (65.6 to 81.5)
  • Every effort. RQGM leads at low, medium and high effort, on the held-out tasks and on all 113. Intervals shown are Wilson 95% confidence intervals.
  • Comparison with prior work. The two methods we build on, both at ICLR 2026, report single runs, citing cost. HGM, an Oral, beat DGM by 2 of 60 SWE-bench tasks; DGM reached 50.0% on a 200-task subset against 51.0% for the best open-source system. Our margins are at least as large.
  • Seeds. We repeated the comparison over 3 seeds per method; RQGM leads with a significant overall odds ratio of 1.41 when accounting for task and reasoning effort. Full results in Q3, part 2.
  • Polyglot. Our claim in Fig. 3 is efficiency and not absolute performance, a stronger coder at 71.7% against 69.9% with 1.35 to 1.72× fewer search tokens. Furthermore, as mentioned above the HGM reports similar margins.

W2. The gain might be prompt injection

Whether the panel uses static or learned reviewers, the RQGM writers lead (App. C.7).

Panel, 100 papers per writerHGM-H writerRQGM writersRatio
Full panel of Tab. 221.8%38.8 to 40.5%1.78 to 1.86×
Without the RQGM reviewer25.7%42.0 to 43.7%1.64 to 1.70×
External only, Sakana and DGM-H6.5%22.5%3.46×
  • External reviewers. The paired-bootstrap CI of the ratio is 2.27 to 6.57. Paired over the same 100 prompts, DGM-H accepts 31 generalist and 28 specialist papers that it rejects from HGM-H, and 0 the other way (exact sign test, p < 10⁻⁸).
  • Scope. The paper never claims scientifically better papers: paper writing measures acceptance by automated reviewers, the only search signal available without ground truth (App. C.7).
  • Scan. We checked all 300 panel papers for reviewer-directed instructions, role tags, hidden markup, invisible Unicode, acceptance claims and copied rubric text, and read every hit. We did not find anything that stood out as prompt injection.
  • Writer code. Neither evolved writer prompt asks the model to address or instruct the reviewer; both ask for the paper itself with no meta-commentary outside the bounds of the paper, and the RQGM prompt adds "do not mention that you are trying to get accepted". Both are verbatim in App. C.9.

Continued in part 2

AuthorsOpenReview ↗

Rebuttal Comment 2

Dear Reviewer JHVg (continued)


Q1. More details on the RQGM setup

Every part of the setup is in the supplementary code, rqgm/ in the zip

Each archive node is one Git commit holding the full workspace: every role and the meta-agent, which may edit any of them. Scoring code sits outside, in rqgm/src/.

<node commit>/
├─ .rqgm/contract.json   roles, judges and seats
├─ meta/                 meta-agent and its instructions
├─ task_agents/
│  ├─ solver/            coder, writer or prover
│  └─ reviewer/          reviewer, or grader for proofs
├─ shared/               code shared by all roles
└─ experience/           train data and ancestors' results

A child is a checkout of its parent that the meta-agent edits; its roles are scored by the seated evaluator. At a checkpoint, challengers are scored on the anchors, the best is promoted based on the previously mentioned rule, an adversarial pool is minted by extracting scored artefacts that any evaluator accepted, and records that depended on the old evaluator are erased. S stands for rqgm/src/rqgm below:

QuestionWhere in the zip
Workspace creationS/git_store/node_store.py:1328-1419
Meta-agent editsrqgm/src/rqgm_execution/expansion_runner.py:149-541
Roles and reviewer callsexperiments/domains/&lt;domain&gt;/forms/mini_swe/wiring.json declares them; each call runs task_agents/shared/runtime.py:219-266 in a container; S/settlement.py:384 resolves the seated reviewer
Train, validation and testdeepswe/data/split.json; train reaches only the meta-agent (experience_capture.py:63-148); test is used once (tools/operations/heldout.py)
Promotion and erasureS/transitions.py:410-505, S/declared_rules.py:152-265, S/ledger/run_projection.py:138-151
Challenge poolS/domain/standard.py:304-357: outputs the displaced reviewer accepted become items its successor must reject
ComputeS/spend.py:53-72 charges every call of every role to one blended-token budget

Shared workspaces. Nodes interact only through lineage: a child inherits its parent's whole tree, reviewer code included.


W3. The evaluator may influence the prompt, the parent or the parent's parent

Validation and test data cannot leak, whatever the evaluator does. The splits are disjoint (DeepSWE: 28 train, 57 validation and 28 test tasks in experiments/domains/deepswe/data/split.json; APReS by paper). With X for rqgm/src/rqgm_execution, a meta-agent receives only the parent's code; read-only train inputs and labels; its ancestors' train results (X/experience_capture.py:226-262); and one outcome value per judged ancestor production, with no sample or content (X/experience_capture.py:122-148). X/expansion_runner.py:369-370 stages nothing else. Ground truth comes from fixed scorers outside the node that no agent can edit (e.g. wiring.json:86-108), and test tasks are used once, after selection.

The evaluator can shape which agents are generated, through parent selection and those outcome values, but never an agent's ground-truth score. Each record carries the evaluator slots that affected it, and selective erasure removes it when that evaluator is displaced (Sec. 3, App. F.4).


Q3. Multiple independent seeds

We repeated the DeepSWE comparison over 3 seeds per method; RQGM leads, significantly overall. One GPT-6 Astra seed pair costs about 50.7k USD, so we ran 3 RQGM and 3 HGM-H searches on Claude Haiku 5.5, each to 1,024M blended tokens. HGM-H keeps a reviewer whose seat stays frozen at the root, so the comparison isolates evaluator replacement. Each selected coder attempts the 28 held-out tasks once per effort (Tab. 4 and 7; per-seed curves in Fig. 10).

EffortHGM-H, solved per seedRQGM, solved per seedHGM-HRQGMOdds ratio (95% CI)
Low18 / 14 / 1216 / 17 / 1152.4%52.4%1.01 (0.56 to 1.81)
Medium14 / 16 / 1415 / 17 / 1652.4%57.1%1.41 (0.79 to 2.51)
High14 / 16 / 1017 / 12 / 1947.6%57.1%1.83 (1.05 to 3.20)
All46 / 46 / 3648 / 46 / 4650.8%55.6%1.41 (1.01 to 1.97)
  • The RQGM solves 140 of 252 attempts against 128, ties at low effort, leads at medium and high, and ties or wins on every seed. The odds ratio is from a mixed logistic model with random effects for task and search run, which shows the results are significant (Tab. 4 and 8).
  • We also provide results for Proof writing where the RQGM prover leads the two-stage baseline on every seed (see Tab. 5 and our reply to cyom), and on reward design where the RQGM scores 0.365 ± 0.053 against 0.306 ± 0.128 (Tab. 1).

Thank you again for the careful review. If our response has addressed your concerns, we kindly ask you to consider raising your score.

The Authors

Dear Reviewer JHVg (follow-up)

As the discussion period comes to an end, we would greatly appreciate your feedback on whether our response addresses your concerns. We are also happy to answer any further questions or requests.

The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Rebuttal Overview

General Comment

Dear Reviewers, ACs and PCs,

We thank the reviewers for their feedback. 4rVo finds that "the research question is important, and the method directly addresses it", cyom that "the problem matters beyond Gödel Machines", and all three find the core design sound: "a clear and well-motivated framework" (4rVo), "a reasonable design choice" (JHVg) and a mechanism that "makes the idea actionable" (cyom).

The reviews raised three main questions: whether the DeepSWE gain holds beyond one run on 28 held-out tasks (JHVg, cyom), whether co-evolution beats evolving a strong evaluator and then freezing it (cyom), and whether the evaluator actually shapes the search (4rVo, cyom). To answer them, we ran 48 new experiments: 12 searches (about 12.3B blended tokens) and 36 held-out evaluations, counted per coder and reasoning effort (over 1.1B blended tokens). We also re-analysed the logs of every run perturbing the search in order to provide qualitative insight into the dynamics of our search algorithm.


1. The full DeepSWE benchmark [JHVg, cyom]. The paper scored the two GPT-6 Astra winners only on the 28 held-out tasks. We now also score them on all 113 DeepSWE tasks at low, medium and high reasoning effort (Fig. 1, left). Consistent with the held-out results, the RQGM outperformed all baselines across reasoning efforts.

2. More DeepSWE seeds [JHVg, cyom]. Adding another GPT-6 Astra seed costs about $50k. To mitigate this cost, we chose to repeat the comparison on Claude Haiku 5.5 with 3 RQGM and 3 HGM-H seeds, each searched to 1,024M blended tokens. Each selected coder attempts the 28 held-out tasks at three efforts (Tab. 4 and 7, Fig. 10).

  • Across seeds and reasoning efforts, the RQGM outperforms baselines, consistent with the Astra results. Pooled over seeds, RQGM passes 55.6% of held-out attempts against 50.8% for HGM-H, and its lead grows with reasoning effort: a tie at low (52.4% each), 57.1% against 52.4% at medium and 57.1% against 47.6% at high (Tab. 4), consistent with the Astra results.

3. Evolve-then-freeze baseline [cyom]. On proof writing with Claude Haiku 5.5, a two-stage baseline first evolves a grader on the anchors and freezes it, then evolves the prover against that grader, over 3 seeds at the same 1,024M blended tokens as RQGM, inclusive of stage 1 (Tab. 5 and 9, Figs. 8 and 9).

  • On the 7 IMO-hard held-out problems, the RQGM shows improvement over baselines irrespective of which grader panel is selected for review. It also completes more proofs (36 of 42 against 31), and its Pass@7 is 50% against 30%.

Continued in part 2.

AuthorsOpenReview ↗

Continuation

General Comment (continued)

4. Paper-writing evaluation and budget [JHVg]. JHVg was concerned that our own reviewers judge our writers. Scored only by the two external reviewers, Sakana and DGM-H, the RQGM writers reach 22.5% acceptance against 6.5% for the HGM-H. A scan of all 300 panel papers finds no text directed at the reviewer. Regarding budget, both Astra DeepSWE searches are compared at the same 2,532.7M blended tokens, the length of the full HGM-H run, and pick their winner by the same best-belief rule.

5. The evaluator changes which agents the search builds on [4rVo, cyom]. RQGM chooses which archive node to expand next by sampling from each branch's evidence. In DeepSWE, part of that evidence is the node's reviewer score on a fixed set of labelled patches, its anchor score. From the run logs we re-ran this parent selection offline. We then re-ran it with the reviewer's anchor score removed (Tab. 10 and 23, App. A.3). We find that that co-evolving the evaluator with the task agent plays a significant role in guiding the search towards better nodes. Please see the respective responses for a more detailed discussion.

6. Promotion holds up on a noisy or smaller anchor [cyom]. RQGM replaces the seated evaluator when a candidate scores better on the anchor, a small labelled set. cyom asked whether these promotions are noise. We re-ran every promotion offline from the stored verdicts on 1,000 altered anchors per setting: resampled, with 20% of labels flipped, with only 25% of items, and with all labels shuffled as a control that carries no information. Most candidates in a run may already be decent, so to keep results comparable across seeds we rank the evaluator each anchor promotes among all of the run's candidates by accuracy on the full anchor (Tab. 6 and 11).

Percentile of promoted evaluatorRootResampled anchorShuffled labels
DeepSWE, GPT-6 Astra69.593.862.5
DeepSWE, Haiku (3 seeds)64.186.350.5
Proof, Haiku (3 seeds)6.387.758.5

Flipped labels and smaller anchors give similar results, while shuffled labels promote evaluators closer to a random pick. At 9 of the 11 Astra promotions, the recorded reviewer is also the most frequent pick under resampled anchors. These results show that the anchors are helpful in guiding the search for the RQGM.


Roadmap of revisions

  • Sec. 5.5, new. The 113-task evaluation, the Haiku DeepSWE seeds (Tab. 4), the two-stage baseline (Tab. 5) and anchor reliability (Tab. 6).
  • App. A, new. Per-seed convergence (Figs. 8 to 10, Tab. 7), significance of the overall gains (Tab. 8), proofs per judge (Tab. 9), evaluator influence (Tab. 10) and every anchor perturbation (Tab. 11).
  • App. E.4. The winning DeepSWE chain (Tab. 23) and each reviewer's patch discrimination (Tab. 24).
  • App. C.4, C.5 and C.7. Tokens and configuration of every run, the budget and winner rule, the injection scan and the external-only panel.
  • App. F.4 and F.6. The scope of the validity guarantee and the pooled form of the adversarial objective.
  • Fig. 2, App. B and C.2. A caption that walks through Fig. 2, the expanded method (Fig. 11), and each domain's evidence and call flow (Figs. 12 and 13).

All revisions are in blue. We thank the reviewers again and look forward to the rest of the discussion.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.