acceptodds
Under review as a conference paper at ICLR 2027

Verify Less, Evolve More: Training Idea-Level Critics for Verification-Efficient ML Evolving Agents

Abstract

As large language models become more powerful, self-evolving agents are able to tackle challenging tasks including AI for machine learning (AI4ML). In AI4ML, while empirical verification is available, it often requires computationally costly model training and evaluation, limiting the speed and scale of agent evolution. Yet verification efficiency remains underexplored, and frontier models provide only limited gains when used directly as idea selectors. We address this gap with specialized idea-level critic models that predict whether a proposed ML modification will improve upon the current solution, allowing agents to screen ideas and concentrate verification resources on the most promising candidates. We train the critic models through supervised fine-tuning on high-quality critiques synthesized by Gemini-3.1-Pro, followed by GRPO to further improve their predictive accuracy. Empirically, our critic models outperform Gemini-3.1-Pro in static idea evaluation, and these gains extend to agent inference, continual learning, and policy training. During inference-time evolution, they improve final solution quality under the same verification budget by selecting more promising ideas, with further gains from continual learning. During policy training, they serve as learned reward models, reserving empirical verification for uncertain cases and enabling substantially more policy updates with the same verification resources. Together, these results show that idea-level critic models help ML agents discover better solutions and learn stronger proposal policies under limited verification budgets.

Primary area:
foundation or frontier models, including LLMs
Keywords:
ML Agents, Self-Improving Agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3HLWOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes training “idea-level” critic models that predict whether a proposed modification to an ML solution will beat its parent (Improve / Tie / Lose). The goal is to let ML evolving agents spend expensive empirical verification only on promising candidates.

Critical strengths

The problem is practical and well motivated. Verification cost is a real bottleneck for evolutionary ML agents, and learned pre-execution screening is a natural lever.

Critical weaknesses
  1. The train/test split and task overlap are underspecified. The paper does not say how the 1.5K benchmark examples were held out. Each parent has 20 children sharing identical task text, criteria and parent summary. A random example-level split would therefore put siblings in both train and test. Please clarify, and report results with a parent-level or task-level split. Downstream, 10 of the 30 MLS-Lite tasks were used in critic training (Table 13). For those tasks, the OpenEvolve seed parent is the same baseline used as the starting parent during data collection. Please clarify, and report held-out-only results throughout. Please also report the label distribution of the benchmark and a majority-class baseline.

  2. The inference-time gains are modest, and the controls are incomplete. The improvements are about 1.5–2.7 domain-average points over no screening or Gemini screening. No seeds or variance are reported for a stochastic evolutionary process with 30 verifications per task. Untrained base Qwen selectors already reach 39.22 / 39.25. The largest per-domain gain (Language Models, 18.04 → 32.57) is matched by the untrained base selectors (32.48 / 32.54), so it cannot be attributed to critic training. Two baselines are missing: random selection among the 16 candidates, which isolates the effect of generating 16 options; DeepSeek-GRM as a selector, given its strong static numbers (Dir-3 58.1 and F1 73.6 for the 27B model, above the 9B SFT critic). Cost is not accounted for. Please report total compute or API cost, and wall-clock time.

Questions for authors

N/A

4 comments

Response

We thank the reviewer for their comments and suggestions. We address the questions proposed by the reviewer as below:

W1: Train–Test Splits, Task Overlap, and Majority-Class Baselines

“The paper does not say how the 1.5K benchmark examples were held out. Each parent has 20 children sharing identical task text, criteria and parent summary. A random example-level split would therefore put siblings in both train and test. Please clarify, and report results with a parent-level or task-level split. … Please also report the label distribution of the benchmark and a majority-class baseline.”

We thank the reviewer for requesting these clarifications. Table R1 reports the label distribution of the 1,500-example benchmark. Combining BigWin, MediumWin, and SmallWin gives an Improve proportion of 58.27%. The majority-class baselines therefore achieve 58.27% for both Improve Accuracy and Dir-3 Accuracy, and 26.87% for 5-Ladder Accuracy by always predicting BigWin. Table R2 compares our trained critic models against these baselines; both models outperform the majority-class baseline on all three metrics. For the static benchmark, the parents in train and test set are not overlapped. However, tasks can overlap between the two partitions. We will clarify this distinction in the manuscript.

“Downstream, 10 of the 30 MLS-Lite tasks were used in critic training (Table 13). For those tasks, the OpenEvolve seed parent is the same baseline used as the starting parent during data collection. Please clarify, and report held-out-only results throughout.”

For downstream OpenEvolve evaluation, we acknowledge that the overlapping tasks use the same initial baseline solutions as data collection. These results measure performance within previously encountered tasks and should be distinguished from held-out-task performance. To make this distinction explicit, Table R3 separately reports averages over the seven domains containing training-task overlap and the four domains without such overlap, giving equal weight to each domain.

Compared with their corresponding base-model selectors, Critic-9B and Critic-35B improve the held-in domain average by 2.96 and 3.18 points, respectively. On held-out domains, the improvements are smaller but remain positive, at 1.12 and 0.28 points. Relative to OpenEvolve without screening, the two trained critic models improve the held-out domain average by 5.19 and 4.47 points. Overall, the gains from critic training are larger on held-in domains and smaller but positive on held-out domains. This pattern is consistent with the critic learning task-relevant knowledge from execution feedback, which transfers only partially to unseen domains.

Table R1: Label Distribution of Evaluation Benchmark

LabelPercentage
BigWin26.87%
MediumWin16.73%
SmallWin14.67%
Tie15.33%
Lose26.40%
Total100.00%

Table R2: Static Evaluation of Trained Critic Models Against Majority-Class Baselines

MetricMajority-class baselineCritic-9BCritic-35B
Improve Accuracy58.27%66.7% (+8.43 %)68.5% (+10.23 %)
Dir-3 Accuracy58.27%60.7% (+2.43 %)63.3% (+5.03 %)
5-Ladder Accuracy26.87%41.5% (+14.63 %)45.4% (+18.53 %)

Table R3: Domain-Averaged OpenEvolve Performance on Held-In and Held-Out Domains

Domain splitNoneGemini-3.1-ProGemini-3.5-FlashBase-9BCritic-9BBase-35BCritic-35B
Held-in Domain Average (7)41.1041.6941.7539.4842.4439.4742.65
Held-out Domain Average (4)34.6936.6335.4538.7639.8838.8839.16

W2: OpenEvolve Improvement Modest and More Baselines

“The improvements are about 1.5–2.7 domain-average points over no screening or Gemini screening. … Untrained base Qwen selectors already reach 39.22 / 39.25. The largest per-domain gain (Language Models, 18.04 → 32.57) is matched by the untrained base selectors (32.48 / 32.54), so it cannot be attributed to critic training.”

We thank the reviewer for raising these questions. Under the same verification budget, adding our critic model improves the domain-average score by up to 2.74 points over OpenEvolve without screening. We agree that the large gain in Language Models (18.04 → 32.57) is also achieved by the untrained base selectors and therefore cannot be attributed to post-training; it indicates that idea-level screening within our pipeline, which compares neutral parent and child summaries under task-specific criteria and aggregates multiple critiques before execution, can already be effective without training. Post-training provides consistent gains on top of this pipeline: compared with their same-size base selectors, Critic-9B and Critic-35B achieve higher scores in 10 of the 11 domains in Table 2 of the paper, and higher three-seed mean scores on 5 and 7 of the 7 tasks in our seed evaluation below (Table R4), respectively. For context, GPT-6 Astra scores 3.8 points higher than GPT-5.6 Sol on the public MLS-Bench Lite leaderboard with the Codex harness held fixed (source: https://mls-bench.com/leaderboard). Although the evaluation protocols differ, this provides a rough sense of scale.

To examine robustness across random seeds, we completed three-seed evaluations on the seven least computationally demanding tasks in the benchmark, selected solely by execution cost so that all methods could be evaluated across three seeds within the available compute budget and rebuttal timeline, and report the mean and standard deviation for each method. We also added two baselines. Random selects a proposal at random from the candidate pool, while GRM-27B uses the off-the-shelf DeepSeek-GRM-27B generalist generative reward model as the candidate selector. Random performs on par with no screening, with seven-task averages of 54.58 and 54.38, respectively, indicating that the gains from screening do not come merely from generating a larger candidate pool. Five of these seven tasks overlap with the tasks used in critic training. The evaluated tasks and their detailed results are provided in Table R5.

The gains vary by task. On ML Clustering Algorithm, OpenEvolve without screening reaches 56.23, compared with 56.76 for Critic-9B, suggesting limited additional benefit from screening under the current proposer. On Time Series Imputation, Critic-9B and Critic-35B achieve 66.49 ± 0.25 and 66.30 ± 0.85, respectively, compared with 60.20 ± 2.59 without screening. Both trained critic models also improve over their corresponding base-model selectors on this task.

Table R4: Number of Tasks with Higher Three-Seed Mean Scores than Each Baseline.

Higher mean score than…Critic-9BCritic-35B
Same-size base selector5/77/7
No screening5/76/7
Random selection5/76/7
Gemini selector5/73/7
GRM-27B selector5/75/7

Table R5: OpenEvolve Performance on Seven Tasks (Mean ± Standard Deviation over Three Seeds)

TaskNoneRandomGeminiBase-9BCritic-9BBase-35BCritic-35BGRM-27B
AI4Bio Mutation Effect Prediction49.47 ± 0.9049.48 ± 1.4450.35 ± 1.1649.36 ± 1.9348.79 ± 1.5449.69 ± 0.9949.86 ± 1.6351.22 ± 1.28
Graph Generation73.74 ± 1.9974.06 ± 1.9774.77 ± 2.6172.26 ± 5.7170.97 ± 1.8069.11 ± 1.0074.33 ± 2.0674.10 ± 4.04
ML Clustering Algorithm56.23 ± 0.2255.74 ± 0.4556.55 ± 1.0356.10 ± 0.2056.76 ± 0.4356.14 ± 0.2256.43 ±0.2856.21 ± 0.38
ML Dimensionality Reduction47.60 ± 2.7647.50 ± 3.8746.60 ± 2.1843.44 ±2.5249.32 ± 0.9037.48 ± 0.4639.67 ± 1.8740.57 ± 0.98
Multi-objective Optimization44.97 ± 0.5044.47 ± 0.9148.77 ± 0.4544.41 ± 0.1049.48 ± 0.1749.25 ± 0.0250.66 ± 0.6648.53 ± 1.60
Exogenous Forecasting48.47 ± 1.1248.11 ± 1.3747.73 ± 0.4448.30 ± 0.5649.88 ± 1.4847.13 ± 0.2349.36 ± 1.0348.83 ± 0.79
Time Series Imputation60.20 ± 2.5962.70 ± 3.0865.78 ± 0.4763.40 ± 0.3166.49 ± 0.2558.24 ± 1.2866.30 ± 0.8559.38 ± 4.64

W2: OpenEvolve Improvement Modest and More Baselines (Cont'd)

“Cost is not accounted for. Please report total compute or API cost, and wall-clock time.”

Table R6 reports the per-task inference cost in OpenEvolve, using median values across all tasks. Generating, implementing, and summarizing the 16 candidates costs 126.66 USD per task, and this cost is shared by all screening settings. On top of this, using Gemini-3.1-Pro as the selector costs 330.80 USD in API calls. In contrast, Critic-9B and Critic-35B require only 0.11 and 0.31 H200 GPU-hours, or 0.54 and 1.51 USD even at a higher-end rate of 4.866 USD per GPU-hour (https://vast.ai/pricing/gpu/H200, highest price over last 90 days), which overstates our screening cost. Thus, the total cost is 330.26 USD (72.2%) and 329.29 USD (72.0%) less than the Gemini-3.1-Pro selector. The additional cost compared to no screening should be weighed against the verification cost, which is identical across settings and depends heavily on the task: a single verification on MLS-Lite-Bench takes from about one minute to nearly ten hours, and each task uses 30 verifications. For computationally expensive tasks, the additional inference cost is therefore small relative to the verification cost that screening helps allocate.

Table R6: Inference API and GPU Cost for OpenEvolve (Median across all tasks)

Cost componentNoneGemini-3.1-ProBase-9BBase-35BCritic-9BCritic-35B
H200 GPUs used for local screening0—1212
Local screening cost (GPU-hours)0.00—0.310.910.110.31
Idea proposal API cost (USD)3.5656.9656.9656.9656.9656.96
Code implementation API cost (USD)3.7359.6859.6859.6859.6859.68
Code summary API cost (USD)0.6310.0210.0210.0210.0210.02
Candidate screening API cost (USD)0.00330.800.000.000.000.00
Candidate screening GPU cost (USD)0.000.001.514.430.541.51
Total inference cost (USD)7.92457.46128.17131.09127.20128.17

Dear Reviewer 3HLW,

Thank you again for your review. Our November 13 response clarifies the train/test split and reports the label distribution with majority-class baselines (Tables R1–R2). It also reports held-in and held-out domain results separately (Table R3), adds three-seed results with Random and DeepSeek-GRM-27B selector baselines (Tables R4–R5), and gives the total inference cost (Table R6). We would appreciate your feedback on whether these address your questions. Thank you for your time.

Reviewer_dUR4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper trains idea-level critics to predict whether a proposed ML modification will improve the current solution. The trained critics are used for idea selection and policy training. The results show better accuracy under a fixed verification budget.

Critical strengths
  1. Verification is very expensive for ML agents, reducing evaluation cost is useful. The problem is important.
  2. The method make sense,and is simple.
Critical weaknesses
  1. I think the efficiency comparison is not fair. The method generates 16 ideas and runs 16 critic evaluations for each. Although the number of real verifications is the same, these extra generation costs are not fully considered. I suggest reporting the total cost.

  2. The policy training comparison needs more ablations. Critic-in-loop GRPO run about 110 steps, while verify-all GRPO only run about 40 steps. It can be determined whether the improvement are come from reward or come from more training.

  3. In Table 2, the trained critics only improve the average score by around 2 points over the base model. improvements are relatively small.

Questions for authors
  1. Can the authors provide the total cost, including LLM calls and critic inference?

  2. Are the improvements in table 2 robust under different random seeds?

  3. How does critic-in-loop GRPO perform when using the same number of training steps as verify-all GRPO?

5 comments

Response

We thank the reviewer for their questions and suggestions. We address the comments of the reviewers as below:

W1 and Q1: Total cost

Although the number of real verifications is the same, these extra generation costs are not fully considered; Can the authors provide the total cost, including LLM calls and critic inference?

We thank the reviewer for this suggestion. We report the total cost for both settings in which the critic model is used: inference-time evolution in OpenEvolve and policy training.

[Inference-time evolution (OpenEvolve).] Table R1 reports the per-task inference cost in OpenEvolve, using median values across all tasks. Generating, implementing, and summarizing the 16 candidates costs 126.66 USD per task, and this cost is shared by all screening settings. On top of this, using Gemini-3.1-Pro as the selector costs 330.80 USD in API calls. In contrast, Critic-9B and Critic-35B require only 0.11 and 0.31 H200 GPU-hours, or 0.54 USD and 1.51 USD even at a higher-end rate of 4.866 USD per GPU-hour (https://vast.ai/pricing/gpu/H200, highest price over last 90 days), which overstates our screening cost. Thus, the total cost is 330.26 USD (72.2%) and 329.29 USD (72.0%) less than the Gemini-3.1-Pro selector. The additional cost compared to no screening should be weighed against the verification cost, which is identical across settings and depends heavily on the task: a single verification on MLS-Lite-Bench takes from about one minute to nearly ten hours, and each task uses 30 verifications. For computationally expensive tasks, the additional inference cost is therefore small relative to the verification cost that screening helps allocate.

Table R1: Inference API and GPU Cost for OpenEvolve (Median across all tasks)

Cost componentNoneGemini-3.1-ProBase-9BBase-35BCritic-9BCritic-35B
H200 GPUs used for local screening0—1212
Local screening cost (GPU-hours)0.00—0.310.910.110.31
Idea proposal API cost (USD)3.5656.9656.9656.9656.9656.96
Code implementation API cost (USD)3.7359.6859.6859.6859.6859.68
Code summary API cost (USD)0.6310.0210.0210.0210.0210.02
Candidate screening API cost (USD)0.00330.800.000.000.000.00
Candidate screening GPU cost (USD)0.000.001.514.430.541.51
Total inference cost (USD)7.92457.46128.17131.09127.20128.17

W1 and Q1: Total cost (Cont'd)

[Policy training.] Table R2 below estimates the GPU-hours spent on verification and reward assignment during policy training. Each verification has a 30-minute limit, and we use the observed median execution time of approximately 20 minutes to estimate verifier GPU usage. Critic inference cost is estimated from the per-critique GPU cost measured in our OpenEvolve experiments. We assume that the per-step policy update cost is identical across the two settings and exclude it from all estimates. Including critic model inference, the estimated cost over 40 training steps decreases from 3,413 GPU-hours for verify-all to 1,241 and 1,246 GPU-hours with the 9B and 35B critic models, respectively, corresponding to approximately 64% savings. These numbers exclude environment initialization and cluster queueing. Running fewer verification jobs also reduces the demand for environment setup and cluster scheduling, offering an additional practical benefit when scaling training.

Table R3 below estimates the API costs of initial code implementation and code summarization. Both methods require Gemini to implement the candidate ideas, while critic-in-loop additionally generates code summaries as inputs to the critic model. Table R4 below combines these API costs with the verification and reward-assignment GPU costs. Because verify-all relies more heavily on GPU-based verification, we use a lower-end rate of 3.00 USD per GPU-hour, the lowest H200 rental rate on Vast.ai over the past 90 days (source: https://vast.ai/pricing/gpu/H200). This rate understates the cost of verify-all and therefore gives a conservative estimate of our savings. At the same 40-step training budget, the combined estimated cost decreases from 10,497.01 USD for verify-all to 4,145.03 USD and 4,157.81 USD for critic-in-loop with the 9B and 35B models, respectively. This corresponds to savings of approximately 60.5% and 60.4%, showing that the reduction in verification GPU usage outweighs the additional summarization API cost. Under the same verification budget, critic-in-loop runs approximately 110 steps at a combined cost of 11,398.80 USD and 11,434.02 USD, approximately 9% higher than verify-all at 40 steps.

Table R2: Computation Cost for Policy Training for Verify-All vs. Critic-in-Loop

Cost componentVerify-all, 1 stepCritic-in-loop 9B, 1 stepCritic-in-loop 35B, 1 stepVerify-all, 40 stepsCritic-in-loop 9B, 40 stepsCritic-in-loop 35B, 40 stepsCritic-in-loop 9B, 110 stepsCritic-in-loop 35B, 110 steps
Empirical verifications2569393~10k~3,700~3,700~10k~10k
Verifier GPU-hours85.3330.9830.983,413.331,239.041,239.043,407.363,407.36
Critic model inference GPU-hours0.000.060.170.002.356.616.4518.19
Verification and reward assignment GPU-hours85.3331.0431.153,413.331,241.391,245.653,413.813,425.55

Table R3: Gemini API Costs During Policy Training

Cost componentVerify-all, 1 stepCritic-in-loop, 1 stepVerify-all, 40 stepsCritic-in-loop, 40 stepsCritic-in-loop, 110 steps
Initial code implementation calls25625610,24010,24028,160
Code summary calls0256010,24028,160
Initial code implementation API cost (USD)6.436.43257.02257.02706.81
Code summary API cost (USD)0.004.100.00163.84450.56
API cost subtotal (USD)6.4310.52257.02420.861,157.37

Table R4: Total Cost(USD) Comparison at 3.00 USD per GPU-hour

Cost componentVerify-all, 40 stepsCritic-in-loop 9B, 40 stepsCritic-in-loop 35B, 40 stepsCritic-in-loop 9B, 110 stepsCritic-in-loop 35B, 110 steps
Verification and reward assignment GPU cost (USD)10,239.993,724.173,736.9510,241.4310,276.65
Implementation and summary API cost (USD)257.02420.86420.861,157.371,157.37
Combined estimated cost (USD)10,497.014,145.034,157.8111,398.8011,434.02

W2 and Q3: Critic-in-Loop vs Verify-All Under Same Steps

It can be determined whether the improvement are come from reward or come from more training; How does critic-in-loop GRPO perform when using the same number of training steps as verify-all GRPO?

We thank the reviewer for suggesting this comparison. We added a matched-step ablation on ML Clustering Algorithm and Graph Generation for both model sizes, reporting mean scores over three sampling runs in Table R5 below. At 40 training steps, verify-all performs better in three of the four settings. Critic-in-loop performs better for the 35B policy on ML Clustering Algorithm, achieving 32.77 compared with 31.47. These results are consistent with the benefit of obtaining empirical reward signals for every candidate when the number of training updates is fixed. The trade-off between verification cost and reward reliability is adjustable through the uncertainty threshold. Lowering this threshold routes more candidates to empirical verification, increasing reliance on execution-derived rewards at a higher verification cost per step. When performance within a limited number of updates is the priority, a larger verification budget per step can therefore be allocated through more conservative routing. Our reported configuration prioritizes verification efficiency, with approximately 36.3% of candidates requiring empirical verification.

Under this configuration, extending critic-in-loop training from 40 to 110 steps improves performance in all four settings and exceeds verify-all at 40 steps under the same approximate budget of 10,000 empirical verifications. The additional updates contribute to the final gains, and enabling these updates through reduced verification requirements is an intended benefit of our approach. The critic model provides useful training rewards for many candidates, allowing the policy to learn from substantially more proposals within the same verification budget.

Table R5: Ablation on Verify-all vs Critic-in-loop under 40 Steps

9BVerify-all 40Critic-in-loop 40Critic-in-loop 110
ml-clustering-algorithm23.7020.0732.91
graph-generation21.0317.9233.40
35BVerify-all 40Critic-in-loop 40Critic-in-loop 110
ml-clustering-algorithm31.4732.7737.02
graph-generation16.0110.5331.70
Reviewer_WiikOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the high cost of empirical execution and verification in evolving machine learning agents. The authors collect executed parent–child solution pairs from MLE-Dojo and MLS-Bench, derive relative-improvement labels from execution outcomes, and train idea-level critics through SFT and GRPO. The critics compare candidates against their parents using task-specific criteria and are applied to candidate screening in OpenEvolve, policy training with uncertainty-based fallback to empirical verification, and test-time adaptation during search. Experiments show that the trained critics improve static prediction and downstream search performance while enabling more policy updates under a fixed empirical-verification budget.

Critical strengths
  1. The paper addresses an important bottleneck in ML evolving agents, where empirical verification can require expensive model training and evaluation.
  2. The combination of execution-derived supervision, task-specific reasoning criteria, and SFT followed by GRPO provides a concrete approach to training specialized idea-level critics.
  3. The paper investigates critic training, candidate selection, continual learning, and policy optimization. The soft-rubric ablation also provides useful evidence about the effect of structured reasoning on critic optimization.
Critical weaknesses
  1. The framework requires independently training and deploying an idea-level critichowever, the paper does not establish why the same functionality cannot be incorporated into the idea proposer's reasoning or self-evaluation process. Although pretrained models are evaluated as separate critics, these comparisons do not demonstrate the necessity of a dedicated model over integrated self-critique or self-ranking. Given the additional training and inference costs, the motivation for this architectural choice remains insufficiently justified.
  2. The improvements in downstream OpenEvolve performance are relatively modest. The trained 9B and 35B critics improve the average score by only 2.29 and 2.13 points over their respective base-model selectors. Moreover, Critic-35B achieves higher static accuracy than Critic-9B but slightly worse downstream performance (41.38 versus 41.51). These results raise questions about how effectively the improved critic accuracy translates into practical benefits and whether the additional training and larger model sizes are justified.
  3. The critic is optimized primarily for three-way outcome classification, whereas its main inference-time application requires ranking multiple candidates using five-level predictions. Improving average classification accuracy does not necessarily improve the identification of the best candidate, especially when rare but substantial improvements are more valuable than common small improvements.
  4. Critic-35B achieves identical downstream performance with Critic-9B, suggesting that the paper's critic objective do not necessarily translate into better candidate selection or final agent performance. Given the higher inference cost of the larger model, the paper faces scalability issues.
Questions for authors
  1. The critic predicts discretized improvement categories, why not directly predict the expected improvement score relative to the parent solution? Have the authors compared classification-based ranking with regression-based scoring or pairwise ranking objectives in terms of candidate selection and downstream evolution performance?
  2. Why does Critic-35B achieve higher static classification accuracy than Critic-9B but nearly identical downstream OpenEvolve performance?
  3. If the motivation of the critic is to save compute, how much faster can critic-guided search or policy training reach the same target performance compared with the corresponding critic-free baseline?
7 comments

Response

We thank the reviewer for their feedback. We address the comments and questions of the reviewers as below:

W1: The motivation for architectural choice

why the same functionality cannot be incorporated into the idea proposer's reasoning or self-evaluation process; Given the additional training and inference costs, the motivation for this architectural choice remains insufficiently justified

We thank the reviewer for raising this question. Our motivation for training a dedicated critic model is that strong idea-generation capabilities do not necessarily imply reliable assessment of the empirical value of those ideas. In ML tasks, a conceptually plausible modification may succeed or fail depending on the dataset, parent implementation, optimization dynamics, and task constraints. This separation follows standard practice in RLHF and verifier-guided generation, where the reward model is trained separately from the policy on preference or outcome data rather than relying on the policy’s own judgment[1, 2]. We therefore investigate whether execution-derived supervision can improve this assessment capability beyond directly prompting the proposer to evaluate its own ideas.

  • Our experiments already include a same-model self-evaluation baseline. In Section 4.1, Gemini-3.1-Pro serves as both the idea proposer and the selector evaluating its own generated candidates. This baseline uses the same candidate-generation and selection procedure as our trained critics. Although the selector is invoked in a separate call rather than within the proposal context, it uses the same underlying model as the proposer and therefore tests whether the proposer’s own assessment capability suffices. Table 2 of the paper reports domain-average scores of 38.77 without screening, 39.85 with Gemini self-evaluation, and 41.51/41.38 with our trained 9B/35B critics.
  • The data-construction process further illustrates the difficulty of this assessment task. Within our SFT/evaluation pool, approximately 55.5% of examples received no correct five-level prediction across eight initial, unassisted Gemini critiques, suggesting Gemini-3.1-Pro’s five-level prediction is at most 44.5%. In contrast, after post-training, our pipeline achieve Dir-3 accuracies of 60.7%/63.3%, shown in Table 1 of the paper.
  • A dedicated, open-weight critic also enables test-time alignment as agent evolution proceeds. While keeping the API-based proposer unchanged, we can update the critic using newly acquired execution feedback to adapt its assessments during the current search. Section 4.2 directly evaluates this capability: the critic is updated using outcomes from verifications already performed by OpenEvolve, without additional verifier calls. Thus, the critic is a trainable component that can continue adapting from interaction with the environment.

The critic model is trained once per checkpoint and can be reused across tasks and OpenEvolve runs, allowing its training cost to be amortized. In Table R1 below, we show the inference cost from API and GPU usage; numbers are medians across all tasks. Under our OpenEvolve screening protocol, using Gemini-3.1-Pro as the selector incurred 330.80 USD in API costs. Critic-9B and Critic-35B used 0.11 and 0.31 H200 GPU-hours, respectively (GPU-hours are summed over all GPUs used). At a reference rate of 4.866 USD per H200 GPU-hour (https://vast.ai/pricing/gpu/H200, highest price among last 90 days), their estimated screening costs were 0.54 and 1.51 USD, respectively. All screening settings additionally propose, implement, and summarize 16 candidate ideas per round instead of one, which increases these API costs from 7.92 to 126.66 USD per task; we include these costs in the table. Even including them, the total inference cost per task is 127.20 USD with Critic-9B and 128.17 USD with Critic-35B, compared with 457.46 USD with the Gemini-3.1-Pro selector.

Table R1: Comparison of Inference API and GPU Cost for OpenEvolve(Median among all tasks)

Cost componentNoneGemini-3.1-ProBase-9BBase-35BCritic-9BCritic-35B
H200 GPUs used for local screening0—1212
Local screening cost (GPU-hours)0.00—0.310.910.110.31
Idea proposal API cost (USD)3.5656.9656.9656.9656.9656.96
Code implementation API cost (USD)3.7359.6859.6859.6859.6859.68
Code summary API cost (USD)0.6310.0210.0210.0210.0210.02
Candidate screening API cost (USD)0.00330.800.000.000.000.00
Candidate screening GPU cost (USD)0.000.001.514.430.541.51
Total inference cost (USD)7.92457.46128.17131.09127.20128.17

[1]: Training language models to follow instructions with human feedback.

[2]: Inference-Time Scaling for Generalist Reward Modeling

W2, W4, and Q2: OpenEvolve Improvement Modest and 9B vs 35B Performance

The improvements in downstream OpenEvolve performance are relatively modest.

In our setup, under the same verification budget, adding a critic model improves the OpenEvolve domain-average score by up to 2.74 points over the setting without screening. To put this gain in context, GPT-6 Astra scores 3.8 points higher than GPT-5.6 Sol on the public MLS-Bench Lite leaderboard under the same Codex harness (source: https://mls-bench.com/leaderboard). Given the differences in evaluation protocols, we view this as a rough reference for the magnitude of the gain from adding a critic model.

Critic-35B achieves identical downstream performance with Critic-9B, suggesting that the paper's critic objective do not necessarily translate into better candidate selection or final agent performance. Why does Critic-35B achieve higher static classification accuracy than Critic-9B but nearly identical downstream OpenEvolve performance?

We thank the reviewer for raising this point. Although Critic-35B does not show a downstream advantage over Critic-9B on MLS-Lite-Bench, critic model training improves downstream performance at both model sizes. Compared with their respective base-model selectors, Critic-9B and Critic-35B each outperforms its corresponding base model in 10 of the 11 domains. These results provide evidence that our training approach can translate into practical downstream benefits, even though the additional benefit of the larger model is not apparent in this evaluation.

One possible contributor to the similar downstream scores is the difference in task composition. Many of our critic model training tasks come from MLE-Dojo, whereas Table 2 of the paper evaluates on MLS-Lite-Bench. To see whether the relative performance of the two models changes on a benchmark more represented in training, we evaluated both on three MLE-Dojo tasks held out from critic model training, using the same OpenEvolve configuration as Section 4.1 with 40 evolution rounds and three seeds. We report per-task min–max normalized scores, mapping the best final result across methods to 1.0 and the worst to 0.0 and scaling intermediate results linearly. The results are shown in Table R2 below. Critic-35B outperforms Critic-9B on TGS Salt Identification and AI-Generated Detection and ties it on Amazon Employee Access. Their average normalized scores are 1.000 and 0.913, respectively.

Table R2: Task Performances on MLE-Dojo with OpenEvolve

TaskNoneGeminiBase 9BCritic-9BBase 35BCritic-35B
TGS Salt Identification0.221.000.010.9101.00
AI-Generated Detection0.170.250.750.8301.00
Amazon Employee Access0.000.000.991.000.991.00

More discussion on the rationale behind our choice of a classification objective is presented in the next part of the response.

W3 And Q1: Is classification the right objective and why not predict the expected improvement score?

Improving average classification accuracy does not necessarily improve the identification of the best candidate

We thank the reviewer for the opportunity to clarify the roles of five-level supervision and three-way optimization. Our goal is to learn a reusable assessment of whether a proposed modification improves upon its parent, together with a coarse indication of the improvement magnitude. This assessment supports both inference-time screening and reward provision for policy training. Such design allows easy integrations of the critic model into different agent frameworks.

The five-level labels serve two purposes. First, they provide SFT with both improvement direction and coarse-grained magnitude information. With this supervision, we aim to train the critic model to act as a scorer, assigning each proposed modification a coarse ordinal score that reflects its anticipated improvement over the parent. Second, five-level labels serve as a better quality-filter for teacher-generated critiques. A critique is retained only when its final verdict matches the execution-derived label. This imposes a stricter acceptance criterion on the retained training samples than requiring agreement on improvement direction alone. In GRPO stage, we use three-level directional labels for two reasons. First, we prioritize getting the improvement direction right, because determining whether a modification improves or worsens the parent solution is more important for screening and policy training than distinguishing a MediumWin from a BigWin. Second, this choice also makes the supervision more scalable. A three-way label can be obtained directly from the empirical outcomes of a single parent–child pair, whereas our five-level labeling procedure additionally requires task-specific calibration of improvement magnitudes using a broader collection of execution outcomes. Although GRPO uses only three-way rewards, it does not erode the finer magnitude distinctions learned during SFT: after GRPO, 5-Ladder accuracy increases from 31.5% to 41.5% for the 9B critic and from 38.5% to 45.4% for the 35B critic (Table 1 of the paper). We will clarify this design rationale and the role of each supervision level in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.