- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes training “idea-level” critic models that predict whether a proposed modification to an ML solution will beat its parent (Improve / Tie / Lose). The goal is to let ML evolving agents spend expensive empirical verification only on promising candidates.
Critical strengths
The problem is practical and well motivated. Verification cost is a real bottleneck for evolutionary ML agents, and learned pre-execution screening is a natural lever.
Critical weaknesses
-
The train/test split and task overlap are underspecified. The paper does not say how the 1.5K benchmark examples were held out. Each parent has 20 children sharing identical task text, criteria and parent summary. A random example-level split would therefore put siblings in both train and test. Please clarify, and report results with a parent-level or task-level split. Downstream, 10 of the 30 MLS-Lite tasks were used in critic training (Table 13). For those tasks, the OpenEvolve seed parent is the same baseline used as the starting parent during data collection. Please clarify, and report held-out-only results throughout. Please also report the label distribution of the benchmark and a majority-class baseline.
-
The inference-time gains are modest, and the controls are incomplete. The improvements are about 1.5–2.7 domain-average points over no screening or Gemini screening. No seeds or variance are reported for a stochastic evolutionary process with 30 verifications per task. Untrained base Qwen selectors already reach 39.22 / 39.25. The largest per-domain gain (Language Models, 18.04 → 32.57) is matched by the untrained base selectors (32.48 / 32.54), so it cannot be attributed to critic training. Two baselines are missing: random selection among the 16 candidates, which isolates the effect of generating 16 options; DeepSeek-GRM as a selector, given its strong static numbers (Dir-3 58.1 and F1 73.6 for the 27B model, above the 9B SFT critic). Cost is not accounted for. Please report total compute or API cost, and wall-clock time.
Questions for authors
N/A
4 comments
W2: OpenEvolve Improvement Modest and More Baselines
“The improvements are about 1.5–2.7 domain-average points over no screening or Gemini screening. … Untrained base Qwen selectors already reach 39.22 / 39.25. The largest per-domain gain (Language Models, 18.04 → 32.57) is matched by the untrained base selectors (32.48 / 32.54), so it cannot be attributed to critic training.”
We thank the reviewer for raising these questions. Under the same verification budget, adding our critic model improves the domain-average score by up to 2.74 points over OpenEvolve without screening. We agree that the large gain in Language Models (18.04 → 32.57) is also achieved by the untrained base selectors and therefore cannot be attributed to post-training; it indicates that idea-level screening within our pipeline, which compares neutral parent and child summaries under task-specific criteria and aggregates multiple critiques before execution, can already be effective without training. Post-training provides consistent gains on top of this pipeline: compared with their same-size base selectors, Critic-9B and Critic-35B achieve higher scores in 10 of the 11 domains in Table 2 of the paper, and higher three-seed mean scores on 5 and 7 of the 7 tasks in our seed evaluation below (Table R4), respectively. For context, GPT-6 Astra scores 3.8 points higher than GPT-5.6 Sol on the public MLS-Bench Lite leaderboard with the Codex harness held fixed (source: https://mls-bench.com/leaderboard). Although the evaluation protocols differ, this provides a rough sense of scale.
To examine robustness across random seeds, we completed three-seed evaluations on the seven least computationally demanding tasks in the benchmark, selected solely by execution cost so that all methods could be evaluated across three seeds within the available compute budget and rebuttal timeline, and report the mean and standard deviation for each method. We also added two baselines. Random selects a proposal at random from the candidate pool, while GRM-27B uses the off-the-shelf DeepSeek-GRM-27B generalist generative reward model as the candidate selector. Random performs on par with no screening, with seven-task averages of 54.58 and 54.38, respectively, indicating that the gains from screening do not come merely from generating a larger candidate pool. Five of these seven tasks overlap with the tasks used in critic training. The evaluated tasks and their detailed results are provided in Table R5.
The gains vary by task. On ML Clustering Algorithm, OpenEvolve without screening reaches 56.23, compared with 56.76 for Critic-9B, suggesting limited additional benefit from screening under the current proposer. On Time Series Imputation, Critic-9B and Critic-35B achieve 66.49 ± 0.25 and 66.30 ± 0.85, respectively, compared with 60.20 ± 2.59 without screening. Both trained critic models also improve over their corresponding base-model selectors on this task.
Table R4: Number of Tasks with Higher Three-Seed Mean Scores than Each Baseline.
| Higher mean score than… | Critic-9B | Critic-35B |
|---|---|---|
| Same-size base selector | 5/7 | 7/7 |
| No screening | 5/7 | 6/7 |
| Random selection | 5/7 | 6/7 |
| Gemini selector | 5/7 | 3/7 |
| GRM-27B selector | 5/7 | 5/7 |
Table R5: OpenEvolve Performance on Seven Tasks (Mean ± Standard Deviation over Three Seeds)
| Task | None | Random | Gemini | Base-9B | Critic-9B | Base-35B | Critic-35B | GRM-27B |
|---|---|---|---|---|---|---|---|---|
| AI4Bio Mutation Effect Prediction | 49.47 ± 0.90 | 49.48 ± 1.44 | 50.35 ± 1.16 | 49.36 ± 1.93 | 48.79 ± 1.54 | 49.69 ± 0.99 | 49.86 ± 1.63 | 51.22 ± 1.28 |
| Graph Generation | 73.74 ± 1.99 | 74.06 ± 1.97 | 74.77 ± 2.61 | 72.26 ± 5.71 | 70.97 ± 1.80 | 69.11 ± 1.00 | 74.33 ± 2.06 | 74.10 ± 4.04 |
| ML Clustering Algorithm | 56.23 ± 0.22 | 55.74 ± 0.45 | 56.55 ± 1.03 | 56.10 ± 0.20 | 56.76 ± 0.43 | 56.14 ± 0.22 | 56.43 ±0.28 | 56.21 ± 0.38 |
| ML Dimensionality Reduction | 47.60 ± 2.76 | 47.50 ± 3.87 | 46.60 ± 2.18 | 43.44 ±2.52 | 49.32 ± 0.90 | 37.48 ± 0.46 | 39.67 ± 1.87 | 40.57 ± 0.98 |
| Multi-objective Optimization | 44.97 ± 0.50 | 44.47 ± 0.91 | 48.77 ± 0.45 | 44.41 ± 0.10 | 49.48 ± 0.17 | 49.25 ± 0.02 | 50.66 ± 0.66 | 48.53 ± 1.60 |
| Exogenous Forecasting | 48.47 ± 1.12 | 48.11 ± 1.37 | 47.73 ± 0.44 | 48.30 ± 0.56 | 49.88 ± 1.48 | 47.13 ± 0.23 | 49.36 ± 1.03 | 48.83 ± 0.79 |
| Time Series Imputation | 60.20 ± 2.59 | 62.70 ± 3.08 | 65.78 ± 0.47 | 63.40 ± 0.31 | 66.49 ± 0.25 | 58.24 ± 1.28 | 66.30 ± 0.85 | 59.38 ± 4.64 |
W2: OpenEvolve Improvement Modest and More Baselines (Cont'd)
“Cost is not accounted for. Please report total compute or API cost, and wall-clock time.”
Table R6 reports the per-task inference cost in OpenEvolve, using median values across all tasks. Generating, implementing, and summarizing the 16 candidates costs 126.66 USD per task, and this cost is shared by all screening settings. On top of this, using Gemini-3.1-Pro as the selector costs 330.80 USD in API calls. In contrast, Critic-9B and Critic-35B require only 0.11 and 0.31 H200 GPU-hours, or 0.54 and 1.51 USD even at a higher-end rate of 4.866 USD per GPU-hour (https://vast.ai/pricing/gpu/H200, highest price over last 90 days), which overstates our screening cost. Thus, the total cost is 330.26 USD (72.2%) and 329.29 USD (72.0%) less than the Gemini-3.1-Pro selector. The additional cost compared to no screening should be weighed against the verification cost, which is identical across settings and depends heavily on the task: a single verification on MLS-Lite-Bench takes from about one minute to nearly ten hours, and each task uses 30 verifications. For computationally expensive tasks, the additional inference cost is therefore small relative to the verification cost that screening helps allocate.
Table R6: Inference API and GPU Cost for OpenEvolve (Median across all tasks)
| Cost component | None | Gemini-3.1-Pro | Base-9B | Base-35B | Critic-9B | Critic-35B |
|---|---|---|---|---|---|---|
| H200 GPUs used for local screening | 0 | — | 1 | 2 | 1 | 2 |
| Local screening cost (GPU-hours) | 0.00 | — | 0.31 | 0.91 | 0.11 | 0.31 |
| Idea proposal API cost (USD) | 3.56 | 56.96 | 56.96 | 56.96 | 56.96 | 56.96 |
| Code implementation API cost (USD) | 3.73 | 59.68 | 59.68 | 59.68 | 59.68 | 59.68 |
| Code summary API cost (USD) | 0.63 | 10.02 | 10.02 | 10.02 | 10.02 | 10.02 |
| Candidate screening API cost (USD) | 0.00 | 330.80 | 0.00 | 0.00 | 0.00 | 0.00 |
| Candidate screening GPU cost (USD) | 0.00 | 0.00 | 1.51 | 4.43 | 0.54 | 1.51 |
| Total inference cost (USD) | 7.92 | 457.46 | 128.17 | 131.09 | 127.20 | 128.17 |
Dear Reviewer 3HLW,
Thank you again for your review. Our November 13 response clarifies the train/test split and reports the label distribution with majority-class baselines (Tables R1–R2). It also reports held-in and held-out domain results separately (Table R3), adds three-seed results with Random and DeepSeek-GRM-27B selector baselines (Tables R4–R5), and gives the total inference cost (Table R6). We would appreciate your feedback on whether these address your questions. Thank you for your time.
Response
We thank the reviewer for their comments and suggestions. We address the questions proposed by the reviewer as below:
W1: Train–Test Splits, Task Overlap, and Majority-Class Baselines
We thank the reviewer for requesting these clarifications. Table R1 reports the label distribution of the 1,500-example benchmark. Combining BigWin, MediumWin, and SmallWin gives an Improve proportion of 58.27%. The majority-class baselines therefore achieve 58.27% for both Improve Accuracy and Dir-3 Accuracy, and 26.87% for 5-Ladder Accuracy by always predicting BigWin. Table R2 compares our trained critic models against these baselines; both models outperform the majority-class baseline on all three metrics. For the static benchmark, the parents in train and test set are not overlapped. However, tasks can overlap between the two partitions. We will clarify this distinction in the manuscript.
For downstream OpenEvolve evaluation, we acknowledge that the overlapping tasks use the same initial baseline solutions as data collection. These results measure performance within previously encountered tasks and should be distinguished from held-out-task performance. To make this distinction explicit, Table R3 separately reports averages over the seven domains containing training-task overlap and the four domains without such overlap, giving equal weight to each domain.
Compared with their corresponding base-model selectors, Critic-9B and Critic-35B improve the held-in domain average by 2.96 and 3.18 points, respectively. On held-out domains, the improvements are smaller but remain positive, at 1.12 and 0.28 points. Relative to OpenEvolve without screening, the two trained critic models improve the held-out domain average by 5.19 and 4.47 points. Overall, the gains from critic training are larger on held-in domains and smaller but positive on held-out domains. This pattern is consistent with the critic learning task-relevant knowledge from execution feedback, which transfers only partially to unseen domains.
Table R1: Label Distribution of Evaluation Benchmark
Table R2: Static Evaluation of Trained Critic Models Against Majority-Class Baselines
Table R3: Domain-Averaged OpenEvolve Performance on Held-In and Held-Out Domains