- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
EvalEvo organizes autonomous training around prospective evidence acquisition, cross-version training decisions, and evolution of the evaluation/training system. An evaluator collects reliability, diagnostic, and prescriptive evidence; a trainer ranks intervention directions using confidence, historical success, and exploration terms; a global DAG records recipe and evidence dependencies. The paper reports 100-run AutoResearch experiments and eight-run RL searches for math and coding, including component ablations, selected compute accounting, and detailed evolution traces. In RL, all candidate runs restart from the same base checkpoint rather than inheriting weights.
Critical strengths
-
A practically relevant decision problem with informative process evidence. The paper focuses on deciding which expensive training experiment to run, and its trajectories illustrate why scalar scores alone may be insufficient. The math trace distinguishes long-response reasoning failures from incorrect reward treatment; the coding trace separates balanced data coverage from a bundled tail guard. These examples make the proposed evaluation-to-intervention workflow concrete and explain how a lower-scoring version can still provide useful evidence.
-
Evaluation across distinct training settings with explicit budget distinctions. The study covers language-model pretraining and two RL domains. Under the stated budgets, EvalEvo reaches 0.9899 bpb versus 0.9980 for AutoResearch, and improves Math/Coding Avg@8 to 76.71/53.50. Appendix A.3 explicitly separates weight-updating training from inference-only decision support and reports costs to first exceed a common reference. The controlled probe and system-evolution ablations also help characterize the full system.
Critical weaknesses
- Adaptive benchmark access leaves generalization and reliability unresolved. Appendix A.4 explicitly allows the trainer to inspect benchmark scores and evaluation rollouts, then uses this feedback for interventions and model selection; Table 2 reports the best of eight runs on those same benchmarks. This is a clearly disclosed benchmark-feedback optimization protocol, but keeping scoring rules fixed does not provide an independent final test. With only 78 math and 175 coding questions, one decoding seed, and no repeated autonomous-search trajectories, improvements of 1.43 and 2.21 points over the human recipe may include adaptive selection effects. The paper does not establish that these gains persist on an untouched evaluation set or under search/training randomness. I am not claiming that evaluation examples were directly added to training.
- The evidence does not isolate the distinctive contribution from existing adaptive trainers. EvoTrainer already studies rollout-level diagnosis, evolving training harnesses, backtesting, and reusable skills (Chen et al., 2026, https://arxiv.org/abs/2606.03108). Appendix A.2 explains why the authors did not reproduce it, which is reasonable to disclose, but the incremental contribution of prospective probing and DAG-based decisions still needs a controlled test. AutoResearch is not a matched ablation of cross-version access: it differs in other evaluation and decision mechanisms. Moreover, w/o System Evolution simultaneously freezes procedures, tools, and the reusable experience store. Thus the large ablation gap cannot be attributed to system adaptation alone, and the proposed ranking rule is not separately evaluated.
Questions for authors
- Independent evaluation and variation across searches. Can the final selected recipes be evaluated on a benchmark partition or additional problems never exposed to either agent, with paired repeated runs or appropriate uncertainty estimates? Please distinguish training randomness, rollout randomness, and autonomous-search randomness, and state whether any evaluation-derived content entered generated training data.
- Controlled attribution and executable decision details. Could the authors compare global DAG access with a sequential-history variant while holding the evaluator, tools, experience store, agent model, and budgets fixed? Please provide concrete candidate-generation/scoring prompts, the rule defining relevant previous trials n_t(d), and a trace of candidates, confidence values, pre-run hypotheses, and verified outcomes. These details would help assess the prospective decision mechanism beyond the general adaptive-training workflow.
- Full-trajectory resource accounting. Appendix A.3 usefully reports GPU-hours to first exceed the human reference. What are the complete eight-run training and decision-support totals, the agent-side inference costs, and their corresponding values for the ablations? How are short-run parameter-updating diagnostic interventions counted against the eight completed-run limit? A full account would clarify the scope of the fixed-budget claim.
4 comments
Response to Reviewer vgTX Part [2]: Controlled Attribution and Executable Decision Details
Reviewer's Concern:
Could the authors compare global DAG access with a sequential-history variant while holding the evaluator, tools, experience store, agent model, and budgets fixed? Please provide concrete candidate-generation/scoring prompts, the rule defining relevant previous trials, and a trace of candidates, confidence values, pre-run hypotheses, and verified outcomes.
We sincerely thank the reviewer for these constructive suggestions. We appreciate the opportunity to clarify how EvalEvo generates, ranks, and verifies prospective training decisions, and to highlight the concrete decision traces already provided in our Appendix.
(1) Executable candidate generation and scoring.
EvalEvo does not select interventions through unconstrained LLM exploration. As described in Sections 3.3–3.4 and Appendix A.4, it starts with a predefined intervention space covering training hyperparameters, reward functions, RL algorithms, and training recipes.
The LLM examines the current reliability, diagnostic, and prescriptive evidence together with historical findings to select, refine, or extend candidate directions.
Candidates are ranked using:
Specifically:
- : The LLM assesses confidence in a direction based on diagnostic evidence, prior knowledge, and accumulated experience, producing a score in .
- : The LLM identifies semantically relevant previous trials based on their intervention directions and historical context, rather than simply counting all earlier experiments.
- : Counts the relevant trials with verified positive outcomes.
This combines evidence-supported confidence, historical effectiveness, and exploration, prioritizing promising interventions without permanently excluding previously unsuccessful directions.
Crucially, the selected decision includes a falsifiable hypothesis recorded before training. After the run, the Evaluator checks the observed outcome against this hypothesis using the established verification criteria. Both positive and unsuccessful findings are retained for subsequent decisions.
(2) Global DAG versus sequential search.
AutoResearch provides a representative sequential, single-incumbent search baseline. In contrast, EvalEvo's global DAG explicitly represents cross-version dependencies, allowing training recipes and diagnostic evidence to be reused from different branches, rather than restricting exploration to the latest or best-performing version.
Under the same 100-experiment budget, EvalEvo achieves 0.9899 bpb versus 0.9980 for AutoResearch, surpassing its final result after only 23 experiments (Table 1).
(3) Concrete decision traces already documented.
Our paper provides actual version-level decision traces, rather than only final performance scores:
- Appendix B, Table 8: Records interventions and outcomes for v22–v24. After full-context attention increases MFU from 21.1% to 38.9%, EvalEvo revisits previously ineffective width expansion, improving bpb from 0.9950 to 0.9905.
- Appendix C, Tables 9–10: Document all eight Math/Coding interventions, their source versions, dependency types, and decision contexts.
- Figure 5 and Appendix D: Visualize cross-version dependencies and report complete eight-round evaluation outcomes, including unsuccessful interventions.
For example, M8 combines M2's training recipe (75.05% Avg@8) with reward-integrity evidence from the lower-performing M7 (73.61%), achieving 76.71%. This illustrates how EvalEvo uses complementary historical findings instead of merely continuing a single evolutionary path.
(4) Controlled component attribution.
The w/o Probe ablation removes targeted evidence acquisition while retaining the remaining decision mechanisms, reducing Math/Coding Avg@8 from 76.71/53.50 to 74.86/50.71 (Table 4).
The w/o System Evolution ablation retains the DAG but freezes adaptive procedures, decision tools, and experience accumulation (Appendix A.5). Its performance gap therefore reflects their combined contribution, rather than an isolated DAG effect.
We agree that AutoResearch is a sequential-search baseline, not an otherwise identical DAG-only ablation. Nevertheless, the controlled probing ablation, explicit decision formulation, and documented cross-branch trajectories jointly substantiate the prospective evidence-to-decision mechanism.
We thank the reviewer for emphasizing implementation transparency. We will clarify the LLM-based candidate selection, semantic relevance assessment, confidence scoring, and pre-run hypothesis verification procedures in the revised manuscript.
Response to Reviewer vgTX Part [3]: Full-Trajectory Resource Accounting
Reviewer's Concern:
What are the complete eight-run training and decision-support totals, the agent-side inference costs, and their corresponding values for the ablations? How are short-run parameter-updating diagnostic interventions counted against the eight completed-run limit?
We sincerely thank the reviewer for emphasizing the importance of transparent compute accounting. We would like to clarify two key points: (i) EvalEvo does not perform additional parameter-updating diagnostic runs outside the eight-run budget; and (ii) its measured GPU-based decision-support overhead is explicitly accounted for and remains small relative to training compute.
(1) Strict training-budget enforcement.
As specified in Appendix A.3, all autonomous methods receive eight completed weight-updating runs per domain, each capped at 3,200 optimizer steps on 32 NVIDIA H20 GPUs. All candidates start from the same Qwen3.5-9B checkpoint.
Importantly, EvalEvo performs no additional parameter-updating diagnostic training outside these eight runs. Targeted evidence acquisition between training runs consists of inference-only operations, such as rollout analysis, failure diagnosis, and hypothesis verification.
Thus, diagnostic probing does not introduce hidden training opportunities. Any parameter-updating intervention is part of the prescribed training budget rather than an additional uncounted experiment.
(2) Explicit accounting of decision-support computation.
Appendix A.3 separates realized compute into:
- Training GPU-hours: Rollout generation, forward/backward passes, and optimizer updates within RL training runs.
- Decision-support GPU-hours: Inference-only evidence acquisition, diagnosis, verification, and adaptive evaluation.
Tables 5–6 report cumulative GPU-hours until reaching a common performance threshold—the human-engineered RL reference:
| Task | Method | Round | Training GPU-h | Decision GPU-h |
|---|---|---|---|---|
| Math | EvalEvo | 3 | 4,512 | 17.92 |
| Math | AutoResearch | Not reached | — | — |
| Coding | EvalEvo | 2 | 6,240 | 27.36 |
| Coding | AutoResearch | 3 | 7,104 | 0 |
The measured decision-support overhead is only 0.40% on Math and 0.44% on Coding.
More importantly, EvalEvo remains more compute-efficient after including this overhead. On Coding, it exceeds the human-engineered reference using 6,267.36 total GPU-hours, compared with 7,104 GPU-hours for AutoResearch, representing an 11.8% reduction. On Math, EvalEvo crosses the reference within three rounds, whereas AutoResearch does not reach it within eight.
These results show that EvalEvo's efficiency advantage is not an artifact of excluding GPU-based diagnostic computation.
(3) Clarifying full-trajectory and agent-side costs.
We agree that the scope of the reported measurements should be explicit. Tables 5–6 present cumulative costs to the first common performance milestone, rather than complete eight-run totals. This choice permits compute comparison at matched achieved quality, rather than at potentially different final performance levels.
The ablations follow the same eight-run training limits and evaluation protocols (Appendix A.5). However, their complete trajectory-specific GPU-hour totals, as well as a separate breakdown of agent-side LLM inference costs, are not reported in the current tables. Accordingly, the 0.40%/0.44% figures characterize measured GPU-based decision support at the reported milestones, not an independently established upper bound on all agent orchestration costs.
In summary, EvalEvo introduces no hidden parameter-updating experiments, respects the same eight-run training budget, and explicitly accounts for additional inference-only diagnostic GPU computation. Its demonstrated advantage in reaching the human reference persists after that measured overhead is included. We appreciate the reviewer's suggestion and will clarify these accounting boundaries and the scope of the reported compute figures in the revision.
Thanks for the response. It clarifies that both agents receive only aggregate benchmark scores, that diagnostics use training-side or separately generated data, and that between-run probes do not update model parameters outside the eight-run budget. The M8 example shows how the system combined M2’s recipe with diagnostic evidence from M7. The compute reply also clarifies that 0.40% and 0.44% are the measured decision-support GPU overheads relative to training compute at the first round that surpassed the human recipe.
A few questions remain. The same Math and Coding problems are used to guide the search and report its best results, so it remains unclear whether the gains hold on unseen problems or across independent searches. A matched comparison with sequential history and one complete candidate-ranking record would help clarify what the DAG and scoring rule contribute. Full eight-round costs, ablation costs, and agent-side inference usage are still unreported.
Response to Reviewer vgTX Part [1]: Clarifying Strict Benchmark Isolation and Evaluation Reliability
Reviewer's Concern:
We sincerely thank the reviewer for this important question and for the careful assessment of our experimental reliability. We would like to clarify a crucial implementation detail: headline benchmark evaluation is strictly black-box, and neither the Evaluator nor the Trainer has access to benchmark content.
(1) Strict benchmark isolation: Scores only, no data leakage.
Both agents receive only aggregate benchmark scores, with no access to individual questions, ground-truth answers, per-problem predictions, correctness labels, or benchmark evaluation rollouts.
Crucially, all rollout analyses, failure diagnoses, and targeted probes operate exclusively on training data or agent-generated diagnostic data, independently of the headline benchmarks. No benchmark-derived problem content enters generated training data, and neither agent can design interventions tailored to specific benchmark questions.
Thus, benchmarks serve solely as black-box performance evaluators, not as sources of training or diagnostic data. We will clarify this distinction in Appendix A.4, where the term "evaluation rollouts" may have caused confusion.
(2) Controlled evaluation and reliability verification.
All methods share fixed evaluation protocols and matched training budgets (Sections 4.1–4.2; Appendices A.1–A.4). Math training uses 6,428 BigMath-Hard problems, with an overlapping AIME 2024 example removed; Coding uses 9,872 cleaned TACO problems and evaluates on 175 LiveCodeBench-v6 problems.
Moreover, Appendix B documents an additional held-out verification check with a tolerance of ϵ=0.002 bpb. Across the first 24 evolved versions, no accepted intervention improved the primary validation score while failing this additional check. This safeguards subsequent training decisions against unreliable score fluctuations.
(3) Distinguishing sources of randomness.
A full independent search replication requires rerunning the entire evidence-acquisition and eight-round weight-updating process, incurring substantial computational cost. Our evaluation therefore emphasizes controlled experimental conditions, multiple rollouts, reliability verification, and transparent reporting of complete evolution trajectories.
In summary, EvalEvo never accesses benchmark-specific content or uses it to construct training data. Combined with fixed evaluation protocols, explicit reliability checks, and documented search trajectories, our results provide concrete evidence for the effectiveness of evidence-guided autonomous training under matched budgets.
We appreciate the reviewer's constructive suggestions and will make these evaluation safeguards more explicit in the revised manuscript.