- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies automated post-training for large language models under fixed compute budgets without human-curated training data. The authors argue that predefined recipe spaces (e.g., AutoPipe) cannot discover new data, while open-ended code-editing agents (e.g., PostTrainBench, AutoTrainess) suffer from low sample efficiency and execution crashes. The authors propose Structured Recipe Optimization (SRO), which operates over a task-adapted recipe space. The Recipe-Space Designer guides an agent to define discrete choices across data sources, synthesis templates, mixtures, and hyperparameters. The Multi-Domain Recipe Optimizer uses domain-structured trial memory and Random Forest surrogates with Expected Hypervolume Improvement (EHVI) to select candidate recipes. The method is evaluated on Qwen3-1.7B-Base and Qwen3-4B-Base across Math, Code, and Science benchmarks under fixed wall-clock time.
Critical strengths
The paper proposes a well-engineered system architecture with several practical strengths:
-
Practical system workflow: Constraining agent search to discrete recipe choices with an execution workflow prevents the script syntax errors and runtime crashes common in open-ended coding agents (Section 3.1, Figure 2).
-
Transparent reporting of failure cases: The appendix documents optimization trajectories and acknowledges notable failure modes, including severe performance drops on AIME and BFCL regressions (Appendix D.3, Appendix D.4).
Critical weaknesses
However, several experimental and methodological aspects require further clarification:
-
Absence of statistical replication: All results (Tables 1–3) are single runs without random seeds, standard deviations, or significance tests. On small test sets like AIME 2025 (30 problems, 3.3% per problem), performance deltas fall within sampling noise.
-
Search-time evaluation overlap: Each trial evaluates on 50% of the target benchmark test samples to select the best checkpoint (Appendix B.1), which is then reported on the full test set. This creates data overlap between search-time selection and final reporting.
-
Severe surrogate sparsity and unstable BO selection: Multi-domain runs complete only 8 to 17 total trials (Table 10). After 5 warm-up trials, the Random Forest surrogate fits on just 3 to 12 data points. Removing EHVI in Table 3 actually improves scores on AIME (10.0 vs 3.3) and GPQA (36.4 vs 30.8).
-
Confounded ablations: The ablation "w/o structured recipe search" removes both the recipe space and the execution workflow (Table 3), attributing general execution stability to the search algorithm.
-
Severe negative transfer in the Science domain: On Qwen3-1.7B (Table 2), multi-domain GPQA drops from 34.2 (single-domain, Table 1) to 16.1 (barely above the 14.1 base model), lagging far behind AutoTrainess (30.8).
Questions for authors
Please refer to the detailed concerns and clarification points listed in the Critical Weaknesses section above.
7 comments
W2: Search-time evaluation overlap. Thank you for highlighting search-time evaluation overlap. We conducted additional strict 50/50 split experiments that separate search-time selection from final reporting: all methods use the same first 50% of examples, fixed across trials, for search and checkpoint selection, while the remaining 50% is reserved exclusively for final evaluation.
The original experiments follow PostTrainBench’s protocol, which permits querying the target benchmark for evaluation during iteration but prohibits training on its test questions or answers. SRO uses the fixed first 50% during search (50 examples for HealthBench-Easy), receiving only scores without inspecting individual evaluation examples or their errors; the submitted results were evaluated on the full benchmarks. The additional experiments assess performance when final evaluation examples are entirely excluded from search and checkpoint selection.
Given the rebuttal time constraints and computational budget, these experiments cover both model sizes and all three methods using DeepSeek-V4-Flash. The two tables below report each method’s same final selected checkpoint on the search and held-out halves, respectively.
50/50 Split — Search-Set Results
| Model | Method | AIME | GSM8K | BFCL | HumanEval | GPQA | HealthBench | Mean |
|---|---|---|---|---|---|---|---|---|
| Qwen3-1.7B | PostTrainBench | 0.0000 | 0.4325 | 0.4400 | 0.4268 | 0.2634 | 0.0967 | 0.2766 |
| Qwen3-1.7B | AutoTrainess | 0.0000 | 0.5448 | 0.9400 | 0.6585 | 0.3170 | 0.1044 | 0.4274 |
| Qwen3-1.7B | SRO (Ours) | 0.0000 | 0.7102 | 0.9400 | 0.6220 | 0.2857 | 0.1365 | 0.4491 |
| Qwen3-4B | PostTrainBench | 0.0667 | 0.5599 | 0.9200 | 0.6951 | 0.2991 | 0.1861 | 0.4545 |
| Qwen3-4B | AutoTrainess | 0.1333 | 0.6252 | 0.9600 | 0.6220 | 0.3259 | 0.1848 | 0.4752 |
| Qwen3-4B | SRO (Ours) | 0.0000 | 0.8134 | 0.9800 | 0.6707 | 0.3527 | 0.2720 | 0.5148 |
50/50 Split — Held-Out Results
| Model | Method | AIME | GSM8K | BFCL | HumanEval | GPQA | HealthBench | Mean |
|---|---|---|---|---|---|---|---|---|
| Qwen3-1.7B | PostTrainBench | 0.0000 | 0.3894 | 0.4000 | 0.2073 | 0.3304 | 0.0699 | 0.2328 |
| Qwen3-1.7B | AutoTrainess | 0.0000 | 0.5273 | 0.9000 | 0.2439 | 0.2589 | 0.0913 | 0.3369 |
| Qwen3-1.7B | SRO (Ours) | 0.0000 | 0.5773 | 0.9000 | 0.2927 | 0.2679 | 0.1167 | 0.3591 |
| Qwen3-4B | PostTrainBench | 0.0000 | 0.6652 | 0.7800 | 0.3902 | 0.3214 | 0.1515 | 0.3847 |
| Qwen3-4B | AutoTrainess | 0.0667 | 0.6591 | 0.8600 | 0.2195 | 0.2902 | 0.1454 | 0.3735 |
| Qwen3-4B | SRO (Ours) | 0.0000 | 0.8318 | 0.9200 | 0.4024 | 0.3080 | 0.2094 | 0.4453 |
With no example overlap between search-time selection and final held-out evaluation, SRO achieves the highest mean score at both model sizes, exceeding the strongest baseline by 2.22 percentage points on 1.7B and 6.06 points on 4B. These results show that SRO’s aggregate advantage persists in this setting when search-feedback examples are excluded from final scoring.
W3: Surrogate sparsity and BO selection stability. Thank you for highlighting surrogate sparsity and unstable BO selection. We first clarify that the five warm-up trials are included in surrogate training, and each subsequent fit uses all valid observations available at that point. Thus, 3–12 refers to the number of post-warm-up trials, not the surrogate’s training sample count.
SRO does not rely solely on BO to search the entire recipe space. It separates candidate construction from trial selection: the LLM proposes candidates using the available choices, task knowledge, and structured trial history; EHVI then selects among them using predicted cross-domain outcomes and uncertainty. These components are complementary: the LLM supplies prior knowledge and contextual reasoning, while EHVI uses accumulated observations to balance exploration and exploitation.
Regarding the higher AIME and GPQA scores after removing EHVI: in the same 4B ablation, AIME increases from 3.3 to 10.0 and GPQA from 30.8 to 36.4, but HealthBench decreases from 26.9 to 14.2 and the six-benchmark average decreases from 53.8 to 51.9. This reflects cross-domain trade-offs; EHVI does not guarantee improvement over the ablated variant on every benchmark.
To further address run-to-run variability in BO selection, we conducted two additional independent full search runs. Across three runs, full SRO retains a higher mean aggregate score than w/o EHVI:
| Model | Full SRO | w/o EHVI selection |
|---|---|---|
| Qwen3-1.7B | 44.88 ± 1.20 | 40.35 ± 0.72 |
| Qwen3-4B | 51.44 ± 2.02 | 48.56 ± 2.93 |
Results are the mean ± sample standard deviation of the six-benchmark average (%) using DeepSeek-V4-Flash. Full SRO improves the mean aggregate score by 4.53 percentage points on 1.7B and 2.88 points on 4B, showing that its aggregate advantage persists across independent repeats rather than appearing only in the original single run.
W4: Separating structured search from execution support. Thank you for suggesting a clearer separation between the contributions of the execution workflow and the search algorithm. The original w/o structured recipe search ablation compares two end-to-end optimization approaches: SRO integrates recipe selection with its executable workflow, whereas an open-ended agent organizes experiments and execution itself. Accordingly, that ablation removes both the structured recipe space and its associated workflow. To further distinguish their contributions, we added w/o structured recipe search (shared workflow), retaining the same executable workflow, data validation, data-generation tools, and 20-hour runtime limit.
The control reuses SRO’s discovered data sources, augmentation operators, and mixture and training choices, together with the same validation, caching, training, and evaluation pipeline. The distinction is how the methods search over these same options: SRO constructs, recombines, and selects candidate recipes using domain-specific evidence; the control receives resource descriptions and directly chooses its next configuration based on its own trial history, without SRO’s structured candidate construction, candidate batches/pool, explicit cross-domain recombination, or EHVI selection. Both methods select the final checkpoint by the unweighted six-benchmark average.
Given the rebuttal time constraints and computational budget, the currently completed shared-workflow controls comprise one run per model size using DeepSeek-V4-Flash. We compare these with SRO’s mean ± sample standard deviation over three independent full search runs, including the original run.
Shared-Workflow Control — Results
Scores are percentages. Control denotes w/o structured recipe search (shared workflow). SRO entries report three-run means ± sample standard deviations; Control entries report single-run scores.
| Model | Method | AIME | GSM8K | BFCL | HumanEval | GPQA | HealthBench | Avg. |
|---|---|---|---|---|---|---|---|---|
| Qwen3-1.7B | Control | 0.00 | 57.92 | 90.00 | 45.73 | 30.13 | 18.35 | 40.36 |
| Qwen3-1.7B | SRO (Ours) | 2.23 ± 3.87 | 68.03 ± 10.26 | 92.33 ± 1.15 | 58.30 ± 0.35 | 24.87 ± 7.62 | 23.53 ± 1.27 | 44.88 ± 1.20 |
| Qwen3-4B | Control | 3.33 | 80.82 | 95.00 | 62.80 | 31.03 | 24.99 | 49.66 |
| Qwen3-4B | SRO (Ours) | 1.10 ± 1.91 | 83.17 ± 3.87 | 93.67 ± 1.53 | 73.37 ± 9.24 | 32.87 ± 2.00 | 24.50 ± 4.07 | 51.44 ± 2.02 |
SRO’s three-run mean exceeds the currently available single-run control score by 4.52 percentage points on 1.7B and 1.78 points on 4B. This comparison supports the additional value of SRO’s structured search mechanism beyond the executable workflow, with the proposer, EHVI, and memory ablations further examining its individual components. The higher six-benchmark averages align with our objective of joint multi-domain optimization. Additional independent runs of this control are underway, and we will report the completed results in the final version.
W5: GPQA degradation and multi-domain trade-offs. Thank you for highlighting the GPQA degradation. SRO’s multi-domain objective is to improve overall performance across all six benchmarks. Across three independent runs, SRO achieves the highest mean six-benchmark average among the evaluated methods at both model sizes. On Qwen3-1.7B, it achieves 44.88 ± 1.20, compared with 37.14 ± 6.93 for AutoTrainess, an improvement of 7.74 percentage points (mean ± sample standard deviation, %).
Regarding GPQA specifically, the three-run mean scores are 24.87 for SRO and 26.70 for AutoTrainess, a difference of 1.83 percentage points. SRO therefore remains below AutoTrainess on this benchmark while achieving a higher aggregate score.
The single-domain experiment specifically optimizes GPQA, whereas multi-domain optimization jointly targets six benchmarks and selects the final checkpoint by their unweighted average, without a non-regression constraint on each benchmark. This permits cross-domain trade-offs: our repeated-run results support improved aggregate multi-domain performance, rather than improvement on every individual benchmark.
We sincerely thank the reviewer for the thorough examination of our methodology and results. We appreciate your suggestions on experimental reliability, search design, and cross-domain trade-offs, which have guided additional validation and clearer explanations. We address each point below.
W1: Statistical replication and small-test-set variability. Thank you for highlighting statistical replication and variability on small test sets. Each experiment in the submission was evaluated with one independent run, given the GPU time and API costs of complete searches. We have now conducted additional independent full search runs for every method and ablation configuration listed below, reporting mean ± sample standard deviation over three independent runs. Given the rebuttal time constraints and computational budget, these experiments cover both model sizes, all three methods, and all four ablation variants using DeepSeek-V4-Flash.
These repeated runs follow the search and final-evaluation protocols used in the submission and are reported separately from the additional 50/50 split experiments. All scores are percentages. Avg. is the unweighted mean across the six benchmarks.
Repeated-run main results. SRO achieves the highest mean aggregate score at both model sizes, with lower aggregate run-to-run variability than both baselines. It exceeds the strongest baseline at each model size by 6.61 percentage points on 1.7B and 6.98 points on 4B. Its aggregate standard deviations are 1.20 and 2.02, compared with 6.78–6.93 and 3.39–4.99 for the baselines.
Repeated Runs — Main Results
Repeated-run ablation results. Full SRO achieves the highest mean aggregate score at both model sizes, exceeding the four ablated variants by 2.36–10.66 percentage points on 1.7B and 2.06–7.73 points on 4B. The table below reports means and standard deviations for every benchmark, showing both average gains and run-to-run variability.
Repeated Runs — Ablation Results
Regarding AIME 2025’s small test set and sampling noise: with 30 problems, each problem corresponds to approximately 3.3 percentage points in a single evaluation. We therefore do not treat changes of one or two solved problems as evidence of statistically significant improvement. The repeated experiments measure variability across independent search runs; our aggregate conclusion rests on the six-benchmark average at both model sizes, rather than small differences on AIME alone.