- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper jointly evolves reasoning tasks and their generation harness. It turns solver failures into reusable skills during generation and revises prompts and workflows after each batch. Experiments show increasing difficulty for a fixed solver and downstream gains from training on the generated data.
Critical strengths
- Jointly adapting tasks and the generation process is a sensible extension of fixed-harness synthesis.
- The ablation compares both update schedules and their combination, which produces the hardest tasks for the evaluated solver.
- The frozen-harness experiment on held-out math seeds supports reuse beyond the original task chains .
- The generated data improves multiple benchmarks under both SFT and GRPO, supporting its training utility.
Critical weaknesses
1.The downstream gains are not isolated to harness evolution. There is no direct training comparison between this system’s fixed- and evolved-harness data, and equal example counts do not ensure equal training tokens.
-
Difficulty is measured against the solver used during optimization. It remains unclear whether the generated tasks are also harder for independent solvers.
-
The additional search budget is not controlled. Reported costs exclude harness updates and candidate search and evaluation, leaving the comparison under equal total budgets unresolved.
-
Task validity needs stronger evidence. Multiple verifier models are used, but disagreement handling, independent correctness audits, and per-round validity rates are not reported.
Questions for authors
- Can the authors compare downstream training on their own fixed- and evolved-harness data, with matched domain mixtures and training tokens?
- Does the full method still outperform fixed-harness synthesis under the same total generation budget, including all update, search, and evaluation costs?
- How are verifier disagreements res
3 comments
Thank you for the rebuttal and for addressing my questions. After considering your response, I raised my overall score.
Response to Reviewer Thk1
Thank you for taking the time to examine the additional experiments and reconsider your assessment. Your questions prompted us to compare the same student model trained on data from our fixed and evolved harnesses, and to include candidate evaluation and retries in the end-to-end generation cost. These additions make it easier for readers to judge both what harness evolution contributes and what it costs. We appreciate your careful reading and constructive feedback.
Response to Reviewer Thk1
Thank you for recognizing our method and downstream utility.
Q1: Own-data training attribution
We compare Qwen3.5-35B-A3B SFT on our Fixed/Full data from the same seeds, with matched example counts and domain proportions.
Table R12: SFT mean-16 accuracy (%).
With matched example counts and domain proportions, Full improves all four benchmarks by 2.1/1.8/3.0/2.0 pp. This direct comparison demonstrates better downstream performance from our evolved data at the same dataset size.
Q2: Complete generation budgets
Both methods produce 20,000 tasks passing production verification; independent GLM evaluates each entire corpus. Self-improvement costs include online updates, Meta Agent, candidate generation, both-side evaluation, rejections, and retries.
Table R13: Reported USD/task and corresponding difficulty.
After including online updates, Meta Agent, candidate search, and both-side evaluation, Full is still 1.5% cheaper overall with 18.9 pp lower GLM accuracy. The difficulty gain in this comparison is therefore obtained at lower end-to-end expenditure, rather than through a larger total budget.
Q3: Verification and human auditing
Answer construction and cross-verification use tools; difficulty evaluation is tool-free. After Check, Qwen3.8-Max and DeepSeek-V4-Pro cross-solve with tools. Both matching permits automated acceptance; disagreement or neither matching escalates to GPT6-Sol Ultra via Codex CLI, then human review if unresolved.
We separately audited Full outputs from 30 chains/domain at R10/R14:
Table R14: Human validity audit.
Human auditing found 176/180 tasks valid (97.8%), with 1/2/1 invalid tasks in mathematics/STEM/coding. Pooling R10/R14, this provides human evidence for the validity of late-round outputs across all three domains.
Q4: Transfer to an independent solver
GLM 5.3 never participated in generation, Verify, Meta Agent, Gate, or selection. DeepSeek/Qwen participated in feedback and provide complementary comparisons. Evaluation covers 250 chains (100 math/50 STEM/100 coding), with four attempts/task per checkpoint.
Table R15(a): Solver accuracy (%).
At R14, Full lowers GLM/DeepSeek/Qwen accuracy by 22.9/18.6/17.8 pp versus Task-only. The independent GLM results show difficulty transfer to a solver excluded from optimization.
Table R15(b): Distinct independent-GLM cohorts.
The 250-chain R14 cohort and 20,000-task production corpora are separate; the latter are evaluated in full.