acceptodds
Under review as a conference paper at ICLR 2027

Recursive Harness Self-Improvement for Frontier Reasoning Data Synthesis

Abstract

Generating progressively harder reasoning problems requires synthesis procedures that adapt as the task distribution evolves. Existing task-level recursion reuses generated problems as seeds but leaves the construction harness unchanged. We present task–harness co-evolution, a framework for recursive harness self-improvement(RSI) in reasoning-data synthesis. Online self-improvement converts intermediate solver failures into reusable skills during generation. Post-task self-improvement revises skills, prompts, and workflows after each batch, adopting candidates only when they generate harder valid tasks within a bounded cost increase. Model weights and verification criteria remain fixed. Across mathematics, coding, and science, mean solver accuracy decreases from 100.0% to 54.8% over fourteen evolution rounds. Ablations show that combining both update schedules produces harder tasks than fixed-harness recursion or either schedule alone. The resulting data improves downstream SFT and GRPO performance. In particular, a 27B student fine-tuned on 10K synthesized mathematics examples achieves 62.5% mean-16 accuracy on APEX, competitive with selected frontier-model references.These results support adapting the synthesis harness alongside the tasks to generate increasingly challenging data with downstream training value.

Primary area:
foundation or frontier models, including LLMs
Keywords:
ai for science, data synthesis, agentic ai

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Thk1OpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper jointly evolves reasoning tasks and their generation harness. It turns solver failures into reusable skills during generation and revises prompts and workflows after each batch. Experiments show increasing difficulty for a fixed solver and downstream gains from training on the generated data.

Critical strengths
  1. Jointly adapting tasks and the generation process is a sensible extension of fixed-harness synthesis.
  2. The ablation compares both update schedules and their combination, which produces the hardest tasks for the evaluated solver.
  3. The frozen-harness experiment on held-out math seeds supports reuse beyond the original task chains .
  4. The generated data improves multiple benchmarks under both SFT and GRPO, supporting its training utility.
Critical weaknesses

1.The downstream gains are not isolated to harness evolution. There is no direct training comparison between this system’s fixed- and evolved-harness data, and equal example counts do not ensure equal training tokens.

  1. Difficulty is measured against the solver used during optimization. It remains unclear whether the generated tasks are also harder for independent solvers.

  2. The additional search budget is not controlled. Reported costs exclude harness updates and candidate search and evaluation, leaving the comparison under equal total budgets unresolved.

  3. Task validity needs stronger evidence. Multiple verifier models are used, but disagreement handling, independent correctness audits, and per-round validity rates are not reported.

Questions for authors
  1. Can the authors compare downstream training on their own fixed- and evolved-harness data, with matched domain mixtures and training tokens?
  2. Does the full method still outperform fixed-harness synthesis under the same total generation budget, including all update, search, and evaluation costs?
  3. How are verifier disagreements res
3 comments
AuthorsOpenReview ↗

Response to Reviewer Thk1

Thank you for recognizing our method and downstream utility.

Q1: Own-data training attribution

We compare Qwen3.5-35B-A3B SFT on our Fixed/Full data from the same seeds, with matched example counts and domain proportions.

Table R12: SFT mean-16 accuracy (%).

Student / training dataAPEXAPEX ShortlistHMMT Feb 2026LCB v6
Starting Qwen3.5-35B-A3B4.244.380.974.8
SFT on our Fixed-harness data5.247.983.777.1
SFT on our Full RSI data7.349.786.779.1
Full − Fixed (pp)+2.1+1.8+3.0+2.0

With matched example counts and domain proportions, Full improves all four benchmarks by 2.1/1.8/3.0/2.0 pp. This direct comparison demonstrates better downstream performance from our evolved data at the same dataset size.

Q2: Complete generation budgets

Both methods produce 20,000 tasks passing production verification; independent GLM evaluates each entire corpus. Self-improvement costs include online updates, Meta Agent, candidate generation, both-side evaluation, rejections, and retries.

Table R13: Reported USD/task and corresponding difficulty.

ConfigurationRetained tasksProduction (USD/task)Self-improvement (USD/task)End-to-end (USD/task)GLM 5.3 full-corpus accuracy
Fixed-harness20,0001.2840.0001.28478.3%
Full RSI20,0001.0340.2311.26559.4%

After including online updates, Meta Agent, candidate search, and both-side evaluation, Full is still 1.5% cheaper overall with 18.9 pp lower GLM accuracy. The difficulty gain in this comparison is therefore obtained at lower end-to-end expenditure, rather than through a larger total budget.

Q3: Verification and human auditing

Answer construction and cross-verification use tools; difficulty evaluation is tool-free. After Check, Qwen3.8-Max and DeepSeek-V4-Pro cross-solve with tools. Both matching permits automated acceptance; disagreement or neither matching escalates to GPT6-Sol Ultra via Codex CLI, then human review if unresolved.

We separately audited Full outputs from 30 chains/domain at R10/R14:

Table R14: Human validity audit.

DomainSeed chainsAudited tasksValidInvalidValidity rate
Mathematics306059198.3%
STEM306058296.7%
Coding306059198.3%
Total90180176497.8%

Human auditing found 176/180 tasks valid (97.8%), with 1/2/1 invalid tasks in mathematics/STEM/coding. Pooling R10/R14, this provides human evidence for the validity of late-round outputs across all three domains.

Q4: Transfer to an independent solver

GLM 5.3 never participated in generation, Verify, Meta Agent, Gate, or selection. DeepSeek/Qwen participated in feedback and provide complementary comparisons. Evaluation covers 250 chains (100 math/50 STEM/100 coding), with four attempts/task per checkpoint.

Table R15(a): Solver accuracy (%).

SolverConfigurationR0R4R8R12R14
GLM 5.3 (independent)Task-only95.586.374.873.670.8
GLM 5.3 (independent)Online-only95.585.972.468.063.6
GLM 5.3 (independent)Post-task-only95.583.569.760.756.3
GLM 5.3 (independent)Full RSI95.585.166.251.147.9
DeepSeek-V4-ProTask-only98.386.672.371.773.4
DeepSeek-V4-ProOnline-only98.385.471.067.166.3
DeepSeek-V4-ProPost-task-only98.384.969.163.958.7
DeepSeek-V4-ProFull RSI98.384.168.359.454.8
Qwen3.8-MaxTask-only99.490.483.780.882.7
Qwen3.8-MaxOnline-only99.491.685.380.277.3
Qwen3.8-MaxPost-task-only99.488.283.178.672.4
Qwen3.8-MaxFull RSI99.485.580.072.264.9

At R14, Full lowers GLM/DeepSeek/Qwen accuracy by 22.9/18.6/17.8 pp versus Task-only. The independent GLM results show difficulty transfer to a solver excluded from optimization.

Table R15(b): Distinct independent-GLM cohorts.

Independent GLM 5.3 evaluation cohortFixed / Task-onlyFull RSIAccuracy reduction
R14 outputs from 250 seed chains across three domains70.8%47.9%22.9 pp
Full-corpus evaluation of 20,000 production tasks per method78.3%59.4%18.9 pp

The 250-chain R14 cohort and 20,000-task production corpora are separate; the latter are evaluated in full.

Thank you for the rebuttal and for addressing my questions. After considering your response, I raised my overall score.

AuthorsOpenReview ↗

Response to Reviewer Thk1

Thank you for taking the time to examine the additional experiments and reconsider your assessment. Your questions prompted us to compare the same student model trained on data from our fixed and evolved harnesses, and to include candidate evaluation and retries in the end-to-end generation cost. These additions make it easier for readers to judge both what harness evolution contributes and what it costs. We appreciate your careful reading and constructive feedback.

Reviewer_MhmxOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper generates progressively harder reasoning problems by updating both the problems and the procedure used to construct them. Solver failures guide new construction strategies, while batch-level feedback drives changes to prompts and workflows. The resulting data is then used for SFT and RL training.

Critical strengths
  • The idea is reasonable, and the ablations suggest that combining immediate skill updates with batch-level harness revision produces harder problems than either mechanism alone.
  • The frozen-harness experiment on unseen seeds is a useful result: the evolved procedure transfers beyond its original problem chains. The downstream training gains are also encouraging.
Critical weaknesses

The biggest missing comparison is training students on their own fixed-harness versus evolved-harness data under matched training tokens. Their dataset helps, but the downstream experiments don’t establish how much harness evolution contributes to those gains.

  • The cost comparison leaves out the meta-agent, harness search, and candidate evaluations. Without total costs and a matched-budget baseline, we don’t know whether the method beats spending the same compute on a simpler generation pipeline.
  • Difficulty is measured using the same solver that guides the evolution. It is unclear how much the generated problems challenge other models versus targeting this solver's particular weaknesses. Validity audits and rejection/fallback statistics are also missing.
  • There are no independent-run uncertainty estimates. Harness changes are accepted whenever measured accuracy decreases, without accounting for evaluation noise, so the robustness of the improvements is difficult to assess.
  • More about the overall approach, it reads primarily as an engineering recipe, with limited analysis of what the system actually discovers
Questions for authors

See weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer Mhmx

Thank you for the constructive feedback.

Q1: Own-data SFT

Our Qwen3.5-35B-A3B SFT compares Fixed/Full data from the same seeds, matching example counts and domain mix.

Table R6: SFT mean-16 accuracy (%).

Student / training dataAPEXAPEX ShortlistHMMT Feb 2026LCB v6
Starting Qwen3.5-35B-A3B4.244.380.974.8
SFT on our Fixed-harness data5.247.983.777.1
SFT on our Full RSI data7.349.786.779.1
Full − Fixed (pp)+2.1+1.8+3.0+2.0

Full improves all four benchmarks at the same dataset size and domain mix.

Q2: Complete generation costs

Each method has 20,000 pipeline-accepted outputs. Overhead includes online updates, Meta Agent, candidate generation, both-side evaluation, rejections and retries.

Table R7: USD/task; full-corpus GLM.

ConfigurationRetained tasksProduction (USD/task)Self-improvement (USD/task)End-to-end (USD/task)GLM 5.3 full-corpus accuracy
Fixed-harness20,0001.2840.0001.28478.3%
Full RSI20,0001.0340.2311.26559.4%

Including all update/search overhead, Full is 1.5% cheaper and lowers GLM accuracy by 18.9 pp for the same output count.

Q3: Independent evaluation and validity

GLM was excluded from generation, Verify, Meta Agent, Gate and selection; DeepSeek/Qwen participated. We evaluate 250 chains (100 math/50 STEM/100 coding), four attempts/task/checkpoint.

Table R8(a): Solver accuracy (%).

SolverConfigurationR0R4R8R12R14
GLM 5.3 (independent)Task-only95.586.374.873.670.8
GLM 5.3 (independent)Online-only95.585.972.468.063.6
GLM 5.3 (independent)Post-task-only95.583.569.760.756.3
GLM 5.3 (independent)Full RSI95.585.166.251.147.9
DeepSeek-V4-ProTask-only98.386.672.371.773.4
DeepSeek-V4-ProOnline-only98.385.471.067.166.3
DeepSeek-V4-ProPost-task-only98.384.969.163.958.7
DeepSeek-V4-ProFull RSI98.384.168.359.454.8
Qwen3.8-MaxTask-only99.490.483.780.882.7
Qwen3.8-MaxOnline-only99.491.685.380.277.3
Qwen3.8-MaxPost-task-only99.488.283.178.672.4
Qwen3.8-MaxFull RSI99.485.580.072.264.9

At R14, Full lowers GLM/DeepSeek/Qwen accuracy by 22.9/18.6/17.8 pp.

Table R8(b): GLM cohorts.

Independent GLM 5.3 evaluation cohortFixed / Task-onlyFull RSIAccuracy reduction
R14 outputs from 250 seed chains across three domains70.8%47.9%22.9 pp
Full-corpus evaluation of 20,000 production tasks per method78.3%59.4%18.9 pp

We separately audited 30 Full chains/domain at R10/R14:

Table R9: Human validity audit.

DomainSeed chainsAudited tasksValidInvalidValidity rate
Mathematics306059198.3%
STEM306058296.7%
Coding306059198.3%
Total90180176497.8%

Validity is 97.8% (R10/R14 pooled). All candidates rejected or budget exhausted: retain Online, with at most two retries.

Q4: Gate noise and stability

Gate uses 150 shared inputs (50/domain), matched models/tools/budgets, and four solves per output. Repeated solving and cross-domain coverage aim to reduce noise. Subsequent recursion, independent GLM evaluation, and frozen transfer assess overall benefits.

Q5: Mechanisms and reuse

Online updates skills during generation; post-task updates skills/prompts/workflows using batch feedback.

Table R10: R14 component ablations (%).

ConfigurationDeepSeek-V4-ProGLM 5.3 (independent)Qwen3.8-Max
Task-only73.470.882.7
Online-only66.363.677.3
Post-task-only58.756.372.4
Full RSI54.847.964.9
Task-only − Full (pp)18.622.917.8

Task-only fixes the harness; Online/Post-task enable one update each. Full beats both.

Table R11: Frozen transfer (%).

Frozen harnessNew-chain R0R2R4R6R8R10
Initial H₀100.095.587.078.070.570.0
Evolved H₁₄100.090.083.570.062.552.0

On 50 unseen math seeds, frozen H14 lowers DeepSeek's new-chain R10 accuracy by 18.0 pp versus H0, supporting reuse. We observe Online adding high-quality skills and Post-task pruning ineffective skills, revising workflows, and moving Check earlier.

I consider my original concerns regarding the cost and single-solver limitations to have been addressed by the author's response. However, as mentioned in my original review, I still consider this work to be more of an engineering effort than a scientific contribution. I stand by my original score of 2

AuthorsOpenReview ↗

Response to Reviewer Mhmx

Thank you for clarifying that the additional experiments addressed your concerns about cost and reliance on a single solver. Your remaining question is what the work shows beyond building a more effective generation system. Our central finding is that not only can tasks become harder over successive rounds, but the procedure for constructing them can itself improve, and that improvement can transfer to new seeds even after the procedure is frozen. Three points explain this distinction.

  1. Tasks and their generation procedure co-evolve. In task-only recursion, each round passes its generated tasks to the next round, but the procedure that constructs them stays fixed. Our method carries forward both the tasks and a generation harness revised in light of solver failures and verification feedback. This distinction raises a testable question: if we replace the seed tasks, does the advantage disappear with the original task chains, or does it transfer with the harness?
  2. Online evolution and post-task evolution build on one another, and their combination outperforms either alone. Online evolution turns solver failures into construction skills that can guide the very next generation decision. Post-task evolution starts from the resulting online harness and uses feedback from the completed batch to decide which skills to retain or remove and how to revise prompts or workflows. The two forms of evolution thus draw on different evidence: an individual failure and the outcomes of a whole batch. At round 14, GLM 5.3, which played no role in generation or selection, achieved 70.8% accuracy on Task-only outputs, 63.6% with online evolution alone, 56.3% with post-task evolution alone, and 47.9% with both. In this comparison, their combination produced harder tasks for the independent solver than either form of evolution alone.
  3. The frozen-harness experiment tests whether the gain transfers beyond the original task chains. We froze the initial and evolved harnesses, applied them to the same 50 unseen math seeds, and allowed only the tasks to evolve. After ten rounds, DeepSeek-V4-Pro achieved 70.0% accuracy on tasks produced by the initial harness versus 52.0% on those produced by the evolved harness. This is our most direct answer to the scientific question: neither condition inherited harder tasks from the original chains, and neither harness continued to adapt. From the same new starting points, the evolved procedure still constructed harder tasks.
Reviewer_LYbbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper uses LLM agents to generate harder reasoning questions and improve the generation process. Given a set of seed questions, a generator modifies them, a solver tries to answer them, and LLM verifiers assess their validity. A meta-agent converts solver failures into reusable construction skills. It also revises prompts and workflows after each batch. Broader updates are accepted when they produce harder questions while meeting validity and cost constraints. The generated questions seed subsequent rounds. Model weights remain fixed in all previous processes. In the downstream experiments, the authors evaluate the generated data by using it to train student models.

Critical strengths
  1. The paper makes previous attempts reusable.
  2. The two update schedules handle different types of information. It makes the process more interpretable.
  3. The model training in the downstream tasks is useful and shows the effectiveness of the method.
Critical weaknesses
  1. The task difficulty evaluation process is weak and unpredictable. It relies on a single model and may become targeted at the evaluator model's weaknesses, but not a real hard task.
  2. The verification process is also insufficient. Although Appendix F lists a few LLMs for verification, the paper does not clearly establish domain-specific adjudication procedures, disagreement handling, etc. Also, a fixed verifier can make systematic mistakes in the verification process, especially on distinguishing between “the solver failed” and “the reference answer is wrong” is essential here.
  3. Post-task improvement includes candidate generation, evaluation, and retries. The ablation changes both adaptation and search effort. A fixed harness, given the same total budget, might close some of the gap.
Questions for authors
  1. How do you establish trustworthy answers to problems that the generating model struggles to solve?
  2. Does the harness discover generally harder reasoning tasks or exploit one solver’s weaknesses?
  3. Which modification actually produces the gains?
  4. How much improvement comes from adaptation versus additional search and selection?
2 comments
AuthorsOpenReview ↗

Response to Reviewer LYbb

Thank you for recognizing experience reuse, the two update schedules, and downstream utility.

Q1: Answer reliability

Answer construction and verification use tools; difficulty evaluation is tool-free. Check failures return for revision. Tool-enabled Qwen3.8-Max and DeepSeek-V4-Pro cross-solve: both matching the reference permits automated acceptance; disagreement or neither matching escalates to GPT6-Sol Ultra via Codex CLI, then human review if unresolved.

We separately audited Full outputs from 30 chains/domain at R10/R14 (180 tasks):

Table R1: Human validity audit.

DomainSeed chainsAudited tasksValidInvalidValidity rate
Mathematics306059198.3%
STEM306058296.7%
Coding306059198.3%
Total90180176497.8%

Human auditing found 176/180 tasks valid (97.8%). Invalid counts were 1/2/1 for mathematics/STEM/coding, pooling R10/R14.

Q2: Difficulty beyond optimization solvers

GLM 5.3 never participated in generation, Verify, Meta Agent, Gate, or selection. We also report DeepSeek/Qwen, which contributed generation feedback. Evaluation covers 250 chains (100 math/50 STEM/100 coding), four attempts/task at each checkpoint. Task-only fixes the harness; Online/Post-task enable one update each.

Table R2: Solver accuracy (%).

SolverConfigurationR0R4R8R12R14
GLM 5.3 (independent)Task-only95.586.374.873.670.8
GLM 5.3 (independent)Online-only95.585.972.468.063.6
GLM 5.3 (independent)Post-task-only95.583.569.760.756.3
GLM 5.3 (independent)Full RSI95.585.166.251.147.9
DeepSeek-V4-ProTask-only98.386.672.371.773.4
DeepSeek-V4-ProOnline-only98.385.471.067.166.3
DeepSeek-V4-ProPost-task-only98.384.969.163.958.7
DeepSeek-V4-ProFull RSI98.384.168.359.454.8
Qwen3.8-MaxTask-only99.490.483.780.882.7
Qwen3.8-MaxOnline-only99.491.685.380.277.3
Qwen3.8-MaxPost-task-only99.488.283.178.672.4
Qwen3.8-MaxFull RSI99.485.580.072.264.9

At R14, Full reduces GLM/DeepSeek/Qwen accuracy by 22.9/18.6/17.8 pp versus Task-only. The independent GLM results show difficulty transfer to a solver excluded from optimization.

Q3: Update contributions

Table R3: R14 component ablations (%).

ConfigurationDeepSeek-V4-ProGLM 5.3 (independent)Qwen3.8-Max
Task-only73.470.882.7
Online-only66.363.677.3
Post-task-only58.756.372.4
Full RSI54.847.964.9
Task-only − Full (pp)18.622.917.8

Full improves over both single-update variants. We also froze H0/H14 on 50 shared math seeds excluded from H14 evolution/selection; only tasks evolved.

Table R4: Frozen transfer; DeepSeek accuracy (%).

Frozen harnessNew-chain R0R2R4R6R8R10
Initial H₀100.095.587.078.070.570.0
Evolved H₁₄100.090.083.570.062.552.0

At new-chain R10, H14 yields 52.0% versus H0's 70.0%. The frozen evolved harness generates harder tasks on unseen seeds, supporting reuse of learned construction strategies. Observed updates include adding high-quality skills, pruning ineffective skills, and moving Check earlier.

Q4: Adaptation versus extra search

Both methods produce 20,000 tasks passing production verification; GLM evaluates each full corpus. Overhead includes online updates, Meta Agent, candidate generation, both-side evaluation, rejections, and retries.

Table R5: Reported costs (USD/task) and difficulty.

ConfigurationRetained tasksProduction (USD/task)Self-improvement (USD/task)End-to-end (USD/task)GLM 5.3 full-corpus accuracy
Fixed-harness20,0001.2840.0001.28478.3%
Full RSI20,0001.0340.2311.26559.4%

Including all self-improvement overhead, Full is 1.5% cheaper overall with 18.9 pp lower GLM accuracy. Production savings of USD 0.250/task cover the USD 0.231/task update overhead. Together with frozen-harness transfer, this supports reusable improvements that generate harder tasks at lower total expenditure in this comparison.

AuthorsOpenReview ↗

Response to Reviewer LYbb

Thank you again for raising the questions of reference-answer reliability and whether the generated tasks are harder for other solvers. We reported two additional results in our response: human reviewers judged 176 of 180 audited tasks valid (97.8%), and at round 14, GLM 5.3, an independent solver not involved in generation or selection, achieved 70.8% accuracy on Task-only tasks versus 47.9% on Full RSI tasks. If time permits, we would appreciate hearing whether these results address your concerns or whether there is a point we could clarify.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary for the Area Chair

We thank the Area Chair and reviewers for their careful reading. The main scientific question was whether recursive generation improves only the tasks in a particular chain or also improves the procedure that constructs them. We summarize the evidence most relevant to that distinction.

  1. The evolved harness transfers to new task chains. We froze the initial and evolved harnesses and applied both to the same 50 unseen math seeds. After ten rounds with no further harness evolution, DeepSeek-V4-Pro achieved 70.0% accuracy on tasks from the initial harness versus 52.0% on tasks from the evolved harness. Because both conditions started from the same new seeds, the evolved harness did not inherit harder tasks from its original chains.
  2. The two forms of evolution are complementary. Across 250 seed chains spanning math, STEM, and coding, round-14 accuracy for GLM 5.3 was 70.8% with task-only recursion, 63.6% with online evolution alone, 56.3% with post-task evolution alone, and 47.9% with both. GLM took no part in generation, verification, or selection.
  3. Human auditing supports task validity. Human auditors examined 180 retained tasks from rounds 10 and 14 and judged 176 valid (97.8%): 59/60 in math, 58/60 in STEM, and 59/60 in coding.
  4. The difficulty gain did not require a higher cost per retained task. Each method produced 20,000 retained tasks. Including meta-agent calls, candidate generation, evaluation of both harnesses, and retries, the end-to-end cost was USD 1.265 per task for Full RSI versus USD 1.284 for the fixed harness. GLM accuracy on the two complete corpora was 59.4% and 78.3%, respectively.
  5. The generated data also improved downstream training. With the same student model, training example count, domain mix, and SFT configuration, training on Full RSI data rather than our own fixed-harness data improved accuracy by 2.1 points on APEX, 1.8 on APEX Shortlist, 3.0 on HMMT, and 2.0 on LCB v6.

Following the discussion, Reviewer Thk1 raised their overall rating. Reviewer Mhmx indicated that the cost and single-solver concerns had been addressed, while maintaining their reservation about the scientific contribution. On that remaining point, we believe the frozen-harness comparison is the most informative result: even after adaptation stopped, the evolved procedure retained its advantage on new seeds.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.