acceptodds
Desk rejected at ICLR 2027

SKILL2TASK: SELF-EVOLVING AGENTIC TASK SYNTHESIS VIA SKILL GRAPH AND PROGRESSIVE VALIDATION

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Tool-using language agents are increasingly expected to complete long-horizon workflows in realistic computing environments, and training them at scale requires a large and diverse supply of executable, verifiable tasks. Yet existing pipelines generate this data with a fixed, one-shot synthesizer built from a hand-designed harness that cannot keep improving from the failures surfaced during construction. We present Skill2Task, a self-evolving framework that turns reusable community skills into executable general-agent tasks and makes the synthesis harness itself the object of optimization. It composes skills through a skill-state graph, progressively validates each environment, instruction, and verifier before use, and feeds construction and rollout diagnostics back to revise the harness each round, which raises the success rate and efficiency of task construction over successive rounds. This loop yields 10,916 verifiable tasks and 10,880 curated trajectories, and supervised fine-tuning on them consistently improves two open backbones of different scales across four agent benchmarks, raising their average benchmark score by up to 18% relative, with the largest gains on skill-composition tasks, where accuracy more than doubles. These results show that evolving the task-synthesis harness itself, rather than freezing it, yields broader and more reliable training data than one-shot synthesis.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agent, supervised fine tuning, self-evolve

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_c6uAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Skill2Task, a framework for synthesizing executable agent tasks from community skills. It builds a skill-state graph using embeddings and clustering, generates tasks through progressive validation and repair, and updates the synthesis harness based on failure feedback. The authors collect 10,880 successful trajectories for SFT on two Qwen3.5 models and report improvements across four agent benchmarks.

Critical strengths
  • The paper presents a reasonably complete pipeline for automated agent task synthesis, including skill composition, task validation, harness refinement, and trajectory collection.

  • The experimental results are encouraging, with consistent improvements across four agent benchmarks and two model sizes.

Critical weaknesses
  • The graph is constructed by clustering skill preconditions and postconditions using Qwen embeddings. However, semantic similarity does not necessarily imply logical compatibility. Two states may be close in embedding space without one satisfying the requirements of the other. The paper lacks direct evaluation of whether the resulting connections and sampled skill paths are logically valid.

  • During progressive validation, the framework can modify tasks to resolve compatibility issues. For example, the task repair case in Appendix A.15 replaces part of the original workflow with direct processing of an existing file. Such repairs may remove the dependencies between the original skills. The paper does not evaluate whether these dependencies are preserved in the final tasks.

  • The evaluation is limited to two Qwen models. Both models belong to the same family, making it difficult to assess how well the synthesized data generalizes. I would like to see the generated tasks evaluated by independent state-of-the-art API agents, such as GPT or Claude, to better assess their difficulty, quality, and actual skill requirements. The GLM-5.2 teacher rollouts alone are not sufficient for this purpose.

  • Table 2 only compares graph-based sampling against random skill combinations. Since random combinations are likely to produce incompatible workflows, this comparison is not particularly convincing. Stronger baselines, such as LLM-based skill planning or simple embedding-based retrieval without graph construction, would help establish whether the proposed graph is actually necessary.

  • Although the valid-bundle rate improves from 42.3% to 91.5%, the corresponding downstream performance gains are much smaller. In addition, the same 200 held-out paths are repeatedly used to select harness updates, raising concerns about overfitting to this validation set. Evaluation on a separate test set and comparison with a stronger static harness under a matched compute budget would make the results more convincing.

Questions for authors
  1. How do the authors verify the logical compatibility of connected skills? Can they provide an evaluation of graph edge correctness and sampled path validity?

  2. How much of the original skill dependency structure is preserved after task repair? Do the final tasks genuinely require all the sampled skills?

  3. Can the authors evaluate the synthesized tasks using independent SOTA API agents and test whether the training gains transfer to non-Qwen models?

  4. How does the proposed graph compare with LLM-based skill planning or simpler semantic retrieval methods?

3 comments

Thank you for your great efforts in reviewing this paper. Please let us know if you still have further concerns, or if you are not satisfied with the current responses, so that we can further update the response as soon as possible.

[W1] Embedding similarity does not establish logical compatibility of graph connections or paths.

[A]: Thank you for asking about skill compatibility. An LLM first extracts each skill's pre/post-execution scenarios; embeddings cluster similar scenarios into shared nodes, with skills forming directed edges. Paths connect output to input scenarios, while downstream validation checks concrete execution requirements. We used gpt-5.6-sol to assess all 10,916 released tasks from their skill sequences, source documents, scenarios, and final instructions. Compatibility required the upstream output scenario to support the downstream requirements. 28,830/30,796 adjacent transitions (93.62%) were judged compatible and 10,414/10,916 workflows (95.40%) coherent, counting uncertain cases as non-passes.

[W2] Task repair may bypass original skill dependencies.

[A]: Thank you for highlighting dependency preservation. Appendix A.15 supplies an existing HDF5 artifact, bypassing upstream modeling/simulation; we will identify this dependency loss explicitly. Two annotators compared the sampled skills, pre-repair requirements, and final instructions/assets for 100 repaired tasks. A dependency counted only if the upstream contribution remained necessary downstream; pre-supplied results did not count. 294/320 dependencies (91.88%) and 84/100 complete chains were retained. Thus, preservation is not universal; we will distinguish sampled path length from retained dependency length.

[W3] Independent API-agent evaluation and transfer beyond Qwen are missing.

[A]: Thank you for suggesting these evaluations. Each model below made four attempts per task on the same 200 tasks, stratified by domain and path length without teacher-success filtering. Tools, skill access, and budgets were matched; references and verifiers were hidden. We report avg@4, the mean binary success rate over all 800 attempts per model. Qwen is the original, pre-SFT checkpoint.

MetricGLM-5.2gpt-5.6-solQwen3.5-35B-A3B
avg@4 (%)63.2574.3852.75

Gemma-4-E4B-it improves on all four benchmarks after Skill2Task SFT, with a +5.18 mean gain under identical evaluation settings (avg@8 on PinchBench, avg@4 elsewhere). Training uses 10,240 retained trajectories and 251M supervised tokens per epoch, a different data budget from Qwen. This single run provides cross-family evidence without establishing training-seed significance.

ModelPinchBenchSkillsBenchSWE-bench MultilingualTB2.0 VerifiedAvg.
Gemma-4-E4B-it (before SFT)54.737.1912.338.4320.67
Gemma-4-E4B-it + Skill2Task SFT60.1712.0717.6713.4825.85

[W4] Random composition is insufficient to establish the graph's value over stronger planners.

[A]: Thank you for suggesting these alternatives. (1) LLM planning: supplying all 84,728 skills in one prompt is impractical; retrieval-assisted planning reduces this burden but requires retrieval and planning for each task. (2) Simple embedding retrieval: similar skill descriptions do not establish execution order or output-to-input compatibility.

Our graph offers (1) scenario-based composition: embeddings cluster LLM-extracted pre/post scenarios into nodes, with skill edges connecting output to input scenarios; concrete compatibility is checked downstream. (2) Reusable, controllable sampling: the graph supports repeated sampling, with path length controlling the intended number of composed skills. These features motivate the design; Table 2 establishes its benefit over random composition, not empirical superiority to the proposed planners.

[W5] Repeated selection-set use risks overfitting; construction gains exceed downstream gains.

[A]: Thank you for asking about generalization and downstream impact. Section 4.3 and Appendix A.9 specify 300 held-in diagnostic paths and 200 disjoint held-out selection paths. Held-in feedback drives edits, while held-out yield rises 42.3% → 91.5%, supporting transfer beyond diagnostic paths. Repeated selection still makes this a validation set, so these results do not establish generalization to an untouched test set. Evolution primarily improves construction efficiency: repair iterations fall 7.42 → 3.91. Downstream gains are a separate measure: Figure 3b caps training-task counts yet improves 46.40 → 49.14. Downstream benchmarks provide neither update feedback nor acceptance criteria, avoiding direct optimization for those evaluations.

Thank you again for your review. We hope that our rebuttal could address your questions and concerns. As the discussion phase is nearing its end, we would be grateful to hear your feedback and wondered if you might still have any concerns we could address. It would be appreciated if you could raise your score on our paper if we address your concerns. We thank you again for your effort in reviewing our paper.

Thanks for the rebuttal and the additional experiments. Most of my concerns have been addressed, and I appreciate the effort.

However, I still have two concerns about the lack of comparisons with stronger alternatives to graph-based sampling, such as retrieval-assisted LLM planning, and the lack of evaluation on independent paths that were never used for diagnosis or update selection. I agree with your explanations, but these comparisons are still needed to establish the value of the graph and self-evolution components.

Also, my “Weaknesses” and “Questions for Authors” are not entirely repetitive, and some specific questions are still only partly answered. For example, the dependency-retention numbers do not fully answer whether the final tasks actually require all the sampled skills.

Given these remaining gaps, I don’t think the current evidence is sufficient for acceptance at ICLR, so I will maintain my score.

Reviewer_HYANOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Skill2Task, a framework that converts community skills into executable agent tasks and improves the task-synthesis harness using construction and rollout feedback. Skills are composed through a state-transition graph, followed by separate generation, validation, and repair stages for the environment, task instruction, and verifier. Successful trajectories are further curated by masking erroneous intermediate actions during SFT. Experiments on two Qwen3.5 models show improvements across four agent benchmarks

Critical strengths

The staged construction pipeline addresses concrete problems in executable task synthesis. Separating environment, instruction, and verifier validation makes failures easier to diagnose and repair.

Recollecting baseline trajectories with the same teacher reduces a major confound in synthetic-data comparisons. Skill2Task improves both backbones across all four benchmarks while using fewer training tokens than the baselines. The component ablations and the per-round training comparison also provide evidence beyond construction success rates.

Critical weaknesses

Appendix A.5 lists 87 SkillsBench tasks evaluated over four runs, and Appendix A.8 states that every task receives a binary score. Under this protocol, the mean percentage must be a multiple of 100/348 (the denominator is 4 × 87 = 348), approximately 0.2874 points. The reported 11.17% is not obtainable after rounding to two decimal places; the nearest attainable value is 11.21%. The reason for this discrepancy is unclear. This may reflect an omitted aggregation rule or a different evaluation subset, but the discrepancy needs to be resolved to make the results reproducible.

The evaluation of harness evolution needs a clearer separation between what is optimized and what measures progress. Section 3.4 allows the update operator to modify validators, while updates are selected using the valid-bundle rate. It is unclear whether the criteria used to compute this rate remain fixed across rounds. If these criteria change, the increase from 42.3% to 91.5% could partly reflect changes in acceptance behavior. The final-round audit does not establish that construction quality was measured consistently throughout evolution.

Questions for authors

For the evolution experiment, which validator components can change, and which evaluation criteria remain fixed when computing the valid-bundle rate? Please specify the numerical teacher-solve-rate band used for acceptance.

3 comments

Thank you for your great efforts in reviewing this paper. Please let us know if you still have further concerns, or if you are not satisfied with the current responses, so that we can further update the response as soon as possible.

[W1] 11.17% is incompatible with 87 binary tasks evaluated over four runs.

[A]: Thank you for pointing out this discrepancy. SkillsBench task scores are not all binary: they range from 0 to 1 and can include fractional credit. Two actual evaluation cases illustrate the scoring rules:

TaskScoring criteriaRecorded score
setup-fuzzing-pySix equally weighted test groups: general setup/discovery and one suite for each of five libraries (arrow, ujson, black, ipython, minisgl). Library suites check the environment, testing notes, fuzzing artifacts, and driver execution. Only the arrow and ipython suites passed.2/6, stored as 0.33
crystallographic-wyckoff-position-analysisEleven CIF inputs, each requiring exact agreement with the expected Wyckoff multiplicities and coordinates. Four inputs passed.4/11, stored as 0.36

Consequently, avg@4 is not restricted to multiples of 100/348. We will correct Appendix A.8 and provide the original four-run task scores and aggregation code so that 11.17% can be checked directly.

[W2] Editable validators may change acceptance standards; the solve-rate band is unspecified.

[A]: Thank you for asking about evaluation consistency. To account for changes in construction-time validation prompts, we reassessed all six rounds with one frozen evaluator, fixing its code, prompts, model settings, thresholds, and repair-free budget. Round IDs and original verdicts were hidden; failed constructions remained in the denominator. Yields for Rounds 0–5 were 41.10/57.40/65.30/78.70/84.40/89.80%, supporting improvement under a common standard. This retrospective check does not imply that the original selection evaluator was frozen. We will report both curves separately; the solve-rate band is specified in tomE W11.

Thank you again for your review. We hope that our rebuttal could address your questions and concerns. As the discussion phase is nearing its end, we would be grateful to hear your feedback and wondered if you might still have any concerns we could address. It would be appreciated if you could raise your score on our paper if we address your concerns. We thank you again for your effort in reviewing our paper.

Thank you again for your review. We hope that our rebuttal could address your questions and concerns. As the discussion phase is nearing its end, we would be grateful to hear your feedback and wondered if you might still have any concerns we could address. It would be appreciated if you could raise your score on our paper if we address your concerns. We thank you again for your effort in reviewing our paper.

Reviewer_tomEOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces Skill2Task, a pipeline that turns community skills (SkillsMP, ClawHub) into executable, verifiable training tasks for general agents. Skills are organized into a "skill-state graph" (nodes = clustered pre/postcondition scenarios, edges = skills), and sampled paths become multi-skill tasks. Each task is built through three staged generate–validate–repair loops (environment → instruction → verifier). The main pitch is that the synthesis harness itself is revised each round by an LLM reading construction diagnostics, gated on a held-out valid-bundle rate (42.3% → 91.5% over five rounds). SFT on the resulting 10,880 trajectories (from 10,916 tasks) improves Qwen3.5-9B and 35B-A3B on four agent benchmarks, beating ClawGym and SWE-smith data re-rolled with the same teacher, with the largest gains on SkillsBench.

Critical strengths
  • [Section 4.2, A.6] Strong results, with a fairer comparison than is typical for data papers.
    • Baselines are re-rolled with the same GLM-5.2 teacher and the same success-only filter. I really appreciate this control.
    • Improves both backbones on all four benchmarks with fewer trajectories and tokens (0.76x ClawGym, 0.50x SWE-smith).
    • SkillsBench / TB2.0 gains are several times the run-to-run deviation (Table A.5).
  • [Section 3.3, A.14, A.15] The staged validation + verifier audit is the most valuable part, in my opinion.
    • Verifiers are probed against correct, unchanged, incorrect, and shortcut states. Verifier quality is arguably the weakest link in synthetic agent tasks.
    • Blind audit by two non-author PhD students: 97% reliable, κ = 0.87.
    • Table A.9 (88% of verifiers need repair vs. 8% of instructions) is a neat, honest finding on its own.
    • Gating harness evolution on construction quality (not the benchmarks) avoids the obvious overfitting route.
Critical weaknesses
  • [Section 4.3, Figure 3] The headline contribution (self-evolving harness) has the weakest downstream evidence.
    • The only downstream evidence is Fig. 3b: 46.40 → 49.14, plateauing at Round 2, at an unstated training budget.
    • Table 1 compares against external datasets, so it doesn't isolate evolution from the graph or the staged validation.
    • Adversarially, Round 0 is a "weak static harness". The accepted edits (Table A.10) read like fixes an engineer would make after looking at a few dozen failures (don't reinstall python, pin seeds, weights sum to 1). The real question is whether it beats a reasonably engineered static harness, and at what cost.
    • Novelty leans on this piece: the graph is close to SkillSynth (Fan et al., 2026), and the loop resembles Meta-Harness / Self-Harness applied to synthesis.
    • The meaning of 42.3% is inconsistent: "first-attempt" in the intro, "after repair" in the Fig. 3 caption.
  • [Section 4.2, Table 1] The baseline gap may partly reflect harness/format alignment rather than data quality.
    • Skill2Task trajectories use the OpenClaw prompt, and PinchBench + SkillsBench are also evaluated in OpenClaw.
    • The baselines don't just fail to help, they collapse: ClawGym drops SWE-bench Multilingual 55.33 → 33.33, and SWE-smith nearly halves SkillsBench. "Lacks multi-skill composition" explains no gain, not a collapse.
    • If the rollout agent sees SKILL.md files, the model learns exactly what SkillsBench's with-skill setting rewards.
    • There's one SFT run per setting, and std is over eval seeds only, so PinchBench and some Table 2 differences are within noise.

[Overall] I enjoyed this paper; the validation pipeline and teacher-consistent comparison are useful contributions. My hesitations are that the component the paper puts front and center is the least supported, and that the baseline gap may be partly about format. Showing the evolution gain at full scale against a reasonable static harness, and clarifying how baselines were rolled out, would make me more confident.

Questions for authors

Clarifications:

  1. [A.6] Which harness and tool interface were used to re-roll ClawGym / SWE-smith? Was the SFT recipe tuned on Skill2Task data? What explains the 22-point SWE-bench Multilingual drop from ClawGym data?
  2. [Section 3.5] Does the rollout agent see the skill files for the sampled path? If so, how much of the SkillsBench gain comes from that?
  3. [Section 4.3] Is 42.3% → 91.5% before or after repair? What are first-attempt rates per stage per round, and which harness is Table A.9 from? What's the Fig. 3b budget, how many proposals were rejected, and what's the cost per valid task at Round 0 vs. 5?
  4. [Section 3.5] Only ~25% of rollouts pass. How many distinct tasks have ≥1 passing trajectory? Where do the "reference correct states" used to validate verifiers come from? If they come from the same generator LLM, could the verifier and reference share a misreading?
  5. [Section 5.3] Was the DeepSeek step labeler checked against human labels?

Suggestions:

  1. Round 0 vs. Round 5 at full scale, and/or a human-engineered static harness baseline.
  2. GLM-5.2's own benchmark scores as an upper reference.

Minor:

  • [Abstract] "Accuracy more than doubles" holds for 9B (11.17 → 22.45), but 35B is ~1.4x (19.83 → 28.39).
  • [Section 4.2] "Outperforms on every benchmark": 9B PinchBench is 80.89 vs. 80.34 with std ~3.
  • [Figure 2] "Meta-Harness" label vs. "update operator U" in the text. The label is also the name of Lee et al. (2026).
  • [Algorithm 1] Missing the acceptance gate, and deploys every round although the pool is from Round 5 only.
  • [Section 3.4] What's the "fixed band" for the solve-rate guardrail?
  • [Eq. 1] r, v metadata are never used later?
  • [Section 4.4] What's the valid-bundle yield for randomly combined skills?
  • [A.13] Please also report the path-length distribution of the released pool.
  • [A.7] "Filter mentioned in Section 4": I think this is Section 3.2.

Suggested Citations:

4 comments

Thank you for your great efforts in reviewing this paper. Please let us know if you still have further concerns, or if you are not satisfied with the current responses, so that we can further update the response as soon as possible.

[W1] Does evolution outperform a reasonably engineered static harness at full scale?

[A]: Thank you for suggesting this comparison. Round 0 was already manually engineered with three-stage generation–validation–repair loops, but diverse synthesis runs expose recurring failures that are difficult to anticipate during design. Self-evolution automates their diagnosis and correction, reducing repeated manual refinement. Section 4.3 shows repair iterations falling 7.42 → 3.91 and the benchmark mean improving 46.40 → 49.14 at matched task counts, without benchmark-guided updates. This supports synthesis efficiency and downstream utility; a full-scale comparison against further human optimization remains untested.

[W2] What training budget does Figure 3b use, and what does its gain establish?

[A]: Thank you for requesting the budget. Each round uses 2,000 validated bundles, with SFT capped at 1,800 trajectories and 46M supervised tokens per epoch; candidate paths, teacher, architecture, and curation are fixed. The 46.40 → 49.14 (+2.74) gain therefore extends beyond increased task count. We will clarify that the trend is non-monotonic around Round 2 and improves subsequently.

[W3] How does the contribution differ from SkillSynth and runtime-harness optimization?

[A]: Thank you for asking about novelty. Building on SkillSynth's scenario-mediated composition, we combine separate environment, instruction, and verifier validation with feedback-driven refinement of task generation. Where runtime-harness optimization improves task execution, our updates improve the producer of training tasks and are selected by construction quality, without downstream benchmark feedback. We will emphasize this integrated contribution and discuss SWE-Gym, R2E-Gym, and BugPilot.

[W4] Were baseline rollouts and SFT settings comparable?

[A]: Thank you for asking about these controls. All sources use GLM-5.2/OpenClaw 2026.5.7, shared tool interfaces and serialization, four attempts per task, and their original environments/checkers. SFT settings—three epochs, batch 64, peak LR 2 × 10⁻⁵—were selected on a separate development workload. Only Skill2Task uses step masking; removing it gives 81.16/26.55/56.92/33.99 on PinchBench/SkillsBench/SWE-bench Multilingual/TB2.0, respectively, helping separate curation effects.

[W5] Why does ClawGym training reduce SWE-bench Multilingual from 55.33 to 33.33?

[A]: Thank you for highlighting this drop. Tool-call parseability is 99.2% versus 99.4% for the backbone. One SFT epoch instead of three improves the score to 49.25, still below 55.33. This suggests training-duration sensitivity but does not fully explain the decline or establish equally favorable baseline settings. We will remove the unsupported attribution to missing composition.

[W6] Does access to SKILL.md files explain the SkillsBench gain?

[A]: Thank you for asking about skill access. The solver can read sampled SKILL.md files via a read-only directory and index; references and verifiers are hidden. Matched recollection/SFT with versus without access scores 28.39 versus 27.01, both above 19.83, under identical with-skill evaluation. Access contributes without explaining the entire gain. Gains under mini-swe-agent and Terminus 2 also support transfer beyond OpenClaw.

[W7] Do small margins and evaluation-seed deviations establish training stability?

[A]: Thank you for highlighting this distinction. Original deviations cover evaluation seeds only; the 0.55 PinchBench margin does not establish significance. With one SFT run per setting, training-seed variability remains unmeasured. We will describe small margins as numerical leads rather than statistically established advantages.

[W8] What are the GLM-5.2 teacher's benchmark scores?

[A]: Thank you for suggesting a teacher reference. Under matched evaluation, GLM-5.2 scores 87.80 on PinchBench, 53.19 on SkillsBench, 80.00 on SWE-bench Multilingual, and 67.42 on TB2.0 Verified, averaging 72.10. These provide an empirical reference, not a formal upper bound.

[W9] Is 42.3% → 91.5% measured before or after repair? What are the stage-level rates?

[A]: Thank you for identifying this ambiguity. 42.3% → 91.5% compares the initial and evolved harnesses (Round 0 vs. Round 5), with both rates measured after task-level repair. We will correct the first-attempt wording in the introduction and Section 5.2. In the table, round denotes harness evolution. All rates use 200 held-out paths as the denominator; stage-level first-pass rates count paths passing that stage before local repair, with paths blocked upstream counted as non-passes. Under this binary, equally weighted protocol, rates change in 0.5-percentage-point increments; the original 42.3% must be reconciled with the per-path records.

Harness roundEnvironment first pass (%)Instruction first pass (%)Verifier first pass (%)Post-repair bundles (%)
052.5080.5028.5042.50
159.0088.5039.5059.00
264.0089.0046.0067.00
368.0089.5056.5080.50
471.0089.0061.5086.50
572.5089.5066.0091.50

[W10] Which harness and sample does Table A.9 describe?

[A]: Thank you for requesting the provenance. Table A.9 covers 1,000 successful Round 0 constructions: 542 environments, 81 instructions, and 880 verifiers required repair. Conditioning on eventual success means these are not complements of W9's first-pass rates.

[W11] How many proposals were rejected, and what was the solve-rate guardrail?

[A]: Thank you for asking about update selection. Of eight proposals, five were accepted and three rejected: two for lower yield and one for violating the preset 0.15–0.35 trajectory-level teacher success band. The band is measured using four rollouts on each of 100 accepted selection bundles, with a fixed target path-length distribution.

[W12] What is the cost per valid task before and after evolution?

[A]: Thank you for requesting the cost comparison. The paper reports a reduction in average repair iterations per successfully constructed task from 7.42 to 3.91, indicating less repair work. It does not establish a monetary cost per valid task; fewer repair iterations alone cannot quantify dollar savings because calls and execution steps can have different costs.

[W13] How many distinct tasks contribute successful trajectories?

[A]: Thank you for asking about coverage. 10,880 retained trajectories represent 10,880 distinct tasks, covering 99.67% of 10,916 bundles, with one successful trajectory retained per task. We will clarify that 10,880/43,664 measures retention, not raw rollout success.

[W14] Where do reference correct states come from, and could they share verifier errors?

[A]: Thank you for raising this concern. References come from a separate GLM-5.2 solving call that sees instructions/assets but not the verifier. Shared-model misreadings remain possible; an independent human check found 96/100 references consistent with their tasks, separately from the 97% verifier-behavior audit.

[W15] Was the DeepSeek step labeler checked against human judgments?

[A]: Thank you for requesting this check. Humans reviewed 200 randomly sampled DeepSeek-flagged erroneous steps with context, confirming 191 (95.50% precision). This measures the precision of masking decisions; the downstream ablation assesses their training benefit.

[W16] What is the valid-bundle yield for randomly combined skills?

[A]: Thank you for requesting this construction metric. Under matched construction attempts, random combinations yield 340/500 valid bundles (68.00%), compared with 451/500 (90.20%) for graph sampling, a 22.20-point difference.

[W17] What is the path-length distribution of the released pool?

[A]: Thank you for requesting the released-pool statistics. Counts for sampled path lengths 1–7 are 153/2,148/766/1,146/1,612/1,757/3,334, totaling 10,916. We will add this distribution to Appendix A.13 and distinguish sampled length from dependencies retained after repair.

[W18] Which wording, algorithm, notation, and cross-reference corrections will be made?

[A]: Thank you for identifying these issues. (1) “More than doubles” will refer specifically to 9B (11.17 → 22.45); 35B improves 19.83 → 28.39. (2) Figure 2 will use “Harness update operator U,” reserving “Meta-Harness” for the cited work. (3) Algorithm 1 will show acceptance/rejection and distinguish evolution diagnostics from final-pool collection using Round 5. (4) We will remove unused r/v metadata and correct Appendix A.7's filtering reference to Section 3.2.

Thank you again for your review. We hope that our rebuttal could address your questions and concerns. As the discussion phase is nearing its end, we would be grateful to hear your feedback and wondered if you might still have any concerns we could address. It would be appreciated if you could raise your score on our paper if we address your concerns. We thank you again for your effort in reviewing our paper.

Thank you again for your review. We hope that our rebuttal could address your questions and concerns. As the discussion phase is nearing its end, we would be grateful to hear your feedback and wondered if you might still have any concerns we could address. It would be appreciated if you could raise your score on our paper if we address your concerns. We thank you again for your effort in reviewing our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.