- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
WEFT combines stateful tool-environment synthesis, compositional task generation, multiple interaction views and harnesses, execution-driven component repair, prefix-preserving demonstration sampling, atomic-turn GRPO, and shared MCP serving. Its central idea is sensible: a failed rollout can reflect an environment, task, or verifier defect, so treating every failure as a policy error can corrupt training.
Critical strengths
- Meaningful 14B improvements. Table 1 reports gains over Agent-World-14B of 6.41 points on BFCL V4, 2.23 on τ²-Bench, and 12.27 on Claw-Eval. The Claw-Eval improvement is particularly compelling.
- Useful evidence for interaction diversity. In Table 2, adding OpenClaw and Hermes increases Toolathlon-Verified from 32.41 to 45.99 and AutomationBench from 13.67 to 26.50. Evaluation through benchmark-native harnesses makes this relevant evidence of transfer.
Critical weaknesses
- Several ablations do not fully separate the proposed factor from data or compute allocation
For harness and view diversity, a fixed task set does not establish equal numbers of demonstrations, training tokens, or optimizer updates. The cumulative harness gains could partly reflect increased training exposure.
For rejection sampling, the same retry limit applied per atomic task versus per complete trajectory does not imply equal generation cost. Fixing accepted-trace count is helpful, but efficiency requires generated-token and tool-call accounting.
For RL, atomic-turn training changes both the advantage calculation and the rollout procedure: candidates share restored states, and successful candidates preferentially supply subsequent history. It is unclear whether trajectory-level GRPO uses the same continuation rule and rollout topology.
- The central supervision-reliability claim is not directly validated
The reduction from 1.76% to 0.96% tool-call errors measures errors in selected teacher trajectories, including recoverable errors. It does not directly measure incorrect failure attribution, verifier false positives, or verifier false negatives.
The downstream improvements are valuable evidence that evolution helps. They do not distinguish among several explanations: repaired implementations, better demonstrations, changed task difficulty, or changed verifier permissiveness. Fixing task identities does not by itself establish that intended task semantics remain invariant across revisions.
The correlation filter has a related limitation: agreement between an LLM judge and an executable verifier does not establish that either is correct, particularly when they may share assumptions.
Questions for authors
- What independently verified evidence shows that self-evolution reduces erroneous supervision while preserving the intended task objectives? In particular, does attribution accuracy improve, and do verifier false-positive and false-negative rates decrease?
Rebuttal
Thank you for your careful review and for recognizing WEFT’s core motivation, the meaningful 14B gains, particularly on Claw-Eval, and the transfer evidence from benchmark-native evaluation.
Response to W1: Training volume and diversity gains, sampling costs, and RL rollout differences.
The task set and SFT settings stay fixed; added task formats and harnesses increase demonstrations and training updates (Table R1). We address the scarcity of executable, verifiable tasks. From the same 11,884 tasks, each added format or harness yields 7,486–8,813 accepted trajectories. Mixing Agentic and SimUser improves WEFT-35B-A3B by a mean 3.18 percentage points over Agentic alone; OpenClaw and Hermes add 9.71 more across Toolathlon-Verified, AutomationBench, and Claw-Eval under native harnesses (Table 2).
Prefix-preserving sampling uses 28% fewer generated tokens and 30% fewer tool calls per task (Table R2). We measured both costs per task as suggested. Both methods allow k = 3 additional retries: per full trajectory for end-to-end sampling, or per atomic-task turn for prefix-preserving sampling. The downstream comparison in Figure 5 fixes accepted-trace counts.
The GRPO variants use different rollouts but the same group size, training steps, and number of turn segments. Trajectory-level GRPO uses G independent full rollouts and trajectory-level advantages. Atomic-turn GRPO shares restored history and state, uses within-turn advantages, and continues from a uniformly selected passing candidate (a failed one if none pass). All candidates train the model; both yield G × T segments for T turns (Table 3).
Response to W2: Attribution accuracy, verifier errors, task-objective preservation, and filter reliability.
WEFT identifies the correct failure cause in 92.0% of audited cases (Table R3). Auditors checked WEFT's assignments to policy, backend, task, initial state, or verifier using 100 cases per category across three rounds, blind to downstream scores. This directly audits attribution, not tool-call errors.
The revised verifier better distinguishes valid from invalid executions in the independent audit (Table R4). We compare rounds 0 and 3 on 200 trajectories labeled against original objectives, with verifier outputs hidden (percentages use all 200). Both false acceptances and false rejections decrease.
We repair faulty components without relaxing the original task requirements. After each repair, the checker evaluates fresh rollouts against the original objectives; 680 of 6,771 proposals were rejected for relaxing requirements. Appendix F.1–F.5 gives concrete cases: payment-ID binding is corrected while amount and merchant constraints remain unchanged.
Filter reliability. Thank you for this important question. Verifiers are hard to validate, so we used official AutomationBench verifiers as an independent reference. We rebuilt verifiers for 600 tasks from descriptions and environments alone, then filtered them. On the same rollouts, retained verifiers were more reliable: 87.5% agreement vs. 47.3% for removed ones (Table R5).
Response to Q1: Independent audits of attribution and verifier errors, and checks on task objectives.
Please see W2: human auditing finds 92.0% correct failure attribution across three rounds (Table R3); the revised verifier makes fewer false acceptances and rejections against original-objective labels (Table R4). Appendix F.1–F.5 shows repairs that preserve the original requirements.
Appendices B.1, C.1, F, and G provide training settings, GRPO rollout details, cases, and reliability audits, respectively. Thank you again for your constructive feedback. We hope this evidence addresses your concerns and would appreciate your reconsideration of the score.