- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes EVOSYN, a feedback-driven framework for agentic data synthesis. It iteratively generates verified teacher trajectories, fine-tunes a student model using SFT, and adjusts subsequent data generation based on the student's weaknesses. The method is evaluated on τ²-Bench and CAR-Bench, showing improvements in task success rates.
Critical strengths
-
The paper is clearly written and well organized.
-
The experiments are extensive, including multiple benchmarks, model scales, ablation studies, and analyses of synthetic data quality.
Critical weaknesses
-
The novelty is unclear. The method seems to be iterative teacher-student distillation with adaptive data generation. What is the key novelty beyond existing adaptive data synthesis methods? Also, since training tasks are generated in the same environments as the benchmarks, they may be highly similar to test tasks, raising concerns about whether the reported performance reflects real generalization.
-
Repeated SFT on synthetic data may hurt the model's generalization ability. However, the paper does not evaluate performance on unrelated tasks or benchmarks.
-
The experiments do not clearly separate the effects of the synthesis strategy, data size, and training budget. Table 2 matches the candidate generation budget, but EVOSYN retains more examples for SFT. The size-matched comparison only measures trajectory coverage, not downstream task success. A comparison with matched training data sizes and training budgets is needed to better isolate the benefit of the synthesis strategy.
-
The need to add task factors dynamically is not directly tested. Although the paper includes an ablation without the Reflector, a more direct baseline would provide all final factors, such as num_write and num_read, from the first round. It would then adjust only the sampling proportions based on student feedback. This would help show whether the gains come from dynamic factor discovery or simply from having a more complete set of task attributes.
-
It is unclear how new factors are discovered. Appendix D shows that the Reflector’s input can already include suggested factor names and values, and the prompt asks it to prioritize these suggestions. The authors should explain how the diagnostic probes and suggestions are produced and whether they rely on predefined candidates. Otherwise, it is hard to tell whether the system discovers new task attributes or simply turns attributes already supplied in its input into generation controls.
Questions for authors
See weaknesses.
7 comments
3. Generalization: unseen tasks within executable environments.
We appreciate the concern regarding training–test similarity. Importantly, sharing an environment does not imply sharing task instances or test supervision. Our setting evaluates in-environment generalization: the agent learns from synthesized interactions and is evaluated on held-out tasks under the same tool interfaces and domain policies.
As specified in Appendix B.1:
- Synthesis feedback is obtained from a held-out diagnostic set disjoint from the official test set.
- Official benchmark test tasks are reserved for final evaluation; test labels are never used for synthesis or training.
- Improvements are consistently observed across four distinct evaluation domains (Airline, Retail, Telecom, and CAR-Bench), rather than a single task distribution.
Beyond task success, Appendix B.9 provides complementary evidence of structural generalization: under matched sample counts, EvoSyn covers 1.32× more distinct executable interaction sequences on CAR-Bench and 2.19× more on Airline than Static Synthesis. These findings indicate that the method expands interaction coverage rather than merely accumulating repetitive trajectories.
Reply to W2 -- General Capability Retention Under Repeated Synthetic SFT
We thank the reviewer for raising this important concern. To examine whether repeated synthetic SFT compromises general-purpose capabilities, we conducted additional evaluations on four independent benchmarks covering instruction following, mathematical reasoning, and general knowledge. We compare the original Qwen3-8B, an R-Zero-style synthesis baseline, EvoSyn-R1, and EvoSyn-Final under identical evaluation protocols, without additional training or benchmark-specific tuning.
1. Generalization results on unseen benchmarks.
All models are trained only on CAR-Bench synthetic data, while the following benchmarks are entirely outside the training domain.
| Benchmark | Qwen3-8B Base | R-Zero Style | EvoSyn-R1 | EvoSyn-Final |
|---|---|---|---|---|
| IFEval (Instruction Following) | 80.0 | 75.4 | 80.6 | 80.6 |
| GSM8K (Mathematical Reasoning) | 87.6 | 91.4 | 91.1 | 91.1 |
| BBH (Complex Reasoning) | 76.9 | 79.4 | 76.5 | 80.1 |
| MMLU-Pro (General Knowledge) | 69.1 | 69.2 | 66.3 | 67.9 |
2. Repeated SFT preserves general capabilities without measurable cumulative degradation.
The results provide three key observations:
- General capability retention: Compared with the original Qwen3-8B, EvoSyn-Final maintains comparable IFEval performance (80.6 vs. 80.0) and MMLU-Pro performance (67.9 vs. 69.1), while significantly improving GSM8K by +3.5 pp (95% CI [+1.6, +5.4]) and BBH by +3.2 pp ([+0.7, +5.9]).
- No measurable additional degradation across rounds: From EvoSyn-R1 to EvoSyn-Final, performance remains unchanged on IFEval (80.6 → 80.6) and GSM8K (91.1 → 91.1), improves on MMLU-Pro (66.3 → 67.9), and significantly improves on BBH (76.5 → 80.1, +3.6 pp). None of these four benchmarks exhibits a statistically significant decline from R1 to Final.
- Comparison with learner-feedback synthesis: EvoSyn-Final significantly outperforms the R-Zero-style baseline on IFEval (+5.2 pp, 95% CI [+1.9, +8.3]) while achieving comparable GSM8K and MMLU-Pro performance.
These additional experiments show that EvoSyn's repeated synthesis-learning rounds do not introduce measurable cumulative degradation across the evaluated general-purpose capabilities. Instead, the final model preserves instruction-following and general-knowledge performance while improving mathematical and complex reasoning. These results provide complementary evidence that EvoSyn's specialized agentic improvements need not come at the expense of broader LLM capabilities. We will incorporate these evaluations and their protocols into the revised manuscript.
Reply to W3 -- Isolating the Benefit of Synthesis Strategy Beyond Data Size and Training Budget
To address this, we conducted a new Budget-Matched Static Synthesis experiment, explicitly controlling the final SFT data volume and aggregate training-token budget while keeping the Static synthesis strategy unchanged.
1. New controlled experiment: matched data and training budgets.
We expanded Static Synthesis using its original fixed generation procedure, without learner feedback or synthesis-space evolution. The resulting baseline matches EvoSyn's final SFT dataset size and processes 235.3M training tokens, slightly exceeding EvoSyn's 227.6M (+3.4%) across the complete six-epoch training runs.
| Method | SFT Examples | R1 / R2 / R3 Training Tokens | Total Tokens | Success Rate |
|---|---|---|---|---|
| Static (Original) | 639 | 37.5M / 20.0M / 49.3M | 106.8M | 61.3% |
| Static (Budget-Matched) | ~1,287 | 37.5M / 97.6M / 100.2M | 235.3M | 64.6% |
| EvoSyn | 1,287 | 37.5M / 91.1M / 99.0M | 227.6M | 72.6% |
Importantly, R1 is identical across methods, and Budget-Matched Static receives more training tokens than EvoSyn in both subsequent rounds.
2. The synthesis strategy retains a substantial independent advantage.
Increasing Static's data and compute improves its success rate by +3.3 pp (61.3% → 64.6%), confirming that training volume contributes to performance. However, EvoSyn still outperforms Budget-Matched Static by +8.0 pp (72.6% vs. 64.6%), retaining approximately 71% of its original performance advantage despite using slightly fewer training tokens.
The structural-quality results further explain this persistent gap:
| Metric | Static (Budget-Matched) | EvoSyn |
|---|---|---|
| Factor Coverage | 0.812 | 0.893 |
| Factor Balance | 0.447 | 0.508 |
| Interaction Coverage | 0.358 | 0.373 |
| Trajectory Diversity | 0.784 | 0.938 |
These results complement Figure 12, which already demonstrates 1.32× and 2.19× greater size-matched structural support on CAR-Bench and Airline, respectively. Crucially, our new experiment extends the evidence to actual downstream task success under comparable training budgets, rather than structural coverage alone.
Reply to W4 -- Does Dynamic Factor Discovery Matter Beyond a Complete Synthesis Space?
We thank the reviewer for this insightful suggestion. To directly distinguish dynamic factor discovery from simply having a complete set of task attributes, we implemented the proposed Oracle-Full-Space + Adaptive Allocation baseline on CAR-Bench.
1. New experiment: Dynamic expansion vs. oracle-defined synthesis space.
The Oracle baseline receives all factors eventually discovered by EvoSyn (including num_write and num_read) from R1, while retaining feedback-guided allocation through the Designer. Its synthesis space remains fixed, whereas EvoSyn progressively introduces new factors based on learner feedback.
| Method | R1 | R2 | R3 | R1→R3 Gain |
|---|---|---|---|---|
| Fixed-State Synthesis | 60.7% | 62.7% | 61.3% | +0.6 pp |
| Oracle-Full-Space + Adaptive Allocation | 64.0% | 67.3% | 69.3% | +5.3 pp |
| EvoSyn (Dynamic Expansion) | 60.7% | 65.5% | 72.6% | +11.9 pp |
2. Key finding: Dynamic expansion ultimately surpasses the oracle baseline.
Despite having privileged access to all final factors, Oracle's initial advantage progressively disappears:
- Early advantage: Oracle outperforms EvoSyn by +3.3 pp in R1, confirming the usefulness of additional factors.
- Stronger evolution: EvoSyn achieves +11.9 pp improvement from R1 to R3, compared with +5.3 pp for Oracle.
- Final advantage: EvoSyn reaches 72.6% vs. 69.3% (+3.3 pp), with a substantially larger R2→R3 improvement (+7.1 vs. +2.0 pp).
3. Why dynamic introduction matters.
These findings distinguish factor availability from learning-guided factor introduction. Rather than exposing all factors initially, EvoSyn progressively introduces num_write and num_read as the learner's weaknesses shift from write-intensive execution to read-intensive decision making (Figure 3). This is further supported by our feedback-conditioned discovery results (2/2 vs. 0/6, Appendix B.6) and factor controllability (99.6–100%, Appendix B.7).
The oracle-controlled comparison provides direct evidence that EvoSyn's benefit extends beyond a richer predefined synthesis space. Its stronger multi-round improvement and higher final performance support the value of introducing controllable factors in response to evolving learner weaknesses. We will incorporate this new comparison into the revised manuscript.
Response to Reviewer yNng
We sincerely thank you for the careful review and constructive feedback. Your insightful questions and suggestions have helped us further clarify the technical novelty of EvoSyn and better characterize its generalization and factor-discovery mechanisms. We hope the following detailed responses, additional analyses, and new experiments adequately address your main concerns and provide stronger evidence for the contributions of our work.
Reply to Weakness
Reply to W1 -- Clarifying the Technical Novelty and Generalization of EvoSyn
We sincerely thank you for raising these important questions. Our novelty lies not in iterative distillation itself, but in making the controllable synthesis representation explicitly revisable through learner feedback. We clarify this distinction and provide supporting evidence below.
1. Technical novelty: evolving the synthesis space, not merely the generated tasks.
Existing adaptive synthesis methods typically use feedback to generate harder tasks, explore new environments, or adjust task distributions. EvoSyn addresses a different question: what if the learner's remaining weaknesses cannot be adequately expressed by the current synthesis controls?
Formally, EvoSyn maintains a synthesis state Sr=(Or,r), where Or defines the controllable interaction factors and r specifies their generation allocation. Feedback drives two distinct decisions:
This distinction goes beyond conventional adaptive generation:
Importantly, our claim is not unrestricted ontology invention. EvoSyn identifies measurable, previously unrepresented interaction factors and promotes them into reusable synthesis controls based on learner feedback.
2. Mechanistic evidence: discovery, controllability, and learning utility.
Our experiments validate this distinction at multiple levels:
num_writeandnum_read(2/2), whereas the no-feedback control fails to recover either (0/6).num_writeandnum_readprovide 0.168 and 0.134 nats of additional learner-failure information after conditioning on the existing synthesis representation.These results establish a concrete distinction: EvoSyn does not simply use feedback to generate different examples; it converts learner weaknesses into new, verifiable synthesis capabilities.