- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes DataFoundry, which improves synthetic training-data preparation by iteratively revising the preparation pipeline before full-scale generation. A coding-agent controller coordinates runtime compilation, pilot-set assessment, and component-specific adapter design. Experiments evaluate downstream fine-tuning performance across mathematics, finance, law, and medicine, primarily using Qwen2.5-7B, with additional Llama-3.1-8B results. The framework addresses a relevant problem and reports promising downstream improvements. However, the experiments do not sufficiently isolate the benefits of the proposed mechanism, and its methodological contribution remains under-specified.
Critical strengths
-
The evaluation measures downstream training utility across multiple domains and model families (Qwen2.5 / Llama3.1).
-
The framework connects recurring data defects to actionable changes in the generation pipeline.
Critical weaknesses
-
The experimental comparisons do not isolate data quality from data quantity and computational budget. Appendix E reports substantially different dataset sizes across methods. For example, DataFoundry generates 32,692 mathematics samples, compared with 3,501 for DataPrep-Skill using the same coding-agent backbone. Training for three epochs over these different datasets does not equalize optimization steps, training tokens, or the proportion of domain data relative to Dolly-15k. Although DataFoundry uses fewer samples in some other domains, matched-size and matched-token experiments are still necessary to identify the source of its gains. The paper also lacks an end-to-end accounting of compilation, pilot generation, assessment, and evolution costs, as well as repeated-run uncertainty estimates. Consequently, the reported improvements do not establish that recursive preparator evolution is superior under comparable resource budgets.
-
The added value of the proposed evolution mechanism is insufficiently specified and experimentally demonstrated. Compilation, assessment, and deficiency tracing are described primarily as coding-agent workflows, without enough detail on assessment calibration, revision acceptance, or safeguards against harmful changes. The comparisons do not isolate contextual criterion selection, component-level attribution, or adapter-based updates from simpler iterative prompt refinement or unrestricted runtime revision under the same budget. Moreover, performance drops substantially after the third round, leaving the claimed protection against uncontrolled drift unvalidated. The main text contains relatively little substantive methodological detail, and Figure 1 largely repeats the motivation while adding limited technical information. These gaps make it difficult to identify and assess the contribution beyond assembling a modular agent-driven feedback loop.
Questions for authors
-
Do the improvements persist when generated datasets are matched by sample count or training tokens, with equal optimization budgets and repeated-run confidence intervals?
-
How does DataFoundry compare with equally budgeted iterative prompt refinement, unrestricted runtime revision, and construction followed by filtering?
-
Which components account for the gains? Can the authors ablate contextual criterion selection, deficiency-to-component tracing, and adapter-based preservation of the baseline specification?
-
How are assessment scores calibrated, revisions accepted or rejected, and evolution stopped without consulting downstream test results? Can the authors report independent quality measurements across rounds and the complete preparation cost?