acceptodds
Under review as a conference paper at ICLR 2027

DataFoundry: Evolving Data Preparators via Recursive Self-Improvement

Abstract

Domain adaptation of large language models increasingly depends on constructing high-quality training data, yet existing data-preparation pipelines typically address quality only after generation through post-hoc filtering. This creates a fundamental mismatch: data-quality issues often originate from the construction process itself, while quality control is applied only to its outputs. We introduce DataFoundry, a framework for evolving data preparators through recursive self-improvement before large-scale data production. DataFoundry represents a data preparator as an evolvable runtime specification and instantiates its evolution with a Skills-as-Modules architecture, in which a central Controller orchestrates modular skills to compile executable runtimes, diagnose deficiencies on small pilot sets using domain-appropriate criteria, and translate diagnostic feedback into adapters that revise individual preparation components while preserving stable interfaces. We evaluate DataFoundry on DataPrep-Bench across mathematics, finance, law, and medicine, and find that recursively evolved preparators produce training data with higher downstream utility than baselines. Experiments across different backbones further demonstrate that these improvements are not tied to a particular model, while analyses and case studies further reveal the framework's optimization dynamics and illustrate how its evolution unfolds in practice.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
data preparation, recursive self-improvement, agent skills

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KZNMOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes DataFoundry, which improves synthetic training-data preparation by iteratively revising the preparation pipeline before full-scale generation. A coding-agent controller coordinates runtime compilation, pilot-set assessment, and component-specific adapter design. Experiments evaluate downstream fine-tuning performance across mathematics, finance, law, and medicine, primarily using Qwen2.5-7B, with additional Llama-3.1-8B results. The framework addresses a relevant problem and reports promising downstream improvements. However, the experiments do not sufficiently isolate the benefits of the proposed mechanism, and its methodological contribution remains under-specified.

Critical strengths
  • The evaluation measures downstream training utility across multiple domains and model families (Qwen2.5 / Llama3.1).

  • The framework connects recurring data defects to actionable changes in the generation pipeline.

Critical weaknesses
  • The experimental comparisons do not isolate data quality from data quantity and computational budget. Appendix E reports substantially different dataset sizes across methods. For example, DataFoundry generates 32,692 mathematics samples, compared with 3,501 for DataPrep-Skill using the same coding-agent backbone. Training for three epochs over these different datasets does not equalize optimization steps, training tokens, or the proportion of domain data relative to Dolly-15k. Although DataFoundry uses fewer samples in some other domains, matched-size and matched-token experiments are still necessary to identify the source of its gains. The paper also lacks an end-to-end accounting of compilation, pilot generation, assessment, and evolution costs, as well as repeated-run uncertainty estimates. Consequently, the reported improvements do not establish that recursive preparator evolution is superior under comparable resource budgets.

  • The added value of the proposed evolution mechanism is insufficiently specified and experimentally demonstrated. Compilation, assessment, and deficiency tracing are described primarily as coding-agent workflows, without enough detail on assessment calibration, revision acceptance, or safeguards against harmful changes. The comparisons do not isolate contextual criterion selection, component-level attribution, or adapter-based updates from simpler iterative prompt refinement or unrestricted runtime revision under the same budget. Moreover, performance drops substantially after the third round, leaving the claimed protection against uncontrolled drift unvalidated. The main text contains relatively little substantive methodological detail, and Figure 1 largely repeats the motivation while adding limited technical information. These gaps make it difficult to identify and assess the contribution beyond assembling a modular agent-driven feedback loop.

Questions for authors
  • Do the improvements persist when generated datasets are matched by sample count or training tokens, with equal optimization budgets and repeated-run confidence intervals?

  • How does DataFoundry compare with equally budgeted iterative prompt refinement, unrestricted runtime revision, and construction followed by filtering?

  • Which components account for the gains? Can the authors ablate contextual criterion selection, deficiency-to-component tracing, and adapter-based preservation of the baseline specification?

  • How are assessment scores calibrated, revisions accepted or rejected, and evolution stopped without consulting downstream test results? Can the authors report independent quality measurements across rounds and the complete preparation cost?

Reviewer_hDwvOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces DataFoundry, a framework for preparing domain-specific SFT data. Instead of the common "construct-then-filter" paradigm, DataFoundry represents a data-preparation pipeline as an evolvable runtime specification and improves it through recursive self-improvement before large-scale production. A central controller, implemented as a coding agent (GPT-5.6 Sol via Codex CLI), orchestrates three modular skills: a compiler that turns the specification into executable code, an assessor that scores a small pilot set against domain-selected criteria, and an adapter designer that translates diagnosed deficiencies into component-level revisions while keeping interfaces and the baseline specification frozen. After two evolution rounds, the evolved runtime produces the full training set. Experiments on DataPrep-Bench across Math, Finance, Law, and Medical, with Qwen2.5-7B and Llama-3.1-8B as downstream models, show that the resulting data yields higher domain-average benchmark scores than DataFlow-based, LLM-based, agent-based, and skill-based generators.

Critical strengths

1. A clearly articulated and practically relevant problem framing. The paper identifies a real mismatch in current data-preparation practice: quality issues originate in the construction process, while quality control is applied only post-hoc to outputs, with pipelines routinely discarding 50–90% of generated data. Shifting intervention upstream is a meaningful direction, and the pilot-set/evolution-loop instantiation is a concrete, reproducible way to operationalize it. This framing is consequential to my assessment because it is what makes the paper worth reviewing seriously despite my concerns below.

2. Broad empirical coverage with controlled same-backbone comparisons. The evaluation spans four domains, two downstream base models, and a wide range of generator baselines, including agent-based and skill-based generators using the identical GPT-5.6 Sol backbone. The gains over these same-backbone baselines (3.4–14.4 points on domain averages) rule out the simplest explanation that the method is merely "a stronger model generating data," and the component-level case studies (Figures 4–5) give a legible account of what the evolution loop changes.

Critical weaknesses

1. The central motivating claim is never tested, and data volume is uncontrolled across comparisons. The paper's core premise (Figure 1) is that post-hoc filtering is fundamentally mismatched with where quality issues arise, and that upstream evolution of the pipeline is the remedy. However, no experiment compares the evolution loop against the natural alternative at matched compute: spending the same budget on a larger candidate pool plus stronger filtering. This confound is not hypothetical — per Table 7, synthesized dataset sizes vary by two orders of magnitude across methods (from 4.8K for the GPT-5.3-codex agent to 509K for DataFlow; DataFoundry yields 88K), so volume/quality trade-offs contaminate every headline comparison, in both directions. Given that the paper's stated contribution is conceptual ("where to intervene"), the absence of a budget-matched filtering baseline leaves the main claim empirically unsupported.

2. The framework's contribution over naive iterative refinement is not isolated, and the optimization target is an unvalidated proxy. The same-backbone baselines only show that some feedback loop beats none: per Appendix D, the agent-based and skill-based generators receive no evaluation feedback and simply iterate until self-judged completion. The decisive baseline — an agent given the same pilot-set assessment reports and allowed to freely modify the pipeline, without the skills-as-modules decomposition and adapter mechanism — is missing. As a result, the measured gains may reflect (a) iterative refinement per se, and (b) substantially more inference compute (compilation, pilot generation, per-sample assessment, rewriting), neither of which is novel. Relatedly, the loop is driven entirely by LLM-judged proxy criteria whose correlation with downstream utility is never measured; the observed Round-3 degradation (e.g., Finance 66.1→58.3) indicates the process is not a stable self-improvement loop, yet the stopping point is chosen post hoc and the failure mode receives no analysis. These are consequential because they bear directly on whether the paper teaches us anything beyond "coding agents benefit from feedback."

Questions for authors
  1. Under a matched total compute budget (e.g., equal wall-clock/API cost), how does DataFoundry compare to the base runtime scaled to a larger candidate pool with post-hoc filtering applied? This directly tests the claim motivating Figure 1.
  2. If a coding agent receives the same assessment reports but modifies the pipeline directly (no skills decomposition, no adapter, no frozen baseline specification), does performance differ materially? This isolates the contribution of the framework's structure from iterative refinement itself.
  3. What is the measured correlation between the assessor's per-sample criterion scores and actual downstream utility (e.g., via a small-scale leave-out training study)? Why does Round 3 degrade, and is there a principled, feedback-driven stopping criterion rather than a post-hoc choice of two rounds?
Reviewer_NZukOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem of preparing high-quality domain-specific post-training data for LLMs. Unlike conventional data preparation pipelines that primarily rely on post-hoc filtering, the authors propose DATAFOUNDRY, a framework that recursively improves the data preparation process itself before large-scale data generation.

DATAFOUNDRY formulates data preparation as an evolvable runtime and introduces a Skills-as-Modules architecture to implement recursive self-improvement (RSI). The framework consists of three main components: 1) runtime compilation, 2) contextualized data quality assessment, and 3) adapter-based runtime evolution. Through iterative evaluation on small pilot datasets, the framework identifies deficiencies and updates the data preparation pipeline before generating the final training data.

The authors evaluate DATAFOUNDRY on DataPrep-Bench across four domains: mathematics, finance, law, and medicine. The experimental results show improvements in downstream fine-tuning performance over several baseline data generation methods, with additional analyses of RSI iterations and generalization across different downstream models.

Critical strengths
  1. A meaningful and practically relevant research problem. The paper addresses an important challenge in domain-specific LLM post-training: how to systematically prepare high-quality training data. Moving beyond post-hoc data filtering to improve the data preparation pipeline itself is a reasonable and potentially useful direction, particularly when scaling data generation across different specialized domains.

  2. Promising empirical results across multiple domains. The experiments show that DATAFOUNDRY can improve the downstream utility of generated training data across mathematics, finance, law, and medicine. The evaluation also considers different downstream base models and multiple RSI iterations, providing useful initial evidence for the effectiveness and transferability of the proposed approach.

Critical weaknesses
  1. Limited methodological novelty and insufficient clarity of the proposed framework.

While the overall approach is reasonable, the core methodology appears to be a relatively straightforward application of recursive self-improvement to data preparation. The Skills-as-Modules architecture largely follows a standard iterative process of compiling a runtime, evaluating generated data, and updating the runtime based on feedback. I find it difficult to identify the key technical contribution that distinguishes DATAFOUNDRY from existing agent-based data preparation and self-improving workflow frameworks.

In addition, the method section is not particularly easy to follow. The description relies heavily on abstract function definitions and module names, without sufficiently explaining the concrete operations and design choices that make the proposed framework effective. A clearer algorithmic description and a more explicit discussion of what is technically new would strengthen the paper.

  1. Insufficient clarity and control in the experimental comparisons.

Although the main results show promising improvements, I have concerns about how the baseline comparisons are conducted and interpreted. The paper includes a large number of baseline generators with different underlying models and generation mechanisms. While some implementation and training details are provided in the appendix, the experimental setup does not clearly establish whether these methods are compared under sufficiently controlled data generation and computational budgets.

In particular, Appendix E shows substantial differences in the number of generated training samples across methods. This makes it difficult to determine whether the observed performance differences primarily reflect improvements in data preparation quality, differences in training data quantity, or other factors such as the generation backbone and computational resources.

I would therefore like to see more controlled comparisons, particularly under matched data budgets and clearer baseline configurations. Without these controls, it is difficult to fully assess the effectiveness and generality of the proposed framework.

Questions for authors
  1. What is the key methodological novelty of DATAFOUNDRY beyond existing RSI and agent-based data preparation frameworks? The proposed Compile–Construct–Assess–Evolve loop appears conceptually similar to general self-improving agent workflows. Could the authors clarify which components represent novel algorithmic contributions, rather than an application of existing techniques to data preparation?

  2. Could the authors provide a clearer and more concrete description of the Skills-as-Modules implementation? In particular, how does the Evolution-Adapter-Designer identify which runtime component should be modified, and how are the proposed modifications generated and validated? A step-by-step example with concrete inputs, intermediate outputs, and runtime changes would make the approach easier to understand and reproduce.

  3. How are the baseline methods compared in terms of data generation budgets and computational resources? Appendix E reports substantial differences in the number of generated training samples. Could the authors provide additional experiments with matched training data sizes and, ideally, comparable generation budgets to isolate the contribution of DATAFOUNDRY's data preparation strategy?

  4. How much of the improvement comes specifically from recursive runtime evolution? It would be helpful to more explicitly distinguish the contributions of the initial Skills-as-Modules architecture, contextualized assessment, and iterative adapter updates. Could the authors provide further controlled ablations to clarify the importance of these individual components?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.