acceptodds
Under review as a conference paper at ICLR 2027

Agentic Data Engineering for LLMs: System Design and a Controlled Empirical Study

Abstract

LLM data engineering requires iterative experimentation to determine how data selection, rewards, and curricula affect model learning, making it a natural setting for AI-for-AI (AI4AI) research. Can agents use experimental feedback to progressively improve strategies, and do the resulting strategies improve target-model performance on held-out data? We study how agents use experimental feedback to develop data-engineering strategies under fixed training and evaluation protocols. Our Agentic Data Engineering (ADE) framework lets agents pursue, revise, or combine research directions using cumulative findings. Strategies are selected by in-loop scores and evaluated on held-out benchmarks. Across four tasks—math and code data selection, math reward design, and math curriculum learning—selected strategies achieve absolute gains over generic baselines of up to 13.34% on AIME25 Pass@32 and 5.85% averaged across four held-out benchmarks under fixed training conditions. In-loop scores improve over successive experiments, while held-out trajectories show that later in-loop gains do not always improve generalization. Comparisons of research organization and feedback access favor combining evidence analysis with cumulative findings. Research trajectories show how earlier findings inform strategy revisions and combinations across research directions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
AI-for-AI, autonomous machine learning research, LLM Data Engineering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rSz6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes to explore using LLM agents for data engineering tasks, which can become a motivating starting point for AI-for-AI research. To that end, it proposes a framework, agentic data engineering (ADE), that lets agents pursue, revise, or combine research directions using past findings. Results compare the proposed approach with varying numbers of coordinators, a module that reviews past attempts and proposes which new attempts to make, with a base model and a “generic” baseline.

=== post rebuttal ===

Thank you for the response. However, in my opinion, I think my primary concern (W1) is not well-addressed because the paper in its state does feel like putting different components together and try what happens, rather than providing some insight on how to approach the problem. Hence, I choose to maintain my score.

Critical strengths

I believe the paper’s strengths lie in its engineering efforts in connecting different modules for the agentic framework, and it details the purpose of each of these modules in the paper and explains them to the audience. The highlight on the importance of evaluating agents on held-out data makes sense.

Critical weaknesses
  1. After reading the paper twice, I feel the paper is more like a report documenting how different components are put together for the data engineering agentic framework, and I feel no strong signal on why these components are needed. I also wasn’t able to find anything on what a “generic” baseline is until reading the appendix. Therefore, I think the paper’s presentation and writing are confusing.

  2. Based on Appendix D.2, the “generic” baseline for each of the for tasks are:

Math data selection: random subset; Code data selection: random subset; Reward design: outcome-only reward; Curriculum learning: random ordering;

Since three out of four are basically random baselines, I think a name with “generic” baseline is confusing and misleading to the reader. The baseline is not generic, it is a random baseline, which is a very weak baseline since it makes no assumptions and has no heuristics on search, so saying it is a generic baseline conveys a meaning that the baseline is stronger than it actually is. I would be more convinced that the baseline is generic if it were a tree-search or beam-search.

  1. For the same reason as (2), the paper’s baseline is a weak baseline.
Questions for authors

The paper raises a good point that agentic methods should be evaluated on held-out data. However, if the data distribution of the held-out data is in-distribution, then statistically they would be similar to evaluating on data in the current research progress. I wonder how the held-out data distribution differs from the distribution in the current research progress?

6 comments
AuthorsOpenReview ↗

Weakness 1: Thank you for your comment about evidence for research organization and feedback

I feel no strong signal on why these components are needed. I also wasn’t able to find anything on what a “generic” baseline is until reading the appendix.

We respectfully clarify that the generic baseline for every task is explicitly identified in Table 2 of the main paper (p. 6, lines 270–275), immediately above the main results in Table 3. Its “Generic baseline” column specifies random subset selection for Math and Code Data Selection, outcome-only reward for Reward Design, and random ordering for Curriculum Learning. Appendix D.2 supplies the implementation details. “Generic baseline” denotes the reference intervention trained under the same task conditions as the agent-developed strategies; the common label covers these different intervention types and distinguishes them from the base model before task-specific training.

We appreciate the reviewer’s emphasis on empirical support for the framework’s design. The submitted paper already evaluates research organization and feedback access in Section 4.3 and Figure 5 (p. 8; detailed results in Table 21). Unified Agent uses the same researcher model in one persistent session, testing ADE’s structured organization of planning, implementation, and analysis. Independent Search evaluates the same number of candidates without access to prior candidates’ findings, testing the value of evidence reuse. Outcome-only Feedback and Raw Evidence Feedback further examine feedback and analysis arrangements (Appendix D.6). ADE achieves higher in-loop and held-out macro scores on both metrics than these alternatives in the original Math Data Selection study.

During the rebuttal, we added Unified Agent and Independent Search experiments for Code Data Selection, Math Reward Design, and Math Curriculum Learning. The table includes the original Math comparisons and these new comparator results; all ADE N=1 results are reused from Table 3. Each method evaluates five candidates per repetition under the same task-specific training and evaluation protocol. Results report the Best strategy selected by in-loop metrics across three repetitions; held-out macro averages the four benchmarks for each task.

TaskResult sourceMethodIn-loop: Pass@K / Average@K (%)Held-out macro: Pass@K / Average@K (%)
Math SelectionOriginal paperIndependent Search50.00 / 15.5262.83 / 44.09
Math SelectionOriginal paperUnified Agent50.00 / 14.3862.97 / 42.75
Math SelectionOriginal paperADE N=156.67 / 17.8165.58 / 45.32
Code SelectionAdded in rebuttalIndependent Search40.09 / 28.0230.66 / 22.49
Code SelectionAdded in rebuttalUnified Agent42.37 / 29.6130.74 / 22.33
Code SelectionOriginal paperADE N=145.79 / 31.7433.53 / 25.38
Reward DesignAdded in rebuttalIndependent Search48.60 / 33.0712.82 / 7.22
Reward DesignAdded in rebuttalUnified Agent49.80 / 35.6713.05 / 6.77
Reward DesignOriginal paperADE N=150.20 / 36.3314.93 / 7.80
Curriculum LearningAdded in rebuttalIndependent Search48.40 / 33.2712.03 / 7.28
Curriculum LearningAdded in rebuttalUnified Agent49.40 / 34.8010.88 / 5.94
Curriculum LearningOriginal paperADE N=151.80 / 36.4013.95 / 7.75

ADE achieves higher in-loop and held-out macro scores on both metrics than both alternatives across all four tasks, extending the evidence for structured research organization and reuse of experimental findings beyond the original Math setting. Section 4.4 and Figure 6 provide complementary process evidence: redundancy findings motivate repetition control, and an unsuccessful shortening experiment motivates length matching in the subsequent strategy. Together, the controlled comparisons measure the benefit of these research arrangements, while the trajectory shows how earlier findings inform later decisions.

AuthorsOpenReview ↗

Weakness 2: Thank you for your comment about baseline strength and search policies

Weakness 2

Three of the four reference baselines are random and use no search heuristics. The term “generic” suggests a stronger baseline; tree search or beam search would be more convincing alternatives.

We appreciate the reviewer’s suggestion to examine stronger baselines. “Generic baseline” denotes the task-specific reference intervention used for training under matched conditions. Table 2 (p. 6, lines 270–275), immediately above Table 3, names these interventions explicitly: random subset selection, outcome-only reward, and random ordering. The label describes their reference role. Tree and beam search instead govern candidate exploration and selection.

Our research-process comparisons test whether experimental feedback and research organization improve strategy development under matched candidate budgets. This addresses the Introduction’s questions about feedback-guided research and held-out effectiveness. Under Principle 1, the agent chooses which directions to pursue, revisit, or combine; the Harness enforces valid actions and fixed experimental conditions (Sections 3.2–3.3). Rankings inform the next research decision, while in-loop scores determine the final reported strategy.

The four ablations target these choices directly: (1) Unified Agent uses the same researcher model in one persistent session; (2) Independent Search removes access to previous findings; (3) Outcome-only Feedback permits iterative revision using scalar outcomes and execution facts; and (4) Raw Evidence Feedback additionally supplies raw experimental records. Full ADE combines independent evidence analysis with cumulative research summaries (Appendix D.6). Comparing tree- and beam-search policies is outside this study’s scope: it would assess alternative candidate-expansion and retention policies, whereas these controls assess the value of research organization and evidence use.

The following Math ablation results are already reported in Section 4.3, Figure 5, and Table 21 of the submitted paper. Each method has three repetitions of five candidates under the same training and evaluation protocol. Results report the Best strategy selected by in-loop metrics; held-out macro averages four benchmarks.

Method (original Math ablation)In-loop: Pass@32 / Average@32 (%)Held-out macro: Pass@K / Average@K (%)
Independent Search50.00 / 15.5262.83 / 44.09
Unified Agent50.00 / 14.3862.97 / 42.75
Outcome-only Feedback46.67 / 14.1760.12 / 43.55
Raw Evidence Feedback50.00 / 15.1060.57 / 43.19
ADE N=156.67 / 17.8165.58 / 45.32

ADE exceeds all four alternatives on both in-loop and held-out macro metrics. Raw Evidence Feedback improves both in-loop metrics over Outcome-only Feedback, but its held-out improvement is mixed: Pass@K increases while Average@K decreases. ADE exceeds both feedback variants on both held-out metrics, supporting the workflow that combines evidence analysis with cumulative findings.

Our rebuttal extends Unified Agent and Independent Search to Code Data Selection, Reward Design, and Curriculum Learning, with the full results in our response to your Weakness 1. ADE achieves higher in-loop and held-out macro scores on both metrics than both alternatives in all three tasks, broadening this evidence beyond Math Data Selection.

To directly address the concern about strategy competitiveness, we added two established-method comparisons during the rebuttal: T-SHIRT for Math Data Selection and E2H-C for Math Curriculum Learning. T-SHIRT uses the same 3,840-example pool, 384-example selection budget, and 13-epoch SFT recipe; E2H-C uses difficulty-based scheduling configured with ADE’s 100-step training budget. Both follow ADE’s task-specific target models and evaluation protocol. Comparator rows are individual completed runs; ADE rows reuse the original in-loop-selected N=1 Best results.

TaskResult sourceMethodIn-loop: Pass@K / Average@K (%)Held-out macro: Pass@K / Average@K (%)
Math Data SelectionAdded in rebuttalT-SHIRT53.33 / 15.2163.60 / 44.16
Math Data SelectionOriginal paperADE56.67 / 17.8165.58 / 45.32
Math Curriculum LearningAdded in rebuttalE2H-C51.60 / 34.5311.30 / 7.33
Math Curriculum LearningOriginal paperADE51.80 / 36.4013.95 / 7.75

ADE achieves higher in-loop and held-out macro scores on both metrics than T-SHIRT and E2H-C in these comparisons. These comparisons support both the research workflow and the competitiveness of its outputs.

AuthorsOpenReview ↗

Weakness 3: Thank you for your comment about baseline strength

Weakness 3

For the same reason as (2), the paper’s baseline is a weak baseline.

We appreciate the reviewer’s emphasis on stronger comparisons. As stated in the Introduction (lines 87–94), our contribution is a framework and a controlled empirical study of agent-directed LLM post-training data engineering. The central questions concern how agents use experimental feedback to develop strategies and whether the resulting improvements generalize. The strategies are concrete outputs through which we evaluate this research capability.

This focus on research capability and its mechanisms is shared by concurrent work. RSIBench-Data, Sections 4–5 compares researcher agents under shared infrastructure and analyzes iterative research decisions; What Is Missing from AI Post-Training AI, Section 4 / Table 2 studies experience-driven research and ablates experiment journals, skill libraries, and evaluator agents. PostTrainBench, Sections 3–5 evaluates autonomous post-training agents and analyzes reasoning effort, time budgets, agent scaffolds, and research behavior. These studies examine autonomous research capability, research organization, and the use of experimental evidence.

ADE’s original Section 4.3 ablations directly evaluate these dimensions, comparing Unified Agent, Independent Search, Outcome-only Feedback, and Raw Evidence Feedback under matched candidate budgets. ADE achieves higher in-loop and held-out macro scores on both metrics than all four Math alternatives. Our new Unified Agent and Independent Search comparisons extend the finding to Code Data Selection, Reward Design, and Curriculum Learning, as reported in our response to your Weakness 1.

We additionally evaluated T-SHIRT and E2H-C to assess the practical competitiveness of the strategies themselves. ADE achieves higher in-loop and held-out macro Pass@K and Average@K than these comparators in the evaluated Math Data Selection and Curriculum Learning settings; the full results and settings appear in our response to your Weakness 2. Thus, the evidence covers both the research process that produces strategies and the effectiveness of its outputs.

Reviewer_Kp6yOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed an agentic framework for data engineering tasks including data selection, rewards and curriculum learning. For the agentic data engineering tasks, the authors proposed two principals: 1/ agents can decide the research directions under fixed experimental protocols 2/ the evaluation set is outside the agentic data engineering loop. The performance improved in well-defined tasks like math and coding task.

Critical strengths

The authors formulazied principals for agentic data engineering tasks. The principals looks reasonable.The overall agentic framework is clear.

Critical weaknesses

1/ The experimental results are incomplete. The proposed agentic framework used GPT-5.6 sol codex to design the experiment strategies under the proposed two layer framework for training Qwen2.5-7B-Instruct model. The baselines only included base model and generic training strategies (like random subset or Random ordering). No other competative baselines are compared. For example, an intuitive baseline is to use GPT-5.6 sol codex to design training strategy for Qwen2.5-7B-Instruct model without guidance. 2/ In Table 4, it lists the different strategies can be used by the agent. I think previous work such as auto-glueon (https://auto.gluon.ai/stable/index.html) already used some training strategies for data engineering task. It lacks novelty that discovered by the agents. 3. Minor: Line 164, no definition of S_{in} and S_{online} in mathematic formula.

Questions for authors

(See weaknesses

6 comments
AuthorsOpenReview ↗

Weakness 1: Thank you for your comment about same-model strategy design without ADE organization

Weakness 1

The baselines only included the base model and generic training strategies, such as random subset selection or random ordering, without comparisons to other competitive baselines. An intuitive baseline is to use GPT-5.6 Sol Codex to design training strategies for Qwen2.5-7B-Instruct without guidance.

We thank the reviewer for suggesting this comparison. Unified Agent uses GPT-5.6 Sol Codex—the same researcher model proposed in your comment—and the Math comparison is already reported in Section 4.3, Figure 5, and Table 21. It plans, implements, and analyzes experiments within a persistent session, with access to experimental evidence and its own notes, but without ADE’s prescribed role handoffs and structured Plan/Run memory. The strategy interface, execution infrastructure, and five-candidate budget are held fixed (Appendix D.6).

During the rebuttal, we added Unified Agent experiments for Code Data Selection, Math Reward Design, and Math Curriculum Learning. All ADE N=1 results below are reused from Table 3 of the submitted paper. The table reports each method’s Best strategy selected using in-loop metrics under the paper’s reporting protocol. Held-out macro averages the four benchmarks for each task.

TaskResult sourceMethodIn-loop: Pass@K / Average@K (%)Held-out macro: Pass@K / Average@K (%)
Math SelectionOriginal paperUnified Agent50.00 / 14.3862.97 / 42.75
Math SelectionOriginal paperADE N=156.67 / 17.8165.58 / 45.32
Code SelectionAdded in rebuttalUnified Agent42.37 / 29.6130.74 / 22.33
Code SelectionOriginal paperADE N=145.79 / 31.7433.53 / 25.38
Reward DesignAdded in rebuttalUnified Agent49.80 / 35.6713.05 / 6.77
Reward DesignOriginal paperADE N=150.20 / 36.3314.93 / 7.80
Curriculum LearningAdded in rebuttalUnified Agent49.40 / 34.8010.88 / 5.94
Curriculum LearningOriginal paperADE N=151.80 / 36.4013.95 / 7.75

ADE achieves higher in-loop and held-out macro scores on both metrics across all four tasks. These same-model, matched-budget comparisons support the contribution of ADE’s structured research workflow beyond the capabilities of the underlying researcher model alone.

To address the competitiveness of the resulting strategies, we additionally evaluated T-SHIRT and E2H-C during the rebuttal. Both methods are adapted to ADE’s task-specific target models and evaluation protocols. For Math Data Selection, T-SHIRT selects 384 examples from the same 3,840-example pool and uses the 13-epoch SFT recipe. For Math Curriculum Learning, E2H-C uses difficulty-based scheduling configured with ADE’s 100-step training budget. The comparator rows are individual completed runs; the ADE rows reuse the original Table 3 N=1 Best results selected by in-loop metrics.

TaskResult sourceMethodIn-loop: Pass@K / Average@K (%)Held-out macro: Pass@K / Average@K (%)
Math Data SelectionAdded in rebuttalT-SHIRT53.33 / 15.2163.60 / 44.16
Math Data SelectionOriginal paperADE56.67 / 17.8165.58 / 45.32
Math Curriculum LearningAdded in rebuttalE2H-C51.60 / 34.5311.30 / 7.33
Math Curriculum LearningOriginal paperADE51.80 / 36.4013.95 / 7.75

ADE achieves higher in-loop and held-out macro scores on both metrics than these two established-method comparators, providing complementary evidence of the practical competitiveness of the strategies produced by its research workflow.

Together, these comparisons support both the value of ADE’s structured research workflow and the practical competitiveness of its resulting strategies. They suggest that feedback-guided adaptation to a particular source corpus, target model, and training objective is a promising direction for LLM data engineering. Established methods such as T-SHIRT and E2H-C could also serve as starting points in an extended search space, with experimental feedback guiding which approaches to retain, revise, or combine.

AuthorsOpenReview ↗

Weakness 2: Thank you for your comment about (2) novelty and the relation to AutoGluon; (3) definition of the evaluation functions

Weakness 2

Table 4 lists strategies used by the agent. Prior work such as AutoGluon already uses training strategies for data engineering, making the novelty of the agent-developed strategies unclear.

We appreciate the reviewer’s suggestion to consider AutoGluon, a relevant contribution to automated machine learning whose Tabular workflow emphasizes model training, bagging, and stacking. ADE studies agents that develop and revise executable LLM post-training data interventions from experimental feedback: the strategy logic itself can change across experiments. We will add an AutoGluon reference to our related-work discussion of data engineering for automated machine learning, clarifying this distinction and the shared goal of automation.

As already stated in the Introduction (lines 87–94), our contribution is a framework and a controlled empirical study of agent-directed LLM post-training data engineering. The framework supports long-CoT data selection for math and code SFT, curriculum scheduling across RFT steps, and reward-function design for generated reasoning and answers. Agents use cumulative findings to choose which research directions to pursue, revise, or combine. Holding the target model and training protocol fixed within each task allows us to study these research decisions and evaluate their effects on held-out performance.

Table 4 names and summarizes strategies developed and selected during research; their executable implementations appear in Appendix J.1–J.4. These are research outputs, rather than a predefined strategy menu. Section 4.4 and Figure 6 show how one such output emerges: redundancy findings motivate repetition control, while an unsuccessful shortening experiment motivates matching examples by length. Together, these findings shape Stratified Repetition Control, which improves in-loop Pass@3 by 3.19 percentage points over its parent. This example shows feedback refining both the research hypothesis and the implemented intervention.

The controlled comparisons in Section 4.3 favor ADE’s research organization and access to previous findings; Sections 4.1–4.2 assess strategy effectiveness and whether research progress transfers to held-out data. The additional comparisons reported in our response to Weakness 1 extend this evidence across tasks and against established methods. These results support the contribution at both levels: the process by which agents develop post-training strategies and the practical value of its outputs.

Weakness 3

Line 164, no definition of S_{in} and S_{online} in mathematic formula.

We thank the reviewer for pointing out this notation gap. In Equation (1), SonlineS_{\mathrm{online}} evaluates checkpoints during training to select θp,k⋆\theta^\star_{p,k}, while SinS_{\mathrm{in}} evaluates the selected checkpoint to produce the in-loop performance record used to compare Trials. Both take a model checkpoint, the in-loop dataset DinD_{\mathrm{in}}, and a per-example response-sampling budget as inputs, using the task-defined performance metrics.

Section 3.1 already explains these two evaluation stages and defines their sampling budgets, KonlineK_{\mathrm{online}} and KofflineK_{\mathrm{offline}}; Appendix E specifies the metrics, and Appendix D.7 specifies the ordering used for Trial comparison. We will explicitly define both evaluation functions immediately after Equation (1) so that their meanings and roles are clear at first use.

AuthorsOpenReview ↗

Following up on your concerns — we would welcome your feedback

Thank you again for your time and feedback. We wanted to briefly follow up on the concerns underlying your assessment:

  • Does ADE add value beyond the same GPT model? (Weakness 1) The suggested comparison already appears as Unified Agent in Section 4.3, using the same model without ADE’s prescribed role handoffs and structured memory. We extended it to the other three tasks: ADE achieves higher in-loop and held-out macro scores on both metrics across all four tasks under matched candidate budgets. The added T-SHIRT and E2H-C comparisons also support strategy competitiveness.
  • What is new relative to AutoGluon? (Weakness 2) AutoGluon automates predictive-model training, including bagging and stacking. ADE studies agents that develop and revise executable LLM post-training data-selection, curriculum, and reward strategies from experimental findings. Our contribution is this research framework and its controlled empirical study. Table 4 presents research outputs; Figure 6 traces feedback-guided revisions, and Section 4.3 evaluates the research workflow. The shared goal of automation therefore leaves a distinct research question and contribution.
  • Evaluation notation (Weakness 3). We clarified both functions’ roles and will define them explicitly beside Equation (1).

We would really appreciate your thoughts on these clarifications and results. If they address your concerns, we would be grateful for your reconsideration of the assessment. As the discussion period draws to a close, we would especially appreciate hearing your thoughts on what we could clarify or improve further. Thank you again for your time and help in strengthening the paper.

Reviewer_z7CvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Agentic Data Engineering (ADE), a multi-agent framework that allows AI agents to develop and improve data engineering strategies through iterative experiments. The system uses different agents for planning, implementation, and result analysis, along with shared memory to guide future research decisions. It also separates in-loop evaluation from held-out evaluation to measure whether the discovered strategies generalize to unseen data. The authors evaluate ADE on four tasks covering math and code data selection, reward design, and curriculum learning. The results show improvements over generic baselines, and additional ablation studies and research trajectory analyses examine the effects of memory, feedback, and agent organization on the research process.

Critical strengths
  1. The paper studies an important question of whether AI agents can use experimental feedback to improve their research decisions, rather than simply running many experiments and selecting the best result. I think this is a meaningful problem for AI4AI research, and the separation between in-loop and held-out evaluation is a useful design choice.
  2. The proposed ADE framework combines multiple research agents, shared memory, and controlled experiments into a complete workflow. The authors evaluate the system across four data engineering tasks and provide ablation studies and research trajectories to examine how feedback and memory affect the research process. These analyses provide useful insights into agent-driven research and support the value of the proposed framework.
Critical weaknesses
  1. One of the main goals of the paper is to show that agents can use experimental feedback to develop better research strategies over time. However, the improvement in best-so-far in-loop scores in Figure 3 may partly come from trying more strategies rather than learning from previous experiments. Although the authors provide ablation studies and research trajectories, the evidence is still limited in showing how much feedback-guided decisions contribute to the improvements.
  2. In Table 3, the authors mainly compare ADE-developed strategies with generic baselines, such as random data selection, outcome-only rewards, and random curriculum ordering. While these comparisons show that ADE can find useful strategies, they do not clearly demonstrate how competitive the discovered strategies are against established data engineering methods. This makes it difficult to assess the practical significance of the reported improvements.
Questions for authors
  1. In Section 4.2, the authors use the improvement in best-so-far in-loop scores to show that ADE develops better strategies over successive experiments. However, best-so-far scores can naturally increase as more strategies are tested, even without learning from previous experiments. Although Section 4.3 includes an Independent Search baseline, could the authors clarify how much of the improvement comes from using previous findings rather than simply exploring more candidates? Does this pattern hold across all four data engineering tasks?
  2. In Section 4.3 and Figure 5, the authors compare ADE with Independent Search, Unified Agent, and different feedback settings to study the value of memory and research organization. However, these comparisons are conducted only on Math Data Selection. Given that reward design and curriculum learning involve different types of research decisions, is there evidence that the same benefits of shared memory and feedback analysis also hold for these tasks?
  3. In Table 3, the authors mainly compare ADE-developed strategies with generic baselines, such as random subset selection and random curriculum ordering. While these results show that ADE can find useful strategies, it is unclear how competitive they are against recent data engineering methods. For instance, T-SHIRT [1] provides a stronger data selection approach for instruction tuning, while E2H Reasoner [2] studies difficulty-based curricula for RL training. Could the authors compare ADE's discovered strategies with such recent methods under the same training settings and data budgets?
  4. In Section 4.2 and Figure 3, the authors observe that held-out performance does not always improve with in-loop performance. For Math Data Selection at C = 3, AIME25 Pass@32 reaches 46.67% but later drops to 41.11%, even as the in-loop score continues to increase. Does this suggest that the current in-loop evaluation is not a reliable signal for selecting strategies that generalize well? How sensitive are the final results to this selection process, and could the authors clarify how often similar behavior occurs across runs and tasks?

[1] Fu, Yanjun, Faisal Hamman, and Sanghamitra Dutta. "T-shirt: Token-selective hierarchical data selection for instruction tuning." Advances in Neural Information Processing Systems 38 (2026): 113932-113958. [2] Parashar, Shubham, et al. "Curriculum reinforcement learning from easy to hard tasks improves llm reasoning." International Conference on Learning Representations. Vol. 2026. 2026.

7 comments
AuthorsOpenReview ↗

Weakness 1: Thank you for your comment about feedback-guided improvement — Part 1: Existing evidence

Weakness 1 — Part 1: Existing evidence

The improvement in best-so-far in-loop scores in Figure 3 may partly come from trying more strategies rather than learning from previous experiments. Although the authors provide ablation studies and research trajectories, the evidence is still limited in showing how much feedback-guided decisions contribute to the improvements.

We thank the reviewer for raising this concern. We agree that it is important to analyze how much feedback-guided decisions contribute to strategy improvement beyond simply evaluating more candidates. We already address this question through the controlled comparisons in Section 4.3 and the feedback-linked research trajectories in Section 4.4, and further strengthen this evidence with the additional three-task ablation experiments reported below. This distinction is already stated in Introduction Question 1 (lines 43–47), including the explicit observation that “a strong final strategy may reflect best-of-many sampling rather than feedback-guided research” (lines 46–47). The Introduction also states that answering our research questions requires assessing progress over experiments, links between prior feedback and subsequent decisions, and held-out effectiveness (lines 50–52). Our evaluation examines these complementary aspects in Sections 4.2–4.4.

In the submitted paper, Section 4.2 and Figure 3 track ADE’s improvement as experiments accumulate, while Section 4.3 and Figure 5 already include a matched-budget Independent Search comparison on Math Data Selection to assess the benefit of using previous findings. Figure 3 tracks the best in-loop scores over successive completed experiments and the held-out performance of the selected strategies. It establishes the magnitude and timing of the observed improvement and shows whether that improvement generalizes. Section 4.3 complements these trajectories with a controlled comparison: Independent Search evaluates the same number of candidates without access to prior candidates’ findings, allowing us to assess the benefit of feedback use at a matched candidate budget. This directly addresses the reviewer’s question—whether ADE achieves better results because it uses previous experimental findings, beyond the opportunity to test more strategies.

The Independent Search comparison controls for the number of candidates evaluated. As specified in Section 4.3 and Appendices D.6–D.7, ADE with one Coordinator and Independent Search each evaluate five candidates per repetition under the same target-model initialization, candidate pool, training protocol, evaluation splits, and selection rule. Independent Search develops candidates without access to findings from the other candidates, whereas ADE uses accumulated findings to guide subsequent experiments. Independent Search also selects its best candidate using the same in-loop criteria, so the comparison explicitly accounts for the benefit of best-of-many selection. The original Math result, already reported in Figure 5 and Table 21, gives a 2.75-percentage-point advantage in held-out macro Pass@K over Independent Search.

Part 2 reports the additional three-task comparisons and connects these results to the feedback-guided decisions illustrated in Figure 6.

AuthorsOpenReview ↗

Weakness 1: Thank you for your comment about feedback-guided improvement — Part 2: Extended comparisons

Weakness 1 — Part 2: Extended comparisons

The evidence is still limited in showing how much feedback-guided decisions contribute to the improvements.

For this rebuttal, we extend the existing Math comparison to three additional tasks: Code Data Selection, Math Reward Design, and Math Curriculum Learning, using the same standard protocol. These additional comparisons strengthen the quantitative evidence for feedback-guided improvement beyond the original Math setting. The table combines the original Math comparison with three new Independent Search results; all ADE values are from the submitted paper. For every task, each method uses three repetitions of five candidates; the reported result is the Best strategy selected by in-loop metrics across the three repetition winners, rather than an average over repetitions. ADE outperforms Independent Search on both held-out macro metrics in all four tasks. Held-out macro averages the four held-out benchmarks for each task; gains are absolute percentage-point differences (pp).

TaskComparison sourceIndependent Search in-loop: Pass@K / Average@K (%)ADE in-loop: Pass@K / Average@K (%)In-loop gain: Pass@K / Average@K (pp)Independent Search held-out macro: Pass@K / Average@K (%)ADE held-out macro: Pass@K / Average@K (%)Held-out macro gain: Pass@K / Average@K (pp)
Math Data SelectionSubmitted paper, Figure 5 / Table 2150.00 / 15.5256.67 / 17.81+6.67 / +2.2962.83 / 44.0965.58 / 45.32+2.75 / +1.23
Code Data SelectionNew supplemental experiment40.09 / 28.0245.79 / 31.74+5.70 / +3.7230.66 / 22.4933.53 / 25.38+2.87 / +2.89
Math Reward DesignNew supplemental experiment48.60 / 33.0750.20 / 36.33+1.60 / +3.2612.82 / 7.2214.93 / 7.80+2.11 / +0.58
Math Curriculum LearningNew supplemental experiment48.40 / 33.2751.80 / 36.40+3.40 / +3.1312.03 / 7.2813.95 / 7.75+1.92 / +0.47

These matched-budget differences quantify the observed advantage of feedback-guided ADE over independent candidate generation: held-out macro Pass@K improves by 1.92–2.87 percentage points, with Average@K improving in every task. In-loop Pass@K also improves by 6.67, 5.70, 1.60, and 3.40 percentage points, respectively. Because the candidate budgets are matched, the gains support the value of using accumulated findings beyond simply evaluating more candidates, and their presence on held-out benchmarks shows that the advantage extends beyond the research feedback set.

The trajectory already presented in Section 4.4 and Figure 6 illustrates how feedback changes subsequent decisions. In Code Data Selection, redundancy findings motivate repetition control, while an unsuccessful shortening experiment motivates matching examples by length. The resulting Stratified Repetition Control strategy therefore tests whether reducing repetition at comparable length improves learning, and increases in-loop Pass@3 by 3.19 percentage points over its parent. This example shows how an unsuccessful experiment can inform a more targeted subsequent intervention. It complements the matched-budget comparisons: the comparisons quantify the performance advantage, while the trajectory makes the feedback-to-decision link explicit.

AuthorsOpenReview ↗

Weakness 2: Thank you for your comment about established data-engineering methods

Weakness 2

In Table 3, the authors mainly compare ADE-developed strategies with generic baselines. While these comparisons show that ADE can find useful strategies, they do not clearly demonstrate how competitive the discovered strategies are against established data engineering methods.

We thank the reviewer for raising this concern. As stated in the Introduction (lines 87–94), our central contribution is the ADE framework and a controlled study of research agents, iterative improvement, and the mechanisms supporting feedback use. Section 4.1 evaluates the effectiveness of ADE-developed strategies, while Sections 4.2–4.4 examine iterative improvement, research organization, and feedback-guided decisions. Comparisons with established methods further assess the practical competitiveness of these strategies. Established data-engineering methods can also serve as candidate strategies within this research process, supplied through method descriptions, reusable skills, or research memory and then evaluated or adapted to the task. This is a possible extension of ADE’s search space; the present experiments evaluate T-SHIRT and E2H-C as external comparators.

Concurrent work likewise examines autonomous research capability and its mechanisms. RSIBench-Data, Sections 4–5 compares researcher agents and analyzes iterative research decisions; What Is Missing from AI Post-Training AI, Section 4 / Table 2 studies experience-driven research and ablates experiment journals, skill libraries, and evaluator agents; and PostTrainBench, Sections 3–5 evaluates autonomous post-training agents, agent scaffolds, budgets, and research behavior. These studies examine researcher capability, research organization, and the use of experimental evidence.

We appreciate the reviewer’s suggestion to complement the framework-level evaluation with established-method comparisons, which help clarify the practical value of ADE-developed strategies. Following this suggestion, we have completed additional comparisons with the two methods specifically mentioned in your Question 3: T-SHIRT: Token-Selective Hierarchical Data Selection for Instruction Tuning for Math Data Selection, and the E2H-C variant from Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning for Math Curriculum Learning. We adapt both methods to ADE’s task-specific target models and evaluation protocol. T-SHIRT selects 384 examples from the same 3,840-example pool and uses the 13-epoch SFT recipe. E2H-C uses difficulty-based scheduling configured with ADE’s 100-step training budget. Both comparisons evaluate the selected checkpoint on the in-loop set and four held-out benchmarks.

Held-out macro averages the four benchmarks for each task. The comparator rows are individual completed runs, while the ADE rows reuse the submitted paper’s N=1 Best strategy selected by in-loop metrics.

TaskMethodIn-loop: Pass@K / Average@K (%)Held-out macro: Pass@K / Average@K (%)
Math Data SelectionT-SHIRT53.33 / 15.2163.60 / 44.16
Math Data SelectionADE56.67 / 17.8165.58 / 45.32
Math Curriculum LearningE2H-C51.60 / 34.5311.30 / 7.33
Math Curriculum LearningADE51.80 / 36.4013.95 / 7.75

ADE achieves higher in-loop and held-out macro scores on both metrics than the two established-method comparators.

The ADE-developed curriculum shares E2H’s use of difficulty-structured training, while combining foundation–bridge–challenge cycling with coverage-aware sampling and same-subject reinforcement (Appendix J.4). This illustrates how agent-directed research can arrive at strategies that share established curriculum principles while developing task-specific implementations.

These results suggest that feedback-guided, task-specific strategy development is a promising direction for LLM data engineering. Its practical value lies in evaluating and refining data interventions for a particular source corpus, target model, and training objective. Established methods can provide useful starting points within this process, while experimental feedback guides which strategies to retain, revise, or combine.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to the Area Chair and all reviewers

Dear Area Chair and Reviewers,

Thank you for your feedback. As the discussion period draws to a close, we summarize the key clarifications and additional evidence in our responses:

  • Feedback-guided improvement — z7Cv. Section 4.3 already compares ADE with Independent Search under matched candidate budgets, complementing the trajectories in Sections 4.2 and 4.4. We extended this comparison to the other three tasks. ADE achieves higher in-loop and held-out macro scores on both metrics across all four tasks.
  • Same-model baseline— Kp6y. The suggested same-model baseline is already evaluated as Unified Agent in Section 4.3. We extended this comparison to the other three tasks, with consistent advantages for ADE. We also clarified that Table 4 reports strategies developed during research and explained how ADE differs from AutoGluon.
  • Baseline definitions and design evidence — rSz6. The submitted paper already defines every task’s generic baseline in main-text Table 2 and evaluates research organization and feedback use through four ablations in Section 4.3. Our response points to this existing evidence and supplements it with comparisons on three additional tasks.
  • Established-method comparisons — z7Cv. We added comparisons with T-SHIRT and E2H-C. ADE achieves higher in-loop and held-out macro Pass@K and Average@K in both comparisons. These results also address the broader baseline-strength concerns raised by Kp6y and rSz6.
  • Generalization and held-out evaluation — z7Cv, rSz6. We clarified the evaluation splits and added a 24-run analysis quantifying the divergence already shown in Figure 3 between in-loop improvement and held-out performance.

We would appreciate your feedback on these responses and welcome any remaining questions. If the clarifications and additional evidence resolve your concerns, we kindly ask you to reconsider your assessment and scores accordingly.

Thank you again for your time and consideration.

AuthorsOpenReview ↗

Revised PDF: additional evidence and clarifications

Dear Area Chair and Reviewers,

Thank you for the time, care, and thoughtful questions you have brought to this discussion. We are glad that our responses have helped address some of the concerns raised here. We have updated the PDF to incorporate the evidence from our responses; additions are marked in blue to make them easy to find.

  • Appendix K.1 extends the Independent Search and Unified Agent comparisons to all four tasks, with in-loop and per-benchmark held-out results. The Math comparisons and ADE results were already in the submitted paper; the other comparator results are new.
  • Appendix K.2 gives the full T-SHIRT and E2H-C comparisons under the corresponding ADE task settings. Appendix K.3 quantifies the held-out regressions already illustrated in Figure 3 across 24 original runs.
  • Section 3.1 now defines the two evaluation functions beside Equation (1). We also cite AutoGluon-Tabular briefly in Appendix A. The original baseline definitions in Table 2, research-process ablations in Section 4.3, and strategy examples in Table 4 and Appendix J remain available for the questions raised in the discussion.

We hope the revised PDF makes the evidence easier to assess. With the discussion period nearing its end, we would be grateful if you could let us know whether these additions address your concerns, or point us to any question that remains unresolved. Your feedback has helped us strengthen the paper, and we appreciate your continued consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.