- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes to explore using LLM agents for data engineering tasks, which can become a motivating starting point for AI-for-AI research. To that end, it proposes a framework, agentic data engineering (ADE), that lets agents pursue, revise, or combine research directions using past findings. Results compare the proposed approach with varying numbers of coordinators, a module that reviews past attempts and proposes which new attempts to make, with a base model and a “generic” baseline.
=== post rebuttal ===
Thank you for the response. However, in my opinion, I think my primary concern (W1) is not well-addressed because the paper in its state does feel like putting different components together and try what happens, rather than providing some insight on how to approach the problem. Hence, I choose to maintain my score.
Critical strengths
I believe the paper’s strengths lie in its engineering efforts in connecting different modules for the agentic framework, and it details the purpose of each of these modules in the paper and explains them to the audience. The highlight on the importance of evaluating agents on held-out data makes sense.
Critical weaknesses
-
After reading the paper twice, I feel the paper is more like a report documenting how different components are put together for the data engineering agentic framework, and I feel no strong signal on why these components are needed. I also wasn’t able to find anything on what a “generic” baseline is until reading the appendix. Therefore, I think the paper’s presentation and writing are confusing.
-
Based on Appendix D.2, the “generic” baseline for each of the for tasks are:
Math data selection: random subset; Code data selection: random subset; Reward design: outcome-only reward; Curriculum learning: random ordering;
Since three out of four are basically random baselines, I think a name with “generic” baseline is confusing and misleading to the reader. The baseline is not generic, it is a random baseline, which is a very weak baseline since it makes no assumptions and has no heuristics on search, so saying it is a generic baseline conveys a meaning that the baseline is stronger than it actually is. I would be more convinced that the baseline is generic if it were a tree-search or beam-search.
- For the same reason as (2), the paper’s baseline is a weak baseline.
Questions for authors
The paper raises a good point that agentic methods should be evaluated on held-out data. However, if the data distribution of the held-out data is in-distribution, then statistically they would be similar to evaluating on data in the current research progress. I wonder how the held-out data distribution differs from the distribution in the current research progress?
6 comments
Weakness 2: Thank you for your comment about baseline strength and search policies
Weakness 2
Three of the four reference baselines are random and use no search heuristics. The term “generic” suggests a stronger baseline; tree search or beam search would be more convincing alternatives.
We appreciate the reviewer’s suggestion to examine stronger baselines. “Generic baseline” denotes the task-specific reference intervention used for training under matched conditions. Table 2 (p. 6, lines 270–275), immediately above Table 3, names these interventions explicitly: random subset selection, outcome-only reward, and random ordering. The label describes their reference role. Tree and beam search instead govern candidate exploration and selection.
Our research-process comparisons test whether experimental feedback and research organization improve strategy development under matched candidate budgets. This addresses the Introduction’s questions about feedback-guided research and held-out effectiveness. Under Principle 1, the agent chooses which directions to pursue, revisit, or combine; the Harness enforces valid actions and fixed experimental conditions (Sections 3.2–3.3). Rankings inform the next research decision, while in-loop scores determine the final reported strategy.
The four ablations target these choices directly: (1) Unified Agent uses the same researcher model in one persistent session; (2) Independent Search removes access to previous findings; (3) Outcome-only Feedback permits iterative revision using scalar outcomes and execution facts; and (4) Raw Evidence Feedback additionally supplies raw experimental records. Full ADE combines independent evidence analysis with cumulative research summaries (Appendix D.6). Comparing tree- and beam-search policies is outside this study’s scope: it would assess alternative candidate-expansion and retention policies, whereas these controls assess the value of research organization and evidence use.
The following Math ablation results are already reported in Section 4.3, Figure 5, and Table 21 of the submitted paper. Each method has three repetitions of five candidates under the same training and evaluation protocol. Results report the Best strategy selected by in-loop metrics; held-out macro averages four benchmarks.
| Method (original Math ablation) | In-loop: Pass@32 / Average@32 (%) | Held-out macro: Pass@K / Average@K (%) |
|---|---|---|
| Independent Search | 50.00 / 15.52 | 62.83 / 44.09 |
| Unified Agent | 50.00 / 14.38 | 62.97 / 42.75 |
| Outcome-only Feedback | 46.67 / 14.17 | 60.12 / 43.55 |
| Raw Evidence Feedback | 50.00 / 15.10 | 60.57 / 43.19 |
| ADE N=1 | 56.67 / 17.81 | 65.58 / 45.32 |
ADE exceeds all four alternatives on both in-loop and held-out macro metrics. Raw Evidence Feedback improves both in-loop metrics over Outcome-only Feedback, but its held-out improvement is mixed: Pass@K increases while Average@K decreases. ADE exceeds both feedback variants on both held-out metrics, supporting the workflow that combines evidence analysis with cumulative findings.
Our rebuttal extends Unified Agent and Independent Search to Code Data Selection, Reward Design, and Curriculum Learning, with the full results in our response to your Weakness 1. ADE achieves higher in-loop and held-out macro scores on both metrics than both alternatives in all three tasks, broadening this evidence beyond Math Data Selection.
To directly address the concern about strategy competitiveness, we added two established-method comparisons during the rebuttal: T-SHIRT for Math Data Selection and E2H-C for Math Curriculum Learning. T-SHIRT uses the same 3,840-example pool, 384-example selection budget, and 13-epoch SFT recipe; E2H-C uses difficulty-based scheduling configured with ADE’s 100-step training budget. Both follow ADE’s task-specific target models and evaluation protocol. Comparator rows are individual completed runs; ADE rows reuse the original in-loop-selected N=1 Best results.
| Task | Result source | Method | In-loop: Pass@K / Average@K (%) | Held-out macro: Pass@K / Average@K (%) |
|---|---|---|---|---|
| Math Data Selection | Added in rebuttal | T-SHIRT | 53.33 / 15.21 | 63.60 / 44.16 |
| Math Data Selection | Original paper | ADE | 56.67 / 17.81 | 65.58 / 45.32 |
| Math Curriculum Learning | Added in rebuttal | E2H-C | 51.60 / 34.53 | 11.30 / 7.33 |
| Math Curriculum Learning | Original paper | ADE | 51.80 / 36.40 | 13.95 / 7.75 |
ADE achieves higher in-loop and held-out macro scores on both metrics than T-SHIRT and E2H-C in these comparisons. These comparisons support both the research workflow and the competitiveness of its outputs.
Weakness 3: Thank you for your comment about baseline strength
Weakness 3
For the same reason as (2), the paper’s baseline is a weak baseline.
We appreciate the reviewer’s emphasis on stronger comparisons. As stated in the Introduction (lines 87–94), our contribution is a framework and a controlled empirical study of agent-directed LLM post-training data engineering. The central questions concern how agents use experimental feedback to develop strategies and whether the resulting improvements generalize. The strategies are concrete outputs through which we evaluate this research capability.
This focus on research capability and its mechanisms is shared by concurrent work. RSIBench-Data, Sections 4–5 compares researcher agents under shared infrastructure and analyzes iterative research decisions; What Is Missing from AI Post-Training AI, Section 4 / Table 2 studies experience-driven research and ablates experiment journals, skill libraries, and evaluator agents. PostTrainBench, Sections 3–5 evaluates autonomous post-training agents and analyzes reasoning effort, time budgets, agent scaffolds, and research behavior. These studies examine autonomous research capability, research organization, and the use of experimental evidence.
ADE’s original Section 4.3 ablations directly evaluate these dimensions, comparing Unified Agent, Independent Search, Outcome-only Feedback, and Raw Evidence Feedback under matched candidate budgets. ADE achieves higher in-loop and held-out macro scores on both metrics than all four Math alternatives. Our new Unified Agent and Independent Search comparisons extend the finding to Code Data Selection, Reward Design, and Curriculum Learning, as reported in our response to your Weakness 1.
We additionally evaluated T-SHIRT and E2H-C to assess the practical competitiveness of the strategies themselves. ADE achieves higher in-loop and held-out macro Pass@K and Average@K than these comparators in the evaluated Math Data Selection and Curriculum Learning settings; the full results and settings appear in our response to your Weakness 2. Thus, the evidence covers both the research process that produces strategies and the effectiveness of its outputs.
Weakness 1: Thank you for your comment about evidence for research organization and feedback
We respectfully clarify that the generic baseline for every task is explicitly identified in Table 2 of the main paper (p. 6, lines 270–275), immediately above the main results in Table 3. Its “Generic baseline” column specifies random subset selection for Math and Code Data Selection, outcome-only reward for Reward Design, and random ordering for Curriculum Learning. Appendix D.2 supplies the implementation details. “Generic baseline” denotes the reference intervention trained under the same task conditions as the agent-developed strategies; the common label covers these different intervention types and distinguishes them from the base model before task-specific training.
We appreciate the reviewer’s emphasis on empirical support for the framework’s design. The submitted paper already evaluates research organization and feedback access in Section 4.3 and Figure 5 (p. 8; detailed results in Table 21). Unified Agent uses the same researcher model in one persistent session, testing ADE’s structured organization of planning, implementation, and analysis. Independent Search evaluates the same number of candidates without access to prior candidates’ findings, testing the value of evidence reuse. Outcome-only Feedback and Raw Evidence Feedback further examine feedback and analysis arrangements (Appendix D.6). ADE achieves higher in-loop and held-out macro scores on both metrics than these alternatives in the original Math Data Selection study.
During the rebuttal, we added Unified Agent and Independent Search experiments for Code Data Selection, Math Reward Design, and Math Curriculum Learning. The table includes the original Math comparisons and these new comparator results; all ADE N=1 results are reused from Table 3. Each method evaluates five candidates per repetition under the same task-specific training and evaluation protocol. Results report the Best strategy selected by in-loop metrics across three repetitions; held-out macro averages the four benchmarks for each task.
ADE achieves higher in-loop and held-out macro scores on both metrics than both alternatives across all four tasks, extending the evidence for structured research organization and reuse of experimental findings beyond the original Math setting. Section 4.4 and Figure 6 provide complementary process evidence: redundancy findings motivate repetition control, and an unsuccessful shortening experiment motivates length matching in the subsequent strategy. Together, the controlled comparisons measure the benefit of these research arrangements, while the trajectory shows how earlier findings inform later decisions.