- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes DataMaster, an autonomous data-engineering framework that optimizes executable data states while keeping the underlying learning algorithm fixed. It integrates a tree-structured search process with Red nodes for external data discovery, Black nodes for data refinement, a shared Data Pool, and Global Memory. Experiments on MLE-Bench Lite and PostTrainBench demonstrate improvements over several baselines, achieving a 68.18% medal rate on MLE-Bench Lite and a 28.35% average score on PostTrainBench.
Critical strengths
- Well-motivated problem formulation. The paper addresses autonomous data engineering as a primary optimization objective, which is a practically relevant direction for agentic machine learning systems.
- Promising empirical results. DataMaster achieves improvements across classical ML tasks and LLM post-training, with additional component analyses and data-leakage audits supporting the evaluation.
Critical weaknesses
- Limited methodological novelty. The proposed framework mainly combines existing techniques, including tree search, shared data storage, and memory-augmented agents. The methodological advances beyond prior agent frameworks are not sufficiently clear.
- Unclear UCB scheduling mechanism. Section 3.5 and Appendix D assign infinite priority to unvisited frontier nodes, making reward-guided selection unclear. The reward definition also lacks explicit handling of lower-is-better evaluation metrics.
- Insufficient ablation studies. The component ablations are conducted on only 10 MLE-Bench Lite tasks, without corresponding ablations on PostTrainBench. This limits evidence for the contribution of individual components in the open-world setting.
- Confounded data-only optimization. Although the paper emphasizes a fixed learning algorithm, Appendix B.2.4 allows changes to learning rate, batch size, and training epochs on PostTrainBench, making it difficult to attribute all improvements exclusively to data engineering.
- High computational cost. According to Table 8, DataMaster requires substantially more runtime and API cost than several baselines. Cost-matched comparisons are needed to establish whether the improvements come from better data engineering rather than greater search expenditure.
Questions for authors
Please see the weaknesses.
2 comments
Response to Reviewer va8v [2/2]
[W5] High Computational Cost
- Cost-controlled evaluation. We additionally evaluate DataMaster with its API expenditure capped at $0.50, using the same 10-task MLE-Bench Lite subset:
| Method | API Cost | Agent Tokens (K) | Medal Rate | Gold Rate |
|---|---|---|---|---|
| Claude Code | $0.56 | 634.5 | 50% | 30% |
| ML-Master 2.0 | $0.29 | 462.9 | 60% | 30% |
| DataMaster @ $0.50 | $0.50 | 985.2 | 90% | 70% |
| DataMaster Full | $1.61 | 3,286.6 | 90% | 80% |
Reducing API expenditure by 68.9% preserves DataMaster's 90% Medal Rate and retains 70% Gold Rate, demonstrating that most of its gains can be achieved under a substantially smaller agent budget.
- Cost–performance tradeoff. The API cost represents only a small incremental overhead on top of downstream model training, whose compute budget is fixed across methods. DataMaster's full agent-side API cost is only $1.61 per task. We argue that minimizing agent expenditure is not an objective in itself: a modest additional investment in data discovery and validation is worthwhile when it yields better training data and higher downstream performance. Indeed, the full-budget configuration further improves Gold Medal Rate from 70% to 80% at the same 90% Medal Rate.
Finally: We really appreciate your insightful suggestion. We hope the above clarifications resolve the concerns. If the response is helpful, we would be grateful if you could consider adjusting your score accordingly.
Response to Reviewer va8v [1/2]
We sincerely appreciate your time and thoughtful comments. Below, we respond to each point in detail.
[W1] Limited Methodological Novelty
DataMaster's contribution is a new formulation and system design for autonomous data engineering, rather than a new tree-search algorithm or memory primitive. We formulate data engineering as a constrained search over executable data states under a fixed learning algorithm, where data discovery, refinement, and downstream validation jointly drive optimization.
Its components are specifically designed to address the structural challenges of this problem, rather than being a generic combination of existing techniques.
Together, they establish a closed-loop process in which discovered data is refined, validated through downstream training, and used to guide subsequent search.
[W2] UCB Scheduling Mechanism
Optimistic initialization of unvisited nodes. Assigning infinite priority to unvisited nodes is a standard optimistic initialization strategy in tree search, ensuring that every newly generated frontier node receives an initial evaluation. This is only a one-time initialization: after its first execution, the node is scheduled according to Eq. (2), where historical rewards and visit counts guide subsequent selection. Thus, reward-guided scheduling remains central throughout the search.
Direction-aware reward normalization. We clarify that Black-node rewards must account for the direction of the underlying evaluation metric. We define a task-specific normalized utility y~v∈[0,1], such that higher values consistently indicate better performance, including for lower-is-better metrics. The immediate reward is defined as:
This ensures that UCB consistently maximizes utility across tasks, regardless of their original metric directions.
[W3] Ablation Studies on PostTrainBench
We extend the component ablations to PostTrainBench to examine whether DataMaster's components contribute consistently in the open-world LLM post-training setting. Specifically, we evaluate all seven tasks under the same Qwen3-1.7B-Base initialization, training recipe, and compute budget.
For w/o Red, we use a fixed candidate data pool; w/o Black disables task-specific data refinement while retaining necessary format adaptation; w/o Memory disables cross-node retrieval of historical outcomes and reusable findings.
These results isolate the contributions of external data discovery, data refinement, and cumulative memory in open-world LLM post-training, complementing the existing MLE-Bench Lite ablations.
[W4] Confounded Data-only Optimization
Fixed learning algorithm with necessary configuration adjustments. DataMaster fixes the model architecture, training objective, and optimization algorithm. Limited adjustments to learning rate, batch size, and epochs are necessary to accommodate changes in dataset size and memory requirements; otherwise, training may become unstable, fail to converge, or encounter out-of-memory errors.
Empirical hyperparameter adjustment statistics. We audit Black-node execution traces and find that hyperparameter adjustments are infrequent:
Learning-rate adjustments are particularly rare, while batch-size and epoch adjustments primarily accommodate data-volume and memory constraints. These statistics indicate that DataMaster's primary search behavior remains data construction rather than hyperparameter optimization.