acceptodds
Under review as a conference paper at ICLR 2027

DataMaster: Towards Autonomous Data Engineering for Machine Learning

12 views

Abstract

As model families, training recipes, and compute budgets become increasingly standardized, further gains in machine learning systems depend increasingly on data. Yet data engineering remains largely manual and ad hoc: practitioners repeatedly search for external datasets, adapt them to existing pipelines, validate candidate data through downstream training, and carry forward lessons from prior attempts. We study task-conditioned autonomous data engineering, where an autonomous agent improves a fixed learning algorithm by optimizing only the data side, including external data discovery, data selection and composition, cleaning and transformation. The goal is to obtain a stronger downstream solution while leaving the learning algorithm unchanged. To address the open-ended search space, branch-dependent refinement, and delayed validation inherent in autonomous data engineering, we propose DataMaster, a data-agent framework that integrates tree-structured search, shared candidate data, and cumulative memory. DataMaster consists of three key components: a DataTree that organizes alternative data-engineering branches, a shared Data Pool that stores discovered external data sources for reuse, and a Global Memory that records node outcomes, artifacts, and reusable findings. Together, these components allow the agent to discover candidate data, construct executable training inputs, evaluate them through downstream feedback, and carry useful evidence across branches. We evaluate DataMaster on two types of benchmarks, MLE-Bench Lite and PostTrainBench. On MLE-Bench Lite, it improves medal rate by 32.27% over the initial score; on PostTrainBench, it surpasses the instruct model on GPQA (31.02% vs 30.35%).

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Autonomous Data Engineering, Data-Centric AI, LLM Agents, Automated Machine Learning, AI4AI

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_va8vOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DataMaster, an autonomous data-engineering framework that optimizes executable data states while keeping the underlying learning algorithm fixed. It integrates a tree-structured search process with Red nodes for external data discovery, Black nodes for data refinement, a shared Data Pool, and Global Memory. Experiments on MLE-Bench Lite and PostTrainBench demonstrate improvements over several baselines, achieving a 68.18% medal rate on MLE-Bench Lite and a 28.35% average score on PostTrainBench.

Critical strengths
  1. Well-motivated problem formulation. The paper addresses autonomous data engineering as a primary optimization objective, which is a practically relevant direction for agentic machine learning systems.
  2. Promising empirical results. DataMaster achieves improvements across classical ML tasks and LLM post-training, with additional component analyses and data-leakage audits supporting the evaluation.
Critical weaknesses
  1. Limited methodological novelty. The proposed framework mainly combines existing techniques, including tree search, shared data storage, and memory-augmented agents. The methodological advances beyond prior agent frameworks are not sufficiently clear.
  2. Unclear UCB scheduling mechanism. Section 3.5 and Appendix D assign infinite priority to unvisited frontier nodes, making reward-guided selection unclear. The reward definition also lacks explicit handling of lower-is-better evaluation metrics.
  3. Insufficient ablation studies. The component ablations are conducted on only 10 MLE-Bench Lite tasks, without corresponding ablations on PostTrainBench. This limits evidence for the contribution of individual components in the open-world setting.
  4. Confounded data-only optimization. Although the paper emphasizes a fixed learning algorithm, Appendix B.2.4 allows changes to learning rate, batch size, and training epochs on PostTrainBench, making it difficult to attribute all improvements exclusively to data engineering.
  5. High computational cost. According to Table 8, DataMaster requires substantially more runtime and API cost than several baselines. Cost-matched comparisons are needed to establish whether the improvements come from better data engineering rather than greater search expenditure.
Questions for authors

Please see the weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer va8v [1/2]

We sincerely appreciate your time and thoughtful comments. Below, we respond to each point in detail.


[W1] Limited Methodological Novelty

  1. DataMaster's contribution is a new formulation and system design for autonomous data engineering, rather than a new tree-search algorithm or memory primitive. We formulate data engineering as a constrained search over executable data states under a fixed learning algorithm, where data discovery, refinement, and downstream validation jointly drive optimization.

  2. Its components are specifically designed to address the structural challenges of this problem, rather than being a generic combination of existing techniques.

  • DataTree preserves alternative data-engineering trajectories;
  • Red/Black nodes separate open-world discovery from data refinement and downstream validation;
  • the shared Data Pool enables cross-branch data reuse;
  • and Global Memory accumulates outcomes and reusable findings across iterations.

Together, they establish a closed-loop process in which discovered data is refined, validated through downstream training, and used to guide subsequent search.


[W2] UCB Scheduling Mechanism

  1. Optimistic initialization of unvisited nodes. Assigning infinite priority to unvisited nodes is a standard optimistic initialization strategy in tree search, ensuring that every newly generated frontier node receives an initial evaluation. This is only a one-time initialization: after its first execution, the node is scheduled according to Eq. (2), where historical rewards and visit counts guide subsequent selection. Thus, reward-guided scheduling remains central throughout the search.

  2. Direction-aware reward normalization. We clarify that Black-node rewards must account for the direction of the underlying evaluation metric. We define a task-specific normalized utility y~v∈[0,1]\tilde y_v \in [0,1], such that higher values consistently indicate better performance, including for lower-is-better metrics. The immediate reward is defined as:

  • Successful Red node: rv=ϵr_v = \epsilon
  • Black node: rv=y~vr_v = \tilde y_v
  • Failed node: rv=0r_v = 0

This ensures that UCB consistently maximizes utility across tasks, regardless of their original metric directions.


[W3] Ablation Studies on PostTrainBench

We extend the component ablations to PostTrainBench to examine whether DataMaster's components contribute consistently in the open-world LLM post-training setting. Specifically, we evaluate all seven tasks under the same Qwen3-1.7B-Base initialization, training recipe, and compute budget.

SettingAIMEArenaHardBFCLGPQAGSM8KHealthBenchHumanEvalAvg.
w/o Red0.001.2026.0026.3441.7027.1035.9822.62
w/o Black0.001.6029.0027.2344.2029.8038.4124.32
w/o Memory0.001.6030.0028.5745.0030.6039.6325.06
Full DataMaster3.332.1934.2731.0249.4334.9343.2928.35

For w/o Red, we use a fixed candidate data pool; w/o Black disables task-specific data refinement while retaining necessary format adaptation; w/o Memory disables cross-node retrieval of historical outcomes and reusable findings.

These results isolate the contributions of external data discovery, data refinement, and cumulative memory in open-world LLM post-training, complementing the existing MLE-Bench Lite ablations.


[W4] Confounded Data-only Optimization

  1. Fixed learning algorithm with necessary configuration adjustments. DataMaster fixes the model architecture, training objective, and optimization algorithm. Limited adjustments to learning rate, batch size, and epochs are necessary to accommodate changes in dataset size and memory requirements; otherwise, training may become unstable, fail to converge, or encounter out-of-memory errors.

  2. Empirical hyperparameter adjustment statistics. We audit Black-node execution traces and find that hyperparameter adjustments are infrequent:

HyperparameterModified NodesFraction
Learning Rate82.3%
Batch Size308.8%
Epochs4212.4%

Learning-rate adjustments are particularly rare, while batch-size and epoch adjustments primarily accommodate data-volume and memory constraints. These statistics indicate that DataMaster's primary search behavior remains data construction rather than hyperparameter optimization.

AuthorsOpenReview ↗

Response to Reviewer va8v [2/2]

[W5] High Computational Cost

  1. Cost-controlled evaluation. We additionally evaluate DataMaster with its API expenditure capped at $0.50, using the same 10-task MLE-Bench Lite subset:
MethodAPI CostAgent Tokens (K)Medal RateGold Rate
Claude Code$0.56634.550%30%
ML-Master 2.0$0.29462.960%30%
DataMaster @ $0.50$0.50985.290%70%
DataMaster Full$1.613,286.690%80%

Reducing API expenditure by 68.9% preserves DataMaster's 90% Medal Rate and retains 70% Gold Rate, demonstrating that most of its gains can be achieved under a substantially smaller agent budget.

  1. Cost–performance tradeoff. The API cost represents only a small incremental overhead on top of downstream model training, whose compute budget is fixed across methods. DataMaster's full agent-side API cost is only $1.61 per task. We argue that minimizing agent expenditure is not an objective in itself: a modest additional investment in data discovery and validation is worthwhile when it yields better training data and higher downstream performance. Indeed, the full-budget configuration further improves Gold Medal Rate from 70% to 80% at the same 90% Medal Rate.

Finally: We really appreciate your insightful suggestion. We hope the above clarifications resolve the concerns. If the response is helpful, we would be grateful if you could consider adjusting your score accordingly.

Reviewer_1UroOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies task-conditioned autonomous data engineering: given a downstream task, a fixed learning algorithm, and a compute budget, the agent maximizes performance by acting only on the data side — dataset discovery, selection, composition, cleaning, transformation, and DataLoader construction (Eq. 1).

It proposes DataMaster, which casts this as tree search over executable data states: a DataTree of red nodes (dataset discovery, no training) and black nodes (compose/clean/adapt, then train for a score); a shared Data Pool of discovered-dataset manifests reusable across branches; and a Global Memory of node outcomes and failures, with a UCB1-inspired scheduler.

Validation covers two settings under a 12-hour per-task limit with GLM-5: MLE-Bench Lite (22 tasks), where medal rate rises 36.36% → 68.18% above all four baselines, and PostTrainBench (7 capabilities), where the macro-average rises 8.47% → 28.35% and GPQA slightly exceeds the Qwen3-1.7B-Instruct reference. Supporting analyses include a backbone sweep, a component ablation on 10 tasks, discovery/cost diagnostics, and a leakage audit.

Critical strengths

S1. The regime is well-chosen, and the red/black split answers its one hard constraint. In data-side optimization the only trustworthy signal (downstream training) is also the only expensive operation;DataMaster designs around exactly that asymmetry, making discovery cheap and shareable while confining scoring to black nodes. This is the part of the contribution that would survive even if the headline numbers did not.

S2. The memory ablation is a genuinely interesting result. Removing Global Memory collapses overcome rate 69.96% → 28.57% and best gold rate 80% → 20% (Table 2), pointing to cumulative cross-branch memory — not the tree — as the active ingredient.

S3. The discovery funnel is rarely-reported evidence. App. C.1 traces 1,344 queries → 233 unique datasets → 130 used → 20 retained in best data states, quantifying how lossy open-world data search is, independent of whether the gains hold.

S4. Cost is reported against the authors' own interest. Table 8 discloses 718.7 min / 2.69M input tokens for DataMaster vs. DatasetResearch's 71.7 min / 38.5K at 59.09% medal rate — creditable, even though it also undercuts the comparison (W2).

S5. Leakage is treated as a first-class risk. URL blocklists, test-hash dedup, and provenance records are built in, with audits on GPQA n-grams and all 22 MLE-Bench tasks (Table 7). Incomplete (W2), but reporting 1.06% 3-gram overlap rather than rounding to "no leakage" sets a better norm.

Critical weaknesses

W1. Every headline number is a single run, and the paper's own table shows noise as large as the claimed effects. No seeds, repetitions, or confidence intervals appear anywhere. The four rows of Table 2 should share an identical initial node, yet report initial medal 50/60/50/50 and gold 20/20/20/30 — a quantity that ought to be constant moves 10 pp, establishing ≥10 pp single-run variation on this 10-task set and meaning the reported gains (+30/+10/+0/+40 pp) start from different baselines. "Only the full system reaches 90%" is then a one-task difference inside demonstrated noise. Likewise the flagship claim: 31.02% vs. 30.35% on GPQA is 0.67 pp over 448 questions, inside one standard error. With single runs I cannot separate any component, backbone, or initialization conclusion from noise — which is most of the empirical content.

W2. The headline comparisons are confounded in ways that plausibly account for the entire gain.

  • Not compute-matched. Table 8: DataMaster 718.7 min / 2,690K input tokens vs. Claude Code 127.6 / 522K, Codex 95.4 / 97K, DatasetResearch 71.7 / 38.5K — 6–10× wall-clock, up to 70× tokens, no iso-compute condition. The baselines also look mis-run: Claude Code lands exactly on the initial score (zero net gain in 12 h) and Codex scores below doing nothing.
  • No real PostTrainBench control. "Initial Score" is the base model with no fine-tuning, so +19.88 pp largely measures that SFT happened; no off-the-shelf-mixture SFT baseline exists. The Codex row is copied from the benchmark paper under GPT-5.2-Codex while DataMaster runs GLM-5.
  • The "fixed algorithm" premise leaks. Table 5 lets black nodes tune learning rate, batch size, epochs, with no ablation separating curation from hyperparameter search — while Fig. 2b shows GPQA improving by adding MMLU science subsets, then MMLU-Pro + MedQA, eval-adjacent data the §4.4 safeguards cannot detect, and the audit covers GPQA only, not GSM8K/HumanEval/HealthBench where gains are largest.

So the central claim — matching expert-curated post-training data — rests on a 0.67 pp margin from a search that converged on benchmark-adjacent corpora, against baselines given a fraction of the compute.

Questions for authors
  1. Why do the four rows of Table 2 report different initial-node medal (50/60/50/50) and gold (20/20/20/30) rates, when the initial node should be identical across ablation variants? Is this run-to-run nondeterminism, and which setting (Full-code or Algo-only) was used?
  2. Can you report ≥3 seeds with per-task variance for Table 1 (at minimum MLE-Bench Lite medal rate) and Table 2? If initial-node spread is ~10 pp, what separates the 90% vs. 80% full-vs-no-red conclusion from noise?
  3. Can you provide a compute-matched comparison — baselines run to the same ~719 min and token budget as DataMaster? And why does Claude Code land exactly on the initial score while Codex falls below it; were those runs verified to execute to budget?
  4. Can you add the missing PostTrainBench control: SFT of Qwen3-1.7B-Base on a standard public instruction mixture under the same recipe? How much of 8.47% → 28.35% survives it?
  5. Holding the curated dataset fixed, what gain comes from the learning-rate / batch-size / epoch freedom granted in Table 5? Without this, how should the "fixed learning algorithm" claim be read?
  6. Can the leakage audit be extended to GSM8K, HumanEval, HealthBench Easy, and BFCL, and can you list the 20 datasets retained in the best data states (App. C.1) with their licenses? Do you consider training on eval-adjacent corpora (MMLU science subsets, MMLU-Pro, MedQA per Fig. 2b) in scope for your safeguards?
  7. The tree and the UCB scheduler are never ablated. How does DataMaster compare against random node selection, pure greedy, and a flat loop with the same Data Pool and Memory but no tree? Since red nodes receive a constant pseudo-reward ε, what do the statistics in Eq. 2 estimate?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 1Uro [1/2]

We sincerely appreciate your time and thoughtful comments. Below, we respond to each point in detail.


[W1 / Q1 / Q2] Experimental Stability and Initialization Consistency

We provide additional evidence through expanded ablations, repeated independent runs, and evaluation-stability analysis.

  1. Initialization consistency. As specified in Section 4.3, Table 2 reports component ablations on 10 MLE-Bench Lite tasks. We acknowledge that the initial-node results vary across variants (50–60% Medal Rate and 20–30% Gold Rate). We will clarify the original initialization procedure and the Full-code/Algo-only setting. Our expanded ablation uses matched initial nodes and evaluation protocols to eliminate initialization differences as a confounding factor.

  2. Expanded component ablations. We extend the original 10-task ablation to all 22 MLE-Bench Lite tasks, reducing sensitivity to individual task outcomes.

SettingOriginal 10 Tasks (Medal / Gold)Full 22 Tasks (Medal / Gold)
w/o Red80% / 70%59.09% / 36.36%
w/o Black70% / 50%54.55% / 31.82%
w/o Memory50% / 20%45.45% / 22.73%
Full DataMaster90% / 80%68.18% / 45.45%

Results show that Full DataMaster outperforms w/o Red, w/o Black, and w/o Memory by 9.09, 13.63, and 22.73 percentage points in Medal Rate, respectively. The component contributions therefore persist on the larger task set, supporting the complementary roles of data discovery, refinement, and Global Memory.

  1. Stability across independent runs. We additionally evaluate the complete DataMaster pipeline across three independent runs:
BenchmarkMetricMean ± Std
MLE-Bench Lite (10 tasks)Medal Rate86.67 ± 5.77%
MLE-Bench Lite (10 tasks)Gold Medal Rate70.00 ± 0.00%
PostTrainBenchAverage Score30.47 ± 0.60%

These results suggest that DataMaster maintains competitive performance across independent search trajectories. The stable Gold Medal Rate and limited variation in PostTrainBench provide complementary evidence of reproducibility.

4. GPQA evaluation stability. We further investigate the 0.67 pp difference between DataMaster (31.02%) and Qwen3-1.7B-Instruct (30.35%) by repeating evaluation with the same checkpoints and evaluation protocol.

ModelEval 1Eval 2Eval 3Mean ± Std
DataMaster31.02%31.47%31.25%31.25 ± 0.22%
Qwen3-1.7B-Instruct30.35%30.13%30.58%30.36 ± 0.22%

The illustrative evaluations yield a consistent numerical advantage for DataMaster, with an average difference of 0.89 percentage points. This suggests that the observed advantage is not solely driven by a particular decoding run. We will additionally assess uncertainty over the 448 GPQA questions before making claims of statistical superiority.


[W2.1 / Q3] Budget Fairness and Baseline Execution

1. Additional cost-controlled comparison. We evaluate DataMaster under a reduced $0.50 API-cost budget and compare it with GLM-5-based baselines on the same 10-task MLE-Bench Lite subset:

MethodAPI CostAgent Tokens (K)Medal RateGold Rate
Claude Code$0.56634.550%30%
ML-Master 2.0$0.29462.960%30%
DataMaster @ $0.50$0.50985.290%70%
DataMaster Full$1.613,286.690%80%

Reducing API expenditure by 68.9% preserves DataMaster's 90% Medal Rate, while Gold Rate decreases by only 10 pp. Under comparable API expenditure, DataMaster outperforms Claude Code by 40 pp and ML-Master 2.0 by 30 pp in Medal Rate.

2. Updated CLI baselines and execution behavior. We additionally examine newer CLI configurations on PostTrainBench (Qwen3-1.7B):

AgentBackboneScoreSource
DataMasterGLM-528.35%Our paper
Claude CodeGLM-5.228.5%Official PostTrainBench
Codex CLIGLM-5.227.1%Our new evaluation

All methods in our original experiments were allocated the same 12-hour end-to-end limit and downstream training-compute budget. However, the realized execution times differed across agents. The official PostTrainBench project also documents premature termination in Codex CLI and introduces reprompted configurations to encourage agents to utilize their remaining budget.

These observations highlight the sensitivity of baseline performance and resource utilization to CLI versions and execution policies. We will clarify the original CLI versions and termination conditions, while distinguishing API-cost-controlled comparisons from wall-clock- and token-matched comparisons.

AuthorsOpenReview ↗

Response to Reviewer 1Uro [2/2]

[W2.2/Q4] Contribution Beyond Standard SFT

  1. Controlled raw-data SFT baseline. To isolate the contribution of autonomous data curation from supervised fine-tuning itself, we conduct an additional control using the original data sources associated with DataMaster's best-performing checkpoint. We train Qwen3-1.7B-Base on the same volume of raw data, without DataMaster's task-specific filtering, selection, and transformation.
MethodTraining DataAvg. Score
Qwen3-1.7B-BaseNo SFT8.47%
Raw-Data SFTSame sources and data size20.40%
DataMasterAutonomously curated data28.35%
Qwen3-1.7B-InstructOfficial post-training46.91%

DataMaster improves upon Raw-Data SFT by 7.95 percentage points, demonstrating that the performance gain cannot be explained by SFT alone or simply acquiring the underlying data sources.

  1. Analysis of curation benefits. Our analysis identifies two major limitations of training directly on the raw data:
  • Format alignment: Raw data often lacks the input/output structure required by downstream tasks, whereas DataMaster transforms examples into task-compatible training formats.
  • Data selection: Raw datasets contain irrelevant or low-quality examples, while DataMaster filters and selects training samples based on task requirements and downstream feedback.

[W2.3 / Q5] Fixed Learning Algorithm and Hyperparameter Adjustments

  1. DataMaster strictly fixes the model architecture, training objective, and optimization algorithm. Limited hyperparameter adjustments are necessary to accommodate changing dataset sizes and memory constraints; otherwise, training may become unstable, fail to converge, or encounter out-of-memory errors.

  2. We audit Black-node execution traces and find that training hyperparameter adjustments are rare:

HyperparameterNodes ModifiedFraction
Learning Rate82.3%
Batch Size308.7%
Epochs/Iterations4212.4%
  • Learning rate is modified in only 2.3% of Black nodes, while the limited batch-size and epoch adjustments primarily accommodate changes in dataset volume and training feasibility. This indicates that DataMaster's search is predominantly focused on data construction rather than hyperparameter optimization.

[W2.4 / Q6] Data Contamination and Dataset Provenance

  1. Contamination filtering covers all training data, not only GPQA. As described in §4.4, all candidate datasets pass through an independent, agent-invisible contamination filter before entering training. The filter performs exact and fuzzy matching against protected evaluation samples. GPQA was presented as a representative quantitative audit rather than the only benchmark covered by these safeguards.

  2. Extended contamination audit. We additionally report the contamination statistics across all seven PostTrainBench benchmarks:

BenchmarkExact MatchFuzzy Match (≥90%)5-gram Overlap
GPQA00.00%0.08%
GSM8K00.00%0.04%
HumanEval00.00%0.08%
HealthBench Easy00.00%0.07%
BFCL00.00%0.01%
AIME 202500.00%0.03%
ArenaHard Writing00.00%0.06%

The same screening procedure applies to benchmark-adjacent sources, including MMLU science, MMLU-Pro, and MedQA. These additional results quantify residual overlap in the training data admitted by the filtering pipeline.

  1. Appendix reference. We could not locate the referenced Appendix C.1 or the list of 20 retained datasets in the submitted manuscript. Appendix C covers search-cost and tool-use diagnostics and is followed directly by Appendix D.

[Q7] DataTree and UCB Scheduling

  1. Search strategy ablations. We compare UCB against alternative scheduling strategies and remove the tree structure:
SettingMedal RateGold Rate
Round-robin50%40%
Random50%50%
Flat search (w/o Tree)70%70%
DataMaster (UCB + DataTree)90%80%

DataMaster achieves the best results, demonstrating the benefits of adaptive scheduling and tree-structured search.

  1. UCB reward interpretation. In Eq. (2), Rv/NvR_v/N_v estimates the empirical average reward accumulated by branch vv, while the second term ctlog⁡NPar(v)/Nvc_t\sqrt{\log N_{\mathrm{Par}(v)}/N_v} encourages exploration of less-visited branches. Successful Red nodes receive a small pseudo-reward (ϵ=0.01\epsilon=0.01), Black nodes receive actual downstream evaluation scores (yvy_v), and failed nodes receive zero. Both rewards and visit counts are propagated along the ancestor path. Thus, Red nodes receive limited credit for successful discovery, while Black-node evaluations provide downstream performance feedback.

Finally: We really appreciate your insightful suggestion. We hope the above clarifications resolve the concerns. If the response is helpful, we would be grateful if you could consider adjusting your score accordingly.

Reviewer_1G4qOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper formalizes task-conditioned autonomous data engineering as a constrained optimization problem in which an agent, holding the learning algorithm fixed, searches over executable “data states” (sources, subsets, cleaning and transformation logic) to maximize downstream performance under a budget. The authors propose DataMaster, which organizes the process as a DataTree, where one type of node discover external datasets and another node curate existing dataset.

Critical strengths

Data curation is the upstream of the entire model training cycle, yet there's no "science" for it. Agent seems to be very suitable for this task given the heuristic, trial-and-error nature of this task. Furthermore, the paper incorporates dataset finding into the loop.

Critical weaknesses

One of the main issue for agent-based data recipe optimization is that, agent often prefers tuning the hyperparameters for existing data curation methods instead of exploring new data curation techniques. The paper’s search machinery addresses where to look for data, but offers no mechanism regarding whether the agent explores the space of curation methods.

Questions for authors

Could the authors characterize the distribution of black-node operations across runs? Specifically, what fraction introduce a new curation method versus adjust parameters of an existing pipeline?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 1G4q

We sincerely appreciate your time and thoughtful comments. Below, we respond to each point in detail.


[W1] Lack of explicit exploration of data curation methods

  1. DataMaster supports open-ended exploration of curation methods rather than restricting Black nodes to predefined pipelines. Black nodes can autonomously implement new data-processing strategies through executable code, including cleaning, selection, transformation, sampling, composition, and feature construction. DataTree enables branch-wise exploration of alternative strategies, while Global Memory preserves previous implementations and evaluation feedback to guide subsequent exploration and refinement.

  2. Our trace-level audit further confirms the diversity of Black-node operations:

OperationNodesImproving RateGain (Δ)
Cleaning / Transformation25432.3%+0.61
Selection32232.0%+0.44
Sampling / Balancing31421.7%+0.36
Composition31215.4%+0.31
Feature Construction11430.7%−0.21

These results show that Black nodes explore diverse data-construction operations, with cleaning/transformation and selection yielding the strongest average improvements. While DataMaster does not explicitly reward method novelty, its open-ended action space and branch-wise search enable exploration beyond a fixed curation pipeline.


[Q1] Distribution of method exploration versus parameter tuning

To directly quantify methodological exploration, we classify Black-node transitions based on their executable code and configuration changes into three categories:

Transition TypeFractionImproving Rate
New Method Exploration38.6%34.2%
Method Refinement36.8%29.7%
Parameter Tuning24.6%21.5%

Method exploration and refinement account for 75.4% of Black-node transitions, compared with only 24.6% devoted to parameter tuning. Moreover, new-method exploration achieves the highest improving rate (34.2%), exceeding parameter tuning by 12.7 percentage points. These results demonstrate that Black nodes actively explore and improve curation strategies rather than predominantly tuning existing pipelines.


Finally: We really appreciate your insightful suggestion. We hope the above clarifications resolve the concerns. If the response is helpful, we would be grateful if you could consider adjusting your score accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.