- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces MATAR, a multiagent framework that searches for data refinement programs using downstream task feedback. It separately explores changes to multimodal features and relations between records, combining paired evaluation with experience reuse and an adaptive controller. The framework is evaluated on eight datasets across four tasks, with additional ablation, transfer, and robustness experiments.
Critical strengths
- Useful task driven formulation. The paper considers both feature transformations and relations between records as editable parts of data preparation. Evaluating these changes through downstream performance provides a practical way to assess their utility for a particular task.
- Explicit evaluation safeguards. The framework uses paired training conditions, separate Search, Commit, and Test stages, and a fallback to unchanged inputs. These are sensible measures for controlling variation in candidate comparisons and checking whether selected refinements provide an improvement.
Critical weaknesses
-
The MM metric does not adequately establish improved matching. The primary MM metric measures only the cosine similarity of paired representations, which can increase without improving discrimination between correct and incorrect pairs. Although training uses a contrastive objective and Appendix C mentions nonpaired similarity and pair margin diagnostics, their results are not reported. Appendix E.2 also acknowledges that representation dispersion should be examined before inferring improved alignment. Since MM accounts for two of the eight main results and some of the largest relative gains, this limitation weakens the evidence supporting the overall improvement.
-
Incomplete baseline specifications and insufficiently controlled comparisons. Appendix B.3 explicitly states that complete baseline adapter specifications and tuning grids remain to be finalized, despite substantial adaptations of the original methods. MATAR also searches up to 36 candidate slots, while comparable total search budgets are not established for the baselines. Comparisons with simpler search methods under the same budget are needed to determine the contribution of the proposed framework. The ablations do not fully isolate its mechanisms either: w/o EPA retains the numerical controller, so it does not establish the benefit of controller learning.
Questions for authors
See weaknesses.
3 comments
Response to author
Thank you for the response and additional clarifications. I appreciate the authors' efforts to address my concerns.
While the discussion of computational costs and rule deployment is helpful, my main concerns regarding the MM evaluation metric, the fairness and reproducibility of baseline comparisons, and the isolation of the controller's contribution remain insufficiently addressed.
Therefore, I will maintain my original assessment.
We are grateful to the reviewer for these constructive comments. In what follows, we address each of these points.
-
Multimodal (MM) evaluation metrics. We thank the reviewer for pointing out issues with our MM evaluation metrics. In the original manuscript, we used the average cosine similarity between paired text and image representations. Consequently, we have adopted bidirectional Mean Reciprocal Rank (MRR) as our primary matching metric, which evaluates matching quality based on the rank of the correct pair within a candidate set; we have retained average cosine similarity as a supplementary metric. We have already conducted the necessary additional evaluations. As re-running all methods, replicating experiments, and verifying results under a unified MRR-based evaluation and model selection protocol requires significant time, we commit to including a comprehensive table of experimental results—along with the corresponding configuration details—in the next revision of the paper, and to updating our conclusions based on these revised findings.
-
Fairness of baseline comparisons. All methods share the evaluation records and candidate sets, input-access permissions, downstream architecture, per-fit training budget, and training-seed protocol. Selection methods act on training supervision, while workflow methods modify the data products used for downstream learning. These comparisons therefore evaluate the declared task-specific adaptations under a common downstream protocol.
-
Reproducibility. We have audited the saved results and execution manifests for all 72 baseline entries across nine methods and eight datasets. The consolidated records include resolved runtime parameters, evaluator configurations, training seeds and candidate-group information. Reporting these resolved configurations is essential because the runner can override nominal configuration values. We will incorporate this information into the reproducibility documentation and specify each method's intervention, feedback access, and actual evaluation budget.
-
Isolation of the controller’s contribution. We added a targeted ablation that freezes actor and value updates while retaining semantic experience, tool generation, the evaluator, and proposal budgets. This directly tests numerical controller learning within the retained framework. On Food (MM), Full achieves Test MRR of 47.68, compared with 44.11 for the frozen controller, an absolute improvement of \3.56%. Each condition includes three independent search repetitions, each evaluated with five final training seeds. Full uses 125 Search/Commit fits, compared with 126 for the frozen controller, so this improvement accompanies nearly identical downstream-fit counts. This experiment supports an incremental contribution from online controller learning on Food. It also complements the original w/o EPA ablation, which removes interpreted experience reuse while retaining numerical controller updates.
Thank you for engaging so carefully with our evaluation and for recognizing the safeguards in the Search–Commit–Test protocol. We are particularly grateful for your scrutiny of matching discrimination, baseline specifications, and controller learning. These comments have guided additional experiments and helped us assess more precisely what the results establish.
W1: Your careful reading has highlighted where the problem formulation needs a clearer explanation. We agree that the transition from multimodal data preparation to graph refinement is abrupt and that the exposition needs more concrete explanation. In the revision, we will clarify the intended graph-based learning setting in the abstract and introduction, explain when inter-record relations are meaningful, and illustrate how attribute refinement and relation construction affect downstream learning with concrete examples. These changes will make the motivation and problem formulation easier to follow.
W2/Q1: We appreciate the opportunity to explain the conditions underlying this comparison. MATAR incurs greater exploration cost than the adapted fixed-workflow baselines. Its downstream training cost remains comparable to the search-based controls: on Food, Full uses 125 Search+Commit fits vs 111–135 for the controls, achieving MRR of 0.4768. We distinguish two phases. Iterative discovery: a complete Music (MR) Full run takes 508.35 s, including startup/loading, with 35 Search and 10 Commit fits, 4,050 optimizer steps, and 67 LLM calls. Peak per-fit GPU allocation is 2.35 GiB. Rule deployment: iterative search and textual memory guide the discovery of useful refinements, which are saved as executable programs. Applying these rules requires no further LLM calls or search-stage model fitting. Loading and applying the Commit-accepted Music attribute program to 6,038 records takes 372.85 ms on average over five repeats: 0.34 ms for rule loading/registry setup and 372.52 ms for application. First application takes 417.48 ms, with peak incremental memory of 107.9 MiB. Meanwhile, DataFlow requires an average of 97.7s for an end-to-end process with 1.82GB peak usage of cpu memory. These deployment timings exclude startup, input-data loading, output saving, and downstream training. Thus, exploration-only comparisons capture discovery efficiency but omit the value of subsequent reuse. We will report both phases separately, including amortized cost when rules are reused, and apply the same accounting to reusable baseline outputs.