acceptodds
Under review as a conference paper at ICLR 2027

MATAR: Multi-Agent Task-Adaptive Refinement for Multimodal Data

Abstract

The utility of multimodal data depends on the downstream task, yet data preparation often relies on generic quality criteria, sample selection, or fixed refinement procedures. These approaches leave limited scope to adapt record attributes and inter-record relations to task requirements. We propose , a task-adaptive multi-agent framework that uses downstream evidence to optimize executable refinement programs. Attribute and relation programs are searched independently, with task feedback selecting one final output. Task-adaptive tool optimization revises program structure and parameters and generates validated numerical feature operations when existing tools are insufficient. Paired replay evaluation compares revisions under matched training conditions, while experiential policy adaptation uses recent outcomes and an online controller to guide subsequent proposals with frozen LLM parameters. Across eight datasets spanning four downstream tasks, achieves an average relative gain of 23.9% over the strongest baseline per dataset under the evaluated protocols. Further experiments support the contributions of the three components and show that experience encoded in refinement programs can benefit other datasets. The findings suggest that effective data refinement should consider both attribute calibration and relational structure, with task-specific evaluation guiding their use and transfer.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Data Governance, Task Adaptation, Multi-Agent Systems

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_aWt4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MATAR, a multiagent framework that searches for data refinement programs using downstream task feedback. It separately explores changes to multimodal features and relations between records, combining paired evaluation with experience reuse and an adaptive controller. The framework is evaluated on eight datasets across four tasks, with additional ablation, transfer, and robustness experiments.

Critical strengths
  1. Useful task driven formulation. The paper considers both feature transformations and relations between records as editable parts of data preparation. Evaluating these changes through downstream performance provides a practical way to assess their utility for a particular task.
  2. Explicit evaluation safeguards. The framework uses paired training conditions, separate Search, Commit, and Test stages, and a fallback to unchanged inputs. These are sensible measures for controlling variation in candidate comparisons and checking whether selected refinements provide an improvement.
Critical weaknesses
  1. The MM metric does not adequately establish improved matching. The primary MM metric measures only the cosine similarity of paired representations, which can increase without improving discrimination between correct and incorrect pairs. Although training uses a contrastive objective and Appendix C mentions nonpaired similarity and pair margin diagnostics, their results are not reported. Appendix E.2 also acknowledges that representation dispersion should be examined before inferring improved alignment. Since MM accounts for two of the eight main results and some of the largest relative gains, this limitation weakens the evidence supporting the overall improvement.

  2. Incomplete baseline specifications and insufficiently controlled comparisons. Appendix B.3 explicitly states that complete baseline adapter specifications and tuning grids remain to be finalized, despite substantial adaptations of the original methods. MATAR also searches up to 36 candidate slots, while comparable total search budgets are not established for the baselines. Comparisons with simpler search methods under the same budget are needed to determine the contribution of the proposed framework. The ablations do not fully isolate its mechanisms either: w/o EPA retains the numerical controller, so it does not establish the benefit of controller learning.

Questions for authors

See weaknesses.

3 comments

Thank you for engaging so carefully with our evaluation and for recognizing the safeguards in the Search–Commit–Test protocol. We are particularly grateful for your scrutiny of matching discrimination, baseline specifications, and controller learning. These comments have guided additional experiments and helped us assess more precisely what the results establish.

W1: Your careful reading has highlighted where the problem formulation needs a clearer explanation. We agree that the transition from multimodal data preparation to graph refinement is abrupt and that the exposition needs more concrete explanation. In the revision, we will clarify the intended graph-based learning setting in the abstract and introduction, explain when inter-record relations are meaningful, and illustrate how attribute refinement and relation construction affect downstream learning with concrete examples. These changes will make the motivation and problem formulation easier to follow.

W2/Q1: We appreciate the opportunity to explain the conditions underlying this comparison. MATAR incurs greater exploration cost than the adapted fixed-workflow baselines. Its downstream training cost remains comparable to the search-based controls: on Food, Full uses 125 Search+Commit fits vs 111–135 for the controls, achieving MRR of 0.4768. We distinguish two phases. Iterative discovery: a complete Music (MR) Full run takes 508.35 s, including startup/loading, with 35 Search and 10 Commit fits, 4,050 optimizer steps, and 67 LLM calls. Peak per-fit GPU allocation is 2.35 GiB. Rule deployment: iterative search and textual memory guide the discovery of useful refinements, which are saved as executable programs. Applying these rules requires no further LLM calls or search-stage model fitting. Loading and applying the Commit-accepted Music attribute program to 6,038 records takes 372.85 ms on average over five repeats: 0.34 ms for rule loading/registry setup and 372.52 ms for application. First application takes 417.48 ms, with peak incremental memory of 107.9 MiB. Meanwhile, DataFlow requires an average of 97.7s for an end-to-end process with 1.82GB peak usage of cpu memory. These deployment timings exclude startup, input-data loading, output saving, and downstream training. Thus, exploration-only comparisons capture discovery efficiency but omit the value of subsequent reuse. We will report both phases separately, including amortized cost when rules are reused, and apply the same accounting to reusable baseline outputs.

Reviewer_aWt4OpenReview ↗

Response to author

Thank you for the response and additional clarifications. I appreciate the authors' efforts to address my concerns.

While the discussion of computational costs and rule deployment is helpful, my main concerns regarding the MM evaluation metric, the fairness and reproducibility of baseline comparisons, and the isolation of the controller's contribution remain insufficiently addressed.

Therefore, I will maintain my original assessment.

We are grateful to the reviewer for these constructive comments. In what follows, we address each of these points.

  1. Multimodal (MM) evaluation metrics. We thank the reviewer for pointing out issues with our MM evaluation metrics. In the original manuscript, we used the average cosine similarity between paired text and image representations. Consequently, we have adopted bidirectional Mean Reciprocal Rank (MRR) as our primary matching metric, which evaluates matching quality based on the rank of the correct pair within a candidate set; we have retained average cosine similarity as a supplementary metric. We have already conducted the necessary additional evaluations. As re-running all methods, replicating experiments, and verifying results under a unified MRR-based evaluation and model selection protocol requires significant time, we commit to including a comprehensive table of experimental results—along with the corresponding configuration details—in the next revision of the paper, and to updating our conclusions based on these revised findings.

  2. Fairness of baseline comparisons. All methods share the evaluation records and candidate sets, input-access permissions, downstream architecture, per-fit training budget, and training-seed protocol. Selection methods act on training supervision, while workflow methods modify the data products used for downstream learning. These comparisons therefore evaluate the declared task-specific adaptations under a common downstream protocol.

  3. Reproducibility. We have audited the saved results and execution manifests for all 72 baseline entries across nine methods and eight datasets. The consolidated records include resolved runtime parameters, evaluator configurations, training seeds and candidate-group information. Reporting these resolved configurations is essential because the runner can override nominal configuration values. We will incorporate this information into the reproducibility documentation and specify each method's intervention, feedback access, and actual evaluation budget.

  4. Isolation of the controller’s contribution. We added a targeted ablation that freezes actor and value updates while retaining semantic experience, tool generation, the evaluator, and proposal budgets. This directly tests numerical controller learning within the retained framework. On Food (MM), Full achieves Test MRR of 47.68, compared with 44.11 for the frozen controller, an absolute improvement of \3.56%. Each condition includes three independent search repetitions, each evaluated with five final training seeds. Full uses 125 Search/Commit fits, compared with 126 for the frozen controller, so this improvement accompanies nearly identical downstream-fit counts. This experiment supports an incremental contribution from online controller learning on Food. It also complements the original w/o EPA ablation, which removes interpreted experience reuse while retaining numerical controller updates.

Reviewer_VKRiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes MATAR, a multi-agent framework for task-adaptive multimodal data refinement. It searches over executable processing programs that modify either data attributes or graph relations, allowing changes to operations, parameters, and ordering. The two refinement paths are searched independently, with candidate programs evaluated through downstream MGAT performance. The framework combines task-adaptive program search, paired refinement evaluation, and experience feedback. Experiments cover eight datasets and four task types from OpenMAG, with comparisons against nine adapted baselines, component ablations, and cross-dataset transfer of frozen processing programs.

Critical strengths
  1. The paper propose a useful formulation of task-driven data preparation. It supports both attribute transformation and relation construction for multimodal graph learning.
  2. The main experiments (Table 1) report improvements across 8 datasets and 4 task types. Ablations provide evidence for the effectiveness of the framework (TATO, PRE, EPA). Transfer expeirments suggest that selected processing programs can benefit other datasets.
Critical weaknesses
  1. Abrupt problem framing and insufficiently concrete exposition. The abstract and introduction frame the problem broadly as multimodal data preparation, but quickly make inter-record relations and graph refinement central without sufficiently explaining when such relations are meaningful or why graph-based learning is the intended setting.
  2. The methodological contribution over existing workflow optimization is not sufficiently isolated. Task-guided data optimization, executable pipeline generation, and iterative experience feedback have precedents in prior work. Comparisons with matched operator spaces, downstream feedback, and total evaluation budgets would strengthen the attribution of gains to the proposed method.
Questions for authors
  1. How closely are MATAR and the adapted DataFlow baseline matched in their available operators, program expressiveness, and total downstream evaluation budgets? Would MATAR still has its advantage over a simpler search procedure or single-agent implementation given the same operators and feedback?
1 comment

We sincerely thank you for your careful reading and thoughtful assessment of our work. Your comments on the problem framing and the basis for comparison have prompted us to examine how we present and support our contribution. We value the attention you have given to these issues and appreciate the opportunity to address them through clearer explanations and additional evidence.

W1: Your careful reading has highlighted where the problem formulation needs a clearer explanation. We agree that the transition from multimodal data preparation to graph refinement is abrupt and that the exposition needs more concrete explanation. In the revision, we will clarify the intended graph-based learning setting in the abstract and introduction, explain when inter-record relations are meaningful, and illustrate how attribute refinement and relation construction affect downstream learning with concrete examples. These changes will make the motivation and problem formulation easier to follow.

W2/Q1: We appreciate the opportunity to explain the conditions underlying this comparison. MATAR incurs greater exploration cost than the adapted fixed-workflow baselines. Its downstream training cost remains comparable to the search-based controls: on Food, Full uses 125 Search+Commit fits vs 111–135 for the controls, achieving MRR of 0.4768. We distinguish two phases. Iterative discovery: a complete Music (MR) Full run takes 508.35 s, including startup/loading, with 35 Search and 10 Commit fits, 4,050 optimizer steps, and 67 LLM calls. Peak per-fit GPU allocation is 2.35 GiB. Rule deployment: iterative search and textual memory guide the discovery of useful refinements, which are saved as executable programs. Applying these rules requires no further LLM calls or search-stage model fitting. Loading and applying the Commit-accepted Music attribute program to 6,038 records takes 372.85 ms on average over five repeats: 0.34 ms for rule loading/registry setup and 372.52 ms for application. First application takes 417.48 ms, with peak incremental memory of 107.9 MiB. Meanwhile, DataFlow requires an average of 97.7s for an end-to-end process with 1.82GB peak usage of cpu memory. These deployment timings exclude startup, input-data loading, output saving, and downstream training. Thus, exploration-only comparisons capture discovery efficiency but omit the value of subsequent reuse. We will report both phases separately, including amortized cost when rules are reused, and apply the same accounting to reusable baseline outputs.

Reviewer_f42BOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MATAR, a task-adaptive multi-agent framework for multimodal data optimization. The method represents data processing as executable programs and separately searches and optimizes data features and relational structures. The authors conduct experiments on four types of downstream tasks and eight datasets, report an average relative performance improvement of 23.9%, and validate the effectiveness of the method through ablation studies, cross-dataset transfer, and robustness experiments.

Critical strengths

S1. The research question is valuable.

This paper explores whether a multi-agent system can continuously improve its own data processing strategies using execution feedback and historical experience, which is a meaningful research direction. Compared with using a fixed data processing pipeline, allowing models to autonomously identify problems, modify processing programs, and accumulate experience based on downstream task performance has certain research value and practical application potential.

S2. The experimental coverage is relatively comprehensive.

This paper evaluates the method on four different downstream tasks and eight datasets, and provides module ablation, hyperparameter analysis, cross-dataset transfer, and noise robustness experiments. These experiments demonstrate the methodʼs performance from different perspectives; in particular, the cross-dataset transfer experiments preliminarily reflect the reuse potential of the learned data processing programs.

Critical weaknesses

W1. The source of the performance gains remains insufficiently clear.

MATAR can not only add new features and construct neighbor graphs, but also combine multiple processing operations. In contrast, the various sample filtering methods in the comparison mainly adjust only which samples are used for training and their weights. Therefore, MATARʼs advantage may partly come from these additional data-processing capabilities rather than from its core mechanism. For example, the non-link prediction task in the paper initially has no input relation graph; in this case, simply adding a neighbor graph may already improve MGATʼs performance. It is recommended to add control experiments with simple graph construction and feature processing, and to let random search or a simple single-agent method use the same processing tools as MATAR while receiving a comparable model training budget. Only in this way can it be determined whether MATAR can still find better results when other methods are also allowed to adopt the same data-processing schemes.

Questions for authors

Q1. MATAR involves multiple rounds of program generation, data processing, and downstream model evaluation. What are the computational and time overheads of this pipeline? Could you report the overall and component-wise running costs, as well as the processing cost of applying the generated data processing programs to datasets of different scales?

Q2. The paper achieves continuous strategy improvement through historical experience and an online controller, but it currently lacks direct analysis of this process. Could you show how performance changes over iterations during the search, and separately validate the contribution of the online controller?

Q3. Can this framework handle the situation well when the task definition changes from a single task to multiple tasks? Does this mean that we need to pre-split the task input or can the agent complete it well in one go? Q4. MATAR relies on LLMs to generate and modify data processing tools, but the capabilities of these tools themselves may limit the final data optimization performance. Can the current experiments demonstrate the capability ceiling of this automatic improvement mechanism? For example, when existing tools cannot effectively handle a certain type of data problem, can the model generate better tools through feedback, or will it be constrained by the capabilities of the initial model and tool library?

1 comment

Thank you for the time and care you devoted to reviewing our work. We appreciate your recognition of the research question and the breadth of our evaluation. Your questions about the sources of improvement, computational cost, and tool refinement have guided our additional analyses and helped us explain the framework more clearly.

W1: We appreciate your suggestions on clarifying the source of the gains. On Cartoon (MM), Test MRR is 8.32% for Preserve, 11.05% for a feature-free random ring, 18.69% for untuned equal-weight text/image fused 1-NN, and 45.79% for Full MATAR. Adding MM-Gen feature processing to the same fused graph gives 19.47%. Full thus improves over these graph and feature-processing controls by at least 26.31% MRR.

Q1: We are pleased that we can have the opportunity to clarify the practical cost. We report two cost phases. Iterative discovery: on Music (MR), a complete Full run takes 508.35 s including startup/loading: Search takes 454.89 s (attribute/relation: 237.26/216.36 s), and Commit/Test 47.58 s after loading. It uses 35 Search and 10 Commit model fits, 4,050 optimizer steps, and 67 LLM calls (482,771 input/33,418 output tokens); Test restores five checkpoints. Peak per-fit GPU allocation is 2.35 GiB. Search cost is O(P C_apply + F C_fit + C_LLM), for P program evaluations and F downstream fits. For N records of width d, q bounded numerical transformations cost O(qNd) time and O(Nd) memory; each exact k-NN construction costs O(N²d) time, with blockwise memory O(Nd+bN+Nk), where b=256 and k≤32. Rule deployment: selected rules are saved as executable programs; applying them requires no LLM calls or search-stage model fitting. The Commit-accepted Music attribute-path program (6,038 records) takes 372.85 ms : rule loading/registry setup 0.34 ms, application 372.52 ms. First application: 417.48 ms and peak incremental memory: 107.9 MiB. We also report search progress. Summing the retained-program Search MRR gains over Preserve from the attribute and relation paths gives 33.12, 40.50, 41.64, and 42.42 MRR points on Cartoon at iterations 1, 3, 6, and 9, respectively. These summed path gains describe continued candidate improvement during Search, with additional gains after the initial iterations.

Q2: We are grateful for your interest in the refinement process. To isolate online-controller learning, we freeze only actor/value updates while retaining semantic experience, tool generation, the evaluator, and proposal budgets. On Food (MM), Full versus frozen-controller Test MRR is 0.4768 vs 0.4411, a mean gain of 3.56%. Their Search/Commit fits are 95/30 versus 96/30. Full MATAR vs frozen-controller obtains 0.4768 vs 0.4411 Test MRR on Food (MM), and 41.89 vs 40.10 Macro-F1 on Movies (EC), +1.79% on Movies.

Q3: It is a pleasure to discuss this broader application setting. MATAR requires separate runs for multiple tasks; its agents cannot automatically decompose and jointly optimize a multi-task request. Users specify each task and evaluator beforehand and can reuse the same records without disjoint data subsets. Paired feedback ranks edits against the task objective, and Commit checks improvement on that objective, making selection directly responsive to the intended task. Joint optimization of conflicting objectives requires a weighting or constraint rule, which the current evaluator does not define. Therefore, in multi-task scenarios, it is still necessary to pre-split the task inputs. We will make further optimizations for the scenario you’ve described.

Q4: We value your question about extending the tool library. On DY (LP), MATAR generated a column-standardization/tanh residual to address unequal text-feature scales. In a search repetition, inserting this operation before an unchanged text-scale-0.5 step increases mean Search AUC from 72.764 to 74.008 (+1.244%). The generated tool remains in the selected program. This extends the initial library with a useful numerical transformation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.