acceptodds
Under review as a conference paper at ICLR 2027

Which Sources Should Participate? Target-Specific Multi-Source Configuration

Abstract

Multi-source learning often treats source participation as fixed, either using all available sources or ranking sources independently for a target. We instead study source participation as a target-specific configuration problem over non-empty subsets of complete source domains. Empirical analysis reveals three recurring patterns. Adding sources can reduce target performance even when more training data become available, near-tied validation configurations frequently reverse on test data, and larger configurations do not reliably outperform smaller ones when validation support is comparable. These observations motivate ARCS, an Admissible Risk-aware Configuration Selection rule that constructs a near-optimal validation set, resolves ambiguity through parsimony, and uses worst-class validation performance only as a localized safeguard. We evaluate ARCS across multiple real-world heterogeneous multi-source benchmarks spanning wearable sensing, clinical multi-site data, and multimodal sentiment analysis, where it remains competitive while selecting compact source configurations. Further analyses show that the main gain comes from validation admissibility combined with parsimony, that performance remains stable under reduced target evidence, and that approximate candidate search can reduce configuration evaluation without changing the final selection principle. Overall, our results suggest that source participation should be treated as a target-specific configuration decision rather than a fixed assumption in multi-source learning.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Multimodal learning, Heterogeneous multimodal data, Multi-source learning, Source configuration transfer, Target-dependent transfer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ePgQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary

This paper studies target-specific source selection in multi-source learning. In this paper, they propose the ARCS that selects a subset of source domains based on validation performance, class-level risk, and parsimony. Experiments on wearable sensing, clinical, and multimodal sentiment datasets show that ARCS can achieve competitive performance with fewer participating sources. Extensive results have verified the effectiveness of the proposed method.

Critical strengths

Strengths

  1. Pretraining with multi-source datasets is a popular paradigm, especially for the multi-modal datasets as mentioned in this paper. More training sources do not always lead to better performance, and selecting suitable sources for different targets is an important problem.
  2. The proposed method is simple and easy to understand.
  3. The experiments cover different domains and learning models.
Critical weaknesses

Weaknesses

  1. The experiments are limited and not so informative and readible. I guess a good way is to provide detailed numbers in appendix, and use more figrues, plus highlighting takeawyas in the experiment section.
  2. I think the proposed method is relatively simple, mainly combining validation-based selection with some heuristic rules. I wonder what is the key methodological novelty compared with existing source selection methods? Would it be possible to provide more analysis on why this particular combination is effective?
  3. I notice that ARCS needs to evaluate different source configurations before selection. However, the number of possible configurations grows exponentially with the number of sources. I wonder how practical this method would be when there are tens or hundreds of source domains? Would it be possible to provide more experiments on the computational cost and scalability?
  4. Is this generalize to pretraining? How to combine multi-source datasets is more siginificant in the pretraining/mid-training scenarios, rather than downstream applications. Related works [1-3] have discussed the scaling and mixed training of different multi-modal datasets, it's worth discussing here. This kind of limits the scope the paper, proper limitation/related work section can help maintain a supportive aspect.
  5. I notice that the improvements are not always consistent across datasets. For example, AllSource performs better on MOSEI, and the improvement over DATE on PPMI is not statistically significant. I wonder under what conditions ARCS is expected to provide clear benefits? It would be helpful to provide more analysis of when source selection is more useful than simply using all sources.
  6. How about other metrics? Since here you report the Macro-F1, it's depend on how you choose the threshold. For discriminative tasks, it's better to use AUROC and AUPRC.

[1] MegaScale-Data: Scaling Dataloader for Multisource Large Foundation Model Training. European Conference on Computer Systems'2026.

[2] Improving knowledge tracing through multi-source scaling with decoder-only transformers. ACM Transactions on Intelligent Systems and Technology'2026.

[2] OSF: On Pre-training and Scaling of Sleep Foundation Models. ICML'26.

Questions for authors

See weakness.

1 comment
AuthorsOpenReview ↗

Improved experimental presentation and readability, with additional AUROC/AP evaluation and clearer discussion of key findings

1. Experimental presentation and readability

We have replaced Figure 2 with Table 2 for clearer presentation. We have revised the experimental presentation to make the findings clearer and easier to follow. Specifically, we have expanded the experimental details in Section 6 and Appendices A–F. We have also made the key takeaways and their implications more explicit in the experimental analysis, highlighting what the results reveal about source configuration selection.

We also provide an anonymized repository at https://anonymous.4open.science/r/arcs-review-package-66CC/.

2. Methodological novelty and contribution

Our main contribution is a systematic investigation of target-specific source configuration selection, providing empirical insights into the challenges of selecting appropriate source combinations and how such decisions can be made. We examine source configuration utility, the reliability of target-validation evidence, and the trade-offs involved in selection decisions. Building on these findings, we introduce ARCS as an explicit selection rule and systematically evaluate its behavior across different benchmarks, learners, and selection criteria.

3. Computational cost and scalability

Appendix I reports an approximate-search experiment on PPMI using the four learners. At beam width B=5B=5, the number of evaluated configurations decreases by 21.33%, while ARCS matches exhaustive selection in 45 of 48 evaluation units (93.75%). These results demonstrate that ARCS can be combined with approximate search to reduce configuration evaluations while largely preserving exhaustive selection decisions. The overall computational cost depends not only on the number of evaluated configurations but also on the underlying learner, training data volume, and model training and evaluation costs. Scalability to substantially larger source pools remains an important direction for further evaluation.

4. Scope and relation to multi-source pretraining

Our study focuses on target-specific source configuration selection, where limited target-validation evidence is available to guide which sources should jointly participate in learning. This differs from large-scale pretraining and mid-training, where source mixing often involves data proportions and training-budget allocation without direct feedback from a specific downstream target. We have clarified this distinction and discussed the recommended three relevant works in the revised Related Work (Section 2) and Limitations section.

5. Source selection under different conditions

The varying results across benchmarks are an important part of our empirical findings. Our analyses across HAR4-Fixed, PPMI, and MOSEI show that source configuration utility depends on the target, participating sources, and underlying learner, while identifying beneficial configurations remains challenging under limited validation evidence. We have clarified these findings and their implications in the revised experimental analysis and Discussion.

6. Additional evaluation metrics

We have supplemented the Macro-F1 results with AUROC and average precision (AP) evaluations across HAR4-Fixed, PPMI, and MOSEI (Appendix G). ARCS achieves competitive AUROC and AP on HAR4-Fixed, closely approaching BestSingleton, while AllSource performs best on PPMI and MOSEI. These findings show that neither selecting fewer sources nor including all sources consistently yields the best predictive performance. Instead, source configuration utility varies across benchmarks, further highlighting the importance of target-specific source configuration selection.

MethodHAR4-Fixed AUROCHAR4-Fixed APPPMI AUROCPPMI APMOSEI AUROCMOSEI AP
AllSource0.89750.81840.98270.90310.69670.6789
BestSingleton0.92580.86400.97630.84960.66600.6544
ValidationBest0.92440.86120.97470.84280.68870.6724
DATE0.90730.82840.97340.85010.68260.6692
A2XP0.91870.85550.97970.88090.67600.6628
SEA/SUTE0.90270.82110.97490.85210.68460.6692
PANDA0.90750.83340.98030.87300.68630.6724
PreSTA0.89980.82660.94870.75610.65380.6378
ARCS (Ours)0.92570.86320.97250.83570.68710.6712
Reviewer_435iOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies target-specific source configuration selection. ARCS retains configurations with near-optimal validation scores, applies a worst-class performance filter, and favors smaller source subsets. Experiments cover activity recognition, clinical prediction, and multimodal sentiment analysis using five underlying learners.

Critical strengths
  1. The paper addresses a relevant problem: source utility can depend on both the target and the other participating sources.
  2. The fixed source-data budget in HAR4-Fixed helps distinguish source composition from data quantity.
  3. The evaluation covers multiple settings, and the authors acknowledge mixed results, modest risk-gate contributions, and approximate-search trade-offs.
Critical weaknesses
  1. ARCS combines validation tolerance, worst-class filtering, and parsimony. Its advantage over simpler alternatives, such as cardinality-penalized or uncertainty-aware validation selection, remains insufficiently established.

  2. Section 4 uses target-test behavior to motivate the rule, while Appendix F justifies ε = 0.01 through its highest reported Macro-F1. Whether these choices were fixed using independent development data needs clarification. Validation-only configuration selection does not resolve this concern at the method-development level.

  3. ValidationBest is absent from the PPMI and MOSEI comparisons. On HAR4-Fixed, ARCS improves Macro-F1 from 0.7422 to 0.7465, while the risk gate changes only 3 of 60 selections. Comparisons against ValidationBest and TargetOnly across benchmarks, with paired uncertainty estimates, are needed. Held-out worst-class results would also strengthen the risk-screening rationale.

  4. HAR feature and label construction, the PPMI prediction task, MOSEI grouping rules, and adaptations of the literature-based learners and selectors remain inadequately specified. References to a “canonical Setting B” or “tabular-core implementations” do not provide enough information to assess reproducibility and comparison fidelity.

  5. Approximate search is evaluated with only six sources. The tested beam setting achieving complete agreement with exhaustive selection still evaluates 60 of 63 configurations. Larger source pools and measured search and training costs would better establish practical scalability.

Questions for authors
  1. Were ε and the selection rule fixed independently of the reported target-test results?
  2. In the reduced-evidence experiment, which target splits are subsampled, and are candidate learners retrained?
1 comment
AuthorsOpenReview ↗

Additional experiments on selection rules, comparisons with simpler selection rules, additional baselines and held-out evaluation, with clarified research contributions and improved reproducibility

1. Choice of the admissibility tolerance

We selected ϵ=0.01\epsilon=0.01 during method development on HAR4-Fixed and subsequently applied the fixed rule to PPMI and MOSEI without benchmark-specific tuning. The results show that ϵ=0.01\epsilon=0.01 provides a competitive common operating point. This supports the use of a unified selection rule rather than choosing benchmark-specific parameters based on test performance.

2. Details of the reduced-validation-evidence experiment

We have clarified and extended the experiment in Section 6.3 and Appendix J. Only the target-validation partition is subsampled, while trained models, andidate predictions, and target-training/test partitions remain fixed. No learners are retrained. The revised experiment covers three benchmarks using LR, MLP64, and GGA, with four evidence levels and five resampling runs per unit. ARCS and ValidationBest use matched validation subsets, and the results reveal benchmark-dependent predictive performance and selection stability.

3. Comparisons with simpler selection rules

We have added comparisons with cardinality-penalized selection (λ=0.01\lambda=0.01) and two uncertainty-aware validation selection rules, One-SE and bootstrap LCB, using the final four learners across all three benchmarks (Appendix K). The results show that simpler rules can outperform ARCS in some settings, but none consistently dominates across benchmarks. Cardinality-penalized selection performs better on HAR4-Fixed and PPMI but worse on MOSEI, while One-SE shows a similar benchmark-dependent pattern. These findings further demonstrate that source configuration selection involves trade-offs between predictive performance and source participation, rather than a universally superior selection rule.

MethodHAR4-Fixed F1#Src.PPMI F1#Src.MOSEI F1#Src.
ValidationBest0.81901.790.68502.520.62911.92
Cardinality Penalty (λ=0.01\lambda=0.01)0.81981.420.68992.060.62451.67
One-SE0.82381.080.66971.210.61241.17
Bootstrap LCB0.81801.830.68702.460.62971.94
ARCS0.81871.630.68802.290.62971.83

4. Additional baselines and held-out evaluation

We have added matched TargetOnly and ValidationBest comparisons, paired uncertainty estimates, and held-out WorstClass analysis using the four learners across all three benchmarks (Appendix L). Both ARCS and ValidationBest achieve higher average Macro-F1 than TargetOnly. Their predictive performance is closely matched, with target-clustered paired bootstrap confidence intervals including zero across all three benchmarks. Held-out WorstClass analysis further shows benchmark-dependent effects, improving minimum-class F1 on MOSEI but reducing it on HAR4-Fixed. These findings provide additional evidence on the behavior and limitations of source configuration selection under different evaluation criteria.

BenchmarkTargetOnlyValidationBestARCSDifference (95% CI)
HAR4-Fixed0.75080.78470.7843-0.0004 [-0.0010, 0.0002]
PPMI0.57130.68500.6880+0.0030 [-0.0041, 0.0158]
MOSEI0.56400.62910.6297+0.0006 [-0.0011, 0.0016]

5. Benchmark construction and reproducibility

We have substantially revised Appendices A–F to clarify benchmark construction, data partitions, preprocessing, learner implementations, adapted selection strategies, and evaluation protocols.

We also provide an anonymized repository at https://anonymous.4open.science/r/arcs-review-package-66CC/.

6. Approximate search and computational cost

Appendix I reports an approximate-search experiment on PPMI using the four learners. At beam width B=5B=5, the number of evaluated configurations decreases by 21.33%, while ARCS matches exhaustive selection in 45 of 48 evaluation units (93.75%). These results demonstrate that ARCS can be combined with approximate search to reduce configuration evaluations while largely preserving exhaustive selection decisions. The overall computational cost depends not only on the number of evaluated configurations but also on the underlying learner, training data volume, and model training and evaluation costs. Scalability to substantially larger source pools remains an important direction for further evaluation.

Our contribution goes beyond proposing a selection rule by systematically investigating the challenges of identifying suitable source combinations and making reliable selection decisions with limited target-validation evidence. We hope the substantial improvements made in response to your feedback will encourage you to reconsider your assessment.

Reviewer_GkKaOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduce Admissible Risk-aware Configuration Selection (ARCS), a rule for selecting source configurations in multi-source settings for a given downstream task. Their work is motivated by their experiments which show that including more sources do not consistently improve performance. The empirical results (Tables 2 and 3) seem to support their method. The work is limited by insufficient reproducibility and its writing, which makes the paper hard to read and understand. Without a clear way for a reader to implement the method or the experiments, it’s difficult to assess the validity of the method. Additionally, it seems like the paper is missing the mandatory AI disclosure statement. Therefore, I recommend rejection for this manuscript.

Critical strengths
  • Motivation and observations. The authors outline their motivation based on experimentation (Section 4).
  • Empirical results. The results in Tables 2 and 3 seem to support the method outperforming prior work.
Critical weaknesses
  • Reproducibility. Difficulty in parsing the paper aside (see below), there are insufficient details for a reader to implement ARCS from Sections 3, 4, and 5. The same goes for reimplementing the experiments the authors ran (even with the details in Appendices A-F). The authors should refer to the Machine Learning Reproducibility Checklist and share an anonymized code repository.
  • Writing. I found the writing difficult to read. For example, even after reading Section 3, I am confused by the core definitions for the problem setting (e.g. what exactly is “completeness” for a source domain, what are the components of a configuration etc.).
Questions for authors

Please see weaknesses.

2 comments
AuthorsOpenReview ↗

Substantial manuscript revisions to improve clarity, experimental consistency, and reproducibility

We sincerely thank the reviewer for the thoughtful and constructive feedback. Motivated by these comments, we have undertaken substantial revisions throughout the paper, including a thorough improvement of the writing and presentation, as well as a comprehensive examination of experimental consistency and reproducibility. We have also added the Discussion section to synthesize our findings on source configuration utility, the reliability of target-validation evidence, and the trade-offs involved in selection decisions. These revisions go beyond addressing individual comments and have helped us strengthen the paper as a whole.

We would be very grateful if you could take the time to review our revised manuscript and we believe the manuscript has become substantially clearer, more rigorous, and more complete.

1. Writing and problem formulation

We have reorganized the Introduction to clarify the progression from the research problem and its distinction from related settings (Table 1), to empirical evidence of configuration-dependent source utility (Figure 1 and Table 2), and then to the challenges of selecting configurations using limited target-validation evidence (Section 4).

We have also revised the Related Work and Problem Formulation (Sections 2–3) to clarify the scope and core definitions. In particular, we removed the ambiguous notion of source "completeness" and explicitly define a source configuration as a non-empty subset of candidate source domains. Each source participates as a whole unit, rather than being assigned an individual weight or selected at the sample or modality level. Section 3 now specifies the source pool, target training and validation data, underlying learner, candidate models, and configuration selection procedure.

2. Experimental reproducibility

We have substantially expanded the experimental and supplementary details, including benchmark construction, data partitions, preprocessing, learner implementations, adapted comparison strategies, and evaluation protocols in Section 6 and Appendices A–F. We also clarified the ARCS implementation and conducted a comprehensive consistency audit of the four-learner, nine-selector evaluation, checking candidate configurations, selection behavior, prediction assets, and reported metrics. The revised manuscript explicitly distinguishes the motivation study, empirical observation analysis, and main evaluation, and documents relevant protocol and reproducibility limitations.

We also provide an anonymized repository at https://anonymous.4open.science/r/arcs-review-package-66CC/ containing the experimental code, configurations, and reproduction instructions.

3. AI disclosure and reproducibility statements

We have added dedicated AI Use Statement and Reproducibility Statement sections to the revised manuscript.

Our central contribution is not simply a new selection rule, but a systematic understanding of why selecting the right combination of sources is difficult and how such decisions can be made with limited validation evidence. We hope these substantial revisions will encourage the reviewer to reconsider their assessment.

Thank you to the authors for their comments to my review, the edits to their manuscript, and the anonymized code repository. Given that your response was submitted on the day of the rebuttal deadline, I have limited time and bandwidth to engage with the rewritten manuscript. My thoughts are below:

Writing and Problem Formulation The writing is still hard to parse: for example, “[w]e select source domains as whole units, meaning that each source is either included or excluded” seems like an unnecessarily complicated way of saying “We only use complete source domains.”

Code Repository. The README.md file from the Snapshot from Nov 17, 2026 mentions folders and files that do not exist. For example: example.py, examples/ and tests/ provide the runnable illustration. I could not find a examples/ or tests/ folder, both of which seem critical for actually running the experiments. (e.g. synthetic_validation.csv is under / whereas line 10 of example.py calls data = pd.read_csv(root / 'examples' / 'synthetic_validation.csv')).

AI Disclosure. Thank you for including the AI Disclosure and Reproducibility Statement. However, I still find the manuscript hard to read and reproduce (see above).

Given the late rebuttal, the lack of substantial clarity from a quick read of the updated manuscript, and the irreproducible code repository, I will maintain my score of Clear Rejection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.