- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Summary
This paper studies target-specific source selection in multi-source learning. In this paper, they propose the ARCS that selects a subset of source domains based on validation performance, class-level risk, and parsimony. Experiments on wearable sensing, clinical, and multimodal sentiment datasets show that ARCS can achieve competitive performance with fewer participating sources. Extensive results have verified the effectiveness of the proposed method.
Critical strengths
Strengths
- Pretraining with multi-source datasets is a popular paradigm, especially for the multi-modal datasets as mentioned in this paper. More training sources do not always lead to better performance, and selecting suitable sources for different targets is an important problem.
- The proposed method is simple and easy to understand.
- The experiments cover different domains and learning models.
Critical weaknesses
Weaknesses
- The experiments are limited and not so informative and readible. I guess a good way is to provide detailed numbers in appendix, and use more figrues, plus highlighting takeawyas in the experiment section.
- I think the proposed method is relatively simple, mainly combining validation-based selection with some heuristic rules. I wonder what is the key methodological novelty compared with existing source selection methods? Would it be possible to provide more analysis on why this particular combination is effective?
- I notice that ARCS needs to evaluate different source configurations before selection. However, the number of possible configurations grows exponentially with the number of sources. I wonder how practical this method would be when there are tens or hundreds of source domains? Would it be possible to provide more experiments on the computational cost and scalability?
- Is this generalize to pretraining? How to combine multi-source datasets is more siginificant in the pretraining/mid-training scenarios, rather than downstream applications. Related works [1-3] have discussed the scaling and mixed training of different multi-modal datasets, it's worth discussing here. This kind of limits the scope the paper, proper limitation/related work section can help maintain a supportive aspect.
- I notice that the improvements are not always consistent across datasets. For example, AllSource performs better on MOSEI, and the improvement over DATE on PPMI is not statistically significant. I wonder under what conditions ARCS is expected to provide clear benefits? It would be helpful to provide more analysis of when source selection is more useful than simply using all sources.
- How about other metrics? Since here you report the Macro-F1, it's depend on how you choose the threshold. For discriminative tasks, it's better to use AUROC and AUPRC.
[1] MegaScale-Data: Scaling Dataloader for Multisource Large Foundation Model Training. European Conference on Computer Systems'2026.
[2] Improving knowledge tracing through multi-source scaling with decoder-only transformers. ACM Transactions on Intelligent Systems and Technology'2026.
[2] OSF: On Pre-training and Scaling of Sleep Foundation Models. ICML'26.
Questions for authors
See weakness.
Improved experimental presentation and readability, with additional AUROC/AP evaluation and clearer discussion of key findings
1. Experimental presentation and readability
We have replaced Figure 2 with Table 2 for clearer presentation. We have revised the experimental presentation to make the findings clearer and easier to follow. Specifically, we have expanded the experimental details in Section 6 and Appendices A–F. We have also made the key takeaways and their implications more explicit in the experimental analysis, highlighting what the results reveal about source configuration selection.
We also provide an anonymized repository at https://anonymous.4open.science/r/arcs-review-package-66CC/.
2. Methodological novelty and contribution
Our main contribution is a systematic investigation of target-specific source configuration selection, providing empirical insights into the challenges of selecting appropriate source combinations and how such decisions can be made. We examine source configuration utility, the reliability of target-validation evidence, and the trade-offs involved in selection decisions. Building on these findings, we introduce ARCS as an explicit selection rule and systematically evaluate its behavior across different benchmarks, learners, and selection criteria.
3. Computational cost and scalability
Appendix I reports an approximate-search experiment on PPMI using the four learners. At beam width B=5, the number of evaluated configurations decreases by 21.33%, while ARCS matches exhaustive selection in 45 of 48 evaluation units (93.75%). These results demonstrate that ARCS can be combined with approximate search to reduce configuration evaluations while largely preserving exhaustive selection decisions. The overall computational cost depends not only on the number of evaluated configurations but also on the underlying learner, training data volume, and model training and evaluation costs. Scalability to substantially larger source pools remains an important direction for further evaluation.
4. Scope and relation to multi-source pretraining
Our study focuses on target-specific source configuration selection, where limited target-validation evidence is available to guide which sources should jointly participate in learning. This differs from large-scale pretraining and mid-training, where source mixing often involves data proportions and training-budget allocation without direct feedback from a specific downstream target. We have clarified this distinction and discussed the recommended three relevant works in the revised Related Work (Section 2) and Limitations section.
5. Source selection under different conditions
The varying results across benchmarks are an important part of our empirical findings. Our analyses across HAR4-Fixed, PPMI, and MOSEI show that source configuration utility depends on the target, participating sources, and underlying learner, while identifying beneficial configurations remains challenging under limited validation evidence. We have clarified these findings and their implications in the revised experimental analysis and Discussion.
6. Additional evaluation metrics
We have supplemented the Macro-F1 results with AUROC and average precision (AP) evaluations across HAR4-Fixed, PPMI, and MOSEI (Appendix G). ARCS achieves competitive AUROC and AP on HAR4-Fixed, closely approaching BestSingleton, while AllSource performs best on PPMI and MOSEI. These findings show that neither selecting fewer sources nor including all sources consistently yields the best predictive performance. Instead, source configuration utility varies across benchmarks, further highlighting the importance of target-specific source configuration selection.