- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies multimodal data selection before training. This work defines utility as a joint function of the source, target, model, and training recipe, and evaluates it under a sealed protocol in which each decision is fixed before candidate utility is revealed. The evaluation spans three paired audio and text datasets for sentiment and emotion (CMU-MOSI, MELD, and CH-SIMS) and four downstream learners: logistic regression, GaussianNB, a neural MLP, and learned gated fusion. The central finding is that such signals suffice for routing between sources but fail to rank buckets within a source, and that utility is learner dependent. This paper therefore proposes a compromise that routes coarsely, forms a shortlist, and evaluates only a few candidates with the full downstream recipe.
Critical strengths
The motivation is sound: multimodal training is highly resource-intensive, so identifying which data are critical before training begins is of key practical importance.
Critical weaknesses
- The title and abstract claim to study prospective selection of multimodal data, but all evidence comes from three audio-text sentiment corpora with one task type and one modality pair. The paper then draws general conclusions about multimodal data selection. The title and the conclusions claim more than the evidence covers.
- All results are from the validation set. Section 4.1 states that only training and validation splits are used, and no test-set result is reported anywhere in the paper. Additionally, the best positive pilot is run and selected on the validation set by validation utility, and then reports the utility on that same validation set.
- All utilities come from frozen encoders with light heads (Section 4.5), and no end-to-end training is done. It is therefore not clear whether the conclusions hold when the model is larger, or when the encoder is tuned.
- This method combines existing parts: static features, a short probe, adapted LEEP/LogME, and a shortlist-and-pilot procedure. It gives no new method for ranking buckets within a source and no theory that explains or predicts this failure. The contribution therefore depends only on the evaluation results, and these results have limited novelty and reliability.
- The presentation is poor.
- Figure 1 does not help to explain the work. It puts the text of Section 4 into boxes. It does not show the main idea of the method or how routing, bucket ranking, and the shortlist relate to each other.
- The tables are badly formatted. Table 3 has broken lines, misaligned columns, and mixed units in one column. The last row of Table 5 has only one value.
- Most numbers have 5–6 decimal places (Table 2, Table 4, Section 5.2), and the precision changes between tables (Table 3 vs. Table 5).
Questions for authors
Please refer to the weaknesses.
Thank you for the detailed response. I have read it and appreciate the added analyses and the decision to withdraw the original positive effectiveness claim. The test-set comparison, matched-shortlist baselines, and cost accounting address some of my questions. The revised results reinforce my concern that the original validation-selected gain did not establish a benefit from adapted LEEP. I also still find the central granularity claim insufficiently supported. The added label-corruption control shows sensitivity to an imposed difference, but does not establish that the selectors can rank naturally meaningful within-source subsets. These are concerns about the evidence and scope, rather than requests for another clarification. I am keeping my score unchanged.