acceptodds
Under review as a conference paper at ICLR 2027

Before Training, Which Multimodal Data Will Help? Resolution Limits Across Granularities and Learners

Abstract

Choosing a multimodal training corpus poses a catch: we learn whether it helps only after paying for the training run. When there are more corpora than we can try, can frozen representations tell us which ones are worth that cost? We test this question on three paired audio and text datasets for sentiment and emotion. Our sealed evaluation fixes each decision before candidate utility is revealed and repeats the comparison with four downstream learners. The answer depends on selection granularity. With logistic regression, centroid distance chooses the oracle source in 44 of 45 decisions, yet fails to rank buckets from that source. The separation between sources is 22.3 times the typical within-source spread; matching bucket size and class balance does not recover the ranking. Utility also changes with the learner and fusion rule, while candidate-specific versions of LEEP and LogME remain unreliable as direct bucket rankers. We use these signals to form a shortlist rather than replace downstream evaluation. Before opening a new repartition family, we fix a procedure that routes by distance, ranks by adapted LEEP, evaluates the top two candidates with the full downstream recipe, and keeps the target-only model if neither helps. Across three targets, it improves validation macro F1 by 0.01084 on average, with a 95% hierarchical bootstrap interval of , while avoiding 87.5% of full candidate runs. The result locates a practical boundary: prospective scores can guide a training budget, but they are not universal measures of dataset quality.

Primary area:
datasets and benchmarks
Keywords:
multimodal data selection, dataset utility prediction, transferability estimation, data valuation, source selection, audio-text learning, datasets and benchmarks, compute-efficient learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_9mPvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies multimodal data selection before training. This work defines utility as a joint function of the source, target, model, and training recipe, and evaluates it under a sealed protocol in which each decision is fixed before candidate utility is revealed. The evaluation spans three paired audio and text datasets for sentiment and emotion (CMU-MOSI, MELD, and CH-SIMS) and four downstream learners: logistic regression, GaussianNB, a neural MLP, and learned gated fusion. The central finding is that such signals suffice for routing between sources but fail to rank buckets within a source, and that utility is learner dependent. This paper therefore proposes a compromise that routes coarsely, forms a shortlist, and evaluates only a few candidates with the full downstream recipe.

Critical strengths

The motivation is sound: multimodal training is highly resource-intensive, so identifying which data are critical before training begins is of key practical importance.

Critical weaknesses
  • The title and abstract claim to study prospective selection of multimodal data, but all evidence comes from three audio-text sentiment corpora with one task type and one modality pair. The paper then draws general conclusions about multimodal data selection. The title and the conclusions claim more than the evidence covers.
  • All results are from the validation set. Section 4.1 states that only training and validation splits are used, and no test-set result is reported anywhere in the paper. Additionally, the best positive pilot is run and selected on the validation set by validation utility, and then reports the utility on that same validation set.
  • All utilities come from frozen encoders with light heads (Section 4.5), and no end-to-end training is done. It is therefore not clear whether the conclusions hold when the model is larger, or when the encoder is tuned.
  • This method combines existing parts: static features, a short probe, adapted LEEP/LogME, and a shortlist-and-pilot procedure. It gives no new method for ranking buckets within a source and no theory that explains or predicts this failure. The contribution therefore depends only on the evaluation results, and these results have limited novelty and reliability.
  • The presentation is poor.
    • Figure 1 does not help to explain the work. It puts the text of Section 4 into boxes. It does not show the main idea of the method or how routing, bucket ranking, and the shortlist relate to each other.
    • The tables are badly formatted. Table 3 has broken lines, misaligned columns, and mixed units in one column. The last row of Table 5 has only one value.
    • Most numbers have 5–6 decimal places (Table 2, Table 4, Section 5.2), and the precision changes between tables (Table 3 vs. Table 5).
Questions for authors

Please refer to the weaknesses.

Reviewer_wri9OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The work establishes how to determine (prior to full training), whether given text-audio datasets will improve the performance of a given model and training method if additional data (buckets from different corpora) is included in the training data. The method enables establishing a shortlist of text/audio sources to add to the data to increase performance. Withholding corpora when there is no benefit to adding their data is also discussed and investigated.

Critical strengths

S1. The research question is well motivated and of practical importance.

Critical weaknesses

W1. Presentation and problem framing. The presentation quality of the paper is quite poor and needs to be improved, to the degree that I vote to reject the paper. While the approach is intuitive, the presentation makes it very hard to follow.

  • Density and missing definitions. At times the text is extremely dense in terms of information, and key concepts often need to be assumed (their definition is missing and/or can only be inferred from context provided two sections later). Some concepts, such as "source", are conflicting: Section 3 defines a source as a bucket of the corpora (i.e. a subset of text-audio samples), while the vast remainder of the paper treats it as an entire corpus (lines 265–266). Intuitively, one might even assume "source" refers to the modality (i.e. text or audio). This also includes terms such as "corpus routing" (line 100), "within-corpus ranking" (line 100), "positive pilot" (line 86), "prospective stage" (line 133), "released manifests" (line 151), "development partition" (line 155), "hash salted sample identifiers" (line 156), "repartition family" (line 187), etc. which are used before they are defined or are not defined at all. At other times the terminology is clear but the underlying construction is not: how are "audio centroid distances" computed? Is the centroid taken over each bucket? Which metric is used to quantify the distance between centroids?
  • Unclear overarching task. After reading the paper twice, I am still not entirely sure what the overarching task is. Do the authors perform transfer experiments (line 152), or do they train from scratch on different unions of the target data and the auxiliary corpora? The framework is never clearly stated, and the task has to be read between the lines: on the one hand the authors mention transfer learning (line 152), on the other hand the method describes fixed-target augmentations (lines 116, 442).
  • Abstract. The abstract assumes far too much knowledge of the paper's content (centroids, buckets, sources, LEEP/LogME, repartition family, target-only model) and needs revision.
  • Figures. Some figures have legends overlapping the plot contents.

W2. Incremental gains. A average gain of 0.01 in F1 score seems incremental, considering that, by construction, the performance difference between the proposed methodology and the anchor-only model is non-negative: "[...] uses validation utility to keep a positive pilot or fall back to the anchor-only model" (lines 140–141).

Questions for authors
  1. How are the buckets created?
  2. What is the cost of your method? How does it compare to exhaustive train-on-all-corpora approaches?
  3. Does the method (if applicable) translate from training from scratch with different data augmentations (target anchors + bucket) to sequential transfer learning, where a model has already been trained and we identify which bucket benefits performance the most when complemented with additional multimodal data?
Reviewer_PYveOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether you can tell which candidate multimodal (audio-text) data will help a target model before running any transfer experiments, using only cheap scores computed from frozen pretrained representations. The authors use a small set of affective-computing corpora, each taking a turn as the target. They split selection into two levels: which source corpus to draw from, and which subset of it to use. They test several kinds of selectors, from simple similarity statistics to adapted transferability measures. The evaluation protocol is strict: scores and decision rules are fixed before any outcomes are seen. Representation distance reliably picks the right source but not the best subset within it. Promising selectors fail to replicate on new data, and which data helps depends on the downstream learner.

Critical strengths
  1. The evaluation discipline and honest reporting are better than usual for data selection papers. Scores are written before utilities exist. Thresholds are set on a separate family. A fresh family is opened only after the workflow is frozen. Salts are explicitly not treated as independent replicates. The paper also reports its own failures plainly: Curve 19 does no better than random, the classifier gain reverses across families, and the abstention rule ends up rejecting everything. Section 5.2 is a good cautionary example of how a selector can look like a win on one repartition and not replicate. The idea of splitting "which corpus" from "which subset" is also useful, even if I disagree below with what the paper concludes from it.
Critical weaknesses
  1. The positive result comes mostly from the fallback rule, and it is selected and reported on the same validation set.

In the k=2 workflow, the realized utility of each decision is max⁡(0,Ua,Ub)\max(0, U_a, U_b). Here UaU_a and UbU_b are the validation macro F1 gains of the two pilots, and the same validation number is then reported as the outcome. So every decision is ≥0\geq 0 by construction. "Every target mean is positive" and the bootstrap interval excluding zero follow automatically for any shortlist rule. Any resample of non-negative values with a few positives has a positive mean. These statements carry almost no evidence that adapted LEEP is doing anything.

The missing baseline is the obvious one: pick k buckets at random from the routed source and apply the same keep-best-positive-or-fallback rule. I did a back of the napkin check using the spread implied by the logistic rows of Table 4: uniform mean about 0.0008, and implied mean oracle about 0.019 over 8 buckets (utility + regret), so within-source spread is roughly σ≈0.012\sigma \approx 0.012. Under that spread, random-2 with fallback gives about 0.007 to 0.010, and random-4 gives about 0.012 to 0.015. The paper reports 0.01084 and 0.01320. Oracle coverage tells a similar story. A random 2-of-8 shortlist contains the best bucket 25% of the time (about 3.5 of 15 decisions after routing), versus 5/15 for adapted LEEP. For k=4 it is 50% (about 7.5 of 15) versus 10/15. With 15 correlated decisions from 3 targets, I cannot tell these apart. My estimate uses the spread from Table 4, which may differ from the fresh family's. But the authors already have the utility of every candidate in the fresh family, so they can compute the exact random-shortlist baseline without any new training runs.

On top of that, taking the max of two noisy validation estimates is biased upward. Test splits exist for all three corpora but are never used. The 87.5% "avoided runs" also leaves out the adapted LEEP cost. Adapted LEEP fits two logistic heads on every candidate (Table 7), the same model class as the downstream. The paper counts each exact pilot as ten heads only because the evaluation averages five seeds for stability. Even under that scenario, the saving on 16 candidates is 67.5% ((32+20) vs 160) rather than 87.5%. In practice a person adopting this approach would more likely run each candidate once. Then a full run costs about the same two heads that adapted LEEP already fits on every candidate, and scoring all candidates costs roughly as much as evaluating them all directly. Basically, this undercuts the "Before Training" framing a bit. The only training-free signal, centroid distance, picks the source but cannot rank buckets within it. The positive result depends on fitting heads to every candidate and then running exact pilots, and both of those are training.

  1. The within-source "resolution limit" looks built into the benchmark, and there is no noise ceiling or positive control.

As far as I can tell from Sec. 4.1 and 4.4, buckets are hash partitions of salted sample IDs. That makes the buckets of one source exchangeable random draws from the same distribution. Any distribution-level static score (centroid distance, label entropy, class-conditional distance) can then differ between buckets only through sampling noise. That is exactly what the split-half tau=-0.152 shows. The 22.3x ratio between source gap and bucket spread mostly reflects bucket size, since the spread of a centroid estimate shrinks like 1/n1/\sqrt{n}.

The target side has the same problem. UiU_i is a macro F1 difference on small validation sets. With standard splits, that is 229 utterances for MOSI and 456 for CH-SIMS. The 5 seeds do not resample the validation set. A rough estimate of the sampling error of a paired macro F1 difference, with a few percent of predictions flipping on a couple hundred examples, is around 0.01 to 0.015. That is the same size as the within-source spread of utility. The paper never reports how reliable the ground-truth bucket ranking is (seed-to-seed SD of ui,su_{i,s}, or rank stability under a bootstrap of validation examples). So "distance cannot rank buckets" may simply mean "there is no stable ranking to recover".

To support a claim about granularity, the paper needs a positive control: buckets with real within-source differences, such as injected label noise, speaker, topic, or recording-condition splits, or corrupted audio. Then we would see whether the same selectors can rank subsets when real differences exist. Without that, the title's "resolution limits across granularities" is really about random partitions.

The routing result is also weaker than "44 of 45" suggests. Each target has only 2 eligible sources, so routing is a binary choice. The 45 decisions all come from re-partitioning the same 3 corpora with different salts and families. Re-partitioning does not change the corpora, so a source's mean centroid barely moves and each target is routed the same way almost every time. This is closer to 3 binary calls than 45 independent successes, and since one corpus differs in language from the other two, some of those calls are likely easy. Under the other learners, route accuracy is 12/15 (MLP), 10/15 (GaussianNB), and 8/15 (gated fusion), and the last is close to chance for a binary choice.

Questions for authors
  1. In the k=2 and k=4 workflow, is the validation F1 used to pick the winning pilot the same number that is reported? What is the test-split utility of the selected model?
  2. On the fresh family, what do these alternatives give under the same keep-best-positive-or-fallback rule: random k from the routed source, top k by distance, and top k by Curve 19? Please include utility, oracle coverage, and the fraction of decisions where the shortlist contains the oracle. This is the comparison that decides whether adapted LEEP contributes anything.
  3. How exactly are buckets formed? Are they uniform random hash partitions within each source, with no stratification by speaker, video, or topic?
  4. How reliable is UiU_i? What does the seed control in the logistic recipe? What is the seed-to-seed SD of ui,su_{i,s}? How stable is the within-source bucket ranking under a bootstrap over validation examples?
  5. What is the exact formula for adapted LEEP? How many heads are fit per candidate (2, or 10 given the five anchor seeds)? Is that cost counted in the 87.5%?
  6. In Table 4, were the selectors for GaussianNB, MLP, and gated fusion retrained on those learners utilities, or carried over frozen from logistic?
  7. What exactly was the pre-locked "success criterion" (in section 4.4)?
1 comment

Thank you for the detailed response. I have read it and appreciate the added analyses and the decision to withdraw the original positive effectiveness claim. The test-set comparison, matched-shortlist baselines, and cost accounting address some of my questions. The revised results reinforce my concern that the original validation-selected gain did not establish a benefit from adapted LEEP. I also still find the central granularity claim insufficiently supported. The added label-corruption control shows sensitivity to an imposed difference, but does not establish that the selectors can rank naturally meaningful within-source subsets. These are concerns about the evidence and scope, rather than requests for another clarification. I am keeping my score unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.