acceptodds
Under review as a conference paper at ICLR 2027

Cohort-Based Active Modality Acquisition

Abstract

Real-world multimodal machine learning often faces missing, costly-to-acquire modalities, raising the problem of which samples to prioritize for additional acquisition under a budget. Prior work mainly studies per-sample or training-time acquisition while test-time, cohort-level acquisition is less explored. We propose Cohort-based Active Modality Acquisition (CAMA), a novel test-time cohort-level modality acquisition setting, and introduce imputation-based acquisition strategies that estimate the expected utility of acquiring a missing modality, along with upper-bound heuristics for benchmarking. Experiments on datasets with up to 15 modalities demonstrate that our proposed imputation-based strategies can more effectively guide the acquisition of an additional modality for selected samples com- pared with methods relying solely on pre-acquisition information, entropy-based guidance, or random selection. We showcase the real-world relevance and scalability of our method by demonstrating its ability to guide the acquisition of proteomics data for disease prediction in a large prospective cohort, the UK Biobank (UKB). Our work provides an effective approach for optimizing modality acquisition at the cohort level, enabling more effective use of resources in constrained settings.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
active modality acquisition, multimodal learning, missing modalities, cohort-level optimization, test-time acquisition, modality imputation, resource allocation, uncertainty estimation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_opKJOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes CAMA, a test-time cohort-level modality acquisition framework that allocates a fixed budget to acquiring one specified missing modality. Generative imputation estimates post-acquisition predictions, which guide sample selection to improve cohort-level AUROC or AUPRC.

Experiments on four datasets with up to 15 modalities demonstrate improvements over random selection and simple baselines in several settings.

Critical strengths

S1: The problem of allocating costly modality acquisitions under a limited budget is well-motivated and practically relevant.

S2: The proposed method shows promising improvements over random selection across several datasets.

Critical weaknesses
  • W1: Test-time information acquisition has been studied in EDDI [R1], while DyMo [R2] selects reliable recovered modalities at inference time. Their test-time selection and generative imputation alone are not new. The authors should clarify the methodological novelty arising specifically from cohort-level allocation.

  • W2: The evaluation excludes splits where full acquisition reduces test performance (Section 5). This conditions the results on outcomes unavailable before acquisition. Since additional modalities can introduce noise or conflicting signals [R3], such cases should be evaluated rather than excluded. Please report exclusion counts and unfiltered absolute performance.

  • W3: The acquisition functions mainly adapt established probability, entropy, and divergence criteria. Large KL divergence between pre-acquisition and imputed predictions may reflect unreliable imputation rather than useful acquisition. Stronger baselines, such as EDDI [R1] and DyMo [R2], and analysis of imputation reliability are needed.

  • W4: Sample-level visualizations could explain whether acquisition priorities predict realized benefits.

[R1] Ma, Chao, et al. "EDDI: Efficient dynamic discovery of high-value information with partial VAE." ICML 2019.

[R2] Du, Siyi, et al. "Inference-time dynamic modality selection for incomplete multimodal classification." ICLR 2026.

[R3] Dai, Yinglong, et al. "Analysis of multimodal data fusion from an information theory perspective." Information Sciences 2023.

Questions for authors
  • Q1: What is the methodological novelty compared with prior information acquisition and recovered-modality selection?

  • Q2: Do the conclusions hold when negative-gain splits are retained? How many splits are excluded?

  • Q3: Can the authors analyze whether KL-based priorities remain reliable under inaccurate imputation and compare against stronger acquisition baselines?

  • Q4: Could the authors visualize acquisition process for representative successes and failures?

Reviewer_Mx7ROpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces CAMA, a test-time mechanism to actively select a subset of samples within a cohort for each it would be beneficial to acquire a specific missing modality. The work is framed from the perspective of healthcare and it is validated on 3/4 healthcare datasets and several tasks (lacks some clarity here), but it could potentially be extended to other application fields. The proposal explores the use of active feature selection mechanisms to rank the samples that would benefit more from also having access to the missing modality during inference time. For this the paper describes two directions, the first one works as a baseline, where the ranking of samples if performed based on uncertainty quantification strategies. The second direction explores the use of generative mechanisms to input the latent representation of the missing modality, and compare the logic scores before and after the imputation, once more leveraging uncertainty quantification methods to determine for which samples the access to the additional modality would be beneficial. The paper compares this two settings against the oracle based methods, where the true classes are known, and quantifies the gains based on the variations in the AUROC and AUPRC. While the field of application and potential impact of the proposal seems promising, the paper has some experimental and conceptual limitations that would require further analysis prior to acceptance and publication. Moreover, the description of the methodology and the presentation of the experiments in the main corpus has some clarity issues, making it hard to assess and fully understand the paper without going back and forth between it and the additional pages.

Critical strengths
  1. This paper addresses a relevant topic, focusing on identifying if a missing modality would be beneficial to change the decision of a classification model. Given that some modalities can be expensive, particularly in the healthcare field, this is a timely and relevant research question.
Critical weaknesses
  1. A core element of the proposal is the use of an imputation strategy to simulate the gains in the evaluation metric. Such assumption only holds if there is a significant overlap/redundancy between the available and missing modalities, i.e., if the existing modalities actually contain some clues about the missing one. Otherwise, there is no guarantee that a generative approach will create meaningful modality embedding and not an hallucination, even if guided by a discriminative process too. There is a discussion about this in the experimental section, but no experimental evidence actually supports that the embeddings generated under this framework are actually more robust or meaningful than generating a single deterministic embedding, for example, or that they are capturing the predictive component of the missing modality. This concern is also directly related with which type of mechanism responsible for the missing modality. The paper frames the missing from the perspective of “modality cost”, but in the real world missing occurs for many reasons, and it would be relevant do understand in which paradigms (MAR, MNAR, and MCAR) does the proposal hold.

  2. The pipeline is always assessed with a fixed missing modality, and in most of the settings just using two modalities. While the authors recognize that this is a limitation of the paper, given the variety of modalities that exist in the selected datasets, it makes it difficult to isolate whether the proposal would actually bring benefits to a real multimodal setting. Moreover, for some of the pairs even the gain with the Oracle is quite low, maybe suggesting that the same modality does not contribute to all tasks. Surprisingly, some of the settings where the Oracle most benefit from the assess to the new modalities are those where the imputation based strategy goes to negative values - which supports my previous concern that imputation may have severe limitations when the existing and missing modalities do not share much information.

Questions for authors

This paper has potential, but that it is still in a preliminary form and requires further work and experimental validation before acceptance. Nevertheless, I have a few questions that I would like to see clarified:

  1. Could the authors discuss in which missing paradigms does their proposal work and in which would it fail/have limitations?

  2. It is unclear how the multimodal fusion is actually perform. Is this a transformer layer with masking for the missing modality?

  3. The generative model seems to be recovering one modality from the other. In the binary modality setting this is clear, but what happens when you are inputing one modality from two others, as reported in Section I? Moreover, how do you validate the generated embeddings in terms of distribution w.r.t the original one?

  4. Please report what is the actual modality budget used in the results.

  5. The impact of manipulating the data flow is not negligible (see Table 7). Can you discuss why the data flow and balancing the set seem to have this impact in the results? Moreover, where does the KL-divergence score comes from? Is this from the BC-VAE against the DDPM?

  6. What happens when the missing modality is not recoverable from the existing one(s), but still conveys relevant information? How would the proposal flag which cases are more relevant?

Suggestion: Please consider rearranging the information in the paper, as it is hard to collect the full proposal and interpret the results from the document as it is.

Reviewer_uaYBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper defines Cohort-based Active Modality Acquisition (CAMA). A cohort of test samples has a subset of modalities available. A fixed budget allows acquiring one target modality for β samples. The goal is to choose the subset that maximizes a global metric such as AUROC or AUPRC. The authors formalize this as a constrained subset selection problem. Experiments cover UK Biobank (proteomics for SLE), MIMIC Symile, MIMIC HAIM, and MOSEI, with up to 15 modalities.

Critical strengths
  1. The problem is well chosen. Cohort-level, test-time acquisition of one fixed modality is a real constraint in biobanks and hospitals, and the UKB proteomics example makes the motivation concrete. The formalization in Eq. 3–5 is clean and makes the objective precise. This is a useful framing even if the methods are simple.

  2. The imputation-based KL acquisition function is simple and does not need labels at test time. The idea of using a generative model to sample the counterfactual score, then ranking by the expected divergence, is a sensible way to approximate expected information gain. The fact that it beats entropy and probability baselines on several datasets is a useful empirical result.

Critical weaknesses
  1. The evaluation metric is not sound. G_full can exceed 1, and the oracle reaches 2.787. The cause is that s^avail and s^acquired come from the same network under different input masks, so mixing them can shift the score distribution. The paper's own "upper-bound heuristics" are sometimes beaten by the proposed method. An upper bound that can be beaten is not an upper bound. This means the reported gains may be partly a calibration artifact, not an information gain.

  2. The empirical support is thin and selective. Only the KL variant beats simple baselines. On Symile, imputation Rank is 0.378 against random 0.376, and Probability is 0.426 against baseline 0.458. On UKB, KL is 0.641 against random 0.528, a modest gain. On MOSEI with image imputed by text, KL is 0.551 against random 0.521. The main table averages over modality combinations and hides this. In addition, the paper excludes tasks with negative gain at the split level. That selects the cases where acquisition helps and biases the evaluation.

Questions for authors
  1. How is G_full defined when M_post < M_pre? You exclude those tasks. How many splits were excluded, and what happens if they are included?

  2. The oracle exceeds 1 by a large margin. Can you explain the mechanism?

  3. In Table 5, imputation KL (0.855) is close to true KL (0.882), and in Table 3 it is close to the true KL upper bound. How can an estimate approach an upper bound that uses the true acquired scores?

  4. How many imputations K do you use? You mention 100 DDPM samples. How sensitive is the ranking to K, and what is the wall-clock cost at 100k samples?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.