acceptodds
Withdrawn from ICLR 2027

DACE: Divide-and-Compose Evidence for Missing-Modality Recognition

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal recognition must remain robust when modalities are unavailable at test time. Existing methods either reconstruct entire missing representations or fuse modalities without distinguishing transferable from modality-specific information. We introduce Divide-and-Compose Evidence (DACE), a lightweight recognition head that achieves robustness through learned evidence separation and selective cross-modal transport. For each observed modality, a complementary router divides its encoded feature into two branches: specific evidence that captures modality-unique cues and remains local, and shared evidence that expresses transferable cross-modal information. When a modality is missing, DACE recovers only its shared slot by transforming and averaging the shared branches from observed sources, leaving modality-specific content unestimated. The recovered and observed evidence is then composed and refined through availability-conditioned attention and observed-pair agreement. Experimental results demonstrate consistent robustness: across three benchmarks and 27 missingness conditions, DACE achieves the highest score in every comparison, and a single trained model maintains strong performance when test rates shift from 10% to 90%. On naturally incomplete medical data, DACE reaches 84.30% accuracy and 75.60% AUROC, demonstrating robustness on real-world incomplete medical data.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Missing-modality learning, Multimodal classification, Multimodal fusion, Representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_aPvUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DACE (Divide-and-Compose Evidence) for multimodal recognition under missing modalities. A complementary gate divides each observed representation into branches assigned specific and shared roles. When a modality is absent, learned cross-modal transformations estimate only its shared branch, leaving its specific branch unestimated. The evidence is composed according to the availability mask and refined using mask-conditioned attention and observed-pair agreement.

DACE is evaluated on MM-IMDb, Food-101, Hateful Memes, and naturally incomplete MMIST-ccRCC. The three image-text benchmarks use frozen CLIP ViT-B/16-QuickGELU features and a recognition head with fewer than one million trainable parameters. DACE has the highest displayed score in all 27 main image-text conditions. The paper also evaluates cross-rate transfer and natural medical missingness. The displayed ranking is encouraging, although the historical baseline scores do not constitute a fully controlled comparison.

The design principle is useful, but the experiments do not yet isolate whether the gains arise from meaningful shared/private information separation, increased head capacity, or the training treatment of missing subsets. The theoretical properties characterize the architecture; they do not by themselves establish the semantic interpretation of the branches.

Critical strengths

The divide–recover–compose design is conceptually coherent and addresses a genuine weakness of full-feature reconstruction. Recovering only evidence expected to transfer between modalities is a sensible inductive bias: genuinely modality-private information need not be reconstructible from other inputs. The staged ablation is directionally encouraging, with Divide giving the largest improvement and Compose/refinement adding gains. However, the capacity increase at the Divide stage prevents a clean attribution of that improvement to decomposition alone. The evaluation covers three image-text tasks, three missingness patterns, three rates, cross-rate transfer, and a naturally incomplete medical dataset. Table 2 is especially useful because it evaluates a model trained at mixed 70% missingness across different test conditions. This breadth supports the practical value of the architecture. The medical experiment is useful evidence on naturally incomplete data, while the small, imbalanced cohort limits broader conclusions.

Critical weaknesses
  1. The semantic interpretation of the specific/shared branches is insufficiently validated. Equation (2) enforces a complementary algebraic partition, but does not establish that one branch contains transferable cross-modal information and the other modality-private cues. The paper explicitly avoids assuming a unique semantic decomposition, including beyond initialization; I am not asking for a proof of unique identifiability. The unresolved issue is whether the learned branches empirically perform the semantic roles used to motivate the method. Figure 4 provides qualitative support, but t-SNE separation or proximity cannot establish modality specificity or successful cross-modal transport. Section 4.5 and the caption broadly identify the processing stages; reproducibility would improve by specifying the exact tensors, samples, normalization, and whether all compared representations use one common projection. The claimed semantic interpretation needs direct measurements beyond the visualization. Useful tests include predicting the paired modality's features or cross-modal retrieval from each branch with matched-capacity probes, branch-removal interventions, and a controlled task with known shared and modality-private factors. Such tests should distinguish transferable information from task-specific private cues. Existing shared-specific approaches, especially ShaSpec, also deserve a more precise mechanism-level comparison.

  2. Capacity, training-risk, and baseline-protocol differences confound attribution of the performance gains. Table 5 already reports parameter counts and reveals a substantial confound in the staged ablation: on MM-IMDb, Direct has 0.027968M parameters and Divide has 0.749888M; on Hateful Memes, the corresponding counts are 0.002432M and 0.724352M. The largest performance jump therefore coincides with a large capacity increase. A parameter-matched nonlinear head without the specific/shared split, together with fixed or random complementary gates, is needed to isolate the proposed division mechanism. Section 4.1 also states that FF/SF use one sampled incomplete mask per training example, whereas DACE evaluates the weighted subset risk exactly. This is a meaningful training difference even when the nominal missingness rates agree. Please compare local controls under the same subset-risk objective and explain which original modalities remain accessible during training. The current description raises a data-access and attribution question; it does not establish leakage. The main comparison reproduces published MoRA/SyP-era results while the local models use frozen CLIP features. The paper acknowledges this provenance and describes the missingness cases, rates, seeds, and dataset splits. The remaining issue is whether exact sample masks, paired-data access, model selection, and training budgets are comparable. DACE's highest displayed values are valid observations, but stronger claims of controlled superiority require a reproducible comparison. A focused rerun of a strong recent baseline under shared data splits, missingness masks, backbone where feasible, and model-selection rules would be more informative than requiring every method to adopt an identical freezing policy that may change its native design. Report each method's adaptation and compute budget. A relevant shared-specific baseline such as ShaSpec would further test the claimed mechanism; Unseen Modality Interaction is relevant context, but should not automatically be treated as the same type of explicit factorization. Two reporting inconsistencies also need resolution. FF on Hateful Memes at mixed 50% is 68.94 in Table 1 but 71.13 in Table 4. In addition, Section 4.2 interprets Food-101 as visually dominant, although the reported 90% conditions give 78.89 with text missing versus 87.15 with images missing. Please reconcile the FF settings and correct the missing-modality interpretation, checking both the I/T labels and the narrative.

[1] Hu Wang, Yuanhong Chen, Congbo Ma, Jodie Avery, Louise Hull, and Gustavo Carneiro. Multimodal learning with missing modality via shared-specific feature modelling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15878–15887, 2023. [2] Yunhua Zhang, Hazel Doughty, and Cees Snoek. Learning unseen modality interaction. volume 36, pp. 54716–54726, 2023.

Questions for authors

Clarification requests

  1. Can the authors reconcile the FF Hateful Memes scores in Tables 1 and 4 and the Food-101 modality interpretation in Section 4.2? If these results use different protocols, please identify those differences explicitly.

  2. For the exact subset-risk training in Equation (10), are complete original pairs accessible before masks are applied? Can FF/SF and the capacity-matched nonlinear control be trained under the same objective and data access? Please distinguish this from the fixed train/validation/test splits already documented in Appendix C.

  3. What direct evidence shows that the designated shared branch is more transferable than the specific branch? Please identify the exact tensors and common projection protocol used in Figure 4, and avoid interpreting t-SNE geometry as proof of semantic separation.

  4. Which published baseline scores use the same actual missingness masks and paired training information as DACE? For the principal comparison, please report variability across the three stated seeds and, where feasible, a strong baseline rerun with matched masks. Small margins should be interpreted relative to that variability.

Reviewer_AU49OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a Divide-and-Compose Evidence (DACE) framework that decomposes the encoded representation of each modality into two complementary components: specific evidence, which captures modality-specific information and remains local to the modality, and shared evidence, which encodes transferable cross-modal information. When a modality is unavailable, DACE reconstructs only its shared evidence by transforming and aggregating the shared representations from the observed modalities. The recovered and observed evidence are subsequently composed and refined through an availability-conditioned attention mechanism and an observed-pair agreement strategy before being utilized for downstream prediction/classification tasks. The paper further provides theoretical analyses of the proposed framework in terms of evidence conservation, missing-input invariance, and bounds on shared-evidence reconstruction error and residual magnitude. The results from two sets of experiments on imaging-text recognition (three datasets) and naturally incomplete medical data (one dataset) demonstrate the superior performance of DACE compared to baselines considered.

Critical strengths

The idea of dividing encoded features for each modality into two branches: specific evidence that captures modality-unique information and remains local, and shared evidence seems potentially useful in real-world settings. The experimental results demonstrate the superior performance of the proposed method compared to the baselines included. Though, my enthusiasm is dampened by the weaknesses noted in the next section.

Critical weaknesses
  1. It is unclear whether the proposed DACE framework is valid under different missing data mechanisms, e.g., MCAR, MAR, and MNAR. Actually there is no discussion about this in the paper.

  2. In Section 2, there is insufficient review of a growing body literature on a) learning both shared and modality-specific representation in multimodal learning with missing data (https://arxiv.org/abs/2409.07825; https://arxiv.org/abs/2502.11724; https://arxiv.org/abs/2504.16352; https://arxiv.org/abs/2403.06197). b) spare Mixture of Experts (MoE) based imputation models including those that can learn interactions between modalities (https://arxiv.org/abs/2605.27431), which are very much relevant to the proposed method. As such, the claim by the authors that “Existing methods either reconstruct entire missing representations or fuse modalities without distinguishing transferable from modality-specific information” is not accurate.

  3. Regarding the theoretical analysis in Appendix B, I have some difficulty understanding the relevance of the theoretical results to the incomplete multimodal data fusion, particularly in terms of a) prediction/classification performance using incomplete multimodal data, and b) the validity/performance of the proposed method under different missing data mechanisms.

  4. In the first set of experiments, it seems that artificial missing values are generated (masked) under MCAR. Since MCAR is often viewed unrealistic in real data settings, the results from these experiments (Tables 1 and 2) might not provide useful information about real-world usefulness of the proposed method.

  5. In all experiments, existing state-of-the-art methods such as those mentioned in 2) are not included as baselines. This is particularly problematic in the second set of experiments using naturally incomplete medically data, where only FF and SF are included.

Questions for authors
  1. What is the reason for using frozen encoder for each modality? How does this compare with end-to-end algorithms for integration of incomplete multimodal data such as those existing multimodal learning methods mentioned 2) in Critical Weaknesses?

  2. It is unclear whether and why the proposed modeling framework can effectively learn modality-specific and shared branches defined in Equation (2), as there may be some identifiability issues when estimating two latent representations the sum of which is another latent representation.

  3. Can DACE handle missing data within individual data modalities? This is often encountered in incomplete multimodal clinical data such as those used in the second set of experiments in this paper.

Reviewer_CSbGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates multimodal recognition under missing-modality conditions and proposes Divide-and-Compose Evidence (DACE), a lightweight framework that separates observed representations into modality-specific and shared evidence. The method preserves specific evidence locally, estimates the shared components of missing modalities through learned cross-modal transformations, and combines the resulting information using availability-conditioned composition and refinement. The paper also analyzes evidence conservation, missing-input invariance, recovery-error bounds, and bounded residual refinement. Experiments on MM-IMDb, UPMC Food-101, and Hateful Memes cover 27 missingness conditions, with additional evaluations of cross-condition robustness and naturally incomplete medical data. DACE achieves the highest reported scores across the main benchmark comparisons and demonstrates encouraging robustness under varying modality availability.

Critical strengths

The paper addresses a relevant and practical problem in multimodal learning, and its motivation for preserving modality-specific information while selectively recovering transferable evidence is intuitive and well aligned with the missing-modality setting. The proposed framework is technically coherent, with complementary routing, shared-slot recovery, and availability-conditioned composition forming a clearly described computational pipeline. The mathematical formulation is reasonably complete, and the theoretical analysis establishes useful structural properties under the stated assumptions. The empirical evaluation is also a notable strength, covering multiple benchmarks, missingness patterns, and cross-condition transfer settings. The consistently favorable results and progressive ablation study reflect substantial effort and provide useful evidence of the effectiveness of DACE as a recognition framework.

Critical weaknesses
  1. The interpretation of the learned specific/shared decomposition is not fully supported by the proposed formulation. A central premise of DACE is that the complementary router separates modality-specific information from transferable shared evidence. However, the two branches are obtained through an elementwise learned gate and are optimized primarily through the downstream recognition objective. Although the complementary construction guarantees that the two branches sum to the original representation, this algebraic property does not establish that they correspond to distinct semantic categories of information. The paper acknowledges that the decomposition need not be unique, but its interpretation sometimes appears stronger than what the formulation directly guarantees. The reported recognition improvements support the usefulness of the routing mechanism, whereas the claim that the learned branches represent genuinely specific and shared evidence remains less firmly established.
  2. The ablation results suggest that the main improvement arises from feature routing rather than the complete divide-and-compose mechanism. The progressive ablation study shows that the Divide stage contributes the largest performance gain across the three benchmarks, while the subsequent Compose and refinement stages provide smaller improvements. Under mixed 50% missingness, adding Compose improves the reported scores by 0.34, 0.14, and 0.39 points on MM-IMDb, Food-101, and Hateful Memes, respectively. These results demonstrate that shared-slot recovery contributes positively, but they also indicate that most of the observed improvement is associated with the initial feature-routing stage. Since the Divide stage additionally introduces trainable nonlinear transformations, the current results do not clearly isolate the role of the intended evidence separation from the general representational benefits of the routing architecture. Consequently, the empirical findings support the effectiveness of the overall design more strongly than they establish the independent importance of selective shared-evidence recovery.
  3. The theoretical results provide structural guarantees but offer limited support for the central semantic claims. The paper establishes several properties of the proposed construction, including evidence conservation, missing-input invariance, and bounds on recovery error and residual refinement. These results are mathematically useful, but their interpretation should remain closely tied to the assumptions and definitions under which they hold. In particular, evidence conservation follows from the complementary routing design, while missing-input invariance follows from the dependence of the computation on observed inputs. The shared-slot recovery bound is conditional on the quality of the source estimates and learned transformations. These properties do not directly establish that the model has identified transferable semantic information or that recovered shared slots faithfully represent missing-modality evidence. The theoretical contribution is therefore more convincing as an analysis of architectural behavior than as a validation of the proposed semantic decomposition.
  4. The strength of the empirical comparisons should be interpreted with some caution. DACE achieves consistently favorable performance across the reported benchmark conditions, which is an important empirical result. However, the manuscript combines locally evaluated fusion controls with results reported in previous studies, and these comparisons do not necessarily share identical model adaptation strategies or training configurations. In addition, the local Feature Fusion and Score Fusion controls use sampled missingness masks during training, whereas DACE evaluates the expected loss over the specified observed subsets. These distinctions do not invalidate the reported scores, but they make it difficult to attribute the full performance advantage exclusively to the proposed divide-and-compose architecture. The evidence is strongest for DACE's effectiveness under the reported evaluation protocol, while broader claims of architectural superiority require more careful interpretation.
Questions for authors
  1. Interpretation of evidence decomposition: How should the learned specific and shared branches be interpreted given that their separation is induced by complementary gating and downstream supervision rather than an explicit semantic constraint? In particular, which properties of these branches are guaranteed by the formulation, and which represent intended behavior?
  2. Interpretation of the ablation study: Since most of the performance improvement occurs when the Divide stage is introduced, how do the authors interpret the relative importance of complementary routing, shared-slot recovery, and refinement? What conclusions about the divide-and-compose principle are directly supported by the current ablation results?
  3. Scope of the theoretical guarantees: Could the authors clarify the distinction between the structural properties established by the theoretical analysis and the semantic interpretation of the learned evidence? In particular, how should the conditional shared-slot recovery bound be understood in relation to the actual learned cross-modal representations?
  4. Interpretation of benchmark comparisons: How should readers interpret the comparisons between locally trained methods and previously reported baselines, given the differences in model adaptation and training protocols? A clearer discussion of the scope of these comparisons would help align the conclusions with the existing experimental evidence.
Reviewer_K8rmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies multimodal recognition when some modalities are missing at test time. The authors propose DACE that splits each observed modality feature into specific and shared parts. When a modality is missing, only its shared part is recovered from the observed modalities, and the result is composed and refined with availability-conditioned attention. On the image-text benchmarks, DACE is trained as a lightweight head on frozen CLIP features. The paper reports the highest score in every comparison across diverse missingness settings on three image-text benchmarks, and also evaluates on an incomplete renal-cancer cohort.

Critical strengths

The question of which information should stay with each modality and which can be borrowed from the others is a useful way to frame the missing-modality problem, and I agree that this is the direction to move forward. The evaluation includes a real medical cohort with naturally missing data, and the method can be used without knowing the test-time missing rate in advance.

Critical weaknesses

My main concern is the claim that DACE "achieves the highest score in every comparison". The published numbers in that comparison are copied from the MoRA and SyP papers, while DACE and the FF/SF controls run on frozen CLIP features. On Food-101 a linear head on the same features (the FF control) beats the best published method in more than half the settings, and the paper itself says published and local scores "retain their original experimental provenance". So it is hard to tell how DACE really compares with the published methods until one of them is run in the same pipeline. The fair comparison is then against FF and SF, but those are linear heads with far fewer trainable parameters than DACE, and no generic fusion head of similar size is tried. Without such a head, the experiments cannot separate the divide-and-compose design from the added capacity. The ablation has the same problem, since its largest step (Direct to Divide) also adds most of DACE's trainable parameters.

ShaSpec, which the paper cites, already fills a missing modality with the average of the observed modalities' shared features. DACE's Eq. 3 takes the same average after passing each source through a learned transport map, and Appendix A says the map "starts at the identity". In the ablation, shared-slot recovery (Compose over Divide) adds little on each dataset. The ablation does not include ShaSpec's plain average or full-feature recovery, and no other experiment measures what the transport map adds over that average. The abstract says existing methods "either reconstruct entire missing representations or fuse modalities without distinguishing transferable from modality-specific information", but ShaSpec already keeps the two apart and uses only the shared part for a missing modality, and so does DrFuse (Yao et al., 2024) in the two-modality case. And no seed variance is reported either, although several margins are a fraction of a point.


Yao et al., "DrFuse: Learning Disentangled Representation for Clinical Multi-Modal Fusion with Missing Modality and Modal Inconsistency", AAAI 2024.
Questions for authors

Q1. Could you run at least one published method (e.g., MoRA) with the same CLIP ViT-B/16 backbone, splits and seeds as the local methods?

Q2. With the transport maps fixed at the identity (ShaSpec's rule), how much of Compose's gain remains? And what does full-feature recovery give?

Q3. How does a fusion head of similar size to DACE do on the same features?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.