- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes DACE (Divide-and-Compose Evidence) for multimodal recognition under missing modalities. A complementary gate divides each observed representation into branches assigned specific and shared roles. When a modality is absent, learned cross-modal transformations estimate only its shared branch, leaving its specific branch unestimated. The evidence is composed according to the availability mask and refined using mask-conditioned attention and observed-pair agreement.
DACE is evaluated on MM-IMDb, Food-101, Hateful Memes, and naturally incomplete MMIST-ccRCC. The three image-text benchmarks use frozen CLIP ViT-B/16-QuickGELU features and a recognition head with fewer than one million trainable parameters. DACE has the highest displayed score in all 27 main image-text conditions. The paper also evaluates cross-rate transfer and natural medical missingness. The displayed ranking is encouraging, although the historical baseline scores do not constitute a fully controlled comparison.
The design principle is useful, but the experiments do not yet isolate whether the gains arise from meaningful shared/private information separation, increased head capacity, or the training treatment of missing subsets. The theoretical properties characterize the architecture; they do not by themselves establish the semantic interpretation of the branches.
Critical strengths
The divide–recover–compose design is conceptually coherent and addresses a genuine weakness of full-feature reconstruction. Recovering only evidence expected to transfer between modalities is a sensible inductive bias: genuinely modality-private information need not be reconstructible from other inputs. The staged ablation is directionally encouraging, with Divide giving the largest improvement and Compose/refinement adding gains. However, the capacity increase at the Divide stage prevents a clean attribution of that improvement to decomposition alone. The evaluation covers three image-text tasks, three missingness patterns, three rates, cross-rate transfer, and a naturally incomplete medical dataset. Table 2 is especially useful because it evaluates a model trained at mixed 70% missingness across different test conditions. This breadth supports the practical value of the architecture. The medical experiment is useful evidence on naturally incomplete data, while the small, imbalanced cohort limits broader conclusions.
Critical weaknesses
-
The semantic interpretation of the specific/shared branches is insufficiently validated. Equation (2) enforces a complementary algebraic partition, but does not establish that one branch contains transferable cross-modal information and the other modality-private cues. The paper explicitly avoids assuming a unique semantic decomposition, including beyond initialization; I am not asking for a proof of unique identifiability. The unresolved issue is whether the learned branches empirically perform the semantic roles used to motivate the method. Figure 4 provides qualitative support, but t-SNE separation or proximity cannot establish modality specificity or successful cross-modal transport. Section 4.5 and the caption broadly identify the processing stages; reproducibility would improve by specifying the exact tensors, samples, normalization, and whether all compared representations use one common projection. The claimed semantic interpretation needs direct measurements beyond the visualization. Useful tests include predicting the paired modality's features or cross-modal retrieval from each branch with matched-capacity probes, branch-removal interventions, and a controlled task with known shared and modality-private factors. Such tests should distinguish transferable information from task-specific private cues. Existing shared-specific approaches, especially ShaSpec, also deserve a more precise mechanism-level comparison.
-
Capacity, training-risk, and baseline-protocol differences confound attribution of the performance gains. Table 5 already reports parameter counts and reveals a substantial confound in the staged ablation: on MM-IMDb, Direct has 0.027968M parameters and Divide has 0.749888M; on Hateful Memes, the corresponding counts are 0.002432M and 0.724352M. The largest performance jump therefore coincides with a large capacity increase. A parameter-matched nonlinear head without the specific/shared split, together with fixed or random complementary gates, is needed to isolate the proposed division mechanism. Section 4.1 also states that FF/SF use one sampled incomplete mask per training example, whereas DACE evaluates the weighted subset risk exactly. This is a meaningful training difference even when the nominal missingness rates agree. Please compare local controls under the same subset-risk objective and explain which original modalities remain accessible during training. The current description raises a data-access and attribution question; it does not establish leakage. The main comparison reproduces published MoRA/SyP-era results while the local models use frozen CLIP features. The paper acknowledges this provenance and describes the missingness cases, rates, seeds, and dataset splits. The remaining issue is whether exact sample masks, paired-data access, model selection, and training budgets are comparable. DACE's highest displayed values are valid observations, but stronger claims of controlled superiority require a reproducible comparison. A focused rerun of a strong recent baseline under shared data splits, missingness masks, backbone where feasible, and model-selection rules would be more informative than requiring every method to adopt an identical freezing policy that may change its native design. Report each method's adaptation and compute budget. A relevant shared-specific baseline such as ShaSpec would further test the claimed mechanism; Unseen Modality Interaction is relevant context, but should not automatically be treated as the same type of explicit factorization. Two reporting inconsistencies also need resolution. FF on Hateful Memes at mixed 50% is 68.94 in Table 1 but 71.13 in Table 4. In addition, Section 4.2 interprets Food-101 as visually dominant, although the reported 90% conditions give 78.89 with text missing versus 87.15 with images missing. Please reconcile the FF settings and correct the missing-modality interpretation, checking both the I/T labels and the narrative.
[1] Hu Wang, Yuanhong Chen, Congbo Ma, Jodie Avery, Louise Hull, and Gustavo Carneiro. Multimodal learning with missing modality via shared-specific feature modelling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15878–15887, 2023. [2] Yunhua Zhang, Hazel Doughty, and Cees Snoek. Learning unseen modality interaction. volume 36, pp. 54716–54726, 2023.
Questions for authors
Clarification requests
-
Can the authors reconcile the FF Hateful Memes scores in Tables 1 and 4 and the Food-101 modality interpretation in Section 4.2? If these results use different protocols, please identify those differences explicitly.
-
For the exact subset-risk training in Equation (10), are complete original pairs accessible before masks are applied? Can FF/SF and the capacity-matched nonlinear control be trained under the same objective and data access? Please distinguish this from the fixed train/validation/test splits already documented in Appendix C.
-
What direct evidence shows that the designated shared branch is more transferable than the specific branch? Please identify the exact tensors and common projection protocol used in Figure 4, and avoid interpreting t-SNE geometry as proof of semantic separation.
-
Which published baseline scores use the same actual missingness masks and paired training information as DACE? For the principal comparison, please report variability across the three stated seeds and, where feasible, a strong baseline rerun with matched masks. Small margins should be interpreted relative to that variability.