acceptodds
Withdrawn from ICLR 2027

What Should the Missing Modality Say Structured Candidate Recovery for Incomplete Multimodal Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Missing-modality recovery is often underdetermined: the same partial observa-tion can support several task-relevant interpretations of what an absent modal-ity would express. Deterministic reconstruction may collapse these alternatives,while stochastic recovery produces samples without explicitly associating themwith distinct semantic interpretations. We propose HyDiff, which represents thisambiguity with a finite, data-dependent set of structured candidates. A frozenmultimodal reasoner proposes candidates, a learned compatibility scorer assignsoperational weights, and candidate- and target-specific retrieval supplies instance-grounded priors. Conditional diffusion then recovers each missing target underthe retained candidates, after which task predictions are aggregated with theirweights. Candidate-weight entropy is used for selective prediction; the weightsare defined only over the reasoner-generated set rather than treated as a humaninterpretation posterior. The design therefore separates uncertainty over structuredinterpretations from the realization of each recovered modality and keeps compet-ing semantic alternatives explicit rather than leaving them as anonymous stochasticsamples. Experiments on three multimodal sentiment benchmarks evaluate feature-and modality-level missingness with matched-input, sampling, full-anchor, andreduced-anchor controls. When explicit target-facing fields are removed from theshared anchor, HyDiff retains a smaller but consistent advantage over matchedP-RMF, separating semantic assistance from the recovery architecture; additionaltransfer, completion, candidate-coverage, and efficiency diagnostics are reported inthe appendix.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Abductive Reasoning, Retrieval-Augmented Diffusion, Incomplete Multimodal Learning, Self-Reflective Inference, Multimodal Sentiment Analysis, Missing Modality Imputation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_msFmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes HyDiff, a retrieval-augmented framework for incomplete multimodal learning that aims to recover missing modalities by leveraging relevant information from training instances. Motivated by recent retrieval-augmented approaches, HyDiff constructs semantic anchors from the available modalities to guide the retrieval of relevant information from a global memory bank, followed by diffusion-based missing-modality reconstruction. Although the core idea is potentially valuable and achieves promising performance on three MSA benchmarks, the paper contains numerous issues in methodological presentation, notation, experimental design, and result reporting, which substantially undermine the clarity, soundness and credibility of the work.

Critical strengths

To tackle the insufficient information caused by missing modalities, motivated by recent retrieval-augmented learning approaches, the paper leverages relevant instances from the training set to provide additional contextual information for missing modality reconstruction. Specifically, the proposed HyDiff first derives semantic candidates from the available modalities and constructs semantic anchors to guide retrieval from a global memory bank, which enables the model to access more relevant examples for subsequent modality reconstruction.

Critical weaknesses

1.The overall methodological narrative and presentation are weak, with poor consistency and insufficient standardization. More seriously, numerous variables in the methodology section are not briefly defined or accompanied by explanations of their roles, significantly reducing the overall readability and requiring considerable effort from readers to follow the proposed framework. To be more convincing, it is advisable to coherently introduce the motivation at the beginning of each module, such as highlighting the advantages of certain approaches, drawing inspiration from prior work, or explaining how traditional methods often fall short and thus motivating the proposed approach. Furthermore, I consider that it is necessary to clearly and concisely highlight several specific issues that undermine the clarity and rigor of the paper: 1)In Section 3.1, the modality set is defined as ‘M={v,a,txt}’, whereas the experimental section (e.g., Table 1) adopts the notation ‘[L,V,A]’. This inconsistency in modality notation should be clarified, especially since the meanings of these symbols are not explicitly introduced. Moreover, these modality symbols should be briefly explained when they are first introduced. Since HyDiff is mainly evaluated on MSA benchmarks, the lack of a focused discussion on missing-modality research in MSA in the Related Work section may implicitly assume that readers are familiar with this modality convention. 2)The frequently used variables J and \gamma lack clear and consistent definitions, which makes the methodology difficult to follow. Moreover, although Section 3.3 is described as target-specific, the variables in Equations (7)–(8) do not explicitly include the missing target modality index ‘m’. 3)Tthe experimental section directly refers to the ‘pre-rejection \mu=0.7’ setting, while the meaning of this setting and its sampling procedure are mainly explained in the Appendix. This makes the construction of the missing-modality protocol less transparent when reading the experimental setup. Moreover, the construction of the memory bank, which plays a central role in the retrieval component, should be briefly described in the main methodology. Specifically, the paper should clarify what the keys and values represent, how they are obtained from training samples, and whether the memory bank remains fixed or is updated during training and inference. 4)Although the paper compares against 11 representative baselines, more recent competitive approaches in 2026 are not included (such as TMDC: A Two-Stage Modality Denoising and Complementation Framework for Multimodal Sentiment Analysis with Missing and Noisy Modalities). More importantly, it should more clearly explain how the proposed HyDiff differs from the most relevant SOTA approaches. 5)Regarding the query construction in Section 3.3, the main text does not explain that the retrieval space is optimized using a contrastive alignment loss. Furthermore, this loss is mentioned in the appendix but omitted from the formulation of the overall objective function, leading to a discrepancy between the methodological description and the actual optimization process.

2.With respect to experiments, the reported results raise several credibility concerns summarized as follows: 1)Most notably, in Table 1 and several appendix tables, it’s pretty unusual and abnormal that most reported values have final digits of 0, 2, or 5 when averaged over five random seeds. 2)As reported in Table 1, under the full-modality setting, HyDiff with BERT features reaches an Acc-2 of 89.62. Even if Acc-2 is computed by excluding neutral samples (Note that Acc-2 is generally evaluated under two settings: Negative/Non-negative and Negative/Positive), it still seems unreasonable that this result approaches or even surpasses recent works using the stronger DeBERTa backbone. 3)Figure 3 suffers from insufficient resolution, while the information needed to interpret it is deferred to the Appendix. There is also an inconsistency in that Appendix D specifies a sweep over \mu in [0.1, 0.7], whereas figure 3 includes \mu=0. More notably, MissRAG, the strongest baseline in Table 1, is omitted from Figure 3, making it unclear whether HyDiff consistently outperforms the strongest competing method across different missingness levels.

3.This paper uses too much \vspace, and I am unsure whether this complies with the conference requirements.

Questions for authors

1.The total loss in Equation (13) appears to omit the L_{ret} term mentioned in the Appendix. Is this retrieval loss actually involved in the training process?

2.The paper states that the memory bank is constructed from complete training samples, but its construction and update mechanism remain unclear. How is the memory bank initialized, and is it updated during training?

3.The definition of \mu appears inconsistent between Table 1 and Fig. 3. Since Table 1 uses \mu as the pre-rejection modality-drop probability while Fig. 3 refers to it as the ‘missing rate’, how is \mu precisely defined and implemented throughout the experiments?

4.Are all 11 selected baselines strictly evaluated under the same missing-modality protocol as HyDiff and fairly reimplemented by the authors? Are the results reported in Table 1 averaged over multiple random seeds? More notably, why do most Acc-2 values reported in Table 1 and Tables E.3, E.4, and E.6 in the Appendix exhibit decimal endings of 0, 2, or 5?

5.In Table 1, HyDiff with a BERT text encoder achieves an Acc-2 score of 89.62 under the full-modality setting. Even if Acc-2 is computed by excluding neutral samples (Note that Acc-2 is generally evaluated under two settings: Negative/Non-negative and Negative/Positive), it still seems unreasonable that this result approaches or even surpasses recent works using the stronger DeBERTa backbone. Is this performance consistent with the employed text encoder and experimental setting?

6.Since HyDiff introduces a frozen Qwen2.5-Omni-7B model, memory bank construction, and multi-step diffusion sampling during the recovery stage, its overall computational pipeline appears substantially more complex than most baselines. How does the computational complexity and efficiency of HyDiff compare with existing methods, such as number of parameters, training and inference time, FLOPs, and peak GPU memory?

Reviewer_euDROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes HyDiff, a framework for incomplete multimodal learning that explicitly models multiple plausible interpretations of missing modalities. The method employs a pretrained multimodal reasoner to generate structured semantic candidates, which are weighted according to their compatibility with the observed modalities. Candidate-specific retrieval and conditional diffusion are then used to reconstruct missing features, and the resulting predictions are aggregated. The method is evaluated on three multimodal sentiment analysis benchmarks.

Critical strengths
  • Interesting perspective on missing modalities. The paper addresses an important limitation of missing-modality reconstruction: the available observations may support multiple plausible interpretations, making a single deterministic reconstruction potentially inadequate.

  • Distinction between two sources of uncertainty. A valuable conceptual aspect is the distinction between uncertainty over the semantic interpretation of a missing modality and uncertainty over its actual realization. Explicitly representing these two aspects is a reasonable and interesting approach.

Critical weaknesses

1 Limited methodological novelty

The main concern is the limited novelty of the proposed approach. While the overall framework is reasonably motivated, most of its individual components have already been investigated in previous literature.

For instance, multiple-hypothesis prediction and the preservation of alternative outputs have been explored by Bishop (1994) and Rupprecht et al. (2017). Retrieval-augmented generation and diffusion have also been extensively studied, including Blattmann et al. (2022). More directly related to incomplete multimodal learning, methods such as Knowledge Bridger (Ke et al., 2025), RAG4DMC (He et al., 2026), and ANGA (Guan et al., 2026) have already explored combinations of semantic guidance, retrieval, and modality reconstruction.

Although HyDiff introduces a specific candidate-indexed organization of these components, the contribution appears primarily architectural and engineering-oriented, rather than a substantially novel methodological proposal. The individual mechanisms are largely established, and the paper does not sufficiently demonstrate that their integration constitutes a major conceptual advance.

2. Limited presentation quality

The paper is difficult to follow, particularly in the method section. The mathematical formulation is poorly presented, with numerous symbols, functions, and architectural components introduced without adequate definitions or explanations.

Several examples can be found throughout Section 3:

Equations (5)–(6): (FEXT),(fb(ℓ−1)),(pψr),and(Pstruct)(F_{\mathrm{EXT}}), (fb^{(\ell-1)}), (p_{\psi_r}), and (P_{\mathrm{struct}}) are introduced without sufficiently explaining their precise roles and definitions.

Line 224: (Enchyp),(Encm)(\mathrm{Enc}_{\mathrm{hyp}}), (\mathrm{Enc}m), and the compatibility energy (Eω)(E\omega) are used without adequately describing the corresponding encoders and scoring mechanism.

Equation (6): The candidate count (J)(J) and the entropy notation (H(w))(H(w)) would benefit from explicit definitions at first use.

Equations (7)–(8): The retrieval projection (Φproj)(\Phi_{\mathrm{proj}}), the bounds (Kmin⁡)(K_{\min}) and (Kmax⁡)(K_{\max}), the temperature increment (Δσ)(\Delta\sigma), and the retrieved neighborhood notation are insufficiently explained.

Equation (10): The cross-modal bridge (Φcross)(\Phi_{\mathrm{cross}}), the representation (em)(e_m), and the guidance schedule (ηguide(t))(\eta_{\mathrm{guide}}(t)) are introduced with limited explanation.

Although some notation can be inferred from context and some details are delegated to the appendix, the accumulation of insufficiently introduced components makes the method extremely difficult to follow.

The overall exposition is also excessively dense, with numerous mechanisms introduced in rapid succession and without a sufficiently clear explanation of their interactions. This significantly affects readability and makes it difficult not only to understand the proposed framework but also to independently assess its technical soundness and reproducibility.

3. Limited experimental scope and computational concerns

Although the paper reports improvements over existing approaches, their significance should be considered in relation to the substantial complexity of the proposed framework.

HyDiff relies on a large pretrained multimodal reasoner, Qwen2.5-Omni-7B, which is required also at inference time. This introduces potentially substantial computational and memory overhead. While the paper reports some latency measurements, the complete end-to-end computational cost is not sufficiently characterized in the main text, particularly the contribution of the multimodal reasoner, its iterative candidate generation, and the additional retrieval and diffusion branches.

Furthermore, the role and effective use of the external Omni model are not sufficiently clear. Its integration into the overall framework, particularly the extraction and processing of semantic information, would benefit from a more detailed explanation. More generally, the experimental protocols and setup are difficult to follow.

The controlled experiments also reveal that a substantial portion of the performance improvement comes from task-aligned semantic assistance. When this information is restricted, the remaining architecture-specific improvement is relatively modest (approximately 0.62 accuracy points on average). This raises questions about whether the additional architectural and computational complexity is justified by the observed gains.

Finally, the main experimental evaluation is restricted to three sentiment analysis benchmarks: CMU-MOSI, CMU-MOSEI, and CH-SIMS. These datasets involve the same three modalities (text, audio, and video), and CMU-MOSI in particular is relatively small. Consequently, the evaluation provides limited evidence of general applicability to other multimodal learning problems.

The authors should consider extending the experimental evaluation to larger and more diverse benchmarks, different downstream tasks, and additional modality combinations beyond text, audio, and video. Such experiments would be particularly valuable given the generality claimed for the framework.

Questions for authors

See Weaknesses

Reviewer_98CHOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper designs a multimodal learning framework and address the feature-level and modality-level missingness. It has 3 stages where first stage use Qwen2.5 7B model to propose interpretations, second stage searches the similar examples in the memory bank, and third stage use a generative AI model to generate a synthetic version of the missing clue. Finally a classifier predict positive or negative sentiment by weighted average of votes.

Critical strengths

The paper's motivation is clear. It identified point reconstruction may average distinct valid regions, while stochastic sampling leaves alternatives unstructured. And they want to use HyDiff to retain weighted semantic candidates, performs candidate-conditioned target-wise recovery, and aggregates predictions across branches.

Critical weaknesses

When I try to understand the paper and find related work to understand the field, I found one arxiv paper named "OMG-Agent: Toward Robust Missing Modality Generation with Decoupled Coarse-to-Fine Agentic Workflows". One interesting finding I have is.. current paper's baseline rows in HyDiff Table 1 matches OME-Agent's baseline numbers with +0.02 or +0.05 and the method names are changed. For example, for MOSI, OME-Agent MCTN is 56.1, 55.0, 79.1, 57.5, 81.0, 81.1, 81.4 for xxo, xox, oxx, xoo, oxo, oox, ooo but in HyDiff, the number becomes 56.12, 55.05, 79.15, 57.52, 81.05, 81.10, and 81.42 for CorrKD method. This problem also applies to MOSEI where MCTN is labeled as CorrKD, PMSM is labeled as P-RMF in both MOSEI and MOSI.

Questions for authors

I'd like to ask the authors about the numbers especially the random +0.02 or +0.05 with baseline methods label changes compared to the arxiv paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.