acceptodds
Under review as a conference paper at ICLR 2027

TWIN-MoE: Two-stage Intra-Expert Imputation Network for Incomplete Multimodal Learning

Abstract

Multimodal learning is often deployed under the assumption that all modalities are fully observed, yet real-world data frequently violates this assumption, leaving some modalities missing. Recent Mixture-of-Experts (MoE) approaches address missing modalities either by routing each missingness pattern to a dedicated expert, which scales poorly as the number of modalities grows, or by imputing the missing modality from representations contributed by multiple experts, which requires a costly retrieval mechanism. We argue that such unconstrained inter-expert imputation can be sub-optimal, as it aggregates features from distinct routing-induced semantic regions and may introduce information unrelated to the target modality. Building on a geometric interpretation of sparse MoE routing through expert-specific Voronoi regions, we show that routing induces expert-conditioned representation neighborhoods and derive an imputation-error bound controlled by their local feature dispersion. When these expert-conditioned neighborhoods are more compact than the global representation space, intra-expert aggregation admits a tighter worst-case reconstruction bound than unconstrained global aggregation. Guided by this analysis, we propose TWIN, a two-stage intra-expert imputation framework. Pre-imputation generates a context-conditioned estimate in the encoder embedding space and radially soft-clips it using modality-specific statistics maintained in an observed FIFO buffer, this ensures that the pre-imputed modality stays in empirical observed distribution. Post-imputation refines the missing representation after MoE by aggregating expert-conditioned observed representations within the same sample. Across four missing-modality benchmarks under natural, random, and asymmetric dropout protocols, TWIN achieves state-of-the-art performance while using substantially fewer parameters than competing MoE-based imputation methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Learning, Mixture-of-Expert, Missing Modality

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MjtsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the incomplete multimodal learning problem and proposes TWIN, a two-stage Mixture-of-Experts (MoE) imputation framework. The authors interpret the sparse MoE routing mechanism as a Voronoi geometric partition of the feature space, and theoretically prove that intra-expert imputation has a tighter worst-case error bound than unconstrained global inter-expert imputation.

Critical strengths
  1. The attempt to interpret the semantic locality of MoE routing from a geometric perspective provides intuitive inspiration for missing modality imputation, and the idea is internally consistent to some extent.
  2. The two-stage imputation design has clear logic. The combination of distribution constraint and local aggregation is engineering-wise easy to implement and requires only minor modifications to the MoE backbone.
Critical weaknesses
  1. The theoretical contribution is contrived, severely disconnected from the actual method, and has extremely limited applicability. The conclusion in Proposition 1 that sparse routing is equivalent to Voronoi space partitioning strictly relies on a very specific routing bias setting. Standard sparse MoE uses learnable routing biases end-to-end instead of this fixed form. The paper only vaguely mentions dynamically computing to achieve Voronoi routing, but fails to explain how this modification affects the representational capacity of MoE itself, nor does it verify whether the Voronoi geometric assumption still holds under standard learnable routing.
  2. The extension to Soft MoE completely lacks theoretical support. Soft MoE adopts an expert-choice token routing paradigm and does not have discrete Voronoi cell structures. The migration of the locality principle has no geometric basis and is only empirically tested through simple experiments, resulting in severely insufficient theoretical rigor.
  3. Experimental comparisons are severely incomplete, baseline selection is outdated, and core conclusions are unconvincing. Comparisons with current mainstream technical lines are entirely missing. The paper does not include any generative multimodal completion methods based on diffusion models or flow matching, nor does it compare with recent self-supervised missing modality learning methods based on masked modeling. Against the background that generative completion has become a core track in incomplete multimodal learning, only comparing MoE-based methods from 2021–2025 and early traditional methods completely fails to support the claim of state-of-the-art.
  4. The evaluation dimension is single, and the fairness of comparison is questionable. The paper only evaluates imputation quality by downstream classification metrics, without setting reconstruction metrics (such as MSE, cosine similarity) to directly measure imputation accuracy, making it impossible to conduct horizontal comparison with generative methods focusing on signal reconstruction.
Questions for authors

Please see Weaknesses.

Reviewer_e3puOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TWIN-MoE, a two-stage intra-expert imputation framework for missing modalities. The authors argue that intra-expert aggregation can yield better reconstruction than unconstrained global aggregation. Motivated by this analysis, TWIN-MoE consists of two stages. Pre-imputation generates and refines a context-conditioned estimate to keep the imputed representation within the target modality's feature space. Post-imputation further refines this representation by aggregating expert-conditioned representations of the observed modalities from the same sample.

Experiments on four datasets show improvements over the evaluated baselines.

Critical strengths

S1: Imputing missing modalities within MoE-based methods is well-motivated and practically relevant.

S2: Section 4.1 provides some empirical evidence supporting intra-expert routing.

S3: TWIN-MoE improves upon the baselines evaluated in this paper.

Critical weaknesses
  • W1: The theoretical justification for the method is unconvincing. Although Remark 1 shows that setting cj=ckc_j=c_k eliminates the term ∣∣cj−ck∣∣2||c_j-c_k||_2, it does not account for changes in ∣∣hv,g−cj∣∣2||h_{v,g}-c_j||_2 and ∣∣ck−hv,t∣∣2||c_k-h_{v,t}||_2. Specifically, it does not establish that ∣∣hv,g−cj∣∣2+∣∣cj−hv,t∣∣2||h_{v,g}-c_j||_2+||c_j-h_{v,t}||_2 is smaller than ∣∣hv,g−cj∣∣2+∣∣ck−cj∣∣2+∣∣ck−hv,t∣∣2||h_{v,g}-c_j||_2+||c_k-c_j||_2+||c_k-h_{v,t}||_2 when ckc_k is closer to hv,th_{v,t}. Furthermore, Fig. 3 shows that the centroids of the imputed modalities differ from the original centroids, which appears inconsistent with the authors' assumption.

  • W2: The relative performance of TWIN-SF and TWIN-SP varies across datasets (Tables 1 and 2). What distinguishes these variants, and what explains why one outperforms the other on a given dataset?

  • W3: Tables 1 and 2 should include results under different missing-modality conditions to clarify how missingness affects each method. An analysis of the FIFO queue size is also needed, as this may affect imputation performance.

  • W4: Equations 7–9 appear to assume that the training data form clusters. However, Fig. 3(b) does not show a clear cluster structure, and the loss terms do not appear to guarantee such a structure.

Questions for authors

Q1: Could the authors clarify the theoretical argument and address the concerns in W1?

Q2: Could the authors explain the differences between TWIN-SF and TWIN-SP and discuss their relative performance?

Q3: Could the authors report results for each missing-modality condition and analyze the effect of the FIFO queue size?

Q4: Could the authors justify the cluster assumption underlying Equations 7–9?

1 comment

Dear Reviewer,

As the end of rebuttal is approaching, we kindly request you to engage with us for further discussion and we are open and happy to address any concern you have.

Kind regards

Reviewer_oy73OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes TWIN, a framework for handling missing modalities in multimodal Mixture-of-Experts (MoE) architectures. The core idea is a two-stage imputation strategy consisting of (1) locality- and distribution-aware pre-imputation and (2) post-imputation through expert-conditioned refinement after routing. The method is motivated by the observation that missing-modality representations should respect the local geometry induced by MoE routing. Extensive experiments are conducted across healthcare and multimodal benchmark datasets under several missing-modality settings. I found this paper well motivated and technically interesting. The empirical results are generally good but there are some aspects of the motivation and experimental evidence benefit from further validation and clarification.

Critical strengths
  • Interesting perspective on modality imputation in MoE models. The two-step setup with geometric interpretation. Sparsity component is not necessarily new though (e.g., X-MoE).
  • Strong empirical performance. Across multiple datasets and missingness scenarios TWIN generally achieves competitive performance while often with fewer parameters (only thing 50% missing scenario on one dataset though). Ablation experiments show that pre- and post- imputation together has the most effective.
Critical weaknesses

The central routing-locality hypothesis is not sufficiently validated. The motivation rests on the claim that unconstrained inter-expert imputation can be harmful because representations may be reconstructed using information originating from semantically different routing regions. However, it is unclear whether this phenomenon is empirically established. The supporting evidence relies primarily on UMAP visualizations, which are known to be sensitive to hyperparameter choices and may not faithfully preserve distances or neighborhood structure in the original embedding space. And, stronger quantitative evidence demonstrating that inter-expert imputation degrades representation quality would strengthen the claim.

The relationship between SoftMoE and SparseMoE is unclear. Although the proposed TWIN framework is motivated by locality induced through sparse expert routing, the experimental results frequently favor the SoftMoE variant. Examples include

  • Table 8 (MIMIC-IV), where TWIN-SF generally appears stronger,
  • Table 9 (CMU-MOSI), where TWIN-SF also performs favorably,
  • Additional supplementary results where TWIN-SF frequently outperforms TWIN-SP, and
  • Setting 3 (modality dropout during training/testing), where TWIN-SF again appears especially strong. The paper would benefit from a discussion of when sparsity is beneficial, why soft routing performs better in several settings, and which component of TWIN is primarily responsible for the observed gains.

Minor: Table 14 appears to focus primarily on SparseMoE. Given that TWIN-SF frequently performs better than TWIN-SP in several benchmarks, it would be useful to include analogous analyses for the SoftMoE variant as well.

Questions for authors

Most of my questions are reflected in the weaknesses above, particularly regarding (1) empirical validation of the routing-locality hypothesis, and (2) interpretation of the SoftMoE versus SparseMoE results.

2 comments

Thank you for your response and clarification. I appreciate the effort the authors put into addressing the reviewers' concerns. While my overall assessment has not changed substantially after reviewing the rebuttal and the discussion among reviewers, I continue to view the paper favorably despite some remaining limitations. I will therefore maintain my current Weak Accept score.

Dear reviewer,

Thank you for taking the time to read our rebuttal and share your assessment. We considered your suggestions valuable and constructive to further revise the paper.

Kind regards

Comments on the paper

1 comment

Dear Reviewers,

As the discussion phase is coming to the end, we would like to request if there is any furhter concern that we can address. It will be great to hear your reponse to our rebuttal and looking forward to your engagement.

Kind regards Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.