acceptodds
Under review as a conference paper at ICLR 2027

HiCTE: Hierarchical Cross-Expert Transferability Estimation for Incomplete Multimodal Representation Learning

Abstract

Incomplete multimodal representation learning aims to construct robust representations from partially observed inputs. Existing expert-based methods learn joint representations through cross-expert interaction, but typically transfer entire observed representations to different target experts. Such coarse-grained transfer overlooks target-specific transferability differences at the latent-dimension level and sample-specific transfer requirements. To address these limitations, we propose the Hierarchical Cross-Expert Transferability Estimation (HiCTE) framework. Specifically, HiCTE models each observed modality in a probabilistic latent space and derives reliability from the latent variance. A global estimator then provides initial transferability estimates for each target expert using the latent information and reliability. These estimates are further refined for each sample through local iterative optimization during training, with gradients propagated through the updates to train the global estimator. Based on these estimates, HiCTE constructs cross-expert representations for each target expert and a residual representation, then integrates them with the self representation through adaptive weighting. Extensive experiments on three multimodal benchmarks demonstrate the effectiveness of HiCTE under different missing protocols.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Representation Learning, Incomplete Multimodal Learning, Transferability Estimation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_tik8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HiCTE addresses incomplete multimodal learning by selectively transferring an observed modality's latent dimensions to different target experts. It constructs probabilistic latent representations, uses latent variance as a reliability proxy, and predicts target-conditioned, sample-dependent gates. During training, these gates undergo supervised local refinement, with gradients propagated through the updates to train the global estimator. Self-expert, cross-expert, and residual representations are adaptively combined for prediction. At inference, the method uses the global estimator without label-dependent local optimization. Experiments cover MOSI, MOSEI, and IEMOCAP under fixed modality absence and random missing-rate protocols, with an additional UR-FUNNY evaluation. The results support improvements from selective transfer; local refinement provides smaller additional gains.

Critical strengths
  1. Selective cross-expert transfer has concrete empirical support. The method makes a meaningful distinction between weighting expert outputs and selecting source dimensions before they enter each target expert. Relative to Whole Transfer, fixed-missing average accuracy improves by 1.32, 0.51, and 1.10 percentage points on MOSI, MOSEI, and IEMOCAP. Relative to a target-agnostic shared selector, the corresponding gains are 0.93, 0.44, and 0.53 pp. These controls support the usefulness of selection and its target dependence, beyond comparisons of complete systems alone.

  2. The evaluation separates deployable gains from training-time computation. The full system improves fixed-missing average accuracy over the strongest listed baseline by 1.42, 1.07, and 2.20 pp on the three main datasets. The additional UR-FUNNY comparison explicitly matches features, splits, and missing conditions and reports 63.50 accuracy versus 62.31 for MoMKE. The K=0 and stop-gradient controls help separate the contribution of training-time refinement and gradient propagation to the global estimator used at inference. The paper also reports the extra training cost of local refinement. This provides a useful and transparent assessment of practical behavior.

Critical weaknesses
  1. The added value of the proposed transferability formulation beyond simpler target-conditioned gating remains only partially isolated. MoMKE already processes available inputs through different modality experts and combines their outputs using input-dependent routing. HiCTE's target-specific pre-expert dimension selection is a concrete extension, and the Whole Transfer and Shared Selector controls support selective transfer and target conditioning. The reliability ablations also provide useful component-level evidence. However, these comparisons do not directly establish the full formulation's advantage over a simpler target-conditioned dimension-wise selector trained through the downstream prediction objective, while retaining the same expert and fusion architecture. Such a control would help distinguish the benefit of learned feature selection from the additional value of probabilistic reliability modeling and transferability-specific objectives. K=0 does not provide this comparison: it removes local updates while retaining the reliability-aware formulation and initialization objective. On top of this attribution gap, the benefit attributable specifically to local refinement is 0.18–0.98 pp across the six dataset/protocol average-accuracy comparisons. The consistent positive direction is encouraging, but uncertainty of the paired differences across the three seeds is not reported. The reported refinement cost increases training time by 45.8%, without adding inference-time iteration. Overall, the experiments support a useful selective-transfer method more clearly than the added value of its particular transferability formulation.

  2. The main evaluation and model-selection protocols are insufficiently specified. For the three main datasets, the manuscript should identify which baselines were rerun, their input features, and whether complete modalities were available during training. The matched UR-FUNNY protocol addresses that experiment but does not resolve the main-table questions. In addition, the IEMOCAP description holds out one session for testing and uses the other four for training without specifying validation selection. The data used for the hyperparameter choices in Figure 3 and the checkpoint-selection criterion are also unclear. These omissions limit reproducibility and confidence in the reported differences; they do not establish that test data were used for tuning.

Questions for authors
  1. Clarification using existing runs: Could you report paired per-seed differences, with their variability, for the full model versus K=0, stop-gradient refinement, and the target-agnostic selector? The existing three seeds would be sufficient for this request. These results would clarify the magnitude and stability of the gains from target conditioning and differentiable refinement.

  2. Clarification of evaluation protocol: Please specify the main baselines' features, training-time modality availability, and whether results were rerun or taken from prior reports. How are validation data defined for each IEMOCAP fold, which data and metric determine the Figure 3 hyperparameters and final checkpoints, and are checkpoints shared across missing conditions? A fixed-final-epoch procedure, if used, should simply be stated. The UR-FUNNY matching already described need not be repeated.

Reviewer_tiyTOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes HiCTE, an incomplete multimodal representation learning framework that estimates target-conditioned, dimension-wise cross-expert transferability using variance-derived latent reliability and K-step differentiably unrolled local refinement during training, validated on CMU-MOSI, CMU-MOSEI, IEMOCAP, and UR-FUNNY.

Critical strengths
  • Extending expert-based incomplete multimodal learning from coarse whole-representation routing to dimension-wise, target-conditioned transferability estimation is well-motivated and clearly presented.
  • The ablation studies and diagnostic analyses—such as swapping target selectors and tracking latent variance across increasing test missing rates—are well-designed and help illustrate the intended behavior of each module.
Critical weaknesses
  • Incomplete Experimental Comparisons with Recent Baselines: The experimental evaluation in Table 1 and Figure 2 omits several highly relevant and competitive recent baselines for multimodal sentiment analysis and incomplete representation learning. To convincingly establish state-of-the-art performance, the authors should include quantitative comparisons against strong recent methods from the same period, such as CorrKD (Li et al., CVPR 2024), MPLMM (Guo et al., ACL 2024), DecAlign (Qian et al., ICLR 2026), and CA-ACR (Wang et al., CVPR 2026). Without benchmarking against these recent decoupled alignment and cross-modal reconstruction approaches, it remains unclear whether the proposed cross-expert transferability mechanism truly outperforms current state-of-the-art methods.
  • Train-Test Discrepancy in Local Refinement: During training, the representation paths rely on the K-step locally refined transferability scores optimized using ground-truth labels (Eq. 9 and Eq. 11), whereas at inference time the model falls back to the unrefined initial estimates from the global estimator because labels are unavailable. The paper does not quantify the distribution gap between the initial and refined estimates or evaluate whether downstream adapters overfit to label-refined masks that cannot be produced at test time.
  • Formulation Inconsistency Between Sample-Specific Motivation and Batch-Averaged Updates: While Section 3.2.3 motivates local refinement as capturing sample-specific transfer requirements, Eq. 11 and Algorithm 1 (Lines 17-18) compute the gradient of the mini-batch averaged objective rather than per-sample objectives, which scales down the effective inner step size by the number of valid utterances and makes a sample's refinement trajectory dependent on batch size and batch composition.
  • Heuristic Regularization and Unfair Protocol Assumptions: The transferability objective (Eq. 9) forces the average transferability across dimensions toward an arbitrary constant of 0.5 regardless of modality quality, and the random missing-rate protocol strictly assumes matched training and testing missing rates in the main accuracy benchmarks rather than evaluating accuracy under mismatched test-time corruption.
Questions for authors
  1. Could the authors provide quantitative experimental comparisons on CMU-MOSI and CMU-MOSEI against recent competitive baselines)?
  2. How large is the discrepancy (e.g., L1 or cosine distance) between the initial transferability estimate used at test time and the K-step label-refined estimate used during training, and what happens if a test-time label-free objective (using only the variance and scale regularizers in Eq. 9) is applied for K-step refinement at inference?
  3. Why does Eq. 11 differentiate the mini-batch averaged objective instead of the per-sample objective, and does this cause the effective inner-loop step size to vary with batch size and modality missingness rate?
  4. Following the setup in Figure 5 (training with a fixed missing rate of 0.3 and testing across missing rates from 0.0 to 0.7), what is the actual downstream ACC and F1 performance of HiCTE compared to baselines under mismatched train-test missing rates?
Reviewer_gTHXOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents HiCTE, a hierarchical cross-expert transferability estimation framework for incomplete multimodal representation learning. It introduces probabilistic latent reliability modeling and a global-local iterative optimization mechanism to estimate target-specific transferability at the latent-dimension level. The resulting representations are integrated through adaptive path weighting. Experiments under different datasets show gains over baselines.

Critical strengths

1.The motivation for modeling cross-expert transferability is reasonable. 2.The experimental results are competitive across multiple benchmarks and missing conditions.

Critical weaknesses

1.The methodological novelty is somewhat incremental. The framework mainly combines probabilistic latent modeling, dimension-wise gating, iterative optimization, and adaptive expert fusion, where the relationship of diverse modules should be better clarified. The distinction from existing expert-routing and feature-selection approaches could be better established. 2.The interpretation of latent variance as reliability needs stronger justification. Although the appendix shows that variance increases with missingness, this does not necessarily establish that low-variance dimensions are more informative or transferable for downstream prediction. 3.The performance-efficiency trade-off needs further consideration. The proposed local refinement introduces additional training cost, while some improvements over strong baselines are relatively modest. Moreover, although results are averaged across multiple seeds, standard deviations and statistical significance tests are not reported. 4.Lack of related informative-based incomplete multimodal learning methods comparison, including: [1]CyIN: Cyclic Informative Latent Space for Bridging Complete and Incomplete Multimodal Learning, Ronghao Lin and Qiaolin He and Sijie Mai and Ying Zeng and Aolin Xiong and Li Huang and Yap-Peng Tan and Haifeng Hu,The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025 [2]Huilin Chen, Miaomiao Cai, Fan Liu, Zhiyong Cheng, Richang Hong, and Meng Wang. 2025. I3-MRec: Invariant Learning with Information Bottleneck for Incomplete Modality Recommendation. In Proceedings of the 33rd ACM International Conference on Multimedia (MM '25). Association for Computing Machinery, New York, NY, USA, 6133–6142. https://doi.org/10.1145/3746027.3755410 [3]Aoqiang Zhu, Min Hu, Xiaohua Wang, Jiaoyun Yang, Yiming Tang, and Ning An. 2025. Proxy-Driven Robust Multimodal Sentiment Analysis with Incomplete Data. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 22123–22138, Vienna, Austria. Association for Computational Linguistics. [4]Z. Xie, Y. Yang, J. Wang, X. Liu and X. Li, "Trustworthy Multimodal Fusion for Sentiment Analysis in Ordinal Sentiment Space," in IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 8, pp. 7657-7670, Aug. 2024, doi: 10.1109/TCSVT.2024.3376564. [5]R. Lin, Q. He, Y. Zeng, S. Mai, L. Huang and a. H. Hu, "Affection-guided Bottleneck Diffusion for Missing Modality Issue in Multimodal Affective Computing," in IEEE Transactions on Affective Computing, doi: 10.1109/TAFFC.2025.3632063.

Questions for authors

1.Could the authors compare the prediction performance obtained using the initial global estimates and the locally refined estimates on training or validation samples? 2.How does HiCTE perform when trained at one missing rate but evaluated across different missing rates without retraining? 3.Have the authors investigated more structured missing patterns, such as temporally continuous missing segments or mixed modality-level and feature-level missingness? 4.Could the authors provide visualizations of the learned dimension-wise transferability patterns across different emotion categories or input conditions? 5.How does HiCTE perform under complete multimodal inputs? Can the same trained model handle both complete and incomplete observations? 6.How does the residual representation contribute when the two target-specific transferability estimates are simultaneously high? 7.Could the authors provide the inference latency and memory consumption compared with MoMKE and other lightweight baselines? 8.Could the author compare the proposed method with the pre-mentioned informative-based methods to show the presented effectiveness?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.