acceptodds
Withdrawn from ICLR 2027

SelRef: Selective Privileged Refinement for Multimodal Learning with Missing Modalities

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal learning has achieved remarkable success across a broad range of applications, yet its performance often deteriorates when one or more modalities are missing at inference. Fully observed multimodal data available during training can serve as privileged supervision. Existing distillation-based methods primarily transfer such knowledge by aligning predictions or representations between complete and incomplete inputs, leaving the usefulness of teacher guidance for each observed subset insufficiently explored. To tackle this limitation, we propose SelRef, a unified Selective Privileged Refinement framework that selectively exploits full-modality supervision to refine incomplete-modality predictions. SelRef first constructs a subset-conditioned prediction anchor by adaptively aggregating available modality experts learns an anchor-relative correction from their intermediate representations. During training, Privileged Task Gain and Current-State Compatibility jointly select teacher directions that improve upon the anchor and remain compatible with local descent at the refined prediction. We theoretically characterize the local descent properties of these directions and demonstrate the effectiveness of SelRef on three benchmarks spanning medical image segmentation and multimodal classification under complete and missing modality settings.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Learning, Missing-Modality Learning, Learning Using Privileged Information

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_9GWSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SelRef to handle missing modalities at test time. It first combines predictions from the available modalities, then uses a refiner to improve them. During training, it aligns the refiner’s correction direction with that of a full-modality teacher, using only selective guidance judged to be helpful. The method is comprehensively evaluated on BraTS2023, UPMC Food-101, and MM-IMDb.

Critical strengths

The evaluation is fairly comprehensive, covering three tasks spanning brain tumor segmentation and image-text classification. It includes comparisons with recent methods, and SelRef achieves the best average performance among the compared baselines on all three main benchmarks: BraTS2023, UPMC Food-101, and MM-IMDb.

Critical weaknesses
  1. The benefit of selective privileged guidance, the core contribution of SelRef, is not sufficiently discussed. In Table 3, task-only refinement already improves average Dice substantially over the Prediction Anchor (83.42% vs. 80.55%) and comes close to full SelRef (83.86%). A comparison using the same refiner and full-modality teacher with a stronger general distillation method, such as RLD (Knowledge Distillation with Refined Logits; ICCV 2025), would help clarify why SelRef’s guidance design is particularly useful for missing-modality learning and what it offers beyond standard teacher supervision. More analysis of how Privileged Task Gain and Current-State Compatibility affect teacher guidance and the resulting predictions would further support this core contribution.

  2. It is unclear whether the gains on the main task, medical image segmentation, hold across datasets. On BraTS2020 (Table 8), SelRef performs worse than the best competing methods on ET and TC, while its WT advantage is also marginal. The paper should discuss why the improvements on BraTS2023 do not carry over consistently to BraTS2020.

Questions for authors

My questions mainly reflect the concerns in Critical Weaknesses. I hope the authors can address these concerns either by answering the questions below or in another proper way.

  1. How would SelRef compare with using the same refiner and full-modality teacher, but replacing selective privileged guidance with a stronger general distillation method? For example, Knowledge Distillation with Refined Logits (RLD, ICCV 2025) also handles incorrect predictions from strong teachers. This comparison would help show why SelRef’s guidance is particularly useful when modalities are missing. Further discussion of this point would also be welcome.

  2. What happens if Current-State Compatibility is removed while Privileged Task Gain is kept? Section E.5 shows that 40.97% of task units have positive gain, while 40.91% pass both conditions. Since Compatibility removes very few additional units, what benefit could it provide specifically?

  3. Why are the results different on BraTS2020 and BraTS2023? On BraTS2020 (Table 8), SelRef is below the best competing methods on ET and TC segmentation, and its WT advantage is also marginal. What might explain why the gains on BraTS2023 do not carry over consistently?

  4. Could the authors explain SimMLM’s Food-101 results in Table 2? It has the highest text-only accuracy (86.27%), but its image-only accuracy (65.70%) is much lower than all the other methods (around 77% or higher). This large gap seems unusual. Could it be due to a reporting error, insufficient training, differences in the experimental settings, or a limitation of SimMLM itself?

Reviewer_orEVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SelRef, a unified privileged refinement framework for multimodal learning with missing modalities. SelRef aggregates available modality experts to form a subset-conditioned prediction anchor and learns an anchor-relative correction. Privileged Task Gain identifies task units that may benefit from teacher guidance, while Current-State Compatibility assesses its suitability for the current model. An information-theoretic analysis motivates using the loss difference between the teacher and the observed subset to identify useful teacher predictions.

Critical strengths
  • S1: SelRef addresses a meaningful limitation of multimodal distillation by selecting teacher guidance based on its benefit for the observed modality subset.

  • S2: The paper provides theoretical analysis to support the model design.

  • S3: The evaluation covers both classification and segmentation tasks.

Critical weaknesses
  • W1: In Table 1, SelRef underperforms existing baselines in several missing-modality settings for TC. Its reported Dice improvements are also modest: 0.84 percentage points for ET, 0.27 for TC, and 0.7 for WT. Table 2 shows similarly small gains. Without further evidence, it is difficult to attribute these gains to the proposed design rather than run-to-run variation or hyperparameter choices.

  • W2: The architecture is insufficiently specified, including the modality encoders, modality-specific experts, and mapping module in Figure 1. The substantially lower computational cost also requires explanation, particularly whether it includes training the complete-modality teacher.

  • W3: Visualizations of successful and failed anchor refinements (Equation 2) are needed to assess how the refinement network affects predictions.

  • W4: In Equation 10, LprivL_{priv} involves a product of four weights, but their individual contributions and the consequences of a zero weight are unclear. Its scale relative to LtaskL_{task} is also unspecified.

Questions for authors
  • Q1: Can the authors provide statistical analysis of the results and discuss why in some scenarios SelRef underperforms other baselines.

  • Q2 : Could you specify the architectures and clarify whether the computational cost includes teacher training?

  • Q3 : Could you show successful and failed refinements, comparing predictions before and after refinement?

  • Q4 : How does each weight affect performance, what happens when one is zero, and how does Lpriv\mathcal{L}_{priv} compare in scale to Ltask\mathcal{L}_{task}?

Reviewer_nYBoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript proposes a framework for multimodal prediction when some modalities are missing at inference.

Critical strengths

The proposed method aggregates predictions from the available modality experts into a subset-conditioned anchor, and then learns a residual correction from their intermediate representations. It also introduces a frozen full-modality teacher to supervise the correction direction, with guidance weighted according to whether the teacher improves on the anchor and whether its direction agrees with local descent at the refined prediction. The paper provides a geometric analysis and evaluates the method on brain tumour segmentation and image-text classification.

Critical weaknesses

My main concerns are as follows.

  1. Privileged directional supervision may struggle to train the refinement network. For binary cross-entropy with a hard target, the gradient sign does not change with the prediction, so a positive-gain teacher direction remains descending at every finite score; the compatibility test cannot reject it based on the current state.

  2. The authors need to explain the actual implementation as essential implementation details are missing. Appendix C.4 does not have details despite mentioning 'details' in the title. The manuscript does not adequately describe the architectures and reproducible settings. This prevents readers from reconstructing the method.

  3. The theoretical results do not establish improvement from the implemented refinement. Proposition 1 follows directly from the first-order convexity inequality, while Proposition 2 expresses the usual directional-derivative condition for local descent. Proposition 3 provides a sufficient angular condition, but the training objective does not enforce that condition, and the experiments do not report how often it holds. More fundamentally, these results concern sufficiently small movements in score space, whereas inference applies a learned correction with unrestricted magnitude.

  4. The comparisons do not isolate the proposed advance over existing teacher-guided methods. The classification experiments mainly compare against fusion, representation-learning, and expert-aggregation methods, although selective privileged supervision is the key contribution.

There are some minor writing issues.

  1. Missing words and grammatical errors, e.g., by adaptively aggregating available modality experts learns, the refinement network therefore is initialized, those whose direction remain; Defered ->deferred.

  2. Citation formatting. Several related-work sentences place author year citations directly after phrases without suitable punctuation; Pipoli et al.(Pipoli et al., 2025).

  3. Figure 1 contains small labels and a dense legend, while Figure 3 is particularly small beside Table 4. Both would benefit from larger text and simpler layouts. Figure 2 seems being generated by AI.

Questions for authors

Please see the major weaknesses above.

Reviewer_m6EcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses multimodal learning where all modalities are available during training but some are missing at inference, and uses the full-modality data as privileged supervision. The proposed SelRef combines the available modality experts into an anchor prediction and learns a correction on top of the anchor. During training, a frozen full-modality teacher sets the direction of the correction, but only at the units (voxels, examples or example-label pairs) where the teacher has lower loss than the anchor and where that direction still lowers the loss at the current prediction. Across brain tumor segmentation and multimodal classification, the paper reports the best average scores among the compared methods, with and without missing modalities.

Critical strengths

The paper points out that existing distillation methods mostly align the student with the teacher for every subset of observed modalities without checking whether the guidance actually helps. The paper also measures how often the refinement makes a case worse than the anchor, and mean Dice alone would not show this.

Critical weaknesses
  • The selective teacher guidance is the central new part of SelRef, but the guidance adds little to mean Dice. In the BraTS ablation, task-only refinement reaches 83.42 average Dice and the full method 83.86, and the classification ablations show similarly small gains. On Food-101 and MM-IMDb the anchor alone already beats every baseline on average. The λpriv=0\lambda_{\text{priv}} = 0 point in Fig. 4, which should be the same task-only setting, comes out lower than Table 3's task-only number by more than half of the BraTS gain from guidance.

  • The other evidence is the harmful-correction rate in Appendix E, which drops by a few points. But it comes from one diagnostic run with a bootstrap over test subjects, so it says nothing about variation across training runs, and no run-to-run variance is reported anywhere. On BraTS, guidance without selection is about the same as no guidance, and the paper does not say whether λpriv\lambda_{\text{priv}} was retuned for it. In the post-hoc check in Table 14, the compatibility test almost never excludes a unit that passes the gain test.

  • The main text says the hyperparameter settings are in the appendix, but I couldn't find the learning rates or optimizer settings there, nor the network architectures or whether the baselines other than SFusion were retrained. The hyperparameter sensitivity plot reports test Dice, and I could not find how the final values were chosen.

Questions for authors

Q1. Does the reduction in harmful corrections hold across training seeds, and what is the run-to-run variance in Table 3? Is the task-only point in Fig. 4 the same setting as in Table 3? If so, why the gap?

Q2. Since the selection weights in Eq. 10 also scale Lpriv\mathcal{L}_{\text{priv}}, was λpriv\lambda_{\text{priv}} re-tuned for the variant without guidance selection?

Q3. What learning rates, architectures and other training hyperparameters were used, and how were they chosen? Were the baselines other than SFusion retrained with the same splits and preprocessing?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.