acceptodds
Withdrawn from ICLR 2027

Balancing Multimodal Learning via Selective Knowledge Transfer with Path Discrepancy

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Modality imbalance remains a fundamental challenge in multimodal learning, as heterogeneous modalities exhibit disparate learning dynamics and contribute unequally to joint optimization. Existing methods primarily improve learning balance through optimization regulation, while knowledge transfer complements these efforts by strengthening under-optimized modalities with multimodal supervision, typically through representation alignment. However, alignment-based transfer uniformly reduces cross-modal discrepancy without distinguishing knowledge deficiency caused by insufficient modality learning from modality-conditioned variation arising through cross-modal interaction, thereby conflating information that should be transferred with information that should remain distinctive. We propose Path-Discrepancy Guided Selective Transfer (PDST), which exploits transfer-path consistency to determine how different cross-modal knowledge should be utilized for balanced multimodal learning. By comparing direct transfer with its modality-conditioned counterpart under identical source and target spaces, PDST identifies route-stable information as path-invariant knowledge and interaction-induced variation as path-dependent knowledge. The resulting discrepancy-guided mechanism selectively transfers path-invariant knowledge to compensate for insufficient modality learning while preserving path-dependent knowledge to maintain complementary cross-modal information. Extensive experiments on multiple multimodal datasets demonstrate consistent improvements over representative balanced multimodal learning methods, establishing selective knowledge transfer guided by path discrepancy as an effective approach to mitigating modality imbalance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Learning, Knowledge Distillation, Modality Imbalance

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kHv2OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

PDST introduces a selective knowledge transfer framework for balanced multimodal learning, distinguishing transferable knowledge from interaction-dependent variation through direct and mediated transfer paths. Path-invariant consensus determines the distillation target, while path discrepancy adaptively controls supervision strength. Extensive experiments across six benchmarks demonstrate strong performance, supported by systematic ablations and representation analyses.

Critical strengths
  1. Insightful formulation. Rather than treating all cross-modal discrepancies as knowledge deficiencies, PDST introduces path consistency as a criterion for selective transfer. Jointly determining what to transfer and how strongly to transfer it offers a compelling perspective beyond conventional alignment-based distillation.
  2. Convincing empirical support. Strong results across diverse multimodal benchmarks, together with targeted ablations, transfer-benefit correlations, and representation analyses, provide complementary evidence for the proposed mechanism. Particularly encouraging are the substantial improvements over vanilla distillation and the absence of additional inference overhead.
Critical weaknesses
  1. Limited theoretical characterization. Proposition 1 provides a geometric decomposition, but the connection between path consistency and semantic transferability remains primarily empirical. A clearer discussion of the assumptions underlying this interpretation would strengthen the contribution.
  2. Incomplete practical analysis. Direct measurements of unimodal learning balance and additional training costs would help clarify the mechanism and efficiency, although the existing experimental evidence already supports the effectiveness of selective transfer.
Questions for authors
  1. Could the authors clarify how the mediated transfer path captures modality-conditioned variation without explicitly incorporating the intermediate modality representation?
  2. Can improvements in individual unimodal branches be reported to further substantiate the modality-balancing interpretation?
  3. How much additional training computation does the auxiliary transfer framework require?
Reviewer_C6VnOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper addresses modality imbalance via cross-modal knowledge distillation, where path-discrepancy modeling is proposed to determine the transferability of knowledge in the fused representation and selective distillation is adopted to construct the teacher model and control the distillation strength. Experiments were conducted for evaluating performance on six datasets, covering audio-visual, image-text, and trimodal scenarios, and performance gain reported. Also, ablation studies were carefully conducted.

Critical strengths
  1. The idea of selecting only useful knowledge in the fused representation suitable for distillation under the context of multimodal learning is interesting.
  2. The authors conduct experiments on six datasets, covering audio-visual, image-text, and trimodal scenarios. In addition, they provide comprehensive ablation studies.
Critical weaknesses
  1. The claimed properties of the proposed PDST should be more carefully elaborated and justified. In Proposition 1, the authors discuss the orthogonality and squared-norm relationships between two features, but it is not clear why these properties can prove that qinvq_{inv} represents modality-invariant knowledge. It is not clear how to ensure that these cycle-consistency constraints can learn modality-invariant knowledge.

  2. The paper is not well organized and is very hard to read. For example, the concept of "transfer path" is not clearly explained (may consider using a real case to make the explanation less abstract). The notions like "modality mediation", "route-consistent", "path-dependent", "modality-conditioned", etc. should be carefully elaborated and explain how all these together develop the proposed idea. Another example: In Section 3.2, concepts of "state", "route" should be explained. r is used without definition. But then it is defined later on in Section 3.3. In general, the logical flow should be improved with self-contained context for understanding the paper. Also, the rationale behind the design should be explained before providing the technical details for the whole paper.

Questions for authors

Question 1: Referring to Section 3.2, is the information that some modalities fail to fully learn necessarily the information that the fused representation can provide? Could that still be modality-specific information?

Question 2: You use cycle consistency to ensure that each modality can reconstruct itself, but why not introduce direct-route consistency regularization to ensure cross-modal reconstruction?

Question 3: What are the network architectures of the direct and mediated routes? Are the choices crucial and in what sense? Can these routes be trained end-to-end?

Reviewer_o6GhOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes Path-Discrepancy Guided Selective Transfer (PDST) for balanced multimodal learning. It compares direct and mediated transfer paths, uses their consensus as a distillation target, and adjusts the supervision strength according to path discrepancy. The method is evaluated on six multimodal classification benchmarks.

Critical strengths
  1. The paper considers cross-modal information discrepancies during self-distillation, which is meaningful.
  2. The experiments cover multiple multimodal datasets and report improvements over conventional fusion.
Critical weaknesses
  1. In the related studies, MAG-MS [1] uses multimodal self-distillation to support segmentation with missing modalities, while CMRF [2] uses feature distillation to improve multimodal domain generalization. Although this paper identifies modality imbalance as its motivation, it does not sufficiently explain why its particular self-distillation scheme should improve the fused multimodal classifier. The connection between the distillation objective and improved multimodal classification remains insufficiently justified.
  2. The transport objective does not consider an obvious trivial solution. Setting all transport operators Fs→tF_{s→t} to the identity yields Lcyc=0L_{cyc} = 0 and makes the direct and mediated paths identical, giving δ=0\delta= 0 without requiring the operators to learn cross-modal correspondences. Note that under the stated objectives, LcycL_{cyc} is the only loss term that provides gradients to these operators: the other terms either do not involve them or block gradients to them through stop-gradient operations. The paper does not sufficiently justify the design of the transport operators and their associated training objective. In particular, it does not explain what prevents this degenerate transport configuration or ensures that the learned mappings capture meaningful cross-modal relationships. In this configuration, the path-discrepancy-based mechanism loses its intended selectivity, as every sample receives no path discrepancy. This leaves a central component of the method insufficiently justified.
  3. The paper lacks a clear method figure or algorithm flowchart, making it difficult to understand the complete algorithmic procedure.
  4. The method design lacks sufficient motivation and support.

[1] He Q, Summerfield N, Dong M, et al. Modality-agnostic learning for medical image segmentation using multi-modality self-distillation[C]//2024 IEEE International Symposium on Biomedical Imaging (ISBI). IEEE, 2024: 1-5.

[2] Fan, Yunfeng, et al. "Cross-modal representation flattening for multi-modal domain generalization." Advances in Neural Information Processing Systems 37 (2024): 66773-66795.

Questions for authors
  1. What specifically motivates the use of the proposed self-distillation scheme to improve the classification performance of the fused multimodal classifier?
  2. What prevents all transport operators from becoming identity mappings when cycle consistency is their only direct training signal? How does the objective ensure meaningful cross-modal transport rather than merely cycle-consistent mappings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.