acceptodds
Withdrawn from ICLR 2027

Synergy-Aware Multimodal Disentanglement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Disentangling a multimodal representation into factors for different information types makes it inspectable and reusable: the evidence each factor carries can be probed or recombined. Partial Information Decomposition (PID) provides natural coordinates for this, dividing task information into redundant, unique, and synergistic parts. However, to our knowledge, existing self-supervised multimodal disentanglement stops at a shared/private split, and interaction-oriented methods capture synergy only inside one fused embedding. We propose **URSyn** (**U**nique, **R**edundant, **Syn**ergistic), a self-supervised framework that learns separately accessible factors for all three types: unique factors are repelled from redundant ones, redundant factors are supported by every single modality, and the synergy factor is anchored to the full input while repelled from all single-modality references. We show that idealized versions of these constraints bound the task information of the redundant and synergy factors by redundancy- and synergy-related information quantities, without requiring PID values to be estimated. On a controlled benchmark, each factor best predicts its designated target, and the synergy factor *alone* reaches 83.5% on a joint-only task, exceeding the complete representations of CoMM (70.4%) and InfMasking (77.0%). On six real-world benchmarks, **URSyn** matches or exceeds our CoMM reproduction in mean performance under the same protocol and is close to InfMasking's reported results.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal Representation Learning, Disentangled Representation Learning, Self-Supervised Learning, Partial Information Decomposition, Multimodal Synergy

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_aHMBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a synergy-aware self-supervised framework (URSyn) for multimodal disentanglement. Concretely, it learns separately accessible factors for all three types: unique factors, redundant factors and synergy factor. Experimental results show the effectiveness of the proposed method.

Critical strengths
  1. The paper considers the synergy factor which is usually neglected in the methods with a conventional shared-private split.
  2. Various experiments were conducted to demonstrate the effectiveness of the method.
Critical weaknesses

By stating “URSyn is motivated by two idealized constraints: a redundant factor should be recoverable from every singleton, and a synergy factor should share no information with any singleton.”, it means that only the information that are consistent to all modalities are redundant. This assumption is not always hold in reality.

Questions for authors
  1. By stating “URSyn is motivated by two idealized constraints: a redundant factor should be recoverable from every singleton, and a synergy factor should share no information with any singleton.”, it means that only the information that are consistent to all modalities are redundant. How can we verify that these constraints are valid in real applications? Would this method fail if there is no cross-model consistent shared information?

  2. How are the distinctions between dividing information into redundant, unique, synergy, and categorizing it as task-related or task-irrelevant? Why can the former improve the performance of downstream tasks?

Reviewer_paXDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes URSyn, a self-supervised framework that aims to organize multimodal information into separately accessible unique, redundant, and synergistic factors. Using a shared backbone evaluated on singleton and full-modality inputs, it combines contrastive information preservation with objectives for agreement and geometric separation. Redundant factors are encouraged to agree across modalities, unique factors are repelled from redundant factors, and the synergy factor is anchored to the full-view representation while being repelled from singleton references. A downstream relative-diversity routing rule reweights the frozen factors before prediction. Experiments on Trifeature show target-selective factors and improved performance on the joint-only target, while evaluations on six real-world benchmarks report competitive downstream performance.

Critical strengths
  1. The factor-level analysis on Trifeature. Table 7 shows that each factor best predicts its designated target, and ZSZ_S alone achieves 83.46%83.46\% accuracy on the joint-only task, close to 83.56%83.56\% using all factors. These results provide concrete evidence that the trained model learns representations with different predictive strengths, supporting the usefulness of separately accessible factors on this controlled benchmark.

  2. The evaluation extends to six real-world benchmarks. URSyn matches or improves on the reproduced CoMM baseline in reported mean performance, using the same encoders and fusion backbone. Results are reported over five random seeds with standard deviations. This provides empirical support for the usefulness of the complete framework beyond the controlled Trifeature setting.

Critical weaknesses
  1. The motivation for the synergy objective is not sufficiently justified. The authors aim to concentrate joint-only information in ZSZ_S, but its key separation term penalizes cosine similarity with singleton references. As the authors acknowledge in Section 3.2, this similarity can decrease while the encoded information remains unchanged. Full-view anchoring does not resolve this issue, since the full representation also contains redundant and unique information. The paper therefore needs to explain why these coupled objectives should favor synergistic information rather than a different encoding of singleton-accessible content.

  2. The paper does not clearly explain which prediction task defines the information that its branches are intended to separate. In Section 2.1, the argument assumes that the information shared by augmented views is relevant to the target task. However, the Trifeature evaluation uses the same pretrained model for shape, texture, and joint targets. Information useful for one of these tasks can be irrelevant to another: for example, texture is needed for the texture probes but is irrelevant to predicting shape. The authors need to clarify which target their pretraining assumptions refer to and how the resulting factors should be interpreted across these different tasks. This missing connection weakens the justification for the proposed information-based division of the branches.

Questions for authors
  1. Table 1 asks ZUiZ_{U_i} to preserve Ai=R+UiA_i=R+U_i while satisfying I(ZUi;ZR)=0I(Z_{U_i};Z_R)=0. Does preservation mean retaining all singleton-accessible information, including RR, or maximizing retained information subject to excluding ZRZ_R? These are different objectives, and the intended relationship between the two requirements needs to be stated explicitly.

  2. In Eq. (6), is the full-view reference ZZ updated by the anchor loss? Eq. (7) gives ZZ no separate information-preservation objective, as also reflected in Table 12. What makes ZZ a reference that grounds ZSZ_S in full-view information, rather than another representation that adapts to ZSZ_S?

  3. Figure 1 shows separate forward paths through the shared Transformer but does not show the branch-specific computations. Which operations turn the same singleton backbone output into ZiZ_i, ZUiZ_{U_i}, and ZRiZ_{R_i}, and where do the full-view paths producing ZZ and ZSZ_S diverge? Please make these mappings explicit, including their relationship to the shared projection head.

  4. The paper omits some relevant work on multimodal factorization and information bottlenecks, including MFM (Tsai et al., 2019), MIB (Federici et al., 2020), and ITHP (Xiao et al., 2024). How does URSyn differ from these approaches in the information it seeks to preserve or suppress and the assumptions supporting those choices? Clarifying these connections would help establish the specific contribution of the explicit synergy factor.

Reviewer_h53LOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes URSyn, a self-supervised multimodal representation learning framework that explicitly separates unique, redundant, and synergistic information into individually accessible factors. Inspired by Partial Information Decomposition, URSyn combines contrastive preservation objectives, cross-modal redundancy alignment, and a synergy-specific objective that anchors joint information while repelling unimodal representations. The authors also introduce Relative-Diversity Routing for downstream factor recombination. Experiments on Trifeature and six real-world benchmarks demonstrate improved synergy-specific representation quality and competitive downstream performance against CoMM, InfMasking, and other multimodal learning methods.

Critical strengths
  • Meaningful formulation of synergy-aware disentanglement. URSyn extends shared/private multimodal representation learning by explicitly introducing a dedicated synergy factor. The distinction between capturing synergistic information in a fused representation and making it separately accessible is well motivated and represents a useful methodological contribution.

  • Convincing evidence of factor selectivity on controlled data. On Trifeature, individual factors achieve their strongest performance on the corresponding redundancy, uniqueness, or synergy targets. In particular, the synergy factor alone reaches 83.46% accuracy on the joint-only task, outperforming the complete representations of CoMM and InfMasking. This directly supports the proposed factor-specific learning objective.

  • Broad empirical evaluation and careful theoretical qualification. The experiments cover multiple multimodal benchmarks with five-seed evaluations, objective ablations, and routing analyses. The theoretical proposition also explicitly distinguishes idealized PID-related constraints from the weaker trainable objectives, avoiding unsupported claims of exact information decomposition.

Critical weaknesses
  • The learned factors are not guaranteed to correspond to distinct PID components. The theoretical results provide upper bounds under idealized information-access constraints, but the practical cosine-based objectives do not enforce these conditions. In particular, geometrically separated representations may still carry overlapping information. The factor-removal results in Table 7 also show that unique and redundant information remains recoverable after removing its designated factor. Although the paper acknowledges this limitation, the extent to which URSyn achieves genuine information separation remains unclear.

  • The contribution of the synergy-specific learning mechanism is insufficiently isolated. The controlled experiments demonstrate strong synergy representation quality, but the objective ablations mainly evaluate overall accuracy on real-world datasets rather than factor-level selectivity. Consequently, it remains unclear whether the improvement comes specifically from singleton repulsion, full-view anchoring, entropy regularization, or simply introducing an additional joint representation branch. Matched-capacity controls with simpler factorization objectives would strengthen the methodological claims.

  • Limited evidence of practical advantages over strong multimodal baselines. On real-world benchmarks, URSyn provides relatively modest improvements over CoMM and performs similarly to InfMasking, while requiring substantially greater training computation. Moreover, factor-level interpretability is evaluated mainly on the synthetic Trifeature benchmark, and the improvements from Relative-Diversity Routing are generally small relative to experimental variance. Stronger evidence of factor reuse or robustness on real-world data would better justify the additional complexity.

Questions for authors
  • Can the authors provide additional quantitative measures of information leakage between the learned unique, redundant, and synergy factors, particularly beyond linear probing on Trifeature?

  • How does synergy-factor accuracy change when removing singleton repulsion, full-view anchoring, or entropy regularization individually? Would a CoMM backbone with an additional joint representation head achieve comparable results under matched capacity?

  • Can the authors evaluate factor-specific information contributions on real-world benchmarks through modality interventions, controlled ablations, or factor-level probing?

  • Given the additional training cost, what practical advantages does URSyn provide over InfMasking or capacity-matched CoMM beyond comparable downstream accuracy? Can the authors provide stronger statistical evidence for the routing improvements?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.