acceptodds
Under review as a conference paper at ICLR 2027

SynCo: Learning Cross-Modal Synergy by Contrasting Interaction Residuals

Abstract

Multimodal contrastive learning is a dominant paradigm for learning transferable representations from unlabeled data, but standard objectives primarily capture information that is redundant between modalities. Partial Information Decomposition (PID) shows that task-relevant information in multimodal data decomposes into three components: redundancy shared between modalities, uniqueness specific to each modality, and synergy available only from their joint observation. Recent frameworks extend contrastive learning to capture all three components, yet synergy remains undertrained in practice. We propose SynCo (Synergy Contrastive Learning), a method that directly addresses synergy undertraining through dedicated supervision on an interaction residual. SynCo fits a linear projector to predict the fused representation from independently computed unimodal features, and the resulting interaction residual, which removes the linearly unimodal-predictable component, receives dedicated contrastive supervision at negligible computational cost. On the controlled Trifeature benchmark, SynCo achieves state-of-the-art synergy capture with a gain over the baseline, and on real-world benchmarks from MultiBench, DARai, and MM-IMDb, SynCo consistently outperforms or matches prior methods across diverse modality combinations and task types. The method operates as a plug-in to existing contrastive multimodal frameworks without modifying the underlying fusion architecture and can further improve synergy capture when combined with other methods.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multi Modal, Partial Information Decomposition, Synergy, Fusion

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vkSeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SynCo, which extends the multimodal contrastive learning objective of CoMM by introducing a linear predictor P that predicts the fused representation from individual unimodal representations. The difference between the fused representation and its prediction is used to construct an interaction residual. During training, an MSE loss is used to fit the predictor, while an additional residual InfoNCE loss provides contrastive supervision across two augmented views. Stop-gradient operations are employed to separate the gradient effects of these two losses.

Motivated by Partial Information Decomposition (PID), the authors introduce Hypothesis 1 and derive a lower bound on the conditional mutual information of the residual under this assumption, providing a theoretical motivation for encouraging synergy learning.

The method is evaluated through redundancy, unique-information, and synergy probes on Trifeature, as well as frozen-encoder linear probing on MultiBench, DARai, and MM-IMDb. The paper also reports ablations on projector capacity, loss components, warmup, and computational overhead.

Critical strengths
  1. Simple mechanism with low computational overhead. SynCo does not modify the existing fusion network and only introduces a linear predictor, a prediction error loss, and a residual contrastive loss. The prediction loss updates P, while the interaction loss applies stop-gradient to the predicted representation, making the method relatively easy to integrate into existing training frameworks. Under the specific configuration reported in Appendix C, the per-step runtime increases only from 21.91 ms for CoMM to 21.94 ms for SynCo, indicating good computational efficiency.

  2. Clear results from controlled experiments, with evaluations spanning multiple modalities and tasks. On Trifeature, SynCo achieves a synergy-label linear-probing accuracy of 77.58 ± 0.74%, outperforming CoMM (71.60 ± 1.07%) and the paper's reproduced InfMasking baseline (76.12 ± 1.42%), while maintaining good performance on redundancy and unique-information probes.

Critical weaknesses

The independent contribution of the residual mechanism could be further clarified. The paper already provides reasonably extensive experimental evidence for the effectiveness of SynCo. However, Appendix B.4 does not include a comparison between applying an additional InfoNCE loss directly to the original fused representation and applying it to the interaction residual. Including this simple control experiment would help isolate the contribution of residual decomposition itself to the observed performance gains and further strengthen the justification for the proposed design.

Questions for authors

Q1. Advantages of interaction residuals over direct contrastive supervision. Appendix B.4 already examines the effects of the different loss components in SynCo. Have the authors considered applying an additional InfoNCE loss directly to the original fused representation without residual decomposition? Including this simple comparison would help clarify the distinctive value of the interaction residual design. A convincing result demonstrating the benefit of residual decomposition would address my main concern and could lead me to raise my rating.

Reviewer_NqvzOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the limited learning of synergistic information in multimodal contrastive learning. The authors propose SynCo, which constructs an interaction residual by removing the part of the fused representation that can be linearly predicted from unimodal representations. An additional contrastive loss is applied to the residuals to encourage the learning of cross-modal synergy.

Critical strengths

Improving the learning of synergistic information in multimodal contrastive learning is an important and interesting problem.

The proposed interaction residual provides a simple and intuitive way to encourage cross-modal interactions, without changing the underlying fusion architecture.

SynCo shows consistent improvements over CoMM on both controlled and real-world benchmarks.

Critical weaknesses

W1. Limited evidence for the key assumption. Hypothesis 1 assumes that the linear prediction of the fused representation cannot capture synergistic information. However, this assumption is mainly designed and supported by linear probing results on Trifeature. Linear probing accuracy does not directly measure mutual information, and the assumption may not hold for more general data distributions. Therefore, the generality of the theoretical results remains unclear.

W2. The interaction residual does not necessarily represent synergy. The residual contains information that cannot be recovered by the linear predictor, but this information is not necessarily synergistic. As shown in Table 2, the residual still achieves 81.1% and 57.4% accuracy on redundancy and uniqueness, respectively. This suggests that the residual also contains redundant and unique information, making it unclear whether the proposed method specifically targets synergy.

W3. The connection between the theoretical analysis and the objective is not sufficiently clear. Theorem 1 provides a lower bound on the conditional mutual information I(Zres;Y∣Z^)I(Z_{res};Y\mid\hat Z), while the proposed interaction loss maximizes an InfoNCE lower bound on I(Zres′;Zres′′)I(Z_{res}';Z_{res}''). And Corollary 1 does not fully establish that maximizing residual agreement specifically can improve synergy.

Questions for authors

Q1. Assumption 1 requires augmented views to preserve all task-relevant information while removing task-irrelevant information. Since synergy is defined with respect to a downstream target YY, how can such augmentations be designed or justified when YY is unknown during self-supervised pretraining?

Q2. The theoretical analysis relies on strong augmentation assumptions that may not hold in practice. Could the authors evaluate SynCo under different levels of synergy-preserving and synergy-destroying augmentations to examine how these assumptions affect the results?

Q3. Figure 1 (page 5) is somewhat difficult to follow due to the many connections between different modules.

1 comment

Dear Reviewer, thank you for your time and valuable feedback. We are writing to gently follow up on our submitted rebuttal. We wonder if our response has addressed your concerns and clarified the issues raised.

If you have any remaining questions, please feel free to let us know—we would be more than happy to provide further clarification. If our response has resolved your concerns, we would be very grateful if you could kindly consider updating your score.

Thank you again for your constructive review.

Reviewer_jtXCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that synergy stays undertrained in multimodal contrastive frameworks such as CoMM. SynCo adds a "Synergy Head" to CoMM. A linear projector predicts the fused representation from the unimodal ones, and an extra contrastive loss acts on the residuals of two augmented views. A theorem states that, if the linear prediction holds only redundant and unique information, the residual carries the synergistic task information up to a slack term. Experiments use linear probing on synthetic Trifeature data and several real benchmarks.

Critical strengths
  1. The method is simple, cheap and orthogonal to existing work. It adds one linear layer and two losses, with only about 0.15% extra time per step over CoMM, far less than InfMasking (Table 10). It can also be stacked on InfMasking.
  2. The Trifeature analysis supports the intended mechanism. In Table 2, the concatenated unimodal features and the linear prediction stay at chance on synergy, while the residual reaches 87.24%. Table 5 shows that larger projectors absorb synergy into the prediction. The evaluation covers several modality types.
Critical weaknesses
  1. The information-theoretic justification does not match the evidence or the loss. Hypothesis 1 and Theorem 1 are stated in Shannon mutual information but are backed only by linear probes. On Trifeature the Shannon version plausibly fails, since the unimodal representations determine the synergy label and the projector is generically injective, so Z^\hat Z would in general contain the synergy too, and the conditional term in Theorem 1 would be near zero. Table 2 only shows that synergy is not linearly decodable from Z^\hat Z, which is linearly usable information [1], not Shannon information. Hypothesis 1 is also not checked on any real dataset. A second gap is that Theorem 1 is conditional on Z^\hat Z, while the loss maximizes an unconditional InfoNCE between residuals. Corollary 1 only shows that the shared residual information is task-relevant, not that it is synergistic, and Table 2 shows the residual still carries redundancy and uniqueness. Since the theory is the stated reason the loss is "synergy-targeted", this gap is serious.
  2. The ablations do not isolate the residual subtraction. Unlike the CoMM loss, the new loss works in the raw fusion space. The key missing control is the same InfoNCE on the raw fused views without subtracting Z^\hat Z. Without it, the gain could come from an extra raw-space contrastive term. Figure 2b is partial evidence but not this control. On real benchmarks nothing separates more synergy from generic regularization, and the closely related EMAP [2] is not discussed.
  3. Some results are internally inconsistent, and margins are small relative to variance and the tuning protocol. In Figure 2a, a zero interaction weight gives about 58% synergy. With gradient isolation this should match CoMM, which Tables 1 and 8 report at 71.6%. The interaction weight and warmup were also chosen by maximizing Trifeature synergy, so the headline result seems tuned on the evaluation metric. Several key gaps, such as the synergy gain over InfMasking and on MM-IMDb, are about one standard deviation or less, and SynCo+InfMasking is sometimes below SynCo alone. The abstract's claims of consistent gains and of further improvement when combined overstate this.

References

[1] Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. "A Theory of Usable Information under Computational Constraints." International Conference on Learning Representations (ICLR), 2020.

[2] Jack Hessel and Lillian Lee. "Does my multimodal model learn cross-modal interactions? It's harder to tell than you might think!" Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.

Questions for authors
  1. Can you restate the theory in terms of linearly usable information and connect the conditional bound to the unconditional loss that is optimized?
  2. Can you report the same InfoNCE on the raw fused views without subtracting Z^\hat Z, on Trifeature and at least one real dataset? Can you add an analysis as in [2] on real data?
  3. Why does a zero interaction weight in Figure 2a fall below CoMM in Table 1? Were the weight and warmup chosen on a held-out split, and are the main gains significant across seeds?
1 comment

Dear Reviewer, thank you for your time and valuable feedback. We are writing to gently follow up on our submitted rebuttal. We wonder if our response has addressed your concerns and clarified the issues raised.

If you have any remaining questions, please feel free to let us know—we would be more than happy to provide further clarification. If our response has resolved your concerns, we would be very grateful if you could kindly consider updating your score.

Thank you again for your constructive review.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Revisions

Dear Area Chairs,

As we approach the end of the discussion period, we would like to summarize how our revisions and additional experiments address the reviewers’ concerns. We have posted detailed responses to all questions and incorporated the revisions into the updated manuscript, with changes highlighted in blue. Reviewer vkSe confirmed that the additional comparison addressed their main concern and raised their rating from weak rejection to weak acceptance. We have not yet received responses from Reviewers jtXC and Nqvz to our rebuttal or follow-up messages, so we have not had further discussion of the revisions with them. We remain available to answer any questions or clarify any remaining concerns before the discussion closes.

Reviewer vkSe confirmed that the additional comparison addressed their main concern and raised their rating from weak rejection to weak acceptance. We have not yet received any follow-up comments from Reviewers jtXC and Nqvz and remain available for further questions.

The main revisions are as follows:

  • Theoretical justification — jtXC and Nqvz. We addressed the mismatch between the original theoretical framing and linear-probe evidence by reformulating the residual analysis using linearly usable information. We also clarified the distinction between the theorem’s conditional bound and the unconditional InfoNCE objective, and explained why synergy-focused supervision does not require a pure-synergy residual. These revisions align the theoretical claim with the evaluation and make its scope explicit. Sections 3 and 4.2 and Appendix D have been updated; the method and training objectives remain unchanged.

  • Contribution of residual subtraction — jtXC and vkSe. We addressed the concern that the gains might come merely from an additional contrastive loss by running the requested comparison with InfoNCE applied directly to fused representations, keeping the remaining training settings unchanged. SynCo outperforms this comparison by 5.30 percentage points on Trifeature synergy and 3.80 points on MOSI, supporting the contribution of residual subtraction. The results appear in Appendices B.4 and B.7. Reviewer vkSe explicitly confirmed that this experiment addressed their main concern.

  • Evidence of cross-modal interactions on real data — jtXC. We addressed the request for evidence beyond downstream accuracy by adding an EMAP analysis on MOSI. SynCo’s advantage over additional InfoNCE on fused representations does not persist after additive projection, supporting a contribution from useful non-additive interactions. This analysis strengthens the empirical explanation of the improvement without treating EMAP as a numerical estimate of PID synergy. The procedure and results appear in Appendix B.7.

  • Augmentation assumptions — Nqvz. We addressed concerns about the strong augmentation assumption by clarifying its idealized scope and adding experiments that preserve or disrupt cross-modal relationships at different intervention strengths. SynCo retains an advantage under partial disruption, while both methods approach chance under complete disruption. These experiments directly examine the requested sensitivity and show the limits of robustness when relevant relationships are removed. The results appear in Appendix B.8, with the assumption discussed in Appendix E.

  • Evaluation consistency and hyperparameter selection — jtXC. We addressed the apparent inconsistency in Figure 2 by clarifying that the figure reports validation performance, whereas the main tables report test performance. We also reported a zero-interaction-weight control on the same test set, obtaining 71.6 ± 0.81%, consistent with CoMM. To address the concern about tuning on the evaluation set, we documented held-out validation selection and the use of fixed settings across all seven real-world datasets without dataset-specific retuning.

  • Statistical significance and comparison claims — jtXC. We addressed concerns about small margins and overly broad claims by assessing gains across runs and distinguishing statistically significant improvements from differences in reported means. We also clarified the comparison baselines and revised the wording to reflect that adding SynCo to InfMasking improves most evaluated configurations but does not consistently outperform standalone SynCo.

We thank the reviewers and Area Chairs for their time and consideration. We remain available to clarify any aspect of the revisions or additional results before the discussion closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.