- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SynCo, which extends the multimodal contrastive learning objective of CoMM by introducing a linear predictor P that predicts the fused representation from individual unimodal representations. The difference between the fused representation and its prediction is used to construct an interaction residual. During training, an MSE loss is used to fit the predictor, while an additional residual InfoNCE loss provides contrastive supervision across two augmented views. Stop-gradient operations are employed to separate the gradient effects of these two losses.
Motivated by Partial Information Decomposition (PID), the authors introduce Hypothesis 1 and derive a lower bound on the conditional mutual information of the residual under this assumption, providing a theoretical motivation for encouraging synergy learning.
The method is evaluated through redundancy, unique-information, and synergy probes on Trifeature, as well as frozen-encoder linear probing on MultiBench, DARai, and MM-IMDb. The paper also reports ablations on projector capacity, loss components, warmup, and computational overhead.
Critical strengths
-
Simple mechanism with low computational overhead. SynCo does not modify the existing fusion network and only introduces a linear predictor, a prediction error loss, and a residual contrastive loss. The prediction loss updates P, while the interaction loss applies stop-gradient to the predicted representation, making the method relatively easy to integrate into existing training frameworks. Under the specific configuration reported in Appendix C, the per-step runtime increases only from 21.91 ms for CoMM to 21.94 ms for SynCo, indicating good computational efficiency.
-
Clear results from controlled experiments, with evaluations spanning multiple modalities and tasks. On Trifeature, SynCo achieves a synergy-label linear-probing accuracy of 77.58 ± 0.74%, outperforming CoMM (71.60 ± 1.07%) and the paper's reproduced InfMasking baseline (76.12 ± 1.42%), while maintaining good performance on redundancy and unique-information probes.
Critical weaknesses
The independent contribution of the residual mechanism could be further clarified. The paper already provides reasonably extensive experimental evidence for the effectiveness of SynCo. However, Appendix B.4 does not include a comparison between applying an additional InfoNCE loss directly to the original fused representation and applying it to the interaction residual. Including this simple control experiment would help isolate the contribution of residual decomposition itself to the observed performance gains and further strengthen the justification for the proposed design.
Questions for authors
Q1. Advantages of interaction residuals over direct contrastive supervision. Appendix B.4 already examines the effects of the different loss components in SynCo. Have the authors considered applying an additional InfoNCE loss directly to the original fused representation without residual decomposition? Including this simple comparison would help clarify the distinctive value of the interaction residual design. A convincing result demonstrating the benefit of residual decomposition would address my main concern and could lead me to raise my rating.
Dear Reviewer, thank you for your time and valuable feedback. We are writing to gently follow up on our submitted rebuttal. We wonder if our response has addressed your concerns and clarified the issues raised.
If you have any remaining questions, please feel free to let us know—we would be more than happy to provide further clarification. If our response has resolved your concerns, we would be very grateful if you could kindly consider updating your score.
Thank you again for your constructive review.