- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a pointwise multimodal information estimation framework based on three common multimodal interaction patterns—namely, synergy, conflict, and joint failure—and estimates pointwise mutual information through unimodal predictions and class priors. Extensive experiments on three multimodal benchmark datasets demonstrate the effectiveness of the proposed PMIB method.
Critical strengths
1、This paper proposes a fusion framework based on pointwise mutual information estimation, which can achieve information accumulation under synergy, reliably select branches in conflict situations, and effectively suppress errors in cases of joint failure.
2、It demonstrates promising performance on multimodal sentiment analysis tasks.
Critical weaknesses
1、For the three-modality interaction patterns in Figure 1 (c), further clarification is needed regarding how the proportions of synergy, conflict, and failure were obtained for each dataset. If these were compiled by the authors themselves, how were they measured?
2、The authors use unimodal predictions and class priors to estimate pointwise mutual information. If the unimodal predictor produces incorrect predictions, is it still valid to estimate pointwise mutual information based on the probabilities predicted from these erroneous unimodal predictions? Furthermore, the authors should provide experimental results using only a single modality to support estimating pointwise mutual information through unimodal predictions and class priors.
3、The authors primarily conducted experiments on multimodal sentiment analysis tasks. For instance, when applying the proposed method to MHD (multimodal humor detection, UR-FUNNY dataset) and MSD (multimodal sarcasm detection, MUStARD dataset), it is necessary to exploit the "conflict" information between different modalities for discrimination. Is the proposed method still effective under such circumstances?
4、Recent papers such as "Learning Optimal Multimodal Information Bottleneck Representations" and "Efficient Quantification of Multimodal Interaction at Sample Level" also use mutual information as a basis to address the challenges of synergy, conflict, and redundancy in multimodal learning. Since this paper likewise employs mutual information to tackle the same problems, what advantages does it offer?
5、The code is not available, resulting in poor reproducibility.
6、Is "multimodala" in the abstract a typographical error?
7、Should the full name of the MSA task be provided where it first appears in line 73?
Questions for authors
The authors need to address all the issues raised in the Weaknesses.