- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper revisits multimodal imbalance from a structural perspective and categorizes it into two levels: representation-level imbalance and fusion-level imbalance. To address these two issues, the authors propose CQFA. The manuscript analyzes the structure of the representation space using effective dimension and principal angle, and explains fusion-level imbalance by drawing on theoretical results from prior work on polysemantic collision in shared fusion neurons. CQFA was compared with a large number of multimodal imbalance methods on CREMA-D, AVE, and Kinetics-400, achieving SOTA results on CREMA-D and Kinetics-400.
Critical strengths
-
Discussing “representation structure issues prior to fusion” and “the problem of weak modal utilization after fusion” separately is valuable. In particular, the paper emphasizes that greater alignment in the representation space does not automatically imply a more balanced fusion stage. This insight complements the large body of existing research on modal competition, which primarily focuses on gradients or optimization dynamics.
-
The authors demonstrate the algorithm’s effectiveness through extensive experimental validation and theoretical analysis, which is quite persuasive.
Critical weaknesses
-
Some theoretical proofs need to be clarified.
-
Additional experiments are needed to demonstrate the algorithm’s universality. Furthermore, there are some inconsistencies in the experimental results.
Questions for authors
-
Line 149 of the manuscript, the authors define the modal subspace mismatch where as a “representation-level imbalance” and further describe this imbalance using the “effective dimension” and “principal angle.” Therefore, I would like the authors to further explain: Why should the fact that different modalities reside in different subspaces be interpreted as “modal imbalance,” rather than as a natural consequence of heterogeneous modalities having distinct and complementary representational structures? In my opinion, it is entirely possible for two modalities to have or even different effective dimensions, yet both be fully learned and make equally important contributions to the final downstream task. Conversely, even if two modalities have the same effective dimension, there may still be very obvious modal competition or a dominant modality. The column in Table 13 of the paper also provides experimental evidence for this view: for the Remix method, for the MLA method, for the QMF method, and for the CQFA method; however, these methods did not achieve better final performance as a result. Therefore, the authors need to provide further explanation regarding these issues.
-
Regarding Equation (2), the minimum principal angle actually reflects only the most aligned pair of directions between the two subspaces. The authors need to further explain why the minimum principal angle can adequately describe the overall mismatch between two subspaces. Additionally, Eq. (20) in the main text uses the “smallest principal angle,” while Appendix G reports the “minimum nonzero principal angle.” Can the authors explain why zero angles were artificially excluded? Does this imply that the metric actually used in the appendix is not entirely the same as that in Eq. (20)?
-
In Tables 1 and 2, the metrics for the AVE dataset are inconsistent. Please explain the cause of this discrepancy. Additionally, the training objective weights given in Table 5 of Appendix A do not match the experimental configuration stated in the AVE sensitivity experiment in Appendix F.
-
Have the authors validated the algorithm’s effectiveness on datasets with other modality combinations (e.g., Visual-Text) and on datasets involving two or more modalities (e.g., Visual-Text-Audio, RGB-Depth)?
-
Is the algorithm applicable to other backbone architectures?
3 comments
Q2. Interpretation of principal angles
We agree that the minimum principal angle measures only the closest pair of directions between two subspaces and cannot characterise their overall mismatch. Its use is motivated by the subspace-constrained modality-gap analysis of Yi et al. (2025), but we have extended our evaluation to whole-spectrum summaries: the mean principal angle, the fraction of angles below 30°, and the mean squared cosine (subspace overlap ratio). The following results are computed for CQFA on the CREMA-D validation split:
| Stage | k | Min nonzero angle (°) | Mean angle (°) | Angles below 30° | Overlap ratio |
|---|---|---|---|---|---|
| Backbone (v/a) | 163 | 19.9° | 56.1° | 10% | 0.35 |
| Projected tokens | 139 | 18.9° | 51.1° | 12% | 0.41 |
| Expert queries | 55 | 22.7° | 53.3° | 4% | 0.37 |
| AVC embeddings | 22 | 5.8° | 30.5° | 59% | 0.68 |
| Fusion streams | 130 | 2.6° | 13.2° | 92% | 0.92 |
The whole-spectrum measures support a substantial increase in alignment at the AVC embeddings and fusion streams relative to earlier stages; they do not imply that every metric changes monotonically between all stages.
The reviewer is also correct that the smallest and minimum nonzero angles are distinct. Following the convention attributed to Yi et al. (2025, Eqs. 28–29), we count angles with , where (approximately ), as near-zero angles, and report their count separately as an estimated shared-subspace dimension . The smallest angle outside this threshold is . The two definitions coincide for all fifteen methods at the backbone stage and for CQFA through its AVC embeddings. Only the fusion-stream entry differs: , the raw , and .
We will distinguish both definitions explicitly in Eq. (20) and Appendix G, report the estimated overlap dimension, and add the whole-spectrum analysis.
W2/Q3. Experimental consistency & Q4/Q5. Other modalities and backbone architectures
W2/Q3. Experimental consistency
AVE results. The AVE discrepancies between tables arose because results from different random-seed runs were reported. In particular, Table 2 explicitly describes a single validation-selected checkpoint per configuration, while Table 1 does not state its reporting protocol. We will document the seed, objective-weight configuration, evaluation split, and checkpoint-selection procedure underlying each reported result, and adopt a consistent reporting protocol in the revision. For the additional AVE runs, eight incorrectly labelled samples overlapping with the training data were removed from the validation split only and the 402-clip test set was unchanged. We will document this validation-set adjustment and its implications for checkpoint selection.
Objective weights. The weights in Table 5, , apply to CREMA-D and Kinetics-400, while AVE uses , as described in Appendix F. Table 5 should have identified these dataset-specific settings. We will correct it accordingly.
We have also conducted three AVE runs under a fixed validation-checkpoint selection protocol, obtaining 95.35 ± 1.03 ACC and 95.57 ± 1.02 F1. We will report the corresponding variability and clarify the relationship between these results and the submitted tables.
Q4/Q5. Other modalities and backbone architectures
We have evaluated CQFA on audio-visual tasks using CLIP and CLAP backbones. We have not yet validated it on other modality combinations, more than two modalities, or alternative backbone architectures.
The query-partitioning and attention-masking mechanisms are not intrinsically audio-visual and operate on modality-specific token representations. Similarly, the learnable-class-prototype ablation in Table 2 indicates that the readout need not depend on text-aligned encoders. These properties suggest potential applicability beyond the evaluated setting, but such architectural compatibility is not evidence of empirical generalisation.
We will state this limitation explicitly and identify evaluation across additional modalities and backbones as future work.
References
[Yi et al., 2025] Lingjie Yi, Raphael Douady, and Chao Chen. Decipher the modality gap in multimodal contrastive learning: From convergent representations to pairwise alignment. arXiv preprint arXiv:2510.03268, 2025.
W1/Q1. Representation-level imbalance
We thank the reviewer for recognising the value of distinguishing representation-level and fusion-level modality imbalance, and for the positive assessment of our experimental evidence. We address the theoretical clarifications and experimental questions below.
We agree that different modality subspaces do not necessarily imply modality imbalance. Heterogeneous modalities naturally encode complementary information, and differences in effective dimension or subspace orientation may coexist with balanced downstream utilisation.
Our intended distinction is between representation-level mismatch, which characterises geometric differences between modality representations, and fusion-level imbalance, which concerns their relative utilisation by the classifier. Representation mismatch is a potential contributor to imbalance, but is neither a necessary nor a sufficient condition for fusion-level underutilisation. Conversely, well-aligned representations do not guarantee balanced fusion.
This distinction also explains the reviewer's observation about Table 13, where methods with smaller effective-dimension differences do not necessarily achieve higher accuracy. The geometric metrics diagnose representation structure rather than downstream performance or modality contribution. CQFA combines representation-oriented supervision in AVC with structured query interactions and fusion in CLS without attempting to optimise geometric metrics directly.
We will clarify this distinction in Section 3.1 and use representation-level mismatch where appropriate to avoid equating geometric differences with harmful imbalance.