acceptodds
Under review as a conference paper at ICLR 2027

Cross-Modal Query Fusion and Alignment: Addressing Modality Imbalance through Query-Guided Fusion

Abstract

Modality imbalance remains a fundamental challenge in multimodal learning, where dominant modalities disproportionately shape optimization while weaker modalities are underutilized. Existing studies mainly analyze this problem through optimization behavior, training losses, and final task performance, paying comparatively little attention to the intrinsic structure of multimodal representations. In this work, we revisit modality imbalance from a structural perspective and identify two coupled failure modes: representation-level imbalance, in which modalities occupy mismatched low-dimensional subspaces, and fusion-level imbalance, in which shared fusion layers suppress weak-modality contributions through destructive cross-modal entanglement. This decomposition suggests that correcting optimization imbalance alone is insufficient, as improved representation alignment does not guarantee balanced utilization after fusion. To address these two sources of imbalance, we propose Cross-Modal Query Fusion and Alignment (CQFA), a query-guided framework that explicitly separates modality-specific representation learning from controlled cross-modal interaction through staged training and query-partitioned fusion. Specifically, staged objectives over modality-specific expert queries stabilize modality-specific representations and strengthen cross-modal correspondence before full fusion, while dual query-guided fusion streams together with logit-level aggregation alleviate fusion-level imbalance by reducing premature entanglement and hard shared bottlenecks. On CREMA-D and Kinetics-400, CQFA improves upon state-of-the-art results, achieving 89.94/89.94 and 71.85/71.66 ACC/F1, respectively.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal Learning, Modality Imbalance, Cross-Modal Fusion, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_m2uMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper revisits multimodal imbalance from a structural perspective and categorizes it into two levels: representation-level imbalance and fusion-level imbalance. To address these two issues, the authors propose CQFA. The manuscript analyzes the structure of the representation space using effective dimension and principal angle, and explains fusion-level imbalance by drawing on theoretical results from prior work on polysemantic collision in shared fusion neurons. CQFA was compared with a large number of multimodal imbalance methods on CREMA-D, AVE, and Kinetics-400, achieving SOTA results on CREMA-D and Kinetics-400.

Critical strengths
  1. Discussing “representation structure issues prior to fusion” and “the problem of weak modal utilization after fusion” separately is valuable. In particular, the paper emphasizes that greater alignment in the representation space does not automatically imply a more balanced fusion stage. This insight complements the large body of existing research on modal competition, which primarily focuses on gradients or optimization dynamics.

  2. The authors demonstrate the algorithm’s effectiveness through extensive experimental validation and theoretical analysis, which is quite persuasive.

Critical weaknesses
  1. Some theoretical proofs need to be clarified.

  2. Additional experiments are needed to demonstrate the algorithm’s universality. Furthermore, there are some inconsistencies in the experimental results.

Questions for authors
  1. Line 149 of the manuscript, the authors define the modal subspace mismatch where Sx≠SyS_x \ne S_y as a “representation-level imbalance” and further describe this imbalance using the “effective dimension” and “principal angle.” Therefore, I would like the authors to further explain: Why should the fact that different modalities reside in different subspaces be interpreted as “modal imbalance,” rather than as a natural consequence of heterogeneous modalities having distinct and complementary representational structures? In my opinion, it is entirely possible for two modalities to have Sx≠SyS_x \ne S_y or even different effective dimensions, yet both be fully learned and make equally important contributions to the final downstream task. Conversely, even if two modalities have the same effective dimension, there may still be very obvious modal competition or a dominant modality. The Δd\Delta d column in Table 13 of the paper also provides experimental evidence for this view: Δd=0\Delta d=0 for the Remix method, Δd=2\Delta d=2 for the MLA method, Δd=4\Delta d=4 for the QMF method, and Δd=12\Delta d=12 for the CQFA method; however, these methods did not achieve better final performance as a result. Therefore, the authors need to provide further explanation regarding these issues.

  2. Regarding Equation (2), the minimum principal angle actually reflects only the most aligned pair of directions between the two subspaces. The authors need to further explain why the minimum principal angle can adequately describe the overall mismatch between two subspaces. Additionally, Eq. (20) in the main text uses the “smallest principal angle,” while Appendix G reports the “minimum nonzero principal angle.” Can the authors explain why zero angles were artificially excluded? Does this imply that the metric actually used in the appendix is not entirely the same as that in Eq. (20)?

  3. In Tables 1 and 2, the metrics for the AVE dataset are inconsistent. Please explain the cause of this discrepancy. Additionally, the training objective weights given in Table 5 of Appendix A do not match the experimental configuration stated in the AVE sensitivity experiment in Appendix F.

  4. Have the authors validated the algorithm’s effectiveness on datasets with other modality combinations (e.g., Visual-Text) and on datasets involving two or more modalities (e.g., Visual-Text-Audio, RGB-Depth)?

  5. Is the algorithm applicable to other backbone architectures?

3 comments
AuthorsOpenReview ↗

W1/Q1. Representation-level imbalance

We thank the reviewer for recognising the value of distinguishing representation-level and fusion-level modality imbalance, and for the positive assessment of our experimental evidence. We address the theoretical clarifications and experimental questions below.

We agree that different modality subspaces do not necessarily imply modality imbalance. Heterogeneous modalities naturally encode complementary information, and differences in effective dimension or subspace orientation may coexist with balanced downstream utilisation.

Our intended distinction is between representation-level mismatch, which characterises geometric differences between modality representations, and fusion-level imbalance, which concerns their relative utilisation by the classifier. Representation mismatch is a potential contributor to imbalance, but is neither a necessary nor a sufficient condition for fusion-level underutilisation. Conversely, well-aligned representations do not guarantee balanced fusion.

This distinction also explains the reviewer's observation about Table 13, where methods with smaller effective-dimension differences do not necessarily achieve higher accuracy. The geometric metrics diagnose representation structure rather than downstream performance or modality contribution. CQFA combines representation-oriented supervision in AVC with structured query interactions and fusion in CLS without attempting to optimise geometric metrics directly.

We will clarify this distinction in Section 3.1 and use representation-level mismatch where appropriate to avoid equating geometric differences with harmful imbalance.

AuthorsOpenReview ↗

Q2. Interpretation of principal angles

We agree that the minimum principal angle measures only the closest pair of directions between two subspaces and cannot characterise their overall mismatch. Its use is motivated by the subspace-constrained modality-gap analysis of Yi et al. (2025), but we have extended our evaluation to whole-spectrum summaries: the mean principal angle, the fraction of angles below 30°, and the mean squared cosine (subspace overlap ratio). The following results are computed for CQFA on the CREMA-D validation split:

StagekMin nonzero angle (°)Mean angle (°)Angles below 30°Overlap ratio
Backbone (v/a)16319.9°56.1°10%0.35
Projected tokens13918.9°51.1°12%0.41
Expert queries5522.7°53.3°4%0.37
AVC embeddings225.8°30.5°59%0.68
Fusion streams1302.6°13.2°92%0.92

The whole-spectrum measures support a substantial increase in alignment at the AVC embeddings and fusion streams relative to earlier stages; they do not imply that every metric changes monotonically between all stages.

The reviewer is also correct that the smallest and minimum nonzero angles are distinct. Following the convention attributed to Yi et al. (2025, Eqs. 28–29), we count angles with cos⁡θ>1−ϵ\cos\theta>1-\epsilon, where ϵ=10−3\epsilon=10^{-3} (approximately θ<2.6∘\theta<2.6^\circ), as near-zero angles, and report their count separately as an estimated shared-subspace dimension doverlapd_{\mathrm{overlap}}. The smallest angle outside this threshold is θmin⁡nz\theta_{\min}^{\mathrm{nz}}. The two definitions coincide for all fifteen methods at the backbone stage and for CQFA through its AVC embeddings. Only the fusion-stream entry differs: doverlap=6d_{\mathrm{overlap}}=6, the raw θmin⁡=1.0∘\theta_{\min}=1.0^\circ, and θmin⁡nz=2.6∘\theta_{\min}^{\mathrm{nz}}=2.6^\circ.

We will distinguish both definitions explicitly in Eq. (20) and Appendix G, report the estimated overlap dimension, and add the whole-spectrum analysis.

AuthorsOpenReview ↗

W2/Q3. Experimental consistency & Q4/Q5. Other modalities and backbone architectures

W2/Q3. Experimental consistency

AVE results. The AVE discrepancies between tables arose because results from different random-seed runs were reported. In particular, Table 2 explicitly describes a single validation-selected checkpoint per configuration, while Table 1 does not state its reporting protocol. We will document the seed, objective-weight configuration, evaluation split, and checkpoint-selection procedure underlying each reported result, and adopt a consistent reporting protocol in the revision. For the additional AVE runs, eight incorrectly labelled samples overlapping with the training data were removed from the validation split only and the 402-clip test set was unchanged. We will document this validation-set adjustment and its implications for checkpoint selection.

Objective weights. The weights in Table 5, (αAVC,αAVM,αCLS)=(1.0,0.5,1.0)(\alpha_{\mathrm{AVC}},\alpha_{\mathrm{AVM}},\alpha_{\mathrm{CLS}})=(1.0,0.5,1.0), apply to CREMA-D and Kinetics-400, while AVE uses (1.0,0.2,1.8)(1.0,0.2,1.8), as described in Appendix F. Table 5 should have identified these dataset-specific settings. We will correct it accordingly.

We have also conducted three AVE runs under a fixed validation-checkpoint selection protocol, obtaining 95.35 ± 1.03 ACC and 95.57 ± 1.02 F1. We will report the corresponding variability and clarify the relationship between these results and the submitted tables.

Q4/Q5. Other modalities and backbone architectures

We have evaluated CQFA on audio-visual tasks using CLIP and CLAP backbones. We have not yet validated it on other modality combinations, more than two modalities, or alternative backbone architectures.

The query-partitioning and attention-masking mechanisms are not intrinsically audio-visual and operate on modality-specific token representations. Similarly, the learnable-class-prototype ablation in Table 2 indicates that the readout need not depend on text-aligned encoders. These properties suggest potential applicability beyond the evaluated setting, but such architectural compatibility is not evidence of empirical generalisation.

We will state this limitation explicitly and identify evaluation across additional modalities and backbones as future work.


References

[Yi et al., 2025] Lingjie Yi, Raphael Douady, and Chao Chen. Decipher the modality gap in multimodal contrastive learning: From convergent representations to pairwise alignment. arXiv preprint arXiv:2510.03268, 2025.

Reviewer_cYxuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates multimodal modality imbalance from a structural perspective and decomposes it into two interrelated sources: (i) representation-level imbalance arising from mismatched low-dimensional, modality-specific subspaces induced by effective-rank collapse; and (ii) fusion-level imbalance, in which cross-modal interference prevents a shared fusion head from fully exploiting the weaker modality. Building on this perspective, the authors propose CQFA, a query-guided architecture that employs partitioned expert and fusion queries together with a three-stage training schedule comprising AVC alignment, AVM matching, and CLS classification. The framework ultimately performs cross-modal fusion through two query-guided streams and aggregates their predictions at the logit level.

Critical strengths
  • It separates representation stabilization from controlled cross-modal interaction through learnable expert and fusion query groups within a shared Q-Former, together with a bottleneck mask that explicitly routes cross-modal exchange.
  • It provides a principled geometric diagnosis of modality imbalance through effective-dimension analysis and a principal-angle-based assessment of cross-modal subspace alignment.
Critical weaknesses
  • The novelty claims appear overstated, as separately modeling modality imbalance at the representation and fusion levels is not entirely new.
  • The core analytical framework has important methodological limitations and does not appear to provide sufficient support for the paper’s empirical conclusions.
Questions for authors
  • Modeling modality imbalance separately at the representation and fusion levels is not entirely novel. Prior studies that model or analyze imbalance at the fusion level, such as MMCosine [1], OLM [2], and Revisit [3], receive insufficient discussion and comparison. Could the authors better clarify the novelty of the proposed formulation and expand the comparison with these closely related works?
  • Is the subspace-alignment analysis in the theoretical section applicable only to architectures based on a Q-Former or incorporating an explicit alignment stage? Previous BML method usually employ fusion strategies such as feature concatenation, where independently extracted modality representations are not necessarily mapped into a comparable shared space. The proposed analysis seems do not provide a "general explanation" of modality imbalance. Could the authors explicitly state its assumptions and scope of applicability?
  • The evaluation metrics derived from the theoretical analysis in Sec 3.1 & Sec 3.4 receive little attention in the main experiments, while the corresponding analysis in the appendix is also relatively limited. This makes it difficult to establish a complete connection between the theoretical formulation and its empirical validation. Could the authors provide a more comprehensive evaluation of these metrics, including detailed computation procedures and results across datasets, baselines, and representation stages?
  • Could the authors include additional late-fusion baselines, such as directly summing the CLIP and CLAP logits without using the Q-Former, to demonstrate the added value of CQFA over a simple ensemble?
  • The proposed framework differs substantially from the architectures originally used in prior modality-balancing methods. How were these methods adapted to the unified experimental framework? The appendix should proide adaptation procedures to demonstrate that the core ideas and mechanisms of each baseline were faithfully preserved, thereby ensuring a fair comparison.

[1] MMCosine: Multi-Modal Cosine Loss Towards Balanced Audio-Visual Fine-Grained Learning. ICASSP, 2023.

[2] Balancing Multimodal Learning via Online Logit Modulation. IJCAI, 2024.

[3] Revisit Modality Imbalance at the Decision Layer. arXiv preprint, 2025.

3 comments
AuthorsOpenReview ↗

W1/Q1. Novelty and related work & W2/Q2. Assumptions and scope of the subspace analysis

We thank the reviewer for recognising the value of separating representation stabilisation from controlled cross-modal interaction. We address the novelty, scope of the geometric analysis, experimental evidence, late-fusion comparison, and baseline adaptations below.

W1/Q1. Novelty and related work

We agree that distinguishing representation-level and fusion-level effects is not itself novel. We will expand Related Work to discuss MMCosine, OLM, and Revisit explicitly, rather than presenting the two-level distinction as an original taxonomy. Our contribution is the combination of a geometric diagnosis with an architectural response, i.e., modality-specific expert queries and AVC supervision address the representations entering fusion, while partitioned fusion queries, stage-specific attention masks, and two directional streams regulate how evidence is combined (Sections 1 and 3.2-3.3).

We highlight that representation mismatch and fusion-level underutilisation are not equivalent, where geometric alignment is neither necessary nor sufficient for balanced modality utilisation. In our experiments, smaller backbone-level modality gaps or effective-dimension differences do not consistently correspond to better classification performance (Tables 3, 4, and 13). CQFA's ablations support the value of the proposed query organisation: removing query partitioning reduces ACC by 3.30/2.29 points and removing bottleneck masking by 2.10/2.17 points on CREMA-D/Kinetics-400 (Table 2). These observations motivate the design but do not establish a causal relationship between any individual geometric statistic and accuracy.

MMCosine, OLM, and Revisit offer important complementary approaches to balancing modality contributions, particularly through classification or decision-level mechanisms. CQFA differs by explicitly structuring the pre-fusion and cross-modal interactions rather than relying solely on adjustments to the decision mechanism. We will make these relationships and the limits of our novelty claim clearer in Sections 1 and 2.

W2/Q2. Assumptions and scope of the subspace analysis

We agree that the geometric analysis is not a general explanation of modality imbalance. Effective dimension is computed within each modality and can be used without a shared embedding space. When modality feature dimensions differ, a normalised effective dimension (e.g., de/dd_e/d) is more interpretable than a raw dimension difference. Principal-angle comparisons, however, require the subspace bases to inhabit a common, meaningfully comparable coordinate system. Equal output dimensionality alone does not guarantee this condition.

Our experiments use CLIP ViT-B/32 and CLAP HTS-AT encoders with 512-dimensional pooled features, including the LoRA adaptations trained for each method. This provides a consistent computational protocol across methods, but it does not, in isolation, establish that independently learned audio and visual coordinates are semantically aligned. We will explicitly state this limitation and carefully revise our wording to ensure that there is no overclaimed novelty or applicability. Neither the effective-dimension diagnostic nor the two-level architectural motivation depends on using a Q-Former.

AuthorsOpenReview ↗

Q3. Diagnostic computation and empirical coverage

We agree that the original manuscript does not document the computation procedure sufficiently and that its geometric evidence is concentrated on CREMA-D. We will add the following protocol to the appendix: extract each method's LoRA-adapted, pooled CLIP visual and CLAP audio features; compute effective dimension from the feature covariance using a 99% cumulative-variance threshold; and obtain principal angles from the singular values of Uv⊤UaU_v^\top U_a, where the bases are formed from retained right-singular directions. Features are L2-normalised for the angle and distance analyses but not for effective dimension. Angles with cosine greater than 1−10−31-10^{-3} are counted separately as estimated overlap dimensions; the minimum nonzero angle excludes these near-zero angles. We will also correct the inconsistent threshold notation.

The recomputed stage-wise CREMA-D validation-split results extend the minimum-angle diagnostic with whole-spectrum summaries:

StagekMin angle (°)Min nonzero angle (°)Estimated overlap dim.Mean angle (°)Angles below 30°Mean cos²
Backbone (v/a)16319.919.9056.110%0.35
Projected tokens13918.918.9051.112%0.41
Expert queries5522.722.7053.34%0.37
AVC embeddings225.85.8030.559%0.68
Fusion streams1301.02.6613.292%0.92

These summaries indicate substantially stronger measured alignment at the AVC embeddings and fusion streams than at earlier stages. They remain diagnostics, not proof that geometric alignment causes improved accuracy or balanced utilisation. We will make the CREMA-D-only scope of the currently available geometric analysis explicit. Additional dataset- and baseline-stage measurements will be included only once computed and verified.

AuthorsOpenReview ↗

Q4. No-Q-Former late-fusion comparison & Q5. Adaptation and fairness of the baseline comparisons

Q4. No-Q-Former late-fusion comparison

A closely related comparison is already present in Table 1: LFM, as adapted in our framework, sums logits from separate CLIP and CLAP classification heads without a Q-Former or cross-modal attention. However, LFM also uses Jensen–Shannon logit-alignment supervision and a dynamic weighting schedule, so it is not identical to an unregularised logits-sum ensemble. We will clarify this distinction and correct the description of LFM's objective in Section 4.2.

LFM reports 88.96 ACC on CREMA-D, 97.26 on AVE, and 70.95 on Kinetics-400. Additional three-run CQFA results are 89.24 ± 0.63 on CREMA-D and 95.35 ± 1.03 on AVE; the three-run Kinetics-400 result is 71.55 ± 0.29 ACC. Thus, relative to LFM's reported results, CQFA shows a modest advantage on CREMA-D, weaker performance on AVE, and a modest advantage on Kinetics-400. The Feature Fusion ablation in Table 2 isolates a different architectural choice and does not replace the requested plain logits-sum control. A direct CLIP+CLAP logits-sum baseline trained under the same protocol remains untested and will be stated as a limitation.

Q5. Adaptation and fairness of the baseline comparisons

We agree that a unified benchmark requires documentation of both what is standardised and what changes in each method. All methods use the same CLIP/CLAP encoders, LoRA configuration, inputs, training budget, and evaluation protocol. Each baseline was adapted to the shared encoders using its characteristic balancing objective, update rule, and fusion mechanism, as documented in our implementation. We will add a method-by-method appendix table documenting the original mechanism, adapted implementation, fusion head, method-specific hyperparameters, and their sources.

Some adaptations are substantive. For example, gradient-modulation methods originally developed for fully trainable encoders act on trainable LoRA and projection parameters in our framework, potentially changing their optimisation behaviour. Similarly, original hyperparameter defaults may not be optimal for CLIP/CLAP with LoRA. A common backbone controls important sources of variation but does not, by itself, guarantee that every adapted baseline is optimally tuned or fully equivalent to its original implementation. We will state this limitation and avoid comparing the resulting absolute scores directly with those from the original architectures.

Reviewer_ZdG9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that modality imbalance consists of representation-level imbalance and fusion-level imbalance. To address them, the authors propose CQFA leveraging learnable Query and Q-Former with three stages (AVC, AVM, and CLS) to align visual and audio modalities. Experimental results showed CQFA could improve the performance on CREMA-D and Kinetics-400 datasets.

Critical strengths
  1. The discussion on decoupling modality imbalance into representation-level imbalance and fusion-level imbalance seems interesting.
Critical weaknesses
  1. The original contribution of this paper is limited.
  • Q-formers have been used in different prior works for visual-language alignment. This work is building on top of it, and thus is considered to be incremental.
  • Also, while quite some discussion on representation-level imbalance is presented in Sections 3.1 and 3.4, it seems that it is not explicitly incorporated into the model training algorithm but only used as a way to quantify the quality of the model after training. The key contribution CQFA is mainly presented in Section 3.3. This make the contribution of advocating the representation-level imbalance less interesting for this paper.
  1. The experimental results cannot demonstrate that CQFA can perform consistently better.
  • CQFA does not give better results than the other baseline consistently. E.g., its performance is not good based for AVE. Thus, it is not clear its performance on other datasets which are also tested in other multimodal fusion papers.
  • Also, standard deviations of the results are not presented. It is hard to conclude that CQFA is always better, which weakens this contribution.
  • As shown in the ablation studies, some version of CQFA can perform better than its full version for AVE. That seems to indicate that the overall design of CQFA should be further optimized.
Questions for authors
  1. Please clarify if the key contribution of introducing representation-level imbalance is for quantifying the model learned. If this is not the case, further clarification will be appreciated.
  2. Why are the three particular datasets being chosen for the experiments while there are more benchmarks used in related papers?
3 comments
AuthorsOpenReview ↗

W1. Novelty relative to Q-Former

We thank the reviewer for recognising that decoupling modality imbalance into representation-level and fusion-level imbalance seems interesting. We address the concerns about the contribution, its implementation, and the experimental evidence below.

We agree that learnable queries and the Q-Former architecture originate from prior work, notably BLIP-2 [Li et al., 2023], which we will cite explicitly. Our contribution is not the Q-Former itself, but its organisation into modality-specific and fusion pathways to address two distinct aspects of modality imbalance (Sections 1 and 3.2).

Unlike a standard shared-query architecture, CQFA partitions queries by modality and role, regulates their interaction through stage-specific attention masks, and uses two directional fusion streams with separate text-space readouts. These choices are designed to preserve modality-specific evidence while controlling how it is combined during classification.

The architectural ablations support the value of this organisation. In Table 2, a capacity-matched shared-query control (29.96M versus 29.97M parameters) performs below CQFA by 1.95 ACC on CREMA-D and 2.49 ACC on Kinetics-400. Removing query partitioning, bottleneck masking, or either fusion stream also reduces performance on these datasets. As such, the contribution lies in the structured fusion mechanism, rather than simply adopting or enlarging a Q-Former.

References [Li et al., 2023] BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. ICML 2023.

AuthorsOpenReview ↗

W2/Q1. Role of representation-level imbalance

We agree that the geometric quantities introduced in Sections 3.1 and 3.4, i.e., effective dimension and principal angle, are diagnostic metrics rather than terms in the training loss. We will make this distinction explicit.

The central motivation is that representation alignment and modality utilisation during fusion are different problems, where improving one does not necessarily resolve the other. This distinction informs CQFA's design. In the AVC configuration, isolated modality-specific query pathways receive per-modality prototype supervision and symmetric audio-visual contrastive supervision, intended to stabilise and align the representations entering fusion. The subsequent query partitioning, attention masks, and dual fusion streams are intended to regulate how those representations are used.

The geometric analysis evaluates the resulting representations and it is not the optimisation mechanism. On the CREMA-D test split, the minimum principal angle decreases from 20.0° at the backbone to 4.8° at the AVC embeddings (Table 11), while removing the AVC objectives reduces accuracy by 4.20 points (Table 6). The separately recomputed validation-split analysis reports 5.8° at the AVC embeddings. These results are consistent with the intended role of representation-level supervision, although they do not establish a direct causal relationship between the geometric metric and accuracy.

We will clarify this in the paper and move the geometric metrics from methodology to the experimental analysis.

AuthorsOpenReview ↗

W3/Q2. Experimental consistency and dataset selection

Performance and dataset choice. We agree that CQFA does not outperform all baselines on AVE and avoid claiming universal superiority. The three datasets were selected to cover emotion recognition (CREMA-D), audio-visual event recognition (AVE), and action recognition (Kinetics-400), across substantially different dataset scales and modality characteristics, going beyond what several comparable approaches have considered.

The results also suggest that CQFA's benefits depend on the extent of modality imbalance. The independently trained unimodal models differ by 1.49 ACC on AVE, compared with 14.27 on CREMA-D and 41.20 on Kinetics-400 (Table 8). This is consistent with AVE presenting a less pronounced unimodal performance imbalance in our setting, but does not establish the absence of fusion-level imbalance. We will clarify the scope of our claims and discuss the limited benchmark coverage.

Variability. We agree that run-to-run variation should be reported. Additional CQFA runs yield 89.24 ± 0.63 ACC on CREMA-D and 95.35 ± 1.03 ACC on AVE (three runs each). These results support a more cautious interpretation of the CREMA-D margin over the strongest baselines, while confirming that AVE remains comparatively weaker. On Kinetics-400, three runs yield 71.55 ± 0.29 ACC and 71.25 ± 0.39 F1, compared with 70.95 ACC for the strongest reported baseline (LFM). We will report the corresponding variability and distinguish these multi-run estimates from the originally submitted results.

AVE ablations. The reviewer correctly observes that some reduced configurations outperform full CQFA on AVE. In particular, removing AVM supervision improves the reported AVE result (Table 6), whereas removing AVC supervision substantially reduces CREMA-D performance. This suggests that not every constraint is equally beneficial across datasets, and that AVM may require dataset-dependent weighting. We regard this as an important limitation rather than evidence that every component must improve performance universally. We will clarify the AVE experimental configuration and reporting protocol in our revision.

AVE reporting consistency. The differing full-CQFA results across Tables 1, 2, 6 and 10 arise from different single-seed runs of the same configuration, rather than different aggregation protocols. This was insufficiently documented. We will identify the runs underlying each table, clarify the checkpoint-selection procedure, and ensure that comparisons are reported consistently in the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.