Cross-Modal Query Fusion and Alignment: Addressing Modality Imbalance through Query-Guided Fusion

Modality imbalance remains a fundamental challenge in multimodal learning, where dominant modalities disproportionately shape optimization while weaker modalities are underutilized. Existing studies mainly analyze this problem through optimization behavior, training losses, and final task performance, paying comparatively little attention to the intrinsic structure of multimodal representations. In this work, we revisit modality imbalance from a structural perspective and identify two coupled failure modes: representation-level imbalance, in which modalities occupy mismatched low-dimensional subspaces, and fusion-level imbalance, in which shared fusion layers suppress weak-modality contributions through destructive cross-modal entanglement. This decomposition suggests that correcting optimization imbalance alone is insufficient, as improved representation alignment does not guarantee balanced utilization after fusion. To address these two sources of imbalance, we propose Cross-Modal Query Fusion and Alignment (CQFA), a query-guided framework that explicitly separates modality-specific representation learning from controlled cross-modal interaction through staged training and query-partitioned fusion. Specifically, staged objectives over modality-specific expert queries stabilize modality-specific representations and strengthen cross-modal correspondence before full fusion, while dual query-guided fusion streams together with logit-level aggregation alleviate fusion-level imbalance by reducing premature entanglement and hard shared bottlenecks. On CREMA-D and Kinetics-400, CQFA improves upon state-of-the-art results, achieving 89.94/89.94 and 71.85/71.66 ACC/F1, respectively.