Feature-Pullback: Rethinking Multimodal Imbalance Through the Geometry of Feature Space

Modality imbalance prevents multimodal models from fully exploiting heterogeneous information and has become a major issue that limits joint learning performance. Existing studies reduce disparities between modalities by regulating gradients, learning speeds, or contributions of the modality. These methods expect modalities to receive the same optimization signal but rarely examine the feature-space updates induced by this signal in modality representation spaces. To this end, we revisit modality imbalance from the perspective of featurespace geometry. We empirically show that optimization signals undergo modality-specific matrix-valued responses as they propagate through the encoders, inducing feature updates with different directions, scales, and reachability. First, a controlled modality-manifold experiment shows that changing only the geometry of modality representations yields an 8.86% performance improvement. Further local optimization analysis reveals that task signals must pass through modality-specific feature response operators before being converted into feature updates. It also establishes a correspondence between response distortion in feature space and pullback geometry in parameter space. Based on this insight, we propose Feature-Pullback, which uses feature probes and layer-wise Kronecker structures to obtain a scalable approximation. It corrects the backward updates of each modality without altering the forward network. Experiments on four audio-visual and image-text benchmarks show that the method consistently improves modality learning and fusion performance and achieves new state-of-the-art results on all benchmarks.