Revisiting Geometric Symmetry Non-Conservation for Multimodal Test-Time Adaptation

Without access to source data, multimodal test-time adaptation (MTTA) has emerged as a promising paradigm for adapting multimodal models to target domains under complex distribution shift. Despite the documented success of MTTA, an overlooked yet persistent issue remains, which we term geometric symmetry non-conservation. This phenomenon refers to the breakdown of pretraining-established geometric symmetry between representations of different modalities under complex distribution shift. We uncover the cause of geometric symmetry non-conservation through a graphical explanation and a theoretical derivation. Furthermore, we propose ManiGest, short for Manifold-constrained Geometry Restoration, to alleviate the impact of complex distribution shift on the geometric symmetry between representations of different modalities. ManiGest restores the geometric symmetry under the constraint of probabilistic Birkhoff polytope manifold, and leverages the Riemannian distance on the manifold of symmetric positive definite matrices to preserve intra-modal geometry. Extensive results on multiple benchmarks demonstrate that ManiGest serves as an effective plug-and-play module for improving robustness against complex distribution shift.