acceptodds
Withdrawn from ICLR 2027

Revisiting Geometric Symmetry Non-Conservation for Multimodal Test-Time Adaptation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Without access to source data, multimodal test-time adaptation (MTTA) has emerged as a promising paradigm for adapting multimodal models to target domains under complex distribution shift. Despite the documented success of MTTA, an overlooked yet persistent issue remains, which we term geometric symmetry non-conservation. This phenomenon refers to the breakdown of pretraining-established geometric symmetry between representations of different modalities under complex distribution shift. We uncover the cause of geometric symmetry non-conservation through a graphical explanation and a theoretical derivation. Furthermore, we propose ManiGest, short for Manifold-constrained Geometry Restoration, to alleviate the impact of complex distribution shift on the geometric symmetry between representations of different modalities. ManiGest restores the geometric symmetry under the constraint of probabilistic Birkhoff polytope manifold, and leverages the Riemannian distance on the manifold of symmetric positive definite matrices to preserve intra-modal geometry. Extensive results on multiple benchmarks demonstrate that ManiGest serves as an effective plug-and-play module for improving robustness against complex distribution shift.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal learning, Test-time Adaptation, Distribution Shift, Geometric Symmetry Non-Conservation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fUS3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies multimodal test-time adaptation (MTTA) under modal-specific distribution shift. The authors observe that the corrupted modality becomes more compact in representation space, which breaks the geometric symmetry between modalities, and attempt to explain this phenomenon through a graphical model and a theorem. They propose ManiGest, a plug-and-play module with two losses: BGSR, a token-level optimal transport loss between modalities, and SIGP, an AIRM distance between shallow-layer and last-layer kernel matrices. ManiGest is added to READ, TSA, and ABPEM, and shows accuracy gains on Kinetics50-C and VGGSound-C.

Critical strengths
  1. Plug-and-play module with consistent average gains.
  • ManiGest improves the average accuracy of three MTTA baselines on two datasets under both audio and video corruptions.
  • Per-corruption results, component ablations, and sensitivity analyses are provided.
  1. Representation-level view on MTTA.
  • The paper focuses on the geometry of modality-specific representations, which is less studied than fusion-level methods.
  • The compactness change of the corrupted modality is illustrated through t-SNE visualizations across 21 corruption types.
Critical weaknesses
  1. The support for geometric symmetry non-conservation seems insufficient.
  • Theorem 3.1: Eq. (13) assumes an identical noise vector for both samples, which may not represent sample-specific corruptions. Moreover, this assumption does not generally guarantee an increase in expected cosine similarity. The proof also appears to have several issues: Eq. (15) treats ⟨μ_i, μ_j⟩ as the cosine of the means, Eq. (20) appears to use an incorrect Taylor approximation, and Eq. (21) bounds an expectation with a high-probability bound. In addition, stating a deterministic expectation inequality “with high probability” is not well defined.
  • Graphical model: In a fixed corrupted test domain, the samples can remain conditionally independent given c. Moreover, statistical dependence does not necessarily imply increased representation compactness.
  • Definition and measurement: “Geometric symmetry” is not formally defined. The Symmetry Score measures paired cross-modal distance, which seems different from the compactness described in the text. Also, t-SNE may not be reliable for comparing compactness.
  • Correlation and causation: The correlation in Fig. 2 may simply reflect that stronger corruption causes both larger representation change and larger accuracy drop. Also, SIGP alone gives a gain similar to BGSR (+2.0 vs. +2.3 with READ), so it is not clear that the gain comes from symmetry restoration.
  1. The method design and experimental validation could be strengthened.
  • Direction of BGSR: Although each term uses stop-gradient, the bidirectional BGSR objective still updates both modalities. It is unclear how this design prevents the clean modality from being aligned toward corrupted representations.
  • Reference of SIGP: The shallow-layer reference is computed from the same corrupted input, so it may not reflect the pretraining geometry.
  • Gains and variance: Gains are small in several settings (e.g., +0.1 and +0.4 on Kinetics50-C audio), and results are reported without variability across independent runs. Also, the +8.1 gain on VGGSound-C audio is over a reproduced ABPEM (27.1) that is lower than READ (32.4).
  • Ablation values: In Table 5, the BGSR-only and SIGP-only rows for ABPEM are identical in 12 of the 15 corruptions. These values may need to be checked.
  • Missing comparisons: Simpler alternatives (e.g., pooled cosine alignment, Euclidean distance instead of AIRM) are not compared, and related work on the modality gap (Liang et al., 2022) is not discussed. The computational cost is also not reported.
Questions for authors

Clarification Questions

  1. Noise assumption: Is ε in Eq. (13) shared by all samples? Under which conditions does Theorem 3.1 hold, and does it still hold with independent noise per sample?
  2. Symmetry measurement: Can the authors report the Symmetry Score of each baseline with and without ManiGest?
  3. Direction of BGSR: When the corrupted modality is known (oracle setting), how does one-way BGSR compare with the proposed bidirectional loss?
  4. Ablation values: Can the authors confirm the BGSR-only and SIGP-only results for ABPEM in Table 5?
  5. Hyperparameter selection: How are hyperparameters selected “in a label-free manner using the target test data”? The text refers to Table 1, but Table 1 shows results.
  6. Cost and variance: What are the additional time and memory costs, and how much do results vary across independent runs?

Suggestions

  1. If possible, please evaluate other modality pairs or backbones, and bimodal corruption settings (e.g., AVRobustBench).
  2. Please compare with simpler alignment losses and distances to justify the OT and SPD-manifold design.
  3. Please clarify the terminology of the “probabilistic Birkhoff polytope,” which appears to correspond to the standard transportation polytope in OT.
Reviewer_jEfYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ManiGest for multimodal TTA. The paper observes that corruption changes the feature distributions of different modalities and affects their geometric relationship. Based on this observation, the method restores the geometric relationship between modalities while preserving the feature structure within each modality. The experiments combine ManiGest with READ, TSA, and ABPEM and evaluate it on Kinetics50-C and VGGSound-C.

Critical strengths

1.The visualization in this paper is excellent. The changes in feature distributions under different corruptions are clearly presented, making the research motivation easy to understand. Meanwhile, the framework diagram clearly illustrates the two modules and their respective functions. I believe these visualizations effectively help readers grasp the authors' main ideas.

2.The experiments cover two datasets and various types of audio and video corruption, demonstrating a comprehensive experimental scope. Moreover, when combined with READ, TSA, and ABPEM, the proposed method consistently improves average accuracy. Ablation studies further reveal the individual contributions of each module as well as the effectiveness of their combination.

Critical weaknesses

1.This paper does not compare with DASP (CVPR 2026). This method also studies multimodal TTA under modality-specific corruption and conducts experiments on Kinetics50-C and VGGSound-C. Why wasn't this method compared? I hope the authors can include this comparison under the same settings. I think this is important for assessing the advantages of ManiGest.

2.The efficiency of this method is not clearly demonstrated. The method introduces token-level optimal transport and additional matrix operations, but the paper does not report the additional runtime and memory usage. Therefore, I hope the authors can compare the computational cost before and after adding ManiGest, using the same batch size and hardware. This would help to understand the practical value of the method.

Questions for authors

For this paper, my assessment is weak reject. The visualization is very good, and the experiments show improvements when ManiGest is combined with several existing methods. However, the paper misses a relevant published method, and the computational cost also need further clarification. At the same time, I will also look at the opinions of other reviewers to see if I have missed anything. The specific issues can be found in “Weaknesses”.

Reviewer_34DWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies multimodal test-time adaptation under modality-specific distribution shifts, arguing that corruption can make the affected modality’s representations more compact and disrupt their geometric correspondence with other modalities. It supports this observation with empirical analysis and a theoretical argument under a shared additive-noise model, then proposes ManiGest, a plug-in framework combining confidence-weighted token-level optimal transport for cross-modal alignment with an SPD-based regularizer that preserves within-modality geometry using shallow-layer features as a reference. Experiments on Kinetics50-C and VGGSound-C, covering 15 video and six audio corruptions, show improved average accuracy when ManiGest is added to READ, TSA, and ABPEM. Component ablations support both regularizers, while sensitivity analyses and visualizations provide additional evidence for the approach; validation remains focused on synthetic corruptions affecting one modality at a time.

Critical strengths
  1. A compelling geometric perspective: The paper links corruption-induced changes in representation geometry to performance degradation, motivating a concrete target for multimodal adaptation.
  2. Consistent gains across baselines: ManiGest improves average accuracy across three adaptation methods on two benchmarks, with ablations supporting both cross-modal alignment and within-modality geometry preservation.
Critical weaknesses
  1. i) The theoretical proof contains substantive errors. Equation (15) equates the inner product of mean vectors with their cosine similarity, and the Taylor expansion in Eq. (20) is incorrect. The concentration argument also does not establish the stated probability guarantee. These errors undermine Theorem 3.1 and the theoretical explanation of geometric contraction. ii) The motivation, metric, and objective describe different geometric properties. The motivation concerns inter-class compactness, the symmetry score measures centered cross-modal distances, and BGSR aligns tokens within individual samples. The paper does not show that optimizing BGSR restores the inter-sample geometry underlying its central claim.
  2. i) The reliability of small gains is unclear. Several average improvements are only 0.1–0.5 percentage points, but no standard deviations or confidence intervals across repeated runs are reported. Given the sensitivity of online adaptation to sample order and batch composition, these results do not establish consistent improvements over the baselines or simpler ablations. ii) Computational overhead is not evaluated. The method adds token-level optimal transport and SPD matrix operations without reporting adaptation time, throughput, or peak GPU memory. It is therefore difficult to judge whether the gains justify the additional test-time cost. iii) The evaluation leaves important failure conditions untested. Experiments focus on corruption of one modality at a time. Although adaptation continues within each corruption stream, simultaneous corruption and sequences that switch corruption types or affected modalities are not clearly evaluated. These settings directly test the method’s reliance on reliable cross-modal guidance.
Questions for authors
  1. Hyperparameter selection: What label-free criterion is used to select α\alpha, β\beta, and τ\tau? Are the settings fixed across corruptions, and does selection involve revisiting test data or accessing future samples? Appendix A.2 refers to Table 1 for these settings, but that table reports accuracy.
  2. Implementation and batch dependence: What batch size, optimal transport solver, convergence criterion, and SPD stabilization settings are used? For a batch of one, the self-similarity matrices make SIGP identically zero. How much of the reported benefit remains at small batch sizes?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.