- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies multimodal test-time adaptation (MTTA) under modal-specific distribution shift. The authors observe that the corrupted modality becomes more compact in representation space, which breaks the geometric symmetry between modalities, and attempt to explain this phenomenon through a graphical model and a theorem. They propose ManiGest, a plug-and-play module with two losses: BGSR, a token-level optimal transport loss between modalities, and SIGP, an AIRM distance between shallow-layer and last-layer kernel matrices. ManiGest is added to READ, TSA, and ABPEM, and shows accuracy gains on Kinetics50-C and VGGSound-C.
Critical strengths
- Plug-and-play module with consistent average gains.
- ManiGest improves the average accuracy of three MTTA baselines on two datasets under both audio and video corruptions.
- Per-corruption results, component ablations, and sensitivity analyses are provided.
- Representation-level view on MTTA.
- The paper focuses on the geometry of modality-specific representations, which is less studied than fusion-level methods.
- The compactness change of the corrupted modality is illustrated through t-SNE visualizations across 21 corruption types.
Critical weaknesses
- The support for geometric symmetry non-conservation seems insufficient.
- Theorem 3.1: Eq. (13) assumes an identical noise vector for both samples, which may not represent sample-specific corruptions. Moreover, this assumption does not generally guarantee an increase in expected cosine similarity. The proof also appears to have several issues: Eq. (15) treats ⟨μ_i, μ_j⟩ as the cosine of the means, Eq. (20) appears to use an incorrect Taylor approximation, and Eq. (21) bounds an expectation with a high-probability bound. In addition, stating a deterministic expectation inequality “with high probability” is not well defined.
- Graphical model: In a fixed corrupted test domain, the samples can remain conditionally independent given c. Moreover, statistical dependence does not necessarily imply increased representation compactness.
- Definition and measurement: “Geometric symmetry” is not formally defined. The Symmetry Score measures paired cross-modal distance, which seems different from the compactness described in the text. Also, t-SNE may not be reliable for comparing compactness.
- Correlation and causation: The correlation in Fig. 2 may simply reflect that stronger corruption causes both larger representation change and larger accuracy drop. Also, SIGP alone gives a gain similar to BGSR (+2.0 vs. +2.3 with READ), so it is not clear that the gain comes from symmetry restoration.
- The method design and experimental validation could be strengthened.
- Direction of BGSR: Although each term uses stop-gradient, the bidirectional BGSR objective still updates both modalities. It is unclear how this design prevents the clean modality from being aligned toward corrupted representations.
- Reference of SIGP: The shallow-layer reference is computed from the same corrupted input, so it may not reflect the pretraining geometry.
- Gains and variance: Gains are small in several settings (e.g., +0.1 and +0.4 on Kinetics50-C audio), and results are reported without variability across independent runs. Also, the +8.1 gain on VGGSound-C audio is over a reproduced ABPEM (27.1) that is lower than READ (32.4).
- Ablation values: In Table 5, the BGSR-only and SIGP-only rows for ABPEM are identical in 12 of the 15 corruptions. These values may need to be checked.
- Missing comparisons: Simpler alternatives (e.g., pooled cosine alignment, Euclidean distance instead of AIRM) are not compared, and related work on the modality gap (Liang et al., 2022) is not discussed. The computational cost is also not reported.
Questions for authors
Clarification Questions
- Noise assumption: Is ε in Eq. (13) shared by all samples? Under which conditions does Theorem 3.1 hold, and does it still hold with independent noise per sample?
- Symmetry measurement: Can the authors report the Symmetry Score of each baseline with and without ManiGest?
- Direction of BGSR: When the corrupted modality is known (oracle setting), how does one-way BGSR compare with the proposed bidirectional loss?
- Ablation values: Can the authors confirm the BGSR-only and SIGP-only results for ABPEM in Table 5?
- Hyperparameter selection: How are hyperparameters selected “in a label-free manner using the target test data”? The text refers to Table 1, but Table 1 shows results.
- Cost and variance: What are the additional time and memory costs, and how much do results vary across independent runs?
Suggestions
- If possible, please evaluate other modality pairs or backbones, and bimodal corruption settings (e.g., AVRobustBench).
- Please compare with simpler alignment losses and distances to justify the OT and SPD-manifold design.
- Please clarify the terminology of the “probabilistic Birkhoff polytope,” which appears to correspond to the standard transportation polytope in OT.