acceptodds
Under review as a conference paper at ICLR 2027

Representation Editing for Multimodal Test-Time Adaptation

Abstract

Multimodal test-time adaptation (TTA) aims to adapt a pretrained multimodal model online to distribution shift across modalities using unlabeled test data, showing broad potential in real-world applications. However, existing methods primarily focus on adjusting fused features to bridge the source-target gap, lacking explicit control over intermediate representation misalignment, which is a key driver of performance drop under distribution shift. In this work, we tackle this challenge from the perspective of representation engineering. Unlike previous TTA methods that update fusion weights in place, we propose FourIer Representation Editor (FIRE), a novel multimodal TTA approach that directly edits semantically rich intermediate representations. Specifically, we first adopt representation editors into each intermediate layer of the unimodal encoders, enabling layer-wise calibration of unimodal representations. To further enhance the diversity and stability of the low-rank editing subspaces, each representation editor performs frequency domain mixing via the fast Fourier transform to construct structured bases. Moreover, we introduce multi-level adaptation objectives to optimize these editors, jointly promoting cross-modal semantic alignment, source-target statistical alignment, and asymmetric prediction consistency. In this way, FIRE yields aligned unimodal representations for fusion and further improves prediction reliability. Extensive experiments on two widely used multimodal benchmarks under various corruption types demonstrate the superiority of our approach over existing multimodal TTA methods. The source code is available at https://anonymous.4open.science/r/FIRE-BF81.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Test-time Adaptation, Multimodal Test-time Adaptation, Representation Editing, Multimodal Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_2NCjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces the Fourier Representation Editor (FIRE), a novel approach to multimodal test-time adaptation (TTA). Moving beyond existing methods that primarily adjust fused features, FIRE intervenes from a representation engineering perspective by directly editing semantically rich intermediate representations.

Critical strengths
  1. Algorithmic Innovation and Performance: The paper successfully applies Fourier representation editing to multimodal TTA for the first time, achieving results that consistently outperform current state-of-the-art methods.

  2. Comprehensive Evaluation: The authors conducted extensive experiments evaluating the framework against a wide variety of corruption types (e.g., on Kinetics50-C and VGGSound-C), rigorously validating the strong empirical performance of FIRE.

Critical weaknesses

1.The proposed method is not a strictly source-free TTA approach, as it explicitly requires the pre-computation and caching of source data statistics (means and standard deviations)[cite: 6, 7]. This dependency is conceptually similar to Domain Adaptation methods like DUA (https://ieeexplore.ieee.org/abstract/document/9879821) and diminishes its practical applicability. In real-world blind deployments, caching source distributions in advance is often unrealistic and introduces additional computational overhead.

  1. FIRE applies its representation editing equally across all intermediate layers. This uniform treatment can harm overall computational efficiency and potentially lead to the accumulation of errors through the network. The framework would benefit significantly from an adaptive layer-selection mechanism. For instance, employing eXplainable AI (XAI) techniques, such as Grad-CAM, to identify and selectively edit only the most feature-relevant layers could optimize both computational efficiency and adaptation accuracy.
Questions for authors

see issues raised in weaknesses

3 comments
AuthorsOpenReview ↗

We clarified FIRE’s source-free TTA setting and provided additional experiments on computational efficiency and adaptive layer selection.

Thank you very much for your thorough review of our work and your insightful suggestions.

W.1. Source-Free TTA and Cached Source-Domain Statistics

We would like to clarify the distinction between accessing raw source-domain data at test time and using precomputed source-domain statistics in TTA. Before deployment, FIRE computes the mean and standard deviation of layer-wise representations offline using a minibatch of source samples. No raw source samples are accessed during test-time adaptation. Thus, FIRE remains consistent with the source-free TTA setting adopted by methods such as FOA. These statistics are computed only once before deployment, introducing minimal additional overhead. Although DUA also leverages statistical information, it differs from FIRE in the parameters being updated, how the statistics are utilized, and the specific representation calibration mechanism.

W.2. Computational Efficiency and Adaptation Strategies

Thanks to the low-rank design of the representation editors, FIRE maintains high computational efficiency even when applied to all intermediate layers. As shown in Table A, we additionally report the runtime of various TTA methods for a single run on VGGSound-C. The results demonstrate that FIRE requires fewer updated parameters. Despite the additional computational overhead, FIRE achieves substantial performance improvements.

We agree that adaptive layer selection could further improve efficiency, as also discussed in the Limitations section. Moreover, the results in Table 5 confirm that different layers contribute differently. However, approaches such as Grad-CAM may introduce additional computational overhead. We are currently exploring this direction. We experimented with dynamically selecting the Top-K layers with the largest distribution shifts in each modality for representation editing. As shown in Table B, while this improves efficiency, it leads to a performance drop. Further investigation is needed to better understand this trade-off.

Table A. Computational overhead.

MethodSourceTentSARREADSuMiBriMPRFIRE
Time (s)66.8161.1236.6135.2424.4186.2219.0
Params (M)00.2180.2181.7720.2180.1690.068

Table B. Adaptive layer selection.

MethodACCTime (s)
All layers58.2219.0
Adaptive57.5189.6

The relevant discussion and experimental results on computational efficiency have been added to Appendix E.2.

I thank the authors for their responses and supplementary experiments, and I will retain my score.

Dear Reviewer 2NCj,

Thank you for engaging with our response and for the positive feedback. We appreciate your time and support.

Reviewer_3c8ROpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This manuscript proposes FIRE, a test-time adaptation method for audio-visual CAV-MAE models on Kinetics50-C and VGGSound-C. A low-rank editor is inserted after every transformer layer of each unimodal encoder and edits the features in a Fourier-enhanced orthonormal subspace, while the backbone stays frozen. The editors are trained online with three losses: cross-modal contrastive alignment, layer-wise source-target moment matching, and asymmetric masked-modality consistency. At severity 5 FIRE beats 12 baselines, e.g. 47.0% vs 41.5% of PTA on Kinetics50-C multimodal shift.

Critical strengths
  1. To my knowledge this is a pioneer work to treat multimodal TTA as layer-wise editing with a fully frozen backbone. The no-op initialization plus the Lipschitz bound in Theorem 1 gives a stability guarantee that most entropy-minimization baselines lack.

  2. The evaluation is comprehensive: 2 benchmarks, unimodal and 90-combination shifts, 12 baselines including 2025/26 SOTA (ABPEM, SuMi, TSA, PTA, BriMPR). And the performance analysis is clear.

  3. Parameters efficient: 0.068M at rank 2 vs 0.218M of Tent (Table 6), still best.

Critical weaknesses
  1. Novelty is incremental. The editor is ReFT (Wu et al., 2024b), and the Fourier basis is already used in PEFT (FoLA, DFT-LoRA, FFP). BriMPR (AAAI'26) also aligns unimodal features to the source and uses inter-modal contrast, and FIRE beats it on VGGSound-C by only +0.5/+0.4.

  2. Results are means over 5 seeds, but no std and no significance test appear anywhere.

  3. Compute cost is not reported: it seems Eq. 9 needs two more masked passes per batch plus an editor per layer, but Table 6 only gives parameters.

  4. Only severity 5 is tested and the real-world shift benchmarks of PTA/BriMPR are missing.

Questions for authors
  1. Please report std and a paired significance test over the 5 seeds, especially for the small margins on VGGSound-C (video 58.2 vs 57.7; audio 36.9 vs 36.5).

  2. Compared with Tent and READ, what is the wall-clock and throughput overhead from the two extra masked passes?

  3. How exactly is 'mask 75% of one modality' done for audio and video?

5 comments
AuthorsOpenReview ↗

We clarified FIRE’s methodological differences and implementation details, and provided additional experiments on statistical variability, computational efficiency, and real-world distribution shifts.

We sincerely appreciate your positive assessment of our work and constructive suggestions.

W.1. Differences from Related Methods

FIRE does not directly reuse ReFT or FourierFT. Unlike FourierFT, which performs frequency-domain parameterization in the weight space, FIRE uses Fourier transforms to construct orthogonal editing subspaces that directly modify intermediate encoder representations. Unlike the implicit prompt-based adaptation of BriMPR and FOA, FIRE performs explicit layer-wise representation calibration. The analyses in Table 5 and ablation results in Table 7 support the effectiveness of our design.

W.2. & Q.1. Standard Deviation and Statistical Significance

Following prior experimental protocols and reporting conventions, all results in our paper are averaged over five random seeds. We additionally report the standard deviations of FIRE in Table A. FIRE achieves the highest mean accuracy across all six settings, with low std across random seeds.

Compared with strong baselines, FIRE achieves substantial improvements on Kinetics50-C, while the gains on VGGSound-C are relatively modest. Notably, strong baselines also show limited gains on VGGSound-C, highlighting the difficulty of further improving performance on this dataset. Although the improvements vary across datasets and corruption settings, the overall results show that FIRE consistently achieves numerical gains under diverse distribution shifts, particularly in more challenging multimodal shift scenarios.

Table A. Mean ± standard.

DatasetMethodAudioVideoBoth
Kinetics50-CREAD71.5±0.3263.4±0.4835.9±0.35
BriMPR72.0±0.1865.9±0.1740.9±0.25
FIRE73.4±0.1268.6±0.1647.0±0.18
VGGSound-CREAD32.4±0.1556.7±0.6714.6±0.20
BriMPR36.5±0.1757.7±0.1120.7±0.25
FIRE36.9±0.1558.2±0.1322.0±0.24

W.3. & Q.2. Computational Overhead

In addition to parameter counts, we compare the runtime of various TTA methods for a single run on VGGSound-C using RTX 3090 GPU, as shown in Table B. FIRE* denotes the runtime excluding the masking operation. Due to layer-wise representation editing, FIRE has a longer runtime than Tent and READ, but remains more efficient than SAR and SuMi. Compared with the augmentation-based method DeYO, FIRE incurs slightly lower computational overhead. Despite the additional computational overhead, FIRE achieves substantial performance improvements.

Table B. Computational overhead.

MethodSourceTentSARDeYOREADSuMiBriMPRFIREFIRE*
Time (s)66.8161.1236.6220.4135.2424.4186.2219.0189.3
Params (M)00.2180.2180.2181.7720.2180.1690.0680.068

The relevant discussion and experimental results have been added to Appendix E.2.

W.4. Experimental Setup and Real-world shifts

To ensure comparability with prior work, we follow the widely adopted severity level 5 evaluation protocol in multimodal TTA, covering 15 video corruptions, 6 audio corruptions, and 90 bimodal corruption combinations. We agree that evaluations at other severity levels and in real-world scenarios would provide additional insights. Nevertheless, our current experiments already cover diverse corruption mechanisms and both unimodal and bimodal distribution shifts, rather than a single corruption type. For real-world distribution shifts, as the code for methods such as BriMPR is not publicly available, we reproduced them following their papers and referring to the open-source model of CASP [1], on which they are based. As shown in Table C, FIRE achieves the best performance under real-world cross-dataset distribution shifts. We attribute this advantage primarily to FIRE's explicit representation editing mechanism, which reduces sensitivity to such shifts.

Table C. Results on real-world distribution shifts.

MethodMOSI →\rightarrow SIMSSIMS →\rightarrow MOSI
Source46.040.1
READ32.440.9
SuMi44.441.6
BriMPR58.240.4
FIRE60.242.0

The relevant discussion and experimental results on real-world distribution shifts have been added to Appendix E.3.

[1] Bridging the gap for test-time multimodal sentiment analysis, AAAI, 2025.

Q.3. Masking Implementation

Regarding 75% masking, it means masking 75% of the input information from the selected modality while leaving the other modality unchanged. Specifically, we first convert the audio and video inputs into token sequences using their respective patch embedding layers. We then randomly mask the token sequence of the selected modality by retaining only 25% of its tokens and discarding the remaining 75%, while keeping the other modality intact.

We provide a mask legend on the left side of Figure 2 and have added pseudocode for the masking procedure in Appendix E.4.

Dear Reviewer 3c8R,

We sincerely thank you again for your valuable feedback, which has helped us improve the quality of our work. As the discussion deadline is approaching, we kindly request that you review our rebuttal. We have also uploaded a revised PDF of the paper and its appendix, addressing your concerns to the best of our ability. We are happy to address any remaining concerns.

Thank you again for your time and effort!

Reviewer_G2idOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a Multimodal TTA approach, namely FIRE, which specifies how to modify intermediate representations by learning small Fourier parameterised editors and train them with feature alignment and consistency losses.

Critical strengths

FIRE introduce low-rank representation editing mechanism with a Fourier based orthogonal basis construction which is not used before in TTA setting.

FIRE improves over all the baselines it reports on the six aggregate conditions.

Critical weaknesses

My biggest issue with the paper is the positioning. The novelty in the paper appear much narrower than the claimed.

The novelty from the paper as stated by the authors "Specifically, we first introduce representation editors and place them after each transformer layer in every unimodal encoder, jointly optimizing unimodal representations layer-wise. By layer-wise calibrating unimodal representations, editors align the target distribution with the source, thereby facilitating reliable fusion. Notably, only the editors are updated in TTA, while the pretrained model remains entirely frozen."

Most of the claim is not exclusive to FIRE. I would not say that FIRE is the first to align intermediate representations to improve downstream tasks. The novelty is mainly in the "usage" of low-rank representation editing mechanism with a Fourier based orthogonal basis construction. The term "usage" is because this mechanism is not novel as well, however the usage of it for test time adaptation seems new. Therefore, I think it is crucial that the authors tone down their broad claim in the paper.

Also, its advantage in comparison to SOTA is modest in most of the settings, with some exceptions.

One of the recent emerging trend in TTA is the usage or reinforecement learning, however the paper is not discussing or comparing to any prior work in this domain. I appreciate that TTRL can actually be paired with FIRE as their objectives are complementary, neveretheless, both are serving similar purpose, so there should be a comparison against these methods.

The work is inceremental but it is useful, I am happy to increase my score if my concerns are addressed.

Questions for authors

Why only the mean accross seeds is reported, would you also please provide the variance, as it would matter given the small gain in some benchmarks.

Is it possible to compare to at least one test time reinforcement approaches, e.g. GRPO-TTA?

3 comments
AuthorsOpenReview ↗

We clarified FIRE’s novelty, reported statistical variability, and conducted additional comparisons with GRPO-TTA.

Thank you very much for your thorough review of our work and your insightful suggestions.

W.1. Novelty of FIRE

We acknowledge that FIRE is inspired by ReFT and that Fourier transforms have been explored in parameter-efficient fine-tuning. In "we first introduce representation editors", "first" refers to the order of operations within our framework, rather than a claim of methodological priority.

The key contribution of FIRE is a representation-level adaptation framework specifically designed for multimodal TTA. It introduces layer-wise representation editing into multimodal TTA and directly calibrates intermediate representations through Fourier-enhanced, low-rank orthogonal subspaces. The results in Tables 5 and 7 support the effectiveness of this design.

Q.1. Standard Deviation and Statistical Significance

Following prior experimental protocols and reporting conventions, all results in our paper are averaged over five random seeds. We additionally provide the standard deviations of our method in Table A. FIRE achieves the highest mean accuracy across all six settings, with low std across random seeds.

Compared with strong baselines, FIRE achieves substantial improvements on Kinetics50-C, while the gains on VGGSound-C are relatively modest. Notably, strong baselines also show limited improvements on VGGSound-C, highlighting the difficulty of further improving performance on this dataset. Although the gains vary across datasets and corruption settings, the overall results demonstrate that FIRE consistently achieves numerical improvements under diverse distribution shifts, with particularly notable gains in more challenging multimodal shift scenarios.

Table A. Mean ± std.

DatasetMethodAudioVideoBoth
Kinetics50-CREAD71.5±0.3263.4±0.4835.9±0.35
BriMPR72.0±0.1865.9±0.1740.9±0.25
FIRE73.4±0.1268.6±0.1647.0±0.18
VGGSound-CREAD32.4±0.1556.7±0.6714.6±0.20
BriMPR36.5±0.1757.7±0.1120.7±0.25
FIRE36.9±0.1558.2±0.1322.0±0.24

W.2. & Q.2. Comparison with GRPO-TTA

TTRL primarily targets LLM reasoning, while GRPO-TTA [1] relies on CLIP-based image-text similarities and reward mechanisms, which differ substantially from our CAV-MAE-based audio-visual classification setting. Direct adaptation thus requires redesigning the candidate generation and reward mechanisms. Notably, GRPO-TTA does not provide publicly available code.

We therefore reimplemented and adapted GRPO-TTA following its core principles. Specifically, we treat classification predictions as a policy distribution and select the Top-KK candidate classes for group-relative policy optimization. We replace the original CLIP similarity-based rewards with rewards based on cross-modal prediction consistency and differences in predicted probabilities. Finally, we optimize the Q/K/V parameters in the READ fusion attention module using the GRPO clipped objective.

As shown in Table B, compared with Source, GRPO-TTA achieves marginal improvements under video shifts but suffers performance degradation under audio shifts. This may stem from unreliable candidate predictions and reward signals under severe noisy distribution shifts. This suggests that reinforcement learning-based TTA is a promising research direction that warrants further investigation and careful design in future work.

Table B. Comparison with GRPO-TTA on Kinetics50-C (left) and VGGSound-C (right).

MethodAudioVideoAudioVideo
Source69.460.525.056.2
READ71.563.432.456.7
GRPO-TTA*67.860.715.556.3
FIRE73.468.636.958.2

[1] GRPO-TTA: Test-time visual tuning for vision-language models via GRPO-driven reinforcement learning, arXiv, 2026.

The relevant discussion and experimental results on GRPO-TTA have been added to Appendix E.1.

I thank the authors for their detailed response. I still believe that the novelty is limited. I am still standing by my initial jusdgment that the work is incremental but useful. However, most of my concerns are addressed and therefor I am increasing my score. Thanks.

Dear Reviewer G2id,

Thank you for engaging with our response and for updating your score. We appreciate your time and valuable feedback.

Comments on the paper

1 comment

We sincerely thank all reviewers for their constructive feedback and valuable suggestions. We have carefully revised the manuscript to address the concerns raised in the reviews. The main revisions are summarized below:

  1. Added comparisons with reinforcement learning-based test-time adaptation (Appendix E.1). We reimplemented and adapted GRPO-TTA for audio-visual classification and conducted additional experiments to further demonstrate the effectiveness of FIRE under challenging distribution shifts.

  2. Expanded computational efficiency analysis (Appendix E.2). We supplemented comparisons of runtime and trainable parameters with various test-time adaptation methods, providing further insights into the computational overhead of FIRE and its trade-off between adaptation performance and efficiency.

  3. Extended evaluations to real-world distribution shifts (Appendix E.3). We conducted additional cross-dataset adaptation experiments on CMU-MOSI and CH-SIMS, comparing FIRE with existing methods to further validate its effectiveness under real-world distribution shifts.

Additionally, we clarified the implementation details of the random masking strategy and provided corresponding pseudocode in Appendix E.4 to improve the clarity and reproducibility of our method.

The revisions in the manuscript are highlighted in red for ease of reference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.