acceptodds
Under review as a conference paper at ICLR 2027

MODE-TTA: Multi-Mode Density Modeling for Test-Time Adaptation of 3D Vision-Language Foundation Models

Abstract

3D vision–language foundation models (VLFMs) for point clouds exhibit strong zero-shot generalization capabilities, yet their performance often deteriorates under real-world distribution shifts. Test-time adaptation (TTA) seeks to address this challenge by adapting models during inference using only unlabeled target data. Recent cache-based TTA methods address distribution shifts in online settings by maintaining a memory of representative test samples and refining predictions based on feature affinities between the current sample and the cached instances. However, their effectiveness is largely dependent on the quality of pseudo-labels, which may deteriorate under substantial domain shifts. Moreover, they do not capture the underlying structure and dynamics of the incoming test distribution. To overcome these limitations, we propose MODE-TTA, a streaming density modeling framework that explicitly models class-conditional feature distributions. Each class is represented by a Gaussian Mixture Model (GMM) to capture the multi-cluster structure of 3D features, with mixture parameters updated incrementally via a hierarchical expectation–maximization procedure guided by soft zero-shot assignments. To further improve cross-modal alignment, we introduce a lightweight residual adaptation for text embeddings. Extensive experiments across diverse distribution shifts demonstrate that MODE-TTA consistently outperforms competitive baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Test-Time Adaptation, 3D Vision-Language Models, Point Clouds, Distribution Shift

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZuYVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies online test-time adaptation for 3D vision-language foundation models under distribution shift. The proposed MODE-TTA models each class-conditional feature distribution using a Gaussian Mixture Model and updates the mixture parameters online through a hierarchical EM procedure guided by soft zero-shot assignments.

The method also introduces residual text alignment to improve the consistency between class text embeddings and the learned target-domain feature distributions. Experiments on several corrupted and clean 3D datasets with Uni3D and ULIP-2 show consistent improvements over cache-based and Gaussian-based TTA baselines.

Critical strengths
  1. The paper addresses a relevant problem and proposes a clear extension beyond prototype-based TTA by explicitly modeling multi-modal class-conditional feature distributions.

  2. The empirical evaluation is broad, covering several corruption benchmarks, clean datasets, and two different 3D vision-language backbones. The ablation studies also show that the GMM modeling is the main source of improvement.

Critical weaknesses
  1. The methodological novelty is somewhat limited. The main idea is to replace prototype or single-Gaussian modeling with a GMM and combine it with residual text adaptation. While reasonable and effective, the overall framework feels more like a natural extension of prior density-based TTA methods than a fundamentally new formulation.

  2. The method relies on zero-shot predictions as soft assignments when updating the GMM. Under severe domain shift, these assignments may themselves be inaccurate, which could lead to error accumulation in the online density estimates. The paper would benefit from a clearer analysis of robustness to noisy pseudo-labels.

Questions for authors
  1. How robust is MODE-TTA when the initial zero-shot predictions are unreliable? Is there any analysis showing how errors in the soft class assignments affect the online GMM updates?

  2. Could the authors clarify the main conceptual difference from prior density-based TTA methods such as DOTA and FreeTTA, beyond using multiple Gaussian components per class?

  3. How sensitive is the method to the number of mixture components M across different datasets and backbones?

1 comment
AuthorsOpenReview ↗

Methodological Novelty, Pseudo-Label Robustness, and Mixture Sensitivity

Thank you for the constructive review. We appreciate the recognition of MODE-TTA's multimodal density modeling and empirical improvements. We address the concerns with additional experiments using Uni3D-L, severity 5, batch size 1, and M=8M=8 unless explicitly varied for sensitivity analysis. Results are mean ± std over 3 stream orders × 15 corruptions, with matched features and streams.

W1/Q2 — Methodological novelty beyond DOTA and FreeTTA. The novelty of MODE-TTA goes beyond replacing a single Gaussian with a GMM. We propose a coupled probabilistic adaptation framework in which streaming density estimation supports both target-domain classification and cross-modal alignment:

  1. Hierarchical probabilistic adaptation: Unlike the single-Gaussian modeling of DOTA and FreeTTA, MODE-TTA jointly estimates class and mixture responsibilities by combining zero-shot posteriors with component likelihoods. This enables online learning of multimodal class distributions with component-specific means, variances, and mixture weights, without hard pseudo-labels.
  2. Density-driven cross-modal alignment: Rather than independently adapting text prototypes, we formulate a bidirectional likelihood-based objective that aligns persistent text residuals with the evolving class-conditional distributions. Thus, the learned densities serve both as classifiers and as supervision for text adaptation.
  3. Confidence-aware prediction: We integrate the zero-shot and learned density models through entropy-guided fusion and progressive density weighting, accounting for prediction uncertainty and limited early observations.

The central contribution is the formulation of multimodal density estimation as a shared foundation for online classification and text adaptation, rather than simply combining existing adaptation components. Crucially, the learned distributions serve two complementary roles: probabilistic classification and supervision for cross-modal alignment. This formulation distinguishes MODE-TTA from simply combining GMM estimation with existing residual adaptation techniques.

W2/Q1 — Robustness to unreliable zero-shot assignments.

(a) Natural corruption. On SONN-C, zero-shot predictions are incorrect for approximately 62% of samples, yet MODE-TTA improves accuracy by 10.1 points in the additional evaluation. In Tables 1–3, MODE-TTA improves over Source-Only across all 45 corruption settings, including severe shifts (SONN-C Gaussian: 25.99→40.27; MN40-C LiDAR: 22.93→40.31).

W2/Q1 — Robustness to unreliable zero-shot assignments.

(a) Natural corruption. On SONN-C, zero-shot predictions are incorrect for approximately 62% of samples, yet MODE-TTA improves accuracy by 10.1 points. In Tables 1–3, MODE-TTA improves over Source-Only across all 45 corruption settings, including severe shifts (SONN-C Gaussian: 25.99→40.27; MN40-C LiDAR: 22.93→40.31).

(b) Injected assignment noise. We replace zero-shot responsibilities for a fraction pp of samples with one-hot assignments to randomly selected incorrect classes:

Noise pp0%10%20%30%50%
SONN-C, zero-shot38.3438.3438.3438.3438.34
SONN-C, full48.4545.2742.7540.0035.77
SONN-C, density only46.9043.4239.7235.8729.66
MN40-C, zero-shot59.1559.1559.1559.1559.15
MN40-C, full70.86–68.07–62.72

Despite injected errors, MODE-TTA remains above zero-shot through 30% noise on SONN-C and 50% on MN40-C. Entropy-guided fusion mitigates degradation relative to the density branch alone.

(c) Soft vs. hard responsibilities.

AssignmentSONN-CMN40-C
Hard (argmax)40.71±0.6468.28±0.67
Soft (ours)48.45±0.3570.86±0.13
Oracle labels65.03±0.3781.44±0.15

Soft responsibilities improve accuracy by 7.74 / 2.58 points over hard assignments, reducing the impact of incorrect confident predictions. Oracle results quantify the remaining improvement potential. We will add these robustness analyses.

Q3 — Sensitivity to mixture components MM. The main experiments use M=8M=8. We additionally vary M∈{1,2,4,8,12}M\in\{1,2,4,8,12\} across Uni3D and ULIP-2 to assess sensitivity.

The results show that multimodal modeling is substantially better than a single Gaussian in the full Uni3D pipeline. For M≥2M\ge2, accuracy remains relatively stable: within 1.5 points on Uni3D SONN-C, 0.8 on Uni3D MN40-C, and 1.2 on ULIP-2 MN40-C. The largest degradation at M=1M=1 appears related to its interaction with residual alignment and progressive density weighting.

We will expand Fig. 5 with sensitivity results across datasets and backbones, clarifying the robustness of the selected M=8M=8 configuration.

Reviewer_AEU4OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MODE-TTA, an online test-time adaptation framework for 3D vision-language foundation models under distribution shifts. It models class-conditional feature distributions using Gaussian Mixture Models (GMMs), which are incrementally updated through hierarchical EM guided by zero-shot predictions. The method further incorporates noise-based augmentation and residual text alignment to improve distribution estimation and cross-modal alignment. Experiments on ModelNet40-C, ShapeNet-C, and ScanObjectNN-C demonstrate consistent improvements over existing baselines.

Critical strengths
  • GMMs provide a principled way to capture multimodal class-conditional feature distributions beyond single-Gaussian assumptions.

  • Experiments across multiple corruption benchmarks and two 3D vision-language backbones demonstrate consistent improvements, supported by component ablations.

Critical weaknesses
  • Incremental methodological novelty. MODE-TTA primarily extends online EM-based distribution modeling, as explored in FreeTTA [1], from single-Gaussian to GMM-based modeling. Moreover, residual-based text adaptation has been explored in DPE [2]. The paper should better clarify the methodological novelty beyond these existing components and their combination.

  • Limited exploitation of 3D-specific structures. Although evaluated with 3D vision-language models, MODE-TTA operates on global feature embeddings without explicitly exploiting geometric structures or spatial relationships in point clouds. It remains unclear what makes the proposed adaptation mechanism particularly suitable for 3D data rather than a general distribution-based TTA method.

  • Limited robustness to Non-IID streams. MODE-TTA accumulates distribution statistics throughout the test stream without an explicit forgetting mechanism. Consequently, outdated observations may continue to influence predictions when the target distribution changes over time.

  • Insufficient efficiency evaluation of the full method. The reported throughput of approximately 97% is for the GMM-only variant, excluding noise augmentation and residual text adaptation. These additional components introduce extra encoder forward passes and gradient-based optimization, but their individual and end-to-end overheads are not sufficiently characterized.

  • Incomplete discussion of recent related work. The paper does not discuss several relevant recent methods, including PTA [3] and BayesMM [4] for 3D VLMs TTA.

[1] Free on the Fly: Enhancing Flexibility in Test-Time Adaptation with Online EM. CVPR 2025

[2] Dual Prototype Evolving for Test-Time Generalization of Vision-Language Models. NeurIPS 2024

[3] Prototype-Based Test-Time Adaptation of Vision-Language Models. ICML 2026

[4] Adapting Point Cloud Analysis via Multimodal Bayesian Distribution Learning. CVPR 2026

Questions for authors
  • What are its key technical contributions beyond GMM-based modeling and residual text adaptation?

  • Since MODE-TTA operates primarily on global feature embeddings without explicitly modeling geometric structures, what makes the proposed method particularly suitable for 3D data compared with general distribution-based TTA methods for 2D vision-language models?

  • Given that MODE-TTA accumulates distribution statistics without an explicit forgetting mechanism, how does it perform when the test distribution changes sequentially? Could outdated observations hinder adaptation to new domains?

  • Can the authors report the end-to-end latency, throughput, and peak GPU memory consumption of the complete method, separately quantifying the overhead of noise augmentation and residual text alignment?

2 comments
AuthorsOpenReview ↗

Response to Reviewer AEU4: Novelty, Non-IID Robustness, and Efficiency

Thank you for the constructive review. We appreciate the recognition of MODE-TTA's multimodal density modeling and empirical improvements. We address each concern with additional experiments. New i.i.d. results use Uni3D-L, severity 5, batch size 1, and 3 stream orders × 15 corruptions (mean ± std), with matched streams across methods.

W1/Q1 — Novelty beyond FreeTTA and DPE. MODE-TTA introduces a unified probabilistic formulation coupling multimodal density estimation, density-guided text alignment, and adaptive prediction:

  1. Hierarchical streaming EM: Unlike FreeTTA's single Gaussian per class, MODE-TTA learns multiple components with individual diagonal covariances and mixture weights. Joint zero-shot class and mixture responsibilities enable closed-form online updates without hard pseudo-labels.
  2. Density-guided text alignment: Unlike DPE's sample-specific entropy optimization, MODE-TTA learns persistent text residuals through a bidirectional contrastive objective over class-conditional log-likelihoods, directly aligning text embeddings with evolving densities.
  3. Uncertainty-aware fusion: Inverse-entropy weighting and progressive density scaling dynamically combine zero-shot and density predictions without fixed fusion coefficients.

The key novelty is that learned multimodal densities jointly drive classification and text adaptation, rather than independently combining existing EM and residual-learning techniques.

Under matched features and streams:

Our density-based variant improves over DOTA by 4.69 / 5.18 points, demonstrating the benefit of multimodal modeling independently of additional components. We will clarify these distinctions and expand the FreeTTA/DPE discussion.

W2/Q2 — Why 3D? We agree that the formulation operates on global embeddings and is modality-agnostic. Its suitability for 3D stems from modeling multimodal features induced by geometric variability, partial observations, and sensor corruption. Point-cloud augmentation enriches density estimation, while soft assignments reduce reliance on unreliable pseudo-labels. On SONN-C, soft assignments achieve 48.45% versus 40.71% with hard assignments, despite 38.34% zero-shot accuracy. MODE-TTA improves over Source-Only across all 45 corruption settings in Tables 1–3. We will clarify this scope and consider 2D extensions in future work.

W3/Q3 — Non-IID robustness. We evaluate class-sorted streams, Dirichlet label shift (α=0.1\alpha=0.1), and continual corruption shift (15 corruptions sequentially, without reset). We also test exponential forgetting (c←βcc\leftarrow\beta c, β=0.999\beta=0.999).

StreamZSDOTAOurs+forget
SONN-C i.i.d.38.3442.7548.4548.17
SONN-C class-sorted38.3441.7545.7945.37
SONN-C Dir. 0.138.3442.2945.6445.55
SONN-C continual38.34–43.6845.77
MN40-C i.i.d.59.1564.4770.8670.36
MN40-C class-sorted59.1561.74*67.0364.00
MN40-C Dir. 0.159.1563.68*70.1368.77
MN40-C continual59.15–65.3850.84

DOTA results for these settings use one stream order.

Under label shift, MODE-TTA maintains gains of 7.3–11.0 points over zero-shot and 3.4–6.5 over DOTA. Under continual shift, it retains gains of 5.34 / 6.23 points over zero-shot. Forgetting improves SONN-C continual performance but substantially hurts MN40-C, suggesting that indiscriminate discounting can erase useful statistics. We will add these experiments, acknowledge abrupt shifts as a limitation, and explore shift-aware forgetting.

W4/Q4 — Full-method efficiency. We agree that end-to-end efficiency must be reported. We measured CUDA-synchronized latency after warm-up on an RTX PRO 6000 (batch size 1). Accuracies are from Table 5 of the submitted paper.

SONN-Cms/samplesamples/svs. SourcePeak MBAcc.
Source-only19.9750.1100%137738.07
GMM only20.6448.496.7%137748.37
GMM + Aug27.9635.871.4%157049.98
Full31.5331.763.3%157150.90

We will replace "≈97% ... modest reduction" with: "The GMM-only variant retains ~97% of zero-shot throughput, while the full method retains 63%, improving accuracy from 38.07% to 50.90% on SONN-C." We will move this comparison into the main paper.

W5 — Recent related work. We thank the reviewer for identifying PTA (ICML'26) and BayesMM (CVPR'26). PTA maintains confidence-weighted prototypes without modeling multimodal density or covariance. BayesMM models textual and point-cloud distributions using Gaussian assumptions and Bayesian averaging. MODE-TTA instead learns multiple Gaussian components per class and aligns persistent text residuals with learned densities. We will expand the discussion and compare where implementations are available.

Thank you for the detailed response. However, after carefully reviewing the rebuttal, I believe several important concerns remain unresolved.

First, regarding the accuracy-efficiency trade-off, the component ablations show that GMM-based distribution modeling accounts for the majority of the performance improvement (+10.3%), while the two additional components introduced in MODE-TTA, namely noise-based augmentation and residual text alignment, provide relatively modest combined gains (+2.53%). More importantly, these two components introduce additional computational overhead. This raises concerns about whether their marginal benefits justify the increased complexity, particularly given the paper's emphasis on efficient online adaptation.

Second, regarding robustness under non-IID streams, the additional experiments do not fully establish the stability of the proposed method under substantial temporal distribution shifts. The authors only report one Dirichlet label-shift setting α=0.1\alpha=0.1, without evaluating more challenging settings with stronger class correlations, such as α=0.01\alpha=0.01 and α=0.001\alpha=0.001. Therefore, the current experimental evidence remains insufficient to establish its robustness under more challenging non-IID streams.

Finally, regarding comparisons with recent 3D VLM adaptation methods, the rebuttal explains the differences between MODE-TTA and methods such as PTA and BayesMM, but these explanations cannot substitute for quantitative comparisons.

Overall, the concerns regarding methodological novelty, the accuracy-efficiency trade-off, robustness under non-IID settings, and comparisons with recent methods remain insufficiently addressed. Therefore, I will maintain my original rating.

Reviewer_vNjaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies online test-time adaptation of 3D vision–language foundation models under distribution shift. MODE-TTA uses a Gaussian Mixture Model (GMM) for each class-conditional feature distribution. It updates the GMM parameters online using soft assignments derived from zero-shot predictions and combines Residual Text Alignment with logit fusion to improve classification performance. The experiments cover three point cloud corruption benchmarks, component ablations, different backbones, and clean data.

Critical strengths
  1. MODE-TTA achieves higher average accuracy than Uni-Adapter on ModelNet40-C, ShapeNet-C, and ScanObjectNN-C. The reported gain is larger on ScanObjectNN-C.
  2. Even without noise-based augmentation and residual text alignment, MODE-TTA reaches 48.37% accuracy on ScanObjectNN-C, compared with Uni-Adapter's 46.33%. This comparison shows that augmentation and residual learning do not account for the entire gain.
  3. The per-corruption results and component ablations help explain the benchmark averages. The evaluations with ULIP-2 and on clean data also show how the method performs with another backbone and without corruptions.
Critical weaknesses
  1. The variance update in Eq. (9) needs clarification. When a new sample changes the component mean, the squared deviations of previous samples also change. It is unclear how the update accounts for this. Appendix A gives a batch covariance expression. The authors should derive the corresponding recursive update, explain its relationship to Eq. (9), and specify which update the implementation uses.
  2. Appendix E.1 states that the stability curves are averaged over multiple random stream orders. However, the number of runs and the variability across runs are not clearly reported.
  3. The main paper reports approximately 97% throughput retention for the base variant, which omits augmentation and residual learning. It describes the full method used for the main results as incurring only a “modest reduction in speed.” Yet Supplementary Fig. 6 suggests a throughput reduction of approximately 40% relative to Source-Only. The main paper should report the full method’s throughput and relative reduction, and revise this description accordingly.
Questions for authors
  1. The expanded view of the Distribution Modeling module in Fig. 1 is readable, but its visual quality could be improved. Please consider using a vector graphic for a cleaner appearance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.