acceptodds
Withdrawn from ICLR 2027

MM-TTA: Multimodal Test-Time Adaptation with Consistency-Based Model Merging for 3D Object Detection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Robust test-time adaptation (TTA) for 3D object detection (3DOD) plays a vital role in autonomous driving systems when facing severe domain shifts. Existing TTA methods for 3DOD mainly rely on the LiDAR modality and are vulnerable to various types of domain shifts. In this paper, we introduce a two-stage multimodal TTA method, referred to as MM-TTA, to improve the adaptation capability of 3DOD models. In the first pre-training stage, we propose the Geometric Consistency Alignment (GCA) strategy to align the 3D perception ranges of source domain multimodal data with those of the target domain, thereby preventing mismatches in parameter and feature dimensions when deploying the pretrained fusion model to the target domain, while maintaining superior performance over LiDAR-based models despite differences in camera and LiDAR configurations between domains. In the second test-time adaptation stage, we further propose the Consistency-Based Model Merging strategy (CMM) for robust transfer learning on the target domain. Instead of choosing diverse historical checkpoints, we only select the most consistent ones for model merging and use the merged model to guide the subsequent fine-tuning. This is because historical fusion models with low similarity are more likely to exhibit performance discrepancies than complementarity, which can introduce interference and degrade the merged model. To reduce the computational costs from model selection, we also propose the sign consistency score to measure the similarity between model pairs, which is computed only once without repeatedly accessing historical models. The CMM strategy achieves better performance than previous methods with fewer time steps and significantly reduces the computational costs introduced by model selection and numerous parameters of the fusion model itself. Our method is evaluated on five datasets and eight corruption types, demonstrating its effectiveness under various domain shifts in real-world scenarios (with a 15.24% improvement over the second-best method in a cross-dataset scenario).

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Transfer Learning, Domain Adaptation, 3D Object Detection, Multimodal Fusion

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bQPWOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MM-TTA, a two-stage multimodal test-time adaptation method for 3D object detection that aims to improve the robustness of LiDAR-camera fusion models under domain shifts. In the first pre-training stage, Geometric Consistency Alignment (GCA) aligns the source-domain 3D perception space with that of the target domain by refining source point clouds, selecting valid camera images, filtering out-of-range labels, and adjusting camera configurations, so that the pretrained fusion model can be directly deployed on the target domain without parameter or feature-dimension mismatch. In the second TTA stage, Consistency-Based Model Merging (CMM) maintains a task-vector bank and a similarity matrix, uses a proposed Sign Consistency Score to measure model-pair similarity, selects the most mutually consistent historical models via a maximum clique, and merges them into a super-model that generates pseudo-labels for subsequent fine-tuning; historical models are periodically cleared to control memory and overfitting. Experiments on five datasets and eight corruption types show improvements over prior TTA and offline domain adaptation methods, including a 15.24% gain over the second-best method in a cross-dataset scenario.

Critical strengths

The advantage of this paper lies in its early introduction of LiDAR-image fusion models into test-time adaptation for 3D object detection, breaking through the previous limitation that 3D TTA was mostly confined to the LiDAR-only modality.

Critical weaknesses

Unfortunately, I find both stages of the paper quite incremental. More critically, in the first stage, when adapting the perception space, the authors also remove the ground-truth labels that fall outside the range. I believe this violates the original intent of model transfer in the field of 3D object detection, as it deliberately skips the most difficult part to solve.

Questions for authors

See the weakness.

Reviewer_UQKLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the test-time adaptation problem in multi-modal 3D object detection and proposes a two-stage multi-modal TTA method, MM-TTA. In the pre-training stage, we introduce a Geometric Consistency Alignment strategy that aligns the 3D perception spaces of source and target domain data, enabling the deployment of pre-trained fusion models to the target domain. In the adaptation stage, we propose a Consistency-Based Model Merging strategy, which measures model similarity through symbolic consistency scores and selects the most consistent models for merging; the merged model then guides the generation of subsequent pseudo-labels. Finally, MM-TTA is evaluated on multiple cross-domain datasets, achieving promising results across all benchmarks.

Critical strengths

Introducing multimodal information in the 3D detection TTA field is in line with practical application scenarios. The proposed Consistency-Based Model Merging module is different from previous model merging strategies. The paper presents a novel idea of selecting similar models for merging instead of models with significant differences. The experiments are comprehensive, not only conducting experiments on common cross-domain tasks, but also on some corruption scenarios. Finally, good performance has been achieved.

Critical weaknesses

1.The proposed MM-TTA framework employs a multimodal fusion detector, whereas the main competing methods, such as MOS and CodeMerge, are based on LiDAR-only detectors. Although Figure 7 compares different model merging strategies, it remains unclear whether MM-TTA would maintain its advantage if MOS and CodeMerge were fully implemented using the same multimodal architecture.

2.The proposed GCA strategy relies on information about the target domain to align the 3D perception spaces during source-domain pretraining. However, it is unclear whether this information consists only of predefined sensor configurations and calibration parameters or requires access to actual target-domain samples before deployment. If target test samples are accessed during pretraining, this may conflict with a strict TTA setting.

3.The authors argue that historical fusion models with low similarity are more likely to exhibit performance discrepancies rather than complementary knowledge, and that merging highly consistent models leads to better detection performance. The authors should quantitatively evaluate the detection performance of historical models with different similarity levels to determine whether low-similarity models primarily exhibit performance gaps rather than complementary strengths. Furthermore, directly comparing the performance of models obtained by merging high-similarity and low-similarity checkpoints would provide stronger evidence for the proposed merging strategy.

4.Section 3.1 states that the task-vector bank and similarity matrix are cleared periodically, after which the initialization, accumulation, and model merging stages are repeated. However, it is unclear whether the detector parameters are preserved across cycles or reset to the source-pretrained model. CMM selects historical models based on parameter similarity without explicitly considering scene distribution shifts. When the test stream transitions to a substantially different scene, highly similar historical models may no longer be suitable for the new environment, potentially generating unreliable pseudo-labels and causing performance degradation.

Questions for authors
  1. The appendix reports detection results for multiple categories, including Car, Pedestrian, and Cyclist, demonstrating that the proposed framework supports multi-class detection. However, the main experimental results primarily focus on the Car category. Given the potential benefits of multimodal fusion for detecting smaller objects, could the authors include representative multi-class results in the main paper to provide a more comprehensive evaluation?

  2. For the Waymo-to-nuScenes task, the authors adopt the nuScenes evaluation metrics, whereas previous cross-domain 3D object detection studies have commonly used KITTI-style evaluation metrics. Could the authors explain the rationale for this choice? Would it be possible to additionally report results using KITTI-style metrics to facilitate comparisons with previous methods?

  3. Appendix Table 9 shows that a projection dimension of 8192 achieves the highest AP3D (70.67), whereas 2048 achieves an AP3D of 70.09. However, the accompanying text states that 2048 achieves the highest APBEV and AP3D. Could the authors clarify this inconsistency and explain why 2048 is selected as the default projection dimension?

  4. Does the proposed framework save historical checkpoints during test-time adaptation and evaluate all of them after the adaptation process, ultimately selecting the highest result as the experimental outcome? If so, should the computational cost of evaluating these checkpoints also be included in the efficiency analysis?

  5. Section 3.1 describes a sequential test stream, while Appendix A.3.1 specifies a batch size of 4 during test-time adaptation. Could the authors clarify whether evaluation predictions are generated before or after the model is updated using the corresponding batch? In particular, does the method follow a test-then-train or train-then-test protocol? Additionally, are samples processed in chronological order, and are previously observed frames ever retrospectively re-evaluated?

  6. How is the NDS calculated in this paper? Does it follow the standard nuScenes evaluation protocol? According to the standard NDS formula, the reported NDS values in Appendix Table 3 appear inconsistent with the corresponding mAP and TP error metrics (mATE, mASE, mAOE, mAVE, and mAAE). Could the authors clarify this discrepancy and verify whether the reported results are correct?

Reviewer_rXe6OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies test-time adaptation (TTA) for 3D object detection with a LiDAR–camera fusion model (BEVFusion). Prior 3D TTA work adapts single-modality detectors: DPO and MOS adapt LiDAR-only detectors, and CodeMerge adapts both a LiDAR-only (SECOND) and a camera-only (SparseDrive) model, but a fusion model.. MM-TTA has two stages:

  • Geometric Consistency Alignment (GCA) during source pre-training. The target domain's 3D perception range is the region of LiDAR points that project into at least one target camera. Source point clouds and labels are cropped to that range, and source images that see none of the cropped points are dropped. The fusion model trained this way runs on the target domain without changing parameter or feature sizes.

  • Consistency-Based Model Merging (CMM) at test time. The model self-trains on ST3D-style pseudo-labels. Task vectors of the regression layer, after random projection, are stored for historical models and compared pairwise with a proposed sign consistency score. A maximum clique of mutually consistent models, thresholded at the q-th quantile of scores, is merged TIES-style with weights based on similarity. The merged model produces pseudo-labels. The bank is cleared every K + L steps.

Critical strengths
  • Under-explored problem. Deployed driving stacks are multimodal, but no prior 3D TTA method adapts a LiDAR–camera fusion detector.

  • CMM's motivation has some evidence. Fig. 10 shows qualitatively that early-step models are worse rather than complementary. In Fig. 7(a), replacing CMM's selection with MOS's synergy weights or CodeMerge's ridge leverage score degrades performance sharply.

  • Lightweight model selection. Similarities come from stored, projected task vectors, so historical models never need extra forward passes. Throughput is 1.34 samples/s higher than MOS

Critical weaknesses
  1. Table 1 compares MM-TTA's fusion model against LiDAR-only baselines.

Table 1's W →\rightarrow K and N →\rightarrow K baselines are taken from CodeMerge's Table 3, and those results are for a LiDAR-only detector. No Adapt and Oracle (line 337-340) are also defined as LiDAR-based models. Neither the table nor its caption says this is for lidar-only or fusion. MM-TTA, by contrast, claims to be TTA for a LiDAR–camera fusion model (BEVFusion). Even if the camera branch is seen only as a way to improve LiDAR features, MM-TTA still consumes images at test time: camera BEV features are fused with LiDAR features before the detection head (Fig. 2), and the view transform is adapted online (Fig. 6). MM-TTA uses more input than the baselines. This leaves two possibilities, and neither supports the paper's claims:

  • Read as a LiDAR-only benchmark, Table 1 says nothing about fusion TTA. If Table 1 is meant to compare TTA methods on LiDAR-only detection, MM-TTA's row doesn't belong in it. The paper then has no comparison against other TTA methods applied to a fusion model, which is the setting it claims to address.

  • Read as a fusion result, the improvement can't be attributed to CMM. The fusion detector has an extra modality and already beats the LiDAR model before any adaptation (Table 8: +1.66 / +2.56 (APBEVAP_{BEV}/ AP3DAP_{3D}) on W →\rightarrow K betwen No Adapt. and BEVFusion). Its final margins are modest: +3.78 / +6.88 AP3DAP_{3D} over CodeMerge on W →\rightarrow K / N →\rightarrow K, and only +0.0035 mAP over Tent on W →\rightarrow N. Without baselines run on the same fusion model, the margin can't be split between TTA and the capability of the fusion detector, so the claim that MM-TTA outperforms prior TTA methods is unsupportedunsupported.

  1. GCA is limited in novelty, ignores the main source of shift, and has weak supporting evidence.

It's standard range alignment. GCA crops the source point-cloud range and picks which source cameras to keep. Cross-domain 3D detection work does this routinely. UDGA [1] simply converted them to a unified detection range and coordinates. This doesn't justify presenting GCA as a main contribution.

It conflicts with the TTA setting. GCA needs the target's sensor layout and target samples before source pre-training (Section 3.2, Step 1), so a separate source model must be trained for each target domain. Standard TTA, including the MOS / DPO / CodeMerge protocol the paper follows, deploys one source model to unknown target streams.

Its measured benefit is small and inconsistently reported. Before TTA, the GCA-pre-trained BEVFusion beats LiDAR NoAdaptNo Adapt by only +1.66 / +2.56 (APBEVAP_{BEV}/ AP3DAP_{3D}) (Table 8). Yet, Section 4.3 claims GCA gives "nearly 16.23 higher AP3DAP_{3D} than other methods at the beginning of the TTA stage" (Fig. 7(c)). No ablation separates GCA from CMM.

  1. CMM is close to CodeMerge with the selection rule reversed, and it may reduce to merging recent checkpoints. Most of CMM is inherited. The checkpoint bank and K/L schedule follow MOS. The randomly projected representation follows CodeMerge's fingerprints. The merge rule (Eq. 2) has the same form as CodeMerge's Eq. 11. The actual changes are fingerprints computed from weight changes, selection by a consistency clique instead of by diversity, and similarity-based weights.

  2. There is a conceptual conflict (consistency versus diversity) between MM-TTA and previoud methods (MOS, CodeMerge). MOS weights models with the inverse Gram matrix, which deliberately favours diversediverse, less redundant models. CodeMerge selects by ridge leverage scores, which also favours informative, non−redundantnon-redundant models. MM-TTA's CMM keeps the models that are most consistentconsistent with each other and weights them by similarity row sums, which is the opposite principle.

[1] Chang, Gyusam, Jiwon Lee, Donghyun Kim, Jinkyu Kim, Dongwook Lee, Daehyun Ji, Sujin Jang, and Sangpil Kim. "Unified domain generalization and adaptation for multi-view 3d object detection." Advances in Neural Information Processing Systems 37 (2024): 58498-58524.

Questions for authors
  1. Run the baseline TTA methods (DPO, MOS, CodeMerge) on the same GCA-pre-trained BEVFusion. Report a fusion model for NoAdaptNo Adapt and a supervised fusion model for OracleOracle. To back the claim that CMM outperforms prior TTA methods, please evaluate CMM on LiDAR-only (SECOND model) for a direct comparison with the published numbers. Moreover, add an input-modality column to Table 1.

  2. Compare GCA with, or combine it with, camera/BEV domain-generalization methods. Please ablate GCA and CMM separately.

  3. Please explain why the favor of consistency in MM-TTA (fusion model) should beat the favor of diversity (lidar-only model).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.