acceptodds
Withdrawn from ICLR 2027

ReDi-TTA: Refinement-guided Distillation for Multi-Modal Test-Time Adaptation in 3D Semantic Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-modal test-time adaptation (MM-TTA) enables 3D semantic segmentation models to adapt to deployment shifts using unlabeled target observations without access to source data. Existing methods exploit predictive reliability to construct adaptation supervision. However, reducing fused predictions to hard pseudo-labels can still preserve an incorrect class decision, even when the full distributions of base models and teachers retain complementary support for alternative classes. To address this shortcoming, we propose Refinement-Guided Distillation for Test-Time Adaptation (ReDi-TTA), which exploits class-wise evidence through probability refinement and role-decoupled student learning while retaining the base adaptation procedure. First, Cross-modal Class Probability Refinement (CCPR) reweights each base prediction using class-aligned probabilities from both exponential moving average teachers, producing branch-specific soft targets that preserve information beyond the highest-scoring class. Second, Role-Decoupled Bimodal Classification (RBC) introduces dedicated student heads operating on aligned image and point-cloud features. These heads are excluded from target construction and EMA teacher updates, preventing student optimization from altering the guidance generation pathway. Third, Proximal Soft-Target Distillation (PSD) addresses the evolving feature representations during base adaptation by fitting pre-update soft targets on detached features recomputed after the base update, while regularizing student-head changes with a proximal constraint. Together, these designs transform refined class-wise evidence into a persistent student learning signal, allowing future predictions to benefit from refinement rather than replacing current-batch outputs. Extensive experiments demonstrate that ReDi-TTA achieves state-of-the-art performance across multiple MM-TTA benchmarks.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Multi-modal learning, Test-time adaptation, 3D semantic segmentation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YfsUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies online adaptation of image–point-cloud semantic segmentation models through a separate student learning pathway. ReDi-TTA constructs branch-specific soft targets by multiplying base and EMA-teacher probabilities, trains dedicated bimodal linear heads, and fits them on detached post-update features with a proximal penalty. The original base adaptation remains intact, and predictions are recorded before each batch update. Experiments on three domain transfers and three adaptation frameworks report improvements over native baselines and direct probability refinement.

Critical strengths
  1. Separating student fitting from target construction and teacher updates provides a clear way to add refinement-guided learning without changing the base adaptation objective. And it provides a detailed explanation of which parameters will receive gradients, as well as how the learned heads affect subsequent predictions.
  2. The evaluation covers geographic, cross-sensor, and synthetic-to-real shifts, and applies the same student design to PLATO-TTA, Latte, and MM-TTA. Table 1 reports positive fused-score gains in all nine framework–transfer combinations, suggesting useful applicability beyond one adaptation framework.
Critical weaknesses
  1. The headline results are internally inconsistent: Table 1 exceeds the student xM in Tables 4–6 by exactly 0.75 points on every transfer, and Section 4.3 contradicts Tables 8–9 about feature refresh and fusion. Final predictions, confusion matrices, and run identifiers are needed to establish which results support the claims.
  2. The student contribution is not cleanly isolated. Tables 1 and 3 imply only 0.46–0.72-point PLATO gains over direct refinement, but Equation (15) shows that multiplying branch targets counts teacher evidence twice, unlike the direct reference. Comparisons with matched teacher multiplicity, fusion, and a single bimodal head would clarify the benefit of persistent student learning.
  3. PSD is motivated by encoder drift, but Table 8 tests feature refresh only on U→S, with a 0.424-point margin and no controlled regularization-strength sweep or drift measurements. Paired controls across transfers would better establish when post-update fitting and the proximal penalty improve stability.
Questions for authors
  1. Which final runs support the xM scores in Table 1 and the component conclusions in Section 4.3? Please reconcile the 0.75-point discrepancies with Tables 4–6 and the conflicting PSD and fusion results using identified final prediction records and confusion matrices.
  2. How much benefit remains from persistent student learning when direct refinement uses the same teacher multiplicity and fusion rule? Could the authors provide this comparison across transfers and include a single bimodal student head, since Equations (6)–(7) make the fused predictor equivalent to a softmax over summed linear logits?
  3. Does fitting post-update features improve stability across all three transfers, particularly when encoder drift is substantial? Could the author compare the fitting effects before and after the update, adjust the proximal strength, and conduct a correlation analysis between the results and the measured feature drift?
Reviewer_rsRSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ReDi-TTA, a Multi-Modal Test-Time Adaptation (MM-TTA) method for 3D semantic segmentation. Specifically, it points out the existing limitation of using hard pseudo-labels, which motivates it to introduce soft targets via Cross-modal Class Probability Refinement (CCPR). To achieve better cross-modal fusion, it introduces Role-Decoupled Bimodal Classification (RBC), which inserts a dedicated linear student classifier to output predictions for evaluation. These outputs are regularized by the Proximal Soft-Target Distillation (PSD), which introduces an additional forward pass to retrieve the latest features after the immediate adaptation step so that the student classifiers can be better maintained.

Critical strengths

The proposed method is simple but effective, especially for CCPR, solely by which it can achieve more than 3.0% improvement as in Table 2 with trivial additional cost introduced. The empirical results provided are comprehensive (in both the main manuscript and the appendices), covering various evaluation perspectives and recent competitive baselines.

Critical weaknesses
  • Limited add-ons from RBC and PSD. The claimed contributions of this manuscript include CCPR, RBC, and PSD, but it seems that direct pseudo-label refinement from CCPR is already empirically significant, while RBC and PSD contribute much less. In Table 3, using CCPR with PLATO achieves a noticeable improvement of over 3.0%, but introducing RBC and PSD further only leads to a minor increase of less than 0.5%. Therefore, empirically speaking, the specific design of additional student heads and distillation seems to make limited contributions.
  • Unclear motivations and design rationale. Specifically, the motivations of RBC and PSD are from the two challenges mentioned in lines 83-89, but to me it seems more like a personal choice rather than a universal understanding supported by existing literature or empirical observations. For the first challenge, the authors claim that classifiers used for target generation should not be updated with these targets, but there aren’t specific ablation runs supporting this claim or quantifying the actual impact of this act. For the second challenge, it somehow contradicts the individual control ablation in Table 8, where disabling the pre-update features in PSD leads to a minor drop of 0.4%. As admitted in C.1, this gap is too trivial to justify whether the challenges mentioned are sufficiently critical to motivate the design choices in RBC and PSD.
  • Inconsistent numbers in main comparison. While it is thorough to present the class-wise performance in appendices, the overall performance on xM is not consistent across Table 1 and Tables 4-6. Specifically, the score of "xM" in Table 1 is exactly 0.75 higher than those corresponding in Tables 4- 6 (53.65/56.49/48.03 vs. 52.90/55.74/47.28), but all other figures from "2D" and "3D" are identical. A justification should be presented to confirm which numbers are the correct results.
  • Confusing roles and presentation. Some major points are listed as follows:
  1. "Base", "teacher", and "student" are used in a way that conflicts with common usage in EMA. Till the end, it is still not perfectly clear to me which module is updatable and which is not.
  2. Figure 2 does not match the text description. Specifically, H^tm\hat{H}_t^m is in the RCD box but defined in PSD in the text. ZtmZ_t^m is undefined in the text (which I think should be StmS_t^m instead?).
  3. In line 87, "student classifiers" is still undefined, and therefore the challenge does not sound straightforward at first glance. A clearer phrasing could help.
  • Missing AI usage statement. The manuscript does not include an AI usage statement, which seems to be mandatory according to ICLR 2027 submission policy. Although I have not factored this into my assessment, I flag this concern here and leave it to be handled by ACs.
Questions for authors
  • For some key ablation runs mentioned in the weakness section (e.g., empirical justification for both challenges and the contributions of specific designs in RBC and PSD), can the authors supplement additional results on all three benchmarks rather than solely on U-S?
  • Which xM numbers are correct for the complete model: Table 1 or Tables 4–6?
  • What happens if we simply use QQ from CCPR as a soft pseudo-target with existing MM-TTA methods?
  • In run No. 2 of Table 2, does the "direct refinement" mean using the CCPR soft target to update the model for PLATO-TTA? Does the "Refinement" in the output columns mean that this run directly uses CCPR's output for evaluation?
  • What is the computational overhead and efficiency in comparison with existing methods?
Reviewer_E5YWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ReDi-TTA, a framework for multi-modal test-time adaptation in 3D semantic segmentation. It refines class-wise probability distributions from base models and EMA teachers and distills them into separate bimodal student classifiers while preserving the original adaptation procedure. Experiments on three domain adaptation benchmarks demonstrate improvements over existing methods.

Critical strengths

Consistent Improvements Across Benchmarks: The proposed method consistently improves upon the corresponding baseline methods across all three domain adaptation benchmarks.

Compatibility with Different TTA Methods: The approach is evaluated using PLATO-TTA, Latte, and MM-TTA, demonstrating its integration into diverse adaptation pipelines.

Critical weaknesses

Limited Novelty: The proposed method mainly combines established techniques, including probability multiplication, knowledge distillation, EMA teachers, and linear classifier adaptation. The authors could more clearly distinguish their contributions from those of existing methods.

Limited Benefits of Some Components: As shown in Table 2, direct CCPR refinement alone achieves approximately 88% of the full method's improvement over PLATO-TTA on the USA→Singapore route. This raises questions about whether the additional complexity introduced by RBC and PSD is justified by the relatively small additional gain.

Clarity: The paper does not clearly specify which underlying EMA update strategy is preserved. Furthermore, although the individual components are relatively straightforward, the method is presented using unnecessarily complicated terminology, making it difficult to follow. A clearer, step-by-step explanation connecting the motivation, challenges, and proposed components would significantly improve readability. “Role-Decoupled Bimodal Classification” is a heavy name for what is essentially a linear classifier head that takes concatenated features from both modalities as input and is excluded from EMA updates. The motivation for the decoupling is clear and valid, but the naming adds cognitive overhead without adding precision.

Reproducibility and Statistical Reliability: Experiments are conducted with a single random seed. In particular, the improvements over PLATO-TTA on A2D2→SemanticKITTI (+1.06 mIoU) and SYNTHIA→SemanticKITTI (+1.11 mIoU) are relatively small. Reporting results across multiple seeds, together with standard deviations, would help establish the reliability of these improvements.

Missing Efficiency Analysis: The paper does not provide a quantitative analysis of the computational overhead introduced by the additional student classifiers and feature recomputation. Since the method targets online test-time adaptation, runtime and memory consumption are important for assessing its practical applicability.

Inconsistent PSD Ablation Results: Section 4.3 reports that using pre-update features achieves 53.23 mIoU, outperforming the complete configuration's 52.91 mIoU. However, Table 8 in Appendix C.1 reports 53.65 mIoU for the complete model and 53.226 mIoU for pre-update features, indicating the opposite conclusion. The authors should clarify this discrepancy, as it directly affects the interpretation of the PSD ablation results.

Questions for authors

Contribution of RBC and PSD: Given that direct CCPR refinement already achieves approximately 88% of the full method's improvement over PLATO-TTA on USA→Singapore (Table 2), could the authors further justify the additional complexity of RBC and PSD? How do these components compare with simpler alternatives across all three benchmarks?

Computational Efficiency: What is the additional runtime and GPU memory overhead introduced by ReDi-TTA compared with the underlying TTA frameworks? In particular, how much computational cost is introduced by L-BFGS optimization and post-update feature recomputation?

Statistical Reliability: Could the authors provide results across multiple random seeds, including standard deviations, particularly for A2D2→SemanticKITTI and SYNTHIA→SemanticKITTI, where the improvements over PLATO-TTA are relatively small?

PSD Ablation Inconsistency: In Section 4.3, the authors state that using pre-update features achieves 53.23 mIoU, outperforming the complete configuration at 52.91 mIoU. However, Table 8 in Appendix C.1 reports 53.65 mIoU for the complete model and 53.226 mIoU for pre-update features. Could the authors clarify the origin of the 52.91 result and which conclusion is correct?

EMA Teacher Updates: Could the authors clarify which EMA teacher update strategy is inherited from the underlying TTA framework, how the teachers are initialized, and whether the same update mechanism is used when integrating ReDi-TTA with PLATO-TTA, Latte, and MM-TTA?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.