acceptodds
Under review as a conference paper at ICLR 2027

Decoupled Streams for Open-Set Test-Time Adaptation of Vision-Language Models

Abstract

Open-set test-time adaptation requires a model to improve classification on in-distribution (ID) samples while reliably rejecting unknown out-of-distribution (OOD) inputs. Existing approaches for vision-language models optimize these two objectives through the same trainable parameters, implicitly assuming that their updates remain compatible throughout adaptation. We show that this compatibility progressively weakens during adaptation: classification and ID/OOD separation gradients become less aligned, while recent gradient-rebalancing and modular adapter-routing strategies do not consistently improve both objectives jointly. This motivates a simple, yet effective alternative: rather than reconciling the objectives within a shared adaptation space, we separate their optimization. We introduce DSTA (Decoupled Streams for Open-Set Test-Time Adaptation), which equips vision-language models with two independent sets of low-rank adapters for classification and ID/OOD separation over a shared frozen backbone, while retaining their interaction through OOD-guided sample selection. In this decoupled regime, DSTA stabilizes the separation stream with a novel margin-based objective that prevents excessive ID/OOD gap increase, and proposes automatic OOD logit-scale calibration to avoid compressed confidence scores. Across clean and corrupted open-set benchmarks, DSTA improves the joint behavior of ID classification and OOD detection, with particularly strong gains in unknown-sample rejection under distribution shift. These benefits remain evident across backbone sizes, supporting parameter-space decoupling as an effective design principle for open-set test-time adaptation. Code will be released upon acceptance.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
test-time adaptation, open-set test-time adaptation, vision-language models, CLIP, out-of-distribution detection, parameter-efficient adaptation, LoRA, gradient interference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xPuxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the problem of open-set test-time adapatation of vision-language models. Through gradient cosine similarity analysis, the author discovered that the gradient directions for the two objectives gradually cease to be highly aligned during the adaptation process. Consequently, the author proposes DSTA, which employs two separate sets of low-rank Adapters to decouple the optimization parameters for the two objectives.

Critical strengths
  1. This paper is well organized and clearly written.

  2. The technical details are also easy to follow.

  3. The experiments are extensive and helpful to validate the effectiveness of the model.

Critical weaknesses
  1. As shown in Table 2, the improvements against baselines seem to be mirrror.

  2. The pseudo-labeling for OOD samples may be subject to confirmation bias. Please further investigate the influence.

  3. The stability argument for the margin-based objective is not sufficiently substantiated.

  4. The ratio between ID and OOD samples is fixed, which is overly idealistic, as the actual ratio is more complex.

Questions for authors

Please see the above weaknesses.

6 comments
AuthorsOpenReview ↗

Response (part 1)

We appreciate the reviewer's positive comments on our work, particularly for highlighting the clarity of the presentation, the accessibility of the technical details, and the experimental evaluation supporting the effectiveness of DSTA. We address the reviewer's concerns point by point below.

1. Performance Improvements over Existing Baselines.

The results in Tab. 2 demonstrate substantial improvements over existing baselines.

On clean benchmarks, DSTA achieves the best aggregate results across all four metrics, reducing FPR95 from 35.3% to 29.3% and improving H-score from 68.6% to 72.0% compared with UniEnt.

Contrary to the reviewer's observation, the gains are particularly substantial on individual benchmarks: compared with CLIPTTA, DSTA reduces FPR95 from 7.6% to 0.1% on CIFAR-100 and from 9.7% to 0.8% on ImageNet, while also improving classification accuracy.

Under corruptions, DSTA further improves aggregate FPR95 by 5.1 points and H-score by 4.6 points over CLIPTTA.

AuthorsOpenReview ↗

Response (part 2)

2. Confirmation Bias in OOD Pseudo-Labeling.

Potential confirmation bias is a general challenge in learning from unlabeled data, and not a limitation specific to DSTA. Without ground-truth labels, existing methods necessarily rely on estimated ID/OOD membership or confidence-based sample selection, and therefore remain susceptible to confirmation bias arising from incorrect assignments. This includes the existing open-set TTA methods evaluated as baselines in our work: OSTTA (Lee et al., 2023), SoTTA (Gong et al., 2023), STAMP (Yu et al., 2024), UniEnt (Gao et al., 2024), and CLIPTTA (Lafon et al., 2025).

As in prior open-set TTA methods, addressing confirmation bias is beyond DSTA's methodological focus. Addressing this broader challenge constitutes a separate research direction and falls outside the scope of the present work.

AuthorsOpenReview ↗

Response (part 3)

3. Theoretical and Empirical Evidence for the Stability of the Margin-Based Objective.

We would like to clarify that both analytical and empirical evidence supporting the stability of the margin-based objective are already in the submitted paper, specifically in Supplementary Sec. A.1 (Eqs. 4–6, Figs. 3–4) and Tab. 3 of the main paper.

Analytically, CLIPTTA's original OCE objective has a gradient magnitude (2∣Δ∣2|\Delta|), which grows with the separation gap and continuously encourages further separation. In contrast, our margin objective has a bounded gradient magnitude (≤1)(\leq 1) that vanishes once (Δ≥m\Delta \geq m), preventing excessive optimization beyond the desired separation margin (Fig. 3).

Empirically, Fig. 4 of the main paper shows that OCE leads to an unstable separation gap and AUC degradation to 55.7%, whereas the margin objective maintains stable separation and AUC of 99.8%. Tab. 3 further confirms this improvement, with AUC increasing from 55.7% to 99.5% and FPR95 decreasing from 94.2% to 1.4% under identical decoupled LN adaptation.

Reviewer_aSmAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a method for open-set test-time adaptation (OS-TTA) of vision-language models (VLMs), such as CLIP. Its central motivation is that the gradients of the ID classification and OOD detection objectives become less aligned during adaptation, potentially limiting performance when both objectives update the same parameters. The proposed method therefore uses two independent sets of LoRA adapters, one for each objective. However, optimizing the separation stream independently can destabilize adaptation because the original objective encourages unrestricted growth of the gap between mean ID and OOD scores. To address this issue, the method introduces a margin-based smooth L1 loss that stops encouraging further separation once a target gap is reached. Combined with automatic OOD logit-scale calibration, this design improves classification accuracy and OOD detection over methods such as CLIPTTA on clean semantic OOD benchmarks, while yielding more mixed results under corruption.

Critical strengths
  • The proposed separation of classification adaptation and OOD detection objectives into two independent streams is interesting.

  • The method outperforms CLIPTTA in both ID classification accuracy and OOD detection, particularly on clean semantic OOD benchmarks.

  • The main paper and supplementary material provide extensive ablations of objective decoupling, the margin loss, logit-scale calibration, and LoRA configuration. These experiments generally support the design choices and clarify how the components contribute to performance.

Critical weaknesses
  • The method builds substantially on CLIPTTA, retaining its contrastive adaptation loss, OOD scoring and thresholding framework, and class-wise confident memory. The main innovations are separate LoRA adapters for classification and OOD separation, together with a margin-based loss that stabilizes the separation stream. Although these changes yield meaningful empirical gains, the contribution feels more like an incremental architectural and optimization refinement than a fundamentally new approach.

  • The gradient analysis offers some motivation for decoupling the two streams, but does not convincingly demonstrate harmful interference: cosine similarity remains positive for most optimization steps, indicating reduced alignment rather than direct gradient conflict.

  • The experiments lack key baselines needed to isolate the sources of improvement. In particular, automatic OOD logit-scale calibration should also be evaluated with CLIPTTA to determine how much of the gain it explains. Table 4 should additionally include a shared-stream baseline in which the classification objective updates attention adapters and the separation objective updates MLP adapters, to distinguish the benefit of adapter allocation from that of using independent streams.

Questions for authors
  • Beyond the architectural and optimization refinements to CLIPTTA, what generalizable insight do you consider the paper’s principal contribution, and which experiments most directly support it?

  • Since gradient cosine similarity remains positive for most optimization steps, what evidence shows that reduced alignment causes harmful interference rather than simply weaker mutual support? Can you connect the observed gradient behavior to degradation in classification or OOD detection?

  • How does CLIPTTA perform when equipped with the same automatic OOD logit-scale calibration? Would its inclusion substantially narrow the gap to DSTA?

  • Can you extend Table 4 with a single-stream baseline in which the classification objective updates attention adapters and the separation objective updates MLP adapters, using the same margin loss and calibration as DSTA? This would help distinguish the benefit of objective-specific adapter allocation from that of maintaining independent streams.

6 comments
AuthorsOpenReview ↗

Response (part 1)

We appreciate the reviewer's acknowledgment of the positive aspects of our work, particularly highlighting the proposed objective decoupling, the improvements over CLIPTTA in both classification and OOD detection, and the supporting ablation studies. We appreciate the constructive feedback and address the reviewer's concerns point by point below.

1. Novelty and Contributions Beyond CLIPTTA

We respectfully disagree with the characterization of DSTA as an incremental refinement. This assessment focuses on the individual methodological components without fully accounting for the finding that motivates their combination. Our work is the first to identify that classification and ID/OOD separation progressively lose mutual gradient support during shared-parameter adaptation (Q1a). This finding motivates DSTA, which addresses the identified limitation by independently optimizing the two objectives through decoupled adaptation streams. Importantly, addressing a previously unrecognized limitation does not necessarily require introducing more complex losses or additional adaptation mechanisms. Instead, our contribution lies in identifying the limitation and designing a targeted solution using established components. Our experiments (Q1b-Q3b) further further support the effectiveness of this design and demonstrate the benefits of parameter decoupling.

Q1a. Generalizable Insight and Principal Contribution of DSTA.

Our first contribution is a systematic analysis of the interaction between classification and ID/OOD separation during shared-parameter adaptation. We show that the two objectives progressively lose mutual gradient support, a trend that persists even with gradient-rebalancing methods such as PCGrad, CAGrad, and FairGrad (Fig. 1 of the main paper). Although their gradients remain predominantly positively aligned, this declining alignment reveals diminishing mutual support. Our central insight is that independently optimizing these objectives can be more effective than balancing them within a shared parameter space. Building on this insight, our second contribution is DSTA, which decouples the trainable parameters of classification and ID/OOD separation.

Q1b. Experimental Evidence Supporting the Principal Insight

Experimental support is already provided by Tab. 2 of the main paper, where DSTA outperforms existing non-decoupled approaches across all aggregate metrics on clean benchmarks and achieves the best aggregate FPR95 and H-score under corruptions. Further evidence comes from the additional experiment reported in Q3b, which compares DSTA against a single-stream baseline with no decoupling. DSTA consistently outperforms this baseline, hence demonstrating that maintaining independent streams provides benefits beyond all other methodological components.

AuthorsOpenReview ↗

Response (part 2)

2. Interpretation and Implications of Gradient Alignment

Q2a: Distinguishing diminishing mutual support from harmful gradient interference

Our gradient cosine similarity analysis (Fig. 1 of the main paper) does not equate decreasing positive gradient alignment with harmful gradient conflict. The practical concern is that, as mutual support diminishes, a shared update may become increasingly difficult to balance across both objectives. Hence, our argument concerns the increasing difficulty of jointly optimizing the two objectives, which motivates DSTA's approach of optimizing them independently.

Q2b: Relationship between gradient behavior and classification/OOD detection performance

It may seem natural to measure the correlation between batch-level ACC/AUC and gradient cosine similarity. However, interpreting such a correlation as evidence of a direct relationship would be methodologically incorrect, since gradient alignment describes the relationship between the objectives at a given optimization step, whereas ACC and AUC reflect the accumulated effects of previous adaptation updates. Consequently, performance may continue improving even as gradient alignment progressively decreases, making a direct correlation misleading.

AuthorsOpenReview ↗

Response (part 3)

3. Additional Baselines to Isolate Methodological Contributions

Q3a: Equipping CLIPTTA with DSTA's automatic OOD logit-scale calibration.

We run the suggested experiment in order to isolate the contribution of calibration on ImageNet/Places365. We equipped CLIPTTA with the same automatic OOD logit-scale calibration used in DSTA, increasing its OOD logit scale from the original fixed value of 1.0 to the automatically calibrated value of 14.9.

\textbf{Method} & \textbf{ACC}\uparrow & \textbf{AUC}\uparrow & \textbf{FPR95}\downarrow \\\\ \hline \text{CLIPTTA} & 67.9 & 97.1 & 11.5 \\\\ \text{CLIPTTA + Calibration} & 57.9 & 99.4 & 1.8 \\\\ \textbf{DSTA (ours)} & \mathbf{69.2} & \mathbf{99.8} & \mathbf{0.8} \end{array}$$ Calibration narrows the OOD detection gap for CLIPTTA, improving AUC and FPR95, but at the cost of a sharp drop in classification accuracy. This result is consistent with a trade-off under shared-parameter optimization: strengthening the separation objective improves OOD detection but compromises classification. We have included this additional result in Tab. 3 of the revised paper. ### Q3b: Isolating the benefits of independent streams from adapter allocation. We implemented the single-stream baseline suggested by the reviewer (named ``Single-stream (Attn/MLP)''): it assigns classification updates to attention adapters and separation updates to MLP adapters, while retaining DSTA's margin loss and automatic OOD logit-scale calibration. $$\begin{array}{l|ccc} \textbf{Method} & \textbf{ACC}\uparrow & \textbf{AUC}\uparrow & \textbf{FPR95}\downarrow \\\\ \hline \text{CLIPTTA} & 67.6 & 97.7 & 9.7 \\\\ \text{Single-stream (Attn/MLP)} & 68.0 & 98.9 & 4.0 \\\\ \textbf{DSTA (ours)} & \mathbf{69.2} & \mathbf{99.8} & \mathbf{0.8} \end{array}$$ The single-stream baseline underperforms DSTA in both classification and ID/OOD separation, indicating that objective-specific adapter allocation alone is insufficient to reproduce DSTA's gains. We included this baseline in the revised Tab. 4 (see ``ATTN/MLP Separate Updates'').
Reviewer_73RKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies open-set test-time adaptation of vision-language models. DSTA uses two separate sets of adapters over a shared frozen backbone: one for known-class classification and one for ID/OOD separation. It also adds a margin loss and logit-scale calibration

Critical strengths

DSTA performs well on several open-set benchmarks. On clean ImageNet, it improves accuracy from 67.6% to 69.2%, AUC from 97.7% to 99.8%, and FPR95 from 9.7% to 0.8% relative to CLIPTTA. The paper also provides more than a single overall score: it includes component and parameter-budget controls, results with a second backbone, and runtime measurements.

Critical weaknesses
  1. The contribution of decoupling to the full method is not sufficiently isolated. Table 3 provides some evidence for decoupling when both shared and separate LayerNorm updates use the margin loss. However, separating the updates without that loss drops ImageNet AUC from 97.7% to 55.7% and raises FPR95 from 9.7% to 94.2%. The final DSTA model also changes adapter placement and adds logit calibration. The capacity-matched comparisons in Table 4 do not hold all these choices fixed while changing only whether the adapters are shared. The results therefore support the complete design more clearly than they support decoupling as its main source of improvement.
  2. The test batches always contain equal numbers of ID and OOD samples. Each batch has 128 of each. This follows the evaluation protocol used for the baselines, so the reported comparison is fair within that protocol. However, DSTA uses batch scores to set its separation threshold, and its behavior when unknown samples are rare, absent, or become more frequent is not reported.
  3. ImageNet-C is an important exception to the overall trend. DSTA is worse than CLIPTTA on all four reported metrics in this setting: accuracy is 36.3% versus 39.0%, AUC is 82.9% versus 89.0%, FPR95 is 55.0% versus 43.2%, and H-score is 48.5% versus 55.1%. The aggregate improvement should therefore be interpreted alongside this failure case.
Questions for authors

Regarding weakness 1: What explains the sharp drop when LayerNorm updates are decoupled without the margin loss? Have the authors tested a shared-adapter version with the same margin loss, logit calibration, LoRA placement, and a comparable parameter budget as the final DSTA model? If available, this result would clarify the independent contribution of decoupling. Regarding weakness 2: How does the batch-based threshold behave when the ID/OOD ratio changes, particularly in batches with few or no OOD samples? Have the authors measured performance under such stream compositions? Regarding weakness 3: Which corruption types account for the regression on ImageNet-C? Are per-corruption results or an analysis of this failure available?

6 comments
AuthorsOpenReview ↗

Response (part 1)

We thank the reviewer for recognizing the strengths of our work, particularly for recognizing DSTA's performance across several open-set benchmarks and the breadth of our experimental evaluation and analyses. We appreciate the constructive feedback and address the concerns below with additional experimental evidence and analysis.

1. The contribution of Decoupling

Q1a. Performance drop when LayerNorm updates are decoupled without the margin loss.

We analyzed this behavior both theoretically and empirically in the initial version of the submitted paper, in Sec. A.1 (Eqs. 4–6, Figs. 3–4) of the supplementary material and Tab. 3 of the main paper.

The collapse arises from CLIPTTA's original OCE objective, (LOCE=−Δ2)(L_{\mathrm{OCE}}=-\Delta^2), whose gradient magnitude (2∣Δ∣)(2|\Delta|) increases with the ID/OOD separation gap. When the objectives are decoupled, classification updates no longer counterbalance separation updates on the same parameters. Consequently, OCE continuously amplifies the estimated separation gap, eventually destabilizing OOD detection. As shown in Tab. 3, decoupling LN updates while retaining OCE reduces AUC from 97.7% to 55.7%, as the separation gap eventually enters an unstable regime. Replacing OCE with the margin loss under the same decoupled LN configuration prevents excessive gap growth, restoring AUC to 99.5.

In addition, Fig. 4 of the supplementary material further shows that the uncontrolled growth of the separation gap under OCE and the corresponding AUC deterioration, compared with stable separation under the margin objective. In particular, on ImageNet/Places, the separation gap under OCE exceeds Δ≈3.7\Delta \approx 3.7 before becoming unstable, with AUC eventually dropping to 55.7%. In contrast, the margin objective (m=3m=3) maintains a controlled separation gap and an AUC of 99.8%.

Q1b. Independent contribution of Decoupling.

We agree about the need to isolate the contribution of parameter decoupling while testing a shared-adapter approach. We conducted an additional experiment on ImageNet/Places (named “Shared adapters”), using shared adapters while keeping the same margin loss, logit calibration, and LoRA placement as DSTA, and maintaining a comparable trainable parameter budget ($\sim$1.3M parameters).

\hline \text{Method} & \text{ACC}\uparrow & \text{AUC}\uparrow & \text{FPR95}\downarrow \\\\ \hline \text{CLIPTTA} & 67.6 & 97.7 & 9.7 \\\\ \text{Shared adapters} & 67.6 & 99.5 & 1.7 \\\\ \text{DSTA (ours)} & \mathbf{69.2} & \mathbf{99.8} & \mathbf{0.8} \\\\ \hline \end{array}$$ The shared-adapter approach performs better than CLIPTTA and worse that our DSTA across all three metrics. With all other methodological components matched and a comparable parameter budget, DSTA improves ACC by 1.6 points, AUC by 0.3 points, and reduces FPR95 by 0.9 points relative to shared adapters. This is an even clearer signal that supports the idea of decoupling proposed in our approach. We have included this additional comparison in Tab. 4 of the revised paper.
AuthorsOpenReview ↗

Response (part 2)

2. Robustness to Varying ID/OOD Batch Compositions

DSTA, like other distribution-based ID/OOD partitioning methods within the same class of methods as ours, (e.g., CLIPTTA (Lafon et al., 2025) and UniEnt (Gao et al., 2024)), relies on confidence-score distributions of ID/OOD groups. All these methods cannot be characterized by zero or a extremely low number of data points. Consequently, the reliance on sufficient sample sizes is a limitation of all batch-based, distribution-based ID/OOD partitioning techniques, and resolving it remains an important open problem for the field. Nevertheless, to address the underlying concern regarding robustness, we conducted additional experiments using imbalanced ID/OOD compositions (25/75$%$ and 75/25$%$) on image classification (CIFAR-100/SVHN) and action recognition in videos (HAC-H/EK, HAC-A/EK, and HAC-C/EK). These experiments evaluate our method's limits while remaining within the operating bounds of distribution-based partitioning.

Q2a. Behavior of the Batch-Based Threshold Under Varying ID/OOD Ratios.

We report below the mean and standard deviation of Otsu thresholds estimated by DSTA on CIFAR-100/SVHN and HAC/EK under different ID/OOD ratios. Note: 50/50$%$ is the original setup of previous work and of our paper.

\hline \textbf{Domain} & \textbf{ID/OOD} & \textbf{DSTA} \\\\ \hline \text{CIFAR-100/SVHN} & 25/75 & -3.143 \pm 0.374 \\\\ & 50/50 & -3.071 \pm 0.319 \\\\ & 75/25 & -2.935 \pm 0.321 \\\\ \hline \text{HAC-H/EK} & 25/75 & -0.993 \pm 0.159 \\\\ & 50/50 & -0.899 \pm 0.096 \\\\ & 75/25 & -0.843 \pm 0.074 \\\\ \hline \text{HAC-A/EK} & 25/75 & -0.924 \pm 0.167 \\\\ & 50/50 & -0.796 \pm 0.089 \\\\ & 75/25 & -0.721 \pm 0.059 \\\\ \hline \text{HAC-C/EK} & 25/75 & -0.847 \pm 0.128 \\\\ & 50/50 & -0.732 \pm 0.056 \\\\ & 75/25 & -0.759 \pm 0.036 \\\\ \hline \end{array}$$ Overall, the estimated threshold tends to increase with the proportion of ID samples, as their generally higher confidence scores become more prevalent in the batch, shifting the Otsu partition toward higher values. This indicates that DSTA responds to changes in batch composition without requiring knowledge of the true ID/OOD ratio. We have included this analysis in Sec. A.14 of the revised supplementary material. ### Q2b. Performance under varying ID/OOD batch compositions. We report below the performance of DSTA on CIFAR-100/SVHN and HAC/EK under different ID/OOD ratios. $$\begin{array}{l|c|c|c|c} \hline \textbf{Domain} & \textbf{ID/OOD} & \textbf{ACC}\uparrow & \textbf{AUC}\uparrow & \textbf{FPR95}\downarrow \\\\ \hline \text{CIFAR-100/SVHN} & 25/75 & 74.2 & 100.0 & 0.1 \\\\ & 50/50 & 75.5 & 100.0 & 0.1 \\\\ & 75/25 & 75.5 & 99.9 & 0.1 \\\\ \hline \text{HAC-H/EK} & 25/75 & 84.2 & 77.3 & 39.9 \\\\ & 50/50 & 84.6 & 81.6 & 53.2 \\\\ & 75/25 & 84.9 & 80.2 & 72.1 \\\\ \hline \text{HAC-A/EK} & 25/75 & 89.5 & 80.5 & 41.9 \\\\ & 50/50 & 88.2 & 82.0 & 60.2 \\\\ & 75/25 & 88.4 & 81.4 & 71.1 \\\\ \hline \text{HAC-C/EK} & 25/75 & 69.2 & 69.0 & 58.3 \\\\ & 50/50 & 66.5 & 57.7 & 89.6 \\\\ & 75/25 & 67.1 & 53.8 & 96.6 \\\\ \hline \end{array}$$ On CIFAR-100/SVHN, OOD detection remains nearly unchanged. On HAC-H/EK and HAC-A/EK, AUC remains relatively consistent across ratios, while HAC-C/EK exhibits greater sensitivity to batch composition. Overall, classification remains stable across ratios, while OOD detection is more sensitive when less OOD samples are present in the batch. We have included these results in Sec. A.14 of the revised supplementary material.
AuthorsOpenReview ↗

Response (part 3)

3. Analysis of ImageNet-C Performance

Q3a. Corruption types accountable for lower performance on ImageNet-C.

To answer this question, we performed a per-corruption analysis of DSTA on ImageNet-C/Places.

\hline \text{Corruption} & \text{ACC}\uparrow & \text{AUC}\uparrow & \text{FPR95}\downarrow \\\\ \hline \text{Gaussian noise} & 23.6 & 60.2 & 92.9 \\\\ \text{Impulse noise} & 23.8 & 58.6 & 91.7 \\\\ \text{Shot noise} & 26.3 & 69.3 & 88.1 \\\\ \hline \text{Brightness} & 58.1 & 95.1 & 23.6 \\\\ \text{Contrast} & 31.6 & 81.0 & 72.3 \\\\ \text{Defocus blur} & 30.8 & 87.9 & 37.5 \\\\ \text{Elastic transform} & 35.8 & 77.5 & 75.5 \\\\ \text{Fog} & 49.2 & 99.4 & 2.4 \\\\ \text{Frost} & 38.8 & 77.5 & 78.3 \\\\ \text{Glass blur} & 31.6 & 96.7 & 14.2 \\\\ \text{JPEG compression} & 39.0 & 78.7 & 84.0 \\\\ \text{Motion blur} & 37.0 & 99.3 & 3.9 \\\\ \text{Pixelate} & 42.8 & 89.9 & 37.4 \\\\ \text{Snow} & 40.7 & 83.6 & 67.3 \\\\ \text{Zoom blur} & 35.3 & 89.4 & 55.1 \\\\ \hline \text{Gaussian-Impulse-Shot noise average} & 24.5 & 62.7 & 90.9 \\\\ \text{Other 12 average} & 39.2 & 88.0 & 46.0 \\\\ \text{Overall average} & 36.3 & 82.9 & 55.0 \\\\ \hline \end{array}$$ This analysis identified a single source of error as the corruption type with the lowest performance: noise (Gaussian, impulse, and shot). We included these results in Sec. A.15 of the revised supplementary material. In addition, we run a comparative analysis with CLIPTTA on the three most challenging corruption types. $$\begin{array}{llccc} \hline \textbf{Corruption} & \textbf{Method} & \textbf{ACC}\uparrow & \textbf{AUC}\uparrow & \textbf{FPR95}\downarrow \\\\ \hline \text{Gaussian noise} & \text{CLIPTTA} & 27.4 & 86.5 & 59.4 \\\\ & \text{DSTA} & 23.6 & 60.2 & 92.9 \\\\ \hline \text{Impulse noise} & \text{CLIPTTA} & 28.6 & 85.2 & 63.7 \\\\ & \text{DSTA} & 23.8 & 58.6 & 91.7 \\\\ \hline \text{Shot noise} & \text{CLIPTTA} & 28.8 & 85.9 & 61.9 \\\\ & \text{DSTA} & 26.3 & 69.3 & 88.1 \\\\ \hline \end{array}$$ These results confirm that DSTA exhibits lower performance than CLIPTTA on Gaussian, impulse, and shot noise corruptions. ### Q3b. Per-corruption qualitative analysis of failure cases. We further investigate here qualitatively the degradation shown in Q3a. Taking Gaussian noise as an illustrative example, we observed a recurring phenomenon: semantically unrelated images (e.g., guenon, ground beetle, banana) consistently show weak preferences for classes such as $\textit{CRT monitor}$ and $\textit{television}$, regardless of their actual content. We hypothesize that severe noise overwhelms discriminative visual features, introducing spurious similarities with these text prototypes. However, predictions remain highly diffuse, with top-5-class probabilities of only 0.4–0.5\% and approximately 98\% of the probability mass spread across the remaining 995 classes. Consequently, the confidence scores $s_i$ fall below the Otsu threshold, causing genuine ID samples to be incorrectly assigned to the OOD group and excluded from classification adaptation. Hence, severe noise can compromise the confidence-based sample selection connecting the two streams. We will include qualitative examples for all three noise corruptions (Gaussian, impulse, and shot noise) in Sec. A.15 of the revised supplementary material. A possible direction to solve this issue is to investigate methods for mitigating spurious associations between corrupted visual features and specific text classes, to reduce their influence on adaptation. We leave this investigation to future work.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of the Responses to Reviewers (Part 1)

We sincerely thank all reviewers for their time and constructive feedback, and the AC for supervising the overall review process. In response, we conducted additional experiments and extended our analyses to address all these concerns.

In this and the above boxes (Part 1 and 2), we summarize the main findings and corresponding manuscript revisions while specific answers and more details have been reported in the individual rebuttals to Reviewers.

1. Additional Baselines to Isolate the Contribution of Decoupling

[Reviewers 73RK and aSmA] We introduced two additional controlled baselines to distinguish the benefits of independent adaptation streams from those of the other DSTA components.

(a) We evaluated a shared-adapter variant retaining DSTA's margin loss, logit calibration, LoRA placement, and a comparable parameter budget. Under these matched conditions, DSTA improves ACC by 1.6 points, AUC by 0.3 points, and reduces FPR95 by 0.9 points, directly supporting the independent contribution of parameter decoupling.

(b) We evaluated an ATTN/MLP Separate Updates baseline, which assigns classification and separation updates to different adapter sets within a single adaptation stream. Despite using DSTA's margin loss and calibration, this baseline remains below DSTA across all three metrics. These comparisons show that DSTA's gains cannot be reproduced by its individual components or objective-specific adapter allocation alone. Both baselines have been included in Tab. 4 of the revised paper.

2. Further Analysis of the Margin Loss and Logit Calibration

[Reviewers 73RK, aSmA, and xPux] (a) We clarified the theoretical and empirical evidence supporting the proposed margin-based separation objective. Importantly, the submitted paper already included an analysis showing that CLIPTTA's original separation objective can drive excessive growth of the ID/OOD separation gap when the objectives are decoupled. The margin loss addresses this behavior by limiting further optimization once the desired separation margin is reached. Under the same decoupled LayerNorm configuration, replacing the original objective with the margin loss increases AUC from 55.7% to 99.5%.

(b) We also conducted a new calibration experiment using CLIPTTA with DSTA's automatic OOD logit-scale calibration. While calibration improves AUC from 97.1% to 99.4% and reduces FPR95 from 11.5% to 1.8%, classification accuracy decreases from 67.9% to 57.9%. This demonstrates that improving OOD detection through calibration comes at the expense of classification accuracy under shared-parameter adaptation. The additional calibration results have been incorporated into Table 3, complementing the stability analysis already provided in Supplementary Sec. A.1.

3. Robustness Beyond the Standard ID/OOD Batch Composition

[Reviewers 73RK and xPux] We investigated DSTA's behavior under imbalanced ID/OOD batch compositions. We would like to emphasize that the balanced 50/50 ID/OOD composition follows the established evaluation protocol adopted by the compared open-set TTA methods. Evaluating different proportions therefore extends the standard experimental setting rather than addressing an assumption introduced specifically by DSTA.

Nevertheless, (a) we conducted additional experiments with 25/75 and 75/25 ID/OOD ratios, alongside the original 50/50 setting, on CIFAR-100/SVHN and three video action-recognition benchmarks (HAC-H/EK, HAC-A/EK, and HAC-C/EK). The results show that classification accuracy remains overall stable across the evaluated compositions. OOD detection is nearly unchanged on CIFAR-100/SVHN, while the video benchmarks show more variation, particularly when fewer OOD samples are present.

(b) We also analyzed how the estimated Otsu threshold changes with batch composition, showing that DSTA adjusts its threshold without requiring knowledge of the true ID/OOD ratio. These experiments provide additional evidence under nonstandard, imbalanced evaluation conditions, while also clarifying the limitations of batch-based distribution-dependent partitioning. The new experiments and threshold analysis are included in Supplementary Sec. A.14.

AuthorsOpenReview ↗

Summary of the Responses to Reviewers (Part 2)

4. Detailed Analysis of ImageNet-C Performance

[Reviewer 73RK] We performed additional experiments to better understand DSTA's lower performance on ImageNet-C. We first evaluated all 15 ImageNet-C corruption types individually. The results identify Gaussian, impulse, and shot noise as particularly challenging, with DSTA showing lower performance than CLIPTTA on these three corruptions. We then conducted a qualitative analysis of incorrectly rejected ID samples under these noise types. The examples show highly diffuse class probabilities and low confidence scores that fall below the estimated Otsu threshold, causing genuine ID samples to be assigned to the OOD group and excluded from classification adaptation. These findings provide a more detailed account of the method's behavior under severe noise and identify a direction for future improvement. The per-corruption results and qualitative examples have been added to Supplementary Sec. A.15.

5. Clarification of the Main Contribution and Methodological Scope

[Reviewer aSmA] We further clarified that DSTA's contribution extends beyond its individual methodological components. Our central finding is that classification and ID/OOD separation progressively lose mutual gradient support during shared-parameter adaptation. The gradients remain predominantly positively aligned; our argument concerns their decreasing alignment, rather than necessarily harmful gradient conflict. This observation motivates the use of independent adaptation streams instead of attempting to balance the two objectives within the same trainable parameter space. The additional controlled baselines strengthen the experimental support for this design. The contribution lies in identifying the underlying optimization limitation and addressing it through a targeted, relatively simple architectural change, without requiring increasingly complex adaptation mechanisms. [Reviewer xPux] Incorrect ID/OOD assignments are a potential limitation of unlabeled adaptation methods generally, including existing open-set TTA approaches. While investigating this issue further is important, resolving confirmation bias is a separate research problem beyond DSTA's present methodological focus.

Summary of Changes to the Manuscript

Newly added material and updates are highlighted in magenta.

(a) Table 3: Added the CLIPTTA calibration experiment.

(b) Table 4: Added the shared-adapter and ATTN/MLP Separate Updates baselines.

(c) Sec. 3: Clarified that declining gradient alignment serves as a motivation for decoupling.

(d) Sec. A.14: Added performance comparisons across 25/75, 50/50, and 75/25 ID/OOD ratios, together with an analysis of the estimated Otsu thresholds.

(e) Sec. A.15 : Added per-corruption results and qualitative examples of failure cases.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.