acceptodds
Under review as a conference paper at ICLR 2027

RoAR: Robust Test-Time Refinement with Unknown Priors for Vision-Language Models

Abstract

Test-time adaptation improves the zero-shot predictions of vision-language models using only the unlabeled test pool, and its gains are measured on pools whose priors are fixed: classes are roughly balanced, every class in the label set is present, and every image belongs to the label set. A deployed model receives a pool whose priors are unknown. We show that strong gains on the original benchmark pools need not persist when their class composition changes. The strongest evaluated baseline on the original pools falls below zero-shot in mean accuracy at the strongest tested level of class imbalance, class absence, and open-set contamination. We release an evaluation protocol that turns the class composition of a test pool into controlled prior axes, and we propose **RoAR** (Robust Anchored Refinement), a transductive test-time refinement for pools with unknown priors. It pairs a vision-language model with a self-supervised encoder, screens potentially unknown images before aggregation, and holds a selectively reweighted zero-shot anchor fixed during refinement. In two-view pooling, anchor confidence sets the total prototype exponent, while geometric scores determine the relative weights of the prototype views. With a single configuration shared across all datasets and prior settings, RoAR matches the strongest existing method on the original pools, improving over zero-shot by 7.7 percentage points on ImageNet and ten cross-domain datasets. Under severe class imbalance or absent classes, where that method drops below zero-shot, RoAR outperforms it by about 8 percentage points.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
test-time adaptation, vision-language models, open-set recognition

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GGzPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies TTA under unknown class priors, including class imbalance, class absence, and open-set samples. It proposes an evaluation protocol for these prior shifts and RoAR, a test-time refinement method that uses prototype information while anchoring to the original zero-shot predictions.

Critical strengths
  1. The paper addresses a meaningful and practically relevant limitation of existing TTA benchmarks. The explicit consideration of class imbalance, class absence, and open-set contamination provides a useful framework for studying how test-time adaptation behaves when the test-pool prior is unknown, rather than assuming that all candidate classes are adequately represented in the pool.
Critical weaknesses
  1. The paper is hard to follow. Although the overall idea of using a zero-shot prediction as an anchor and refining it with test-pool-derived prototypes is interesting, the method is presented in a highly technical and fragmented manner. In particular, the roles of the anchor, the two prototype views, the prototype exponent, geometric scores, and the fallback whitened view are not clearly motivated before being introduced. A more intuitive overview or step-by-step description of the refinement process would make the method much easier to understand and evaluate.

  2. The novelty of RoAR relative to prior open-set TTA is unclear. The related-work section mainly mentions AdaND and does not adequately cover existing OSTTA methods that identify unknown/OOD samples and prevent their contamination of adaptation. The authors should clarify whether RoAR's main contribution is handling unknown priors more broadly, including class imbalance and class absence, and explain why existing OSTTA methods are insufficient under this setting.

The baseline selection also underrepresents open-set methods. Most baselines (e.g., StatA, TransCLIP, GDA, COSMIC, and SOTA) are general VLM TTA or refinement methods, while only ZS-NTTA appears to directly address noisy/out-of-label samples. Given the central role of unknown-sample screening in RoAR, comparisons with more existing open-set TTA and test-time OOD filtering methods would strengthen the evaluation.

[1]Gao, Zhengqing, Xu-Yao Zhang, and Cheng-Lin Liu. "Unified entropy optimization for open-set test-time adaptation." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

[2]Sreenivas, Manogna, and Soma Biswas. "Efficient open set single image test time adaptation of vision language models." arXiv preprint arXiv:2406.00481 (2024).

[3]Luo, Wei, et al. "ProtoDCS: Towards Robust and Efficient Open-Set Test-Time Adaptation for Vision-Language Models." arXiv preprint arXiv:2602.23653 (2026).

[4]Feng, Mingxu, Fengqiang Wan, and Yang Yang. "Open-Set Test-Time Adaptation in VLMs via Noise-Immune Self-Purification.”

Questions for authors

1.The paper positions open-set contamination as one of the three central unknown-prior settings. How does RoAR’s screening-and-prototype construction differ fundamentally from existing open-set TTA methods that already identify and filter unknown samples before adaptation? In particular, which component of RoAR should be considered the main methodological novelty beyond combining existing negative-label and visual-clustering ideas?

2.Since open-set contamination is a central setting in the proposed protocol, could the authors clarify why the evaluation does not include more open-set TTA or test-time OOD filtering methods beyond ZS-NTTA? How should RoAR’s reported gains be interpreted relative to those methods?

5 comments

Weakness 1

We thank the reviewer for this comment. We will reorganize Section 3 into three steps and state more clearly what each step does and why it is needed.

  1. Membership screening rejects unknowns using negative-label scores and DINOv2 clusters (Sec. 3.1).
  2. The VLM anchor is adjusted once using the predicted class distribution of the retained images (Sec. 3.2).
  3. Refinement combines the prototype predictions of the raw and residual views with the fixed anchor (Secs. 3.3–3.4).

Weakness 2

Yes. The main contribution of RoAR is handling unknown priors broadly rather than open-set rejection. RoAR handles class imbalance, class absence and unknown contamination with one configuration and without information about the prior (Tables 1–3 of the paper, and Table G.4 when all three occur together). Existing OSTTA methods set their rejection criterion assuming that unknowns are present, so they reject 21–51% of the known images even on pools without unknowns (Q2, Table b). Pools with class imbalance or absent classes also contain no unknowns, and OSTTA methods have no mechanism for handling frequency changes among the known classes. On the ρ=1000 and 50% absent pools the compared OSTTA methods reject 29–52% of the known images, while RoAR rejects none (0.00%).

The submission did not sufficiently cover closely related OSTTA work such as UniEnt, ROSITA, ProtoDCS and NISP. We will add these works to the Related Work section and have added comparisons with the methods that have runnable implementations (Q2). Q1 summarizes the differences and Q2 reports the comparison results.

Question 1

The core methodological contribution of RoAR is not the screening but the refinement rule that combines a fixed VLM anchor with prototype evidence from an auxiliary visual representation. Negative-label scoring and clustering are existing components. RoAR's contribution lies in how they are combined and in the refinement rule built on top of them. Existing OSTTA methods keep updating the model after filtering unknowns, so unknowns that pass the filter and incorrect pseudo-labels can affect later predictions through these updates. RoAR does not update the encoder. It constructs the VLM anchor once per pool and keeps it fixed during refinement. The reference with which the prototype predictions are combined therefore stays the same even as the prototype estimates change. With a single configuration, RoAR maintained known accuracy across the evaluated conditions (presence and proportion of unknowns, class imbalance and class absence).

RoAR differs from existing OSTTA methods in two ways.

  1. Adaptation. Most existing methods compute an entropy or contrastive loss on selected samples and update prompts, normalization layers or adapters. RoAR performs no gradient update. It fixes the VLM posterior as the anchor and combines it with raw and residual prototypes estimated from the pool.
  2. Unit of rejection. Existing methods mainly apply a stream-adaptive threshold to per-image scores. RoAR uses both the cluster mean and the individual membership score. A DINOv2 cluster is rejected only when its mean score falls below the cutoff, and images with high individual scores are kept even within a rejected cluster. Table b in Q2 reports the known false rejection rate of this rule.

Question 2

The OSTTA comparison in the submission was insufficient. In addition to ZS-NTTA from the submission, we evaluated ROSITA, UniEnt, DDE, OpenXpert and NISP. We could not compare with ProtoDCS because no runnable public code is available. We mixed Places images into the full known test splits of ImageNet and the ten cross-domain datasets. Each method used its released implementation and its own input configuration.

At deployment it is not known whether unknowns are present at all, so we also evaluated clean pools without unknowns (0%) with the same configuration. Existing OSTTA methods reject a substantial share of the known images even when no unknowns are present.

Table a. Known accuracy / H-score by unknown proportion (%)

Method0% (clean)25%50%75%
ROSITA38.1855.61 / 68.1557.33 / 69.0959.17 / 69.36
UniEnt37.3855.25 / 67.3856.92 / 68.1758.68 / 68.49
ZS-NTTA57.2256.93 / 54.0756.90 / 60.4657.84 / 63.52
DDE48.6151.73 / 60.1352.89 / 61.2454.13 / 62.35
OpenXpert39.9347.25 / 60.3858.56 / 67.9860.97 / 68.04
NISP†57.9457.55 / 60.1858.65 / 58.6159.75 / 51.74
RoAR76.5375.73 / 56.4875.44 / 64.3175.09 / 66.53

The H-score is undefined on clean pools because they contain no unknowns.

Table b. Known false rejection rate by unknown proportion (%, ↓)

Method0% (clean)Per-dataset max at 0%25%50%75%
ROSITA47.2056.91 (DTD)15.5912.679.17
UniEnt48.8155.61 (DTD)16.2913.439.93
ZS-NTTA23.1985.99 (Aircraft)23.3822.4319.34
DDE29.5391.43 (EuroSAT)24.6222.7119.74
OpenXpert50.7761.47 (DTD)39.6719.5812.45
NISP†21.4272.94 (Aircraft)19.9717.3714.70
RoAR0.000.000.040.390.48

The compared methods reject 21–51% of the known images on clean pools, and their false rejection rate generally increases as the unknown proportion decreases. ROSITA and UniEnt reach high AUROC on the Places mixture pools (94.9 and 93.4) but still reject many known images on clean pools. A score's ability to separate known from unknown images therefore has to be evaluated together with how well known images are preserved at the actual rejection threshold. RoAR keeps the known false rejection rate at or below 0.5% at every evaluated proportion.

To check the effect of the unknown source, we replaced Places with iNaturalist, a semantically distant unknown source. We mixed all 10,000 iNaturalist OOD images into the full known test split of each dataset and kept the other settings unchanged.

Table c. iNaturalist unknowns (%)

MethodKnown accuracy ↑H-score ↑Known false rejection ↓
ROSITA55.4366.2917.31
UniEnt55.3866.7517.29
ZS-NTTA56.5663.1221.72
DDE59.6070.6611.74
OpenXpert58.3966.3018.34
NISP†57.3756.8020.47
RoAR74.6978.480.94

With iNaturalist unknowns and the same configuration, RoAR keeps the known false rejection rate below 1% and obtains the highest known accuracy and H-score among all compared methods.

In every cell a rejected known image counts as an error. † NISP uses our implementation that restores the parts missing from the released code. We will add the works suggested by the reviewer to the Related Work section.

Thank you again for your review. We would appreciate any feedback on whether our responses have addressed your concerns. If anything remains unclear or insufficient, we would be glad to provide further clarification or additional results.

Reviewer_YtLYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a real blind spot in VLM test-time adaptation: methods are evaluated only on pools with a fixed, known prior (balanced, closed-set, every class present), but a deployed model faces a pool whose class composition is unknown. The authors decompose the prior into three controllable axes — class imbalance π, absent classes P, open-set contamination ε — and release a reproducible protocol (pool construction, draws, seeds). They show that the strongest baseline, SOTA, falls below zero-shot at the extreme of each axis (e.g., 64.9 vs 69.5 at ρ=1000; 61.4 vs 68.8 on known images at 75% unknown). They propose RoAR, which fixes a zero-shot anchor, reweights it once only when predictions are concentrated, and pools it with two-view DINOv2 prototypes plus cluster-level unknown screening, using a single configuration across 11 datasets and three backbones.

Critical strengths
  1. The problem is well-chosen and under-explored; it is the right kind of correction for the subfield, in the spirit of TTA-VLM's "illusion of progress." 2.The evaluation protocol is clean and reproducible, and correctly treats the pool (not the image) as the unit at which the prior is specified.
Critical weaknesses
  • Novelty is incremental. RoAR combines existing pieces (NegLabel-style OOD screening, DINOv2 prototypes as in COSMIC/SOTA, KL-barycenter pooling, text-anchoring as in StatA). The genuine contribution is the combination and its systematic validation under unknown priors, not a conceptual leap.
  • Open-set test skews far-OOD. Unknowns are Places/iNaturalist; on SUN397 (near-OOD, Table G.2) all-images accuracy collapses to 17.9 at 75% unknown. The open-set robustness conclusion mainly holds for far-OOD and near-OOD remains an unresolved risk that should be acknowledged in the main text.
  • Open-set all-images gain is mostly rejection, not refinement. Table G.4: RoAR without rejection reaches only 17.9 at 75% unknown; the abstract's "only method above zero-shot everywhere" should be qualified by this distinction (known-image gain = refinement, all-image gain = rejection).
Questions for authors

The evaluation protocol is a net-positive contribution and the empirical work is honest and well-attributed. The drags are incremental novelty, unaddressed comparison fairness, the missing "no tuning against altered-pool" statement, far-OOD skew of the open-set claim, and unreported compute. Adding a CLIP-only variant, compute costs, a near-OOD boundary statement, and an explicit constant-selection declaration would justify raising the score to 7–8. The work's main value is the reproducible infrastructure that forces the community to ask about prior robustness.

3 comments

We thank the reviewer for recognizing the value of the evaluation protocol and the importance of the problem. We agree that negative-label scoring, DINOv2 prototypes and geometric pooling are existing components. The core methodological contribution is the refinement rule that combines a fixed VLM anchor with prototype evidence from an auxiliary visual representation. The anchor is adjusted once and then held fixed, its confidence sets how much weight the prototypes receive, and geometry only allocates that weight across the views. Table 4 and Table G.3 of the paper isolate the effect of this rule from the screening. Alongside the method, we treat class imbalance, class absence and unknown contamination within one evaluation framework and apply the same refinement configuration to pools whose composition is not known in advance.

1. CLIP-only variant

To check whether RoAR can be applied without an additional visual encoder, we evaluated a CLIP-only variant that replaces the DINOv2 features with the 512-dimensional image embeddings of the frozen CLIP model.

ConditionZero-shotCLIP-only variantGain (pp)
Original pools68.7870.81+2.03
Imbalance, ρ=100069.5171.46+1.95
50% absent classes68.7170.74+2.02
Places unknowns 25%68.7870.60+1.81
Places unknowns 50%68.7870.23+1.44
Places unknowns 75%68.7869.39+0.60

The CLIP-only variant stays above zero-shot in every condition, with gains of 0.6 to 2.0 pp.

2. Computational cost on ImageNet

We measured the per-image running time and the peak GPU memory on the full 50,000-image ImageNet validation set.

Methodms/imageGPU memory (GB)ImageNet accuracy
CLIP zero-shot (1 view)8.530.8568.32
CLIP zero-shot (16 views)22.720.8570.2
GDA22.761.3370.71
StatA23.1910.3672.36
TransCLIP23.1511.2472.70
SOTA16.0211.6878.0
TPT89715.868.94
COSMIC4325.272.3
RoAR30.124.2180.4

RoAR runs both CLIP and DINOv2 in inference mode only and performs no gradient update, so the second encoder adds little time. RoAR takes about 7 ms more per image than GDA, StatA and TransCLIP, and uses less GPU memory than StatA, TransCLIP and SOTA. It is about 30 times faster than TPT, which optimizes prompts by backpropagation at test time.

3. Near-OOD limitation and all-image accuracy

Near-OOD. As the SUN397 result noted by the reviewer shows, detecting semantically close unknowns is a limitation of RoAR. Unknown images can pass the screening when they are visually similar to known images or when the negative vocabulary does not capture the difference. We therefore do not generalize the far-OOD results to near-OOD. In the revision we will state in the main text and in the Limitations that RoAR does not reliably reject near-OOD unknowns.

Near-OOD images are close to the known classes both visually and semantically, so they readily receive confident known predictions and are hard to separate from known images during adaptation.

All-image accuracy. We agree. On open-set pools the all-image gain comes mostly from rejecting unknowns, and the known-image gain comes mostly from the refinement. Table 4 of the paper separates the two at 75% unknown: the known-image gain is +3.0 pp without the rejection step and +4.8 pp with it. On pools without unknowns RoAR rejects no known image (0.00%), so the gains in Tables 1 and 2 come from the refinement alone. In the revision we will report the two gains separately and qualify the statement that RoAR stays above zero-shot on every axis: on open-set pools it refers to known-image accuracy, and the all-image gain is attributed to rejection.

4. Choice of constants and transfer of the fixed configuration

No constant was tuned against the class-imbalance or absent-class pools, and no constant was tuned on the ten cross-domain datasets. All selection used ImageNet only. The only altered pools used for selection were the ImageNet pools with Places unknowns, and only for the screening constant below. The constants were chosen as follows.

  • From prior work. The number of negative groups (100) and the mining quantile (0.95) follow the released NegLabel configuration.
  • Refinement. The anchor reweighting, the weight that combines the original prediction with the prototypes and the fallback setting were selected on the original ImageNet pool only, using its ground-truth labels. This pool is the official validation split and is also the ImageNet test pool in our evaluation, not a separately held-out tuning set. The selected configuration was then fixed and evaluated on the other datasets. The refinement itself never uses labels.
  • Screening. The negative vocabulary fraction (30%) was set on ImageNet, after checking the trade-off between unknown detection and known false rejection on its Places mixture pools and on its pool without unknowns. The cross-domain datasets were not used for this choice.

The appendix of the revision will list the value of each constant and the evidence for its choice.

The final configuration was applied identically to every evaluated dataset and prior condition. It was also used across CLIP, SigLIP and SigLIP2 without per-model or per-dataset retuning.

Sensitivity. We checked whether the gain depends on one particular combination of constants. We evaluated 64 configurations that jointly vary 18 settings within ranges fixed in advance. On the original, ρ=1000 and 50% absent pools of the eleven datasets, the mean known accuracy of every configuration was at or above zero-shot. This analysis covers known classification at seed 0, and the detailed results will be given in the appendix.

New datasets. After fixing the configuration, we evaluated four datasets added after submission (AID, Stanford Dogs, CUB-200 and HAR) with the screening and refinement settings unchanged. The mean gains over zero-shot are +8.99, +5.79 and +6.14 pp on the original, ρ=1000 and 50% absent pools.

DatasetOriginal: Zero-shot / RoARρ=1000: Zero-shot / RoAR50% absent: Zero-shot / RoAR
AID61.32 / 71.4173.21 / 78.3562.35 / 67.05
Stanford Dogs65.15 / 77.9670.04 / 80.4164.68 / 74.41
CUB-20057.23 / 68.2559.93 / 66.7258.06 / 68.14
HAR87.24 / 89.2885.93 / 86.7887.56 / 87.61

The fixed configuration therefore transfers to new datasets without change, which indicates that the gains of RoAR are not fitted to the original benchmarks.

Thank you again for your review. We would appreciate any feedback on whether our responses have addressed your concerns. If anything remains unclear or insufficient, we would be glad to provide further clarification or additional results.

Reviewer_ViwjOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RoAR, a test-time refinement method for vision-language models under unknown test-pool priors. It combines CLIP predictions with DINOv2 features through membership screening, anchor reweighting, and prototype refinement. Experiments across multiple datasets demonstrate strong performance under class imbalance, absent classes, and open-set contamination.

Critical strengths

RoAR achieves substantial improvements over existing TTA methods across multiple datasets.

Critical weaknesses
  1. RoAR involves around 21 numerical settings, yet their selection procedure is unclear. With such a massive number of tunable parameters, it is hard for me to be convinced of the work's true generalization ability. Several thresholds are highly sensitive, raising concerns about robustness and potential benchmark-specific tuning.
  2. Several components rely on heuristics without sufficient justification. In particular, histogram-based anchor reweighting may amplify existing prediction biases, and its independent contribution is not adequately validated.
  3. The evaluation focuses on specifically constructed prior shifts that closely match the proposed motivation. The effectiveness under more realistic distributions, particularly near-OOD scenarios, remains unclear.
  4. RoAR requires an additional DINOv2-L encoder, clustering, eigendecomposition, and iterative prototype refinement, yet provides no systematic efficiency analysis. Its practical advantages over simpler TTA methods remain unclear.
Questions for authors
  1. How were the numerous hyperparameters selected without a validation set? Were the evaluation datasets involved in determining their values?
  2. Why should frequently predicted classes receive additional weight in Eq. (4)? Could this amplify existing prediction biases? Please provide an ablation with η = 0.
  3. What is the specific advantage of DINOv2-based clustering over existing per-image OOD detection methods such as NegLabel? How does this strategy perform when unknown classes are semantically close to known classes?
4 comments

We thank the reviewer for the comments. We address each point below.

Question 1 / Weakness 1. Choice of constants and generalization

No constant was tuned against the class-imbalance or absent-class pools, and no constant was tuned on the ten cross-domain datasets. All selection used ImageNet only. The only altered pools used for selection were the ImageNet pools with Places unknowns, and only for the screening constant below. The constants were chosen as follows.

  • From prior work. The number of negative groups (100) and the mining quantile (0.95) follow the released NegLabel configuration.
  • Refinement. The anchor reweighting, the weight that combines the original prediction with the prototypes and the fallback setting were selected on the original ImageNet pool only, using its ground-truth labels. This pool is the official validation split and is also the ImageNet test pool in our evaluation, not a separately held-out tuning set. The selected configuration was then fixed and evaluated on the other datasets. The refinement itself never uses labels.
  • Screening. The negative vocabulary fraction (30%) was set on ImageNet, after checking the trade-off between unknown detection and known false rejection on its Places mixture pools and on its pool without unknowns. The cross-domain datasets were not used for this choice.

The appendix of the revision will list the value of each constant and the evidence for its choice. The final configuration was applied identically to every evaluated dataset and prior condition. It was also used across CLIP, SigLIP and SigLIP2 without per-model or per-dataset retuning.

New datasets. After fixing the configuration, we evaluated four datasets added after submission (AID, Stanford Dogs, CUB-200 and HAR) with the screening and refinement settings unchanged.

DatasetOriginal: Zero-shot / RoARρ=1000: Zero-shot / RoAR50% absent: Zero-shot / RoAR
AID61.32 / 71.4173.21 / 78.3562.35 / 67.05
Stanford Dogs65.15 / 77.9670.04 / 80.4164.68 / 74.41
CUB-20057.23 / 68.2559.93 / 66.7258.06 / 68.14
HAR87.24 / 89.2885.93 / 86.7887.56 / 87.61

The mean gains over zero-shot are +8.99, +5.79 and +6.14 pp on the original, ρ=1000 and 50% absent pools. With the same configuration, SigLIP gains +3.43 to +6.19 pp and SigLIP2 gains +3.08 to +5.30 pp on these four datasets across the three conditions (seed 0).

Sensitivity. We checked whether the gain depends on one particular combination of constants. We evaluated 64 configurations that jointly vary 18 settings within ranges fixed in advance. The table gives the largest drop in mean known accuracy from the default configuration (pp).

DatasetsOriginalρ=100050% absent
Original eleven1.031.670.67
Four new0.651.180.61

For every configuration, the mean known accuracy was at or above zero-shot in all three conditions (seed 0). We did not re-select the default after this analysis.

The screening threshold that appeared sensitive mainly moves the operating point of unknown detection. On the 50% unknown pools, moving the cluster cut from 0.4 to 0.5 to 0.6 raises unknown recall from 56.5% to 63.8% to 71.4%, while known accuracy stays within 0.6 pp (75.42%, 75.43% and 74.90%). These numbers use the full test splits of the eleven datasets and seed 0.

Question 2. The reweighting in Eq. (4) and the η = 0 ablation

Eq. (4) lets the anchor reflect the class composition of the pool. In pools with class imbalance or absent classes, the zero-shot predictions concentrate on the classes that are actually frequent, so the predicted histogram carries information about the class proportions of the pool. Eq. (4) has the algebraic form of a label-shift correction with this histogram as the prior estimate. The posterior odds of two classes are multiplied by the ratio of their histogram values raised to the power η, and η = 0 returns the zero-shot posterior.

As the reviewer points out, the histogram also contains misclassifications and can amplify bias. We limit this in three ways. The adjustment is applied only in pools whose predictions concentrate on few classes (effective number of predicted classes below half of all classes). It is damped with η = 1/2. It is applied once per pool and then held fixed, so it does not accumulate over iterations. The table sets only η to 0 and keeps the screening and refinement unchanged (mean over the original eleven datasets).

ConditionZero-shotη=0RoAR (η=1/2)Difference (pp)
Original68.7876.5176.510.00
ρ=100069.5172.2073.02+0.82
50% absent68.7173.2173.28+0.06
Places unknown 50%68.7875.3575.350.00

With η = 0 RoAR still gains +2.69 pp over zero-shot at ρ=1000, so its gain does not depend on the reweighting.

We also examined the tail classes on the same ρ=1000 pools, where the tail is the lowest quarter of the present classes by number of images. RoAR preserves the tail classes best among the compared TTA methods.

MethodTail-class accuracyBalanced accuracyMacro F1Accuracy
Zero-shot67.468.655.969.5
GDA66.868.856.470.3
StatA61.365.855.169.8
SOTA64.466.552.064.8
RoAR67.771.060.073.0

Question 3. Clustering versus per-image OOD decisions such as NegLabel, and near-OOD

The benefit of clustering is not stronger detection but fewer false rejections of known images. A per-image rule rejects every known image with a low score. The cluster rule rejects a cluster of visually similar images only when its mean score is below the cutoff, and even then it keeps images with a high individual score. This difference matters when it is not known whether the pool contains unknowns. The table compares RoAR with per-image rejection rules and existing OSTTA methods on clean pools without unknowns

MethodClean known accuracyClean known false rejection ↓
ROSITA38.1847.20
UniEnt37.3848.81
ZS-NTTA57.2223.19
DDE48.6129.53
OpenXpert39.9350.77
NISP†57.9421.42
Per-image rejection (RoAR score)75.411.62
Per-image rejection (NegLabel score)76.250.44
RoAR (cluster + veto)76.530.00

These results use the full known test splits of the eleven datasets with seed 0, and a rejected known image counts as an error. This is the same experiment as in our response to Reviewer GGzP (Q2). The per-image rows use the same features and refinement as RoAR. The first also uses the RoAR score, so it differs from RoAR only in the rejection rule. † NISP uses our implementation that restores the parts missing from the released code

With the cluster rule, known false rejection is 0% on all eleven datasets for both the RoAR score and the NegLabel score, whereas per-image NegLabel rejects 2.0% of the known images on DTD and 1.6% on EuroSAT. The per-image rule has higher unknown recall in exchange (73.65% against 63.84% on the 50% Places pools), and we do not claim superior detection. Existing OSTTA methods set their rejection criterion assuming that unknowns are present, so they reject 21–51% of the known images even on clean pools.

The screening is also replaceable. When it is replaced by per-image NegLabel, the known-accuracy gain from the refinement remains +7.67 pp, so the gain of RoAR comes from the refinement rule and not from a particular screening rule. The original NegLabel threshold is set with ID validation data, so this comparison uses 0.5, which needs no validation data.

Detecting semantically close unknowns is a limitation of RoAR. When the unknowns are semantically close to the known classes, most of them pass the screening. Near-OOD images are close to the known classes both visually and semantically, so they readily receive confident known predictions and fall into the same clusters as known images. The cluster rule still rejects no known image in this case, and known accuracy still improves over zero-shot. Existing OSTTA methods do not handle this case well either, as their known accuracy falls below zero-shot on the same pools. We will state this scope in the main text and in the Limitations.

Weakness 2. Justification of the components

Each component has an exact property, listed below. The revision states it right after the corresponding equation.

  • Combination rule (Eq. 9). The weighted geometric mean of the anchor and prototype distributions is the unique minimizer of (1−α)KL(q‖p) + αKL(q‖r), that is, their KL barycenter.
  • Anchor reweighting (Eq. 4). This is a plug-in label-shift correction with the predicted histogram as the prior estimate, damped by the exponent η (Q2).
  • Residual view. The projection removes the rank-k subspace that carries the most within-pseudo-class variance.
  • Cluster cut 1/2. A cluster mean score below 1/2 means that the candidate-class mass is smaller than the negative mass. A known cluster A with expected score μ > 1/2 is rejected with probability at most exp(−2|A|(μ−1/2)²), which decreases exponentially with the cluster size (Hoeffding, under independence).

These properties explain what each operation computes, but they do not determine the values of the constants or guarantee an accuracy gain. Q1 covers the validation of the constants and Q2 the separate contribution of the reweighting.

Weakness 3. Evaluation distributions and near-OOD

We vary class imbalance, absent classes and unknown contamination one axis at a time so that the failure under each change can be isolated. We also evaluated pools in which all three occur together (Table G.4) and pools with iNaturalist as the unknown source. With 10,000 iNaturalist images mixed into the full known test splits, RoAR reaches 74.69% known accuracy and a 78.48% H-score with 0.94% known false rejection. Its known accuracy and H-score are higher than those of every open-set TTA method we compared (response to Reviewer GGzP, Q2). Q3 discusses the near-OOD case.

Weakness 4. Computational cost

We measured the per-image running time and the peak GPU memory on the full 50,000-image ImageNet validation set (RTX 4090).

Methodms/imageGPU memory (GB)ImageNet accuracy
CLIP zero-shot (1 view)8.530.8568.32
CLIP zero-shot (16 views)22.720.8570.2
GDA22.761.3370.71
StatA23.1910.3672.36
TransCLIP23.1511.2472.70
SOTA16.0211.6878.0
TPT89715.868.94
COSMIC4325.272.3
RoAR30.124.2180.4

The clustering, eigendecomposition and prototype refinement that the reviewer mentions take 1.07 s in total for all 50,000 images. This is 0.021 ms per image, less than 0.1% of the total time. The corresponding post-processing takes 0.213 ms for StatA, 0.405 ms for TransCLIP and 0.583 ms for SOTA. Most of the running time of RoAR is image encoding by CLIP and DINOv2, which both run in inference mode with no gradient update. RoAR takes about 7 ms more per image than 16-view CLIP zero-shot and raises ImageNet accuracy by 10.2 pp. It uses less GPU memory than StatA, TransCLIP and SOTA and is about 30 times faster than TPT.

Thank you again for your review. We would appreciate any feedback on whether our responses have addressed your concerns. If anything remains unclear or insufficient, we would be glad to provide further clarification or additional results.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the discussion for the Area Chair

We thank the reviewers and the Area Chair for their time and comments. The revised PDF (changes in red) and our replies address the points as follows.

What the evaluation shows

  • Protocol. The protocol varies class frequency, class presence and unknown share, and the pools and seeds are released.
  • Gains that do not survive a change of prior. Every compared method falls below zero-shot on at least one axis, and SOTA goes from 6.8 points above zero-shot on the original pools to below zero-shot at the strongest level of each axis. RoAR stays above zero-shot in mean accuracy at every evaluated level of every axis and leads the existing test-time methods there by a substantial margin, about 8 points over SOTA on the class axes.

Reviewer Viwj

  • Selection of the constants. Appendix A now states the procedure. All selection used ImageNet, none on the class-imbalance or absent-class pools or the ten cross-domain datasets, and the only altered pools used were ImageNet pools with Places unknowns, for the negative-vocabulary fraction.
  • The robustness claim. We respectfully disagree that the claim is stronger than the evidence. With one configuration, selected on ImageNet as those of ZERO, TPT and StatA were, RoAR has the highest mean in all nine prior conditions on the ten cross-domain datasets, with gains of 3.3 to 7.5 points over zero-shot, and gains 9.0, 5.8 and 6.1 points on four datasets added during the discussion. It stays above zero-shot under class imbalance and absent classes, where the compared methods fall below it, and leads SOTA by a wide margin at the strongest level of each axis. On the tail classes of naturally imbalanced test splits it is more accurate than the compared methods on all four datasets. On clean pools without unknown images, where the open-set methods reject 21 to 51 percent of the known images and fall far below zero-shot, RoAR rejects none and gains 7.7 points. The same configuration extends to SigLIP and SigLIP2, where existing test-time methods show less consistent gains and SOTA falls below zero-shot at the strongest level of each axis. We therefore regard the evidence as supporting robust adaptation under unknown priors.
  • Sensitivity. All 64 configurations that jointly move 18 constants within preset ranges stay at or above zero-shot in mean known accuracy on the class-axis pools, largest dataset-mean drop 1.8 points.
  • Methodological contribution. Each part of the rule is tested under the prior change it targets. A naive prototype classifier from the same features gains 0.5 and 1.6 points under imbalance and absent classes against 3.5 and 4.6 for RoAR, SOTA with its marginal replaced stays 3.9 points below RoAR at ρ = 1000, and SOTA with the same rejection step stays 6.5 points below at 75 percent unknown.
  • Claims and evidence. On 1,098 randomly generated pools RoAR is at or above zero-shot on 94.2 percent, against 27.3 percent for TransCLIP and 37.2 percent for SOTA with the shared input. On four naturally imbalanced test splits it is the most accurate in 11 of 12 class groups.

Reviewer YtLY

  • CLIP-only variant, compute, near-OOD boundary and constant selection. The CLIP-only variant stays above zero-shot in every condition, by 0.6 to 2.0 points. Compute, about 7 ms per image over sixteen-view zero-shot inference and a third of the GPU memory of SOTA. Near-OOD boundary, Section 4 and the Limitations now state that RoAR does not reliably reject semantically close unknowns, and the every-axis statement is restricted to known images on open-set pools, with the all-image gain attributed to rejection. Constant selection, Appendix A declares the procedure, all on ImageNet and none on the altered pools or the cross-domain datasets.
  • Open-set comparison. Six open-set TTA methods were added, and with iNaturalist unknowns RoAR has the highest known accuracy and H-score.
  • Novelty. What is new is the behavior. Each existing method performs well in the setting its design assumes and falls below zero-shot outside it, under a long tail, with absent classes or with unknown images, and the open-set methods also fail on clean pools where every image is known. RoAR is the only evaluated method that adapts robustly across all three changes, keeping the tail classes under imbalance, staying above zero-shot with absent classes and with unknowns, and retaining the known images on clean pools, with one configuration that also extends to SigLIP and SigLIP2.

Reviewer GGzP

  • Readability. Section 3 was rewritten. It states the problem first, then the three decisions, and explains each role before any equation.
  • Open-set TTA. The Related Work covers UniEnt, ROSITA, NISP, ProtoDCS, DDE and OpenXpert, six implementations were evaluated, and Section 4 discusses the trade-off between retaining known images and detecting unknowns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.