- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper studies reliable online cache-based test-time adaptation for CLIP under distribution shift, targeting three coupled issues: entropy-only cache construction admits confident but misclassified samples, online caches become class-imbalanced, and static cached features lack sample-specific local information. ROSE-TTA introduces (i) an uncertainty-aware “doubly robust” cache update that combines entropy with noise-perturbed prediction stability, (ii) a graph-based structural completion module that uses text-embedding class semantics and cached visual evidence to alleviate class imbalance, and (iii) a sample-specific cache refinement mechanism that updates cached features via a forward pseudo-gradient without backpropagation. Experiments on 15 datasets are provided.
Critical strengths
The problem under investigation is both interesting and practically important. The proposed graph-based structural completion together with the sample-specific refinement mechanism is well-designed and closely aligned with the problem characteristics.
Critical weaknesses
- The motivation is insufficient. Cache imbalance and exploitation are identified as common problems in the TTA field, ranging from conventional model-updated methods to model-fixed VLM-based TTA methods. Many existing works, such as RoTTA, DLAE, and MCP, aim to tackle these problems. However, the limitations of and relationships to these prior works are missing, which results in insufficient motivation for this work.
- The experimental comparisons are unfair. Several existing SOTA cache-based and backpropagation-free baselines are omitted. In addition, the efficiency comparisons focused on gradient-based prompt optimization methods are also unfair. These unfair comparisons falsely exaggerate the contributions of this work.
Questions for authors
- The limitations of and relationships to many existing cache-based and backpropagation-free methods are missing (e.g., RoTTA, DLAE, MCP), which leads to insufficient motivation of the work.
- The prediction stability of a sample is evaluated under Gaussian noise. What about using alternative noise generators? The rationale for this design choice is unclear. Neither theoretical analysis nor supporting empirical experiments are provided.
- The technical section is generally readable. However, the definition of in Eq. (6) is not straightforward, which hurts the readability of this section. More detailed specifications should be provided to explain its function.
- In Eq. (8), I understand that is used to update all cached features across all categories without accounting for feature quality. The rationale behind this operation is unclear. In particular, Section 3.2 aims to construct reliable, consistently stable caches. These two objectives appear contradictory.
- Unfair experimental comparisons against existing methods falsely exaggerate the performance of the proposed method.
- Multiple SOTA cache-based and backpropagation-free methods are missing, including MCP, DLAE, ProtoDCS, DOTA, ADAPT, and even Zhang’s ICML’26 work that is already cited in the related work section.
- Efficiency comparisons should focus on cache-based backpropagation-free methods (e.g., MCP, DLAE, ProtoDCS) to demonstrate efficiency, rather than prompt-based methods.
- As mentioned in the Introduction, the most interesting aspect of the proposed method lies in the early adaptation stage. How does graph-based structural completion improve performance when many class caches are empty or populated with unreliable features? This important investigation is absent from the current manuscript.
- Other minor concerns:
- How should Figures 1(a) and 1(b) be interpreted?
- Table 3 lacks ablation baselines. A baseline without the proposed components should be included to verify the effectiveness of each module.
15 comments
Q2: The prediction stability of a sample is evaluated under Gaussian noise. What about using alternative noise generators? The rationale for this design choice is unclear. Neither theoretical analysis nor supporting empirical experiments are provided.
Response: Thank you for your thoughtful comments. Other noise generators can also be used to evaluate prediction stability. We have added an experiment using different noise generators. We have added these results and the discussion to Appendix D.11.
| Method | Noise generator | IN Acc. (%) |
|---|---|---|
| TDA | — | 69.51 |
| ROSE-TTA | Gaussian | 70.78 |
| ROSE-TTA | Uniform | 70.72 |
| ROSE-TTA | Laplace | 70.81 |
From the above results, our method maintains strong performance under different types of noise perturbations.
Q3: The technical section is generally readable. However, the definition of in Eq. (6) is not straightforward, which hurts the readability of this section. More detailed specifications should be provided to explain its function.
Response: Thank you for pointing out the ambiguity. The unified categorical graph combines visual and text relations between classes. This allows classes with few or no cached samples to use visual information from related class caches for prediction.
-
Visual information from the test stream. , where sums the reliability-weighted similarities between samples in and . If is empty, row and column of are zero.
-
Global category information from text. contains similarities between the text embeddings of all classes. In , these similarities weight the visual relations in . A class with an empty cache can therefore use visual relations from classes with similar text embeddings and available cached samples.
-
Complementary predictions. provides global category information to guide feature aggregation in both the text and visual spaces, producing complementary predictions and . Through , incorporates test-specific visual information, while uses text relations to supplement incomplete class caches.
Q4: In Eq. (8), … is used to update all cached features … without accounting for feature quality … Section 3.2 aims to construct reliable, consistently stable caches … These two objectives appear contradictory.
Response: Thank you for raising this concern. Feature quality is considered through entropy and perturbation stability during cache selection. Reliable historical features may still be poorly matched to the current query, so Eq. (8) uses to refine them into for the current prediction. We retain , not , in the cache, so the two objectives do not conflict.
Q6: … the early adaptation stage … How does graph-based structural completion improve performance when many class caches are empty or populated with unreliable features? This important investigation is absent from the current manuscript.
Response: Thank you for highlighting early adaptation. We first explain how graph-based structural completion handles empty class caches and unreliable features, then evaluate its effect on cache predictions and cumulative accuracy. We have added these results and the discussion to Appendix D.8.
-
For empty class caches, provides the similarities between class text embeddings. Through , these similarities allow classes with no cached entries to use visual information from other classes' cached features.
-
For unreliable cached features, assigns smaller reliability weights to entries with higher entropy or greater perturbation sensitivity when constructing .
-
To examine the role of graph-based structural completion in early adaptation, we analyze cache prediction scores and cumulative classification accuracy.
-
We group classes by the number of entries currently stored in their caches. The reported scores are defined as follows:
-
For a test feature , is the normalized cache prediction score for class . This score excludes . Graph-based structural completion adds and , both computed using . Without these terms, ; with them, .
-
The class cache stores entries with pseudo-label . Thus, contains classes whose caches each hold exactly entries.
-
We average the class scores to obtain , separately for each setting. This is the mean score assigned to classes with cache size .
-
-
We have added a comparison of mean cache scores across cache sizes for a test sample at 10% test progress, with and without .
Dataset Graph enhancement ImageNet w/o 0.0000 0.0006 0.0007 0.0008 0.0009 0.0011 ImageNet w/ 0.0007 0.0008 0.0009 0.0009 0.0010 0.0010 Caltech101 w/o 0.0000 0.0081 0.0090 0.0103 0.0118 0.0221 Caltech101 w/ 0.0064 0.0093 0.0095 0.0101 0.0109 0.0142 - From the above results, classes with fewer entries receive higher mean normalized scores with graph-based structural completion in this illustration.
-
We have also added a comparison of cumulative top-1 accuracy over the first 10% of the same test stream.
Dataset Setting Cumulative Acc. at 10% (%) ImageNet w/o 62.68 ImageNet w/ 69.05 Caltech101 w/o 89.24 Caltech101 w/ 93.12 - From the above results, graph-based structural completion improves early adaptation accuracy on both datasets.
Q7: Other minor concerns — M1: How should Figures 1(a) and 1(b) be interpreted?
Response: Thank you for pointing this out. We have clarified Figures 1(a) and 1(b) in the revised manuscript.
- Cache Accuracy (x-axis): formally defined as , measuring the fraction of samples in a nonempty class cache whose pseudo-label matches the ground-truth .
- Frequency (y-axis): The normalized distribution of class caches. It shows the relative frequency of class caches falling into each accuracy bin, normalized to by dividing by the maximum count.
- Both figures show that entropy-based selection can retain incorrect pseudo-labels in the cache, especially during early adaptation.
Q7: Other minor concerns — M2: Table 3 lacks ablation baselines. A baseline without the proposed components should be included to verify the effectiveness of each module.
Response: Thank you for your suggestion. We have added a baseline without the three proposed modules to Table 3. It keeps CLIP frozen and updates the positive cache using entropy alone. The results show that the full method outperforms the baseline and each module contributes to accuracy.
| Metric | Baseline | w/o | w/o , | w/o | ROSE-TTA |
|---|---|---|---|---|---|
| CD Acc. (%) | 67.49 | 69.82 | 68.56 | 69.12 | 70.26 |
| IN+OOD Acc. (%) | 64.96 | 65.67 | 65.34 | 65.76 | 66.25 |
We would appreciate your consideration of these clarifications and additional results in your final evaluation. Please let us know if you have further questions. Thank you again for your detailed and constructive feedback!
We sincerely appreciate your time and insightful comments on our manuscript. Thank you for recognizing the problem and the design of our graph and sample-specific modules. Below, we address your concerns point by point and describe the corresponding revisions.
Response: Thank you for the comment. We have added a detailed comparison with RoTTA, MCP, and DLAE to explain their limitations and relationships to ROSE-TTA:
RoTTA, which also addresses cache class imbalance, balances cached samples to limit frequent-class dominance, but it does not ensure sufficient cache information for classes with few or no cached samples. ROSE-TTA completes cache information by incorporating text information through graph-based structural completion.
MCP, which also improves cache quality, uses entropy and prototype proximity for sample selection, but does not reuse these selection scores as reliability weights in positive-cache retrieval. ROSE-TTA uses perturbation stability and entropy-based confidence in both cache construction and cache utilization, refining the cache information used for prediction.
DLAE improves cache coverage through logit recalibration and exploratory sample selection. However, it still leaves information gaps in the cache unresolved when suitable samples are unavailable. ROSE-TTA completes cache information through graph-based structural completion, using text relations to share existing visual evidence across classes.
We further clarify the motivation and joint design of ROSE-TTA:
Motivation. In streaming settings, reliable cache information can still be incomplete, while completing missing information must account for the reliability of the available information. This motivates us to jointly address cache reliability and completeness end to end, from cache construction through utilization to prediction.
Technical novelty. We introduce a unified framework that jointly completes and refines cache information. Uncertainty-aware cache construction favors reliable entries and passes their reliability estimates to graph-based structural completion. For reliability, these estimates give greater weight to reliable cache information. For completeness, text relations supply the global category information missing from sparse caches. Finally, the unified categorical graph G^ integrates both sources to provide more reliable and complete cache information for prediction. Sample-specific refinement further injects online information from the current query to enrich cache information.
Response: Thank you for these suggestions. We have added accuracy comparisons under shared cache and text settings and efficiency comparisons with additional cache-based methods.
Accuracy comparisons. For a fair comparison,we use a total capacity of 5 cache entries per class. All methods share the same text descriptions. ROSE-TTA ranks first on both metrics across both backbones. We have added these results to Appendix D.9.
Efficiency comparison. ROSE-TTA has the shortest runtime and highest accuracy among the compared adaptation methods. We have added these results to Appendix C.4.