- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies continual test-time adaptation of training-free, CLIP-based open-vocabulary segmentation models over a long unlabeled test stream. Running 11 existing methods for 150 rounds on 8 benchmarks, the authors report a trade-off: methods that gain early eventually collapse to a few classes, while stable methods gain little.
The proposed CAGA adapts only the vision encoder's LayerNorm parameters with a filtered entropy loss. When the gradient norm trends upward, it partially restores the parameters to an earlier state: a recent checkpoint if predictions are still diverse, otherwise a stored reliable anchor.
Experiments use one main backbone (NACLIP ViT-L/14), 8 benchmarks and a single seed, plus an ACDC-only study over 14 backbone variants. CAGA is the only continual method that ends above the unadapted model on all 8 benchmarks, with an average relative gain of 13.6%.
Critical strengths
- A large long-horizon study: 11 baselines 8 benchmarks 150 rounds, with per-corruption results. It shows that many methods, including CTTA- and stability-oriented ones (ViDA, COME, PTTA, DAF), end below the unadapted model on all 8 benchmarks.
- CAGA is the only continual method above No-Adapt on all 8 benchmarks (worst-case gain +6.4%, worst-case peak retention 98.1%). Its restoration step should also be cheap, since it only stores LayerNorm snapshots.
Critical weaknesses
1. Limited novelty, and the gating is not shown to beat simple resets.
Long-horizon collapse and reset-based fixes are already known. RDumb (NeurIPS 2023) showed that most TTA methods eventually fall below a non-adapting model, and that periodic resets match or beat them. PeTTA (NeurIPS 2024) detects drift toward collapse on recurring test streams. Most directly, Lim, Hwang & Lee ("When and Where to Reset Matters for Long-Term Test-Time Adaptation", ICLR 2026) use the entropy of the batch-mean prediction to decide when to reset and how many layers to reset, which is close to CAGA's class-marginal-entropy gate. None of these is cited.
The paper also never shows that its gating beats a simple reset. SAR and CoTTA are only run with their own losses, and periodic reset or restoring to the source weights is never tried on CAGA's base. The only same-base test, "always restore" in Fig. 6(b) (ACDC, one seed), ends only about 0.2 mIoU below full CAGA at round 150.
2. The reported gains are not cleanly attributable to CAGA's adaptation.
On ACDC, CAGA already starts round 0 at about 30 mIoU (Figs. 1a, 5a, 6a, 6b), even with restoration disabled (). That is about 6.5 of its 7.6-point gain over No-Adapt (23.34), and close to episodic MLMP (29.84). TENT and DeYO, at twice CAGA's learning rate, start at the No-Adapt level, while MGP and DAF start near 30 like CAGA (Fig. 5a). This suggests the methods use different prediction pipelines, with CAGA and some baselines using MLMP-style multi-template, multi-layer predictions that No-Adapt does not. If so, the 13.6% gain and the comparisons with TENT, DeYO and SPA mix the effect of the prediction pipeline with that of adaptation.
Against the closest reference, episodic MLMP, CAGA's margin is only 0.6–1.6 mIoU on five of eight benchmarks, and it loses on VOC20-C, even though MLMP was run with 1 step instead of its default 10. All results use a single seed. Three CAGA values in the figures also differ from the tables by 0.3–2.4 mIoU (Question 3).
Questions for authors
Clarifications that could change my rating
-
Using CAGA's own loss, prediction pipeline and learning rate, how does CAGA compare with periodic reset (RDumb), SAR-style recovery, restoring to the source at every step (CoTTA-style), PeTTA, and the adaptive reset of Lim et al. (ICLR 2026)? What does the gating add over these?
-
Which prediction pipeline (templates, multi-layer fusion) does each method use, including No-Adapt? Please report CAGA's pipeline with adaptation turned off, all baselines on the same pipeline, and episodic MLMP at its default 10 steps.
-
Fig. 9 reports 75.86 on VOC20-C and 23.92 on Cityscapes-C (Tables 5 and 4: 73.48 and 24.22), and the ACDC curve in Fig. 1(a) ends near 31.8 (Table 3: 30.95). Which runs do these come from? Please report the mean and standard deviation over at least 3 seeds for CAGA, episodic MLMP and the strongest baselines.
-
Do TENT, DeYO and DAF still collapse at CAGA's learning rate (, vs. for most baselines)? DAF was also run with batch size 1 and learning rate on ACDC, Cityscapes-C and VOC20-C, unlike its paper (batch size 8, ). It ends 28 mIoU below No-Adapt on VOC20-C, where its paper reports a gain of over 8 (with ViT-B/32). Please rerun DAF with its published settings. How were CAGA's hyperparameters chosen, given that most ablations use ACDC, which is also a test benchmark?
-
How early does the trigger fire before accuracy drops? How often does it fire on runs that never collapse, and how often is deep versus shallow restoration used?
-
Most methods barely degrade on closed-set SegFormer (A5). What makes open-vocabulary models collapse? Can you measure vision-language alignment drift after 150 rounds, or test on a vocabulary not seen during adaptation?
6 comments
Rebuttal Part 2/3
W2 / Q2. Pipelines and the gain from adaptation alone
The reviewer's reading of the round-0 gap is correct. TENT, SAR, DeYO, COME, PTTA, SPA, TRIBE and ViDA use the last layer with 1 prompt; MLMP-episodic, MGP, DAF and CAGA fuse 18 layers with 7 prompts (Appendix A3, Table 3). We added No-Adapt (MLMP inference): CAGA's exact pipeline without adaptation.
Correction to No-Adapt. While preparing this, we found that No-Adapt on the seven -C benchmarks had inadvertently been run with 7 prompts, whereas No-Adapt on ACDC and all single-prompt baselines use one. We reran it with a single prompt (0.7–2.0 mIoU lower); CAGA remains above it on all 8 benchmarks, and all tables and figures are updated.
= gain over No-Adapt (MLMP inference), i.e., from adaptation alone (Table 15):
| Benchmark | No-Adapt (MLMP inf.) | MLMP-episodic | CAGA | ||
|---|---|---|---|---|---|
| ACDC | 29.66 | 29.84 | +0.18 | 30.95 | +1.29 |
| Cityscapes-C | 23.20 | 23.40 | +0.20 | 24.22 | +1.02 |
| VOC20-C | 70.35 | 74.09 | +3.74 | 73.48 | +3.13 |
| VOC21-C | 44.41 | 45.44 | +1.03 | 50.01 | +5.60 |
| Context59-C | 24.10 | 24.78 | +0.68 | 26.92 | +2.82 |
| Context60-C | 22.38 | 23.02 | +0.64 | 24.57 | +2.19 |
| COCO-Stuff-C | 12.31 | 12.70 | +0.39 | 13.30 | +0.99 |
| COCO-Object-C | 20.46 | 21.00 | +0.54 | 21.73 | +1.27 |
With the pipeline effect removed, CAGA still improves on all 8 benchmarks, and its adaptation gain is 2–7× that of MLMP-episodic on 7 of 8. The average relative gain is 19.25% against the corrected No-Adapt and 7.7% from adaptation alone; Table 1 now includes No-Adapt (MLMP inference), and Table 15 reports both.
Baselines on the same pipeline. Adding the multi-layer, multi-prompt pipeline to the other baselines would turn them into different methods, but three comparisons on CAGA's pipeline exist: MGP and DAF already use it and fall below No-Adapt (MLMP inference) on 7/8 and 8/8 benchmarks; and CAGA's own base without restoration (DeYO's objective on this pipeline) degrades on 6 of 8 and collapses on ACDC (9.96) and Cityscapes-C. The pipeline raises the starting point but does not provide stability; the restoration target does.
MLMP-episodic with 10 steps (Table 31):
| Method | ACDC | City-C | VOC20-C | VOC21-C | Ctx59-C | Ctx60-C | Stuff-C | Object-C |
|---|---|---|---|---|---|---|---|---|
| MLMP-episodic, 1 step | 29.84 | 23.40 | 74.09 | 45.44 | 24.78 | 23.02 | 12.70 | 21.00 |
| MLMP-episodic, 10 steps | 30.57 | 24.04 | 78.42 | 45.89 | 25.75 | 24.00 | 13.02 | 21.58 |
| CAGA (1 step, R150) | 30.95 | 24.22 | 73.48 | 50.01 | 26.92 | 24.57 | 13.30 | 21.73 |
CAGA is higher on 7 of 8 while the 10-step variant needs 4.6× the FLOPs and 3.5× the GPU time; MLMP-episodic remains better on VOC20-C, and the paper reports both settings.
Q3. Fig. 9, Fig. 1(a), and seeds
- Fig. 9 (75.86 / 23.92) uses a lighter 5-corruption protocol (snow, frost, fog, brightness, contrast), not the 15 corruptions of Tables 4–5 of the submission (Tables 7–8 of the revision); the caption now states this.
- Fig. 1(a) (≈31.8) was a plotting error: the curve came from an earlier development run. It is regenerated from the final run (30.95); no table or reported number is affected.
- Seeds. The differences are not due to randomness: over three seeds on ACDC, CAGA's standard deviation (0.06) is an order of magnitude smaller than its margins over MLMP-episodic (1.13) and the reset variants (≥ 0.7). R150 mIoU (Table 18): CAGA 30.97 ± 0.06, MLMP-episodic 29.84 ± 0.00, SAR 25.31 ± 0.00, SPA 19.33 ± 0.45, DeYO 9.49 ± 0.06. (SAR on ACDC now uses the same argument set as on the other benchmarks, so 25.31 replaces the 25.67 of the submission.)
Rebuttal Part 3/3
Q4. Lower learning rate, DAF, and hyperparameter selection
Baselines at CAGA's learning rate (Fig. 18, Tables 28–29). A lower learning rate delays degradation but does not prevent it: on ACDC the peak of TENT moves from round 19 to 35 and that of DeYO from 30 to 59, after which both keep declining until R150. TENT still ends below No-Adapt on all 8 benchmarks (near zero on 7). DeYO is strong where it remains stable, but at either learning rate it ends below No-Adapt on ACDC (22.26 at ), Cityscapes-C (4.90) and COCO-Stuff-C, whereas CAGA is above No-Adapt on all 8 without per-dataset tuning.
DAF with its published settings (batch 8, lr , SGD; Table 30). Because DAF is trained with SGD rather than Adam, a learning rate of is not comparable for it, so we instead reran it with its published settings, as requested. The main experiments use batch size 1, the online setting we target, so DAF on ACDC, Cityscapes-C and VOC20-C used a linearly scaled lr of ; the other five benchmarks already used the published settings. Rerun with them, VOC20-C reaches 75.09 at round 5 (+7.7 over the single-layer No-Adapt), reproducing DAF's short-horizon gain, but ends at 59.19, below No-Adapt (MLMP inference, 70.35); the same drop appears on all five other benchmarks (e.g., VOC21-C 46.25 → 33.54 vs. 44.41). On ACDC and Cityscapes-C, batch 8 at exceeds GPU memory, so they stay at batch 1.
Hyperparameter selection. They were tuned once on ACDC and used unchanged on the other 7 benchmarks and all 14 backbones; ACDC is also a test benchmark, which the paper now states. The largest gains are on benchmarks never used for tuning, and halving and doubling every hyperparameter on VOC20-C keeps all 12 settings within 0.6 mIoU of the default (Table 32).
Q5. Trigger behaviour
CAGA is not designed as an early-warning detector. The trigger compares the gradient-norm slope with its own recent distribution (robust z-score, Eq. 7), so it is scale-free and fires in about 30% of windows on every stream; each trigger is a mild pull-back (restoration probability 0.01 per element and step). What adapts to the stream is the depth, chosen by (Table 16, Fig. 13): deep restorations make up 90% / 60% on ACDC / Cityscapes-C, where the base collapses without restoration (31% / 21% of its peak at R150), and 0–15% elsewhere. On VOC21-C and COCO-Object-C, which do not collapse (99% / 98% of peak), almost all restorations are shallow (deep share 0.14 / 0.00), and CAGA retains at least 98.1% of its own peak on every benchmark. The wording in the abstract and Secs. 1 and 3.2 no longer suggests an early-warning role.
Q6. Why open-vocabulary models collapse
An OVSS model classifies by comparing adapted visual features with fixed text embeddings, and all pixels share one adapted representation, so updates that favour a few classes pull the features toward their text embeddings and away from the rest. Both suggested analyses support this (Appendix A7):
| ACDC, R150 | mIoU | Class-avg. margin | Classes with negative margin |
|---|---|---|---|
| Source | 29.66 | +0.006 | 48% |
| CAGA | 30.95 | +0.007 | 47% |
| CAGA base, no restoration | 9.96 | −0.097 | 79% |
| TENT / DeYO () | 21.21 / 22.26 | −0.074 / −0.042 | 73% / 73% |
The margin is the cosine similarity to the ground-truth class embedding minus that to the closest wrong class. CAGA keeps the alignment at the source level, while the degrading methods misalign most classes even though their pixel-averaged margin rises (+0.056 / +0.059 vs. +0.014): a few dominant classes improve at the expense of all others.
Unseen vocabulary. Weights adapted for 150 rounds on VOC20-C and evaluated without further adaptation on Pascal-Context59-C (39 of 59 classes unseen, disjoint images): CAGA 24.63 / 45.85 / 13.71 (all / seen / unseen) vs. No-Adapt (MLMP inference) 24.10 / 44.98 / 13.28, while the base without restoration drops to 17.52 / 36.29 / 7.76. It keeps 92% of its peak on VOC20-C itself but loses 42% on the unseen classes, so long-horizon drift damages the open-vocabulary capability more than in-vocabulary accuracy reveals.
We are grateful for this review: the requested experiments have made the paper's claims more precise and better supported.
Rebuttal Part 1/3
We sincerely thank the reviewer for the careful and constructive review. The questions were precise enough that we could answer each with a new experiment; all results are in the revised paper (new content in blue).
W1 / Q1. Novelty and comparison with simple resets
Missing references. We thank the reviewer for pointing us to RDumb, PeTTA and Lim et al. (ICLR 2026); all three are now cited and discussed in Sec. 2. They establish long-horizon collapse and reset-based remedies for closed-set classification; our study shows that the phenomenon is more severe for open-vocabulary segmentation and that where the model is restored to decides whether adaptation gains survive.
Hk never triggers restoration in CAGA; it only selects the depth. PeTTA requires class-wise feature statistics of the source domain, which for CLIP-based OVSS (web-scale pre-training data) are not accessible, so it cannot be instantiated in our setting.
Same-base comparison (Table 2). We kept CAGA's loss, pipeline and learning rate fixed and replaced only its restoration rule: Periodic (full reset to the source every 1,000 steps, RDumb-style), CoTTA-style (per-step stochastic restoration to the source, p=0.01), Random → source (triggered at CAGA's firing rate), and the reset rule of Lim et al. with its regularizer.
No-Adapt (MLMP inference): no adaptation, same inference as CAGA (18 layers, 7 prompts).
CAGA is the best on all 8 benchmarks, by 0.7–5.4 mIoU over the strongest reset variant; on ACDC this holds for all three seeds (30.97±0.06 vs. 29.77–30.26 for Periodic, CoTTA-style and Random → source). Every source-based rule stays within 1.0 mIoU of No-Adapt (MLMP inference): it prevents collapse by giving up most of the adaptation. CAGA restores toward an adapted state verified online (low gradient norm, healthy Hk) and keeps a larger adaptation gain than every reset variant on all 8 benchmarks (e.g., +5.60 vs. at most +0.18 on VOC21-C).
Official RDumb and Lim et al. With their official code and hyperparameters under our protocol (only the two thresholds tied to 1,000 ImageNet classes rescaled), neither improves meaningfully over No-Adapt (peak / R150: RDumb 24.68 / 24.54 and Lim et al. 23.91 / 23.70 on ACDC vs. 23.34; 67.99 / 67.99 and 67.46 / 67.45 on VOC20-C vs. 67.39; Table 19). With batch size 1, Lim et al. resets about every 65 steps, so adaptation cannot accumulate.
Fig. 6(b). The always-restore variant removes only the trigger and still restores toward CAGA's selected target, which is why it ends close to CAGA. We agree that the trigger timing matters less than the restoration target; the contribution statement and Sec. 5.3 now say that the central components are the online selection of a reliable adapted anchor and the entropy-gated restoration depth.