acceptodds
Under review as a conference paper at ICLR 2027

Preventing Collapse in Long-Horizon Continual Test-Time Adaptation for Open-Vocabulary Semantic Segmentation

Abstract

Open-vocabulary semantic segmentation (OVSS) assigns pixel-level labels from free-form text classes but, like other dense prediction models, degrades under distribution shift. Continual test-time adaptation (CTTA) addresses non-stationary shifts by continuously adapting models on unlabeled test streams. However, most existing CTTA methods are designed for closed-set classifiers with fixed prediction heads, and thus do not well match to the open-vocabulary prediction structure. We stress-test CTTA for OVSS over 150 adaptation rounds, an order of magnitude longer than common evaluation protocols. At this horizon, we uncover an adaptation–stability trade-off across existing methods: those that achieve substantial adaptation gains eventually collapse, whereas those that remain stable provide limited improvement. Although episodic resetting prevents collapse, it sacrifices the ability to accumulate useful knowledge over time. To address this, we identify two label-free signals that track collapse progression from the perspectives of optimization dynamics and prediction distribution: gradient-norm trend and class-marginal entropy. Building on these signals, we propose CAGA (Collapse-Aware Gated Anchoring), which preserves beneficial adaptation while selectively restoring the model to a previously reliable state as collapse emerges. CAGA mitigates the adaptation–stability trade-off, consistently preventing long-horizon collapse while sustaining adaptation gains across 8 benchmarks and 14 OVSS backbones. The code will be released upon acceptance.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Test-Time Adaptation, Open-Vocabulary Semantic Segmentation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PRnBOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies continual test-time adaptation of training-free, CLIP-based open-vocabulary segmentation models over a long unlabeled test stream. Running 11 existing methods for 150 rounds on 8 benchmarks, the authors report a trade-off: methods that gain early eventually collapse to a few classes, while stable methods gain little.

The proposed CAGA adapts only the vision encoder's LayerNorm parameters with a filtered entropy loss. When the gradient norm trends upward, it partially restores the parameters to an earlier state: a recent checkpoint if predictions are still diverse, otherwise a stored reliable anchor.

Experiments use one main backbone (NACLIP ViT-L/14), 8 benchmarks and a single seed, plus an ACDC-only study over 14 backbone variants. CAGA is the only continual method that ends above the unadapted model on all 8 benchmarks, with an average relative gain of 13.6%.

Critical strengths
  1. A large long-horizon study: 11 baselines ×\times 8 benchmarks ×\times 150 rounds, with per-corruption results. It shows that many methods, including CTTA- and stability-oriented ones (ViDA, COME, PTTA, DAF), end below the unadapted model on all 8 benchmarks.
  2. CAGA is the only continual method above No-Adapt on all 8 benchmarks (worst-case gain +6.4%, worst-case peak retention 98.1%). Its restoration step should also be cheap, since it only stores LayerNorm snapshots.
Critical weaknesses

1. Limited novelty, and the gating is not shown to beat simple resets.

Long-horizon collapse and reset-based fixes are already known. RDumb (NeurIPS 2023) showed that most TTA methods eventually fall below a non-adapting model, and that periodic resets match or beat them. PeTTA (NeurIPS 2024) detects drift toward collapse on recurring test streams. Most directly, Lim, Hwang & Lee ("When and Where to Reset Matters for Long-Term Test-Time Adaptation", ICLR 2026) use the entropy of the batch-mean prediction to decide when to reset and how many layers to reset, which is close to CAGA's class-marginal-entropy gate. None of these is cited.

The paper also never shows that its gating beats a simple reset. SAR and CoTTA are only run with their own losses, and periodic reset or restoring to the source weights is never tried on CAGA's base. The only same-base test, "always restore" in Fig. 6(b) (ACDC, one seed), ends only about 0.2 mIoU below full CAGA at round 150.

2. The reported gains are not cleanly attributable to CAGA's adaptation.

On ACDC, CAGA already starts round 0 at about 30 mIoU (Figs. 1a, 5a, 6a, 6b), even with restoration disabled (ρ=0\rho = 0). That is about 6.5 of its 7.6-point gain over No-Adapt (23.34), and close to episodic MLMP (29.84). TENT and DeYO, at twice CAGA's learning rate, start at the No-Adapt level, while MGP and DAF start near 30 like CAGA (Fig. 5a). This suggests the methods use different prediction pipelines, with CAGA and some baselines using MLMP-style multi-template, multi-layer predictions that No-Adapt does not. If so, the 13.6% gain and the comparisons with TENT, DeYO and SPA mix the effect of the prediction pipeline with that of adaptation.

Against the closest reference, episodic MLMP, CAGA's margin is only 0.6–1.6 mIoU on five of eight benchmarks, and it loses on VOC20-C, even though MLMP was run with 1 step instead of its default 10. All results use a single seed. Three CAGA values in the figures also differ from the tables by 0.3–2.4 mIoU (Question 3).

Questions for authors

Clarifications that could change my rating

  1. Using CAGA's own loss, prediction pipeline and learning rate, how does CAGA compare with periodic reset (RDumb), SAR-style recovery, restoring to the source at every step (CoTTA-style), PeTTA, and the adaptive reset of Lim et al. (ICLR 2026)? What does the gating add over these?

  2. Which prediction pipeline (templates, multi-layer fusion) does each method use, including No-Adapt? Please report CAGA's pipeline with adaptation turned off, all baselines on the same pipeline, and episodic MLMP at its default 10 steps.

  3. Fig. 9 reports 75.86 on VOC20-C and 23.92 on Cityscapes-C (Tables 5 and 4: 73.48 and 24.22), and the ACDC curve in Fig. 1(a) ends near 31.8 (Table 3: 30.95). Which runs do these come from? Please report the mean and standard deviation over at least 3 seeds for CAGA, episodic MLMP and the strongest baselines.

  4. Do TENT, DeYO and DAF still collapse at CAGA's learning rate (5×10−65\times10^{-6}, vs. 10−510^{-5} for most baselines)? DAF was also run with batch size 1 and learning rate 1.25×10−41.25\times10^{-4} on ACDC, Cityscapes-C and VOC20-C, unlike its paper (batch size 8, 10−310^{-3}). It ends 28 mIoU below No-Adapt on VOC20-C, where its paper reports a gain of over 8 (with ViT-B/32). Please rerun DAF with its published settings. How were CAGA's hyperparameters chosen, given that most ablations use ACDC, which is also a test benchmark?

  5. How early does the trigger fire before accuracy drops? How often does it fire on runs that never collapse, and how often is deep versus shallow restoration used?

  6. Most methods barely degrade on closed-set SegFormer (A5). What makes open-vocabulary models collapse? Can you measure vision-language alignment drift after 150 rounds, or test on a vocabulary not seen during adaptation?

6 comments
AuthorsOpenReview ↗

Rebuttal Part 1/3

We sincerely thank the reviewer for the careful and constructive review. The questions were precise enough that we could answer each with a new experiment; all results are in the revised paper (new content in blue).

W1 / Q1. Novelty and comparison with simple resets

Missing references. We thank the reviewer for pointing us to RDumb, PeTTA and Lim et al. (ICLR 2026); all three are now cited and discussed in Sec. 2. They establish long-horizon collapse and reset-based remedies for closed-set classification; our study shows that the phenomenon is more severe for open-vocabulary segmentation and that where the model is restored to decides whether adaptation gains survive.

TriggerRestoration targetMeaning of "depth"
RDumbfixed periodsource (full reset)–
CoTTAevery step (stochastic)source–
Lim et al.prediction concentrationsource for selected layers, plus a regularizer toward pre-reset parametersnumber of layers reset
CAGAgradient-norm trendan earlier adapted state selected online (anchor or recent snapshot)how far back in time

HkH_k never triggers restoration in CAGA; it only selects the depth. PeTTA requires class-wise feature statistics of the source domain, which for CLIP-based OVSS (web-scale pre-training data) are not accessible, so it cannot be instantiated in our setting.

Same-base comparison (Table 2). We kept CAGA's loss, pipeline and learning rate fixed and replaced only its restoration rule: Periodic (full reset to the source every 1,000 steps, RDumb-style), CoTTA-style (per-step stochastic restoration to the source, p=0.01p=0.01), Random → source (triggered at CAGA's firing rate), and the reset rule of Lim et al. with its regularizer.

R150 mIoUNo-Adapt (MLMP inf.)PeriodicCoTTA-styleRandom → sourceLim et al.CAGA
ACDC29.6630.2729.7729.9430.6630.95
Cityscapes-C23.2023.5623.2823.3523.4024.22
VOC20-C70.3570.9370.4670.6070.6773.48
VOC21-C44.4144.5944.4444.4844.4350.01
Context59-C24.1024.2724.1424.1824.1126.92
Context60-C22.3822.5322.4122.4422.3924.57
COCO-Stuff-C12.3112.4212.3312.3512.3113.30
COCO-Object-C20.4620.5220.4720.4920.4621.73

No-Adapt (MLMP inference): no adaptation, same inference as CAGA (18 layers, 7 prompts).

CAGA is the best on all 8 benchmarks, by 0.7–5.4 mIoU over the strongest reset variant; on ACDC this holds for all three seeds (30.97±0.0630.97 \pm 0.06 vs. 29.77–30.26 for Periodic, CoTTA-style and Random → source). Every source-based rule stays within 1.0 mIoU of No-Adapt (MLMP inference): it prevents collapse by giving up most of the adaptation. CAGA restores toward an adapted state verified online (low gradient norm, healthy HkH_k) and keeps a larger adaptation gain than every reset variant on all 8 benchmarks (e.g., +5.60 vs. at most +0.18 on VOC21-C).

Official RDumb and Lim et al. With their official code and hyperparameters under our protocol (only the two thresholds tied to 1,000 ImageNet classes rescaled), neither improves meaningfully over No-Adapt (peak / R150: RDumb 24.68 / 24.54 and Lim et al. 23.91 / 23.70 on ACDC vs. 23.34; 67.99 / 67.99 and 67.46 / 67.45 on VOC20-C vs. 67.39; Table 19). With batch size 1, Lim et al. resets about every 65 steps, so adaptation cannot accumulate.

Fig. 6(b). The always-restore variant removes only the trigger and still restores toward CAGA's selected target, which is why it ends close to CAGA. We agree that the trigger timing matters less than the restoration target; the contribution statement and Sec. 5.3 now say that the central components are the online selection of a reliable adapted anchor and the entropy-gated restoration depth.

AuthorsOpenReview ↗

Rebuttal Part 2/3

W2 / Q2. Pipelines and the gain from adaptation alone

The reviewer's reading of the round-0 gap is correct. TENT, SAR, DeYO, COME, PTTA, SPA, TRIBE and ViDA use the last layer with 1 prompt; MLMP-episodic, MGP, DAF and CAGA fuse 18 layers with 7 prompts (Appendix A3, Table 3). We added No-Adapt (MLMP inference): CAGA's exact pipeline without adaptation.

Correction to No-Adapt. While preparing this, we found that No-Adapt on the seven -C benchmarks had inadvertently been run with 7 prompts, whereas No-Adapt on ACDC and all single-prompt baselines use one. We reran it with a single prompt (0.7–2.0 mIoU lower); CAGA remains above it on all 8 benchmarks, and all tables and figures are updated.

Δ\Delta = gain over No-Adapt (MLMP inference), i.e., from adaptation alone (Table 15):

BenchmarkNo-Adapt (MLMP inf.)MLMP-episodicΔ\DeltaCAGAΔ\Delta
ACDC29.6629.84+0.1830.95+1.29
Cityscapes-C23.2023.40+0.2024.22+1.02
VOC20-C70.3574.09+3.7473.48+3.13
VOC21-C44.4145.44+1.0350.01+5.60
Context59-C24.1024.78+0.6826.92+2.82
Context60-C22.3823.02+0.6424.57+2.19
COCO-Stuff-C12.3112.70+0.3913.30+0.99
COCO-Object-C20.4621.00+0.5421.73+1.27

With the pipeline effect removed, CAGA still improves on all 8 benchmarks, and its adaptation gain is 2–7× that of MLMP-episodic on 7 of 8. The average relative gain is 19.25% against the corrected No-Adapt and 7.7% from adaptation alone; Table 1 now includes No-Adapt (MLMP inference), and Table 15 reports both.

Baselines on the same pipeline. Adding the multi-layer, multi-prompt pipeline to the other baselines would turn them into different methods, but three comparisons on CAGA's pipeline exist: MGP and DAF already use it and fall below No-Adapt (MLMP inference) on 7/8 and 8/8 benchmarks; and CAGA's own base without restoration (DeYO's objective on this pipeline) degrades on 6 of 8 and collapses on ACDC (9.96) and Cityscapes-C. The pipeline raises the starting point but does not provide stability; the restoration target does.

MLMP-episodic with 10 steps (Table 31):

MethodACDCCity-CVOC20-CVOC21-CCtx59-CCtx60-CStuff-CObject-C
MLMP-episodic, 1 step29.8423.4074.0945.4424.7823.0212.7021.00
MLMP-episodic, 10 steps30.5724.0478.4245.8925.7524.0013.0221.58
CAGA (1 step, R150)30.9524.2273.4850.0126.9224.5713.3021.73

CAGA is higher on 7 of 8 while the 10-step variant needs 4.6× the FLOPs and 3.5× the GPU time; MLMP-episodic remains better on VOC20-C, and the paper reports both settings.

Q3. Fig. 9, Fig. 1(a), and seeds

  • Fig. 9 (75.86 / 23.92) uses a lighter 5-corruption protocol (snow, frost, fog, brightness, contrast), not the 15 corruptions of Tables 4–5 of the submission (Tables 7–8 of the revision); the caption now states this.
  • Fig. 1(a) (≈31.8) was a plotting error: the curve came from an earlier development run. It is regenerated from the final run (30.95); no table or reported number is affected.
  • Seeds. The differences are not due to randomness: over three seeds on ACDC, CAGA's standard deviation (0.06) is an order of magnitude smaller than its margins over MLMP-episodic (1.13) and the reset variants (≥ 0.7). R150 mIoU (Table 18): CAGA 30.97 ± 0.06, MLMP-episodic 29.84 ± 0.00, SAR 25.31 ± 0.00, SPA 19.33 ± 0.45, DeYO 9.49 ± 0.06. (SAR on ACDC now uses the same argument set as on the other benchmarks, so 25.31 replaces the 25.67 of the submission.)
AuthorsOpenReview ↗

Rebuttal Part 3/3

Q4. Lower learning rate, DAF, and hyperparameter selection

Baselines at CAGA's learning rate (Fig. 18, Tables 28–29). A lower learning rate delays degradation but does not prevent it: on ACDC the peak of TENT moves from round 19 to 35 and that of DeYO from 30 to 59, after which both keep declining until R150. TENT still ends below No-Adapt on all 8 benchmarks (near zero on 7). DeYO is strong where it remains stable, but at either learning rate it ends below No-Adapt on ACDC (22.26 at 5×10−65\times10^{-6}), Cityscapes-C (4.90) and COCO-Stuff-C, whereas CAGA is above No-Adapt on all 8 without per-dataset tuning.

DAF with its published settings (batch 8, lr 10−310^{-3}, SGD; Table 30). Because DAF is trained with SGD rather than Adam, a learning rate of 5×10−65\times10^{-6} is not comparable for it, so we instead reran it with its published settings, as requested. The main experiments use batch size 1, the online setting we target, so DAF on ACDC, Cityscapes-C and VOC20-C used a linearly scaled lr of 1.25×10−41.25\times10^{-4}; the other five benchmarks already used the published settings. Rerun with them, VOC20-C reaches 75.09 at round 5 (+7.7 over the single-layer No-Adapt), reproducing DAF's short-horizon gain, but ends at 59.19, below No-Adapt (MLMP inference, 70.35); the same drop appears on all five other benchmarks (e.g., VOC21-C 46.25 → 33.54 vs. 44.41). On ACDC and Cityscapes-C, batch 8 at 1120×5601120\times560 exceeds GPU memory, so they stay at batch 1.

Hyperparameter selection. They were tuned once on ACDC and used unchanged on the other 7 benchmarks and all 14 backbones; ACDC is also a test benchmark, which the paper now states. The largest gains are on benchmarks never used for tuning, and halving and doubling every hyperparameter on VOC20-C keeps all 12 settings within 0.6 mIoU of the default (Table 32).

Q5. Trigger behaviour

CAGA is not designed as an early-warning detector. The trigger compares the gradient-norm slope with its own recent distribution (robust z-score, Eq. 7), so it is scale-free and fires in about 30% of windows on every stream; each trigger is a mild pull-back (restoration probability 0.01 per element and step). What adapts to the stream is the depth, chosen by HkH_k (Table 16, Fig. 13): deep restorations make up 90% / 60% on ACDC / Cityscapes-C, where the base collapses without restoration (31% / 21% of its peak at R150), and 0–15% elsewhere. On VOC21-C and COCO-Object-C, which do not collapse (99% / 98% of peak), almost all restorations are shallow (deep share 0.14 / 0.00), and CAGA retains at least 98.1% of its own peak on every benchmark. The wording in the abstract and Secs. 1 and 3.2 no longer suggests an early-warning role.

Q6. Why open-vocabulary models collapse

An OVSS model classifies by comparing adapted visual features with fixed text embeddings, and all pixels share one adapted representation, so updates that favour a few classes pull the features toward their text embeddings and away from the rest. Both suggested analyses support this (Appendix A7):

ACDC, R150mIoUClass-avg. marginClasses with negative margin
Source29.66+0.00648%
CAGA30.95+0.00747%
CAGA base, no restoration9.96−0.09779%
TENT / DeYO (5×10−65\times10^{-6})21.21 / 22.26−0.074 / −0.04273% / 73%

The margin is the cosine similarity to the ground-truth class embedding minus that to the closest wrong class. CAGA keeps the alignment at the source level, while the degrading methods misalign most classes even though their pixel-averaged margin rises (+0.056 / +0.059 vs. +0.014): a few dominant classes improve at the expense of all others.

Unseen vocabulary. Weights adapted for 150 rounds on VOC20-C and evaluated without further adaptation on Pascal-Context59-C (39 of 59 classes unseen, disjoint images): CAGA 24.63 / 45.85 / 13.71 (all / seen / unseen) vs. No-Adapt (MLMP inference) 24.10 / 44.98 / 13.28, while the base without restoration drops to 17.52 / 36.29 / 7.76. It keeps 92% of its peak on VOC20-C itself but loses 42% on the unseen classes, so long-horizon drift damages the open-vocabulary capability more than in-vocabulary accuracy reveals.

We are grateful for this review: the requested experiments have made the paper's claims more precise and better supported.

Reviewer_hgdfOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies continual test-time adaptation for open-vocabulary semantic segmentation and identifies an adaptation–stability trade-off: methods with substantial short-term gains often collapse under prolonged adaptation, whereas stable methods provide only limited improvements. The authors identify two complementary label-free signals for tracking collapse progression: gradient-norm trend and class-marginal entropy. They propose CAGA, a collapse-aware restoration strategy that selectively restores the model to previously reliable states while preserving beneficial adaptation. Across 8 benchmarks and 14 OVSS backbones, CAGA maintains adaptation gains and prevents collapse throughout 150 adaptation rounds.

Critical strengths
  1. The paper addresses a meaningful problem. Long-horizon evaluation reveals issues that short-term results can obscure. Some methods improve substantially at first but deteriorate severely with continued updates, while others remain stable but offer limited gains. Comparing these behaviors under the same evaluation protocol is more informative than reporting a single performance metric at the end of adaptation.
  2. The experiments are fairly comprehensive. Beyond the main results on eight benchmarks, the authors examine image resampling, held-out corruptions, randomized domain ordering, and domain-set transitions during adaptation. CAGA remains stable under these settings, providing experimental support for the main results.
  3. The method has relatively low deployment requirements. CAGA requires neither target labels nor additional pretrained models. Parameter restoration is restricted to the visual encoder’s LayerNorm parameters, and the same restoration configuration is used across datasets.
Critical weaknesses
  1. The comparisons with simpler restoration strategies are insufficient. Removing restoration causes the model to collapse, which demonstrates that restoration is important. However, in Figure 6(b), always restoring and using either signal alone also produce largely stable behavior, with relatively small gaps from the full method. I therefore cannot determine how much of the benefit comes from parameter restoration itself and how much comes from the proposed gating, dynamic anchor, and rollback-depth design. Comparisons with simpler alternatives, such as periodic restoration and restoration toward the source parameters, are needed under the same base adaptation pipeline.
  2. The two signals do not necessarily indicate model degradation. An increase in gradient norm may simply reflect more difficult samples, while a decrease in class-marginal entropy may result from fewer classes being present in the current scenes. The paper shows correlations between these signals and performance changes but does not sufficiently analyze false positives and false negatives. Moreover, entropy only determines restoration depth and cannot independently trigger restoration. This leaves a gap between the algorithm and the explanation that entropy complements the gradient signal when gradients vanish.
  3. The evidence for continual knowledge accumulation is weaker than the evidence for performance preservation. The current experiments show that CAGA can maintain a good state over a long horizon. However, it is also possible that most useful adaptation occurs early and that subsequent restoration mainly prevents degradation. In the domain-transition experiment, comparisons with a model frozen before the transition and a model re-adapted from the source parameters would help establish how much benefit comes from retaining updates.
  4. Some small performance differences require repeated experiments. All experiments use only seed 0, while parameter restoration involves randomness. In particular, the small gaps between several ablation variants are insufficient to establish that the full method consistently performs better. The main baselines should also be evaluated at lower learning rates to rule out overly aggressive updates as a cause of long-horizon collapse.
Questions for authors
  1. With the base adaptation pipeline held fixed, does the full CAGA method consistently outperform periodic restoration and always restoring toward the source parameters? How much do the dynamic anchor and adaptive rollback depth contribute individually?
  2. When class-marginal entropy decreases because of a genuine change in class composition, how does CAGA avoid unnecessary restoration? If the model has already reached a low-entropy saturated state and the gradient trend is no longer increasing, can the current mechanism still trigger restoration?
  3. Could the authors provide results across multiple random seeds for the key ablations, along with results for the main baselines at lower learning rates, to verify that the observed differences are not due to randomness or update strength?
6 comments
AuthorsOpenReview ↗

Rebuttal Part 1/3

We thank the reviewer for the careful and constructive review. We ran every experiment the reviewer requested, and the results have changed how we present the paper:

  • W1/Q1. With the base pipeline fixed, CAGA outperforms periodic, always-on (CoTTA-style) and random restoration toward the source on all 8 benchmarks; the gain comes mainly from where the model is restored to (the dynamic anchor), and we now say so explicitly.
  • W2/Q2. We separate the data-driven and degradation-driven parts of both signals and correct the inaccurate sentence the reviewer identified.
  • W3. The domain-transition comparison supports the reviewer's reading, and we have narrowed the claim.
  • W4/Q3. Three seeds and lower-learning-rate baselines leave the conclusions unchanged.

All new results are in the revised paper (Table 2 in Sec. 5.3 and Appendices A5–A7).

W1/Q1. Simpler restoration strategies and the contribution of each component

Clarification of Fig. 6(b). Always restore removes only the trigger and still uses CAGA's own target selection (anchor for deep, recent snapshot for shallow restoration, depth chosen by HkH_k); the single-signal variants change both when to restore and how the adapted target is selected. Every variant therefore restores toward an earlier adapted state, never toward the source, which is why they look similar. The reviewer is right that this cannot separate restoration itself from our design, so we ran the requested comparison.

Restoration toward the source. Keeping the base pipeline fixed (loss, reliability filter, LN parameters, learning rate, multi-layer multi-prompt inference), we replaced CAGA's restoration with periodic reset every 1,000 steps (RDumb-style), restoration toward the source at every step, i.e., always restoring (CoTTA-style, p=0.01p=0.01 per parameter), and a random trigger at CAGA's measured firing rate.

Benchmark (R150)No-Adapt (MLMP inf.)Periodic → sourceCoTTA-style → sourceRandom → sourceCAGA
ACDC29.6630.2729.7729.9430.95
Cityscapes-C23.2023.5623.2823.3524.22
VOC20-C70.3570.9370.4670.6073.48
VOC21-C44.4144.5944.4444.4850.01
Context59-C24.1024.2724.1424.1826.92
Context60-C22.3822.5322.4122.4424.57
COCO-Stuff-C12.3112.4212.3312.3513.30
COCO-Object-C20.4620.5220.4720.4921.73

No-Adapt (MLMP inf.): no adaptation, same inference as CAGA (18 layers, 7 prompts).

CAGA is the best on all 8 benchmarks. All three source-based rules stay within about 0.6 mIoU of the non-adapted model with the same inference: they are stable only because they discard almost all of the adaptation, whereas CAGA keeps up to +5.6 mIoU (VOC21-C). This is the adaptation–stability trade-off of the paper, reproduced inside our own pipeline.

Individual contributions (Table 17):

VariantTriggerTargetDepthACDCVOC20-C
CAGAgateanchor / snapshotadaptive30.9573.48
Random → anchorrandomanchor / snapshotadaptive30.9673.48
Random → sourcerandomsource—29.9470.60
Deep onlygateanchoralways deep30.9871.69
Shallow onlygatesnapshotalways shallow32.1973.03
  • Dynamic anchor: the largest contribution. With a random trigger, changing only the target from the source to the anchor gives +1.02 on ACDC and +2.88 on VOC20-C.
  • Trigger: sets how often, not exactly when. A random trigger at the rate measured from CAGA's gate matches CAGA, so the exact timing is not critical; the gate provides the rate itself, with one configuration for all datasets. Never restoring collapses (Fig. 6(a), ρ=0\rho=0).
  • Depth: dataset-dependent. Neither fixed depth is best on both benchmarks: shallow-only is stronger on ACDC, deep-only loses 1.79 on VOC20-C. Which one is preferable cannot be known without labels, and the HkH_k-based choice avoids committing to one.

Revision. The text now states that the main benefit comes from restoring toward an adapted anchor instead of the source, with the gate setting how often restoration is applied and HkH_k setting its depth.

AuthorsOpenReview ↗

Rebuttal Part 2/3

W2/Q2. Do the two signals indicate degradation?

We agree that both signals also respond to the data (harder samples, fewer classes in a scene). The new analysis below shows that each signal has a data component and a degradation component, and that the data component is no longer the main driver once the model is allowed to drift.

Separating data from degradation. On the -C benchmarks every round contains the same images in the same order, so the position in the stream is a direct proxy for the sample. With CAGA, position explains 59–97% of the triggers. Without restoration, this falls to 13–36% on 6 of the 7 benchmarks and to 78% on Cityscapes-C. Sample difficulty is the main driver only while the model is healthy; once it drifts, the gate fires for a different reason.

Gradient norm. We removed restoration (same base; the gate is computed but never applied) and measured the fraction of windows that would trigger:

Rounds (ACDC, no restoration)R1–10R11–20R21–60R61–80R81–150
mIoU30.9331.8925.7218.8311.67
Would-trigger rate0.170.310.390.660.31
Gradient norm5.95.57.015.523.3

The signal thus contains both a difficulty component and a degradation component. CAGA suppresses the degradation, so what remains under CAGA is mostly difficulty: triggers recur on the same corruption types (mainly noise and blur) in every round.

False positives and false negatives. A false positive is cheap by design. During a triggered window each LN element is restored with probability 0.01 per step, and the target is an earlier adapted state, not the source, so a false trigger is a small partial pull-back and never a reset. This is consistent with the Q1 result that the outcome is insensitive to the exact trigger timing. For false negatives, the table shows that the gate keeps firing through every phase of the degradation, including the fully collapsed one.

Q2(a): class composition. Entropy cannot trigger restoration, so a genuine change in class composition alone never causes one; it can only affect the depth after the gradient gate has fired. Decomposing the variation of HkH_k into class composition, corruption type and round on the same fixed streams: without restoration, the round (i.e., drift) explains 76–99% of the variation and class composition at most 9%. Class composition becomes visible under CAGA only because drift has been removed (Appendix A7, Table 20).

Q2(b): low-entropy saturated state. Yes, restoration is still triggered. The trigger compares the current gradient-norm slope with its own recent distribution (the robust z-score of Eq. 7), so it does not require the trend to keep increasing. In the collapsed phase above (R81–150, mIoU 11.67), 31% of windows still trigger. Since HkH_k is then far below γHkmax⁡\gamma H_k^{\max}, each of them is a deep restoration toward the anchor, and the anchor cannot have been overwritten by a collapsed state because anchor updates require Hk≥γaHkmax⁡H_k \ge \gamma_a H_k^{\max}.

The gap between the algorithm and the explanation. The reviewer is right, and we thank the reviewer for catching this: the sentence stating that entropy complements the gradient signal when gradients vanish is inaccurate and has been rewritten. The derivation in Appendix A1 shows that the gradient norm alone has a blind spot in the one-hot limit; this is the reason the restoration depth is decided by HkH_k and not by the gradient. In practice the limit is not reached in any of our runs: even in the fully collapsed run the gradient norm rises from 5.9 to 23.3 instead of vanishing. In the exact one-hot limit the trigger would also lose information, and we now state this as a limitation.

AuthorsOpenReview ↗

Rebuttal Part 3/3

W3. Knowledge accumulation vs. performance preservation

On VOC20-C the stream contains five corruptions for 75 rounds and then switches to five different, unseen corruptions for 75 rounds (Table 14). On the new corruptions:

Model on the 5 new corruptionsFirst roundMean over 75 roundsLast round
No-Adapt (MLMP inf.)66.0966.0966.09
Frozen at the switch (R75)68.4468.4468.44
Reset to source, then CAGA66.2868.9969.30
CAGA, continued68.4868.4668.44
  • What CAGA learned transfers. On unseen corruptions it is +2.35 above No-Adapt (MLMP inference) from the first image; re-adapting from the source needs 12 rounds to catch up and requires knowing when the domain changed.
  • After the switch, CAGA retains but does not improve further, while re-adapting from the source ends 0.86 higher. The reason is identifiable: the anchor is set in the first phase and restoration keeps pulling the model toward it. Started from the source, CAGA itself gains +3.21 on these domains, so the limitation lies in the anchor not being refreshed.

We therefore agree with the reviewer: most useful adaptation happens early, and restoration mainly preserves it. The paper now describes CAGA as retaining, rather than continually accumulating, its gains, lists the limited plasticity after a domain change as a limitation, and notes anchor refresh as future work. The central claim, that CAGA holds its gain over a long horizon where other continual methods lose it, is unaffected.

W4/Q3. Multiple seeds and lower learning rates

Seeds (Table 18; randomness from the restoration mask and patch shuffle, fixed data order):

ACDC, R150Seed 0Seed 1Seed 2Mean ± std
CAGA30.9530.9331.0430.97 ± 0.06
MLMP-episodic29.8429.8429.8429.84 ± 0.00
Periodic → source30.2730.2730.2530.26 ± 0.01
CoTTA-style → source29.7729.7729.7729.77 ± 0.00
Random → source29.9430.1930.0930.07 ± 0.13

The gap to the best source-based variant (0.71) is more than ten times CAGA's standard deviation, and every seed of CAGA is above every seed of every other method. We agree that the gaps in Fig. 6(b) (0.2–0.4 mIoU) are too small to show that the full method is consistently better; those variants all restore toward an adapted state, and the revised paper no longer draws conclusions from them.

Lower learning rates (Fig. 18, Tables 28–29). At CAGA's 5×10−65\times10^{-6}, R150 mIoU:

ACDCCity-CVOC20-CVOC21-CCtx59-CCtx60-CStuff-CObject-C
TENT21.211.190.290.190.010.010.041.39
DeYO22.264.9068.9051.8128.7025.7610.7721.55
CAGA30.9524.2273.4850.0126.9224.5713.3021.73

A lower learning rate delays collapse but does not prevent it: on ACDC the peak of TENT moves from round 19 to 35 and that of DeYO from 30 to 59, after which both decline until R150 and end below No-Adapt (23.34). DeYO is higher than CAGA on three benchmarks here (four at 10−510^{-5}), as in Table 1, but it collapses on ACDC and Cityscapes-C and ends below No-Adapt on COCO-Stuff-C, and nothing observable at test time tells in advance which outcome will occur. Among the remaining baselines, the one exception is SAR on ACDC (31.84 vs. 30.95 at R150), whose curve oscillates between 23.6 and 33.3 and stays below CAGA on average (30.69 vs. 30.98). CAGA itself reaches 31.03 on ACDC at 10−510^{-5}.

Note on No-Adapt. The No-Adapt baseline on the seven -C benchmarks had inadvertently been run with 7 prompt templates instead of one; we reran it with a single prompt (0.7–2.0 mIoU lower), CAGA remains above it on all 8 benchmarks, and all tables and figures are updated.

We thank the reviewer again; these experiments made the paper more precise about where CAGA's benefit comes from.

Reviewer_Zda5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies continual test-time adaptation (CTTA) for open-vocabulary semantic segmentation (OVSS) under a much longer horizon than usual (150 rounds over the same test stream). The authors first show that most existing TTA/CTTA methods either collapse over this horizon (often into near single-class predictions) or stay stable but give only small gains, which they call an adaptation-stability trade-off. They then propose two label-free signals to track collapse: the trend (slope) of the window-averaged gradient norm, and the class-marginal entropy H(pˉ)H(\bar{\mathbf{p}}). Based on these signals, the proposed method CAGA uses a reliability-filtered entropy minimization as the base adaptation, and adds a restoration mechanism. When the normalized gradient-trend score exceeds a threshold, parameters are stochastically restored (CoTTA-style, with probability ρ\rho) either toward a "permanent anchor" (deep restoration, when the marginal entropy has dropped) or toward a recent snapshot (shallow restoration). Experiments on 8 benchmarks and 14 CLIP-based OVSS backbones show that CAGA stays above the non-adapted model on all benchmarks at round 150 and keeps about 99% of its peak performance.

Critical strengths
  1. The problem is relevant and I think the long-horizon observation itself is useful for the community. Figure 1 and Figure 7 make it quite clear that many methods which look good in short evaluations eventually break in the OVSS setting, and the qualitative examples (Figure 8) showing the number of predicted classes help to understand what "collapse" actually looks like.
  2. The experimental coverage is broad: 11 baselines including recent ones (MGP, DAF), 8 benchmarks, and 14 backbones. The same hyperparameters are used across all datasets without per-dataset tuning, which is a good practice.
  3. The method is self-contained. It does not need labels, source data, or an auxiliary pretrained model, and the two monitoring signals are cheap to compute.
  4. CAGA is the only continual method that stays above No-Adapt on all 8 benchmarks at round 150, and its worst-case relative gain is positive (+6.40%). Even if it is not the best on every dataset, this consistency is a meaningful property for deployment.
  5. The paper includes several sanity checks that I appreciated, such as held-out corruptions (Fig. 6d), resampling the images every round (Fig. 6e), randomized domain order, an abrupt domain switch, and the closed-set SegFormer result, which the authors report honestly even though CAGA gives no gain there.
Critical weaknesses

1. Does CAGA keep adapting, or does it just stay near an early safe state?

Restoration pulls the model back, but the base objective is unchanged afterwards, so the model would drift in the same direction again. The paper does not explain how re-entering collapse is prevented. My concern is that CAGA effectively stops adapting after an early peak and stays anchored there. The CAGA curves (Fig. 1, 5, 6d) rise in the first rounds and then stay flat. Also, the gain over MLMP-episodic, which accumulates nothing, is only about 1.4 mIoU on average across the 8 benchmarks (and negative on VOC20-C). This seems small given that "knowledge accumulation" is the main motivation against episodic adaptation.

Please show evidence that adaptation continues, such as the trigger rate over time, the deep/shallow restoration ratio, when the anchor was last updated, and ∥θ−θA∥\|\theta - \theta_A\| over the stream.

2. The long-horizon setting is artificial, and there is no clear comparison under standard protocols.

The 150-round stream repeats the same images and the same fixed domain cycle (e.g., fog → night → rain → snow on ACDC, and the same 15 corruptions on the -C benchmarks) 150 times. This is closer to an artificially augmented loop than to a realistic long deployment, where new images and unseen conditions keep arriving. The randomized-order experiment (Fig. 9) only shuffles the order, so it still cycles over the same set of domains. Because of this, it is hard to tell whether the baseline collapse (and CAGA's advantage) would also appear in a more realistic stream. It is also unclear how much CAGA gives up in the short-horizon regime that prior work evaluates (e.g., 3-5 passes in DAF). From the curves, several baselines are above CAGA in early rounds, and Appendix A4 also notes that strongly adapting methods reach higher peaks before collapsing.

Please report results for all methods at standard horizons (e.g., rounds 1, 5, and 10). It would also help to evaluate CAGA and the baselines on a less repetitive stream, for example one where new images are used every round (as in Fig. 6e, but for all methods) and the domains do not simply cycle, such as a non-repeating or gradually changing domain sequence.

3. Many hyperparameters with limited sensitivity analysis.

The restoration mechanism introduces WW, LL, τ\tau, γ\gamma, γa\gamma_a, and ρ\rho. In addition, the buffer size ∣B∣|\mathcal{B}| and the window for computing uku_k are not specified. However, γ\gamma (which decides deep vs. shallow restoration) and LL (which controls trigger speed) are not analyzed at all. The tested ranges for the other parameters are narrow (e.g., γa∈{0.90,0.95,0.98}\gamma_a \in \{0.90, 0.95, 0.98\}), and everything is tested on ACDC only with a single seed.

Please add a sensitivity analysis for γ\gamma, LL, and ∣B∣|\mathcal{B}|, use wider ranges, and include at least one more dataset (e.g., Cityscapes-C).

Questions for authors

Questions

  1. Please see the requests in Weaknesses 1-3, i.e., adaptation dynamics, short-horizon and less repetitive streams, and the missing hyperparameter.
  2. Most baselines use a learning rate of 1×10−51 \times 10^{-5}, while CAGA uses 5×10−65 \times 10^{-6}. How do the baselines behave at 5×10−65 \times 10^{-6}? Also, why was MLMP-episodic run with 1 step instead of its default 10 steps, and how does it perform with 10 steps?
  3. Are all results from a single seed? Some variance across seeds would help, especially for the small differences in the ablations.

Overall Assessment

I gave Weak Accept, but my position is closer to borderline accept. The long-horizon analysis is useful, and CAGA's consistency across benchmarks is a clear strength. My final decision will mainly depend on the rebuttal to Weaknesses 1 and 2.

6 comments
AuthorsOpenReview ↗

Rebuttal Part 1/3

We thank the reviewer for the careful and constructive review. We ran the experiments requested in W1–W3 and Q2–Q3; all new figures and tables are in the revised appendix (Appendices A6–A8). One correction to the submission: the No-Adapt baseline on the seven -C benchmarks had been run with 7 prompt templates instead of one; all No-Adapt numbers below use the corrected single-prompt setting (note at the end).

W1. Does CAGA keep adapting, or does it stay near an early safe state?

Short answer: CAGA never stops adapting. Collapse is prevented by a feedback loop that keeps firing for the whole stream, not by a one-time rollback.

(a) Why the model does not re-enter collapse. The objective is unchanged after a restoration, but the trajectory that follows is not a replay of the one that was undone:

  • Different data. Updates are strongly domain- and image-specific (see (c)). Collapse is the slow accumulation of a small coherent component μk\mu_k over many windows (Eq. (2)); restoration removes it while the stream moves on.
  • Stochastic restoration. Each parameter is restored with probability ρ\rho, so every restoration lands at a different point between the current state and the target.
  • Reliable states are kept. If the trend rises again, the gate fires again; if the model instead reaches a lower gradient norm with healthy marginal entropy, that state becomes the new anchor. This happens 3–24 times per benchmark, as late as R148 (table below).

The new control isolates the effect. The same base objective without restoration (Fig. 12) degrades on 6 of 8 benchmarks and ends below No-Adapt (MLMP inference), i.e., the non-adapted model with CAGA's inference pipeline (18 layers, 7 prompts), on each of them, whereas CAGA retains at least 98.1% of its own peak on all 8.

BenchmarkNo-Adapt (MLMP inf.)No restoration: peak → R150CAGA R150Trigger rate R1–75 / R76–150Deep shareLast anchor update
ACDC29.6632.05 → 9.9630.950.29 / 0.300.90R8
Cityscapes-C23.2024.56 → 5.2024.220.33 / 0.330.60R8
VOC20-C70.3573.81 → 67.8373.480.30 / 0.290.15R18
Context59-C24.1027.66 → 23.9726.920.36 / 0.370.15R102
Context60-C22.3825.19 → 22.0724.570.37 / 0.360.12R33
COCO-Stuff-C12.3113.61 → 10.0913.300.34 / 0.340.15R148
VOC21-C44.41— → 52.1550.010.31 / 0.300.14R43
COCO-Object-C20.46— → 22.8021.730.31 / 0.300.00R28

On VOC21-C and COCO-Object-C the unrestored objective does not collapse, and CAGA gives up 2.1 and 1.1 mIoU there. This is the price of a safeguard: without labels one cannot know in advance which streams will collapse, and the same configuration recovers 2.5–21 mIoU on the other six.

(b) Requested statistics (Fig. 13). The gate fires in about 30% of windows, at the same rate in both halves of the stream, so restoration neither fades out nor saturates. Deep restorations make up 90% / 60% on ACDC / Cityscapes-C, where the unrestored objective collapses, and 0–15% elsewhere. HkH_k decreases mainly during the first rounds and then changes only gradually; the sharp class concentration that precedes collapse (Fig. 3) does not occur.

AuthorsOpenReview ↗

Rebuttal Part 2/3

(c) Evidence that adaptation continues.

  • Free to adapt most of the time. No restoration is applied in about 70% of windows; in triggered windows each LayerNorm element is restored with probability 0.01 per step, toward an earlier adapted state. The learning rate is never decayed and no parameter is frozen.
  • Late updates are domain-specific. Logging every LayerNorm update on ACDC for three rounds after R150, the domain of the test image (fog / night / rain / snow) can be identified from a single update direction in 92% of cases (chance: 25%; Table 21).
  • Bounded equilibrium, not a fixed point. The LayerNorm parameters stay within 0.4–1.2% of the source norm and keep fluctuating, while without restoration they end 5–17× further away (Fig. 15).

(d) MLMP-episodic. The paper uses 1 step per sample so that every method performs one backward pass per test image. Measured against No-Adapt (MLMP inference), which isolates the gain from adaptation, CAGA's gain is 2–7× that of MLMP-episodic on 7 of 8 benchmarks (e.g., +1.29 vs. +0.18 on ACDC, +5.60 vs. +1.03 on VOC21-C; Table 15); VOC20-C is the exception. With MLMP's default 10 steps (4.6× the FLOPs and 3.5× the GPU time of CAGA), CAGA is still higher on 7 of 8 benchmarks:

MethodACDCCity-CVOC20-CVOC21-CCtx59-CCtx60-CStuff-CObject-C
MLMP-episodic, 1 step29.8423.4074.0945.4424.7823.0212.7021.00
MLMP-episodic, 10 steps30.5724.0478.4245.8925.7524.0013.0221.58
CAGA (1 step, R150)30.9524.2273.4850.0126.9224.5713.3021.73

(e) Revision. We agree that "continually accumulates" overstates what the curves show where the gain converges early. The paper now describes CAGA as retaining, rather than continually accumulating, its gains, and the limitations state that further improvement can be limited once the permanent anchor has settled, which on ACDC and Cityscapes-C happens within the first rounds (Table 16).

W2. Realism of the long-horizon setting and standard protocols

(a) Standard horizons (all methods on all 8 benchmarks: Tables 24–26). Relative gain (%) over No-Adapt, averaged over the 8 benchmarks:

MethodR1R5R10R150
CAGA+11.58+14.92+16.72+19.25
MLMP-episodic+13.59+13.59+13.59+13.59
MGP+11.34+14.50+16.62−27.00
DAF+11.50+12.62+11.18−37.99
DeYO+1.03+7.69+11.13−5.75
SAR+1.12+0.19+8.15+6.19

At R1 CAGA is second only to MLMP-episodic; at R5 and R10, the horizons used by prior work, it has the highest gain of all methods. The other six methods reach at most +3.28% at R10. On individual benchmarks, some baselines are higher in early rounds or reach higher peaks (e.g., MGP on ACDC at R10, Tables 24–26), but none does so consistently and most of them later collapse. (Gains use the corrected No-Adapt, hence +19.25% instead of +13.60% for CAGA.)

(b) Less repetitive streams (Fig. 17, Table 27). Resample: new images every round. Non-repeating: new images, a random domain order and a random severity (1–5) every round, so neither images nor conditions ever repeat. Mean mIoU over rounds:

MethodVOC20-C resampleVOC20-C non-repeatCity-C resampleCity-C non-repeat
CAGA71.9776.4523.8127.75
No-Adapt (MLMP inf.)68.8773.3822.8526.57
SPA75.9979.4313.79—
TRIBE71.3775.127.51—
MGP66.0973.705.776.87
SAR62.9766.8917.5222.05
DeYO62.2965.997.055.20
DAF52.1157.027.3512.04
PTTA33.21—20.82—
COME3.76—1.70—
TENT3.533.852.061.94
ViDA0.45—0.13—

"—": not run.

Baseline collapse is not an artifact of repetition: TENT, COME and ViDA still fall below 5 mIoU, and MGP, TRIBE, DeYO and DAF still degrade on Cityscapes-C. CAGA stays above No-Adapt (MLMP inference) in all four settings and is the best method on both Cityscapes-C streams; SPA is higher on VOC20-C, consistent with the main results, where it is strong on VOC20-C but collapses elsewhere.

AuthorsOpenReview ↗

Rebuttal Part 3/3

W2 (continued)

(c) Unseen conditions and protocol. In Fig. 10 the corruption set switches to a disjoint set at R76, and CAGA stays above No-Adapt without progressive degradation; in Fig. 6d it remains above No-Adapt on held-out corruptions that never contribute gradients. Revisiting the stream is the standard CTTA protocol (CoTTA repeats its sequence 10 times); we only extend its length. Our non-repeating stream still draws from the standard corruption types, and gradually drifting conditions are a natural next step, which the limitations now state.

W3. Hyperparameter sensitivity

Unspecified settings. The uku_k window contains the last 50 triggered trend scores, and the snapshot buffer holds ∣B∣=2000|\mathcal{B}| = 2000 windows (ACDC spans about 1,200, so it never fills). Both are now stated in the paper.

New results (Table 32). We added γ\gamma, LL and the uku_k window, widened γa\gamma_a, and added VOC20-C as a second benchmark (about twice as fast as Cityscapes-C, so every setting finished all 150 rounds).

Parameter (default)TestedACDC (R150)VOC20-C (R150)
γ\gamma (0.9)00 / ∞\infty32.19 / 30.9873.03 / 71.69
γa\gamma_a (0.95)0.80 / 0.85 / 0.90 / 0.98 / 0.9931.76 / 31.52 / 30.96 / 30.80 / 30.80— / — / 72.94 / 73.52 / —
LL (10)5 / 2030.91 / 31.1573.40 / 73.25
WW (50)25 / 10031.02 / 31.2373.54 / 73.32
τ\tau (0.5)0.25 / 1.030.97 / 31.1073.52 / 73.40
ρ\rho (0.01)0.005 / 0.0231.17 / 31.2073.50 / 73.36
uku_k window (50)25 / 10031.03 / 31.2573.39 / 73.35
Default30.9573.48

No setting collapses. The two limiting cases of γ\gamma (never / always deep restoration) stay within 1.3 (ACDC) and 1.8 (VOC20-C) mIoU of the default, and neither is better on both benchmarks, which supports gating between them. All other settings stay within 0.6 mIoU of the default on VOC20-C and within 30.8–31.8 on ACDC; some are slightly better on one benchmark, as expected for one shared configuration across 8 benchmarks and 14 backbones.

Q2. Baselines at 5×10−65 \times 10^{-6}; MLMP-episodic steps

A lower learning rate delays collapse but does not prevent it (Fig. 18, Tables 28–29): on ACDC the peak of TENT moves from round 19 to 35 and that of DeYO from 30 to 59, and both keep declining until R150. At 5×10−65 \times 10^{-6}, TENT still ends below No-Adapt on all 8 benchmarks (near zero on 7), and DeYO below No-Adapt on ACDC (22.26), Cityscapes-C (4.90) and COCO-Stuff-C, versus 30.95 and 24.22 for CAGA on the first two. The one exception is SAR on ACDC (31.84 vs. 30.95 at R150), whose curve oscillates between 23.6 and 33.3 and stays below CAGA on average over the stream (30.69 vs. 30.98). Conversely, CAGA reaches 31.03 on ACDC with the baselines' 10−510^{-5}. MLMP-episodic with 10 steps is covered in W1(d).

Q3. Seeds

The randomness comes only from the stochastic restoration mask and the patch shuffle of the reliability filter (the data order is fixed). Over three seeds on ACDC (Table 18): CAGA 30.97 ± 0.06, MLMP-episodic 29.84 ± 0.00, and the source-based restoration variants 29.77–30.26. Every seed of CAGA is above every seed of the other methods, and the spread is two orders of magnitude smaller than the gaps to collapsing baselines; the sensitivity settings were therefore run with one seed. For the ablations, the revised paper no longer draws conclusions from the small differences between the variants in Fig. 6(b), and instead relies on the same-base restoration comparison (Table 2), whose ordering holds for all three seeds on ACDC.


Note on the No-Adapt baseline. While preparing this response, we found that No-Adapt on the seven -C benchmarks had inadvertently been run with 7 prompt templates, whereas No-Adapt on ACDC and all single-prompt baselines (TENT, SAR, DeYO, COME, PTTA, SPA, TRIBE, ViDA) use one. We reran it with a single prompt; this lowers No-Adapt by 0.7–2.0 mIoU on the -C benchmarks, CAGA remains above it on all 8 benchmarks, and all tables and figures are updated.

We hope these results resolve the reviewer's two main concerns: CAGA keeps adapting under a restoration loop that stays active for the whole stream (W1), and both the baseline collapse and CAGA's advantage persist at standard horizons and on streams that never repeat (W2).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

General Response

We sincerely thank all reviewers for their careful and constructive feedback. We have uploaded a revised draft (new content in blue) and ran every experiment requested. The main changes are:

  1. Same-base comparison with reset methods (Sec. 2, Sec. 5.3 Table 2; Appendix A6). We now cite and discuss RDumb, PeTTA and Lim et al. (ICLR 2026). We keep CAGA's loss, pipeline and learning rate fixed and change only the restoration rule. CAGA beats periodic reset, CoTTA-style restoration, randomly triggered restoration and the reset rule of Lim et al. on all 8 benchmarks. Every rule that restores toward the source stays within 1.01.0 mIoU of the non-adapted model, so it stays stable only by giving up the adaptation. The main benefit comes from restoring toward an adapted anchor, and we now state this explicitly (Sec. 5.3).

  2. Gain from adaptation alone (Appendix A3 Table 3; Appendix A6 Table 15). We add No-Adapt (MLMP inference), which uses the same inference pipeline as CAGA but does no adaptation. Against this baseline CAGA still improves on all 8 benchmarks (average relative gain from adaptation alone: 7.7%). We also corrected the No-Adapt baseline on the -C benchmarks to a single prompt. CAGA remains above No-Adapt on all 8 benchmarks after the correction.

  3. Evidence that adaptation continues (Appendix A6–A7; Figs. 12, 13, 15). Without restoration, the same base objective degrades on 6 of 8 benchmarks. With restoration, CAGA retains at least 98.1%98.1\% of its peak on all 8. The gate fires in about 30%30\% of monitoring windows, at the same rate in both halves of the stream. Late updates are still domain-specific: the domain can be identified from a single update direction in 92%92\% of cases.

  4. Standard horizons and less repetitive streams (Appendix A8, Tables 24–27; Fig. 17). At rounds 5 and 10, CAGA has the highest average gain of all methods. On streams with new images and conditions that never repeat, several baselines still collapse and CAGA stays above No-Adapt.

  5. Robustness checks (Tables 18, 28–32; Fig. 18). Over 3 seeds, CAGA gives 30.97±0.0630.97 \pm 0.06 on ACDC. At CAGA's learning rate, TENT and DeYO still degrade; the lower rate mainly delays it. DAF rerun with its published settings reproduces its reported short-horizon gain and still drops below No-Adapt over 150 rounds. Halving and doubling every hyperparameter changes the result by at most 0.60.6 mIoU on VOC20-C.

  6. Why OVSS models collapse (Appendix A7, Tables 22–23; Fig. 16). The alignment between image and text features stays at the source level under CAGA but breaks down for collapsing methods. CAGA's adapted weights also stay above No-Adapt on 39 classes never seen during adaptation.

  7. Revised claims. We now describe CAGA as retaining, rather than continually accumulating, its adaptation gains. We removed wording that suggested an early-warning role, and we added limited plasticity after a domain change to the limitations (Sec. 6).

We address each reviewer's points in the individual responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.