Preventing Collapse in Long-Horizon Continual Test-Time Adaptation for Open-Vocabulary Semantic Segmentation

Open-vocabulary semantic segmentation (OVSS) assigns pixel-level labels from free-form text classes but, like other dense prediction models, degrades under distribution shift. Continual test-time adaptation (CTTA) addresses non-stationary shifts by continuously adapting models on unlabeled test streams. However, most existing CTTA methods are designed for closed-set classifiers with fixed prediction heads, and thus do not well match to the open-vocabulary prediction structure. We stress-test CTTA for OVSS over 150 adaptation rounds, an order of magnitude longer than common evaluation protocols. At this horizon, we uncover an adaptation–stability trade-off across existing methods: those that achieve substantial adaptation gains eventually collapse, whereas those that remain stable provide limited improvement. Although episodic resetting prevents collapse, it sacrifices the ability to accumulate useful knowledge over time. To address this, we identify two label-free signals that track collapse progression from the perspectives of optimization dynamics and prediction distribution: gradient-norm trend and class-marginal entropy. Building on these signals, we propose CAGA (Collapse-Aware Gated Anchoring), which preserves beneficial adaptation while selectively restoring the model to a previously reliable state as collapse emerges. CAGA mitigates the adaptation–stability trade-off, consistently preventing long-horizon collapse while sustaining adaptation gains across 8 benchmarks and 14 OVSS backbones. The code will be released upon acceptance.