acceptodds
Under review as a conference paper at ICLR 2027

COVSS: A Benchmark for Continual Open-Vocabulary Semantic Segmentation

Abstract

Semantic segmentation studies commonly evaluate continual updating and open-vocabulary recognition separately. Emerging work at their intersection leaves less understood how these capabilities develop together when supervision arrives incrementally. We introduce Continual Open-Vocabulary Semantic Segmentation (COVSS), a task and benchmark for studying this process through multiple class increments of comparable size. Built from existing COCO-Stuff171 and ADE150 annotations, COVSS specifies class partitions, stage-visible supervision, shared comparison conditions, and trajectory-based evaluation on internal and external vocabularies. A reproducible reference implementation combines anonymous scene supervision with dynamic semantic alignment. Experiments reveal weaknesses in both acquisition and retention among conventional continual segmentation methods, while open-vocabulary methods exhibit markedly different learning trajectories. Without dedicated anti-forgetting mechanisms, the COVSS baseline attains strong overall performance and positive net changes in previously learned categories, despite local declines. Pixel-level analysis attributes part of these gains to correcting earlier false positives through changes in competing category responses. These observations establish a benchmark for investigating how continual learning and open-vocabulary prediction interact, and identify challenges for developing scene understanding from successive, partially labeled experiences.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual learning, Open-vocabulary semantic segmentation, Semantic accumulation, Knowledge transfer, Temporal evaluation, Vision–language models.

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WXvWOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Continual Open-Vocabulary Semantic Segmentation (COVSS) task and evaluation protocol. On COCO-Stuff171 and ADE150, classes are divided into ten incremental stages of similar scale. Only identity supervision for current classes is provided at each stage, and a fixed complete internal vocabulary and external datasets are used to evaluate the model’s learning trajectory. The authors provide a reference implementation combining anonymous complement supervision and dynamic semantic alignment aggregation. Through performance matrices composed of class groups and checkpoints, BWT, and fixed-pixel analysis, they investigate the relationships among novel class acquisition, subsequent changes of old classes, and open-vocabulary capability. Experiments report that this baseline achieves favorable final performance among the compared continual training methods and exhibits positive net backward transfer across multiple runs.

Critical strengths
  1. The protocol clearly defines the visibility scope of staged category identities and uses a fixed complete prediction vocabulary at different checkpoints. This facilitates the analysis of category acquisition and subsequent performance changes under relatively consistent conditions, reducing confounding effects caused by variations in the candidate prediction set. Jointly tracking the trajectories of internal categories and the external vocabulary also delivers richer information than merely reporting the final mIoU.
  2. The paper conducts useful diagnostics on several results and constrains its conclusions cautiously. The authors clearly distinguish positive BWT from monotonic improvement, report ten random training runs for the reference baseline, and demonstrate through fixed-pixel and competing-category response analysis that rises in the IoU of old categories can be accompanied by falling recall. These observations provide valuable clues for analyzing prediction revision in continual learning.
Critical weaknesses

The main contribution of this paper lies in integrating existing class-incremental supervision, relatively balanced class partitioning, open-vocabulary testing, and continual learning trajectory measurement. The authors have acknowledged that these components are not individually proposed for the first time; therefore, stronger evidence is required to demonstrate that the combined protocol reveals difficulties or patterns that are hard to observe under existing setups. At present, the arguments against conventional protocols mainly rely on class proportion and pixel coverage statistics, and the main text has not presented sufficient controlled comparisons to isolate the respective contributions of class partitioning, training organization, and open-vocabulary evaluation to the conclusions. There are also specific issues regarding the positioning of related work.

Table 1 marks the class-incremental property of kNN-CLIP as “×”, yet Sections 4.2–4.3 of its original paper explicitly investigate class-incremental settings. Freezing the model and expanding the retrieval bank belong to solution mechanisms and cannot be directly used as evidence that it does not address class-incremental tasks. ConOVS has also studied incremental learning, old knowledge preservation, and out-of-distribution generalization. This paper may define a single continuously updated state as a specific comparison track, but it needs to more clearly illustrate the independent research value brought by this restriction and its practical distinctions from the most closely related work.

The reference implementation also fails to fully bridge this contribution gap. The construction of image-text correlation, spatial aggregation, query aggregation, and decoding pipeline in Equation (7) highly resemble CAT-Seg at the high-level architecture; anonymous supervision merges non-current pixels into one complement query and adopts BCE. The main text needs to explicitly list the newly added operations relative to baseline methods and verify their effects under unified backbone, pretraining, and optimization conditions. In particular, the original CAT-Seg nearly fails in Table 2 while the baseline of this paper achieves substantially better performance, and this large discrepancy needs to be explained through direct controlled ablation studies. The existing cross-checkpoint component pairing mainly demonstrates compatibility and cannot substitute for the above validation.

Questions for authors
  1. Please explain why the ADE150 values in Table 2 do not satisfy Final All = Diag + 0.9 × BWT. Do these metrics adopt different prediction vocabularies, category sets, run sets or aggregation rules? Please provide the complete matrices and corrected results for the relevant methods.
  2. What are the specific additional computational or training mechanisms of DAFA compared with CAT-Seg? Under the same backbone, pretraining, freezing strategy and optimization configuration, what are the respective results of vanilla background/complement supervision and the unseen query setting in this paper? What modifications account for the substantial gap between the two methods in Table 2? If these experiments are already included in the supplementary materials, please specify their locations.
  3. Why is the class-incremental property of kNN-CLIP marked as "×" in Table 1? For the most relevant methods such as ConOVS and retrieval-augmented approaches, what specific information or resource constraints prevent them from being included in the current comparison? Under the defined boundary of identity information, can adapted results be provided, or can the coverage scope of this benchmark track be clarified further?
  4. According to Equation (3), the same image may appear repeatedly across multiple stages. What is the overlap ratio of training images between different stages? After controlling for repeated image exposure, do positive BWT and improvements on old categories still hold?
  5. Can you supplement the in-distribution and out-of-distribution performance of the identically initialized model before training at each stage, and provide joint-training baselines with matched training budgets or sufficient convergence? Current out-of-distribution gains take T0 as the starting point, while continual training and joint training use a total of 800k and 80k iterations respectively. These conditions limit the interpretation of net capability gains and offline performance gaps.
Reviewer_3DcXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces COVSS, a task and benchmark for class-incremental learning with open-vocabulary segmentation models. The construction has four parts:

  • COCO-Stuff-171 and ADE-150 are split into ten near-equal class groups.
  • At each stage, only the current classes are labeled; all other pixels are anonymized.
  • A single continually updated model is evaluated after every stage on the full internal vocabulary and on external vocabularies (VOC-20, PC-59, PC-459).
  • Results are summarized by a category-group trajectory matrix with Final All, Diag, BWT and Pre.
Critical strengths
  1. The question of how continual acquisition and open-vocabulary prediction interact is relevant, and the trajectory-based evaluation is more informative than reporting final accuracy alone.
  2. The pixel-mass analysis of standard CSS splits is a useful observation (Fig. 1): the first 100 ADE classes cover 97.8% of pixels, and 91.8% in the 50-50 setting. It motivates more balanced increments.
  3. The baseline is run ten times and its positive BWT is tested statistically. Order sensitivity is examined, and the paper is careful to state the limits of its mechanism analysis.
Critical weaknesses
  1. The baseline comparisons are not reliable enough to support the benchmark's conclusions.

    • Several reproduced methods appear to fail outright. SimCIS reaches Diag 1.90 / 2.36 and Final All 0.20 / 0.54 on COCO / ADE, so it barely learns even the current classes. PLOP scores far below MiB (Final All 3.82 vs 7.46 on COCO, 4.95 vs 8.99 on ADE), the opposite of the ordering usually reported in the CSS literature. CAT-Seg collapses to 1.41 / 0.85.
    • All methods are forced to 80k iterations per stage. Meanwhile, the CSS methods "retain their official architecture, pretrained weights" (Sec. 5.1), presumably ImageNet-initialized CNN or Mask2Former models, whereas the OVS methods and the baseline use CLIP. The main text does not state any method's backbone.
    • As a result, the central claims are confounded by backbone/pretraining differences and possibly failed reproductions: "conventional continual segmentation methods" show "weaknesses in both acquisition and retention", and "open-vocabulary methods exhibit markedly different learning trajectories". For a benchmark paper, carefully tuned and comparable baselines are the core contribution.
  2. The reference implementation is CAT-Seg under new names, and its key ingredient is not isolated. DAFA corresponds one-to-one to CAT-Seg's cost-aggregation pipeline: a cosine cost volume (the "Raw Alignment Field"), cost embedding, spatial aggregation with visual guidance (SSE), class aggregation with text guidance (SQE), alternated twice (Fig. 2), and an upsampling decoder with guidance features. Since CAT-Seg itself appears in Table 2, the only functional difference behind the 1.41 → 41.40 (COCO) gap seems to be the anonymous complement query. That idea parallels standard background modeling in CSS (e.g., MiB) and the "unknown" class of SSUL (Cha et al., NeurIPS 2021). The main text neither ablates the complement query against CAT-Seg nor states this relationship plainly. Introducing new names (DAFA, Raw Alignment Field, SSE/SQE) obscures how small the change is.

  3. The benchmark's methodological novelty is limited. The data are existing annotations under a new split rule. The paper itself acknowledges that balanced streams already exist (Yin et al., 2025). The trajectory matrix and BWT follow GEM (Lopez-Paz & Ranzato, 2017), and Pre is a form of forward-transfer measurement. The CSS–OVS intersection has been studied by LBD, kNN-CLIP, CUE and ConOVS. The distinctive element is the specific combination of balanced increments, a single updated model and joint internal/external evaluation.

  4. Several protocol choices weaken the reported comparisons.

    • The baseline's main-table scores average ten runs, while the compared methods appear to be single runs.
    • The joint-training "upper bound" uses 80k iterations in total, whereas the continual models receive 80k per stage (800k overall).
    • The order-robustness study uses a different budget (10k per stage) from the main results.
    • VOC-20 and PC-59 overlap heavily with the COCO-Stuff vocabulary, so the "external" gains largely reflect in-vocabulary transfer. The paper acknowledges this.
  5. The mechanism analysis is modest. Eq. (8) restates the fact that IoU = TP / (TP + FP + FN) can increase when FP falls even if TP falls. The supporting interventions are two hand-picked cases (Fig. 3d). The finding that 48% of IoU-increasing transitions show no recall gain is interesting, but it is descriptive and does not explain why the baseline avoids forgetting.

Questions for authors
  1. What backbone, initialization and training recipe does each method in Table 2 use? Can you report the CSS methods with a CLIP backbone, or justify the mismatch?
  2. Why do SimCIS (Diag 1.90) and PLOP (below MiB) underperform so severely? Were they tuned for the 80k-iterations-per-stage schedule? How do they perform on their original benchmarks under your implementation?
  3. Please report CAT-Seg with and without the complement query under identical settings, and run all methods with multiple seeds.
  4. Will you release the split files, stage-view generation code, evaluation scripts and baseline configurations?
  5. How does joint training compare under a matched total training budget?
Reviewer_6cubOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces COVSS, a class-incremental protocol for open-vocabulary semantic segmentation (OVS) built on COCO-Stuff171 and ADE150. The classes of each dataset are split into ten groups of roughly equal size (18 + 9×17 for COCO, 10×15 for ADE). At each stage the model only sees images that contain the current group, and only that group is labeled; all other pixels are marked as "identity unknown" rather than background. A single model is updated across stages and evaluated after each one on the full internal vocabulary, including groups it has not been trained on yet, as well as on VOC20, PC59 and PC459. From the resulting group-by-checkpoint matrix the authors report Final All, Diag (performance when a group is first supervised), BWT (net change afterwards) and Pre (performance before supervision).

A reference baseline builds on CAT-Seg is also proposed. Unknown pixels are assigned to a training-only "unseen" query, a module called DAFA performs spatial and query aggregation, and no replay, distillation or pseudo-labeling is used. With 80k iterations per stage, it is compared against six continual semantic segmentation (CSS) methods, LBD, and four OVS methods. It obtains the best Final All (41.40 on COCO, 36.72 on ADE) and the best external-vocabulary scores, with a BWT of +6.17 / +5.48 that is positive in all ten runs.

Critical strengths
  • The category-group matrix separates first acquisition, subsequent change, and pre-supervision response, which is more informative than a single endpoint score.The trajectory-based evaluation and the mechanism analysis are the most useful parts of the paper. The group-by-checkpoint matrix separates acquisition, later change, and pre-supervision behavior, which a single final score cannot reveal. The paper contrasts naive sequential training of OVS models, where CAT-Seg collapses to 1.41/0.85 Final All, with the complement-supervised baseline. Together with the IoU decomposition and logit-replacement experiments, this gives a concrete, testable account of how old classes can improve.

  • The diagnosis of existing CSS splits is also valuable. The paper shows that the first 100 ADE classes cover 97.80% of training pixels. This makes a case that the standard 100-10 setting mostly tests small extensions of an already well-trained model, and it reasonably motivates balanced increments.

Critical weaknesses
  • The setting does not establish why continual learning is needed; the benchmark is essentially a new protocol over standard data. Continual learning usually assumes that earlier data cannot be revisited. In COVSS, however, the same images reappear across stages and only their earlier labels are hidden. Keeping those labels costs almost nothing, and the paper gives no practical reason to discard them. Replay is allowed and the main text states no memory budget, so keeping all seen labels and fine-tuning cumulatively appears admissible and is likely the strongest approach, yet this baseline is not reported. Joint training is also both cheaper and better: a single 80k-iteration run reaches 43.30/41.56, whereas the baseline needs 10 × 80k iterations to reach 41.40/36.72.

  • The "new" classes are also not new to the model. COCO-Stuff171 is the standard OVS training set, and all its class names are common concepts for CLIP. In Table 3, COCO group C9 rises from 19.3 to 35.1 before it is ever supervised and gains only 3.7 once supervised. This suggests most progress comes from generic dense alignment rather than new knowledge. For context only, since the protocols differ and this is not a fair comparison: a COCO-trained CAT-Seg ViT-L reaches 37.9 mIoU zero-shot on A-150 (Cho et al., 2024), and training-free kNN-CLIP reaches 41.3 (Gui et al., 2024). Both exceed the baseline's 36.72 after ten stages on ADE.

  • The paper claims to study how continual learning and open-vocabulary prediction interact, but its data and constraints do not create a situation that requires continual learning. A more convincing benchmark would use data beyond what pretrained models can already handle, together with explicit memory and compute budgets. For example, on MESS, CAT-Seg-B scores 28.09 in the medical domain, below the 29.51 random baseline (Cho et al., 2024).

  • The main finding is not attributed to any specific design. As described, DAFA closely matches CAT-Seg's cost aggregation: a cosine cost volume, a convolutional embedding, visually guided spatial aggregation, text-guided class aggregation, and an upsampling decoder using intermediate CLIP features. The only visible change is therefore the complement query. CAT-Seg collapses in the same setting while the baseline performs best, yet no ablation separates the effects of the complement query, DAFA, and the training configuration. As a result, the claim that positive transfer arises without anti-forgetting mechanisms cannot be credited to any particular component.

  • The positioning relative to prior work is also underdeveloped. The only method compared that specifically targets continual open-vocabulary segmentation is a reproduced LBD; ConOVS and kNN-CLIP are excluded with a one-sentence rationale. Table 1 marks kNN-CLIP as not class-incremental, yet Sec. 4.2 of kNN-CLIP continually trains FC-CLIP on ADE20K with 5, 10, or 30 classes per step. It finds that results fall below zero-shot (34.1 → 28.9/27.7/26.3), the opposite of this paper's finding, and the paper does not discuss this. ConOVS (Hwang et al., 2025) has also already argued that treating unseen classes as background conflicts with OVS, and it jointly evaluates pre-training, incremental, and zero-shot datasets. Even if the setting is accepted, the method contribution and the advance over prior work remain unclear.

Questions for authors
  • What is the target scenario? COVSS assumes three things at once: labels arrive in class batches; earlier labels cannot be kept, even though the same images reappear; and retraining on all available labels is not an option. Could the authors describe a practical scenario where all three hold? Otherwise, joint training or cumulative retraining seems the more reasonable choice. In the paper's own experiments, joint training is both cheaper and better.

  • Which CLIP backbone is used? Is the initial model raw CLIP or an OVS checkpoint? Which layers are fine-tuned? In Eq. 3, are originally unlabeled pixels mapped to ⊥ or ignored?

  • How does the total compute compare with joint training?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.