- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes the Continual Open-Vocabulary Semantic Segmentation (COVSS) task and evaluation protocol. On COCO-Stuff171 and ADE150, classes are divided into ten incremental stages of similar scale. Only identity supervision for current classes is provided at each stage, and a fixed complete internal vocabulary and external datasets are used to evaluate the model’s learning trajectory. The authors provide a reference implementation combining anonymous complement supervision and dynamic semantic alignment aggregation. Through performance matrices composed of class groups and checkpoints, BWT, and fixed-pixel analysis, they investigate the relationships among novel class acquisition, subsequent changes of old classes, and open-vocabulary capability. Experiments report that this baseline achieves favorable final performance among the compared continual training methods and exhibits positive net backward transfer across multiple runs.
Critical strengths
- The protocol clearly defines the visibility scope of staged category identities and uses a fixed complete prediction vocabulary at different checkpoints. This facilitates the analysis of category acquisition and subsequent performance changes under relatively consistent conditions, reducing confounding effects caused by variations in the candidate prediction set. Jointly tracking the trajectories of internal categories and the external vocabulary also delivers richer information than merely reporting the final mIoU.
- The paper conducts useful diagnostics on several results and constrains its conclusions cautiously. The authors clearly distinguish positive BWT from monotonic improvement, report ten random training runs for the reference baseline, and demonstrate through fixed-pixel and competing-category response analysis that rises in the IoU of old categories can be accompanied by falling recall. These observations provide valuable clues for analyzing prediction revision in continual learning.
Critical weaknesses
The main contribution of this paper lies in integrating existing class-incremental supervision, relatively balanced class partitioning, open-vocabulary testing, and continual learning trajectory measurement. The authors have acknowledged that these components are not individually proposed for the first time; therefore, stronger evidence is required to demonstrate that the combined protocol reveals difficulties or patterns that are hard to observe under existing setups. At present, the arguments against conventional protocols mainly rely on class proportion and pixel coverage statistics, and the main text has not presented sufficient controlled comparisons to isolate the respective contributions of class partitioning, training organization, and open-vocabulary evaluation to the conclusions. There are also specific issues regarding the positioning of related work.
Table 1 marks the class-incremental property of kNN-CLIP as “×”, yet Sections 4.2–4.3 of its original paper explicitly investigate class-incremental settings. Freezing the model and expanding the retrieval bank belong to solution mechanisms and cannot be directly used as evidence that it does not address class-incremental tasks. ConOVS has also studied incremental learning, old knowledge preservation, and out-of-distribution generalization. This paper may define a single continuously updated state as a specific comparison track, but it needs to more clearly illustrate the independent research value brought by this restriction and its practical distinctions from the most closely related work.
The reference implementation also fails to fully bridge this contribution gap. The construction of image-text correlation, spatial aggregation, query aggregation, and decoding pipeline in Equation (7) highly resemble CAT-Seg at the high-level architecture; anonymous supervision merges non-current pixels into one complement query and adopts BCE. The main text needs to explicitly list the newly added operations relative to baseline methods and verify their effects under unified backbone, pretraining, and optimization conditions. In particular, the original CAT-Seg nearly fails in Table 2 while the baseline of this paper achieves substantially better performance, and this large discrepancy needs to be explained through direct controlled ablation studies. The existing cross-checkpoint component pairing mainly demonstrates compatibility and cannot substitute for the above validation.
Questions for authors
- Please explain why the ADE150 values in Table 2 do not satisfy Final All = Diag + 0.9 × BWT. Do these metrics adopt different prediction vocabularies, category sets, run sets or aggregation rules? Please provide the complete matrices and corrected results for the relevant methods.
- What are the specific additional computational or training mechanisms of DAFA compared with CAT-Seg? Under the same backbone, pretraining, freezing strategy and optimization configuration, what are the respective results of vanilla background/complement supervision and the unseen query setting in this paper? What modifications account for the substantial gap between the two methods in Table 2? If these experiments are already included in the supplementary materials, please specify their locations.
- Why is the class-incremental property of kNN-CLIP marked as "×" in Table 1? For the most relevant methods such as ConOVS and retrieval-augmented approaches, what specific information or resource constraints prevent them from being included in the current comparison? Under the defined boundary of identity information, can adapted results be provided, or can the coverage scope of this benchmark track be clarified further?
- According to Equation (3), the same image may appear repeatedly across multiple stages. What is the overlap ratio of training images between different stages? After controlling for repeated image exposure, do positive BWT and improvements on old categories still hold?
- Can you supplement the in-distribution and out-of-distribution performance of the identically initialized model before training at each stage, and provide joint-training baselines with matched training budgets or sufficient convergence? Current out-of-distribution gains take T0 as the starting point, while continual training and joint training use a total of 800k and 80k iterations respectively. These conditions limit the interpretation of net capability gains and offline performance gaps.