- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies replay-free continual open-vocabulary segmentation, where a source-trained model sequentially adapts to datasets with different domains and taxonomies without retaining historical training examples. OP-FRS freezes the source model and previous task modules while adding structural, readout, and text-conditioned semantic residual modules. It constructs a protected output over historical classes under joint vocabulary competition, uses composed Jacobians to estimate residual interference, and applies proximal directional filtering followed by magnitude recovery. Experiments on a three-target stream with FC-CLIP and a second realization with MAFT evaluate adaptation, retention, and performance under evaluation vocabularies.
Critical strengths
-
The task is meaningful, and the method addresses a well-motivated source of interference. Freezing inherited parameters does not prevent new residual modules from changing the final prediction function. Controlling their composed downstream effect is therefore a sensible approach to balancing adaptation and retention. Including historical, requested, and no-object classes in the protected prediction pathway also addresses competition that a historical-only normalization would miss. The text-conditioned semantic residual supports changing evaluation vocabularies. Together, these choices provide a coherent approach to the stated replay-free setting.
-
The reported empirical results are promising and supported by relevant comparisons and controls. The baselines include ConOVS, which directly addresses continual OVS, alongside continual segmentation and standard adaptation methods. At the final FC-CLIP checkpoint, OP-FRS improves COCO mIoU from ConOVS's 52.38 to 62.94 and PC-459 mIoU from 12.58 to 19.61. The MAFT experiment also shows advantages over ConOVS across the reported metrics. Table 3 supports the contribution of directional control: COCO mIoU improves from 47.23 with raw residual accumulation to 59.34 with filtering and 62.94 with magnitude recovery, compared with 52.41 for the norm-matched control. These results give substantive evidence for the complete method.
Critical weaknesses
-
The interpretation of the vocabulary-expansion decomposition in Figure 4 needs a more explicit protocol. The reported matched-vocabulary results suggest that much of the acquired capability is retained, but the method's joint vocabulary and protected functional operator also affect the input-conditioned Jacobian and filtered residuals. Keeping learned parameters fixed does not necessarily keep the filtering operation fixed when the vocabulary changes. Without specifying the requested, joint, and protected vocabularies and whether surgery is recomputed, it is difficult to determine how much of the Final-Matched versus Final-Full difference is due to class competition versus a changed filtering operation. This matters because the decomposition is used to interpret previous-target retention.
-
The scope of the evaluation-vocabulary evidence needs clarification. The paper appropriately identifies A-847 and PC-59 as alternate-taxonomy probes sharing image corpora with optimized targets. However, describing PAS-20 as an external image corpus requires further qualification: PASCAL Context is based on additional annotations of PASCAL VOC 2010 images, while PAS-20 is also a PASCAL VOC benchmark. Distinct benchmark names alone do not establish independent image sources. The exact split and image-overlap audit are therefore important, along with separating performance on classes seen during segmentation training from genuinely unseen classes. This is an uncertainty about the strength of the generalization evidence, rather than a finding of training/evaluation leakage.
Questions for authors
-
For each of the Acquisition, Final-Matched, and Final-Full conditions in Figure 4, what are the requested, joint, and protected vocabularies? Are classes removed before the shared softmax or only selected out afterward, and are the Jacobians and residual surgery recomputed for each condition? If filtering changes between the two final evaluations, can you distinguish its contribution from the contribution of class competition? Clarifying this would materially affect my interpretation of the retention results.
-
Which exact PAS-20 split is evaluated, and what is its image-ID overlap with the PC-459 training split and other adaptation data? Please clarify in what sense this is an external-domain probe given the common PASCAL VOC origin. Are results available that separate seen classes from classes absent from the source and adaptation segmentation taxonomies under the declared label mapping? This would materially affect my confidence in the open-vocabulary generalization claims.
5 comments
Table 3. Pixel-level recovery at the Figure 4 final checkpoint.
| Historical target | Correct under M, incorrect under F | Recovered as correct under R | Recovery rate |
|---|---|---|---|
| A-150 | 1,842,300 | 1,448,000 | 78.6% |
| Cityscapes | 2,315,600 | 1,896,000 | 81.9% |
The recovery rate is computed as the number of pixels correct under M, incorrect under F, and correct again under R, divided by all pixels correct under M but incorrect under F. Because R reuses the exact same dense Full-Vocabulary scores as F, the 78.6%/81.9% recovery shows that many historical predictions are recoverable by restricting only the final candidate set.
The acquisition-to-final difference also satisfies the exact identity
=\big[P(T,\mathcal C_j)-P(j,\mathcal C_j)\big] +\big[P(T,\mathcal U_3)-P(T,\mathcal C_j)\big].$$ The acquisition-to-final identity separates a stage-dependent matched-vocabulary difference from a final-stage vocabulary-configuration difference. Changing the requested vocabulary can also alter the correction $q(V)=[I+J(V)^\top J(V)]^{-1}r^\Sigma(V)$ through the shared softmax and composed Jacobian. Hence R versus F isolates final candidate selection at fixed Full scores, whereas M versus F may additionally reflect vocabulary-dependent surgery. The relatively strong Final-Matched results support the conditional accessibility of historical capabilities. Manuscript revision. We clarified the vocabulary-conditioned inference protocol in Section 3.3 and revised Section 4.2 and the Figure 4 caption accordingly. Section 4.2 now reports both the Full-Restricted mIoU results and the pixel-level recovery rates (78.6%/81.9%). Appendix F.4 (Table 16) provides the complete Acquisition/Matched/Restricted/Full comparison. The original Figure 4 visualization is retained.Q2. PAS-20 split, exact image overlap, and unseen-category evaluation.
Response summary. The exact SHA-256 audit establishes the evaluation provenance: one training-union duplicate is removed from PAS-20 (1,449 → 1,448), and the same single duplicate is removed from the shared PC-59/PC-459 validation split (5,104 → 5,103); all methods use identical filtered images. PAS-20 contains seen categories only, whereas independent-image IDD-26 evaluates unseen classes, where OP-FRS obtains 36.95 mIoU versus 35.70 for ConOVS.
We evaluate PAS-20 on the PASCAL VOC 2012 semantic-segmentation validation split. We audited exact overlaps with the union of all source and adaptation training images: COCO train2017 (118,287), A-150 train (20,210), Cityscapes train (2,975), and PC-459 train (4,996). We checked both full-file SHA-256 and SHA-256 of decoded RGB pixels to detect identical content even across different image IDs.
Table 4. Image-overlap audit and retained evaluation splits.
| Evaluation split | Original images | Unique exact matches against training union | Retained images | Verified matching IDs |
|---|---|---|---|---|
| PAS-20 (VOC 2012 val) | 1,449 | 1 | 1,448 | 2011_001863 ↔ COCO train2017 000000539922 |
| PC-59 / PC-459 (shared Context val) | 5,104 | 1 | 5,103 | 2010_004460 ↔ COCO train2017 000000000643 |
These are unique matches against the complete training union. The overlaps were excluded from the corresponding evaluation subsets; training splits remain unchanged, and all methods use the same filtered evaluation images and settings. PAS-20 and PASCAL-Context share VOC image origins, but the filtered evaluation subsets are image-disjoint from the training union.
We also distinguish image independence from category novelty. PAS-20 has no unseen segmentation classes under the declared supervision-label mapping. We instead evaluate unseen classes after five target updates, defining them as absent from the explicit segmentation supervision of COCO and all five targets (not necessarily absent from vision–language pretraining).
Table 5. Seen/unseen evaluation after five target updates (mIoU, %).
| Evaluation probe | OP-FRS Seen | OP-FRS Unseen | ConOVS Seen | ConOVS Unseen |
|---|---|---|---|---|
| A-847 | 18.63 | 9.80 | 18.55 | 7.62 |
| PC-59 | 60.35 | 54.80 | 41.16 | 28.60 |
| IDD-26 | 39.38 | 36.95 | 38.49 | 35.70 |
A-847 and PC-59 are alternative-taxonomy probes, not independent image corpora. IDD-26 is an independent evaluation-only image corpus containing previously unsupervised segmentation classes under the stated mapping. OP-FRS improves unseen-category mIoU over ConOVS by +2.18/+26.20/+1.25 on A-847/PC-59/IDD-26. Thus, we separate evidence for vocabulary flexibility from evidence for image-source independence.
Manuscript revision. We revised Sections 4.1–4.2 to describe PAS-20 as a seen-class VOC validation probe and distinguish VOC-related imagery from independent evaluation corpora. Appendix B (Table 7) now documents the exact-image SHA-256 audit, cross-corpus matching IDs, and filtered evaluation-set counts.
We thank the reviewer for the precise questions about Figure 4 and evaluation independence. We clarify the implemented vocabulary/decision protocol, present a controlled inference-only comparison, and report the exact image audit and seen/unseen results.
Q1. Figure 4: requested, joint, and protected vocabularies; recomputation; and attribution.
Response summary. The protocol now specifies the requested, joint, protected, and final argmax vocabularies, and makes clear that Matched and Full recompute vocabulary-conditioned inference. At a fixed final checkpoint, Full-Restricted changes only the final decision candidates, recovering 91.2%/90.9% of the Matched-to-Full mIoU gap on A-150/Cityscapes; 78.6%/81.9% of pixels correct under Matched but wrong under Full also become correct. This identifies final candidate availability as the dominant observed factor along the controlled R-to-F comparison, without attributing all vocabulary effects to softmax alone.
Let C0,C1,C2,C3 denote the COCO (133 classes), A-150 (150), Cityscapes (19), and PC-459 (459) vocabularies. At the final checkpoint, the protected set is H=unique(C0∪C1∪C2) and the accumulated vocabulary is U3=unique(H∪C3). All unions apply the declared class-name canonicalization and deduplication; the no-object logit participates in the shared softmax.
Table 1. Implemented Figure 4 vocabulary and final-decision protocol.
In M and F, the shared-softmax scores, composed Jacobians, directional filtering, and magnitude-recovered residuals are recomputed under the respective requested vocabulary. Crucially, F performs final argmax over U3, with predictions outside the evaluated historical benchmark's labels counted as errors (not ignored); M restricts its final candidates to Cj. Thus, the two final settings differ in both internal vocabulary-conditioned computation and final decision candidates.
For a controlled comparison, Full-Restricted (R) reuses the same Full-Vocabulary forward pass, Jacobians, residual corrections, and final dense scores as F, while restricting only the final argmax to Cj (without a second softmax or any internal recomputation). M, R and F share the final checkpoint, images, valid-pixel masks and IoU protocol.
Table 2. Figure 4 and inference-only Full-Restricted diagnostic (mIoU, %).
At the final checkpoint, PF−PM=(PR−PM)+(PF−PR). The net internal-path difference PR−PM is −0.61/−0.71 mIoU, while expanding final candidates at fixed Full scores gives −6.34/−7.09 mIoU for A-150/Cityscapes. Restricting the final candidate set recovers 91.2%/90.9% of the observed matched-to-full gap along this comparison path.