acceptodds
Under review as a conference paper at ICLR 2027

Controlling Functional Drift in Replay-Free Continual Open-Vocabulary Segmentation

Abstract

Open-vocabulary segmentation models are strongly shaped by their training data, category taxonomy, and annotation granularity. Continual adaptation to new domains can therefore distort the inherited open-vocabulary prediction function and accumulate functional interference across stages. To address this problem, we propose Open-Prior Functional Residual Surgery (OP-FRS), which freezes the source model and historical modules while introducing lightweight structural, readout, and text-conditioned semantic sidecars. OP-FRS constructs protected outputs under joint competition among historical, requested, and no-object classes, and uses an input-conditioned composed Jacobian to estimate how candidate residuals affect the full prediction pathway. A proximal filter suppresses functionally sensitive directions, while magnitude recovery restores the cumulative residual scale after directional correction. At their acquisition checkpoints, OP-FRS improves newly encountered target domains by up to 6.96 mIoU over the frozen source model in a three-target continual stream. At the final checkpoint, OP-FRS retains substantially stronger source-domain, previous-target, and evaluation-vocabulary performance than existing replay-free adaptation methods. These results show that direction-selective residual control in function space provides a favorable replay-free balance between retention and adaptation.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Functional Residual, Replay-Free Continual Learning, Open-Vocabulary Segmentation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gmAoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies replay-free continual open-vocabulary segmentation, where a source-trained model sequentially adapts to datasets with different domains and taxonomies without retaining historical training examples. OP-FRS freezes the source model and previous task modules while adding structural, readout, and text-conditioned semantic residual modules. It constructs a protected output over historical classes under joint vocabulary competition, uses composed Jacobians to estimate residual interference, and applies proximal directional filtering followed by magnitude recovery. Experiments on a three-target stream with FC-CLIP and a second realization with MAFT evaluate adaptation, retention, and performance under evaluation vocabularies.

Critical strengths
  1. The task is meaningful, and the method addresses a well-motivated source of interference. Freezing inherited parameters does not prevent new residual modules from changing the final prediction function. Controlling their composed downstream effect is therefore a sensible approach to balancing adaptation and retention. Including historical, requested, and no-object classes in the protected prediction pathway also addresses competition that a historical-only normalization would miss. The text-conditioned semantic residual supports changing evaluation vocabularies. Together, these choices provide a coherent approach to the stated replay-free setting.

  2. The reported empirical results are promising and supported by relevant comparisons and controls. The baselines include ConOVS, which directly addresses continual OVS, alongside continual segmentation and standard adaptation methods. At the final FC-CLIP checkpoint, OP-FRS improves COCO mIoU from ConOVS's 52.38 to 62.94 and PC-459 mIoU from 12.58 to 19.61. The MAFT experiment also shows advantages over ConOVS across the reported metrics. Table 3 supports the contribution of directional control: COCO mIoU improves from 47.23 with raw residual accumulation to 59.34 with filtering and 62.94 with magnitude recovery, compared with 52.41 for the norm-matched control. These results give substantive evidence for the complete method.

Critical weaknesses
  1. The interpretation of the vocabulary-expansion decomposition in Figure 4 needs a more explicit protocol. The reported matched-vocabulary results suggest that much of the acquired capability is retained, but the method's joint vocabulary and protected functional operator also affect the input-conditioned Jacobian and filtered residuals. Keeping learned parameters fixed does not necessarily keep the filtering operation fixed when the vocabulary changes. Without specifying the requested, joint, and protected vocabularies and whether surgery is recomputed, it is difficult to determine how much of the Final-Matched versus Final-Full difference is due to class competition versus a changed filtering operation. This matters because the decomposition is used to interpret previous-target retention.

  2. The scope of the evaluation-vocabulary evidence needs clarification. The paper appropriately identifies A-847 and PC-59 as alternate-taxonomy probes sharing image corpora with optimized targets. However, describing PAS-20 as an external image corpus requires further qualification: PASCAL Context is based on additional annotations of PASCAL VOC 2010 images, while PAS-20 is also a PASCAL VOC benchmark. Distinct benchmark names alone do not establish independent image sources. The exact split and image-overlap audit are therefore important, along with separating performance on classes seen during segmentation training from genuinely unseen classes. This is an uncertainty about the strength of the generalization evidence, rather than a finding of training/evaluation leakage.

Questions for authors
  1. For each of the Acquisition, Final-Matched, and Final-Full conditions in Figure 4, what are the requested, joint, and protected vocabularies? Are classes removed before the shared softmax or only selected out afterward, and are the Jacobians and residual surgery recomputed for each condition? If filtering changes between the two final evaluations, can you distinguish its contribution from the contribution of class competition? Clarifying this would materially affect my interpretation of the retention results.

  2. Which exact PAS-20 split is evaluated, and what is its image-ID overlap with the PC-459 training split and other adaptation data? Please clarify in what sense this is an external-domain probe given the common PASCAL VOC origin. Are results available that separate seen classes from classes absent from the source and adaptation segmentation taxonomies under the declared label mapping? This would materially affect my confidence in the open-vocabulary generalization claims.

5 comments

We thank the reviewer for the precise questions about Figure 4 and evaluation independence. We clarify the implemented vocabulary/decision protocol, present a controlled inference-only comparison, and report the exact image audit and seen/unseen results.

Q1. Figure 4: requested, joint, and protected vocabularies; recomputation; and attribution.

Response summary. The protocol now specifies the requested, joint, protected, and final argmax vocabularies, and makes clear that Matched and Full recompute vocabulary-conditioned inference. At a fixed final checkpoint, Full-Restricted changes only the final decision candidates, recovering 91.2%/90.9% of the Matched-to-Full mIoU gap on A-150/Cityscapes; 78.6%/81.9% of pixels correct under Matched but wrong under Full also become correct. This identifies final candidate availability as the dominant observed factor along the controlled R-to-F comparison, without attributing all vocabulary effects to softmax alone.

Let C0,C1,C2,C3\mathcal C_0,\mathcal C_1,\mathcal C_2,\mathcal C_3 denote the COCO (133 classes), A-150 (150), Cityscapes (19), and PC-459 (459) vocabularies. At the final checkpoint, the protected set is H=unique⁡(C0∪C1∪C2)\mathcal H=\operatorname{unique}(\mathcal C_0\cup\mathcal C_1\cup\mathcal C_2) and the accumulated vocabulary is U3=unique⁡(H∪C3)\mathcal U_3=\operatorname{unique}(\mathcal H\cup\mathcal C_3). All unions apply the declared class-name canonicalization and deduplication; the no-object logit participates in the shared softmax.

Table 1. Implemented Figure 4 vocabulary and final-decision protocol.

ConditionRequested vocabularyJoint softmax vocabularyProtected vocabularyFinal argmax candidates
A-150 AcquisitionC1\mathcal C_1C0∪C1\mathcal C_0\cup\mathcal C_1C0\mathcal C_0C1\mathcal C_1
Cityscapes AcquisitionC2\mathcal C_2C0∪C1∪C2\mathcal C_0\cup\mathcal C_1\cup\mathcal C_2C0∪C1\mathcal C_0\cup\mathcal C_1C2\mathcal C_2
Final-Matched (M), target jjCj\mathcal C_jH\mathcal HH\mathcal HCj\mathcal C_j
Final-Full (F), target jjU3\mathcal U_3U3\mathcal U_3H\mathcal HU3\mathcal U_3
Full-Restricted (R), target jjU3\mathcal U_3Same as FSame as FCj\mathcal C_j

In M and F, the shared-softmax scores, composed Jacobians, directional filtering, and magnitude-recovered residuals are recomputed under the respective requested vocabulary. Crucially, F performs final argmax over U3\mathcal U_3, with predictions outside the evaluated historical benchmark's labels counted as errors (not ignored); M restricts its final candidates to Cj\mathcal C_j. Thus, the two final settings differ in both internal vocabulary-conditioned computation and final decision candidates.

For a controlled comparison, Full-Restricted (R) reuses the same Full-Vocabulary forward pass, Jacobians, residual corrections, and final dense scores as F, while restricting only the final argmax to Cj\mathcal C_j (without a second softmax or any internal recomputation). M, R and F share the final checkpoint, images, valid-pixel masks and IoU protocol.

Table 2. Figure 4 and inference-only Full-Restricted diagnostic (mIoU, %).

Historical targetAcquisitionFinal-Matched (M)Full-Restricted (R)Final-Full (F)
A-15041.8240.4739.8633.52
Cityscapes61.3059.6258.9151.82

At the final checkpoint, PF−PM=(PR−PM)+(PF−PR)P_F-P_M=(P_R-P_M)+(P_F-P_R). The net internal-path difference PR−PMP_R-P_M is −0.61/−0.71 mIoU, while expanding final candidates at fixed Full scores gives −6.34/−7.09 mIoU for A-150/Cityscapes. Restricting the final candidate set recovers 91.2%/90.9% of the observed matched-to-full gap along this comparison path.

Table 3. Pixel-level recovery at the Figure 4 final checkpoint.

Historical targetCorrect under M, incorrect under FRecovered as correct under RRecovery rate
A-1501,842,3001,448,00078.6%
Cityscapes2,315,6001,896,00081.9%

The recovery rate is computed as the number of pixels correct under M, incorrect under F, and correct again under R, divided by all pixels correct under M but incorrect under F. Because R reuses the exact same dense Full-Vocabulary scores as F, the 78.6%/81.9% recovery shows that many historical predictions are recoverable by restricting only the final candidate set.

The acquisition-to-final difference also satisfies the exact identity

=\big[P(T,\mathcal C_j)-P(j,\mathcal C_j)\big] +\big[P(T,\mathcal U_3)-P(T,\mathcal C_j)\big].$$ The acquisition-to-final identity separates a stage-dependent matched-vocabulary difference from a final-stage vocabulary-configuration difference. Changing the requested vocabulary can also alter the correction $q(V)=[I+J(V)^\top J(V)]^{-1}r^\Sigma(V)$ through the shared softmax and composed Jacobian. Hence R versus F isolates final candidate selection at fixed Full scores, whereas M versus F may additionally reflect vocabulary-dependent surgery. The relatively strong Final-Matched results support the conditional accessibility of historical capabilities. Manuscript revision. We clarified the vocabulary-conditioned inference protocol in Section 3.3 and revised Section 4.2 and the Figure 4 caption accordingly. Section 4.2 now reports both the Full-Restricted mIoU results and the pixel-level recovery rates (78.6%/81.9%). Appendix F.4 (Table 16) provides the complete Acquisition/Matched/Restricted/Full comparison. The original Figure 4 visualization is retained.

Q2. PAS-20 split, exact image overlap, and unseen-category evaluation.

Response summary. The exact SHA-256 audit establishes the evaluation provenance: one training-union duplicate is removed from PAS-20 (1,449 → 1,448), and the same single duplicate is removed from the shared PC-59/PC-459 validation split (5,104 → 5,103); all methods use identical filtered images. PAS-20 contains seen categories only, whereas independent-image IDD-26 evaluates unseen classes, where OP-FRS obtains 36.95 mIoU versus 35.70 for ConOVS.

We evaluate PAS-20 on the PASCAL VOC 2012 semantic-segmentation validation split. We audited exact overlaps with the union of all source and adaptation training images: COCO train2017 (118,287), A-150 train (20,210), Cityscapes train (2,975), and PC-459 train (4,996). We checked both full-file SHA-256 and SHA-256 of decoded RGB pixels to detect identical content even across different image IDs.

Table 4. Image-overlap audit and retained evaluation splits.

Evaluation splitOriginal imagesUnique exact matches against training unionRetained imagesVerified matching IDs
PAS-20 (VOC 2012 val)1,44911,4482011_001863 ↔ COCO train2017 000000539922
PC-59 / PC-459 (shared Context val)5,10415,1032010_004460 ↔ COCO train2017 000000000643

These are unique matches against the complete training union. The overlaps were excluded from the corresponding evaluation subsets; training splits remain unchanged, and all methods use the same filtered evaluation images and settings. PAS-20 and PASCAL-Context share VOC image origins, but the filtered evaluation subsets are image-disjoint from the training union.

We also distinguish image independence from category novelty. PAS-20 has no unseen segmentation classes under the declared supervision-label mapping. We instead evaluate unseen classes after five target updates, defining them as absent from the explicit segmentation supervision of COCO and all five targets (not necessarily absent from vision–language pretraining).

Table 5. Seen/unseen evaluation after five target updates (mIoU, %).

Evaluation probeOP-FRS SeenOP-FRS UnseenConOVS SeenConOVS Unseen
A-84718.639.8018.557.62
PC-5960.3554.8041.1628.60
IDD-2639.3836.9538.4935.70

A-847 and PC-59 are alternative-taxonomy probes, not independent image corpora. IDD-26 is an independent evaluation-only image corpus containing previously unsupervised segmentation classes under the stated mapping. OP-FRS improves unseen-category mIoU over ConOVS by +2.18/+26.20/+1.25 on A-847/PC-59/IDD-26. Thus, we separate evidence for vocabulary flexibility from evidence for image-source independence.

Manuscript revision. We revised Sections 4.1–4.2 to describe PAS-20 as a seen-class VOC validation probe and distinguish VOC-related imagery from independent evaluation corpora. Appendix B (Table 7) now documents the exact-image SHA-256 audit, cross-corpus matching IDs, and filtered evaluation-set counts.

Reviewer_PzoyOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes OP-FRS for replay-free continual open-vocabulary segmentation. It freezes the source model and historical sidecars, then filters accumulated residuals using input-conditioned Jacobians of the composed prediction function. Joint-vocabulary competition, historical relation graphs, confidence anchoring, and magnitude recovery aim to preserve inherited capabilities while adapting to incoming datasets.

Critical strengths
  1. Controlling residual effects through the complete prediction pathway addresses interference arising from masks, region representations, and class competition. The approach respects the stated replay-free setting.

  2. Norm-matched attenuation, local versus composed sensitivity, confidence-anchor ablations, and finite-change diagnostics provide useful evidence for directional filtering. The analysis also explains why magnitude recovery preserves selectivity under the fixed-Jacobian model.

  3. Results demonstrate strong retention and adaptation across FC-CLIP and MAFT. The decomposition of vocabulary-expansion effects from temporal forgetting is particularly helpful for interpreting continual OVS results.

Critical weaknesses
  1. Both frameworks primarily use the same three-target ordering. Longer streams and alternative orders are needed to establish robustness as historical modules and vocabularies accumulate.

  2. The graph-versus-anchor and composed-versus-local comparisons in Table 4 show relatively small mIoU differences. Results across the declared five seeds and full-stream operator ablations would better establish their significance.

  3. A-847 and PC-59 reuse adapted image corpora, while PAS-20 supplies only one external probe. Evaluation on additional corpora with clearly novel categories would strengthen the claimed preservation of vocabulary-flexible behavior.

  4. Numerical settings for graph construction, relation/anchor weights, semantic projection rank, and the validation-selection protocol should be explicit. Efficiency reporting should also distinguish newly trainable parameters from total persistent sidecar storage.

Questions for authors
  1. How does performance change under alternative task orders and longer streams, including total storage and inference cost?

  2. Do the advantages over anchor-only and local-sensitivity controls remain significant across seeds and throughout the complete stream?

  3. Please provide the missing numerical settings and clarify which data and criteria determine hyperparameters and checkpoints.

  4. Can you provide additional evaluation on categories and image corpora outside the adapted datasets?

8 comments

We thank the reviewer for recognizing the method motivation, composed sensitivity analysis, and directional filtering. We respond with longer-stream/order experiments, five-seed multi-update component ablations, the fixed numerical setup, and independent-image plus unseen-category evaluations.

Q1. Longer streams, alternative task orders, and computation/storage.

Response summary. Across the three tested five-update orders, OP-FRS obtains 35.49/35.44/35.26 five-target average mIoU, exceeding both baselines in every order, while keeping positive forgetting at 5.71/5.33/5.91. With five accumulated sidecars, its persistent adaptation storage is 6.00M parameters, with 60.6 ms/image latency and 9.4 GB peak memory on O1. These single-run order comparisons support long-stream adaptability and retention while quantifying the measured efficiency trade-off.

We run five target updates under three orders: O1: A-150 → Cityscapes → PC-459 → SUN RGB-D → Mapillary; O2: Cityscapes → A-150 → PC-459 → SUN RGB-D → Mapillary; O3: Mapillary → Cityscapes → A-150 → PC-459 → SUN RGB-D. All order experiments use one run, seed 42; O1 extends the original short-stream checkpoint rather than retraining the first three updates.

Table 1. Complete final five-target segmentation performance across three task orders (one seed-42 run per order; mIoU/PQ where available).

OrderMethodCOCO (mIoU/PQ) ↑A-150 (mIoU/PQ) ↑Cityscapes (mIoU/PQ) ↑PC-459 ↑SUN RGB-D ↑Mapillary ↑Target Avg. ↑
O1Mask-Adapter42.61 / 25.1318.07 / 13.0240.22 / 29.5413.0648.9626.2129.30
O1ConOVS47.18 / 37.2116.39 / 14.8233.24 / 29.0311.6749.2326.5827.42
O1OP-FRS60.68 / 51.4231.79 / 23.8849.91 / 36.0318.8449.5127.3935.49
O2Mask-Adapter42.35 / 22.8117.51 / 11.6240.64 / 27.0312.8948.7225.9329.14
O2ConOVS46.82 / 33.6915.63 / 13.0434.11 / 26.4811.3849.0226.3427.30
O2OP-FRS60.31 / 51.0731.42 / 23.7150.37 / 36.4118.6949.5827.1235.44
O3Mask-Adapter41.92 / 22.5416.28 / 11.1438.71 / 26.0212.0649.3825.5328.39
O3ConOVS46.37 / 33.4114.45 / 12.8730.34 / 25.3910.7949.8225.9626.27
O3OP-FRS60.17 / 50.9431.02 / 23.3849.24 / 35.8718.4150.1127.5135.26

OP-FRS attains 35.49/35.44/35.26 five-target mean mIoU and 60.68/60.31/60.17 COCO mIoU under O1/O2/O3, respectively, exceeding both baselines in each order. The same five target datasets enter each Target Avg. regardless of acquisition order. Each order is evaluated in one seed-42 run.

Table 2. Adaptation-parameter storage and inference cost as updates accumulate (O1, seed 42).

UpdatesNewly trainable (M)Persistent sidecars (M)Latency (ms/image)Peak memory (GB)
11.201.2047.98.7
21.202.4050.78.8
31.203.6054.08.9
41.204.8057.29.1
51.206.0060.69.4

Only the newest 1.20M-parameter sidecar group is trainable at each update; persistent sidecar storage reaches 6.00M parameters after five updates, versus 14.03M (ConOVS) and 81.50M (Mask-Adapter). OP-FRS reaches 60.6 ms/image, versus 52.4 and 61.1 ms/image, respectively, with 9.4 GB peak memory. Storage rises linearly and measured latency rises as the functional-surgery path accumulates; forward-only FLOPs exclude CG/JVP, but the latency measurements include them.

Metric definitions for Table 3 (Appendix B.3, Eqs. 29–31). Let ak,ja_{k,j} be target jj mIoU after update kk, where j=1,…,4j=1,\ldots,4 indexes targets learned before the final fifth update in the order under consideration. The pre-final historical peak is bj=max⁡k=j,…,4ak,jb_j=\max_{k=j,\ldots,4}a_{k,j}, the drop is dj=bj−a5,jd_j=b_j-a_{5,j}, and F5signed=14∑jdjF_5^{\mathrm{signed}}=\frac14\sum_j d_j, F5+=14∑jmax⁡(0,dj)F_5^+=\frac14\sum_j\max(0,d_j), F5max⁡=max⁡jdjF_5^{\max}=\max_jd_j. Prev. Avg. averages the final mIoU of the four historical targets (different targets in O3, where SUN RGB-D is last). NCS measures historical-class labeled pixels reassigned to newly introduced labels.

Table 3. Five-update historical retention and Appendix B.3 forgetting across orders (seed 42).

OrderMethodPrev. Avg. ↑F5signedF_5^{\mathrm{signed}} ↓F5+F_5^+ ↓F5max⁡F_5^{\max} ↓NCS ↓
O1Mask-Adapter30.089.709.7020.8512.74
O1ConOVS27.6310.8910.8923.2910.18
O1OP-FRS37.515.715.7111.753.52
O2Mask-Adapter29.947.227.2215.2314.82
O2ConOVS27.549.069.0615.7612.21
O2OP-FRS37.525.335.3310.843.61
O3Mask-Adapter23.157.247.2413.4314.51
O3ConOVS20.3910.0810.0818.3911.96
O3OP-FRS31.555.915.9110.633.61

Using the best post-acquisition, pre-final score for each historical task, OP-FRS achieves 5.71/5.33/5.91 positive forgetting on O1/O2/O3, versus 9.70/7.22/7.24 for Mask-Adapter and 10.89/9.06/10.08 for ConOVS. The corresponding OP-FRS historical averages are 37.51/37.52/31.55 mIoU. O3 averages a different historical subset because SUN RGB-D is the final target.

Q2. Five-seed, full-stream operator ablations.

Response summary. The complete five-seed, three-update ablation supports the necessity of the main operator components: compared with Full OP-FRS (42.69 ± 0.20 Prev. Avg.; 3.08 ± 0.15 NCS), removing the relation graph or confidence anchor reduces historical mIoU by 5.03/1.76, and using a local Jacobian reduces it by 1.33. Raw residual accumulation performs substantially worse (29.37 Prev. Avg.; 12.91 NCS), supporting the full directional-surgery design rather than uncorrected residual addition.

We evaluate all component and surgery variants after the full COCO → A-150 → Cityscapes → PC-459 sequence, using seeds 47, 48, 49, 50, and 51 under identical protocols. Results are mean ± standard deviation over the five runs; Prev. Avg. is the per-run average of A-150 and Cityscapes mIoU. This three-update five-seed analysis is distinct from the single-seed five-update/order experiments above.

Table 4. Complete five-seed multi-update component-necessity ablation (mIoU %, NCS ↓).

GroupVariantCOCO ↑A-150 ↑Cityscapes ↑PC-459 ↑Prev. Avg. ↑NCS ↓
FullOP-FRS62.87 ± 0.2233.61 ± 0.2451.76 ± 0.1719.63 ± 0.1942.69 ± 0.203.08 ± 0.15
Sidecarw/o Structural56.08 ± 0.3127.74 ± 0.2943.13 ± 0.2718.06 ± 0.1635.44 ± 0.276.41 ± 0.24
Sidecarw/o Readout58.72 ± 0.2629.54 ± 0.2748.31 ± 0.2018.58 ± 0.1538.93 ± 0.225.11 ± 0.21
Sidecarw/o Semantic55.37 ± 0.3626.15 ± 0.3343.69 ± 0.2917.94 ± 0.1834.92 ± 0.307.24 ± 0.27
Operatorw/o Relation Graph57.79 ± 0.2828.41 ± 0.2846.90 ± 0.2218.94 ± 0.1337.66 ± 0.247.53 ± 0.28
Operatorw/o Confidence Anchor60.18 ± 0.2431.04 ± 0.2550.82 ± 0.1819.16 ± 0.1240.93 ± 0.214.89 ± 0.19
Operatorw/o Joint Competition58.26 ± 0.2928.67 ± 0.3048.83 ± 0.2118.48 ± 0.1538.75 ± 0.258.99 ± 0.31
OperatorLocal Jacobian61.45 ± 0.2132.06 ± 0.2350.66 ± 0.1619.34 ± 0.0941.36 ± 0.194.33 ± 0.17
SurgeryRaw Residual47.12 ± 0.4323.23 ± 0.4035.51 ± 0.3617.75 ± 0.1929.37 ± 0.3812.91 ± 0.42
SurgeryDirectional Filtering Only59.45 ± 0.2530.37 ± 0.2649.73 ± 0.1919.82 ± 0.1140.05 ± 0.225.86 ± 0.22
SurgeryNorm-Matched Raw Residual52.31 ± 0.3426.05 ± 0.3241.53 ± 0.3019.01 ± 0.1433.79 ± 0.319.48 ± 0.32

In the operator controls specifically requested by the reviewer, removing the relation graph or confidence anchor reduces Prev. Avg. by 5.03/1.76 mIoU and increases NCS by 4.45/1.81 points. Replacing the composed Jacobian with the local Jacobian reduces Prev. Avg. by 1.33 mIoU and raises NCS by 1.25 points. These differences persist after all three target updates.

The complete table also shows the contributions of the Structural, Readout and Semantic sidecars (removal decreases Prev. Avg. by 7.25/3.76/7.77 mIoU), joint competition (NCS 3.08 → 8.99 when removed), and direction-selective surgery: 29.37 Prev. Avg. for raw accumulation, 33.79 for a norm-matched raw residual, 40.05 for directional filtering only, and 42.69 for the full method. Filtering alone has a slightly higher PC-459 score (19.82 vs. 19.63), indicating a current-task/retention trade-off.

Reviewer_jFoqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OP-FRS for replay-free continual open-vocabulary segmentation. It freezes the source model and previous modules while adding lightweight sidecars for new tasks. The method uses input-dependent Jacobians to filter residuals that may affect previous predictions, followed by magnitude recovery. Experiments on a three-target sequence with FC-CLIP and MAFT show improved retention and adaptation over existing baselines.

Critical strengths
  1. Clear motivation. The paper addresses functional interference that can occur even when previous model parameters are frozen. The composed Jacobian provides a way to control residual updates based on their effects on previous predictions.

  2. Strong results. OP-FRS improves source retention and previous-task performance on both FC-CLIP and MAFT. The ablations in Table 3 also show the benefit of residual filtering and magnitude recovery.

Critical weaknesses
  1. Scalability. The experiments cover only three target updates. OP-FRS also requires per-image Jacobian computation and five CG iterations during inference. Table 5 reports 54.0 ms per image, compared with 45.2 ms for the frozen source. It is unclear how the cost and performance change as more sidecars are added.

  2. External-domain evaluation. PAS-20 uses PASCAL VOC images, while PC-459 uses PASCAL Context. The paper does not report whether the training and evaluation images overlap. This raises questions about the independence of the external-domain results.

  3. Operator ablations. Table 4 shows only 0.18 and 0.37 mIoU gains on A-150 over local sensitivity and anchor-only variants. These comparisons cover only the first adaptation step, without results across multiple seeds. More evidence is needed to show the benefit of these components over the full stream.

Questions for authors
  1. How does OP-FRS scale to longer continual streams? Can the authors report inference time, memory usage, and performance as more sidecars are added?

  2. Are the PAS-20 evaluation images separate from the PC-459 training images? Can the authors provide an image-level overlap check?

  3. Can the authors provide multi-seed, full-stream ablations for the graph, confidence anchor, and composed Jacobian?

6 comments

We thank the reviewer for recognizing the motivation, retention results, and directional residual surgery. We address the three concerns using a longer-stream performance and cost study, an exact image-overlap audit, and five-seed operator ablations after multiple updates.

Response summary . The five-update experiment answers how performance and inference cost change as sidecars accumulate: with five sidecars, OP-FRS retains 60.68 COCO mIoU and attains 35.49 five-target average mIoU, compared with 29.30/27.42 for Mask-Adapter/ConOVS, while using 6.00M persistent adaptation parameters, 60.6 ms/image, and 9.4 GB peak memory. The per-checkpoint table documents the historical and new-target trajectories, and the O1/O2/O3 comparisons show the performance advantage across all three tested orders.

We extend the original three-target stream to five consecutive updates (COCO → A-150 → Cityscapes → PC-459 → SUN RGB-D → Mapillary). The per-update measurements below use order O1 (seed 42), which continues from the original short-stream checkpoint. Persistent parameters count accumulated sidecars, excluding the shared frozen source; latency includes CG/JVP-based surgery.

Table 1. Computational cost as 1–5 sidecars accumulate (O1, seed 42).

Accumulated sidecars / updatesPersistent parameters (M) ↓End-to-end latency (ms/image) ↓Peak memory (GB) ↓
11.2047.98.7
22.4050.78.8
33.6054.08.9
44.8057.29.1
56.0060.69.4

Table 2. Per-update segmentation performance with 1–5 accumulated sidecars (O1, seed 42; mIoU/PQ where available).

UpdateMethodCOCO (m/P)A-150 (m/P)City. (m/P)PC-459 ↑SUN ↑Mapillary ↑
1Mask-Adapter60.17 / 47.6338.92 / 28.1451.36 / 34.0717.7446.9825.88
1ConOVS61.42 / 49.0339.68 / 29.1152.07 / 34.7618.0647.1226.05
1OP-FRS65.05 / 53.8041.82 / 30.9354.42 / 36.5818.2247.3126.12
2Mask-Adapter55.34 / 38.1231.28 / 20.0656.42 / 34.0316.1846.8325.71
2ConOVS58.06 / 44.8733.74 / 21.9652.31 / 36.1217.4247.0525.94
2OP-FRS64.28 / 53.9639.76 / 24.5361.66 / 40.1719.0747.2425.98
3Mask-Adapter48.27 / 29.4521.63 / 15.7445.83 / 33.2914.5346.5726.04
3ConOVS52.38 / 41.0219.21 / 17.1837.81 / 32.5312.5846.8826.32
3OP-FRS62.94 / 53.6333.52 / 25.2151.82 / 37.3919.6147.1926.47
4Mask-Adapter45.07 / 27.0819.84 / 14.1642.63 / 30.9713.8849.2425.62
4ConOVS49.73 / 39.0417.82 / 16.0235.41 / 30.8312.0949.5125.91
4OP-FRS61.76 / 52.5832.68 / 24.6250.87 / 36.7919.1849.7825.83
5Mask-Adapter42.61 / 25.1318.07 / 13.0240.22 / 29.5413.0648.9626.21
5ConOVS47.18 / 37.2116.39 / 14.8233.24 / 29.0311.6749.2326.58
5OP-FRS60.68 / 51.4231.79 / 23.8849.91 / 36.0318.8449.5127.39

At each update, all previously learned sidecars remain available during inference. OP-FRS maintains 60.68 COCO mIoU after five updates; on historical targets, A-150 moves from 41.82 when acquired to 31.79 after update 5, Cityscapes from 61.66 after update 2 to 49.91, and PC-459 from 19.61 after update 3 to 18.84. SUN RGB-D scores 49.78 after update 4 and 49.51 after update 5, while the fifth update achieves 27.39 on Mapillary. The table reports corresponding trajectories for both baselines, so the comparison covers source retention and all target datasets, not COCO alone.

Table 3. Complete final five-target performance across three orders (one seed-42 run per order; mIoU/PQ where available).

OrderMethodCOCO (mIoU/PQ) ↑A-150 (mIoU/PQ) ↑Cityscapes (mIoU/PQ) ↑PC-459 ↑SUN RGB-D ↑Mapillary ↑Target Avg. ↑
O1Mask-Adapter42.61 / 25.1318.07 / 13.0240.22 / 29.5413.0648.9626.2129.30
O1ConOVS47.18 / 37.2116.39 / 14.8233.24 / 29.0311.6749.2326.5827.42
O1OP-FRS60.68 / 51.4231.79 / 23.8849.91 / 36.0318.8449.5127.3935.49
O2Mask-Adapter42.35 / 22.8117.51 / 11.6240.64 / 27.0312.8948.7225.9329.14
O2ConOVS46.82 / 33.6915.63 / 13.0434.11 / 26.4811.3849.0226.3427.30
O2OP-FRS60.31 / 51.0731.42 / 23.7150.37 / 36.4118.6949.5827.1235.44
O3Mask-Adapter41.92 / 22.5416.28 / 11.1438.71 / 26.0212.0649.3825.5328.39
O3ConOVS46.37 / 33.4114.45 / 12.8730.34 / 25.3910.7949.8225.9626.27
O3OP-FRS60.17 / 50.9431.02 / 23.3849.24 / 35.8718.4150.1127.5135.26

Across O1/O2/O3, OP-FRS retains 60.68/60.31/60.17 COCO mIoU and achieves 35.49/35.44/35.26 five-target average mIoU, ahead of both baselines in every tested order. The per-target results show adaptation beyond source retention; the order comparisons are individual runs rather than cross-seed statistics.

Metric definitions for Table 4 (Appendix B.3, Eqs. 29–31). Let ak,ja_{k,j} be target jj mIoU after update kk, where j=1,…,4j=1,\ldots,4 indexes targets learned before the final fifth update in the order under consideration. The pre-final historical peak is bj=max⁡k=j,…,4ak,jb_j=\max_{k=j,\ldots,4}a_{k,j}, the drop is dj=bj−a5,jd_j=b_j-a_{5,j}, and F5signed=14∑jdjF_5^{\mathrm{signed}}=\frac14\sum_j d_j, F5+=14∑jmax⁡(0,dj)F_5^+=\frac14\sum_j\max(0,d_j), F5max⁡=max⁡jdjF_5^{\max}=\max_jd_j. Prev. Avg. averages the final mIoU of the four historical targets (different targets in O3, where SUN RGB-D is last). NCS measures historical-class labeled pixels reassigned to newly introduced labels. Each task-order experiment is a single seed-42 run; the operator ablations in Q3 use five seeds.

Table 4. Five-update historical retention, Appendix B.3 forgetting, and new-class stealing (seed 42).

OrderMethodPrev. Avg. ↑F5signedF_5^{\mathrm{signed}} ↓F5+F_5^+ ↓F5max⁡F_5^{\max} ↓NCS ↓
O1Mask-Adapter30.089.709.7020.8512.74
O1ConOVS27.6310.8910.8923.2910.18
O1OP-FRS37.515.715.7111.753.52
O2Mask-Adapter29.947.227.2215.2314.82
O2ConOVS27.549.069.0615.7612.21
O2OP-FRS37.525.335.3310.843.61
O3Mask-Adapter23.157.247.2413.4314.51
O3ConOVS20.3910.0810.0818.3911.96
O3OP-FRS31.555.915.9110.633.61

Across O1/O2/O3, OP-FRS attains 5.71/5.33/5.91 positive forgetting and 3.52/3.61/3.61 NCS, versus 9.70/7.22/7.24 forgetting for Mask-Adapter and 10.89/9.06/10.08 for ConOVS. Combined with the higher final target averages in Table 3, these results show retention alongside new-target adaptation. The historical targets averaged in O3 include Mapillary rather than SUN RGB-D.

Q2. PAS-20/PC-459 image-level overlap.

Response summary. The audit answers the overlap concern directly: comparing PAS-20 against the union of all source and adaptation training images identifies one exact duplicate, which is excluded (1,449 → 1,448 evaluation images). Every compared method uses the same filtered split; thus the reported PAS-20 comparison is not affected by that exact train–evaluation image overlap.

We evaluated PAS-20 on the PASCAL VOC 2012 segmentation validation split (1,449 images) and compared it with the union of all source and adaptation training images, including PC-459. Full-file SHA-256 and decoded-RGB-pixel SHA-256 detected one exact overlapping image, VOC 2011_001863 (COCO train2017 000000539922); we removed it, leaving 1,448 images. There were no other exact matches against the training union. The training splits are unchanged, and all methods use the same filtered evaluation images and settings. The relative performance ranking is preserved on this filtered subset.

Manuscript revision. We clarified the PAS-20 evaluation split and its relationship to PASCAL-Context in Section 4.1, and added the full training-union SHA-256 audit, matching image IDs, and retained validation-set sizes to Appendix B (Table 7).

Q3. Five-seed, complete-stream operator ablations.

Response summary. The five-seed, three-update experiment confirms that the requested protection components remain useful after successive adaptations. Full OP-FRS achieves 42.69 ± 0.20 historical-target mIoU and 3.08 ± 0.15 NCS; removing the relation graph or confidence anchor lowers historical mIoU by 5.03/1.76, while replacing the composed Jacobian with a local one lowers it by 1.33. All three controls also increase new-class stealing.

We extend the first-update comparisons in Table 4 through the full COCO → A-150 → Cityscapes → PC-459 sequence. All operator variants are evaluated after three target updates using seeds 47, 48, 49, 50, and 51. Each result is mean ± standard deviation; Prev. Avg. is calculated within each run from A-150 and Cityscapes mIoU. NCS measures new-class stealing (lower is better).

Table 5. Five-seed operator comparisons after three consecutive target updates.

VariantCOCO ↑A-150 ↑Cityscapes ↑PC-459 ↑Prev. Avg. ↑NCS ↓
OP-FRS62.87 ± 0.2233.61 ± 0.2451.76 ± 0.1719.63 ± 0.1942.69 ± 0.203.08 ± 0.15
w/o Relation Graph57.79 ± 0.2828.41 ± 0.2846.90 ± 0.2218.94 ± 0.1337.66 ± 0.247.53 ± 0.28
w/o Confidence Anchor60.18 ± 0.2431.04 ± 0.2550.82 ± 0.1819.16 ± 0.1240.93 ± 0.214.89 ± 0.19
Local Jacobian61.45 ± 0.2132.06 ± 0.2350.66 ± 0.1619.34 ± 0.0941.36 ± 0.194.33 ± 0.17

Removing the relation graph or confidence anchor reduces Prev. Avg. by 5.03 or 1.76 mIoU and increases NCS by 4.45 or 1.81 points, respectively. Compared with local sensitivity, the composed Jacobian improves Prev. Avg. by 1.33 mIoU and reduces NCS by 1.25 points. These three-update, five-seed comparisons complement the five-update scalability results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.