Controlling Functional Drift in Replay-Free Continual Open-Vocabulary Segmentation

Open-vocabulary segmentation models are strongly shaped by their training data, category taxonomy, and annotation granularity. Continual adaptation to new domains can therefore distort the inherited open-vocabulary prediction function and accumulate functional interference across stages. To address this problem, we propose Open-Prior Functional Residual Surgery (OP-FRS), which freezes the source model and historical modules while introducing lightweight structural, readout, and text-conditioned semantic sidecars. OP-FRS constructs protected outputs under joint competition among historical, requested, and no-object classes, and uses an input-conditioned composed Jacobian to estimate how candidate residuals affect the full prediction pathway. A proximal filter suppresses functionally sensitive directions, while magnitude recovery restores the cumulative residual scale after directional correction. At their acquisition checkpoints, OP-FRS improves newly encountered target domains by up to 6.96 mIoU over the frozen source model in a three-target continual stream. At the final checkpoint, OP-FRS retains substantially stronger source-domain, previous-target, and evaluation-vocabulary performance than existing replay-free adaptation methods. These results show that direction-selective residual control in function space provides a favorable replay-free balance between retention and adaptation.