PROMPT-FREE TASK-AGNOSTIC CONTINUAL LEARNING ACROSS HETEROGENEOUS BINARY SEGMENTATION TASKS

Continual segmentation is commonly studied under changes in semantic classes or image domains while the underlying prediction objective remains relatively stable. We study a more heterogeneous continual setting in which sequential binary segmentation tasks may differ not only in image distribution but also in the semantic definition of foreground, while retaining the same binary prediction interface. We further consider prompt-free, task-agnostic inference: task boundaries are known during sequential training, but neither task identity nor interactive prompts are available at test time. Using the same shared-path continual learner, a domain-only stream retains 0.809 AA-FG-IoU with 0.118 forgetting, whereas a heterogeneous stream that changes both domain and foreground semantics drops to 0.532 AA-FG-IoU with 0.310 forgetting. Motivated by this degradation, we construct a continual segmentation framework with a frozen DINOv2-B/14 backbone, parameter-isolated segmentation experts, and an adapted distribution-replay selector that activates one expert for each unlabeled test image. On the COD→Polyp→USOD→TOD stream, Hard Top-1 inference reaches 0.806 AA-mIoU with 92.7% task-selection accuracy, compared with 0.814 AA-mIoU under Task-ID Oracle selection, leaving a routing gap of 0.009. These results support a decomposition of heterogeneous continual segmentation into two complementary requirements: preserving task-specific segmentation capabilities and reliably accessing the appropriate capability at inference.