acceptodds
Withdrawn from ICLR 2027

PROMPT-FREE TASK-AGNOSTIC CONTINUAL LEARNING ACROSS HETEROGENEOUS BINARY SEGMENTATION TASKS

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Continual segmentation is commonly studied under changes in semantic classes or image domains while the underlying prediction objective remains relatively stable. We study a more heterogeneous continual setting in which sequential binary segmentation tasks may differ not only in image distribution but also in the semantic definition of foreground, while retaining the same binary prediction interface. We further consider prompt-free, task-agnostic inference: task boundaries are known during sequential training, but neither task identity nor interactive prompts are available at test time. Using the same shared-path continual learner, a domain-only stream retains 0.809 AA-FG-IoU with 0.118 forgetting, whereas a heterogeneous stream that changes both domain and foreground semantics drops to 0.532 AA-FG-IoU with 0.310 forgetting. Motivated by this degradation, we construct a continual segmentation framework with a frozen DINOv2-B/14 backbone, parameter-isolated segmentation experts, and an adapted distribution-replay selector that activates one expert for each unlabeled test image. On the COD→Polyp→USOD→TOD stream, Hard Top-1 inference reaches 0.806 AA-mIoU with 92.7% task-selection accuracy, compared with 0.814 AA-mIoU under Task-ID Oracle selection, leaving a routing gap of 0.009. These results support a decomposition of heterogeneous continual segmentation into two complementary requirements: preserving task-specific segmentation capabilities and reliably accessing the appropriate capability at inference.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Continual Learning,Semantic Segmentation,Computer Vision,Parameter-Efficient Learning,Foundation Models

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_t3LLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper works on solving heterogenous continual binary segmentation task, where both the image domain and the foreground type changes over the stages of training. Motivated by the observation that a shared learner underperforms under heterogeneous condition, the paper places parameter-isolated expert per each tasks, and a task-agnostic selector to decide the domain type to map the expert. The framework achieves better segmentation performance compared to traditional continual learning methods.

Critical strengths

The paper proposes a concrete framework that solves heterogenous continual binary segmentation task, with variety of experiments revealing the failure cases and ablation of each components.

Critical weaknesses

The main concern of the paper is on its novelty. The design of the framework is straightforward from extending SOYO [1]'s parameter selection to expert selection, with standard expert design that does not solve any new visible challenges that raises from the continual setting. The challenges and the corresponding solutions should be clearly stated to identify the contribution of the paper, but it is currently a combination of existing methods without representing what hard problems the paper has solved.

Another is on the outdated baseline methods, as only traditional CL methods (LwF, EWC, ER) are compared. The methods cited in the paper could also be evaluated comparably, including non-continual setups only for evaluation, such as CoSAM [2] (segment anything methods). Other related works not cited in this paper worth being compared to, including Spider [3] that uses a context-based single model, and EVP [4] which uses explicit visual prompts for segmentation on various domains.

[1] Wang, Q., Song, X., He, Y., Han, J., Ding, C., Gao, X., & Gong, Y. (2025, June). Boosting domain incremental learning: Selecting the optimal parameters is all you need. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 4839-4849). IEEE.

[2] Yang, J., Wu, Y., Cen, J., Huang, W., Wang, H., & Zhang, J. (2024). Continual learning for segment anything model adaptation. arXiv preprint arXiv:2412.06418.

[3] Zhao, X., Pang, Y., Ji, W., Sheng, B., Zuo, J., Zhang, L., & Lu, H. (2024). Spider: A unified framework for context-dependent concept segmentation. In Forty-first International Conference on Machine Learning (ICML).

[4] Liu, W., Shen, X., Pun, C. M., & Cun, X. (2023, June). Explicit visual prompting for low-level structure segmentations. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 19434-19445). IEEE.

Questions for authors
  1. On whether single shared pathway is actually insufficient. The paper is motivated from the finding that base methods fails on heterogenous task, but there aren't sufficient comparison between state-of-the-art single-path methods empirically, and the claim may not generalize to other experimental settings. Can the claim be further supported?

  2. What are the main challenges of the problem and the solutions? Currently, the paper combines a series of straightforward techniques from existing works without noticeable modifications or advancements. Please pinpoint them in the paper clearly to better reveal your paper's strength and contributions.

Reviewer_BGuxOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies continual learning over binary segmentation tasks in which both the image domain and foreground change from task to task (camouflaged objects, polyps, underwater salient objects, transparent objects). Task identity is available during training, but at test time the model receives neither the task identity nor any point, box or text prompt. Their motivation study argues that this setting is harder than domain-only continual segmentation.

The framework keeps a frozen DINOv2 backbone and trains one parameter-isolated expert per task, so later tasks cannot overwrite earlier ones. To pick an expert without task identity, it adapts SOYO's selector. It is a task classifier on DINOv2 features, updated at each stage with pseudo-features sampled from statistical summaries of earlier tasks instead of stored data. At inference, one expert is activated per image.

Validation uses a single four-task stream (COD → Polyp → USOD → TOD). Routed inference comes close to using the ground-truth task identity (0.806 vs. 0.814 AA-mIoU) and matches offline joint training, while baselines (fine-tuning, LwF, EWC, experience replay) perform far worse. The paper also compares several selectors on the same expert bank, analyses which tasks get misrouted, and ablates the routing features, the replay mechanism and the adapters.

Critical strengths

1. The same frozen expert bank is evaluated with ground-truth task identity and with automatic routing, and the paper reports task-selection accuracy, the gap Δroute=AAoracle−AArouted\Delta_{\text{route}} = \mathrm{AA}_{\text{oracle}} - \mathrm{AA}_{\text{routed}}, a routing confusion matrix, and per-task routing penalties. This pins down the residual error precisely and is more informative than a single AA/FM pair.

2. Without storing any images or masks, the routed system is within 0.009 AA-mIoU of the oracle and on par with offline joint training on this stream. As a baseline for prompt-free multi-task binary segmentation, this is useful even though the individual components are not new.

Critical weaknesses
  1. The problem is only incrementally new, and the method does not engage with the part claimed to be new:
  • Continual binary segmentation with a changing foreground definition and automatic module selection is already studied in work the paper cites. CoSAM/MoDA (Yang et al., 2024) spans industrial and rail defects, camouflaged objects and polyps, selecting adapters by key matching. SAMCL (Wang et al., 2026b) runs Kvasir→CAMO→ISTD→ISIC→COD (polyp, camouflage, shadow, skin lesion) with a task-agnostic module selector trained on stored embeddings, under the same mIoU/mF1/mMAE and AA/FM protocol used here. CA-SAM routes exemplar-free across nine medical datasets with different targets. The appearance-plus-objective-shift framing is that of CGR (MICCAI 2024) and C2GR, and Spider (ICML 2024) unifies salient, camouflaged, transparent, shadow and polyp segmentation with a continual-learning study. Section 1 places CoSAM among works with a stable prediction formulation, and Sections 1–2 present SAMCL and CA-SAM only as diverse-task or modular methods, without noting that their streams already change the foreground target. Only C2GR is credited with concurrent appearance and objective shift; CGR and Spider are not cited. What remains new is the combination prompt-free (SAMCL uses three ground-truth-sampled points at test time) + exemplar-free (MoDA keeps 10 exemplars per task).
  • The selector is SOYO's compressor/resampler/fusion design with PCA added, the expert is a standard reassemble-and-fuse decoder with bottleneck adapters, and the preservation-vs-access decomposition is the within-task / task-ID prediction decomposition of Kim et al. (NeurIPS 2022) and HiDe-Prompt (NeurIPS 2023). Classic expert routing (Expert Gate, 2017; S-Prompts, 2022) is not discussed.
  1. The experiments do not isolate a contribution beyond "a frozen backbone with isolated experts works":
  • With the backbone and old experts frozen, each expert is identical to an independently trained one, so FM = 0.008 is purely routing error, and Table 1 mainly re-establishes that parameter isolation beats shared-pathway baselines (fine-tuning, LwF/EWC regularization, 40-pair replay).
  • One stream, one order, four tasks, and no seeds or variance, whereas CoSAM reports five seeds and a reversed order and SAMCL reports six orders. Selector differences are 0.001–0.012 AA-mIoU, so variance is needed to interpret Tables 2–4. Neither linear parameter growth nor routing under overlapping domains is tested. Each expert has 12.16M parameters (Table 4). The expert bank therefore exceeds the ~86M DINOv2-B/14 backbone after about seven tasks.
  • Real history outperforms distribution replay (0.943 / 0.811 vs. 0.927 / 0.806), and its storage is not reported.
Questions for authors

Clarifications that could change my assessment:

  1. Please add a table comparing this setting with CoSAM/MoDA, SAMCL, CA-SAM, C2GR and Spider on: foreground change across tasks, prompts at test time, task identity at test time, and stored raw exemplars. Which assumption does this paper relax that none of them does?
  2. How does the system behave when two tasks share an image domain but define foreground differently? Is the COD→USOD confusion concentrated in underwater COD images? If prompt-free routing is ill-posed in that regime, please match the claims to domain-identifiable task sets.
  3. Please report total stored bytes per method (experts included), ER at the same total memory as the expert bank, and MoDA / SAMCL / CA-SAM under your prompt-free, exemplar-free protocol.
  4. What does "Fusion + Real History" store, and how does a storage-matched version compare with distribution replay?
  5. Please report results over several seeds and task orders, and on a longer stream (e.g., 8 tasks), showing how task-selection accuracy, Δroute\Delta_{\text{route}} and total parameters evolve.
  6. What exactly are mIoU (foreground only or foreground/background; per image or per dataset) and AA-FG-IoU in Fig. 1? Is Joint Training a single pathway trained on pooled data? Which shared-path learner produced Fig. 1?
Reviewer_tQabOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies continual learning across binary segmentation tasks with different image domains and foreground definitions. Task identities are known during training but are unavailable at test time. The method uses a frozen DINOv2 backbone, a separate segmentation expert for each task, and a distribution-replay selector adapted from SOYO. It evaluates the system on a four-task stream and compares automatic routing with a task-ID oracle.

Critical strengths

The task-ID oracle and selector comparisons are useful diagnostics (Tables 1 and 2). They help separate the quality of the stored segmentation experts from errors caused by choosing the wrong expert. The reported results show that automatic routing can recover most of the oracle performance on the tested stream.

Critical weaknesses
  1. Limited technical novelty. The method mainly combines a frozen backbone, task-specific adapters and decoders, parameter isolation, and feature replay. These are familiar CL components. Section 3.4 explicitly follows the feature fusion and distribution-replay design of SOYO. The paper needs to identify a substantive technical contribution beyond assembling these components for segmentation.

  2. The segmentation-specific challenge is not established clearly. Sections 1 and 3.1 describe changes in both image domains and foreground definitions. This is a reasonable setting, but it is still unclear what makes it require a different solution from existing modular CL methods. The system preserves one expert per task and selects an expert from global image features. I would like to see which difficulty is specific to segmentation and which part of the method is needed to address it. The performance drop in Figure 1 alone does not answer this question.

  3. The comparisons are seriously insufficient. Table 1 mainly compares against FT, LwF, EWC, and ER, all using a single segmentation pathway. The proposed method adds a separate frozen expert for every task. This makes stronger modular baselines and matched parameter and storage budgets especially important. Relevant methods discussed in Section 2, such as MoDA, CUE, and SAMCL, are missing from the full-system comparisons. The SAMCL-style MLP in Table 2 is only a selector comparison. Moreover, a linear selector already reaches 0.801 AA-mIoU, compared with 0.806 for the proposed selector. These results do not yet establish a convincing advantage over existing approaches. Any comparison should handle differences in prompts and task-ID access explicitly.

  4. The figures are hard to read. The labels and text in Figures 2 and 4 are too small at normal page size. Figure 2 is particularly crowded, which makes the architecture harder to follow. Please enlarge the text and simplify the layouts. This is a presentation concern; my rejection recommendation is mainly driven by points 1–3.

Questions for authors
  1. What technical change is specifically required for heterogeneous binary segmentation, beyond using an existing parameter-isolation and task-selection framework with segmentation decoders? Which experiment shows that this change is necessary?

  2. Can you provide full-system comparisons with relevant modular or continual segmentation methods under compatible prompt and task-ID assumptions? Please account for the growing expert bank in the parameter and storage budgets. These comparisons would matter much more to my assessment than additional comparisons with basic shared-path CL methods.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.