acceptodds
Withdrawn from ICLR 2027

ALME: Adaptive Low-Rank Model Editing for Multi-Domain Few-Shot Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In real-world scenarios, segmentation models often need to adapt continually to emerging specialized domains with only a few annotated examples. However, segmentation foundation models often perform poorly in such domains because they lack the required domain-specific knowledge. Continual adaptation can acquire this knowledge from limited data, but existing methods typically update the model sequentially across domains, which can interfere with previously acquired knowledge and lead to . To address this problem, we propose daptive ow-Rank odel diting (), which decouples domain-specific knowledge acquisition from cross-domain knowledge accumulation. ALME acquires knowledge for each domain independently from the same pretrained model through adaptive low-rank updates, thereby preventing adaptation to a new domain from directly overwriting knowledge acquired from earlier domains. It then calibrates and accumulates the acquired domain knowledge and integrates it with regularization anchored to the pretrained model, preserving previously acquired knowledge and pretrained capabilities while yielding a single deployable model. Experiments with SAM3 across 13 sequential segmentation domains in the 5-shot setting show that ALME achieves 76.81% mIoU, outperforming the strongest non-rehearsal continual learning baseline by 4.92 points.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Few-Shot Segmentation, Segmentation Foundation Models, Model Editing, Low-Rank Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QG32OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Adaptive Low-Rank Model Editing (ALME) for few-shot continual cross-domain segmentation. The method decouples domain-specific knowledge acquisition from cross-domain knowledge accumulation. For each incoming domain, ALME independently adapts a pretrained segmentation foundation model using probe-guided low-rank adaptation. The acquired knowledge is then represented by layer-wise regression statistics, calibrated across domains, and integrated into a single model through pretrained-anchored regularized regression. Experiments are conducted on 13 sequential segmentation datasets using SAM3 under 1-shot, 5-shot, and 10-shot settings. The method achieves 76.81% mIoU in the 5-shot setting, outperforming the strongest non-rehearsal continual learning baseline by 4.92 percentage points.

Critical strengths
  1. Comprehensive experimental evaluation. The paper provides relatively comprehensive experiments across 13 segmentation datasets covering diverse visual domains. Comparisons include several categories of continual learning methods, such as rehearsal-based, regularization-based, and parameter-efficient adaptation approaches. The evaluation also considers different shot settings, multiple performance metrics, ablation studies, and additional analyses of model merging strategies, domain order, and regularization. These experiments provide useful empirical evidence for the effectiveness of the proposed method.

  2. A reasonably structured framework for knowledge accumulation. The proposed framework separates domain-specific knowledge acquisition from cross-domain knowledge integration, avoiding direct sequential updates to previously adapted model parameters. The use of adaptive low-rank updates and regression-based knowledge integration provides a coherent approach to accumulating knowledge from multiple domains. The framework also produces a single deployable model without requiring domain-specific adapters at inference.

Critical weaknesses
  1. Insufficient motivation and unclear significance of the proposed problem setting. The motivation for the proposed few-shot continual cross-domain segmentation setting is not sufficiently established. The paper essentially extends existing domain-incremental segmentation protocols, particularly SAMCL, by restricting each incoming domain to a few annotated samples. However, the authors do not convincingly explain why this particular combination of few-shot learning and continual domain adaptation is necessary or practically important.

More importantly, the paper does not clearly identify the unique challenges introduced by the few-shot constraint beyond the well-established catastrophic forgetting problem in continual learning. The motivation largely relies on general arguments regarding annotation costs and sequentially arriving domains, without sufficient empirical evidence or concrete application scenarios to establish the necessity of this specific setting. Consequently, the significance and practical value of the proposed problem formulation remain unclear.

  1. Questionable evaluation protocol involving ground-truth masks at test time. A more fundamental concern lies in the evaluation protocol. In Section 4.1, the authors explicitly state that three positive point prompts are sampled from the ground-truth masks during both adaptation and evaluation. This indicates that ground-truth information from test images is used to construct the prompts for inference.

Although ground-truth-derived prompts can be used in controlled evaluations of interactive segmentation models, their use is highly questionable in the context of the proposed few-shot continual adaptation setting. Such prompts provide oracle foreground locations that would not normally be available during real-world inference without additional annotation or user interaction. As a result, the reported performance does not directly reflect the model's ability to segment unseen images without oracle guidance.

The fact that the same prompting protocol is applied to all compared methods does not resolve this concern, as it affects the practical interpretation of the entire evaluation. The authors should clearly justify this assumption and provide evaluations using realistic inference-time prompts or a prompt-free protocol. Without such evidence, the practical significance of the reported improvements remains uncertain.

Questions for authors
  1. Missing comparisons with cross-domain few-shot segmentation methods. Although the proposed setting involves few-shot adaptation across substantially different visual domains, the experimental comparisons focus almost exclusively on continual learning approaches. Relevant cross-domain few-shot segmentation (CD-FSS) methods are not included. While conventional CD-FSS methods may not directly support continual learning, comparisons under an appropriately adapted or clearly defined evaluation protocol would help establish the benefits of ALME over existing few-shot domain adaptation techniques.

  2. Limited analysis of intermediate performance under different domain orders. The authors investigate domain-order sensitivity in Section 4.4 and Figure 2. However, the analysis mainly focuses on final performance after all domains have been incorporated. Given the additive and order-independent nature of the proposed knowledge integration, identical final results across different domain orders are largely expected by construction.

In continual learning, performance throughout the adaptation sequence is also important. It would be useful to examine how different domain orders affect intermediate-stage performance, particularly when domains with substantially different visual characteristics are introduced consecutively.

  1. Insufficient analysis of computational and storage efficiency. The method involves an additional probing stage, domain-specific low-rank adaptation, extraction of regression statistics, and closed-form knowledge integration. Although the paper emphasizes compact knowledge accumulation and single-model deployment, the practical trade-offs associated with these additional operations are not sufficiently emphasized in the main evaluation. A more direct comparison of training costs, integration overhead, persistent storage, and inference efficiency against existing continual adaptation methods would strengthen the claimed practical advantages.

  2. Limited clarification of domain shifts versus task differences. The 13 evaluated datasets involve not only different image distributions but also substantially different segmentation objectives, including medical lesions, camouflaged objects, shadows, and structural regions. The paper does not clearly distinguish whether its setting primarily addresses domain-incremental adaptation, task-incremental segmentation, or a mixture of the two. A more precise problem definition would help clarify the scope of the proposed method and the interpretation of the experimental results.

Reviewer_Got4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies continual few-shot adaptation of segmentation foundation models to sequentially arriving specialized domains without retaining historical raw samples. ALME separates domain-specific knowledge acquisition from cross-domain integration. Each domain is adapted independently from a shared pretrained model, with layer-wise LoRA ranks allocated using the singular spectra of short probe updates. The acquired models are summarized through input–output regression statistics, which are calibrated by domain-level feature energy and accumulated to fit a single shared model with regularization toward the pretrained parameters.

Critical strengths
  1. A coherent separation of acquisition and integration with a practical deployment benefit. I find the separation between independently acquiring domain knowledge and subsequently integrating it into a shared model useful. Starting each acquisition from the same pretrained anchor avoids making that acquisition depend on the previously integrated model. Writing the integrated parameters back into the original projections also preserves a single inference architecture without requiring an expanding collection of domain-specific branches or a routing mechanism. These are consequential advantages for deployment and for settings where historical raw images cannot be retained.
  2. Broad empirical coverage and several informative controlled comparisons. The evaluation spans 13 specialized domains and multiple support-set sizes. I particularly appreciate that Table 4 applies different integration strategies to the same acquired models, reducing acquisition quality as a confounding factor. Table 5 matches trainable parameter counts, and Table 6 isolates the effect of statistic calibration. Together with the additional backbone experiments, these results provide meaningful evidence that the complete pipeline improves final prompted-segmentation performance.
Critical weaknesses
  1. The experiments do not yet sufficiently isolate the contribution of the proposed mechanisms. Table 4 controls the acquired models, but ALME’s integration combines energy calibration, a pretrained-parameter anchor, and joint weight/bias fitting. The comparison does not establish how much of the improvement over RegMean comes from each of these choices under matched treatment and tuning. Table 6 is informative about calibration within ALME, but does not resolve the remaining interactions. Similarly, Table 5 matches parameter counts, while spectral allocation requires an additional probe; an equal-total-compute uniform-rank comparison is missing. The probe configuration and rank-dependent LoRA scaling are also insufficiently specified. This matters to my assessment because the current results establish the effectiveness of the combined pipeline more clearly than the distinct contribution and cost-effectiveness of its individual mechanisms.
  2. The evidence for knowledge preservation is less complete than the final average suggests. The final mIoU improvement is meaningful, but the continual-learning metrics show a more mixed picture: ALME’s AIA is 78.56 ± 2.90 versus 79.43 ± 1.23 for DER++, and its FWT is −0.73 ± 2.57 versus 3.22 ± 0.89 for SAMCL. These comparisons do not invalidate the final gain, but they indicate trade-offs that deserve clearer analysis. Positive average BWT alone does not establish that individual earlier domains avoid degradation. I would like to see the domain-by-stage performance matrix before drawing a strong conclusion about retention. From an object detection perspective, I am particularly attentive to improvements in an aggregate score that may conceal failures on specific domains or difficult foreground structures. In addition, the COCO point-prompt result supports preservation on that reference task, rather than preservation of the foundation model’s capabilities more generally.
Questions for authors
  1. Contribution attribution — evidence that could affect my assessment. Can the authors separate energy calibration, pretrained-anchor regularization, and bias fitting through controlled combinations, including a RegMean comparison with matched bias handling and numerical stabilization? Please also clarify the hyperparameter-selection budgets used for the merging baselines. These results would help determine which parts of the proposed integration account for the improvement.
  2. Spectral allocation — implementation clarification and a budget comparison. What are the probe optimizer, number of steps, learning rate, rank bounds, LoRA initialization, and scaling convention? Was the uniform-rank baseline given the same total optimization budget, including the probe cost? If not, an equal-total-compute comparison would materially help assess the benefit of spectral allocation.
  3. Retention and evaluation protocol — clarification and supporting evidence. Can the authors provide the domain-by-stage performance matrix and explain the differences between final mIoU, AIA, and FWT? Are point coordinates fixed across methods for each evaluation sample, and how are multiple instances, connected components, and candidate masks handled? This would help distinguish changes in retained domain performance from effects of the prompting and mask-selection protocol.
Reviewer_wL8iOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses few-shot continual cross-domain adaptation of segmentation foundation models. It argues that sequential model updates entangle domain-specific acquisition with cross-domain accumulation and cause interference. ALME decouples these steps: each domain is adapted independently from the same pretrained anchor, summarized by compact regression statistics, calibrated, and integrated into one deployable model through pretrained-anchored affine regression. The contributions are the problem formulation, the decoupled pipeline without raw-data replay or domain-specific inference modules, and empirical evidence of preserved acquired and pretrained capabilities. On SAM3 across thirteen domains, ALME reaches 76.81% mIoU in the five-shot setting, outperforming the strongest non-rehearsal baseline by 4.92 points while retaining COCO performance. Ablations and SAM/SAM2 extensions support the design.

Critical strengths
  • The method separates per-domain acquisition from cross-domain integration. Each domain is adapted independently from a common pretrained anchor. Its knowledge is compressed into compact statistics. These statistics are then combined through a closed-form solution regularized toward the pretrained model. This design avoids replay of raw data. It also removes domain-specific inference modules. Compared with standard sequential-update continual learning, it offers a clear practical advantage.

  • The empirical evaluation is broad enough to substantiate the main claims. It spans 13 segmentation domains across multiple shot settings. Ablations cover the acquisition reference, statistic calibration, rank allocation, and integration strategy. These ablations help attribute the gains to the proposed design. They mitigate the concern that SAM3 alone explains the results. Overall, the evidence makes the reported improvements credible.

Critical weaknesses
  • The paper combines LoRA, SVD-based rank allocation, RegMean-style sufficient statistics, and pretrained-anchored ridge regression. Eqs. 6 to 8 form a ridge regression problem with a pretrained anchor regularizer, whose solution is a standard normal-equation closed form. This is mathematically close to RegMean and related regression-based merging methods. The paper describes procedural differences from RegMean, but procedural description is not a substitute for ablation or theoretical argument. Without factorial ablation or analysis, whether the core contribution is a principled new method remains unclear. This directly affects the acceptance judgment.

  • Sec. 4.1 states that three positive point prompts are sampled within the ground-truth mask during both adaptation and evaluation. The evaluation therefore uses oracle prompting. Relative comparisons under a shared protocol remain informative, but absolute mIoU should not be read as real deployment performance. The main text reports no robustness tests with fixed grid points, center points, predicted-mask prompts, noisy prompts, or negative prompts. This materially weakens the deployment-value claim, since prompt quality strongly affects segmentation foundation models.

Questions for authors

(1) Sec. 4.1 states that three positive point prompts are sampled within the ground-truth mask during both adaptation and evaluation. Please confirm whether evaluation prompts are indeed sampled from ground-truth masks. If so, do you have results using fixed grid points, center points, predicted-mask prompts, or noisy/negative prompts? This materially affects the absolute performance and deployment-value claims.

(2) The integration step in Eqs. 6 to 8 is a pretrained-anchored ridge regression with a standard closed-form solution. Please clarify which components of ALME are absent from the RegMean baseline in Table 4. Specifically, does the RegMean baseline use the same affine bias fitting, inverse-trace calibration, pretrained anchor regularization, and rank allocation? A factorial ablation isolating these factors would materially affect the novelty assessment.

(3) Please report per-domain results and variance for Table 1 and Table 2, especially the worst-performing domains and intermediate stages. This would clarify whether the gains over the strongest non-rehearsal baseline are uniform across domains or driven by a few datasets.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.