ALME: Adaptive Low-Rank Model Editing for Multi-Domain Few-Shot Segmentation

In real-world scenarios, segmentation models often need to adapt continually to emerging specialized domains with only a few annotated examples. However, segmentation foundation models often perform poorly in such domains because they lack the required domain-specific knowledge. Continual adaptation can acquire this knowledge from limited data, but existing methods typically update the model sequentially across domains, which can interfere with previously acquired knowledge and lead to $\text{catastrophic forgetting}$. To address this problem, we propose $\text{A}$daptive $\text{L}$ow-Rank $\text{M}$odel $\text{E}$diting ($\text{ALME}$), which decouples domain-specific knowledge acquisition from cross-domain knowledge accumulation. ALME acquires knowledge for each domain independently from the same pretrained model through adaptive low-rank updates, thereby preventing adaptation to a new domain from directly overwriting knowledge acquired from earlier domains. It then calibrates and accumulates the acquired domain knowledge and integrates it with regularization anchored to the pretrained model, preserving previously acquired knowledge and pretrained capabilities while yielding a single deployable model. Experiments with SAM3 across 13 sequential segmentation domains in the 5-shot setting show that ALME achieves 76.81% mIoU, outperforming the strongest non-rehearsal continual learning baseline by 4.92 points.