acceptodds
Under review as a conference paper at ICLR 2027

Full Step Parameter-Efficient Continual Image Segmentation via LoRA Adaptation

Abstract

In recent years, prevailing class-incremental image segmentation approaches have heavily relied on knowledge distillation—including logit distillation, feature distillation, and pseudo-labeling—which incur substantial training overhead. Meanwhile, vision segmentation models are rapidly increasing in scale; this discrepancy presents a severe dilemma, as heavy distillation paradigms fail to scale sustainably to large foundation models. In this paper, we propose a full step parameter-efficient continual image segmentation framework based on Low-Rank Adaptation (LoRA). Our approach enables lightweight and efficient adaptation of pre-trained segmentation models to novel classes from the base step through all subsequent incremental steps. Specifically, we devise a task-specific gradient subspace projection mechanism to minimize mutual interference among LoRA adapters across different steps, thereby mitigating catastrophic forgetting. Furthermore, a background classifier calibration strategy is introduced to alleviate background drift, complemented by a prototype-based loss that widens class-wise separation to suppress category confusion. Extensive experiments on ADE20K and Pascal VOC 2012 demonstrate the effectiveness of our method for semantic segmentation. In addition, we present an exploratory investigation into the applicability of LoRA fine-tuning for panoptic segmentation.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Image Segmentation, Parameter Efficient Fine-tuning, Gradient Subspace Projection

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_SHbmOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a LoRA-based framework for class-incremental semantic segmentation that stays parameter-efficient across all steps, including the base step. The authors argue that distillation-based methods are costly, and that ECLIPSE is only efficient in the incremental steps because it fully trains the base step. The method has three main parts. Gradient subspace projection: Before each incremental step, the uncentered covariance of the inputs to each LoRA-adapted layer is computed using GT-based attention masks. The top-kk singular vectors (cumulative energy ρ=0.99\rho = 0.99) define a projector PtlP_t^l, and the AdamW update directions of the LoRA factors are projected onto it during training. Pixel-prototype loss: A hard-negative contrastive loss and a margin loss are computed on class-balanced sampled pixels. Background classifier calibration: The background classifier weights are averaged across steps and rescaled by a factor γ\gamma. Experiments are reported on ADE20K and Pascal VOC 2012 with a Mask2Former (Swin-S/B) model, together with a small study of LoRA fine-tuning for panoptic segmentation.

Critical strengths
  1. The motivation is reasonable. Keeping the whole continual learning pipeline parameter-efficient, including the base step, is a practical goal as segmentation models get larger.
  2. Because the LoRA adapters are merged after each step, the model size does not grow with the number of steps. This is a nice property compared to methods that keep adding prompts or modules.
  3. The method is simple and the paper gives implementation details (learning rates, iterations, hyperparameters) that should help reproduction.
  4. Each component is ablated (Tables 4-6). The appendix also includes some extra analysis, such as the effect of refactoring the attention for PEFT, cross-step vs. intra-step prototypes, and the behavior of γ\gamma.
Critical weaknesses

1. The evaluation protocol is not comparable to the baselines. In Sec. 5.1 the authors say that the ADE20K class order is shuffled with random seed 42. However, as far as I can tell, all baseline numbers in Table 1 come from the standard class order used in prior work. The class order strongly changes how hard the base/incremental split is. I think this also explains the unusual pattern in Table 1: the proposed method gets very high mIoU on the new classes (41-48, close to Joint) but much lower mIoU on the base classes than all baselines. Because of this, I do not think Table 1 supports the claims in the paper. Results under the standard class order, or baselines re-run with the same shuffled order, are needed.

2. Recent state-of-the-art baselines are missing. BalConpas (ECCV'24), CoMBO (CVPR'25), and SimCIS (CVPR'25) are cited in the paper but not compared in Table 1. Using their reported numbers, the Swin-S variant is worse than all three in every setting. The Swin-B variant is also worse in most settings. For example, in 100-50 the "all" mIoU is 41.6 vs. 44.9 (CoMBO) and 48.6 (SimCIS). In 100-10 it is 39.6 vs. 41.1 (CoMBO) and 42.3 (SimCIS). So the claim that the method "outperforms" distillation-based methods does not hold.

I also noticed that the baseline rows in Table 1 match the table in the CoMBO paper, but BalConpas and CoMBO, which appear in the same table, are not included. I would like the authors to explain this. In addition, ADAPT (ICLR'25), which directly targets efficient continual panoptic segmentation, is not cited.

  • BalConpas: Chen, Jinpeng, et al. "Strike a balance in continual panoptic segmentation." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.
  • CoMBO: Fang, Kai, et al. "Combo: Conflict mitigation via branched optimization for class incremental segmentation." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.
  • SimCIS: Zhu, Yuchen, et al. "Rethinking query-based transformer for continual image segmentation." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.
  • ADAPT: Yang, Ze, et al. "ADAPT: Attentive Self-Distillation and Dual-Decoder Prediction Fusion for Continual Panoptic Segmentation." International Conference on Learning Representations. Vol. 2025. 2025.

3. Overlap with prior work in the same line, and limited novelty.

  • ADAPT already pointed out the efficiency problem of distillation-based methods. It freezes most of the model so that the teacher and student share the forward pass, with a similar number of trainable parameters. This directly overlaps with the main motivation of this paper (Figure 1), which assumes that distillation always needs two forward passes.
  • CoMBO already uses class-specific low-rank adapters for continual segmentation.
  • The gradient subspace projection is mostly InfLoRA / LoRA Subtraction applied to segmentation. The only clear difference is using GT-based attention masks when collecting the statistics.

Without comparison to or discussion of these works, it is hard to see what the actual contribution is.

4. Unfair pre-trained initialization. To my knowledge, prior methods (CoMFormer, ECLIPSE, BalConpas, CoMBO, ADAPT) start from an ImageNet pre-trained backbone and train the segmentation model during the base step. In contrast, this paper initializes from a full Mask2Former (including pixel decoder and transformer decoder) that is already trained for segmentation: on Cityscapes for the ADE20K experiments and on ADE20K for the VOC experiments. The backbone (Swin) is also different from the baselines (ResNet).

Table 8 shows that the initialization alone makes a large difference (VOC 15-5: 77.1 vs. 89.2 mIoU). So it is not possible to separate the gain of the proposed method from the gain of the stronger initialization. Also, the main claim of being "parameter-efficient from the base step" relies on starting from an already trained segmentation model. I do not think this is the same condition as prior methods that train the base step from an ImageNet backbone.

Questions for authors
  1. Can you report ADE20K results under the standard class order used by prior work, so that the numbers in Table 1 can be compared directly?
  2. Why are BalConpas, CoMBO, and SimCIS not included in Table 1 even though they are cited? Could you also compare with ADAPT, both in accuracy and in training cost?
  3. What is the performance when the method starts from an ImageNet pre-trained backbone (ResNet-101, same as the baselines) and trains the base step with LoRA? If LoRA on a randomly initialized decoder does not work well, how does this affect the "full step parameter-efficient" claim?
  4. The paper only reports LoRA fine-tuning vs. full training for panoptic segmentation in a non-continual setting (Table 3). Why is there no continual panoptic segmentation comparison like Table 1 (e.g., ADE20K 100-50 / 100-10 / 100-5 / 50-n in PQ) against CoMFormer, ECLIPSE, BalConpas, CoMBO, SimCIS, and ADAPT? Since most recent Mask2Former-based methods report this benchmark, it seems necessary to support the general "continual image segmentation" claim in the title.
Reviewer_Jkv8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes class-incremental semantic segmentation using LoRA to adapt a pretrained Mask2Former throughout training, including the base step. It combines gradient projection onto subspaces estimated from current-step features, a pixel-prototype loss to separate classes, and background classifier calibration to address background drift. Together, these components aim to reduce forgetting without knowledge distillation. Experiments on ADE20K and Pascal VOC 2012 report competitive or higher mIoU against the compared baselines, with ablations examining each component’s contribution.

Critical strengths

The VOC analysis reports only 13.9% trainable parameters throughout learning. On ADE20K 100–50, using LoRA instead of full base-step training reduces final overall mIoU by just 0.4 points (Table 10). This supports a useful trade-off: substantially fewer trainable parameters with little loss in accuracy.

Critical weaknesses

(1) The approach uses segmentation-pretrained checkpoints and Swin backbones, but the paper does not report baselines with matched backbones and initialization. Table 8 shows that changing pretraining alone raises VOC 15–5 overall mIoU by 12.1 points. This makes it difficult to attribute the reported advantage over prior methods to the proposed continual-learning design.

(2) Peak memory and total training time are missing. These costs matter because scalability is the paper’s central motivation. Without measurements covering the complete pipeline and competing methods, the practical benefit remains uncertain.

Questions for authors

(1) Were the baselines in Tables 1–2 evaluated with the same backbone, segmentation pretraining, and class order? What evidence separates the method’s contribution from these differences, given the 12.1-point(89.2-77.1) pretraining effect in Table 8?

(2) How were γ and the choice between cross-step and intra-step prototypes selected? Which validation annotations were available, particularly for previously learned and future classes?

(3) How much of the retention gain in Table 4 comes from restricting update directions versus reducing update magnitude? Does the gain persist against an unprojected LoRA baseline with comparable update magnitudes?

Reviewer_6YS3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes full-step parameter-efficient continual semantic segmentation with LoRA, activation-based update projection, a pixel-prototype loss, and background-classifier calibration. Experiments cover ADE20K and Pascal VOC, with an additional panoptic study. However, unclear baseline alignment and insufficiently controlled ablations prevent the results from establishing the claimed advantage.

Critical strengths
  1. Parameter-efficient adaptation from the base step is a relevant practical goal.
  2. The component ablations report improvements within the authors' experimental setup.
Critical weaknesses
  1. The main results are not shown to be comparable. Section 5.1 shuffles ADE20K classes, while Table 1 matches ECLIPSE's published results under a fixed-order public configuration. Backbone and pretraining also differ; Table 8 alone shows a 12.1-point gain from changing initialization.
  2. The figures do not adequately explain the method. Figure 1 substitutes labels such as 'Efficient backward' for an explanation of the computational savings. In Figure 2, the classifier-merging text is difficult to read at normal page size, while the projection schematic lacks a clear correspondence to Equations (9) and (10). The AI-use statement (p. 9) mentions generated visualization scripts, but does not establish that Figures 1 and 2 were generated by AI. The assessable weakness is their limited technical clarity and insufficient connection to the algorithm.
  3. The GP ablation confounds projection with update scaling. Equation (9) changes both direction and magnitude, whereas Table 4 removes both together. A scale-matched control is needed before attributing the gain to the proposed subspace mechanism.
  4. The efficiency claim lacks end-to-end evidence. Table 12 reports fewer trainable parameters and roughly 10% lower FLOPs, but does not clearly account for the statistics pass and SVD. Peak memory and measured training time are also missing.
  5. The writing remains insufficiently checked despite the stated verification of AI-assisted content (p. 9). The authors disclose AI assistance with translation and revision, yet repeated editing artifacts remain, including 'Eq. equation 4' (p. 5, line 242) and 'Eq. equation 6' (p. 13, line 701). Appendix C discusses 15-5 and 15-1 while Table 11 reports 19-1 and 15-5 (p. 15), and the asserted convergence to a 'tuning-free regime' exceeds the evidence shown. These are concrete editorial and evidential weaknesses; they do not independently identify which passages were AI-generated.
  6. Citation accuracy and attribution need correction in three places. On p. 7, Section 5.1 cites Balazevic et al. (2023), 'Towards In-context Scene Understanding', for ADE20K instead of the original dataset publication. On p. 13, the GR-LoRA author field is malformed as 'Fengqiang Wan Yipeng Lin and Yang Yang'; the correct author order is Yipeng Lin, Fengqiang Wan, and Yang Yang, and the p. 3 citation 'Yipeng Lin & Yang (2026)' consequently needs correction. On p. 3, related work calls Zhu et al. (2025)'s method 'SimCLS' rather than 'SimCIS'. These errors hinder accurate attribution and show insufficient bibliographic checking. They are correctable citation defects, not evidence of fabricated papers.
Questions for authors
  1. Were the ECLIPSE results in Table 1 rerun with the shuffled ADE20K class order stated in Section 5.1? What exact class order was used for each method?
  2. Does the reported advantage persist when the proposed method and the baselines use the same backbone, segmentation-pretraining checkpoint, and comparable training budget?
  3. Does GP still improve performance over ordinary LoRA with matched update scaling and a properly tuned learning rate?
  4. Which LoRA factor and feature dimension are projected in each adapted layer, and what ranks are retained at the stated threshold of 0.99?
  5. How is LoRA injected into native MultiheadAttention, and what checks confirm that the intended adapters participate in the forward pass and receive gradients?
  6. How are the binary ground-truth masks in Equations (5) and (6) converted to the attention-mask convention in Equation (4), particularly for background and ignored pixels?
  7. What validation data select gamma, and where are the 15-1 results supporting the near-unity claim in Appendix C, given that Table 11 reports 19-1 and 15-5?
  8. What are the prototype momentum and hard-negative count, and how are negative prototypes selected when an incremental step introduces only one class?
  9. How were backward FLOPs measured in Table 12, and were the statistics pass, SVD, and projection included? What are the corresponding measured training time and peak memory?
  10. How stable are the gains across random seeds and class orders, and what do per-step old- and new-class results show beyond the final overall mIoU?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.