- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a LoRA-based framework for class-incremental semantic segmentation that stays parameter-efficient across all steps, including the base step. The authors argue that distillation-based methods are costly, and that ECLIPSE is only efficient in the incremental steps because it fully trains the base step. The method has three main parts. Gradient subspace projection: Before each incremental step, the uncentered covariance of the inputs to each LoRA-adapted layer is computed using GT-based attention masks. The top- singular vectors (cumulative energy ) define a projector , and the AdamW update directions of the LoRA factors are projected onto it during training. Pixel-prototype loss: A hard-negative contrastive loss and a margin loss are computed on class-balanced sampled pixels. Background classifier calibration: The background classifier weights are averaged across steps and rescaled by a factor . Experiments are reported on ADE20K and Pascal VOC 2012 with a Mask2Former (Swin-S/B) model, together with a small study of LoRA fine-tuning for panoptic segmentation.
Critical strengths
- The motivation is reasonable. Keeping the whole continual learning pipeline parameter-efficient, including the base step, is a practical goal as segmentation models get larger.
- Because the LoRA adapters are merged after each step, the model size does not grow with the number of steps. This is a nice property compared to methods that keep adding prompts or modules.
- The method is simple and the paper gives implementation details (learning rates, iterations, hyperparameters) that should help reproduction.
- Each component is ablated (Tables 4-6). The appendix also includes some extra analysis, such as the effect of refactoring the attention for PEFT, cross-step vs. intra-step prototypes, and the behavior of .
Critical weaknesses
1. The evaluation protocol is not comparable to the baselines. In Sec. 5.1 the authors say that the ADE20K class order is shuffled with random seed 42. However, as far as I can tell, all baseline numbers in Table 1 come from the standard class order used in prior work. The class order strongly changes how hard the base/incremental split is. I think this also explains the unusual pattern in Table 1: the proposed method gets very high mIoU on the new classes (41-48, close to Joint) but much lower mIoU on the base classes than all baselines. Because of this, I do not think Table 1 supports the claims in the paper. Results under the standard class order, or baselines re-run with the same shuffled order, are needed.
2. Recent state-of-the-art baselines are missing. BalConpas (ECCV'24), CoMBO (CVPR'25), and SimCIS (CVPR'25) are cited in the paper but not compared in Table 1. Using their reported numbers, the Swin-S variant is worse than all three in every setting. The Swin-B variant is also worse in most settings. For example, in 100-50 the "all" mIoU is 41.6 vs. 44.9 (CoMBO) and 48.6 (SimCIS). In 100-10 it is 39.6 vs. 41.1 (CoMBO) and 42.3 (SimCIS). So the claim that the method "outperforms" distillation-based methods does not hold.
I also noticed that the baseline rows in Table 1 match the table in the CoMBO paper, but BalConpas and CoMBO, which appear in the same table, are not included. I would like the authors to explain this. In addition, ADAPT (ICLR'25), which directly targets efficient continual panoptic segmentation, is not cited.
- BalConpas: Chen, Jinpeng, et al. "Strike a balance in continual panoptic segmentation." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.
- CoMBO: Fang, Kai, et al. "Combo: Conflict mitigation via branched optimization for class incremental segmentation." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.
- SimCIS: Zhu, Yuchen, et al. "Rethinking query-based transformer for continual image segmentation." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.
- ADAPT: Yang, Ze, et al. "ADAPT: Attentive Self-Distillation and Dual-Decoder Prediction Fusion for Continual Panoptic Segmentation." International Conference on Learning Representations. Vol. 2025. 2025.
3. Overlap with prior work in the same line, and limited novelty.
- ADAPT already pointed out the efficiency problem of distillation-based methods. It freezes most of the model so that the teacher and student share the forward pass, with a similar number of trainable parameters. This directly overlaps with the main motivation of this paper (Figure 1), which assumes that distillation always needs two forward passes.
- CoMBO already uses class-specific low-rank adapters for continual segmentation.
- The gradient subspace projection is mostly InfLoRA / LoRA Subtraction applied to segmentation. The only clear difference is using GT-based attention masks when collecting the statistics.
Without comparison to or discussion of these works, it is hard to see what the actual contribution is.
4. Unfair pre-trained initialization. To my knowledge, prior methods (CoMFormer, ECLIPSE, BalConpas, CoMBO, ADAPT) start from an ImageNet pre-trained backbone and train the segmentation model during the base step. In contrast, this paper initializes from a full Mask2Former (including pixel decoder and transformer decoder) that is already trained for segmentation: on Cityscapes for the ADE20K experiments and on ADE20K for the VOC experiments. The backbone (Swin) is also different from the baselines (ResNet).
Table 8 shows that the initialization alone makes a large difference (VOC 15-5: 77.1 vs. 89.2 mIoU). So it is not possible to separate the gain of the proposed method from the gain of the stronger initialization. Also, the main claim of being "parameter-efficient from the base step" relies on starting from an already trained segmentation model. I do not think this is the same condition as prior methods that train the base step from an ImageNet backbone.
Questions for authors
- Can you report ADE20K results under the standard class order used by prior work, so that the numbers in Table 1 can be compared directly?
- Why are BalConpas, CoMBO, and SimCIS not included in Table 1 even though they are cited? Could you also compare with ADAPT, both in accuracy and in training cost?
- What is the performance when the method starts from an ImageNet pre-trained backbone (ResNet-101, same as the baselines) and trains the base step with LoRA? If LoRA on a randomly initialized decoder does not work well, how does this affect the "full step parameter-efficient" claim?
- The paper only reports LoRA fine-tuning vs. full training for panoptic segmentation in a non-continual setting (Table 3). Why is there no continual panoptic segmentation comparison like Table 1 (e.g., ADE20K 100-50 / 100-10 / 100-5 / 50-n in PQ) against CoMFormer, ECLIPSE, BalConpas, CoMBO, SimCIS, and ADAPT? Since most recent Mask2Former-based methods report this benchmark, it seems necessary to support the general "continual image segmentation" claim in the title.