- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies class-incremental semantic segmentation without replaying samples from previous tasks. It proposes AKI, a cross-layer knowledge reorganization framework, to mitigate forgetting when learning new classes. AKI comprises three components: RCA uses weight norms to construct response masks and encourages consistent predictions between the full and compact branches; RPA derives class-specific rank-one feature-alignment directions by considering both previously learned knowledge and the needs of the current task; and DEP models class distributions to encourage separation between old and new classes.
Critical strengths
- The problem is important, and the overall design is easy to follow. RCA, RPA, and DEP target forgetting at the response, representation, and class-distribution levels, respectively. The three components appear to play complementary roles.
- The evaluation is fairly extensive. Besides the main results and ablations, the appendix covers the disjoint setting, different backbones, and compatibility with other frameworks. Table 10 also includes random rank-one and simplified-direction controls, which help clarify the role of RPA.
- The derivation of RPA is presented in reasonable detail. The use of a truncated pseudoinverse in the effective feature subspace is explained, and the assumptions and limitations of the analysis are discussed. This gives more support to the projection design than simply introducing it as a heuristic.
Critical weaknesses
- The comparability of the baseline results needs clarification. The paper uses MBS† for comparison, but its differences from the original MBS are not sufficiently clear. If the architecture, training budget, or pseudo-labeling strategy has changed, the impact of those changes should be reported. Otherwise, it is difficult to attribute the improvement to AKI itself.
- The random-mask control for RCA does not isolate the effect of mask selection. According to Appendix C.9, the random mask stays fixed throughout an incremental step, whereas the norm-based mask in RCA is updated during training. Moreover, the number of suppressed responses is matched only at the beginning of the step. The differences in Table 13 may therefore reflect both mask selection and the update strategy. A dynamically updated random mask and a fixed norm-based mask, with layerwise mask counts matched, would make the comparison more informative.
- The actual optimization signal from DEP is not yet clear. DEP computes the Bhattacharyya coefficient in a 768-dimensional feature space. Although working in the log domain helps with intermediate numerical stability, the exponentiated coefficient and its gradients may still be very small. Reporting the range of the DEP loss and its gradient magnitudes during training would help establish whether it provides a meaningful learning signal.
- The independent contribution of RPA needs stronger evidence. SPPA, which is already cited in the paper, also uses low-dimensional projection. The distinction between the two methods and the claimed advantage of RPA over a learned projection should be made clearer. The random rank-one control in Table 10 is useful, but it is not a direct comparison with learned projection methods. In addition, Table 9 reports 79.3 for full AKI and 79.2 for RCA+DEP on VOC 15-1, a gain of only 0.1 mIoU. Results over multiple runs, including the mean and standard deviation, would help determine whether this difference exceeds normal experimental variation.
Questions for authors
- What exactly differs between MBS† and the original MBS? How much does each change affect performance? Does AKI yield a comparable gain when the original MBS training configuration is kept unchanged?
- If the random mask is updated during training and the number of suppressed responses is matched to RCA at each layer throughout training, does the conclusion in Table 13 still hold? Conversely, what happens if the norm-based RCA mask is kept fixed?
- How does RPA differ from learned projection methods such as SPPA in its objective, projection rank, and the feature layers to which it is applied? Do the current ablations adequately support its independent contribution?
Thanks for the author's detailed rebuttal and additional experiments. I retain the initial rating.