- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Role-Decoupled Memory (RDM) to address limited exemplar storage in Class-Incremental Learning (CIL). RDM separates memory into high-fidelity exemplars for representation learning and compressed samples for classifier calibration. To preserve important decision boundaries under compression, the authors introduce an AGOP-based sensitivity metric and develop two compression methods, AGOP-PCA and AGOP-J2K. Experiments on CIFAR-100, TinyImageNet, and ImageNet-100 show that RDM improves distribution coverage and CIL performance under fixed memory budgets.
Critical strengths
-
The paper distinguishes the different memory requirements of representation learning and classifier calibration, offering a fresh perspective on memory allocation in CIL.
-
The AGOP-based sensitivity metric provides a common principle for guiding both AGOP-PCA and AGOP-J2K compression, giving the framework a coherent theoretical foundation.
-
Evaluations across three benchmarks, supported by multiple runs, ablations, and memory allocation analyses, provide convincing evidence of the method's effectiveness.
Critical weaknesses
-
RDM combines role-decoupled memory, classifier calibration, and AGOP-guided compression. Without a baseline using the same calibration stage with raw exemplars, it is unclear which component drives the reported gains.
-
The proposed sensitivity metric lacks comparisons with simpler gradient-based alternatives. Moreover, AGOP-J2K offers only marginal gains over standard JPEG2000, raising questions about the need for additional complexity.
-
Experiments focus on ResNet-18 trained from scratch, without evaluating pretrained models or Vision Transformers. The lack of runtime, memory, and computational cost analysis also makes the practical benefits difficult to assess.
Questions for authors
-
RDM introduces both compressed memory and an additional calibration stage. How much improvement comes from calibration alone? A comparison with standard ER using identical calibration on its existing exemplars would help isolate the contribution of memory design.
-
Why are prototype-margin gradients preferable to simpler alternatives, such as logit or cross-entropy gradients? Given the modest gains of AGOP-J2K over standard JPEG2000, is the added complexity justified?
-
AGOP computation, latent refinement, and classifier calibration introduce additional overhead. Could the authors report training time and GPU costs relative to strong baselines to clarify the accuracy-efficiency trade-off?
-
The baselines are modified under a unified ER framework. Do these changes preserve their original performance, and are the reported gains robust to differences in calibration and training settings?
6 comments
Response to Reviewer GMQ6 (W2 & Q2)
W2 & Q2. Prototype-Margin Sensitivity and the Value of AGOP
Why prototype-margin gradients? Our compression objective is to preserve classification-relevant geometry of historical feature distributions for subsequent classifier calibration. The gradient of prototype margin measures how input reconstruction errors perturb separation along feature-space class contrasts. It provides a feature-geometric sensitivity signal that helps preserve useful historical support for a subsequently refitted classifier, without requiring an already calibrated or unbiased classifier. In contrast, logit and cross-entropy gradients depend on the current classifier weights. As discussed in previous work and our paper, an uncalibrated classifier may exhibit substantial old–new bias in CIL. Using the gradients of such classifier to allocate compression capacity can inherit the very misalignment that calibration seeks to correct, which runs counter to our objective.
Does AGOP provide measurable benefits? Our central contributions are the role-dependent fidelity–coverage trade-off and the resulting role-decoupled memory framework, with AGOP further improving its compressed calibration component. Table 2 already isolates AGOP from standard J2K. To better illustrate the benefit of AGOP, we further examine the paired standard-J2K vs. AGOP-J2K final-accuracy differences over the same five class orders:
| Dataset | Mean paired gain (pp) | Positive runs |
|---|---|---|
| CIFAR-100 | +0.70 ± 0.59 | 4/5 |
| TinyImageNet | +0.43 ± 0.72 | 4/5 |
| ImageNet-100 | +0.72 ± 0.62 | 5/5 |
AGOP-J2K improves accuracy in 13 of 15 paired runs; the average gains are positive across all three datasets. These results show moderate yet positive average gains across all three datasets, supporting the effectiveness of AGOP-guided compression.
To further investigate the underlying mechanism from the perspective of feature coverage, we also compare standard J2K and AGOP-J2K on CIFAR-100 using the same frozen ER backbone, replay buffer, calibration images, sample counts, and total encoded-byte budget, changing only rate–distortion allocation (uniform vs. AGOP-guided). AGOP-J2K reduces normalized mean discrepancy by 15.3%, normalized covariance discrepancy by 22.8%, and MMD$^2$ by 19.5%. Thus, AGOP-guided encoding improves historical feature-distribution preservation under controlled conditions, consistent with its positive accuracy gains.
Response to Reviewer GMQ6 (W3 & Q3)
W3 & Q3. Runtime, Memory and Computational Cost Analysis
Memory overhead. We have already provided a breakdown of RDM's memory decomposition in Table 4. The total overhead (including compressed calibration samples and other method-specific storage) ranges from 3.8% to 18.3%.
Runtime profiling. We measure wall-clock time over the entire CIFAR-100 class-incremental learning sequence, separating ordinary continual training from AGOP sensitivity computation, encoding and processing, latent refinement, and classifier-only fine-tuning. All times are in seconds:
| Stage | AGOP-PCA | AGOP-J2K |
|---|---|---|
| Continual training | 2626.68 | 2705.32 |
| AGOP sensitivity computation | 162.04 | 159.90 |
| PCA/J2K encoding & processing | 23.00 | 59.16 |
| Latent refinement | 248.55 | — |
| Classifier-only FT | 191.29 | 207.51 |
| Total | 3251.56 | 3131.89 |
| Overhead relative to continual training | 23.79% | 15.77% |
Additional cost. The proposed components add 624.88 s (23.79%) for AGOP-PCA and 426.57 s (15.77%) for AGOP-J2K, relative to the corresponding continual-training stage. For reference, as the overall best-performing baseline in our experiment, CEOS incurs a measured runtime overhead of 8.83%. Overall, the extra training cost of our method remains modest and bounded. While RDM incurs a moderately higher training overhead than CEOS (15.8%—23.8% vs. 8.8%), it yields a substantial improvement in final accuracy (e.g. 2.14 pp over CEOS on ImageNet-100) without compromising deployment/inference efficiency. We believe this represents a highly favorable and practical trade-off between accuracy and training cost. We will incorporate this profiling analysis into the revised manuscript.
Response to Reviewer GMQ6 (W1 & Q1)
We sincerely thank the reviewer for the constructive suggestions on isolation of contributions, benefit of AGOP, computational costs, broader applicability, and ensuring fair baseline comparisons.
W1 & Q1. Disentangling Classifier Calibration and Memory Design
Calibration-only controls. We evaluate two additional baselines under the same total historical-data storage budget, using the same five seeds and class orders as Table 1. ER+FT (Shared) fine-tunes the classifier on the existing ER buffer, reusing its raw exemplars for both rehearsal and calibration without extra storage. ER+FT (Disjoint) splits the budget between non-overlapping raw rehearsal and calibration buffers: its rehearsal capacity is the average of those used by AGOP-PCA and AGOP-J2K on the corresponding dataset, and the remainder stores raw calibration exemplars. Both baselines follow the calibration protocol described in the paper.
Contribution of classifier calibration. ER+FT (Shared) improves ER by 9.22, 4.44, and 2.68 pp on CIFAR-100, TinyImageNet, and ImageNet-100, respectively, confirming that classifier calibration alone yields performance gains. Using disjoint raw calibration samples provides a further 1.48–3.24 pp, consistent with the benefit of broader historical coverage.
Contribution of our memory design. Across the three datasets, AGOP-PCA and AGOP-J2K achieve 4.23–7.99 pp higher final accuracy against ER+FT (Shared) and 2.75–5.48 pp against ER+FT (Disjoint). These results show that our proposed role-decoupled compressed memory provides substantial additional gains beyond raw-exemplar classifier calibration under the same storage budget. The contribution of AGOP-guided encoding is examined separately in the encoder ablation (Table 2) and addressed in W2 & Q2.