acceptodds
Under review as a conference paper at ICLR 2027

Fidelity for Rehearsal, Coverage for Calibration: Role-Decoupled Memory for Class-Incremental Learning

Abstract

In Class-Incremental Learning (CIL), limited replay can preserve separable features for historical classes, yet provide insufficient distribution coverage to align the classifier with the historical class distributions in feature space. We observe that representation rehearsal and classifier calibration have different input fidelity requirements: with backbone frozen, calibration can trade input detail for broader historical coverage, provided that compression retains information relevant to classification in feature space. To this end, we propose a role-decoupled memory framework: using high-fidelity exemplars for representation learning, and compressed low-fidelity samples exclusively for subsequent classifier calibration. Drawing on the concept of Average Gradient Outer Product (AGOP), we propose two novel compression schemes, AGOP-PCA and AGOP-guided transform coding, to achieve broad feature coverage and preserve information useful for classification. Experiments across diverse CIL benchmarks show consistent improvements over strong replay, compressed replay and classifier-alignment baselines.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Class-Incremental Learning, Memory-Constrained Continual Learning, Classifier Alignment, Data Compression, Task-Aware Compression

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GMQ6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Role-Decoupled Memory (RDM) to address limited exemplar storage in Class-Incremental Learning (CIL). RDM separates memory into high-fidelity exemplars for representation learning and compressed samples for classifier calibration. To preserve important decision boundaries under compression, the authors introduce an AGOP-based sensitivity metric and develop two compression methods, AGOP-PCA and AGOP-J2K. Experiments on CIFAR-100, TinyImageNet, and ImageNet-100 show that RDM improves distribution coverage and CIL performance under fixed memory budgets.

Critical strengths
  1. The paper distinguishes the different memory requirements of representation learning and classifier calibration, offering a fresh perspective on memory allocation in CIL.

  2. The AGOP-based sensitivity metric provides a common principle for guiding both AGOP-PCA and AGOP-J2K compression, giving the framework a coherent theoretical foundation.

  3. Evaluations across three benchmarks, supported by multiple runs, ablations, and memory allocation analyses, provide convincing evidence of the method's effectiveness.

Critical weaknesses
  1. RDM combines role-decoupled memory, classifier calibration, and AGOP-guided compression. Without a baseline using the same calibration stage with raw exemplars, it is unclear which component drives the reported gains.

  2. The proposed sensitivity metric lacks comparisons with simpler gradient-based alternatives. Moreover, AGOP-J2K offers only marginal gains over standard JPEG2000, raising questions about the need for additional complexity.

  3. Experiments focus on ResNet-18 trained from scratch, without evaluating pretrained models or Vision Transformers. The lack of runtime, memory, and computational cost analysis also makes the practical benefits difficult to assess.

Questions for authors
  1. RDM introduces both compressed memory and an additional calibration stage. How much improvement comes from calibration alone? A comparison with standard ER using identical calibration on its existing exemplars would help isolate the contribution of memory design.

  2. Why are prototype-margin gradients preferable to simpler alternatives, such as logit or cross-entropy gradients? Given the modest gains of AGOP-J2K over standard JPEG2000, is the added complexity justified?

  3. AGOP computation, latent refinement, and classifier calibration introduce additional overhead. Could the authors report training time and GPU costs relative to strong baselines to clarify the accuracy-efficiency trade-off?

  4. The baselines are modified under a unified ER framework. Do these changes preserve their original performance, and are the reported gains robust to differences in calibration and training settings?

6 comments
AuthorsOpenReview ↗

Response to Reviewer GMQ6 (W1 & Q1)

We sincerely thank the reviewer for the constructive suggestions on isolation of contributions, benefit of AGOP, computational costs, broader applicability, and ensuring fair baseline comparisons.

W1 & Q1. Disentangling Classifier Calibration and Memory Design

Calibration-only controls. We evaluate two additional baselines under the same total historical-data storage budget, using the same five seeds and class orders as Table 1. ER+FT (Shared) fine-tunes the classifier on the existing ER buffer, reusing its raw exemplars for both rehearsal and calibration without extra storage. ER+FT (Disjoint) splits the budget between non-overlapping raw rehearsal and calibration buffers: its rehearsal capacity is the average of those used by AGOP-PCA and AGOP-J2K on the corresponding dataset, and the remainder stores raw calibration exemplars. Both baselines follow the calibration protocol described in the paper.

MethodCIFAR-100TinyImageNetImageNet-100
ER34.15 ± 1.4027.43 ± 0.3249.18 ± 0.76
ER+FT (Shared)43.37 ± 0.6831.87 ± 0.5151.86 ± 0.70
ER+FT (Disjoint)46.61 ± 0.1433.35 ± 0.2653.37 ± 0.76
AGOP-PCA51.36 ± 0.7136.10 ± 0.3157.72 ± 0.54
AGOP-J2K50.67 ± 0.5337.03 ± 0.8158.85 ± 0.53

Contribution of classifier calibration. ER+FT (Shared) improves ER by 9.22, 4.44, and 2.68 pp on CIFAR-100, TinyImageNet, and ImageNet-100, respectively, confirming that classifier calibration alone yields performance gains. Using disjoint raw calibration samples provides a further 1.48–3.24 pp, consistent with the benefit of broader historical coverage.

Contribution of our memory design. Across the three datasets, AGOP-PCA and AGOP-J2K achieve 4.23–7.99 pp higher final accuracy against ER+FT (Shared) and 2.75–5.48 pp against ER+FT (Disjoint). These results show that our proposed role-decoupled compressed memory provides substantial additional gains beyond raw-exemplar classifier calibration under the same storage budget. The contribution of AGOP-guided encoding is examined separately in the encoder ablation (Table 2) and addressed in W2 & Q2.

AuthorsOpenReview ↗

Response to Reviewer GMQ6 (W2 & Q2)

W2 & Q2. Prototype-Margin Sensitivity and the Value of AGOP

Why prototype-margin gradients? Our compression objective is to preserve classification-relevant geometry of historical feature distributions for subsequent classifier calibration. The gradient of prototype margin gi,c=JFt(xi)⊤uy,cg_{i,c}=J_{F_t}(x_i)^\top u_{y,c} measures how input reconstruction errors perturb separation along feature-space class contrasts. It provides a feature-geometric sensitivity signal that helps preserve useful historical support for a subsequently refitted classifier, without requiring an already calibrated or unbiased classifier. In contrast, logit and cross-entropy gradients depend on the current classifier weights. As discussed in previous work and our paper, an uncalibrated classifier may exhibit substantial old–new bias in CIL. Using the gradients of such classifier to allocate compression capacity can inherit the very misalignment that calibration seeks to correct, which runs counter to our objective.

Does AGOP provide measurable benefits? Our central contributions are the role-dependent fidelity–coverage trade-off and the resulting role-decoupled memory framework, with AGOP further improving its compressed calibration component. Table 2 already isolates AGOP from standard J2K. To better illustrate the benefit of AGOP, we further examine the paired standard-J2K vs. AGOP-J2K final-accuracy differences over the same five class orders:

DatasetMean paired gain (pp)Positive runs
CIFAR-100+0.70 ± 0.594/5
TinyImageNet+0.43 ± 0.724/5
ImageNet-100+0.72 ± 0.625/5

AGOP-J2K improves accuracy in 13 of 15 paired runs; the average gains are positive across all three datasets. These results show moderate yet positive average gains across all three datasets, supporting the effectiveness of AGOP-guided compression.

To further investigate the underlying mechanism from the perspective of feature coverage, we also compare standard J2K and AGOP-J2K on CIFAR-100 using the same frozen ER backbone, replay buffer, calibration images, sample counts, and total encoded-byte budget, changing only rate–distortion allocation (uniform vs. AGOP-guided). AGOP-J2K reduces normalized mean discrepancy by 15.3%, normalized covariance discrepancy by 22.8%, and MMD$^2$ by 19.5%. Thus, AGOP-guided encoding improves historical feature-distribution preservation under controlled conditions, consistent with its positive accuracy gains.

AuthorsOpenReview ↗

Response to Reviewer GMQ6 (W3 & Q3)

W3 & Q3. Runtime, Memory and Computational Cost Analysis

Memory overhead. We have already provided a breakdown of RDM's memory decomposition in Table 4. The total overhead (including compressed calibration samples and other method-specific storage) ranges from 3.8% to 18.3%.

Runtime profiling. We measure wall-clock time over the entire CIFAR-100 class-incremental learning sequence, separating ordinary continual training from AGOP sensitivity computation, encoding and processing, latent refinement, and classifier-only fine-tuning. All times are in seconds:

StageAGOP-PCAAGOP-J2K
Continual training2626.682705.32
AGOP sensitivity computation162.04159.90
PCA/J2K encoding & processing23.0059.16
Latent refinement248.55—
Classifier-only FT191.29207.51
Total3251.563131.89
Overhead relative to continual training23.79%15.77%

Additional cost. The proposed components add 624.88 s (23.79%) for AGOP-PCA and 426.57 s (15.77%) for AGOP-J2K, relative to the corresponding continual-training stage. For reference, as the overall best-performing baseline in our experiment, CEOS incurs a measured runtime overhead of 8.83%. Overall, the extra training cost of our method remains modest and bounded. While RDM incurs a moderately higher training overhead than CEOS (15.8%—23.8% vs. 8.8%), it yields a substantial improvement in final accuracy (e.g. 2.14 pp over CEOS on ImageNet-100) without compromising deployment/inference efficiency. We believe this represents a highly favorable and practical trade-off between accuracy and training cost. We will incorporate this profiling analysis into the revised manuscript.

Reviewer_wTMPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a role-decoupled memory framework for class-incremental learning. High-fidelity historical samples are used for continual training, while compressed samples are used for classifier calibration with a frozen backbone. The authors further introduce two compression methods based on Average Gradient Outer Product (AGOP) to preserve classification-relevant information. Experiments on three datasets show improvements over existing methods under a fixed memory budget.

Critical strengths

The paper addresses a relevant problem in memory-limited class-incremental learning. Using high-fidelity and compressed historical samples for different training purposes is a reasonable approach. The experiments include competitive baselines, five independent runs, and useful ablation studies.

Critical weaknesses
  1. The contribution of compressed calibration memory is not fully isolated. Figure 1 shows classifier calibration using additional raw or compressed samples, but does not use the same fixed total memory budget as the main experiments. A comparison with standard experience replay followed by the same classifier calibration procedure using only high-fidelity samples is missing. This comparison is needed to separate the benefit of compressed memory from that of classifier calibration itself.

  2. The additional benefit of AGOP-guided compression is limited. Table 2 shows that AGOP-guided JPEG2000 improves final accuracy over standard JPEG2000 by 0.69, 0.43, and 0.71 percentage points on the three datasets. Although the gains are consistent, their reliability is difficult to assess from the reported standard deviations alone. Reporting the paired accuracy differences across the five runs would provide stronger evidence. In addition, Table 3 compares feature-distribution coverage against standard experience replay, but does not include standard JPEG2000. This comparison would help establish whether AGOP itself improves the preservation of historical feature distributions.

  3. The additional computational cost is not reported. Appendix E describes 1,000 additional optimization steps for classifier calibration and another 1,000 steps for latent-code refinement in AGOP-PCA. Computing AGOP sensitivity also introduces additional processing. However, the paper does not report the corresponding training or compression times, making it difficult to assess the computational cost relative to the accuracy improvements.

Questions for authors

1.Under the same total memory budget, what accuracy is obtained by applying classifier-only calibration to standard experience replay using only its high-fidelity samples?

2.Can the authors report the paired accuracy differences between standard JPEG2000 and AGOP-guided JPEG2000, together with their feature-distribution discrepancy results?

3.What are the additional training and compression times introduced by the proposed methods?

3 comments
AuthorsOpenReview ↗

Response to Reviewer wTMP (W1 & Q1)

We sincerely thank the reviewer for the constructive suggestions, including separately evaluating the benefit of memory compression, analyzing pairing accuracy, and reporting computational costs.

W1 & Q1. Calibration Alone vs. Compressed Calibration Memory

Controlled raw-only calibration baselines. To isolate the benefit of classifier calibration from that of compressed calibration memory, we evaluate two additional baselines under the same total historical-data storage budget as our methods:

  • ER+FT (Shared): Standard ER followed by classifier-only fine-tuning on its existing raw replay exemplars. Rehearsal and calibration share the same buffer, without additional storage.
  • ER+FT (Disjoint): The budget is split between disjoint raw rehearsal and calibration buffers. We set the rehearsal capacity to the average of those used by AGOP-PCA and AGOP-J2K on each dataset, allocating the remaining bytes to raw calibration exemplars.

Both baselines follow the same calibration protocol as our methods. We use the same five seeds and class orders as in Table 1. Final accuracy (%) is reported as mean ± std:

MethodCIFAR-100TinyImageNetImageNet-100
ER34.15 ± 1.4027.43 ± 0.3249.18 ± 0.76
ER+FT (Shared)43.37 ± 0.6831.87 ± 0.5151.86 ± 0.70
ER+FT (Disjoint)46.61 ± 0.1433.35 ± 0.2653.37 ± 0.76
AGOP-PCA51.36 ± 0.7136.10 ± 0.3157.72 ± 0.54
AGOP-J2K50.67 ± 0.5337.03 ± 0.8158.85 ± 0.53

Contribution beyond calibration alone. ER+FT (Shared) improves over ER by 2.68–9.22 pp, confirming that classifier calibration itself reduces the error of standard replay. Allocating part of the same budget to disjoint raw calibration samples yields a further 1.48–3.24 pp over Shared. Nevertheless, our proposed role-decoupled calibration methods manage to outperform both ER+FT (Shared) and ER+FT (Disjoint) by 2.75–5.48 pp across the three datasets. Thus, additional classifier optimization explains part of the improvement, but allocating memory to broader compressed calibration support provides substantial further gains under the same storage budget.

AuthorsOpenReview ↗

Response to Reviewer wTMP (W2 & Q2)

W2 & Q2. Paired AGOP-J2K Gains and Feature-Distribution Preservation

Paired accuracy differences. We compare standard J2K and AGOP-J2K using the same five class orders and paired runs as in Table 2. The table reports the final-accuracy difference AGOP-J2K minus standard J2K, in percentage points (pp):

DatasetRun 1Run 2Run 3Run 4Run 5Mean ± std (pp)
CIFAR-100−0.26+1.33+0.93+0.67+0.81+0.70 ± 0.59
TinyImageNet+0.56+0.32+0.29−0.52+1.49+0.43 ± 0.72
ImageNet-100+1.58+0.63+0.05+1.07+0.25+0.72 ± 0.62

AGOP-J2K improves final accuracy in 13 of the 15 paired runs, including all five ImageNet-100 runs. The average gains are positive across all three datasets, supporting the contribution of AGOP-guided compression.

Controlled feature-distribution comparison. To assess whether AGOP itself improves historical feature preservation, we conduct an additional standard J2K vs. AGOP-J2K feature-distribution discrepancy analysis on CIFAR-100. The experiment uses a shared frozen ER-trained backbone for both AGOP sensitivity and feature extraction. Standard J2K and AGOP-J2K use identical replay buffers, calibration images, sample counts, and total encoded-byte budgets, with only rate–distortion allocation differing (uniform vs. AGOP-guided). The resulting discrepancies (lower is better) are:

EncoderMean Δμ\Delta_\mu ↓Cov ΔΣ\Delta_\Sigma ↓Norm. Mean Δˉμ\bar\Delta_\mu ↓Norm. Cov ΔˉΣ\bar\Delta_\Sigma ↓MMD$^2$ ↓
Standard J2K26.295120.9820.3610.3830.028
AGOP-J2K22.246108.3230.3060.2960.023

AGOP-J2K yields lower discrepancies on all five measures, reducing normalized mean discrepancy, normalized covariance discrepancy, and MMD$^2$ by 15.3%, 22.8%, and 19.5%, respectively. This controlled result complements the paired accuracy improvements by showing that AGOP-guided rate allocation better preserves the historical feature distribution under the same sample and byte budgets.

AuthorsOpenReview ↗

Response to Reviewer wTMP (W3 & Q3)

W3 & Q3. Additional Training and Compression Time

Runtime profiling. We measure wall-clock time over the entire CIFAR-100 class-incremental learning sequence, separating ordinary continual training from the extra components introduced by our method. All times are in seconds:

StageAGOP-PCAOverheadAGOP-J2KOverhead
Continual training2626.68—2705.32—
AGOP sensitivity computation162.046.17%159.905.91%
PCA/J2K compression23.000.88%59.162.19%
Latent refinement248.559.46%——
Classifier-only FT191.297.28%207.517.67%
Total3251.5623.79%3131.8915.77%

Additional cost. The proposed components add 624.88 s (23.79%) for AGOP-PCA and 426.57 s (15.77%) for AGOP-J2K, relative to the corresponding continual-training stage. Each component of our method, including AGOP computation, compression, latent refinement and classifier FT, incurs less than 10% overhead. Overall, the additional computational overhead is moderate relative to the main training cost, given the accuracy improvements reported in Table 1. We will include the runtime breakdown in the revised manuscript.

Reviewer_cGbJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This submission intend to adress two issues in class-incremental learning: 1) imited feature replay memory is insufficient and it difficult to align the classifier with such distribution of historical classes; 2) conventional replay methods task a finite set of samples with both representation learning and classifier calibration but, in fact, they impose different requirements on input. To this end, the proposed method retains high-fidelity samples for rehearsal within the backbone network, while maintaining a compressed storage buffer dedicated to fitting the classifier. Upon the completion of each task, the frozen backbone network is used to re-embed the reconstructed images. The reported results shows that proposed method achieve the best performance at tested settings.

Critical strengths
  • Clear Motivation/Insight: This submission find that the feature for replaying and data for classifier fitting have different requirements, as shwon in Fig.1. And this motivates the memory design. Such role decoupling is insightful, offering a entry point for examining rehearsal-based continual learning (CIL) methods.
  • Sound Method: The proposed framework is coherent, with a clear and well-defined division of responsibilities between the two storage components. The AGOP-PCA and AGOP-J2K compression schemes are soundly designed and practically viable. Also, the use of the AGOP to compute prototype-boundary sensitivity is sound and imporves the compression strategy.
  • Sound Performance Improvement: As shown in Table 1, the gains are consistent across three datasets, averaged over five runs.
Critical weaknesses
  • No computational ananlysis: The proposed method introduces non-negligible computational overhead at each step, e.g., 1k steps of SGD calibration and the computation of the AGOP. A more detailed analysis of the computational overhead is important.
  • Ablation study is confused: Table 3 compares various memories that employ different backbone networks. The improvement in distribution matching could stem from an increased sample size, changes in the representation method, or the design of the compression mechanism. While this confirms the effectiveness of the integrated system, it remains unclear which specific component is responsible for the observed improvement.
  • Confused PCA exp: In the ImageNet PCA experiment, the original images account for 96.4% of the budget (4820/5000=0.964). Furthermore, the FP8 basis vectors, mean values, and class scaling factors consume an additional 7.14% of the budget. This means the total usage exceeds the budget limit. Please explain this.
  • The claim is not supported by the results: They provided results does not support for "intrinsic fidelity asymmetry" and the "necessity of AGOP."
Questions for authors

Please see weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer cGbJ (W1)

We sincerely thank the reviewer for the constructive suggestions on computational analysis, better controlled analysis and inconsistency in presented memory budget.

W1. Computational Overhead

Runtime profiling. We measure the wall-clock time of each computational stage over the entire CIFAR-100 class-incremental learning sequence, including continual training, AGOP sensitivity computation, encoding, latent refinement, and classifier-only fine-tuning. The results are reported in seconds:

StageAGOP-PCAAGOP-J2K
Continual training2626.682705.32
AGOP computation162.04159.90
PCA/J2K Encoding & processing23.0059.16
PCA latent refinement248.55—
Classifier-only fine-tuning191.29207.51
Total3251.563131.89
Additional overhead23.79%15.77%

Computational overhead. Relative to their respective continual-training stages, the additional components introduce 23.79% overhead for AGOP-PCA and 15.77% for AGOP-J2K. AGOP sensitivity computation accounts for only approximately 5% of total runtime in both variants. AGOP-PCA incurs additional cost from latent refinement, whereas AGOP-J2K avoids this step. Overall, the additional computational overhead is moderate relative to the main training cost, given the accuracy improvements reported in Table 1. We will include the runtime breakdown in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer cGbJ (W2)

W2. Controlled Feature-Distribution Coverage Analysis

Addressing the confounding factors. As specified in Appendix G, the original Table 3 evaluates each method's memory support against the full training distribution in its own final backbone's feature space. We agree that this system-level comparison does not isolate the effects of backbone differences, memory coverage, and compression strategy. We therefore conduct a fixed-backbone analysis on CIFAR-100 and will replace Table 3 and redraw Figure 3 using this controlled setting. We will retain the original method-specific-backbone results in the revised appendix as complementary system-level evidence.

Controlled comparison. We use the same frozen ER-trained backbone to compute AGOP sensitivity and extract features for all methods, with a common full-data reference and identical discrepancy metrics and MMD kernel settings. In particular, standard J2K and AGOP-J2K use identical high-fidelity replay buffers, calibration images, calibration sample counts, and total encoded-byte budgets; only the rate–distortion allocation differs (uniform vs. AGOP-guided). The results are as follows (lower is better):

MemoryΔμ\Delta_\mu ↓ΔΣ\Delta_\Sigma ↓Δˉμ\bar\Delta_\mu ↓ΔˉΣ\bar\Delta_\Sigma ↓MMD$^2$ ↓
ER115.244193.7900.8060.8770.193
AGOP-PCA15.07790.3800.2070.2510.015
Standard J2K26.295120.9820.3610.3830.028
AGOP-J2K22.246108.3230.3060.2960.023

Findings. The results are consistent with the original Table 3: Both AGOP-PCA and AGOP-J2K substantially reduce the mean/covariance discrepancies and MMD$^2$ relative to ER. AGOP-J2K also improves all five discrepancy measures over standard J2K. In particular, normalized mean discrepancy, normalized covariance discrepancy, and MMD$^2$ decrease by 15.3%, 22.8%, and 19.5%, respectively. This provides direct evidence that AGOP-guided rate allocation improves historical feature-distribution preservation when the backbone, samples, and byte budget are held fixed. The lower distribution discrepancies of both compressed-memory variants relative to ER also illustrate the broader historical support available for classifier calibration. In the revision, Table 3 will report this controlled comparison, while Figure 3 will visualize the four methods using a common two-dimensional PCA projection fitted in the same ER feature space.

AuthorsOpenReview ↗

Response to Reviewer cGbJ (W3)

W3. ImageNet-100 AGOP-PCA memory accounting

We thank the reviewer for identifying an inconsistency in Table 4. Upon verification, we found that the reported sample count (4,820) was mistakenly taken from an debugging configuration with PCA rank r=128r=128, whereas the reported storage percentages correspond to the actual experimental configuration with r=256r=256. The correct number is 4,719 replay samples and 4,719 calibration samples.

Under the actual r=256r=256 ImageNet-100 configuration, the corrected AGOP-PCA storage breakdown is:

Num. SamplesReplayCalibrationOther storage
4,71994.380%0.127%5.475%

The "other storage" category includes the FP8 PCA basis, global mean, per-class AGOP scales, and associated metadata and labels. The actual memory usage therefore satisfies the fixed storage budget. We will correct the sample count in Table 4 and clarify the resolution-dependent storage accounting in Appendix E.

Regarding the reviewer's calculation of 7.14%, it appears to assume that the PCA state and raw replay images use the same spatial resolution D=2562×3D=256^2 \times 3, yielding a percentage of (256+100+1)/5,000=7.14%(256+100+1)/5{,}000=7.14\%. In our ImageNet-100 experiments, following the AlexNet convention (see Appendix E for details), high-fidelity replay images are stored as 256×256256 \times 256 RGB uint8 arrays and cropped to 224×224224 \times 224 during training and evaluation. Hence, calibration samples and the PCA basis use 224×224224 \times 224 inputs to match the actual train/eval input size. Thus, the total budget is 5,000×2562×35{,}000 \times 256^2 \times 3 bytes, while the PCA input dimension is D=2242×3D=224^2 \times 3 rather than 2562×3256^2 \times 3. The basis vectors, means and class scaling factors therefore uses a total budget of 5.467%; including miscellaneous metadata and sample labels yields 5.475%.

AuthorsOpenReview ↗

Response to Reviewer cGbJ (W4)

W4. Evidence for the Fidelity–Coverage Trade-off and AGOP

Role-dependent fidelity–coverage trade-off. Our claim concerns the role-dependent trade-off observed in the evaluated CIL setting; the reviewer's phrase "intrinsic fidelity asymmetry" suggests a broader, universal claim than intended. Figure 1 provides direct evidence: under the same additional byte budget, broader coverage from compressed samples benefits frozen-backbone classifier calibration substantially more than full-model training. Appendix F (Figure 4) provides complementary evidence under a fixed total memory budget. Both AGOP-PCA and AGOP-J2K achieve their highest final accuracy at rcal=1r_{\mathrm{cal}}=1. Allocating too little memory to calibration limits historical coverage, while allocating too much displaces high-fidelity replay, reducing accuracy. These findings support the complementary roles of high-fidelity rehearsal and compressed calibration memory.

Contribution of AGOP. Table 2 has already isolated the effect of AGOP within each encoder: it improves final accuracy by 0.61–2.25 pp for PCA and 0.43–0.71 pp for J2K across the three datasets. We also conduct an additional paired experiment to examine the paired final-accuracy differences between standard J2K and AGOP-J2K over the same five class orders:

DatasetMean paired gain (pp)Positive runs
CIFAR-100+0.70 ± 0.594/5
TinyImageNet+0.43 ± 0.724/5
ImageNet-100+0.72 ± 0.625/5

The results show a consistently positive average gain, and that AGOP-J2K improves upon J2K in 13/15 runs. Moreover, our new fixed-backbone experiment (W2), which holds the ER backbone, replay buffer, calibration images, and encoded-byte budget constant, shows that AGOP-J2K reduces normalized mean discrepancy, normalized covariance discrepancy, and MMD$^2$ over standard J2K by 15.3%, 22.8%, and 19.5%, respectively. Together, these accuracy and distribution-matching results support AGOP as an effective refinement of compressed calibration memory. We will clarify the scope of both contributions in the revised manuscript.

Comments on the paper

1 comment

Dear Reviewers,

As the discussion phase comes to an end, we'd appreciate it if you could take a moment to review the authors' responses. If you have any remaining concerns or questions, please raise them with the authors before the discussion closes.

Best,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.