acceptodds
Under review as a conference paper at ICLR 2027

Not More Parameters, Better Organization: Structured Compression Memory Network for Class-Incremental Learning

Abstract

Class-incremental learning (CIL) aims to learn new classes sequentially while preserving discriminability over previously seen categories. While fine-tuning pre-trained models (PTMs) yields stronger generalization than training from scratch, existing approaches suffer from an inherent trade-off between parameter overhead and feature regulation. Expandable architectures incur steady parameter inflation as tasks grow, whereas fixed-parameter alternatives struggle to maintain reliable recognition of historical classes. To address this dilemma, we propose the Structured Compression Memory Network (SCMNet), a rehearsal-free, fixed-parameter framework built upon PTMs. Instead of growing parameters with tasks, SCMNet spends a fixed budget on organizing the feature and prototype spaces. Specifically, dynamic neighborhood compression constructs label-constrained batch graphs, suppressing intra-class variance while preserving inter-class separability. The adaptive prototype constraint uses stored prototypes as anchors to stabilize historical decision boundaries. These two modules form a mutually reinforcing loop to mitigate performance degradation over long task sequences. Evaluations across six benchmarks and three pre-trained backbones demonstrate that SCMNet surpasses state-of-the-art methods with only a fixed budget of 1.7M trainable parameters, validating the benefits of our structured feature organization strategy. Our code is included in the supplementary materials and will be released publicly upon acceptance.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Class-Incremental Learning, Fine‑Tuning of Pre‑Trained Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QQwdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the core trade-off in pre-trained model (PTM)-based class-incremental learning (CIL): expandable architectures suffer from unbounded parameter growth as tasks accumulate, while fixed-parameter methods struggle to preserve performance on historical classes. The authors propose SCMNet (Structured Compression Memory Network), a rehearsal-free, fixed-parameter framework that reframes CIL as a feature space organization problem rather than a parameter adaptation problem.

SCMNet is built on a frozen pre-trained ViT backbone with lightweight fixed adapters, and consists of two mutually reinforcing modules: 1). Dynamic Neighborhood Compression (DNC): Constructs label-constrained similarity graphs from training batches, and applies two-stage graph convolution to compress intra-class feature variance while preserving inter-class boundaries. 2). Adaptive Prototype Constraint (APC): Maintains a prototype memory bank accumulated across tasks, and uses these prototypes as virtual anchors to stabilize historical decision boundaries. The constraint strength adapts automatically based on how crowded the prototype space is. The two modules form a closed-loop synergy: DNC produces tighter, more reliable prototypes that improve APC supervision, while APC stabilizes the feature space geometry to enable more effective compression in subsequent tasks. With a fixed ~1.7M trainable parameter budget regardless of task length, SCMNet achieves state-of-the-art performance across six benchmarks and three pre-trained backbones. The work also includes in-depth geometric analysis of the feature space to explain the method's effectiveness.

Critical strengths
  • Rigorous and fair experimental design: All methods are evaluated under a unified protocol with the same frozen ViT-B/16 backbone, identical task splits, and consistent metrics (average accuracy Aˉ\bar{A} and last-task accuracy ANA_N). The comparison covers 10+ state-of-the-art methods across six benchmarks and three task lengths (5/10/20 tasks), ensuring comprehensive benchmarking.
  • Comprehensive ablation studies: Component ablations clearly demonstrate the super-additive effect of DNC and APC (their combined gain exceeds the sum of individual gains), verifying their synergistic design. Additional ablations on adapter placement, DNC residual weight α\alpha, and backbone generalization systematically disentangle contributing factors and rule out confounding variables.
  • Strong reproducibility: The appendix includes a detailed reproducibility statement with fixed random seeds, disabled non-deterministic CUDA kernels, and pinned hyperparameters. All backbones and datasets are publicly available, and the authors commit to releasing full code and configuration files.
  • Mechanistic interpretability: The intra/inter-class variance analysis provides concrete geometric evidence for how SCMNet reshapes the feature space, rather than treating the model as a black box. The trend analysis across task sequences clearly illustrates the difference between SCMNet's progressive refinement and SEMA's repeated space re-establishment.
Critical weaknesses
  • The prototype memory bank introduces storage overhead that grows linearly with the total number of classes, which is not quantified or discussed. While trainable parameters are fixed, prototype storage could become a practical bottleneck for ultra-large-scale incremental learning with thousands of classes.
  • Inference-time computational cost (FLOPs, latency) is not reported. It is unclear whether the DNC module introduces any inference overhead relative to simpler fixed-parameter baselines, which is relevant for the deployment motivation of the work.
  • Some design choices lack sufficient justification. For example, the two-stage graph convolution design (channel compression + node calibration) is not compared to alternative aggregation strategies; the rationale for fusing dual expert adapters via element-wise addition (rather than learned routing) is not fully explained.
  • The adaptive weighting formula in APC (λapc=1/(γ+ϵ)\lambda_{apc}=1/(\gamma+\epsilon)) is presented without comparison to other possible adaptive strategies (e.g., linear scaling, exponential decay), leaving it unclear whether this specific formulation is critical to performance.
  • Performance gains are uneven: improvements are very large on benchmarks with large domain gaps from ImageNet pre-training (IN-R, IN-A), but more modest on small-domain-gap datasets (CIFAR, VTAB). The practical impact is strongest for cross-domain incremental learning scenarios.
  • The method is only evaluated on standard disjoint, balanced class-incremental settings. Its behavior in more realistic challenging scenarios (few-shot incremental learning, class-imbalanced increments, online CIL) remains untested.

Based on the comments above, the current manuscript has some limitations, and I give a Weak Reject at current stage.

Questions for authors
  • The paper emphasizes fixed trainable parameters, but the prototype memory bank scales linearly with the total number of classes. Please quantify the per-class storage cost of the prototype bank (e.g., in bytes) and discuss its scalability to very large class sets (e.g., 1000+ classes). If feasible, a brief ablation on prototype compression (e.g., quantization, dimensionality reduction) would help assess the method's practical limits. This would clarify the full storage footprint of the framework.
  • DNC uses a two-stage design: per-sample channel compression via 1×1 convolution followed by graph message passing. Please provide a brief ablation comparing this design to simpler alternatives, such as (a) direct graph convolution without channel compression, or (b) attention-based neighborhood aggregation. This would verify whether the two-stage structure is critical to performance.
  • Since deployment efficiency is a key motivation for fixed-parameter methods, please add a comparison of inference FLOPs and/or latency between SCMNet and representative baselines (SEMA, SR2SR^2-LoRA, SimpleCIL). It would be useful to confirm that the fixed parameter budget translates to fixed inference cost, and quantify any overhead relative to simpler methods.
  • he paper observes larger gains on large-domain-gap benchmarks and smaller gains on small-domain-gap datasets. Please provide a more quantitative analysis of how SCMNet's accuracy gain correlates with the magnitude of domain shift from pre-training. For example, measuring the feature distance between ImageNet-1K and each target dataset, and plotting it against the gain over a baseline. This would help practitioners predict when SCMNet will provide the most benefit.
3 comments
AuthorsOpenReview ↗

Response to Reviewer QQwd (Part 1 - Q1, Q2)

We sincerely thank you for your careful reading and constructive comments. The additions and revisions to the main text have been highlighted in blue and re-uploaded, and we respond to your comments point by point below.

Q1

Reviewer: The paper emphasizes fixed trainable parameters, but the prototype memory bank scales linearly with the total number of classes. Please quantify the per-class storage cost of the prototype bank (e.g., in bytes) and discuss its scalability to very large class sets (e.g., 1000+ classes). If feasible, a brief ablation on prototype compression (e.g., quantization, dimensionality reduction) would help assess the method's practical limits.

Response. Each prototype is a single d=768d=768-dimensional float32 vector, so the per-class cost is 768×4=3072768 \times 4 = 3072 bytes ≈3\approx 3 KB. The full bank is therefore negligible at every scale considered in the paper: IN-R and IN-A (200 classes) each need ≈0.6\approx 0.6 MB, CIFAR-100 needs ≈0.3\approx 0.3 MB, and VTAB needs ≈0.15\approx 0.15 MB. Scaling to 1000 classes costs ≈3\approx 3 MB and to 10,000 classes ≈30\approx 30 MB, next to the frozen backbone's 86M parameters (≈344\approx 344 MB) and even next to the 1.7M trainable parameters (≈6.8\approx 6.8 MB). The bank is storage, not trainable capacity, it is disabled at inference, and it is a cost shared by every prototype-based CIL method. At 3 KB per class there is no practical need for compression; if a tighter footprint were required, quantization to float16 or int8 halves or quarters the cost with no architectural change.

Prototype bank storage budget (float32, d=768d=768)

ComponentStorage
backbone (86M parameters)≈\approx 344 MB
trainable budget (1.7M parameters)≈\approx 6.8 MB
prototype, per class≈\approx 3 KB
prototype bank, 200 classes (IN-R / IN-A)≈\approx 0.6 MB
prototype bank, 1000 classes≈\approx 3 MB
prototype bank, 10000 classes≈\approx 30 MB

Q2

Reviewer: DNC uses a two-stage design: per-sample channel compression via 1$\times$1 convolution followed by graph message passing. Please provide a brief ablation comparing this design to simpler alternatives, such as (a) direct graph convolution without channel compression, or (b) attention-based neighborhood aggregation. This would verify whether the two-stage structure is critical to performance.

Response. Part (a) already exists as the "w/o Channel" row of Table 2, which removes the per-sample channel compression while keeping the graph message passing. It costs only 0.36 to 0.61 points of Aˉ\bar{A} (90.78/89.37/85.46 vs. 91.14/89.91/86.07), so the graph aggregation is the primary contributor and the channel stage is a supporting refinement. The two-stage structure is retained because the channel stage costs almost nothing in parameters yet improves every setting, but it is not where the gain lives. On part (b), an attention-based aggregation differs from ours mainly in the edge weights. Without a label constraint it would assign non-zero weights to cross-class neighbors and pull different classes together, which directly contradicts the intra-class-compression objective. The label-constrained mask, not the choice of cosine versus learned attention weights, is the design element that matters, and the A=IA=I and "w/o Node" rows of Table 2 isolate exactly that element: with the mask removed or the message passing disabled, the gain disappears.

Table 2 (reproduced, excerpt): the two-stage decomposition on IN-R (Aˉ\bar{A} / ANA_N)

Configuration5-Task10-Task20-Task
DNC (A=IA=I) + APC83.23 / 78.0783.42 / 77.6382.32 / 76.45
w/o Node + APC (no message passing)81.61 / 75.2782.78 / 77.9584.58 / 79.37
w/o Channel + APC (no compression)90.78 / 87.2789.37 / 84.9585.46 / 79.45
full (DNC + APC)91.14 / 87.4889.91 / 85.6886.07 / 80.92

AuthorsOpenReview ↗

Response to Reviewer QQwd (Part 2 - Q3, Q4)

Q3

Reviewer: Since deployment efficiency is a key motivation for fixed-parameter methods, please add a comparison of inference FLOPs and/or latency between SCMNet and representative baselines (SEMA, SR$^2$-LoRA, SimpleCIL). It would be useful to confirm that the fixed parameter budget translates to fixed inference cost, and quantify any overhead relative to simpler methods.

Response. Table 6 reports the inference budget. SCMNet costs ≈17.66\approx 17.66G FLOPs versus ≈17.62\approx 17.62G for SEMA, a gap of +0.04+0.04G (below 0.25%), and the DNC enhancer contributes only +0.003+0.003G because at inference its graph degenerates to an identity and only the per-node channel transforms remain. The gap is owed chiefly to SCMNet's larger initial adapter stack (405K vs. 304K), not to DNC. Since DNC adds no per-task computation and the memory bank is disabled at inference, the fixed parameter budget translates directly into a fixed inference budget, whereas SEMA adds +0.01+0.01G FLOPs and +223+223K parameters per expansion and eventually overtakes SCMNet in both.

Table 6 (reproduced): parameter and inference cost budget

MethodViT (frozen)AdaptersAuxiliaryFLOPsΔ\Delta FLOPs/Exp.Expansion
SEMA$\sim$86M$\sim$304K$\sim$600K$\sim$17.62G+0.01+0.01GDynamic
SCMNet$\sim$86M$\sim$405K$\sim$1180K$\sim$17.66G0None (fixed)

Q4

Reviewer: The paper observes larger gains on large-domain-gap benchmarks and smaller gains on small-domain-gap datasets. Please provide a more quantitative analysis of how SCMNet's accuracy gain correlates with the magnitude of domain shift from pre-training. For example, measuring the feature distance between ImageNet-1K and each target dataset, and plotting it against the gain over a baseline.

Response. We do not reduce this to a measured distance-gain curve, because the six benchmarks do not sit on a single domain-gap axis. IN-R shifts through rendition and style, IN-A through natural adversarial examples, VTAB through task diversity, and CIFAR through a distribution close to the pretraining data, so a scalar ImageNet-1K feature distance would collapse these distinct shift types onto one number and would not order the benchmarks meaningfully. The gain also depends on how strongly each baseline responds to the specific shift, so a distance-gain scatter would confound the shift magnitude with the baseline's own sensitivity. What the paper establishes instead is the mechanism on the benchmark where the question matters most. On IN-R, the largest-gap benchmark, Figure 5 shows that DNC's compression is predominantly intra-class, about one and a half times its reduction of the inter-class term, which is the component a domain shift inflates. The same account covers the one boundary case: at 20 tasks CIFAR's small gap leaves little intra-class variance for DNC to compress, while SR$^2$-LoRA's SVD alignment, which benefits from inter-layer drift, overtakes SCMNet there.

Gain over SEMA by domain gap (Aˉ\bar{A}, from Table 1), ordered by gain

Benchmark (domain gap)SCMNetSEMAGain
20-Task IN-A (large)72.1764.53+7.64
5-Task IN-R (large)91.1484.75+6.39
10-Task IN-R (large)89.9183.56+6.35
20-Task IN-R (large)86.0781.75+4.32
5-Task VTAB (small)95.0691.26+3.80
10-Task CIFAR (small)94.8091.37+3.43

The remaining weaknesses other than the four issues above are explained as follows.

(1) Dual-adapter fusion. The dual experts are fused by element-wise addition precisely because it adds zero parameters and zero routing modules; a learned router would reintroduce per-task routing parameters of the kind SEMA's expandable router incurs, contradicting the fixed budget. Table 3 shows adapter capacity is not the driver (single versus the best dual configuration differs by under 0.59 points of Aˉ\bar{A}), so a routing mechanism would add parameters without adding accuracy.

(2) The adaptive weight form. λapc=1/(γ+0.1)\lambda_{apc}=1/(\gamma+0.1) is chosen because it is monotone, bounded (operating between the fixed values we ablate), and needs no extra hyperparameter, unlike linear or exponential schedules which each require additional slope or decay-rate coefficients. The gain reported in Table 7 comes from the relaxation itself, not from the specific functional form.

(3) Untested settings. SCMNet is rehearsal-free and keeps a fixed budget with a prototype bank, so it extends naturally to few-shot and online CIL where rehearsal is restricted; class-imbalanced increments are a separate axis we leave to future work.

Thanks for preparing the detailed response. Some of my concerns have been addressed. However, I still have concerns regarding the rationale behind certain methodological choices, as the current experiments do not fully validate their necessity or advantages over alternative designs. In addition, the relationship between domain shift and performance gains remains insufficiently supported by quantitative evidence. Therefore, I will maintain my original score.

Reviewer_MQ7nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes SCMNet, a rehearsal-free class-incremental learning method with a fixed budget of trainable parameters. It freezes a pretrained ViT backbone, uses a predetermined set of adapters, and adds two components: Dynamic Neighborhood Compression (DNC), which propagates features over label-constrained graphs constructed within training batches, and Adaptive Prototype Constraint (APC), which regularizes the classifier using stored prototypes of earlier classes. The authors describe these components as a feedback loop that compacts class representations while preserving historical boundaries. Table 1 reports competitive accuracy across six incremental-learning configurations, and Table 2 examines the two modules. The main question is whether the reported gains arise from label-conditioned graph propagation, additional pointwise capacity, or conventional prototype regularization.

Critical strengths
  1. The fixed-budget setting is practically relevant. Section 3.2 allocates adapters in advance, with additive dual experts in layers 8–11, rather than introducing new experts and routing modules after every task. Table 6 breaks down the parameter budget for adapters, auxiliary modules, and the classifier. The work therefore targets a meaningful constraint in rehearsal-free continual learning, and its resource accounting makes the design relatively easy to assess.

  2. The empirical performance warrants investigation. Table 1 shows strong results on CIFAR-100, ImageNet-R, ImageNet-A, and VTAB under the reported task splits. In Table 2, the full system achieves 91.14/89.91/86.07 average accuracy on 5/10/20-task ImageNet-R, compared with 84.78/84.47/83.04 when both modules are disabled. These are substantial improvements over the authors' internal baseline. Establishing which mechanism produces them is essential to the paper's contribution.

Critical weaknesses
  1. DNC uses different neighborhood operations during training and inference, and the graph contribution has not been isolated. In Section 3.3, Eqs. (1–3) build the adjacency matrix using ground-truth label equality within each training batch and perform cross-sample message passing. Section 3.6 removes graph construction at inference, where a single-sample graph reduces to A=IA=I. Thus, the class-neighbor aggregation highlighted as the central mechanism is absent when the model makes predictions. Using labels to construct a training graph is not inherently problematic; the unexplored train–test mismatch is. DNC also contains learnable channel transforms WsW_s and WdW_d with a residual path, accounting for approximately 1.18M auxiliary parameters in Table 6. Disabling DNC in Table 2 removes both the graph operation and this extra capacity. A parameter-matched DNC with A=IA=I, together with label-free and random-graph controls, is necessary to show that label-constrained neighborhood compression is responsible for the observed gains.

  2. APC does not directly optimize the old-versus-new boundaries it is claimed to preserve, and several implementation choices remain unresolved. The cross-entropy denominator in Eq. (4) covers only historical classes. Section 3.6 also masks old-class logits in the new-task classification loss. Neither stated objective explicitly compares old classes with new ones, so APC directly constrains competition among old classes rather than all boundaries in the expanded label space. The stored prototypes remain fixed while shared adapters continue to change, creating a possible mismatch between the stored anchors and the current feature space that is not measured. There is also a concrete inconsistency in the hyperparameter discussion: Section 3.3 states that α=4\alpha=4 is best on every benchmark, but Appendix Table 8 reports 91.14 versus 89.72 on 5-task ImageNet-R for α=12\alpha=12 versus 44, and 89.91 versus 88.85 on 10-task ImageNet-R for α=8\alpha=8 versus 44. The rule used to choose these settings, and whether it involved an independent validation set, is not sufficiently documented. These issues affect the proposed retention mechanism and the credibility of the experimental protocol.

Questions for authors
  1. What happens in Tables 1–2 if DNC is trained with A=IA=I while keeping WsW_s, WdW_d, α\alpha, the parameter count, and the training budget unchanged? How much additional gain comes specifically from label-constrained edges rather than label-free edges?
  2. Given the masking in LclsL_{cls} and the old-class-only normalization in LapcL_{apc}, which gradient signal calibrates old-class logits against new-class logits? Please report old-to-new/new-to-old confusion and prototype drift across task age.
  3. How was α\alpha selected for each task split in Appendix Table 8? Was the selection based on held-out validation data? Please reconcile the table with the claim in Section 3.3 and provide variation across class orders or seeds.
6 comments
AuthorsOpenReview ↗

Response to Reviewer MQ7n (Part 1 - Q1)

We sincerely thank you for your careful reading and constructive comments. The additions and revisions to the main text have been highlighted in blue and re-uploaded, and we respond to your comments point by point below.

Q1

Reviewer: What happens in Tables 1 and 2 if DNC is trained with A=IA=I while keeping WsW_s, WdW_d, α\alpha, the parameter count, and the training budget unchanged? How much additional gain comes specifically from label-constrained edges rather than label-free edges?

Response. The parameter-matched control is reported in Table 2. Training DNC with A=IA=I keeps WsW_s, WdW_d, α\alpha, all $\sim$1.18M parameters, and the training budget unchanged, and removes only the cross-sample aggregation, so DNC becomes a per-channel residual MLP of identical size while APC runs as usual. This control falls below the APC-only setting at every task length (Aˉ\bar{A} 83.23/83.42/82.32 vs. 84.90/84.48/83.55), i.e. the same budget spent as an unstructured MLP is mildly harmful ($-1.67/1.67/-1.06/1.06/-$1.23). The full model's margin over this control, $+7.91/7.91/+6.49/6.49/+$3.75 Aˉ\bar{A}, is therefore earned entirely by what the aggregation organizes, not by the capacity it occupies. Disabling node calibration (w/o Node) reproduces the same collapse (down to 81.61 Aˉ\bar{A} at 5 tasks), while disabling channel compression (w/o Channel) costs under one point (0.36 to 0.61), confirming that the gain lives in the organizing mechanism rather than in the pointwise transforms those parameters implement.

We interpret this as organization rather than added capacity. The same budget is spent structuring the feature space, pulling same-class samples toward a shared centroid and collapsing intra-class redundancy instead of enlarging the space. The label mask is what makes that organization directional: it restricts aggregation to same-class neighbors, so it compresses intra-class variance, whereas any label-free graph (random neighbors or raw feature similarity) aggregates across classes and would pull different classes toward one another, the opposite of the intended compression. The A=IA=I control is the strongest form of the label-free control because it keeps no edges at all; since this zero-edge control already sits below the baseline, no label-free aggregation can reverse it, and the benefit is localized to the directional, label-constrained organization.

The organization is also a training-time act with no inference cost. DNC's label-constrained messages run only during training, and the resulting intra-class compaction is absorbed into the adapter and classifier weights that remain active at inference, where a single-sample graph degenerates to A=IA=I. The A=IA=I control and the full model are therefore identical at inference and differ only in whether organization occurred during training, so the full model's margin over the control localizes the entire benefit to training-time label-constrained organization, which is precisely the paper's claim of better organization, not more parameters.

Table 2 (reproduced): component ablation on ImageNet-R (Aˉ\bar{A} / ANA_N)

Configuration5-Task10-Task20-Task
baseline (no DNC, no APC)84.78 / 80.4284.47 / 81.0783.04 / 79.66
APC only84.90 / 80.6584.48 / 79.6783.55 / 79.90
DNC (A=IA=I) + APC83.23 / 78.0783.42 / 77.6382.32 / 76.45
w/o Node + APC (no message passing)81.61 / 75.2782.78 / 77.9584.58 / 79.37
w/o Channel + APC (no compression)90.78 / 87.2789.37 / 84.9585.46 / 79.45
DNC only89.41 / 85.4788.81 / 84.3285.75 / 81.55
full (DNC + APC)91.14 / 87.4889.91 / 85.6886.07 / 80.92
AuthorsOpenReview ↗

Response to Reviewer MQ7n (Part 2 - Q2)

Q2

Reviewer: Given the masking in Lcls\mathcal{L}_{cls} and the old-class-only normalization in Lapc\mathcal{L}_{apc}, which gradient signal calibrates old-class logits against new-class logits? Please report old-to-new/new-to-old confusion and prototype drift across task age.

Response. The masking and the historical-only normalization are deliberate, and the old-versus-new boundary is enforced through the shared parameters rather than through a single loss term. A joint softmax over old and new classes would let new-class gradients flow through the shared denominator into the old-class rows of WW, which is precisely the update path that drives catastrophic forgetting. SCMNet therefore splits stability and plasticity into two orthogonal terms: LclsL_{cls} trains only the new-class rows and the shared feature extractor, while LapcL_{apc} locks the old-class rows through the current, drifted head. The calibrating signal is the coupling these two terms share: both act on the same pre-allocated head WW and the same adapters. LapcL_{apc} holds the old-class rows at their positions relative to the stored prototypes, LclsL_{cls} trains the new-class rows, DNC compacts the new-class features into a tight region, and APC keeps the old-class prototypes separated. Within this single shared feature space the new classes are pushed into the room that remains while the old-class regions do not drift into it. No single objective explicitly compares an old logit with a new one, but the two objectives jointly determine the geometry of the shared space in which the boundary lives.

The drift this design is meant to prevent is measured directly. Figure 5 (and its appendix on the 5/10/20-task protocols) reports the intra-class and inter-class variance decomposition as a quantitative proxy for prototype drift across task age: the inter-class variance measures how far the class means spread, so prototype drift shows up as class means collapsing toward one another. SCMNet stays well below SEMA on both variances and keeps its own curves flat from the first task to the last; at 20 tasks the two methods' inter-to-intra ratios part, SEMA's continuing to decline while SCMNet's climbs slowly and finishes above where it started, so SCMNet's class means stay separated rather than drifting together. The downstream consequence of old-versus-new confusion is a drop in all-class accuracy, and the full model's Aˉ\bar{A} stays far above the baseline at every length (Table 2), with ANA_N rising from 80.42 to 87.48 at 5 tasks and from 81.07 to 85.68 at 10 tasks, so the old-class boundaries are not eroded by the incoming classes.

Old-class retention across task age (IN-R, ANA_N, from Table 2)

Configuration5-Task10-Task20-Task
baseline80.4281.0779.66
DNC only85.4784.3281.55
full (DNC + APC)87.4885.6880.92
AuthorsOpenReview ↗

Response to Reviewer MQ7n (Part 3 - Q3)

Q3

Reviewer: How was α\alpha selected for each task split in Appendix Table 8? Was the selection based on held-out validation data? Please reconcile the table with the claim in Section 3.3 and provide variation across class orders or seeds.

Response. The adopted α\alpha follows a deterministic rule tied to the number of classes per task: α=12\alpha=12 for 40-class tasks (5-task IN-R), α=8\alpha=8 for 20-class tasks (10-task IN-R), and α=4\alpha=4 for the roughly 10-class tasks (the remaining four benchmarks), as shaded in Table 9. The rule is principled rather than tuned: a task that introduces many classes builds one graph over many prototypes, so more relations are available for the message passing and a stronger constraint keeps paying off, while a task with few classes has little structure to organize and is disturbed by a strong constraint. The full sweep in Table 9 documents this rule instead of tuning against it.

The selection is not based on held-out validation data. α\alpha is read off the task structure, the number of classes per task, before training begins, so no validation set is involved and there is no leakage. We state this explicitly in the revised Section 4.1.

On the reconciliation with Section 3.3: the original sentence was ambiguous. It was meant to state that SCMNet with α\alpha fixed at 4 already surpasses every compared method on all six benchmarks, not that α=4\alpha=4 is optimal within our own sweep. The fixed-α\alpha row of Table 9 (89.72/88.85/86.07/94.80/72.17/95.06 in Aˉ\bar{A}) already exceeds the best compared method on every benchmark, so the 91.14 versus 89.72 on 5-task IN-R (α=12\alpha=12 versus α=4\alpha=4) and 89.91 versus 88.85 on 10-task IN-R (α=8\alpha=8 versus α=4\alpha=4) are internal sweep differences, not a contradiction with the claim. The sentence has been rewritten to remove this ambiguity.

On class-order and seed variation, the adopted α\alpha is a deterministic function of the number of classes per task and does not depend on which classes fall into which task, so the rule is insensitive to class ordering by construction. The full protocol, a fixed random class ordering and a fixed seed, follows the standard CIL benchmark and matches SEMA and SR$^2$-LoRA, so the comparison remains fair and the results reproducible; the deterministic α\alpha rule introduces no additional variance source of its own.

Table 9 (reproduced, excerpt): the internal sweep behind the two cited values (Aˉ\bar{A} / ANA_N)

α\alpha5-Task IN-R10-Task IN-R
4 (fixed)89.72 / 85.8888.85 / 84.70
890.68 / 87.0389.91 / 85.68 (adopted)
1291.14 / 87.48 (adopted)87.23 / 83.10

Table 8 (reproduced): SCMNet with α=4\alpha=4 fixed vs. the best compared method, Aˉ\bar{A}

BenchmarkSCMNet (α=4\alpha=4)Best compared
10-Task CIFAR94.8094.67 (KCEA)
5-Task IN-R89.7286.01 (SR$^2$-LoRA)
10-Task IN-R88.8586.01 (KCEA)
20-Task IN-R86.0784.32 (SR$^2$-LoRA)
20-Task IN-A72.1769.41 (SR$^2$-LoRA)
5-Task VTAB95.0694.69 (KCEA)
Reviewer_1R95OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies rehearsal-free class-incremental learning with a frozen pre-trained ViT and proposes SCMNet. Under a fixed budget of about 1.7M trainable parameters, it organizes the feature space directly with two modules. DNC builds a graph with only same-class edges within each training batch and uses graph convolution to compress intra-class variance. APC uses stored old-class prototypes as virtual samples to constrain the classifier, with a weight that adapts to how crowded the prototype space is. Experiments on four datasets, multiple task splits and three backbones show state-of-the-art results, with clear margins on ImageNet-R and ImageNet-A.

Critical strengths
  1. The paper offers a clear and well-motivated perspective on forgetting.
  2. The empirical results are strong and consistent. The margins are clearest on ImageNet-R and ImageNet-A,.
  3. The design is simple, and its inference cost stays constant, which is a practical advantage over expansion-based methods.
Critical weaknesses
  1. The gain of DNC is not separated from its parameter count. DNC adds about 1.18M parameters, more than all adapters combined, yet at inference the graph collapses to an identity graph, so DNC becomes a residual MLP. Since the no-DNC baseline in Table 2 has far fewer parameters, it is unclear whether the gain comes from graph aggregation or simply from extra capacity. This matters because the title emphasizes “not more parameters”.
  2. The benefit of APC and its adaptive weight is not well supported. There is no comparison between a fixed weight and the adaptive one, so it is unclear whether 1/(γ+ϵ) is necessary. APC’s gain also shrinks as sequences get longer, and at 20 tasks on ImageNet-R A_N even drops below DNC alone (80.92 vs. 81.55). Why does an old-class constraint help least in exactly the setting where it should matter most?
  3. Some reported numbers and claims are inconsistent. APC-only A_N on 20-task ImageNet-R is 79.90 in Table 2 but 77.90 for α=0 in Table 8, although the text treats them as the same setting. In addition, Sec. 3.3 claims that α=4 is best on every benchmark, which contradicts Table 8.
Questions for authors
  1. Since DNC collapses to an identity graph at inference, is the gain mainly from the extra 1.18M parameters? I would like to see a parameter-matched control, for example the same DNC trained with A set to the identity, or an MLP of the same size. If most of the gain remains without the graph, the paper’s main claim would need to be reframed.
  2. How does APC perform with a fixed weight chosen on a held-out split? It would also help to report how γ and λ_apc actually evolve over tasks. This would show whether the adaptive weight changes enough to matter.
  3. Why does APC’s benefit shrink with longer sequences, and why does A_N drop at 20 tasks? Old- and new-class accuracy, or forgetting, reported separately for DNC-only and DNC+APC would make this clearer.
  4. APC-only A_N on 20-task IN-R is 79.90 in Table 2 but 77.90 for α=0 in Table 8. Which is correct? Is the identical A_N of 61.49 for α = 0, 2 and 4 on 20-task IN-A a coincidence or a copying error?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 1R95 (Part 1 - Q1, Q2)

We sincerely thank you for your careful reading and constructive comments. The additions and revisions to the main text have been highlighted in blue and re-uploaded, and we respond to your comments point by point below.

Q1

Reviewer: Since DNC collapses to an identity graph at inference, is the gain mainly from the extra 1.18M parameters? I would like to see a parameter-matched control, for example the same DNC trained with A set to the identity, or an MLP of the same size. If most of the gain remains without the graph, the paper's main claim would need to be reframed.

Response. We now provide a parameter-matched control. Training DNC with the adjacency fixed to the identity (A=IA=I) keeps every one of the ~1.18M parameters but removes cross-sample aggregation, reducing DNC to a per-channel MLP of identical size; it then falls below the adapter-only baseline at every task length (83.23/83.42/82.32 vs. 84.78/84.47/83.04 in Aˉ\bar{A}). Capacity alone therefore contributes nothing. Decomposing DNC locates the gain precisely: disabling node calibration (w/o Node) costs up to 9.5 points of Aˉ\bar{A} (81.61 vs. the full 91.14 at 5 tasks), while disabling channel compression (w/o Channel) costs under one point (0.36–0.61). The benefit is graph message passing, not the pointwise transforms those parameters implement.

The graph is a training-time organizer: during training, label-constrained aggregation pulls same-class features toward shared centroids, and this tightened geometry is absorbed into the adapter weights, so at inference the compactness is already encoded in the parameters and the identity graph loses nothing. On the budget, "not more parameters" refers to the budget being fixed across tasks: SEMA appends 223K parameters per task without bound, whereas we spend 1.7M once and organize the space instead of enlarging it. The A=IA=I control is the direct empirical proof that the same budget, spent as an unorganized MLP, underperforms the baseline; organization, not capacity, earns the gain.

Table 2 (reproduced): component ablation on ImageNet-R (Aˉ\bar{A} / ANA_N)

Configuration5-Task10-Task20-Task
baseline (no DNC, no APC)84.78 / 80.4284.47 / 81.0783.04 / 79.66
A=IA=I (parameter-matched MLP)83.23 / 78.0783.42 / 77.6382.32 / 76.45
w/o Node (no graph message passing)81.61 / 75.2782.78 / 77.9584.58 / 79.37
w/o Channel (no compression)90.78 / 87.2789.37 / 84.9585.46 / 79.45
DNC only89.41 / 85.4788.81 / 84.3285.75 / 81.55
full (DNC + APC)91.14 / 87.4889.91 / 85.6886.07 / 80.92

Q2

Reviewer: How does APC perform with a fixed weight chosen on a held-out split? It would also help to report how γ and λ_apc actually evolve over tasks. This would show whether the adaptive weight changes enough to matter.

Response. We add a fixed-weight comparison in Table 7. No constant λ\lambda works across the sequence: λapc=2\lambda_{apc}=2 is best at 5 tasks (91.29 Aˉ\bar{A}) yet worst at 20 (85.19, below the λapc=0\lambda_{apc}=0 DNC-only 85.75), because a strong constant anchor over-regularizes the crowded late-stage space. The optimal constant drifts with task count, so a fixed value cannot win; the adaptive weight is the only setting that stays high throughout (91.14/89.91/86.07) and attains the best average Aˉ\bar{A} (89.04) and ANA_N (84.69).

The adaptive weight is not a free knob: it tracks a real, measurable signal. Figure 6 shows the crowding measure γ\gamma rising monotonically as the prototype space fills (0.66→0.73, 0.61→0.72, 0.62→0.71 on the 5/10/20-task protocols), and λapc=1/(γ+0.1)\lambda_{apc}=1/(\gamma+0.1) relaxing by 8–13% in response, operating between the fixed values {1,2}\{1,2\} we ablate. The fixed-weight grid therefore spans exactly the regime the adaptive coefficient actually inhabits.

Table 7 (reproduced): ablation of the crowding-adaptive coefficient on ImageNet-R (Aˉ\bar{A} / ANA_N)

λapc\lambda_{apc}5-Task10-Task20-TaskAverage
0 (= DNC only)89.41 / 85.4788.81 / 84.3285.75 / 81.5587.99 / 83.78
190.73 / 86.6289.68 / 85.2285.95 / 81.1388.78 / 84.32
291.29 / 87.2589.77 / 85.7185.19 / 80.3488.75 / 84.43
dynamic (adopted)91.14 / 87.4889.91 / 85.6886.07 / 80.9289.04 / 84.69

AuthorsOpenReview ↗

Response to Reviewer 1R95 (Part 2 - Q3, Q4)

Q3

Reviewer: Why does APC's benefit shrink with longer sequences, and why does A_N drop at 20 tasks? Old- and new-class accuracy, or forgetting, reported separately for DNC-only and DNC+APC would make this clearer.

Response. APC is a forward-looking mechanism: its role is to reserve room for the classes still to come, and the number of future classes shrinks as the sequence progresses: 160 remain after task 1 of the 5-task split, but only 10 after task 19 of the 20-task split. The room-reservation it performs is therefore worth most early and converges late, so the shrinking benefit reflects the mechanism behaving as designed.

The metrics separate the two effects cleanly. APC anchors old classes only, so its benefit concentrates in old-class retention, which Aˉ\bar{A} (per-task end accuracy) captures; the Aˉ\bar{A} gain stays positive at every length, +0.32 at 20 tasks (85.75→86.07). ANA_N is terminal all-class accuracy, dominated by the final task's new classes, which are not in the prototype bank during their own training; hence the small decrease (80.92 vs. 81.55). This Aˉ\bar{A}↑ / ANA_N↓ contrast is the signature of an old-class-anchoring module. The adaptive weight works in concert with this: without the relaxation, a fixed strong λ\lambda would over-constrain the crowded 20-task space even further (λapc=2\lambda_{apc}=2 drops to 85.19, below DNC-only), so the adaptivity that makes the anchor appear less active late in training is precisely what prevents late-stage collapse. Even the reduced ANA_N (80.92) still beats the strongest baseline SR²-LoRA (79.54, Table 4).

Tables 2 & 4 (reproduced): APC effect and 20-Task ImageNet-R comparison (Aˉ\bar{A} / ANA_N)

20-Task ImageNet-RAˉ\bar{A}ANA_N
DNC only85.7581.55
SEMA81.7574.53
SR²-LoRA84.3279.54
SCMNet86.0780.92

Q4

Reviewer: APC-only A_N on 20-task IN-R is 79.90 in Table 2 but 77.90 for α=0 in Table 8. Which is correct? Is the identical A_N of 61.49 for α = 0, 2 and 4 on 20-task IN-A a coincidence or a copying error?

Response. (1) The APC-only ANA_N at 20-task IN-R is 79.90, not 77.90. The α=0\alpha=0 row of the sweep is, by construction, the APC-only setting (α=0\alpha=0 disables DNC entirely, leaving only APC), so the two rows agree, and they now do (both 83.55/79.90); the earlier 77.90 was a transcription error. (2) The identical AN=61.49A_N=61.49 for α=0/2/4\alpha=0/2/4 on 20-task IN-A is genuine, not a copy error. ANA_N is a discrete, rounded quantity, and at α≤4\alpha\le 4 the compression is still too weak to flip the hardest adversarial decisions, so ANA_N rounds to the same value, beginning to move only at α≥6\alpha\ge 6 (61.55, then 54.44 at α=14\alpha=14). (3) The "α=4\alpha=4 best on every benchmark" claim referred to the comparison against other methods, not to the internal sweep. As the fixed-α\alpha row of Table 8 shows, with α=4\alpha=4 held constant SCMNet already surpasses every compared method on all six benchmarks; the sweep then raises α\alpha only to harvest further gains where each task carries many classes. We have rewritten the sentence to remove this ambiguity.

Tables 2 & 9 (reproduced): the two referenced values

SourceAˉ\bar{A}ANA_N
Table 2, APC-only row (20-Task IN-R)83.5579.90
Table 9, α=0\alpha=0 row (20-Task IN-R)83.5579.90
α\alpha (20-Task IN-A)Aˉ\bar{A}ANA_N
071.5761.49
272.0461.49
4 (adopted)72.1761.49
671.6961.55
1463.9354.44

Table 8 (reproduced): SCMNet with α=4\alpha=4 fixed vs. the best compared method, Aˉ\bar{A}

BenchmarkSCMNet (α=4\alpha=4)Best compared
10-Task CIFAR94.8094.67 (KCEA)
5-Task IN-R89.7286.01 (SR²-LoRA)
10-Task IN-R88.8586.01 (KCEA)
20-Task IN-R86.0784.32 (SR²-LoRA)
20-Task IN-A72.1769.41 (SR²-LoRA)
5-Task VTAB95.0694.69 (KCEA)

Thank you for the authors' detailed responses and additional experiments. The revisions have addressed most of my concerns,the current results provide stronger support for the proposed design. Therefore, am raising rating to 3.

Comments on the paper

1 comment

Dear Reviewers,

As the discussion phase comes to an end, we'd appreciate it if you could take a moment to review the authors' responses. If you have any remaining concerns or questions, please raise them with the authors before the discussion closes.

Best,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.