- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the core trade-off in pre-trained model (PTM)-based class-incremental learning (CIL): expandable architectures suffer from unbounded parameter growth as tasks accumulate, while fixed-parameter methods struggle to preserve performance on historical classes. The authors propose SCMNet (Structured Compression Memory Network), a rehearsal-free, fixed-parameter framework that reframes CIL as a feature space organization problem rather than a parameter adaptation problem.
SCMNet is built on a frozen pre-trained ViT backbone with lightweight fixed adapters, and consists of two mutually reinforcing modules: 1). Dynamic Neighborhood Compression (DNC): Constructs label-constrained similarity graphs from training batches, and applies two-stage graph convolution to compress intra-class feature variance while preserving inter-class boundaries. 2). Adaptive Prototype Constraint (APC): Maintains a prototype memory bank accumulated across tasks, and uses these prototypes as virtual anchors to stabilize historical decision boundaries. The constraint strength adapts automatically based on how crowded the prototype space is. The two modules form a closed-loop synergy: DNC produces tighter, more reliable prototypes that improve APC supervision, while APC stabilizes the feature space geometry to enable more effective compression in subsequent tasks. With a fixed ~1.7M trainable parameter budget regardless of task length, SCMNet achieves state-of-the-art performance across six benchmarks and three pre-trained backbones. The work also includes in-depth geometric analysis of the feature space to explain the method's effectiveness.
Critical strengths
- Rigorous and fair experimental design: All methods are evaluated under a unified protocol with the same frozen ViT-B/16 backbone, identical task splits, and consistent metrics (average accuracy and last-task accuracy ). The comparison covers 10+ state-of-the-art methods across six benchmarks and three task lengths (5/10/20 tasks), ensuring comprehensive benchmarking.
- Comprehensive ablation studies: Component ablations clearly demonstrate the super-additive effect of DNC and APC (their combined gain exceeds the sum of individual gains), verifying their synergistic design. Additional ablations on adapter placement, DNC residual weight , and backbone generalization systematically disentangle contributing factors and rule out confounding variables.
- Strong reproducibility: The appendix includes a detailed reproducibility statement with fixed random seeds, disabled non-deterministic CUDA kernels, and pinned hyperparameters. All backbones and datasets are publicly available, and the authors commit to releasing full code and configuration files.
- Mechanistic interpretability: The intra/inter-class variance analysis provides concrete geometric evidence for how SCMNet reshapes the feature space, rather than treating the model as a black box. The trend analysis across task sequences clearly illustrates the difference between SCMNet's progressive refinement and SEMA's repeated space re-establishment.
Critical weaknesses
- The prototype memory bank introduces storage overhead that grows linearly with the total number of classes, which is not quantified or discussed. While trainable parameters are fixed, prototype storage could become a practical bottleneck for ultra-large-scale incremental learning with thousands of classes.
- Inference-time computational cost (FLOPs, latency) is not reported. It is unclear whether the DNC module introduces any inference overhead relative to simpler fixed-parameter baselines, which is relevant for the deployment motivation of the work.
- Some design choices lack sufficient justification. For example, the two-stage graph convolution design (channel compression + node calibration) is not compared to alternative aggregation strategies; the rationale for fusing dual expert adapters via element-wise addition (rather than learned routing) is not fully explained.
- The adaptive weighting formula in APC () is presented without comparison to other possible adaptive strategies (e.g., linear scaling, exponential decay), leaving it unclear whether this specific formulation is critical to performance.
- Performance gains are uneven: improvements are very large on benchmarks with large domain gaps from ImageNet pre-training (IN-R, IN-A), but more modest on small-domain-gap datasets (CIFAR, VTAB). The practical impact is strongest for cross-domain incremental learning scenarios.
- The method is only evaluated on standard disjoint, balanced class-incremental settings. Its behavior in more realistic challenging scenarios (few-shot incremental learning, class-imbalanced increments, online CIL) remains untested.
Based on the comments above, the current manuscript has some limitations, and I give a Weak Reject at current stage.
Questions for authors
- The paper emphasizes fixed trainable parameters, but the prototype memory bank scales linearly with the total number of classes. Please quantify the per-class storage cost of the prototype bank (e.g., in bytes) and discuss its scalability to very large class sets (e.g., 1000+ classes). If feasible, a brief ablation on prototype compression (e.g., quantization, dimensionality reduction) would help assess the method's practical limits. This would clarify the full storage footprint of the framework.
- DNC uses a two-stage design: per-sample channel compression via 1×1 convolution followed by graph message passing. Please provide a brief ablation comparing this design to simpler alternatives, such as (a) direct graph convolution without channel compression, or (b) attention-based neighborhood aggregation. This would verify whether the two-stage structure is critical to performance.
- Since deployment efficiency is a key motivation for fixed-parameter methods, please add a comparison of inference FLOPs and/or latency between SCMNet and representative baselines (SEMA, -LoRA, SimpleCIL). It would be useful to confirm that the fixed parameter budget translates to fixed inference cost, and quantify any overhead relative to simpler methods.
- he paper observes larger gains on large-domain-gap benchmarks and smaller gains on small-domain-gap datasets. Please provide a more quantitative analysis of how SCMNet's accuracy gain correlates with the magnitude of domain shift from pre-training. For example, measuring the feature distance between ImageNet-1K and each target dataset, and plotting it against the gain over a baseline. This would help practitioners predict when SCMNet will provide the most benefit.
3 comments
Response to Reviewer QQwd (Part 2 - Q3, Q4)
Q3
Reviewer: Since deployment efficiency is a key motivation for fixed-parameter methods, please add a comparison of inference FLOPs and/or latency between SCMNet and representative baselines (SEMA, SR$^2$-LoRA, SimpleCIL). It would be useful to confirm that the fixed parameter budget translates to fixed inference cost, and quantify any overhead relative to simpler methods.
Response. Table 6 reports the inference budget. SCMNet costs G FLOPs versus G for SEMA, a gap of G (below 0.25%), and the DNC enhancer contributes only G because at inference its graph degenerates to an identity and only the per-node channel transforms remain. The gap is owed chiefly to SCMNet's larger initial adapter stack (405K vs. 304K), not to DNC. Since DNC adds no per-task computation and the memory bank is disabled at inference, the fixed parameter budget translates directly into a fixed inference budget, whereas SEMA adds G FLOPs and K parameters per expansion and eventually overtakes SCMNet in both.
Table 6 (reproduced): parameter and inference cost budget
| Method | ViT (frozen) | Adapters | Auxiliary | FLOPs | FLOPs/Exp. | Expansion |
|---|---|---|---|---|---|---|
| SEMA | $\sim$86M | $\sim$304K | $\sim$600K | $\sim$17.62G | G | Dynamic |
| SCMNet | $\sim$86M | $\sim$405K | $\sim$1180K | $\sim$17.66G | 0 | None (fixed) |
Q4
Reviewer: The paper observes larger gains on large-domain-gap benchmarks and smaller gains on small-domain-gap datasets. Please provide a more quantitative analysis of how SCMNet's accuracy gain correlates with the magnitude of domain shift from pre-training. For example, measuring the feature distance between ImageNet-1K and each target dataset, and plotting it against the gain over a baseline.
Response. We do not reduce this to a measured distance-gain curve, because the six benchmarks do not sit on a single domain-gap axis. IN-R shifts through rendition and style, IN-A through natural adversarial examples, VTAB through task diversity, and CIFAR through a distribution close to the pretraining data, so a scalar ImageNet-1K feature distance would collapse these distinct shift types onto one number and would not order the benchmarks meaningfully. The gain also depends on how strongly each baseline responds to the specific shift, so a distance-gain scatter would confound the shift magnitude with the baseline's own sensitivity. What the paper establishes instead is the mechanism on the benchmark where the question matters most. On IN-R, the largest-gap benchmark, Figure 5 shows that DNC's compression is predominantly intra-class, about one and a half times its reduction of the inter-class term, which is the component a domain shift inflates. The same account covers the one boundary case: at 20 tasks CIFAR's small gap leaves little intra-class variance for DNC to compress, while SR$^2$-LoRA's SVD alignment, which benefits from inter-layer drift, overtakes SCMNet there.
Gain over SEMA by domain gap (, from Table 1), ordered by gain
| Benchmark (domain gap) | SCMNet | SEMA | Gain |
|---|---|---|---|
| 20-Task IN-A (large) | 72.17 | 64.53 | +7.64 |
| 5-Task IN-R (large) | 91.14 | 84.75 | +6.39 |
| 10-Task IN-R (large) | 89.91 | 83.56 | +6.35 |
| 20-Task IN-R (large) | 86.07 | 81.75 | +4.32 |
| 5-Task VTAB (small) | 95.06 | 91.26 | +3.80 |
| 10-Task CIFAR (small) | 94.80 | 91.37 | +3.43 |
The remaining weaknesses other than the four issues above are explained as follows.
(1) Dual-adapter fusion. The dual experts are fused by element-wise addition precisely because it adds zero parameters and zero routing modules; a learned router would reintroduce per-task routing parameters of the kind SEMA's expandable router incurs, contradicting the fixed budget. Table 3 shows adapter capacity is not the driver (single versus the best dual configuration differs by under 0.59 points of ), so a routing mechanism would add parameters without adding accuracy.
(2) The adaptive weight form. is chosen because it is monotone, bounded (operating between the fixed values we ablate), and needs no extra hyperparameter, unlike linear or exponential schedules which each require additional slope or decay-rate coefficients. The gain reported in Table 7 comes from the relaxation itself, not from the specific functional form.
(3) Untested settings. SCMNet is rehearsal-free and keeps a fixed budget with a prototype bank, so it extends naturally to few-shot and online CIL where rehearsal is restricted; class-imbalanced increments are a separate axis we leave to future work.
Thanks for preparing the detailed response. Some of my concerns have been addressed. However, I still have concerns regarding the rationale behind certain methodological choices, as the current experiments do not fully validate their necessity or advantages over alternative designs. In addition, the relationship between domain shift and performance gains remains insufficiently supported by quantitative evidence. Therefore, I will maintain my original score.
Response to Reviewer QQwd (Part 1 - Q1, Q2)
We sincerely thank you for your careful reading and constructive comments. The additions and revisions to the main text have been highlighted in blue and re-uploaded, and we respond to your comments point by point below.
Q1
Response. Each prototype is a single d=768-dimensional float32 vector, so the per-class cost is 768×4=3072 bytes ≈3 KB. The full bank is therefore negligible at every scale considered in the paper: IN-R and IN-A (200 classes) each need ≈0.6 MB, CIFAR-100 needs ≈0.3 MB, and VTAB needs ≈0.15 MB. Scaling to 1000 classes costs ≈3 MB and to 10,000 classes ≈30 MB, next to the frozen backbone's 86M parameters (≈344 MB) and even next to the 1.7M trainable parameters (≈6.8 MB). The bank is storage, not trainable capacity, it is disabled at inference, and it is a cost shared by every prototype-based CIL method. At 3 KB per class there is no practical need for compression; if a tighter footprint were required, quantization to float16 or int8 halves or quarters the cost with no architectural change.
Prototype bank storage budget (float32, d=768)
Q2
Response. Part (a) already exists as the "w/o Channel" row of Table 2, which removes the per-sample channel compression while keeping the graph message passing. It costs only 0.36 to 0.61 points of Aˉ (90.78/89.37/85.46 vs. 91.14/89.91/86.07), so the graph aggregation is the primary contributor and the channel stage is a supporting refinement. The two-stage structure is retained because the channel stage costs almost nothing in parameters yet improves every setting, but it is not where the gain lives. On part (b), an attention-based aggregation differs from ours mainly in the edge weights. Without a label constraint it would assign non-zero weights to cross-class neighbors and pull different classes together, which directly contradicts the intra-class-compression objective. The label-constrained mask, not the choice of cosine versus learned attention weights, is the design element that matters, and the A=I and "w/o Node" rows of Table 2 isolate exactly that element: with the mask removed or the message passing disabled, the gain disappears.
Table 2 (reproduced, excerpt): the two-stage decomposition on IN-R (Aˉ / AN)