- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates what stored logits need to preserve in replay-based class-incremental learning. Through controlled interventions and task-level bias correction, the authors find that logits for later-learned classes can be replaced with fixed constants, whereas class correspondence among previously known classes remains important. Experiments are conducted on CIFAR-100 and TinyImageNet using various ResNet configurations. However, the extent to which these findings generalize beyond fixed-head settings remains unclear, and their potential to improve existing continual learning methods has not been empirically demonstrated.
Critical strengths
- The paper propose a clear and meaningful research question.
- The paper examines whether the original logits are necessary for logit matching and proposes a diagnostic framework to investigate this question.
- The authors conduct targeted experiments to investigate the research.
Critical weaknesses
My main concern is that the paper’s limited implications for broader continual learning methods, along with the unclear practical applicability of its findings to the wider research community, may make it difficult to meet the acceptance threshold.
-
Although the motivation of the paper is understandable, the Introduction does not sufficiently explain how each experimental design addresses the research questions raised earlier. Figure 1 summarizes the experimental results but provides limited intuition about why these specific interventions are necessary and how they collectively support the paper’s main contribution. Explicitly explaining the rationale behind each experiment and its connection to the corresponding research question would help readers better understand the paper’s overall logic and contribution.
-
The experiments are conducted primarily in a controlled setting with a fixed classifier head and explicit task partitions. Although the paper evaluates multiple datasets and ResNet configurations, it remains unclear to what extent the identified mechanisms generalize beyond these settings. Further discussion is needed to clarify whether the findings reflect general principles of replay-based continual learning or phenomena specific to the evaluated configurations. A more explicit discussion of the scope and limitations of the findings would strengthen the paper.
-
Although the paper provides useful diagnostic insights into the role of replayed logits, it does not demonstrate how these findings can be leveraged to improve existing replay-based continual learning (CL) methods. In particular, the paper does not evaluate whether selectively preserving or replacing logits can improve accuracy or reduce memory and computational costs. Developing and empirically validating methods based on these findings would strengthen the practical value and broader impact of the work on the CL community.
Questions for authors
-
In Figure 1, I believe that explaining why these experiments are designed this way is more important at this stage than presenting specific numerical results or accuracy drops.
-
How well do these findings generalize to other continual learning benchmarks and replay methods, and how might they inform the development of more memory-efficient replay strategies?
-
Would selectively replacing or preserving certain logits affect the accuracy-memory trade-off in existing replay-based methods?
-
How can the findings be applied to parameter-efficient fine-tuning methods, such as LoRA-based methods?
-
Does replacing stored logits with fixed constants work across different datasets and settings? The results on TinyImageNet seem less conclusive than those on CIFAR-100. Does this limit the scope of the main claim?
6 comments
Dear authors,
I appreciate the efforts to address my concerns, particularly the new experiments demonstrating the practical implications and broader applicability of the findings. However, some concerns remain regarding the generality and robustness of the results, particularly on TinyImageNet and in LoRA-based settings. In addition, the manuscript has expanded considerably from 29 to 63 pages without indicating where the changes have been made, making the revision difficult to assess. I acknowledge the improvements, but still have reservations about the strength and generality of the contributions.
Best,
Reviewer 7mYS
Thank you for the follow-up, and for noting that the new experiments demonstrate the practical implications and broader applicability of the findings, which was the main concern of your review. On the two points that remain, the evidence is specific, so we state it directly. Your comment does not say which results you find insufficiently robust, so we end with three short questions that would let us answer exactly.
TinyImageNet: the three core findings replicate.
- Later-class matching is a correctable bias. Deleting it costs 8.4 pp [7.1, 9.8] (eight seeds). After one task offset the cost is gone (8.3 → −1.1 pp; six pairs, offset cross-fitted on test folds, marked so in Table S22).
- Class correspondence is needed. Flattening the seen-class scores costs 4.1 pp [3.2, 4.9] and reassigning them 6.1 pp [5.1, 7.1] (four seeds, Table S16), both larger than on CIFAR-100. The reassignment cost stays after the offset (5.9 pp, three pairs).
- The stored later-class values are not needed. In both studies the constant was at least as accurate as the stored scores on average (Full − constant: −0.65 and −0.03 pp), and both 90% intervals exclude a loss of 1 pp (upper limits 0.18 and 0.69 pp). The shortfall is in the equivalence verdict, not in the constant's accuracy: the first interval, [−1.48, 0.18], crosses the margin on the side where the constant is better. The follow-up (ten of twelve planned pairs) passes the ±1 pp rule used throughout the paper, and not a stricter rule adopted for it after an interim look. Figure 2 and Table 3 report exactly this.
- The design consequence is largest here. The constraint without stored scores adds 9.3 pp to label replay in both kinds of stream (Table 2).
LoRA (your Q4): ten of ten planned comparisons. You asked how the findings apply to LoRA-based methods. Section 7 applies them directly. On ViT-B/16 with rank-8 LoRA and a growing head, a constraint that reads no stored score (four bytes per record) improved its baseline in all ten planned comparisons, each on eight paired seeds with Holm correction, across three datasets, streams with and without task boundaries, and baselines with and without stored scores (Table 2). Label replay gains 4.95 to 9.31 pp; the smallest gain of all ten is +1.32 pp [0.96, 1.68]. The result does not rest on TinyImageNet, which overlaps the backbone's pretraining data: on CIFAR-100 and ImageNet-R, label replay still gains 4.95 to 5.28 pp. Two limits are stated in the paper: a dense variant recovers most of the gain on overlapping streams, and the within-task price of the scratch model does not appear here.
Why we consider the contribution strong. A deletion ablation ranks the two parts of DER's target in the wrong order, and the paper shows this with matched controls. Deleting later-class matching costs 15 pp and reassigning seen-class scores 4 pp. After one scalar offset the costs are 1.8 and 4.0 pp, and the order reverses on all three backbones. The larger term is thus a bias correction: a constant replaces its stored values, it lowers within-task accuracy by 4.7 pp, and with an equal budget and the same correction, removing it wins by 2.37 pp [1.83, 2.92]. X-DER showed that these scores go stale, and BiC and WA that recency bias is correctable. None of them shows that the main benefit of DER's later-class matching is that correction, that its stored values are unnecessary, or that it has a within-task cost. The same reading changes how methods compare: ER-ACE's 10.9 pp lead over ER becomes −1.7 pp, while X-DER keeps 9.9 pp (Section 8).
Three questions, so that we can answer exactly.
- On TinyImageNet, which result do you consider not robust: the equivalence verdict for the constant, or one of the effects above (and if so, is the concern the number of seeds)?
- On LoRA, is the concern the consistency of the gains in Table 2, the single backbone and adapter, or the absence of the within-task price?
- On strength, is the remaining doubt about effect sizes, about novelty relative to X-DER, BiC and WA, or about a setting we did not test? If a setting, which one?
Where the changes are. We should have marked them. The main text is ten pages (nine before). Appendices A–E (Tables S1–S29) are the original appendix under the same numbering, and all 33 new pages are Appendices F–H. For your points: W1 and Q1 are answered in the Introduction, Figure 1 and Table 1; W2, Q2 and Q4 in Section 7 and Table 2; W3 in Figure 4c; Q3 in Figure 3b; Q5 in Figure 2 and Table 3. A map for all reviews is posted as "Summary of changes".
If these answers resolve your reservations, we would be grateful if your assessment could reflect that.
Practical gains from replay redesign across settings
Thank you for distinguishing an interesting diagnosis from a practically useful contribution. We apologize for the late reply. We understood your central concern to be that the original paper had not shown how its findings improve replay, especially beyond fixed heads. The revision now includes direct tests of those consequences.
Practical consequences (W3, Q2, Q3). Two design changes follow from the diagnosis, and both are tested.
A constraint without stored scores (Section 7, Table 2). The later-class values are replaceable, so the constraint can be built from the head width at storage (four bytes per record), for outputs that did not exist when the record was stored. On a pretrained ViT-B/16 with LoRA and growing heads it improves its parent in all ten planned comparisons:
On accuracy versus memory: without storing any score, label replay with the constraint closes about 70% of the gap to full stored-logit replay in the ten-task study (CIFAR-100 77.9 → 82.9% against 84.9%; TinyImageNet 70.1 → 79.4% against 83.5%; Table S75). This comparison is descriptive, because the reference uses a preallocated head.
Removing matching when both policies receive correction (Section 6, Figure 4c). Given 500 images, both policies replay 400 and fit task offsets on the other 100. Dropping later-class matching beats keeping it by 2.37 pp [1.83, 2.92] in deployed accuracy and by 4.82 pp within tasks. It also beats replaying all 500 images without correction by 4.19 pp [2.86, 5.52]. The correction needs known task shares.
The same reading also changes how methods compare. Across nine natively trained methods, differences from ER change under task offsets by very different amounts, from 13.40 → 11.11 pp for BiC to 10.88 → −1.69 pp for ER-ACE (Section 8).
W1, Q1. Why each intervention, and Figure 1. The Introduction now derives the design from two obstacles:
Table 1 pairs each edit with the question it answers. Figure 1 now asks three questions of one stored record (are the stored values needed, is the whole pattern needed, does higher accuracy protect discrimination) and answers each with its comparison.
W2, Q2. Beyond fixed heads and task partitions. Section 7 uses growing heads, a pretrained backbone, and overlapping streams without task identity, class list or stage reset. Section 8 adds nine native methods and a separate teacher. One difference is informative: the within-task price appears in the scratch model, not in the pretrained recipes. What transfers is therefore the design idea, not every scratch behavior. Table 3 states where each claim holds.
Q3. Selectively preserving logits. Keep the three largest non-gold scores of each earlier task at their classes and average the remaining non-gold scores: accuracy approaches Full (Full − Top-3: −0.02 pp [−0.44, 0.40]; descriptive). Three randomly chosen scores recover less, and reassigning the reduced target costs 3.58 pp relative to Top-3 (Section 5). Storage was dense; this identifies a promising compact target, not a measured memory saving or a minimum sufficient structure.
Q4. LoRA. Section 7 is this setting (rank-8 LoRA on ViT-B/16). On the overlapping streams, a dense variant that does not use which outputs came later recovers most of the gain; the largest observed gain from the exact support is 0.45 pp. Dense still uses insertion metadata to set its strength.
Q5. Constants across datasets. The constant −2 meets the ±1 pp margin on CIFAR-100 under DER++ and DER, at three memory sizes, on ResNet-32, and at 5 and 10 tasks; equivalence is not established at 20 tasks.
On TinyImageNet, the original eight-seed study is inconclusive. A follow-up with ten of twelve planned pairs passes the original 90% TOST (p = .019) but not a stricter rule adopted after an interim look. The constant 0 fails, and refreshed targets are an exception (6.4 pp cost under X-DER).
The claim is bounded accordingly: stored later-class values are not needed where equivalence was established, and the level of the constant matters.
We hope these additions address the practical-value gap central to your assessment. We would be grateful for your reconsideration in light of the improvements in Figure 4c and Table 2, and the clearer account of where they apply.