- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates whether, when using the store-logit replay, the output for classes not yet observed should be distilled for class incremental learning. By testing different losses and masking strategies, the authors find that the impact depends on the scenario.
Critical strengths
- The experiment findings might be robust as it ran experiments in multiple class orders, optimization seeds, and reported the 95% intervals and error bars in the figures.
Critical weaknesses
- I find this paper very hard to understand. The problem scenario, research question, review of existing work, and algorithm design are not clearly stated or defined.
- Without understanding these points, it is very hard for me and the readers to understand the contribution and the experimental sections.
- For example, what is stored-logit replay? The paper keeps citing [1] when mentioning stored-logit replay, but which instance's logit is stored, how the stored logit is previously used, and how the proposed method differs from existing methods are not clear.
- In the abstract and Intro, the paper states that discarding "output for classes not yet observed" is a harmless simplification. However, why is "output for classes not yet observed" a problem or "harm" in the first place?
- In Section 4, the loss function design is very unclear to me. What is ? How is it different from [1]?
- If I understand correctly, the paper tries to understand and improve stored-logit distillation for class incremental learning. However, the motivation and significance of stored-logit distillation are unclear in this paper. Is it because stored-logit distillation is the state-of-the-art? The main paper does not provide any justification or comparison with other branches of methods, which makes its contribution and significance unclear. Meanwhile, it seems like the proposed method and study are only applicable to vision models and classification tasks. Therefore, the paper needs more justification regarding its scope.
[1] Dark experience for general continual learning: A strong, simple baseline. NeurIPS 2020
Questions for authors
My questions are listed in Critical Weakness. I am willing to adjust my assessment if I have misunderstood this paper.
7 comments
W4. "In Section 4, the loss function design is very unclear to me": every symbol, including , and the difference from [1]
We do not propose a new method. Section 4 takes the DER++ distillation term apart into three versions, so that deleting entries and re-weighting the remaining entries can be studied separately.
| Symbol | Meaning |
|---|---|
| logit of class saved with replayed image | |
| current logit of class for image | |
| difference between current and saved logit | |
| , | number of replayed images in a batch, number of outputs () |
| 1 if class had been seen when image was saved, else 0 | |
| fraction of entries with in the batch | |
| cross-entropy on current-task training data | |
| cross-entropy on replayed images and their saved labels (the second replay term of DER++) | |
| logit distillation, one of the three versions below |
| Version | Definition | What it does |
|---|---|---|
| unmasked | average of over all entries | the DER++ distillation term [1] |
| masked/full | sum of over entries with , divided by | deletes the unseen entries, changes nothing else |
| masked/active | the same sum divided by the number of entries with | deletes them and enlarges every remaining entry by |
The training loss is with and . With the unmasked version this is the DER++ objective. The only differences from [1] are notation (DER++ swaps the names and ), one shared replay minibatch for both replay terms instead of two, and memory insertion during the last epoch of each task. A common way to implement masking corresponds to masked/active, which changes two things at once. Separating the two is the first contribution of the paper.
Revision: Section 3 now defines the setting with the running example. Section 4 states the relation to DER and DER++ and points to a new symbol table in the appendix.
W5. "The motivation and significance of stored-logit distillation are unclear. Is it because stored-logit distillation is the state-of-the-art?"
Stored-logit distillation matters because it is widely used, not because we claim it is state of the art. DER and DER++ are standard replay baselines and building blocks of later methods such as X-DER, and DER++ is a standard replay baseline for pre-trained models in SLCA.
Every such method must decide which saved outputs to keep. Our answer: the outputs of classes learned after storage decide whether masking helps or hurts, so a method should keep or drop them deliberately. The outputs of still-unseen classes can be dropped at little cost.
The broader findings are that (i) deleting distillation terms and re-weighting the rest are different operations, (ii) the value of a constraint depends on which updates it restricts and not on its strength, and (iii) forgetting alone can mislead, so the acquisition of new classes must be measured too. The paper analyzes one widely used objective rather than proposing a new method, so we compare objectives under a controlled recipe instead of ranking method families. In new experiments, masking again raises cross-task errors with separate backbone and classifier learning rates, with KL distillation, with a growing classifier, on CUB-200, ImageNet-R, and Stanford Cars, with a ViT-B/16 pre-trained on ImageNet-21K in SLCA's CUB-200 setting, and inside the official DER++ code.
Revision: Section 6 adds the implications for methods that distill stored logits.
W6. "It seems like the proposed method and study are only applicable to vision models and classification tasks"
The mechanism needs only two things: a classifier with one output per class, and a memory that stores predictions to distill later. Nothing in it is specific to images, but all our experiments are image classification, so the revision states the scope as class-incremental image classification with stored-logit replay.
Revision: scope statement in Section 6.
We upload a revised PDF, with changes marked in magenta. It (1) defines stored-logit replay precisely in Section 3, (2) adds the running example above, (3) adds a symbol table, referenced from Section 4, (4) states the relation to DER and DER++ in Section 4, and (5) states the scope. We hope this clarifies the manuscript, and we would be grateful to hear whether anything remains unclear.
Response to the rebuttal
I thank the authors for their detailed rebuttal. The rebuttal has helped clarify several points. However, my main concern remains unresolved. The study is limited to DER and its variants, with experiments conducted exclusively on class-incremental image classification. While the paper provides an analysis of these methods, I do not find the resulting insights sufficiently novel or generalizable to constitute a substantial contribution to the broader community. I therefore maintain my original score.
Thank you for the frank feedback. We agree that the paper assumed too much familiarity with stored-logit replay. Below we explain the setting with one running example and answer each point. The revised PDF adds this example and a precise definition of the setting in Section 3.
W1 and W2. "What is stored-logit replay?", "which instance's logit is stored, how the stored logit is previously used"
A replayed image is stored together with the model's outputs for that image at the moment of storage. Later, the model is trained to keep its outputs on that image close to the stored ones.
W3. "Why is 'output for classes not yet observed' a problem or 'harm' in the first place?"
Running example: an image saved in task 1 has 100 saved logits, but only 10 belong to classes the model had trained on. The other 90 belong to classes of which the model has not seen a single example, so distilling them looks pointless, and dropping them looks like a harmless cleanup.
The paper tests whether this cleanup is harmless. It is not. When the same image is replayed in task 5, 40 of the 90 dropped entries belong to classes the model has learned since storage. Only 50 still belong to future classes. Our new experiments show that the entries of classes learned since storage decide the outcome. Keeping them and dropping only the future entries recovers 85% to 102% of the accuracy lost by dropping all 90. On average, the entries of truly future classes can be dropped at little cost.