acceptodds
Under review as a conference paper at ICLR 2027

What Does Logit Distillation Preserve in Pre-Trained Class-Incremental Learning?

Abstract

Distilling stored predictions is a standard remedy for forgetting in class-incremental learning. However, in stored-logit replay, many of the stored targets describe classes the model has never trained on: with a fixed classification head, every saved example carries outputs for classes not yet observed. Discarding these unsupervised constraints and distilling only over observed classes appears to be a harmless simplification. In this paper, we investigate what these constraints preserve and show that the simplification is not harmless. Removing these constraints consistently helps the model acquire new classes, but its net effect on accuracy reverses between training settings. Amplifying the reduced distillation signal to the strength of the full constraint does not recover full-distillation accuracy: the remaining loss concentrates on discrimination between classes learned at different times rather than within any single task. Probing the removed terms reveals why. The apparently unsupervised coordinates act strongly on classes learned after the targets were stored, and thereby restrict updates that later tasks depend on. Stored-logit constraints thus preserve cross-task discrimination in a way that their magnitude alone does not explain, and their value must be judged by which updates they restrict rather than by how strongly they constrain.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
class-incremental learning, distillation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_mfn7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether, when using the store-logit replay, the output for classes not yet observed should be distilled for class incremental learning. By testing different losses and masking strategies, the authors find that the impact depends on the scenario.

Critical strengths
  1. The experiment findings might be robust as it ran experiments in multiple class orders, optimization seeds, and reported the 95% intervals and error bars in the figures.
Critical weaknesses
  1. I find this paper very hard to understand. The problem scenario, research question, review of existing work, and algorithm design are not clearly stated or defined.
  • Without understanding these points, it is very hard for me and the readers to understand the contribution and the experimental sections.
  • For example, what is stored-logit replay? The paper keeps citing [1] when mentioning stored-logit replay, but which instance's logit is stored, how the stored logit is previously used, and how the proposed method differs from existing methods are not clear.
  • In the abstract and Intro, the paper states that discarding "output for classes not yet observed" is a harmless simplification. However, why is "output for classes not yet observed" a problem or "harm" in the first place?
  • In Section 4, the loss function design is very unclear to me. What is LrepL_{rep}? How is it different from [1]?
  1. If I understand correctly, the paper tries to understand and improve stored-logit distillation for class incremental learning. However, the motivation and significance of stored-logit distillation are unclear in this paper. Is it because stored-logit distillation is the state-of-the-art? The main paper does not provide any justification or comparison with other branches of methods, which makes its contribution and significance unclear. Meanwhile, it seems like the proposed method and study are only applicable to vision models and classification tasks. Therefore, the paper needs more justification regarding its scope.

[1] Dark experience for general continual learning: A strong, simple baseline. NeurIPS 2020

Questions for authors

My questions are listed in Critical Weakness. I am willing to adjust my assessment if I have misunderstood this paper.

7 comments

Thank you for the frank feedback. We agree that the paper assumed too much familiarity with stored-logit replay. Below we explain the setting with one running example and answer each point. The revised PDF adds this example and a precise definition of the setting in Section 3.

Your pointShort answer
W1 Scenario, question, and design not clearly statedThe paper asks one question: when a stored prediction covers classes the model had not seen yet, should those entries be distilled? The introduction states it in italics at the end of its third paragraph, and Section 3 now defines the setting precisely
W2 What is stored-logit replay, and which logits are stored?A replayed image is saved together with the model's output vector at saving time, and later training keeps the outputs close to that vector (DER [1])
W3 Why would those outputs be a problem?The model has not seen a single example of those classes yet, so dropping them looks harmless. We show it is not, because many of them later belong to learned classes
W4 The loss in Section 4, its symbols, and its difference from [1]No new method. The unmasked loss is the DER++ objective. Two masked versions separate deleting terms from re-weighting the remaining ones. All symbols, including Lrep\mathcal{L}_{\text{rep}}, are defined below
W5 Why does stored-logit distillation matter?DER and DER++ are standard replay baselines and building blocks of later methods. Our results tell such methods which stored outputs to keep
W6 Scope beyond vision classificationThe mechanism needs only a per-class output and stored predictions. Our evidence covers image classification, and the revision states this scope

W1 and W2. "What is stored-logit replay?", "which instance's logit is stored, how the stored logit is previously used"

A replayed image is stored together with the model's outputs for that image at the moment of storage. Later, the model is trained to keep its outputs on that image close to the stored ones.

  • Setting. Classes arrive in groups called tasks. In our CIFAR-100 setting there are 10 tasks with 10 new classes each. After every task, the model must classify all classes seen so far without being told which task an image comes from. A memory keeps 2,000 past training images.
  • What is stored, and when. When an image enters the memory, Dark Experience Replay (DER) [1] also saves its label and the model's output vector for this image, with one logit per class. In our implementation, images enter the memory by reservoir sampling during the last epoch of each task, and the saved logits come from the forward pass just before the optimizer step.
  • How it is used. When the image is replayed later, a loss term Ldis\mathcal{L}_{\text{dis}} keeps the current outputs close to the saved vector. This is logit distillation. DER++ adds Lrep\mathcal{L}_{\text{rep}}, the cross-entropy of the current model on the replayed image and its saved label.
  • Our pipelines. The classifier has one output for each of the 100 classes from the start, which is common with pre-trained models. So every saved vector has 100 entries.

W3. "Why is 'output for classes not yet observed' a problem or 'harm' in the first place?"

Running example: an image saved in task 1 has 100 saved logits, but only 10 belong to classes the model had trained on. The other 90 belong to classes of which the model has not seen a single example, so distilling them looks pointless, and dropping them looks like a harmless cleanup.

The paper tests whether this cleanup is harmless. It is not. When the same image is replayed in task 5, 40 of the 90 dropped entries belong to classes the model has learned since storage. Only 50 still belong to future classes. Our new experiments show that the entries of classes learned since storage decide the outcome. Keeping them and dropping only the future entries recovers 85% to 102% of the accuracy lost by dropping all 90. On average, the entries of truly future classes can be dropped at little cost.

W4. "In Section 4, the loss function design is very unclear to me": every symbol, including Lrep\mathcal{L}_{\text{rep}}, and the difference from [1]

We do not propose a new method. Section 4 takes the DER++ distillation term apart into three versions, so that deleting entries and re-weighting the remaining entries can be studied separately.

SymbolMeaning
zicz_{ic}logit of class cc saved with replayed image ii
fθ(xi)cf_\theta(x_i)_ccurrent logit of class cc for image ii
eic=fθ(xi)c−zice_{ic}=f_\theta(x_i)_c-z_{ic}difference between current and saved logit
BB, CCnumber of replayed images in a batch, number of outputs (C=100C=100)
MicM_{ic}1 if class cc had been seen when image ii was saved, else 0
qqfraction of entries with Mic=1M_{ic}=1 in the batch
Lcur\mathcal{L}_{\text{cur}}cross-entropy on current-task training data
Lrep\mathcal{L}_{\text{rep}}cross-entropy on replayed images and their saved labels (the second replay term of DER++)
Ldis\mathcal{L}_{\text{dis}}logit distillation, one of the three versions below
VersionDefinitionWhat it does
unmaskedaverage of eic2e_{ic}^2 over all B×CB\times C entriesthe DER++ distillation term [1]
masked/fullsum of eic2e_{ic}^2 over entries with Mic=1M_{ic}=1, divided by B×CB\times Cdeletes the unseen entries, changes nothing else
masked/activethe same sum divided by the number of entries with Mic=1M_{ic}=1deletes them and enlarges every remaining entry by 1/q1/q

The training loss is Lcur+α Lrep+β Ldis\mathcal{L}_{\text{cur}}+\alpha\,\mathcal{L}_{\text{rep}}+\beta\,\mathcal{L}_{\text{dis}} with α=.1\alpha=.1 and β=.5\beta=.5. With the unmasked version this is the DER++ objective. The only differences from [1] are notation (DER++ swaps the names α\alpha and β\beta), one shared replay minibatch for both replay terms instead of two, and memory insertion during the last epoch of each task. A common way to implement masking corresponds to masked/active, which changes two things at once. Separating the two is the first contribution of the paper.

Revision: Section 3 now defines the setting with the running example. Section 4 states the relation to DER and DER++ and points to a new symbol table in the appendix.

W5. "The motivation and significance of stored-logit distillation are unclear. Is it because stored-logit distillation is the state-of-the-art?"

Stored-logit distillation matters because it is widely used, not because we claim it is state of the art. DER and DER++ are standard replay baselines and building blocks of later methods such as X-DER, and DER++ is a standard replay baseline for pre-trained models in SLCA.

Every such method must decide which saved outputs to keep. Our answer: the outputs of classes learned after storage decide whether masking helps or hurts, so a method should keep or drop them deliberately. The outputs of still-unseen classes can be dropped at little cost.

The broader findings are that (i) deleting distillation terms and re-weighting the rest are different operations, (ii) the value of a constraint depends on which updates it restricts and not on its strength, and (iii) forgetting alone can mislead, so the acquisition of new classes must be measured too. The paper analyzes one widely used objective rather than proposing a new method, so we compare objectives under a controlled recipe instead of ranking method families. In new experiments, masking again raises cross-task errors with separate backbone and classifier learning rates, with KL distillation, with a growing classifier, on CUB-200, ImageNet-R, and Stanford Cars, with a ViT-B/16 pre-trained on ImageNet-21K in SLCA's CUB-200 setting, and inside the official DER++ code.

Revision: Section 6 adds the implications for methods that distill stored logits.

W6. "It seems like the proposed method and study are only applicable to vision models and classification tasks"

The mechanism needs only two things: a classifier with one output per class, and a memory that stores predictions to distill later. Nothing in it is specific to images, but all our experiments are image classification, so the revision states the scope as class-incremental image classification with stored-logit replay.

Revision: scope statement in Section 6.


We upload a revised PDF, with changes marked in magenta. It (1) defines stored-logit replay precisely in Section 3, (2) adds the running example above, (3) adds a symbol table, referenced from Section 4, (4) states the relation to DER and DER++ in Section 4, and (5) states the scope. We hope this clarifies the manuscript, and we would be grateful to hear whether anything remains unclear.

Reviewer_mfn7OpenReview ↗

Response to the rebuttal

I thank the authors for their detailed rebuttal. The rebuttal has helped clarify several points. However, my main concern remains unresolved. The study is limited to DER and its variants, with experiments conducted exclusively on class-incremental image classification. While the paper provides an analysis of these methods, I do not find the resulting insights sufficiently novel or generalizable to constitute a substantial contribution to the broader community. I therefore maintain my original score.

Reviewer_1kYEOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies which output coordinates should be preserved in stored-logit replay. It separates support masking, loss normalization, and distillation weight within a DER++-style framework. Experiments use pretrained ResNet-18 and ViT-Tiny models on CIFAR-100 and ImageNet-100, with full-parameter fine-tuning. Gradient-norm matching and local diagnostics are used to interpret the role of omitted coordinates in cross-task discrimination.

Critical strengths
  1. Separating support deletion from loss normalization is a useful experimental distinction.
  2. The paired experiments include multiple class orders and seeds. Reporting acquisition, forgetting, and overall accuracy together helps expose trade-offs that forgetting alone can obscure.
Critical weaknesses
  1. The relevance of the findings to representative pretrained CIL methods remains unestablished. The central interventions are confined to a custom fixed-head, stored-logit replay recipe. Pretrained full fine-tuning is an established research setting, as demonstrated by SLCA and CoFiMA [3, 4], but the paper does not demonstrate how its analysis explains these methods or informs their design. For pretrained ResNet, CUB-200 and Stanford Cars would provide relevant fine-grained transfer tests. For pretrained ViT, validation should include representative methods available in PILOT [2] and broader transfer benchmarks such as ImageNet-R, ImageNet-A, CUB-200, and Stanford Cars. The current experiments establish neither robustness across these settings nor an actionable connection to established methods. The supplementary comparisons do not close this gap: the Mammoth experiments train ResNet-18 from scratch without repeating the central interventions; the CL-LoRA-style[5] comparisons do not test the proposed mechanism; and the adapted X-DER omits components of the reference method. The concern is the absence of substantive validation within existing methods, beyond additional benchmark scores.

  2. Known optimization issues in pretrained continual learning are not adequately controlled. The main recipe uses a single initial learning rate of 0.005 for all backbone and classifier parameters. SLCA [3] specifically identifies uniform learning rates, including 0.005, as a source of poor performance in pretrained continual learning and evaluates separate backbone and classifier learning rates, including for DER++. This does not prove that the present results are optimization artifacts, but it identifies a concrete alternative explanation that the paper should test. Without such controls, it is unclear whether the observed acquisition–retention trade-offs and cross-task errors persist under a suitably configured pretrained fine-tuning recipe.

References

[1] Mammoth continual learning framework. Github Repository · Sequential CIFAR-100 configuration

[2] Sun et al. PILOT: A Pre-Trained Model-Based Continual Learning Toolbox. SCIENCE CHINA Information Sciences, 2025. https://arxiv.org/abs/2309.07117)

[3] Zhang et al. SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model. ICCV, 2023. https://arxiv.org/abs/2303.05118

[4] Marouf et al. Weighted Ensemble Models Are Strong Continual Learners. ECCV, 2024. https://arxiv.org/abs/2312.08977

[5] He et al. CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning. CVPR, 2025. https://arxiv.org/abs/2505.24816

Questions for authors
  1. Do the central findings persist on CUB-200 and Stanford Cars for pretrained ResNet, and within representative pretrained ViT methods on broader transfer benchmarks?
  2. What changes when backbone and classifier learning rates are selected separately, following the issue identified by SLCA? Do the findings survive replication under the reference DER++ recipe?
  3. On the same seven order–seed pairs, how do unmasked, unamplified masked, and norm-matched masked training compare? How large does the amplification coefficient become?
  4. Do the conclusions persist with seen-class-only cross-entropy and selective retention of different omitted-coordinate groups? Which established method can directly benefit from the resulting explanation?
4 comments

We thank you for the detailed review. Each question names a specific control, and we ran new experiments for each, with predictions written before the runs started.

Your pointWhat we addedResult
W2, Q2 Uniform learning rate, as identified by SLCASLCA slow-learner rates on CIFAR-100 and ImageNet-100, both pipelinesThe acquisition gain and the cross-task errors persist in all 4 settings. Accuracy falls in every class order, only slightly for ImageNet-100 ViT
Q2 Reference DER++ recipeDER++ in SLCA's CUB-200 setting on ViT-B/16, and masking inside the official Mammoth DER++Masking lowers FAA in both, by 4.94 (0/3) and 15.14 (0/5), and raises cross-task errors in every run
W1, Q1 CUB-200, Stanford Cars, ImageNet-R, ImageNet-AAll four, both pipelinesMasking lowers FAA in all 24 orders (4 benchmarks x 2 pipelines) and raises cross-task errors in every cell
Q3 Same 7 pairs, three arms, coefficient sizeUnamplified masked arm, coefficient per taskNorm matching is worst in both pipelines. Median coefficient 97 (ResNet-18) and 337 (ViT)
Q4 Seen-class-only CE, selective retentionSeen-class-only CE. Add back only N or only FThe effect persists and grows. N decides the effect, F can be dropped
W1, Q4 Which established method benefitsDesign rules derived from the above**F can be dropped at replay. N decides the effect, and two tasks usually predict its sign (75 of 81). ** X-DER: see our later results post
W1 Supplementary comparisonsCentral intervention inside official Mammoth DER++Masking costs 15 pp on average there too (Q2). CL-LoRA and X-DER rows relabeled

Notation. U is unmasked and M is masked without amplification. For a replayed example, N are classes learned after it was stored and F are classes still unseen. Entries are FAA differences in pp: mean over class orders, or paired seeds for Mammoth, [95% CI] (orders or seeds positive / total). The protocol is at the end of part 3.

W2 and Q2. "A single initial learning rate of 0.005 for all backbone and classifier parameters"

The tradeoff is not an artifact of the uniform learning rate. With SLCA's rates (backbone 1e-4, classifier 1e-2), masking still raises acquisition and cross-task errors in all four settings, and lowers accuracy in every class order, only slightly for ImageNet-100 ViT.

Slow learner, M vs UΔFAAΔAcquisitionΔGFAA of U
CIFAR-100, ResNet-18−6.76 [−8.94, −4.58] (0/5)+8.64+8.5154.2
CIFAR-100, ViT-Tiny−3.73 [−4.61, −2.86] (0/5)+2.59+5.8774.2
ImageNet-100, ResNet-18−6.47 [−13.15, +0.22] (0/3)+14.85+8.7669.6
ImageNet-100, ViT-Tiny−0.40 [−1.18, +0.38] (0/3)+1.03+3.5575.5

The learning rate changes the net balance, not the mechanism. This is the setting dependence the paper reports.

Before the runs we predicted that the acquisition gain and the cross-task increase would persist, that the accuracy effect could shrink because a slow backbone limits what the omitted constraints protect, and that CIFAR-100 would stay negative. All three held. The only change of sign is ImageNet-100 ViT, where the gain of +2.9 pp under the uniform rate disappears. CIFAR-100 uses 5 orders with 2 seeds, ImageNet-100 uses 3 orders.

Revision: new appendix on separate learning rates, referenced in Section 5.

Q2. "Do the findings survive replication under the reference DER++ recipe?"

Yes. In both reference settings, masking lowers final accuracy and raises acquisition and cross-task errors in every class order or seed.

Reference settingFAA, U → MΔFAAΔAcquisitionΔG
DER++ in SLCA's CUB-200 setting, ViT-B/16, 3 orders73.25 → 68.31−4.94 [−7.71, −2.16] (0/3)+3.43 (3/3)+6.16 (3/3)
Official Mammoth DER++, seq-CIFAR-100, 5 seeds36.56 → 21.42−15.14 [−15.93, −14.35] (0/5)+5.21 (5/5)+20.44 (5/5)

The effect also appears in the official Mammoth DER++, trained from scratch. Adding only the mask lowers mean FAA by 15.14 pp.

  • DER++ in SLCA's CUB-200 setting, our implementation. ViT-B/16 supervised on ImageNet-21K, backbone 1e-4 and classifier 1e-2, 50 epochs, 1,000 replay examples. Within-task accuracy also rises in every order (+1.22). SLCA reports 77.42 for DER++ in this setting.
  • Official Mammoth DER++. The official code with its best sequential CIFAR-100 configuration (buffer 500, 50 epochs, ResNet-18 from scratch, fixed class order). U is the unmodified code. M changes only the logit-matching term, which then ignores outputs of classes unseen at storage, with the same denominator. The prediction written before the runs held in all five paired seeds.

Revision: both results in a new appendix.

W1 and Q1. "CUB-200 and Stanford Cars would provide relevant fine-grained transfer tests", "ImageNet-R, ImageNet-A, CUB-200, and Stanford Cars"

On all four benchmarks and in both pipelines, masking lowers final accuracy in every class order and raises cross-task errors.

M vs UResNet-18, ΔFAAResNet-18, ΔGViT-Tiny, ΔFAAViT-Tiny, ΔG
CUB-200 (10 x 20 classes)−11.47 (0/3)+11.97−5.07 (0/3)+6.25
ImageNet-R (10 x 20 classes)−12.76 (0/3)+18.20−0.65 (0/3)+12.11
ImageNet-A (10 x 20 classes)−1.89 (0/3)+4.30−1.89 (0/3)+6.22
Stanford Cars (7 x 28 classes)−11.87 (0/3)+14.65−6.22 (0/3)+9.42

The predicted pattern held in all 8 cells: masking raises acquisition and within-task accuracy W, and raises cross-task errors G even more. The CIFAR-100 tradeoff of the paper is the typical case, not an exception.

Each benchmark uses the official split where one exists (a fixed 80/20 split for ImageNet-R and ImageNet-A), 3 random class orders with 2 seeds, and 1,000 replay examples as in SLCA for these benchmarks. Before the runs we predicted that ΔW, ΔG, and ΔAcquisition would all be positive in every dataset and pipeline, and left the sign of ΔFAA open.

Revision: new appendix on transfer benchmarks, summarized in Section 5.

Q3. "On the same seven order and seed pairs, how do unmasked, unamplified masked, and norm-matched masked training compare? How large does the amplification coefficient become?"

Restoring strength moves the learner further from unmasked training, not back toward it. The order is U > M > norm-matched for ResNet-18 and M > U > norm-matched for ViT.

Same 7 pairs, ΔFAAM vs UNorm-matched vs MNorm-matched vs U
ResNet-18−13.89 [−18.79, −8.98] (0/7)−27.21 [−30.00, −24.42] (0/7)−41.10
ViT-Tiny+3.41 [+2.44, +4.38] (7/7)−35.40 [−37.41, −33.38] (0/7)−31.98

The amplification coefficient ∥gU∥/∥gM∥\|g_U\|/\|g_M\|, measured on about 36k replay batches per pair:

TaskResNet-18, median [p10, p90]ViT-Tiny, median [p10, p90]
223.5 [13.2, 39.9]143.6 [33.2, 1252.8]
6111.5 [62.4, 194.1]429.7 [137.8, 3926.5]
10197.3 [89.6, 396.3]358.3 [154.2, 3969.7]
All batches97.3 [27.4, 248.5]336.6 [90.4, 3118.4]

The coefficient is itself a finding: on masked trajectories, the retained coordinates carry only about 1% (ResNet-18) and 0.3% (ViT) of the full distillation gradient. It grows over tasks, as more stored outputs become N outputs. Because a large coefficient could make norm matching look unstable, we also trained static weights of 4, 16, and 64 times the default on the first three pairs, which need no per-batch coefficient. M vs U for ResNet-18 moves from −9.18 at the default weight to −10.66, −17.60, and −46.46. None of these weights brings ResNet-18 back to U. The ViT norm-matched runs come from the paper and ran on a different GPU. Same-configuration U runs on the two GPUs differ by only +0.16 [−0.06, +0.38] pp.

Revision: three-arm table, coefficient table, and static weights in a new appendix.


W1. "The supplementary comparisons do not close this gap"

We agree that these comparisons did not test the mechanism. The new Mammoth experiment does.

  • Mammoth: the central intervention now runs inside the official DER++ code and lowers mean FAA by 15.14 pp there (Q2 above).
  • CL-LoRA-style rows: they serve the relative-repair analysis and do not test the mechanism. The revision says so where they appear.
  • Adapted X-DER: this condition appears only in the appendix tables. The revision renames it "KL target update" and states there that it is a specified recipe adapted from X-DER, not a reproduction of X-DER.

Q4. "Do the conclusions persist with seen-class-only cross-entropy and selective retention of different omitted-coordinate groups?"

Yes to both. With seen-class-only cross-entropy the effect is larger. Selective retention shows that the outputs of classes learned after storage (N) decide the effect, and the outputs of still-unseen classes (F) can be dropped.

CIFAR-100, 5 ordersΔFAA (M vs U)ΔAcquisitionΔG
Seen-class-only CE, ResNet-18−17.20 [−19.47, −14.93] (0/5)+2.06+19.75
Seen-class-only CE, ViT-Tiny−11.07 [−12.01, −10.12] (0/5)+1.37+13.33

The corresponding within-task accuracy differences ΔW are +2.55 (ResNet-18) and +2.26 (ViT-Tiny).

Selective retentionM vs UKeep N (S∪N) vs UKeep F (S∪F) vs ULoss recovered by Nby F
CIFAR-100, ResNet-18−9.22+0.20−8.55102%7%
CIFAR-100, ViT-Tiny−9.36−1.37−8.4185%10%
ImageNet-100, ResNet-18−13.89−0.40−13.7597%1%
ImageNet-100, ViT-Tiny+3.41−1.39+4.02gain removedgain kept

With seen-class-only CE, a future output receives no gradient before its task, so the masked arm is functionally an expanding-head DER++. The selective-retention outcomes match predictions fixed before the runs (at least 0.8 for N and at most 0.2 for F where masking hurts). Under KL distillation on CIFAR-100, keeping N recovers 101% (ResNet-18) and 110% (ViT) of the loss, as also predicted.

Revision: selective retention becomes a new subsection in Section 5. Seen-class-only CE goes to a new appendix.

W1 and Q4. "The paper does not demonstrate how its analysis explains these methods or informs their design", "Which established method can directly benefit?"

Methods that distill stored logits (DER, DER++, X-DER, and DER++ as a replay baseline in pre-trained settings such as SLCA) get three concrete rules.

  1. Outputs of still-unseen classes can be dropped at replay time. On average, dropping only these stays within 1.4 pp of the unmasked objective in all six settings tested, under squared error and KL, and in the four squared-error settings, adding them back to a masked objective changes FAA by less than 1 pp.
  2. Outputs of classes learned after storage decide the net effect. Deleting them costs 8 to 14 pp in three settings and gains 4 pp in one. A head that grows with each task never stores these outputs, consistent with the 11 to 17 pp loss of the expanding-head protocol above. Such methods drop these constraints by design. Storing the full output vector, or refreshing these coordinates as X-DER does, keeps them.
  3. Two tasks usually tell which choice wins. Before the runs we named two early differences, both measured after task 2. The task-prediction difference predicted the sign of the final effect in 75 of 81 new order-level comparisons, and the seen-class accuracy difference (our main candidate) in 74. Five of the six misses of the first come from KL on ImageNet-100, where the early differences were small or faded by the end.

The rules connect to existing designs. X-DER updates the stored targets of all outputs unseen at storage once their classes are learned. LwF and iCaRL refresh the teacher at each task, so N is always empty for them.

SLCA's classifier alignment and CoFiMA's weight ensembling do not distill stored logits, and our analysis does not explain them. The revision states this scope. Where they share a component with our setting, the learning-rate schedule, we tested it directly above.

Revision: the rules and this scope go into Section 6.

Protocol of the new experiments

New runs follow the paper's recipe unless stated (10 tasks, fixed head, 2,000 replay examples, β=.5\beta=.5). For squared error, M keeps the full denominator, so it deletes the unseen-at-storage terms without amplifying the rest. For KL, M renormalizes both distributions over the retained classes. S∪N keeps the outputs seen at storage and N and deletes only F. S∪F deletes only N. Differences are paired within class order and seed and averaged over orders. W is within-task accuracy and G=W−FAAG=W-\text{FAA} measures cross-task errors. New treatment and control runs used the same GPU model and software. Only the ViT norm-matched runs come from the paper and ran on a different GPU (see Q3).


We upload a revised PDF with changes marked in magenta. It adds the slow-learner, transfer-benchmark, ViT-B/16, Mammoth, three-arm, coefficient, seen-class-only, and selective-retention results, states the method scope in Section 6, and relabels the supplementary comparisons. We would be glad to discuss any of these points further.

AuthorsOpenReview ↗

Follow-up on Q2

We extended the ViT-B/16 reference test of Q2 from CUB-200 to ImageNet-R. The setting is unchanged: ViT-B/16 supervised on ImageNet-21K, SLCA's slow-learner rates (backbone 1e-4, classifier 1e-2), 50 epochs, 1,000 replay examples, 10 tasks of 20 classes, and the same 3 class orders. ImageNet-R uses the fixed 80/20 split of our transfer benchmarks.

On ImageNet-R as on CUB-200, masking lowers final accuracy in every class order and raises acquisition, within-task accuracy W, and cross-task errors G in every order. The loss is larger on ImageNet-R.

DER++ in SLCA's setting, ViT-B/16FAA, U → MΔFAAΔAcquisitionΔWΔG
CUB-200, 3 orders (in our response)73.25 → 68.31−4.94 [−7.71, −2.16] (0/3)+3.43 (3/3)+1.22 (3/3)+6.16 (3/3)
ImageNet-R, 3 orders (new)67.40 → 58.94−8.45 [−10.38, −6.53] (0/3)+1.27 (3/3)+4.03 (3/3)+12.48 (3/3)

The unmasked run reaches 67.40%, close to the 67.75% that SLCA reports for DER++ on ImageNet-R with the same backbone. Starting from a baseline at the reference level, adding only the mask costs 8.45 pp.

The early check also held in all three orders. The task-prediction difference after task 2 now predicts the sign of the final effect in 78 of 84 new order-level comparisons.

Revision: The ImageNet-R row is added to Table 4 and to the appendix of the revised PDF.

Reviewer_uZkfOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper invested stored-logit distillation in pre-trained class-incremental learning (CIL). In the DER family, replayed examples carry stored logits over a fixed C-way head, including outputs for classes not yet observed when the example entered memory. The common practice of masking these unseen-at-storage coordinates appears to be a harmless simplification, but it silently changes two things at once: it deletes constraint terms and amplifies the retained ones (by 1/q under full-denominator averaging). The paper decomposes the distillation objective into support, averaging denominator, and nominal weight, and evaluates seven conditions within a DER++-style recipe on CIFAR-100 and ImageNet-100, using both ResNet-18 and ViT-Tiny pipelines.

Critical strengths
  • Separating support from averaging denominator from nominal weight is a useful conceptual contribution.
  • Exact class orders, optimization seeds, training protocols, hardware, and numerical acceptance criteria are all reported. The full numerical record is in the appendix.
  • Matching the masked distillation gradient to the full-gradient norm at the same state and replay batch (Eq. 5) is a direct falsification test of the scalar-strength explanation, and the result is unambiguous: 31–41 pp FAA loss with all seven order–seed pairs negative in both pipelines (Table 22).
Critical weaknesses
  • The ImageNet-100 ViT reversal cannot be attributed to anything: AugReg-21K pre-training, ImageNet-1K fine-tuning, the ViT inductive bias, or the recipe are inseparable. Since explaining this reversal is the paper's central open question, at least one disentangling control (e.g., a different ViT checkpoint source) would seem essential rather than optional.
  • KL-based distillation, temperature scaling, target-updating methods (X-DER is only approximated with an adapted objective), and any head-expansion protocol are all outside the evidence. The abstract-level claim about stored-logit constraints is not supported at that level of generality.
  • The norm-matching intervention (Eq. 5) matches the norm of the distillation component only. It does not account for interference between the distillation and CE gradients (which determines the effective update even at matched component norms), it is detached from any optimizable objective (the norm ratio is not differentiated), and under SGD with momentum, per-batch direction injection does not imply trajectory-level direction control.
Questions for authors
  • Can you predict the sign of the support-restriction effect from measurable early-training quantities (e.g., the ratio of N-group to S-group gradient norms, cross-task logit drift)?
  • Do the core findings (scalar-strength rejection, cross-task concentration of the loss) hold under KL distillation with temperature, which is scale-invariant and used by X-DER?
  • Does adding back only the N-group coordinates (a partial mask) recover most of the lost FAA?
3 comments

We thank you for the precise questions. We ran new controls for each of them, and we wrote down the prediction for each new experiment before its runs started.

Your pointWhat we addedResult
W1 ViT reversal cannot be attributedTwo new checkpoints (IN-21K only, IN-1K only) on two datasets, and SLCA learning ratesThe reversal persists with every checkpoint. It follows the dataset and the learning rate
W2, Q2 KL, target updating, head expansionKL with temperature on two datasets, seen-class-only CE (expanding head)Cross-task errors rise in every setting. Under KL on CIFAR-100, norm matching and the N-only ablation reproduce the paper's results
W3 Norm matching: interference, optimizability, momentumUnamplified masked arm on the same 7 pairs, static weights 4x to 64x, add-back of N, norm matching under KLNone of the tested strengths reproduces unmasked training. Restoring the right coordinates does
Q1 Early predictor of the signThree candidates named in advance, 81 new order-level comparisonsTask-prediction difference after 2 tasks: 75 of 81 correct, 5 of the 6 misses under KL on ImageNet-100. Gradient-norm ratio: does not separate the signs
Q3 Add back only NS∪N and S∪F on CIFAR-100 and the 7 ImageNet-100 pairs, S∪N under KLN recovers 85% to 110% of the loss, also under KL. F recovers 1% to 10% under squared error

Notation. U is unmasked and M is masked without amplification. For a replayed example, N are classes learned after it was stored and F are classes still unseen. Entries are FAA differences in pp: mean over class orders [95% CI] (orders positive / total). The protocol is at the end of part 3.

W1. "The ImageNet-100 ViT reversal cannot be attributed to anything"

The reversal does not require either pre-training stage. Under the original learning rate, masking helps on ImageNet-100 and hurts on CIFAR-100 with every checkpoint.

ViT-Tiny/16ImageNet-100, ΔFAA (M vs U)CIFAR-100, ΔFAA (M vs U)
IN-21K then IN-1K (paper)+2.9 (3/3)−8.86 (0/5)
IN-21K only (new)+3.76 [+2.49, +5.03] (3/3)−6.80 [−11.35, −2.25] (0/3)
IN-1K only, DeiT (new)+2.65 [+1.78, +3.53] (3/3)−3.91 [−5.89, −1.93] (0/3)
IN-21K then IN-1K, SLCA learning rates (new)−0.40 [−1.18, +0.38] (0/3)−3.73 [−4.61, −2.86] (0/5)

What never changes: in every row, masking raises within-task accuracy and raises cross-task errors. The net sign depends on which change is larger. (New rows: ΔW from +2.1 to +10.9 pp, ΔG from +3.0 to +11.6 pp.)

Our attribution rule, written before the runs, stated that if both reduced checkpoints keep the reversal, neither ImageNet-21K pre-training nor ImageNet-1K fine-tuning is necessary for it. This is what we observe. The learning rate does change the net sign on ImageNet-100, while ResNet-18 stays negative under the same learning rates (−6.47). New ImageNet-100 rows use the three support-set orders, CIFAR-100 rows use three orders for the new checkpoints and five for the learning-rate control.

Revision: the limitation "architecture, checkpoint, and recipe are inseparable" (Section 6) is replaced by these controls, with a new appendix.

W2 and Q2. "KL-based distillation, temperature scaling, target-updating methods, and any head-expansion protocol are all outside the evidence"

Under KL distillation and an expanding-head protocol, masking again raises within-task accuracy and cross-task errors in every setting. Accuracy falls in all settings except ImageNet-100 ViT under KL, where the gain seen under squared error disappears. Under KL on CIFAR-100, the other two main findings also hold in every order.

SettingPipelineΔFAAΔWΔTpredT_{\text{pred}}ΔG
KL with T=2T=2, CIFAR-100 (5 orders)ResNet-18−12.16 [−13.08, −11.24] (0/5)+2.34−12.73+14.50
KL with T=2T=2, CIFAR-100ViT-Tiny−6.50 [−6.83, −6.17] (0/5)+2.58−7.23+9.08
KL with T=2T=2, ImageNet-100 (3 orders)ResNet-18−10.27 [−16.19, −4.36] (0/3)+1.35−10.47+11.62
KL with T=2T=2, ImageNet-100ViT-Tiny−0.29 [−1.04, +0.45] (0/3)+3.43−0.35+3.72
Seen-class-only CE (expanding head), CIFAR-100ResNet-18−17.20 [−19.47, −14.93] (0/5)+2.55−17.84+19.75
Seen-class-only CE (expanding head), CIFAR-100ViT-Tiny−11.07 [−12.01, −10.12] (0/5)+2.26−11.27+13.33

On ImageNet-100 ViT, squared error gives a gain of +3.41 on the seven pairs. Under KL, within-task accuracy still rises (+3.43), but cross-task errors rise as much (+3.72), so the net effect is about zero. This is the balance described in the paper, with the KL loss shifting it toward cross-task errors.

KL with T=2T=2, CIFAR-100, 5 ordersResNet-18ViT-Tiny
Norm-matched masked KL vs masked KL−5.46 [−5.98, −4.93] (0/5)−8.41 [−9.69, −7.13] (0/5)
Restore only N (S∪N) vs U+0.17 [−0.62, +0.95] (3/5)+0.68 [+0.17, +1.19] (5/5)
Share of the loss recovered by N101%110%

Under KL too, added strength does not replace the deleted constraints, and restoring only N recovers 101% and 110% of the loss on average, and at least 95% in every order. Both outcomes were predicted before the runs. As under squared error, the norm-matching loss lies in task prediction (−5.46, −8.24), not within-task accuracy (+0.33, −0.87).

Target updating: the add-back result (Q3) shows that the useful deleted constraints are the outputs of classes learned after storage. X-DER updates the stored targets of all outputs unseen at storage, which include these. Inside the official X-DER code, this update gives higher final accuracy than freezing (+4.78 pp) or masking (+4.20 pp) the same targets, in all 5 seeds. These runs do not separate the two groups of outputs, so they do not show why the update helps.

KL with temperature ignores a common shift of an example's logits. Masked KL renormalizes the softmax over classes seen at storage, as LwF and iCaRL do, and S∪N KL over S and N. Norm-matched KL rescales the masked KL gradient to the norm of the full KL gradient on the same batch. With seen-class-only CE, a future output receives no gradient before its task, so the masked arm is functionally an expanding-head DER++. We did not run the full X-DER with our interventions. Section 6 now states the tested scope: fixed stored targets, squared-error or KL distillation, fixed or expanding heads.

Revision: KL and seen-class-only results in Sections 5.3 to 5.5 and a new appendix, scope statement and X-DER in Section 6.

W3. Norm matching "does not account for interference", is "detached from any optimizable objective", and momentum

We agree that per-batch norm matching does not control the total update. Three trajectory-level tests that include interference and momentum give the same answer: none of the tested strengths reproduces unmasked training, while restoring the right coordinates does.

ImageNet-100, ΔFAAResNet-18ViT-Tiny
(a) M vs U, same 7 pairs−13.89 [−18.79, −8.98] (0/7)+3.41 [+2.44, +4.38] (7/7)
(a) Norm-matched vs M, same 7 pairs−27.21 [−30.00, −24.42] (0/7)−35.40 [−37.41,−33.38] (0/7)
(b) Static weight 4x / 16x /64x, M vs U, 3 pairs−10.66 / −17.60 / −46.46+4.14 /−27.03 /diverged
(c) Add back only N, vs U, same 7 pairs−0.40 [−5.01, +4.21]−1.39 [−1.80,−0.97]

(a) The order is U > M > norm-matched for ResNet-18 and M > U > norm-matched for ViT. (b) No static weight from 4x to 64x reproduces unmasked training, and at 16x both pipelines end far below U. (c) Adding back only N, at the default weight, brings the learner back close to U.

Static weights are a standard objective, trained with full SGD momentum and no detached ratio. At the default weight, M vs U on the same three pairs is −9.18 (ResNet-18) and +3.35 (ViT). At 64x the ViT loss became NaN in the first epoch of task 2 in all three pairs. The ViT norm-matched runs come from the paper and ran on a different GPU. Same-configuration U runs on the two GPUs differ by only +0.16 [−0.06, +0.38] pp.

Q1. "Can you predict the sign of the support-restriction effect from measurable early-training quantities?"

In most settings, yes. Before the new runs we named three candidates. The task-prediction accuracy difference after task 2 predicted the sign of the final effect in 75 of 81 new comparisons, and the seen-class accuracy difference (our main candidate) in 74. The existing gradient-norm probes do not separate settings with opposite signs.

Candidate (named before the new runs)Correct sign of final ΔFAA
N-to-S gradient-norm ratio, from the existing probes after task 3does not separate the signs (ViT gives 85 and 87 in two settings with opposite signs)
Seen-class accuracy difference after task 2 (main candidate)74 / 81
Task-prediction accuracy difference after task 2 (cross-task drift)75 / 81

The task-prediction difference was right in 64 of 65 squared-error comparisons and in all 10 KL comparisons on CIFAR-100, but in only 1 of 6 under KL on ImageNet-100.

There, the ResNet-18 differences after task 2 were small (−2.3 to +0.4 pp) although the final loss was 8 to 13 pp, and the ViT differences were positive (+2.0 to +2.9 pp) although the final effect was close to zero (−0.60 to 0.00). The squared-error miss is a ViT-B/16 order with a difference of only +0.25 pp after task 2. An order with a final effect of exactly zero counts as a miss. The early test is therefore a useful check, not a guarantee. The 81 comparisons are all class orders of the 23 new pipeline and setting combinations (KL, checkpoints, and slow learner on two datasets each, slow learner with ViT-B/16, seen-class-only CE, and four transfer benchmarks). A magnitude does not tell the sign, while the early differences usually do, supporting the paper's claim that a constraint's value lies in which updates it restricts.

Revision: new appendix on early prediction, with the counts and where the misses occur. Section 6 reports the count.

Q3. "Does adding back only the N-group coordinates recover most of the lost FAA?"

Yes. Adding back only N recovers 85% to 110% of the loss, under squared error and KL, and adding back only F, under squared error, recovers 1% to 10%. Where masking helps (ImageNet-100 ViT), N also decides the gain.

SettingM vs US∪N vs US∪F vs URecovered by NRecovered by F
CIFAR-100, ResNet-18−9.22+0.20−8.551.020.07
CIFAR-100, ViT-Tiny−9.36−1.37−8.410.850.10
ImageNet-100, ResNet-18−13.89−0.40−13.750.970.01
ImageNet-100, ViT-Tiny+3.41−1.39+4.02gain removedgain kept

The deleted constraints that matter are the outputs of classes learned after storage. The outputs of truly unseen classes can be dropped at almost no cost.

Recovery is (FAAarm−FAAM)/(FAAU−FAAM)(\text{FAA}_{\text{arm}}-\text{FAA}_M)/(\text{FAA}_U-\text{FAA}_M). Before the runs we predicted at least 0.8 for N and at most 0.2 for F where masking hurts, and for ImageNet-100 ViT that S∪F would keep the gain while S∪N would lose it. All four predictions held. Under KL on CIFAR-100, adding back only N recovers 101% (ResNet-18) and 110% (ViT), as predicted. On ImageNet-100, S∪N vs M is +13.49 [+11.05, +15.92] for ResNet-18 and −4.80 [−5.59, −4.01] for ViT, with all 7 pairs in the same direction. Adding back N also closes the task-prediction gap, from −10.75 to −0.03 (CIFAR ResNet), −9.85 to −1.45 (CIFAR ViT), and −14.72 to −0.44 (ImageNet ResNet). This turns the local probes (submission Figure 4, now Appendix M) into a training-level causal result.

Revision: new Section 5.4, "Restoring Later-Learned Constraints Recovers Full Distillation".

Protocol of the new experiments

New runs follow the paper's recipe unless stated (10 tasks, fixed head, 2,000 replay examples, β=.5\beta=.5). For squared error, M keeps the full denominator, so it deletes the unseen-at-storage terms without amplifying the rest. For KL, M renormalizes both distributions over the retained classes. S∪N keeps the outputs seen at storage and N and deletes only F. S∪F deletes only N. Differences are paired within class order and seed and averaged over orders. W is within-task accuracy, TpredT_{\text{pred}} is task-prediction accuracy, and G=W−FAAG=W-\text{FAA} measures cross-task errors. New treatment and control runs used the same GPU model and software. Only the ViT norm-matched runs come from the paper and ran on a different GPU (see W3).


We upload a revised PDF with changes marked in magenta. It adds the add-back subsection to Section 5, replaces the attribution limitation with the checkpoint and learning-rate controls, adds the KL, expanding-head, three-arm, static-weight, and early-prediction results, and states the tested scope in Section 6. We would be glad to answer any further questions.

Comments on the paper

6 comments
AuthorsOpenReview ↗

Response to Reviewer 1kYE

We thank the reviewer for reading the new results and for acknowledging them. We also appreciate the clear statement of the remaining concern. As we read it, the technical questions of the review are no longer in dispute (withdrawn, see our later reply), and one question remains. Is the contribution significant enough for ICLR? We believe it is, for the five reasons below.

1. Which methods in current use face the identified limitation?

Every method (withdrawn) that distills stored logits and deletes, or never stores, the outputs of classes learned after storage. The cost is 5 to 17 pp in three settings taken from current practice.

Where the effect appearsCost of deleting these outputs (FAA)
Official Mammoth DER++ code, mask added only−15.14 pp, 36.56 → 21.42 (0/5 seeds positive)
DER++ as used as a replay baseline in SLCA (ICCV 2023), ViT-B/16 on ImageNet-21K, CUB-200 and ImageNet-R−4.94 and −8.45 pp (0/3 orders each)
Expanding-head protocol, where the head grows with each task−17.20 and −11.07 pp (0/5 orders each)

In the ImageNet-R case, our unmasked baseline reaches 67.40%, close to the 67.75% that SLCA reports for DER++, so the loss starts from a baseline at the published level.

Stored-logit replay remains a building block of current work. X-DER (TPAMI 2022) extends DER++. LiDER (NeurIPS 2022) and STAR (ICLR 2025) are designed as add-ons and report gains on top of DER++ and X-DER. SLCA (ICCV 2023) uses DER++ as its replay baseline for pre-trained models. A hidden strength or weakness of this component affects how all of these results should be read.

2. How does the analysis inform design?

It gives three rules that were not available before. Outputs of still-unseen classes can be dropped at replay. Outputs of classes learned after storage decide the effect, and a growing head never stores them. Two tasks of training usually tell which choice wins (78 of 84 comparisons). The analysis also explains an existing design: X-DER updates exactly these coordinates, and our results show why that matters. (withdrawn)

3. Why is this more than a routine ablation?

Because the result contradicts a common assumption, namely that outputs of classes unseen at storage carry no useful signal and can be deleted or compensated by a stronger loss. Amplifying the remaining distillation to the full gradient norm makes accuracy worse (−27.21 and −35.40 pp against the masked arm). Restoring only the outputs of later-learned classes recovers 85% to 110% of the loss. What matters is which coordinates are constrained, not how strongly. Each of these outcomes was predicted in writing before the runs.

4. On testing the mechanism inside other recent methods

The mechanism is defined only where stored logits are distilled, and within that family we tested it broadly. We agree that our interventions stay within stored-logit replay, and Section 6 of the revision states this boundary. Methods such as SLCA's classifier alignment or prompt-based methods have no stored-logit term to mask, so the intervention does not exist there. Within the family where it exists, we tested squared-error and KL distillation, fixed and expanding heads, uniform and slow-learner rates, three backbones, six datasets, and the official Mammoth code.

5. On the standard for significance

We agree that narrowing claims does not by itself create significance. Our position is that the narrowed claims are significant on their own terms. The ICLR 2027 reviewer guide asks: "Does it contribute new knowledge and sufficient value to the community? Note, this does not necessarily require state-of-the-art results. Submissions bring value to the ICLR community when they convincingly demonstrate new, relevant, impactful knowledge (incl., empirical, theoretical, for practitioners, etc)." Our paper is an analysis paper of this kind. It does not propose a method or compete on a leaderboard. It shows, with predictions written before the runs and consistent results across the settings above, what a widely used objective preserves and what is lost when part of it is deleted.

We respect that the reviewer may weigh significance differently, and we are grateful for the review, which led to most of the controls now in the paper. We hope this helps the AC and the other reviewers assess the remaining question.

We thank you for the detailed comment. It contains concrete requests that we can test, and we are running them now. We first correct three points in our previous reply.

1. We withdraw the sentence "the technical questions of the review are no longer in dispute." You acknowledged the new experiments and did not say that the concerns were resolved. We apologize for the wording.

2. "Every method" was too broad, and we withdraw it. Our evidence supports a narrower statement. In the settings we tested, deleting the stored outputs of classes learned after storage changes final accuracy, and the direction depends on the setting. Masking lowers final accuracy in most of them. The paper also reports a setting where masking helps, ViT-Tiny on ImageNet-100 under the reference recipe (+3.41 pp, 7 of 7 order and seed pairs, Table 3). This gain persists with two other pre-trained checkpoints (Table 33) and disappears under KL distillation and the slow learner (Table 4). In both directions, the outputs of classes learned after storage decide the effect (Section 5.4). We will not claim a cost for methods we have not tested.

3. STAR was published at ICLR 2025. The venue in our reply was our error. We confused it with another paper of the same name when searching for conferences. Thank you for the correction.

Experiments in progress. You ask for two tests. Both are running in the official Mammoth code on Seq-CIFAR-100 with 5 paired seeds, and the only change in each masked arm is the mask itself.

RequestExperiment
Compare updating, retaining, and deleting targets within the complete X-DERX-DER-RPC with three arms: original (update), stored targets kept but not updated (retain), and masked (delete)
Test whether the added mechanisms of STAR and LiDER preserve, weaken, or change the effectMask added inside DER++ + STAR, X-DER-RPC + STAR, and DER++ + LiDER, each against its unmasked version

For STAR we use the hyperparameters of its paper for Seq-CIFAR-100 with buffer 500. For LiDER and X-DER-RPC we use the official Mammoth configurations.

The predictions were written before the runs started, and we will report the results whichever way they turn out. If an added mechanism weakens or removes the effect, we will say so and narrow the paper accordingly.

We also thank you for pointing to Cho et al. (ICLR 2026). We will discuss it in the related work and state how its abundant-memory setting relates to ours.

We will post the results in this thread as soon as the runs finish.

AuthorsOpenReview ↗

Results of the two requested experiments

The runs announced in our previous reply have finished. All use the official Mammoth code on Seq-CIFAR-100 with a ResNet-18 trained from scratch, a buffer of 500, and 5 paired seeds.

Summary. The effect appears inside every method we tested, including LiDER and two STAR combinations. Inside the complete X-DER, the original update is better than both freezing and masking the targets, and the three policies differ in how old and new classes compete.

1. Do the added mechanisms of STAR and LiDER preserve, weaken, or change the effect?

They preserve it. Masking lowers final accuracy and raises cross-task errors G in all 5 seeds of every pair.

Method, mask added onlyFAA, unmasked → maskedΔFAA [95% CI]ΔG
DER++ (reported earlier)36.56 → 21.42−15.14 [−15.93, −14.35] (0/5)+20.44
DER++ + STAR (ICLR 2025)39.70 → 28.59−11.12 [−13.08, −9.15] (0/5)+14.23
DER++ + LiDER (NeurIPS 2022)36.58 → 19.64−16.95 [−17.93, −15.97] (0/5)+20.95
X-DER-RPC48.45 → 44.25−4.20 [−4.96, −3.45] (0/5)+2.47
X-DER-RPC + STAR (ICLR 2025)47.72 → 29.43−18.29 [−19.26, −17.32] (0/5)+17.23

The size differs between methods, and we do not claim an explanation for these differences. For LiDER we used the official Mammoth configuration. For STAR we used the hyperparameters of its paper for Seq-CIFAR-100 with buffer 500. Our split has 10 tasks, as in our earlier Mammoth runs, and the STAR paper uses 5. In our split, the unmasked add-ons do not always improve on their base methods. Our comparison concerns only the effect of the mask within each pair.

2. Updating, retaining, and deleting targets within the complete X-DER

X-DER-RPC updates the stored targets on all outputs that were unseen when the example was stored. We compare three policies on exactly these outputs: the original update, frozen targets (stored values kept, update switched off), and masked targets (no distillation on them). Nothing else is changed.

X-DER-RPC policyFAAWithin-task accuracy WGAccuracy on a new task right after learning itForgetting
Update (original)48.4585.5337.0870.4724.48
Frozen43.6781.3637.7062.1222.65
Masked44.2583.7939.5481.2741.14

X-DER's update is the best of the three policies. Freezing the targets costs 4.78 pp [−5.71, −3.85] and masking them costs 4.20 pp [−4.96, −3.45], each in all 5 seeds.

The policies do not differ in how well a new task is learned in isolation. They differ in how old and new classes compete. With the task identity given, accuracy on a new task right after learning it is 90.3, 90.3, and 90.2. Without it, the three policies separate. Frozen targets favor old classes, with the lowest accuracy on new tasks (62.12) and the lowest forgetting. Masked targets favor new classes, with the highest accuracy on new tasks (81.27) and the highest forgetting. The update lies between the two and reaches the highest final accuracy and the highest W.

One of our written predictions was not supported. We expected the frozen policy to end above the masked one in final accuracy. We observe no such advantage (masked minus frozen: +0.58 [−0.17, +1.33]). G is lower with frozen targets, as predicted (1.85 pp, 5/5 seeds), but W is lower as well (2.43 pp). These runs compare complete policies on all outputs unseen at storage. They do not separate classes learned later from classes still unseen, so we do not claim a mechanism for X-DER from them. You were right that our deletion results in DER++ could not settle this point. We withdraw the one-sentence rationale for X-DER's update from our earlier reply, and the revised paper reports this table instead (Appendix P.5).

What these results establish and what they do not

  • They establish that the effect is not specific to plain DER++. It appears in X-DER-RPC and in the combinations with LiDER and STAR, with the official code and only the mask changed.
  • They do not establish a cost for methods that we have not tested, and they say nothing about methods without a stored-logit term.
  • They do not replace a comparison across many 2025 and 2026 replay methods. Our paper analyzes one widely used objective, and we leave the weight of that scope to you and the AC.

As promised, the related work now discusses Cho et al. (ICLR 2026). Its abundant-memory method replays labels and has no stored-logit term, so our intervention does not apply to it.

Thank you for requesting these experiments. They made the paper more accurate.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.