acceptodds
Under review as a conference paper at ICLR 2027

AdaLMM: Adaptive Layer-wise Model Merging for Continual Learning

Abstract

Replay-based continual learning mitigates catastrophic forgetting by retaining historical examples, but its gains can diminish as the replay memory grows. Parameter-space model merging offers a complementary solution, yet fixed global interpolation coefficients provide limited flexibility in balancing stability and plasticity across the network. To address this limitation, we propose Adaptive Layer-wise Model Merging (AdaLMM), a task-boundary merging framework for replay-based continual learning. AdaLMM first aligns the historical and updated models through activation-based Optimal Transport matching and architecture-compatible symmetry transformations, and then learns layer-wise or block-wise interpolation coefficients directly from replay data. This allows different parts of the network to retain different proportions of the newly learned update. Experiments on Seq-CIFAR-100 and Seq-Tiny-ImageNet with ResNet-18 and Vision Transformer backbones show that AdaLMM improves the balance between retaining previous knowledge and learning new tasks, leading to higher overall accuracy than standard replay across different memory budgets.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Model Merging, Experience Replay, Stability-Plasticity Trade-off, Adaptive Model Interpolation, Catastrophic Forgetting

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_X3CqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AdaLMM is a task-boundary model-merging procedure for replay-based class-incremental learning. It aligns a historical checkpoint to the updated model using activation-based matching and architecture-compatible transformations, then learns an interpolation coefficient for each layer or block by minimizing replay loss. BatchNorm statistics are recalibrated before the merged model is used for inference and later tasks. Experiments combine the procedure with ER, DER++, and ER-ACE on sequential CIFAR-100 and Tiny-ImageNet, using ResNet-18 and a pretrained DeiT backbone.

Critical strengths
  1. The strongest results show a meaningful benefit, especially with the pretrained backbone. In Table 2, AdaLMM improves over CLeWI from 85.29% to 86.40% on CIFAR-100 and from 81.41% to 83.94% on Tiny-ImageNet at the 60% budget, with small reported run-to-run deviations. Table 1 also shows useful improvements over replay on Tiny-ImageNet. Five seeds, architectural diversity, and explicit checkpoint accounting are valuable features of the evaluation.
Critical weaknesses
  1. The evidence for the specific layer-wise contribution is weaker than the evidence for merging overall. In Table 3, replacing a learned global coefficient with layer-wise coefficients improves mean accuracy by 0.21 and 0.27 points on CIFAR-100 and by 0.45 and 0.43 points on Tiny-ImageNet. These differences consistently favor AdaLMM, but five-seed summary statistics alone do not establish how reliable or practically meaningful each advantage is. The same component ablation is not supplied for the pretrained backbone, where the main results are strongest.
  2. The protocol leaves uncertainty about the proposed explanation and comparison advantage. The motivation is a plasticity plateau, yet the main metrics are final average accuracy and forgetting. Those metrics do not distinguish improved learning of the newest task from preservation of earlier tasks; recent-task accuracy before and after merging would address that distinction.
Questions for authors
  1. Can you report paired differences across seeds for layer-wise versus learned global coefficients, and repeat the component ablation for DeiT? Which component explains the largest gains in Table 2?
  2. Was CLeWI's scalar coefficient retuned for the abundant-memory settings with a comparable selection budget? How does that affect the comparison?
  3. What is each method's exact persistent-memory budget for DeiT, including historical checkpoints? Does the comparison remain favorable if replay images are stored compactly?
2 comments
AuthorsOpenReview ↗

Response to Reviewer X3Cq

Thank you for the questions about Table 2.

1. How much do block-wise coefficients add?

On DeiT-S/16 at 60% replay capacity, we fitted either one global coefficient or 15 block-wise coefficients at each merge, using the same fitting budget. We report all six configurations (mean ± sample SD, seeds 42–46).

DatasetCoefficientsAlignmentFinal accuracy (%) ↑Forgetting (pp) ↓
CIFAR-100GlobalOn85.86 ± 0.316.57 ± 0.53
CIFAR-100Block-wiseOn86.28 ± 0.186.51 ± 0.48
Tiny-ImageNetGlobalOff82.29 ± 0.066.16 ± 0.38
Tiny-ImageNetBlock-wiseOff83.24 ± 0.276.13 ± 0.29
Tiny-ImageNetGlobalOn82.70 ± 0.096.43 ± 0.23
Tiny-ImageNetBlock-wiseOn83.76 ± 0.216.41 ± 0.27

The table gives the block-wise minus global accuracy difference for each seed with alignment on (pp).

Dataset / seed4243444546Mean
CIFAR-100+0.45−0.02+0.44+0.52+0.71+0.42
Tiny-ImageNet+1.16+0.90+0.83+0.99+1.43+1.06

On Tiny-ImageNet, block-wise fitting showed the larger gain (+0.95 pp without alignment and +1.06 with it). Alignment added 0.41 pp with a global coefficient and 0.52 with block-wise coefficients. For the original ResNet runs (Table 3), only the mean gains at 60%/80% are verified (+0.21/+0.27 pp on CIFAR-100 and +0.45/+0.43 on Tiny-ImageNet).

2. How does retuning CLeWI affect the comparison?

Table 2 used the published ER setting (α = 0.3). At 60% capacity, we retuned CLeWI using 10 development runs across the two datasets. We selected α using validation data held out from training, then fixed it for all five test seeds. The external search was for CLeWI only. AdaLMM fits its coefficients on replay at each boundary. We report every fixed α tested on the five seeds (test mean ± SD).

DatasetFixed αValidation accuracy (%)Test accuracy (%)Test forgetting (pp)
CIFAR-1000.3 (selected)85.2685.264 ± 0.2766.718 ± 0.232
Tiny-ImageNet0.381.5081.462 ± 0.2648.225 ± 0.235
Tiny-ImageNet0.5 (selected)81.7181.472 ± 0.2228.194 ± 0.282

Compared with the selected CLeWI settings, AdaLMM's mean accuracy was 1.136 pp higher on CIFAR-100 and 2.468 pp higher on Tiny-ImageNet. On Tiny-ImageNet, changing α from 0.3 to 0.5 reduced mean forgetting by 0.031 pp and changed test accuracy by +0.01 pp. Retuning alone does not explain the smaller accuracy gaps here.

3. DeiT memory budgets and accuracy with compact replay

Table 2 already uses compact replay. All three methods store native-resolution uint8 images and use the full replay capacity shown below. CLeWI and AdaLMM each store one additional historical checkpoint without reducing the buffer. ER stores no historical checkpoint.

Dataset / replay capacityER (bytes)CLeWI (bytes)AdaLMM (bytes)
CIFAR-100 / 30,00092,640,008179,456,668179,456,724
CIFAR-100 / 40,000123,520,008210,336,668210,336,724
Tiny-ImageNet / 60,000738,240,008825,210,668825,210,724
Tiny-ImageNet / 80,000984,320,0081,071,290,6681,071,290,724

These are exact persistent bytes beyond the current model, including replay, historical weights and merge coefficients. Including the current model adds 86,816,656 bytes to each CIFAR-100 entry and 86,970,656 bytes to each Tiny-ImageNet entry.

Compared with CLeWI, AdaLMM uses only 56 additional bytes. With compact replay and CLeWI's retuned coefficient at 60% capacity, mean accuracy was nearly identical on CIFAR-100 (86.278% vs. 86.264%) and slightly higher for AdaLMM on Tiny-ImageNet (83.760% vs. 83.442%).

Compared with ER, AdaLMM uses more memory at the same replay capacity. Table 2 nevertheless gives one comparison in which AdaLMM achieved higher accuracy with less storage. On Tiny-ImageNet, AdaLMM with 60,000 examples reached 83.94%, versus 80.28% for ER with 80,000 examples, a gain of 3.66 points using 159.11 MB less. Accuracy at exactly matched total-byte budgets has not been measured.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer X3Cq,

Thank you for taking the time to review our paper and provide valuable feedback. As the discussion period is nearing its conclusion, we would like to kindly follow up on our earlier rebuttal and the additional experimental results.

We would appreciate knowing whether our responses have adequately addressed your concerns. If you have any remaining questions or suggestions, we would be happy to clarify them.

Best regards,
The Authors of Submission 47323

Reviewer_U9q6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AdaLMM, a model-merging method for replay-based continual learning. It aligns historical and updated models at task boundaries and learns layer-wise or block-wise interpolation weights from replay data. The main experiments on sequential CIFAR-100 and Tiny-ImageNet use ResNet-18 and a pretrained Transformer to compare accuracy and forgetting against replay and merging baselines. The main comparisons account for the stored model's memory cost, and the evaluation includes component ablations and training-time measurements.

Critical strengths

Several experiments show useful gains over the underlying replay methods, especially on Tiny-ImageNet and with the Transformer backbone. This provides empirical support for task-boundary merging as a useful addition to replay when historical data are relatively abundant.

I appreciate the accounting for the historical checkpoint's memory cost in the main comparisons. The method also leaves a single merged model with the original architecture for inference, so the extra consolidation work is confined to training.

Critical weaknesses

I am not really convinced that layer-wise adaptation adds much over simpler merging. The learned global control already recovers most of the accuracy gain over ER, and Table 3 shows that layer-wise weights add less than half a percentage point. The extra reduction in forgetting is useful, so there is evidence for the component. The open question is how much it contributes to the larger ViT gains. The same learned-global comparison on ViT, with matched replay data, alignment and fitting budget, would help answer that.

AdaLMM takes ER-ACE on Seq-CIFAR-10 from about 68% to 58% final accuracy with 500 exemplars in Table 5, despite a lower forgetting score. Both methods store the same number of examples here. Lower forgetting need not mean better absolute old-task accuracy, so the paper should show each task's learning curve for both methods and mark accuracy before and after the full consolidation step, including BN recalibration. This would locate the loss without assuming that the merge itself causes it.

Questions for authors

Can you extend the Global-α\alpha comparison to ViT and report the gain from layer-wise weights using matched runs and the same fitting budget?

Could you show old-task and new-task accuracy immediately before and after each merge for the Table 5 ER-ACE case, so we can see where the drop occurs?

3 comments
AuthorsOpenReview ↗

Response to Reviewer U9q6

Thank you for suggesting the learned-global control and the task-by-task analysis. We ran both comparisons over five seeds (42–46).

1. What do block-wise coefficients add on DeiT?

At 60% replay capacity, we compared one learned global coefficient with AdaLMM's 15 learned block-wise coefficients on DeiT-S/16. Both used the same replay protocol, alignment procedure and coefficient-fitting budget. The final replay buffers contained the same examples for each seed. The table gives final average accuracy (mean ± sample SD) and the block-wise minus global difference for each seed, in percentage points.

DatasetGlobal (%)Block-wise (%)Differences for seeds 42, 43, 44, 45, 46 (pp)Mean (pp)
CIFAR-10085.86 ± 0.3186.28 ± 0.18+0.45, −0.02, +0.44, +0.52, +0.71+0.42
Tiny-ImageNet82.70 ± 0.0983.76 ± 0.21+1.16, +0.90, +0.83, +0.99, +1.43+1.06

The gain was modest on CIFAR-100. On Tiny-ImageNet, where Table 2 showed the largest improvement, block-wise fitting gained 0.83–1.43 points across the five seeds. Mean post-merge accuracy on the newest task increased from 87.68% with global fitting to 89.25% with block-wise fitting (+1.57 percentage points).

2. Why did accuracy fall when AdaLMM was added to ER-ACE in Table 5?

In our five-seed reproduction on Seq-CIFAR-10 with 500 replay examples, accuracy fell because the original merge repeatedly shifted predictions for new-class images toward old classes. Immediately before the first merge, both methods scored 84.51% on task 2. After consolidation, AdaLMM's accuracy on task 2 fell to 0.39%, while accuracy on task 1 rose from 82.93% to 96.24%. At the next three boundaries, the newest task again fell below 2.5% after consolidation.

After taskER-ACE old (%)ER-ACE new (%)ER-ACE + original AdaLMM old, pre → post (%)ER-ACE + original AdaLMM new, pre → post (%)
282.9384.5182.93 → 96.2484.51 → 0.39
379.3279.4166.21 → 75.4273.97 → 2.43
473.5368.8358.26 → 70.6256.34 → 0.75
567.7157.3257.66 → 66.7558.77 → 0.08

Old-task accuracy is averaged over earlier tasks. Pre and post refer to the model immediately before fusion and after the full consolidation step, including BN recalibration. ER-ACE is evaluated at the same boundaries. All predictions use all ten classes.

The first merge shows why this happened. Each row of the classifier scores one class. The current model had learned to recognize the two task-2 classes. The historical model had never seen a positive example of either, yet AdaLMM used the same mixing coefficient for every row. In the two checkpoints we inspected, about 46% of each new class's weights came from that historical model. The merged classifier then labeled all 16 new-class images in each diagnostic sample as old classes. Keeping the merged features fixed and restoring just the pre-merge classifier weights brought 10 of those 16 predictions back in each case. Across the five runs, task-2 test accuracy was already 0.02% immediately after fusion, before BN recalibration.

ER-ACE's training rule may make this effect more pronounced. With ER and DER++, a new training image contributes directly to learning that its class should score above the old classes. ER-ACE masks old classes in the loss on incoming images, so it relies on replay for that comparison. We suspect this can give a new class less of a lead over old classes before fusion. In a short control run, switching off the mask improved accuracy before the merge.

The task trajectories also explain the negative forgetting score. Task 4 recovered from 0.75% after its merge to 36.70% at the end, contributing −35.95 points to forgetting when the peak excludes the final evaluation. Task 5 finished at 0.08%. It lowers final average accuracy but is excluded from forgetting.

We revised the merge to preserve the current model's classifier weights for newly introduced classes and corrected the channel alignment. The table gives the results after all five tasks on the same five seeds.

MethodFinal average accuracy (%)Final old-task mean (%)Final newest-task accuracy (%)
ER-ACE65.63 ± 3.4867.7157.32
ER-ACE + AdaLMM, original53.42 ± 5.8366.750.08
ER-ACE + AdaLMM, revised65.19 ± 2.2874.0929.58

The revision raised final average accuracy by 11.77 points, to within 0.44 points of ER-ACE. Its higher mean accuracy on the four old tasks (74.09% vs. 67.71%) offset much of the remaining gap on the newest task (29.58% vs. 57.32%).

Thanks for the extra experiments, I really appreciate the effort. I do think block wise is pretty close to global despite it is a bit higher (i.e. table 1 from your response). So I still think the contribution is limited, and I will keep my score.

AuthorsOpenReview ↗

Response to Reviewer U9q6

Thank you for reviewing the additional experiments and for sharing your assessment. We agree that the learned global coefficient is a strong baseline, and we will make the magnitude of the additional gains from block-wise fitting clear in the revised paper. The DeiT experiments show mean improvements of 0.42 percentage points on CIFAR-100 and 1.06 on Tiny-ImageNet across five seeds, while retaining the original model architecture for inference. Your feedback has helped us clarify both the contribution and the conditions under which it is effective.

Reviewer_BCZnOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes AdaLMM, a task-boundary model-merging method for replay-based class-incremental learning. After each task, it aligns the historical checkpoint to the updated one via activation-correlation optimal transport (including attention symmetry transforms), then learns layer-wise interpolation coefficients from the replay buffer, merges the two models, and re-estimates BN statistics. On Seq-CIFAR-100 and Seq-Tiny-ImageNet with ResNet-18 and DeiT-S/16, plugging AdaLMM into ER, DER++, and ER-ACE improves average accuracy and reduces forgetting under a memory-aligned protocol, with the largest gains on Tiny-ImageNet and small gains for ER-ACE on CIFAR-100.

Critical strengths
  1. The paper is very clear to understand. It clearly identifies the trainable parameters, how to merge, and how to learn a new task.
  2. The experiments include both a DNN (ResNet-18) and a transformer (DeiT-S/16).
Critical weaknesses
  1. The method does not address the concern about motivation. The motivation is that a memory buffer that is too large will reduce plasticity, the ability to learn new tasks. However, the method does not increase plasticity but stability, the ability to preserve old knowledge.
  2. Novelty. The components of AdaLMM are all from other papers. The only new thing is using cross-entropy loss to train the layerwise merging weight α\alpha
  3. In experiments, the paper only compares with one model-merging method. It does not compare with other SOTA model-merging methods (e.g., Weight Space Consolidation, MagMax, MINGLE, cited by the paper).
  4. It seems that under a low-buffer setting, AdaLMM+ER-ACE drops 7–10 points on Seq-CIFAR-10 and has negative forgetting. The result is only in the appendix, with no analysis or explanation.
  5. Lack of experimental details. The paper did not report the buffer strategy, the ratio of new task data in the memory buffer, or the training batch size and replay batch size for ResNet18.
Questions for authors
  1. Can you compare with more model-merging methods like MagMax and MINGLE, which you cited in the paper?
  2. Can you explain how your method addresses the problem that a larger buffer size may reduce plasticity?
  3. Why does AdaLMM+ER-ACE drop that much but still have negative forgetting on Seq-CIFAR-10 under a low-buffer setting?
  4. Can you report more experimental details to reproduce your results? Are these details the same for all the experiments?
6 comments
AuthorsOpenReview ↗

Response to Reviewer BCZn

Thank you for the questions.

1. MagMax and MINGLE

We adapted both methods to DeiT-S/16 Class-IL with learned classifiers. After the last task, we calibrated only the classifier for all three methods, using the same training-buffer examples, split, order and budget (one epoch of Adam, LR 0.0001, batch size 64, no weight decay). Results are final accuracy over all classes without task identity (mean ± sample SD, seeds 42–46).

Method after shared calibrationSeq-CIFAR-100 (%)Seq-Tiny-ImageNet (%)
MagMax (adapted)74.26 ± 0.3976.85 ± 0.25
MINGLE (adapted)69.36 ± 0.8177.38 ± 0.92
AdaLMM86.49 ± 0.1483.89 ± 0.23

2. Plasticity, stability and the contribution of AdaLMM

Final Class-IL accuracy depends on both learning new classes and retaining earlier ones. In Figure 1, ER's accuracy plateaus as the buffer grows on Seq-CIFAR-10, and gradients from incoming and replay data become less aligned. This is consistent with a stability–plasticity trade-off. Here, increasing capacity changes which examples are available for replay while keeping the draw size fixed. We will revise the motivation accordingly. AdaLMM addresses this trade-off at consolidation, after task training.

CLeWI uses fixed interpolation between successive checkpoints. AdaMerging learns layer-wise weights across task experts. AdaLMM uses labeled replay after each task to decide how much of the latest update to retain in each aligned layer or block. The resulting model starts the next task. Our control fits one global coefficient with the same replay loss and fitting budget.

On DeiT at 60% capacity, block-wise fitting gained 0.42 points on CIFAR-100 and 1.06 on Tiny-ImageNet over this control. On Tiny-ImageNet, it recovered 2.06 points of old-task accuracy per merge on average, versus 1.78 with global fitting. It also lost less of the newest task during the merge (3.56 vs. 5.07 points), raising mean post-merge newest-task accuracy from 87.68% to 89.25%.

3. Why did ER-ACE accuracy fall despite negative forgetting?

In a new five-seed reproduction of Table 5 on Seq-CIFAR-10 with 500 replay slots, task-2 accuracy fell from 84.51% before the first merge to 0.39% after consolidation, while task-1 accuracy rose from 82.93% to 96.24%. The original merge mixed every classifier row with the historical model, even for the two new classes it had never seen as positive examples. In two inspected checkpoints, all 16 new-class images in each diagnostic sample were classified as old. Restoring only the pre-merge classifier weights recovered 10 of 16 in each case.

ER-ACE masks old classes in the loss on incoming images and relies on replay to distinguish new classes from old ones. We suspect this can leave new classes with less of a lead over old ones before fusion. Turning off the mask improved pre-merge learning in a short control run.

Task 4 recovered from 0.75% after its merge to 36.70% at the end, contributing −35.95 points when forgetting compares the best earlier accuracy with final accuracy. Task 5 finished at 0.08% and is excluded from forgetting. On the same diagnostic seeds, correcting channel alignment and preserving the new-class classifier rows raised final accuracy from 53.42 ± 5.83% to 65.19 ± 2.28% (ER-ACE reached 65.63 ± 3.48%). The revised model reached 74.09% on old tasks and 29.58% on the newest task, versus 67.71% and 57.32% for ER-ACE.

4. Buffer composition and batch sizes

The ResNet ER and ER-ACE runs maintain a reservoir across tasks. After each update, only unaugmented incoming examples and their labels are submitted for reservoir sampling. Repeated visits across epochs count separately, so slots can hold the same source image. Replay samples occupied slots uniformly without replacement within each draw. The expected fraction from the newest task is its share of all stream visits so far, or 1/t after t equally exposed tasks.

Across our replay experiments, the incoming batch size is 32 and each replay draw contains up to 32 examples. ER concatenates the incoming and replay batches. ER-ACE starts replay at task 2 and sums separate losses for incoming and replay examples. DER++ makes two replay draws, one for logit matching and one for label cross-entropy. AdaLMM fits its merge coefficients in a separate step with a batch size of 256. We train ResNet-18 with SGD (learning rate 0.1) for 20 epochs per task on CIFAR-10 and 50 on CIFAR-100 and Tiny-ImageNet. We train DeiT-S/16 with AdamW (learning rate 0.0001) for five epochs per task.

AuthorsOpenReview ↗

How AdaLMM balances old- and new-task accuracy at consolidation

Figure 1 shows that increasing ER's replay buffer on Seq-CIFAR-10 yields diminishing gains in final accuracy, even as the gradients from incoming and replay examples become less aligned. Final accuracy averages over old and new tasks, so the curve alone cannot tell us which tasks account for the plateau. Our motivation is to give the model a way to manage this trade-off at the end of each task.

By that point, the current model has already learned from the new classes. AdaLMM aligns the previous checkpoint to the current model and fits a mixing coefficient for each layer or block on labeled replay examples from the stream so far. Giving a block more weight from the current model preserves more of its latest update. Giving it more weight from the previous checkpoint can recover accuracy on older classes. The replay loss evaluates both sets of classes together.

What happens at one merge? At the end of task 5 on Seq-CIFAR-100 (ResNet-18, 60% setting, seed 44), we held the two checkpoints fixed and varied the historical weight. Each column gives accuracy (%) on one task in a fixed development split drawn from the training data. All candidates received the same BatchNorm processing. A historical weight of 0 keeps the current weights.

Historical weightT1T2T3T4T5 (new)Mean
072.0267.8470.1065.4882.9671.68
0.1076.3470.3776.0071.4373.6373.55
0.3079.4272.5178.4873.6130.0266.81
Learned per tensor75.9369.7975.0569.8476.4773.41

Increasing the historical weight from 0.10 to 0.30 helped all four old tasks, but task 5 fell from 73.63% to 30.02%. The learned weights recovered accuracy on every old task relative to weight 0 and retained 76.47% on task 5. A fixed weight of 0.10 reached a similar overall mean here, with a different balance across tasks.

Where does the gain over ER come from across a full sequence? We measured each task after learning all ten in a separate Seq-CIFAR-100 reproduction (ResNet-18, seeds 0–4, 60% setting, 30,000 replay slots for ER and 26,347 for AdaLMM + ER).

Final accuracy (%)ERAdaLMM + ER
Task 165.6867.50
Task 259.2061.46
Task 367.0868.18
Task 460.2863.30
Task 562.4264.88
Task 665.0868.38
Task 763.8666.74
Task 860.8066.14
Task 970.3271.60
Task 10 (newest)80.9270.08
All ten tasks65.5666.83

Here the gain over ER came from the nine earlier tasks. Their mean accuracy rose from 63.86% to 66.46%. Accuracy on the newest task fell by 10.84 points. Across all ten tasks the net gain was 1.26 points (computed before rounding).

Does block-wise fitting preserve more of the latest task than a single learned coefficient? On DeiT-S/16 we compared both choices with full alignment at 60% replay capacity. Each variant ran its own full training sequence. We averaged pre- and post-merge accuracies (%) over boundaries from task 2 onward within each run, then over seeds 42–46.

DatasetFittingOld tasks, pre → postNewest task, pre → post
CIFAR-100Global88.62 → 90.1494.00 → 90.94
CIFAR-100Block-wise88.70 → 90.3394.04 → 91.41
Tiny-ImageNetGlobal85.97 → 87.7592.75 → 87.68
Tiny-ImageNetBlock-wise86.20 → 88.2692.81 → 89.25

On Tiny-ImageNet, block-wise fitting raised mean post-merge accuracy on the newest task from 87.68% to 89.25% relative to global fitting, while also recovering more old-task accuracy during consolidation (+2.06 vs. +1.78 points). CIFAR-100 showed the same pattern. Final mean accuracy also rose, from 85.86% to 86.28% on CIFAR-100 and from 82.70% to 83.76% on Tiny-ImageNet. We will describe the effect on the newest task more precisely in the paper as preserving more of what the model has just learned through consolidation.

Reviewer_BCZnOpenReview ↗

Response to the rebuttal

Dear Authors,

Thank you for the detailed responses and additional experiments. Some of my concerns are addressed. However, I still have the following concerns.

  1. I appreciate the additional comparisons with MagMax and MINGLE. However, it remains unclear whether these methods were applied at every task boundary under the same continual learning setting as AdaLMM. Could the authors clarify how these methods were adapted to the sequential learning scenario?

  2. The additional results show that AdaLMM improves old-task accuracy from 63.86% to 66.46%, but reduces new-task accuracy from 80.92% to 70.08%. This suggests that the improvement mainly comes from better stability at the cost of plasticity, rather than addressing the plasticity plateau highlighted in the paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to PCs, SACs, ACs, and Reviewers

Dear PCs, SACs, ACs, and Reviewers,

Thank you for the careful reviews and the discussion. We summarize the additional experiments and clarifications below.

1. Contribution of block-wise coefficients (X3Cq, U9q6, BCZn)

We extended the component ablation to DeiT-S/16 at 60% replay capacity, using five seeds, full alignment, and the same coefficient-fitting budget.

Final accuracy (%)Learned globalAdaLMM block-wiseMean gain (pp)
CIFAR-10085.86 ± 0.3186.28 ± 0.18+0.42
Tiny-ImageNet82.70 ± 0.0983.76 ± 0.21+1.06

Values are mean ± sample SD. We reported individual seed differences. On Tiny-ImageNet, the gain was positive in all five seeds, ranging from 0.83 to 1.43 points. We also repeated the CLeWI comparison with fixed coefficients selected on a training-derived validation split.

2. How consolidation balances old and new knowledge (X3Cq, U9q6, BCZn)

We added task-level measurements before and after consolidation. On Tiny-ImageNet, block-wise fitting raised mean post-merge newest-task accuracy from 87.68% with global fitting to 89.25%, while recovering more old-task accuracy during consolidation (+2.06 vs. +1.78 points). The corresponding newest-task accuracies on CIFAR-100 were 90.94% and 91.41%.

In the ResNet comparison with ER on Seq-CIFAR-100, mean accuracy on the nine earlier tasks increased from 63.86% to 66.46%, raising final average accuracy across all ten tasks from 65.56% to 66.83%. We clarified the motivation around the saturation of replay accuracy and the use of adaptive consolidation to improve average task accuracy by balancing historical knowledge retention and preservation of the latest update.

3. Additional comparisons with MagMax and MINGLE (BCZn)

We added DeiT adaptations of both methods to the same sequential Class-IL benchmarks. Both incorporated each task and evaluated the accumulated model at every task boundary. We documented their model inheritance rules and differences in replay use, training, and persistent storage.

We then applied the same final-classifier calibration to all three methods, using identical training-buffer examples and fitting budgets, with the representation frozen. Calibration was applied once after the final task.

Final calibrated accuracy (%)CIFAR-100Tiny-ImageNet
MagMax adapted74.26 ± 0.3976.85 ± 0.25
MINGLE adapted69.36 ± 0.8177.38 ± 0.92
AdaLMM86.49 ± 0.1483.89 ± 0.23

These five-seed results extend the empirical comparison to two additional merging methods under the stated DeiT adaptation and calibration protocol.

4. Diagnosis and improvement of the ER-ACE case (U9q6, BCZn)

In new five-seed reproductions of Table 5, we recorded old- and new-task accuracy around each consolidation step, including BN recalibration. The measurements and classifier interventions identified the treatment of newly learned classifier rows as a key factor in the sharp new-task loss.

Preserving the current model's new-class classifier weight rows and correcting channel alignment raised final accuracy from 53.42 ± 5.83% to 65.19 ± 2.28%, an improvement of 11.77 points, on Seq-CIFAR-10 with 500 replay examples. We reported this revised policy alongside the original implementation. We also explained the negative forgetting values through the absolute task trajectories and specified the forgetting definition.

5. Memory accounting and reproducibility (X3Cq, BCZn)

We provided explicit persistent-memory accounting, including historical checkpoints. The DeiT ER/CLeWI/AdaLMM comparisons use equal replay-example counts, with images already stored as native-resolution uint8. AdaLMM retains one historical checkpoint. Its accounted storage differs from CLeWI by only 56 bytes. We clarified the budget protocol used by each experiment.

We also supplied the reservoir-sampling policy, measured buffer composition, stream and replay batch sizes, coefficient-fitting batch size, and backbone-specific training configurations.

We will incorporate these experiments, diagnostics, and protocol clarifications into the revised manuscript. AdaLMM retains a single merged model with the original architecture for inference. Thank you for your time and consideration.

The Authors of Submission 47323

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.