- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies replay-free class-incremental learning and proposes FS-EWC. It augments EWC’s historical protection with a regularizer targeting the current task’s Fisher sensitivity within a parameter neighborhood, aiming to learn solutions that better tolerate subsequent updates. The method also averages selected training checkpoints to construct the retained anchor, where Fisher importance is estimated for future tasks. Vision and text classification experiments, together with controlled diagnostics, evaluate the benefits of solution shaping and historical protection.
Critical strengths
The controlled studies support the value of solution shaping and show that better retention can coexist with larger parameter drift (Figs. 3–4; Table 4).
The evaluation spans vision and language, including full-parameter and LoRA settings. Improvements over EWC-LoRA are substantial on ImageNet-R/A and also extend to text classification (Tables 1–3).
Tests with different Fisher estimators, representations, regularization strengths, and neighborhood radii provide useful robustness evidence (Tables 11–12; Fig. 5).
Critical weaknesses
The specific benefit over existing gradient-norm regularization remains insufficiently isolated. Comparisons with EWC+GAM and pointwise ((r=0)) shaping, using matched training budgets and checkpoint averaging, would clarify the contribution of the neighborhood design.
Despite the additional gradient evaluations, training time and peak memory are not reported. This leaves the practical trade-off between performance and computational cost unclear.
The main text could more clearly distinguish the theoretical per-example Fisher objective from the batch-gradient surrogate described in Appendix A.5. The parameter space used for LoRA shaping also needs clarification.
Questions for authors
Additional comparison. Under matched training budgets, historical Fisher processing, and checkpoint averaging, how does FS-EWC compare with EWC+GAM and pointwise ((r=0)) shaping?
Implementation clarification. Under LoRA, which parameter spaces are used for the neighborhood perturbation and shaping gradient: the factors (A,B) or the effective update (BA)? How do these relate to the effective-weight Fisher used for historical protection?
Approximation validation. Appendix A.5 acknowledges the batch-gradient approximation. Could a small-scale comparison with per-example Fisher shaping quantify its effect on retention and sensitivity reduction?
Computational cost. What are the training-time and peak-memory overheads relative to EWC-LoRA? Could the costs of sensitivity shaping and checkpoint averaging be reported separately?
3 comments
Response 2/3
(Weakness 4, Question 3) Approximation validation. Comparison with per-sample Fisher
We adopt the batch-gradient surrogate for computational efficiency, following the use of batched Fisher approximations in existing EWC implementations. Direct per-example Fisher shaping requires individual sample gradients and corresponding second-order computations at each training step, introducing substantial computational overhead.
To evaluate this approximation, we compare the batch surrogate with the scaled per-example objective , defined as
Here, is the per-example loss gradient and is the actual minibatch size. The additional factor in the per-example training objective adjusts its scale.
Following the two-task ImageNet-A data split and base training schedule of Table 4, we compare batch-gradient and per-example Fisher shaping on the first two ImageNet-A tasks, with 20 classes each, using ViT-B/16 with rank-10 LoRA. Both variants use HVPs computed by automatic differentiation, a one-step search with , and . Training time and peak allocated GPU memory are measured on the same GPU.
Both variants achieve 86.32% accuracy after Task 1. The table reports same-seed results, with Task-2 accuracies evaluated over all 40 seen classes.
| Training shaping target | Old-task Acc. (%) | New-task Acc. (%) | BWT (pp) | Time (h) | Peak GPU (GiB) |
|---|---|---|---|---|---|
| Batch surrogate, | 82.41 | 80.71 | −3.91 | 1.18 | 30.69 |
| Scaled per-example, | 82.08 | 80.31 | −4.23 | 6.89 | 12.75 |
The two approaches achieve comparable retention, with the batch surrogate yielding 0.33 percentage points higher old-task accuracy and 0.32 percentage points better BWT. These results indicate comparable retention and new-task accuracy under the evaluated setting.
The batch-gradient implementation reduces runtime from 6.89 to 1.18 hours, providing a 5.84$\times$ speedup. The per-example implementation computes second-order derivatives sequentially for individual samples and accumulates detached results, resulting in lower peak GPU memory but substantially longer runtime.
Together, these results support the batch-gradient surrogate as a practical choice for Fisher sensitivity shaping. Under the evaluated two-task setting, it achieves comparable retention and measured endpoint sensitivity while substantially reducing computation time. This efficiency consideration also motivates the finite-difference implementation adopted in FS-EWC.
Summary of Changes. We have added a controlled comparison between batch-gradient and scaled per-example Fisher shaping in Appendix B.9, reporting retention, runtime, and peak GPU memory.
Response 3/3
(Weakness 3, Question 4) Computational cost. Training Time and Peak Memory
We report a computational cost analysis on ImageNet-A using ViT-B/16 with rank-10 LoRA. All configurations use the same 10-task sequence, 50 epochs per task, Fisher processing, and checkpoint-averaging protocol. We compare EWC-LoRA () and FS-EWCLoRA () with . Both methods average selected LoRA checkpoints from epochs 30, 40, and 50.
Training time and memory overhead.
| Component | Time | GPU allocated memory | CPU snapshot cache | Archived snapshots |
|---|---|---|---|---|
| EWC-LoRA | 69.77 min (1.00×) | 13060.30 MiB | 7.03 MiB | 70.31 MiB |
| FS-EWCLoRA | 224.68 min (3.22×) | 13088.00 MiB | 7.03 MiB | 70.31 MiB |
| Fisher shaping overhead | +154.91 min | +27.70 MiB (0.212%) | No additional | No additional |
| Weight merging operations | 0.807 s | +30 KiB | 7.03 MiB | 70.31 MiB |
Time reports the instrumented training lifecycle. GPU memory for the first two rows is the peak PyTorch allocated memory during formal training; the merging entry is a separately measured phase-local increment. CPU cache is per task, whereas archived snapshot storage is cumulative over ten tasks.
Sensitivity shaping overhead. Compared with EWC-LoRA, FS-EWCLoRA requires 3.22× the measured training lifecycle time, while increasing peak allocated GPU memory by only 27.70 MiB (0.212%). The additional runtime primarily arises from sensitivity shaping, which requires four current-task cross-entropy gradient evaluations per update, compared with one for EWC-LoRA. Considering optimizer updates alone, the measured times are 212.01 and 56.16 minutes, respectively (3.78×).
Weight merging overhead. Across ten tasks, collecting 50 CPU factor snapshots takes 0.551 s, while performing ten averaging operations takes 0.256 s, totaling 0.807 s. The maximum additional GPU allocation during merging is 30 KiB. Five cached LoRA-factor snapshots require 7.03 MiB of CPU tensor storage per task, while archiving these snapshots across ten task checkpoints requires 70.31 MiB of disk storage. The CPU cache is reset between tasks, and the archived snapshots are already included in the stored checkpoints.
Overall, the primary computational trade-off of FS-EWCLoRA is the increased training time from fisher shaping, with only a small additional GPU memory requirement. Checkpoint averaging introduces negligible measured computation time, with its principal storage cost arising from CPU snapshot caching and optional archival.
Summary of Changes We have added a detailed computational and storage cost analysis in Appendix B.6, separating sensitivity shaping overhead, checkpoint-averaging operations, and archived checkpoint storage.
Response 1/3
We would like to express our sincere gratitude to the Reviewer 9HoL for the positive comments on the strengths of our work. We greatly appreciate the opportunity to address the concerns and questions raised. Below, we address each point raised by the reviewer and describe the corresponding revisions made to the manuscript, which are highlighted in blue. We hope that our explanations can address the Reviewer's concerns, and we sincerely welcome any further comments or suggestions.
(Weakness 1, Question 1) Additional comparison with EWC+GAM and pointwise ((r=0)) shaping
Thank you for this suggestion. We use the same ImageNet-A task and model configuration as in Table 1: ten tasks of 20 classes and a pretrained ViT-B/16 with rank-10 LoRA. All methods are trained for 50 epochs per task, totaling 2,550 optimizer steps, with identical historical Fisher processing and checkpoint averaging.
For EWC+GAM [1], we compute the second-order terms as Hessian–vector products using automatic differentiation, with ρ=0.2 and α=0.1. For pointwise shaping, we set r=0 while keeping all other FS-EWC settings unchanged. This removes the neighborhood search while retaining the pointwise shaping penalty.
The table reports results for seed 0. Wall-clock time includes monitoring and checkpoint I/O.
FS-EWC achieves higher FAA and better BWT than EWC+GAM in this comparison. The recorded runtime and peak GPU memory of EWC+GAM are approximately 1.50 and 2.28 times those of FS-EWC, respectively.
Relative to pointwise shaping, FS-EWC improves FAA by 4.15 percentage points and BWT by 2.59 percentage points, with similar recorded runtime and memory usage. These results support the additional benefit of neighborhood shaping over its pointwise counterpart under the shared protocol.
[1] Xingxuan Zhang, Renzhe Xu, Han Yu, Hao Zou, and Peng Cui. “Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalization.” CVPR, 2023, 20247-20257.
(Weakness 2, Question2) Implementation clarification on LoRa spaces for parameter perturbation, shaping gradient and parameter averaging
Under LoRA, the effective weights are W=W0+BA, where the pretrained weights W0 remain frozen and the LoRA factors (A,B) are trainable.
Neighborhood perturbations and shaping gradients are computed in the factor coordinates (A,B). These perturbations are temporary and are used only to estimate the current-task shaping gradient. The original parameter state is restored before evaluating the historical EWC penalty.
Historical protection uses effective-weight Fisher importance to penalize deviations of BA from its retained reference. The EWC penalty is evaluated at the unperturbed model state, and its gradient is propagated to (A,B) through the chain rule. Thus, neighborhood perturbations do not alter the retained EWC reference or the historical Fisher importance.
Both gradient contributions are combined to update the same LoRA factors, ensuring that sensitivity shaping and historical protection are evaluated consistently with respect to the same unperturbed state.
Summary of Changes. We have added comparisons with EWC+GAM and pointwise shaping in Appendix B.8 , using matched historical Fisher processing and checkpoint averaging.