acceptodds
Under review as a conference paper at ICLR 2027

Beyond Parameter Protection: Fisher-Shaped EWC for Replay-Free Continual Learning

Abstract

In replay-free continual learning, Elastic Weight Consolidation (EWC) uses Fisher information matrix (FIM) to protect acquired knowledge by penalizing changes to historically important parameters. However, a local importance estimate at the retained solution does not in general characterize sensitivity after finite parameter displacement. This motivates controlling Fisher sensitivity over a neighborhood of potential future updates. We propose Fisher-Shaped EWC (FS-EWC), which extends the role of FIM from parameter protection to solution formation for future retention. During task learning, FS-EWC minimizes the worst-case empirical FIM trace within a prescribed neighborhood, explicitly favoring solutions that are less sensitive to bounded future displacement. At task completion, trajectory-based model merging constructs the retained solution from multiple optimization states rather than a single terminal iterate, after which FIM is estimated for subsequent consolidation. Experiments across multiple benchmarks show that FS-EWC improves performance over EWC, while controlled studies indicate a more favorable stability–plasticity balance and greater tolerance to parameter drift. These results suggest that effective knowledge retention requires both forming solutions with future retention in mind and protecting acquired knowledge.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, EWC, Fisher Inforamtion matrix

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_9HoLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies replay-free class-incremental learning and proposes FS-EWC. It augments EWC’s historical protection with a regularizer targeting the current task’s Fisher sensitivity within a parameter neighborhood, aiming to learn solutions that better tolerate subsequent updates. The method also averages selected training checkpoints to construct the retained anchor, where Fisher importance is estimated for future tasks. Vision and text classification experiments, together with controlled diagnostics, evaluate the benefits of solution shaping and historical protection.

Critical strengths

The controlled studies support the value of solution shaping and show that better retention can coexist with larger parameter drift (Figs. 3–4; Table 4).

The evaluation spans vision and language, including full-parameter and LoRA settings. Improvements over EWC-LoRA are substantial on ImageNet-R/A and also extend to text classification (Tables 1–3).

Tests with different Fisher estimators, representations, regularization strengths, and neighborhood radii provide useful robustness evidence (Tables 11–12; Fig. 5).

Critical weaknesses

The specific benefit over existing gradient-norm regularization remains insufficiently isolated. Comparisons with EWC+GAM and pointwise ((r=0)) shaping, using matched training budgets and checkpoint averaging, would clarify the contribution of the neighborhood design.

Despite the additional gradient evaluations, training time and peak memory are not reported. This leaves the practical trade-off between performance and computational cost unclear.

The main text could more clearly distinguish the theoretical per-example Fisher objective from the batch-gradient surrogate described in Appendix A.5. The parameter space used for LoRA shaping also needs clarification.

Questions for authors

Additional comparison. Under matched training budgets, historical Fisher processing, and checkpoint averaging, how does FS-EWC compare with EWC+GAM and pointwise ((r=0)) shaping?

Implementation clarification. Under LoRA, which parameter spaces are used for the neighborhood perturbation and shaping gradient: the factors (A,B) or the effective update (BA)? How do these relate to the effective-weight Fisher used for historical protection?

Approximation validation. Appendix A.5 acknowledges the batch-gradient approximation. Could a small-scale comparison with per-example Fisher shaping quantify its effect on retention and sensitivity reduction?

Computational cost. What are the training-time and peak-memory overheads relative to EWC-LoRA? Could the costs of sensitivity shaping and checkpoint averaging be reported separately?

3 comments
AuthorsOpenReview ↗

Response 1/3

We would like to express our sincere gratitude to the Reviewer 9HoL for the positive comments on the strengths of our work. We greatly appreciate the opportunity to address the concerns and questions raised. Below, we address each point raised by the reviewer and describe the corresponding revisions made to the manuscript, which are highlighted in blue. We hope that our explanations can address the Reviewer's concerns, and we sincerely welcome any further comments or suggestions.

(Weakness 1, Question 1) Additional comparison with EWC+GAM and pointwise ((r=0)) shaping

Thank you for this suggestion. We use the same ImageNet-A task and model configuration as in Table 1: ten tasks of 20 classes and a pretrained ViT-B/16 with rank-10 LoRA. All methods are trained for 50 epochs per task, totaling 2,550 optimizer steps, with identical historical Fisher processing and checkpoint averaging.

For EWC+GAM [1], we compute the second-order terms as Hessian–vector products using automatic differentiation, with ρ=0.2\rho=0.2 and α=0.1\alpha=0.1. For pointwise shaping, we set r=0r=0 while keeping all other FS-EWC settings unchanged. This removes the neighborhood search while retaining the pointwise shaping penalty.

The table reports results for seed 0. Wall-clock time includes monitoring and checkpoint I/O.

MethodFAABWTWall-clock time (h)Peak allocated GPU memory (GiB)
FS-EWC, r=0.5r=0.561.42−7.572.2912.77
EWC+GAM53.00−10.703.4329.14
Pointwise shaping, r=0r=057.27−10.162.3212.77

FS-EWC achieves higher FAA and better BWT than EWC+GAM in this comparison. The recorded runtime and peak GPU memory of EWC+GAM are approximately 1.50 and 2.28 times those of FS-EWC, respectively.

Relative to pointwise shaping, FS-EWC improves FAA by 4.15 percentage points and BWT by 2.59 percentage points, with similar recorded runtime and memory usage. These results support the additional benefit of neighborhood shaping over its pointwise counterpart under the shared protocol.

[1] Xingxuan Zhang, Renzhe Xu, Han Yu, Hao Zou, and Peng Cui. “Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalization.” CVPR, 2023, 20247-20257.

(Weakness 2, Question2) Implementation clarification on LoRa spaces for parameter perturbation, shaping gradient and parameter averaging

Under LoRA, the effective weights are W=W0+BAW=W_0+BA, where the pretrained weights W0W_0 remain frozen and the LoRA factors (A,B)(A,B) are trainable.

Neighborhood perturbations and shaping gradients are computed in the factor coordinates (A,B)(A,B). These perturbations are temporary and are used only to estimate the current-task shaping gradient. The original parameter state is restored before evaluating the historical EWC penalty.

Historical protection uses effective-weight Fisher importance to penalize deviations of BABA from its retained reference. The EWC penalty is evaluated at the unperturbed model state, and its gradient is propagated to (A,B)(A,B) through the chain rule. Thus, neighborhood perturbations do not alter the retained EWC reference or the historical Fisher importance.

Both gradient contributions are combined to update the same LoRA factors, ensuring that sensitivity shaping and historical protection are evaluated consistently with respect to the same unperturbed state.

Summary of Changes. We have added comparisons with EWC+GAM and pointwise shaping in Appendix B.8 , using matched historical Fisher processing and checkpoint averaging.

AuthorsOpenReview ↗

Response 2/3

(Weakness 4, Question 3) Approximation validation. Comparison with per-sample Fisher

We adopt the batch-gradient surrogate for computational efficiency, following the use of batched Fisher approximations in existing EWC implementations. Direct per-example Fisher shaping requires individual sample gradients and corresponding second-order computations at each training step, introducing substantial computational overhead.

To evaluate this approximation, we compare the batch surrogate PBP_B with the scaled per-example objective PE/bP_E/b, defined as

PB=∥1b∑i=1bgi∥22,P_B=\|\frac{1}{b}\sum_{i=1}^{b}g_i\|_2^2,

PE=1b∑i=1b∣gi∣22.P_E=\frac{1}{b}\sum_{i=1}^{b}|g_i|_2^2.

Here, gig_i is the per-example loss gradient and bb is the actual minibatch size. The additional 1/b1/b factor in the per-example training objective adjusts its scale.

Following the two-task ImageNet-A data split and base training schedule of Table 4, we compare batch-gradient and per-example Fisher shaping on the first two ImageNet-A tasks, with 20 classes each, using ViT-B/16 with rank-10 LoRA. Both variants use HVPs computed by automatic differentiation, a one-step search with r=0.05r=0.05, and αF=0.001\alpha_F=0.001. Training time and peak allocated GPU memory are measured on the same GPU.

Both variants achieve 86.32% accuracy after Task 1. The table reports same-seed results, with Task-2 accuracies evaluated over all 40 seen classes.

Training shaping targetOld-task Acc. (%)New-task Acc. (%)BWT (pp)Time (h)Peak GPU (GiB)
Batch surrogate, PBP_B82.4180.71−3.911.1830.69
Scaled per-example, PE/bP_E/b82.0880.31−4.236.8912.75

The two approaches achieve comparable retention, with the batch surrogate yielding 0.33 percentage points higher old-task accuracy and 0.32 percentage points better BWT. These results indicate comparable retention and new-task accuracy under the evaluated setting.

The batch-gradient implementation reduces runtime from 6.89 to 1.18 hours, providing a 5.84$\times$ speedup. The per-example implementation computes second-order derivatives sequentially for individual samples and accumulates detached results, resulting in lower peak GPU memory but substantially longer runtime.

Together, these results support the batch-gradient surrogate as a practical choice for Fisher sensitivity shaping. Under the evaluated two-task setting, it achieves comparable retention and measured endpoint sensitivity while substantially reducing computation time. This efficiency consideration also motivates the finite-difference implementation adopted in FS-EWC.

Summary of Changes. We have added a controlled comparison between batch-gradient and scaled per-example Fisher shaping in Appendix B.9, reporting retention, runtime, and peak GPU memory.

AuthorsOpenReview ↗

Response 3/3

(Weakness 3, Question 4) Computational cost. Training Time and Peak Memory

We report a computational cost analysis on ImageNet-A using ViT-B/16 with rank-10 LoRA. All configurations use the same 10-task sequence, 50 epochs per task, Fisher processing, and checkpoint-averaging protocol. We compare EWC-LoRA (α=0\alpha=0) and FS-EWCLoRA (α=5\alpha=5) with λ=100000\lambda=100000. Both methods average selected LoRA checkpoints from epochs 30, 40, and 50.

Training time and memory overhead.

ComponentTimeGPU allocated memoryCPU snapshot cacheArchived snapshots
EWC-LoRA69.77 min (1.00×)13060.30 MiB7.03 MiB70.31 MiB
FS-EWCLoRA224.68 min (3.22×)13088.00 MiB7.03 MiB70.31 MiB
Fisher shaping overhead+154.91 min+27.70 MiB (0.212%)No additionalNo additional
Weight merging operations0.807 s+30 KiB7.03 MiB70.31 MiB

Time reports the instrumented training lifecycle. GPU memory for the first two rows is the peak PyTorch allocated memory during formal training; the merging entry is a separately measured phase-local increment. CPU cache is per task, whereas archived snapshot storage is cumulative over ten tasks.

Sensitivity shaping overhead. Compared with EWC-LoRA, FS-EWCLoRA requires 3.22× the measured training lifecycle time, while increasing peak allocated GPU memory by only 27.70 MiB (0.212%). The additional runtime primarily arises from sensitivity shaping, which requires four current-task cross-entropy gradient evaluations per update, compared with one for EWC-LoRA. Considering optimizer updates alone, the measured times are 212.01 and 56.16 minutes, respectively (3.78×).

Weight merging overhead. Across ten tasks, collecting 50 CPU factor snapshots takes 0.551 s, while performing ten averaging operations takes 0.256 s, totaling 0.807 s. The maximum additional GPU allocation during merging is 30 KiB. Five cached LoRA-factor snapshots require 7.03 MiB of CPU tensor storage per task, while archiving these snapshots across ten task checkpoints requires 70.31 MiB of disk storage. The CPU cache is reset between tasks, and the archived snapshots are already included in the stored checkpoints.

Overall, the primary computational trade-off of FS-EWCLoRA is the increased training time from fisher shaping, with only a small additional GPU memory requirement. Checkpoint averaging introduces negligible measured computation time, with its principal storage cost arising from CPU snapshot caching and optional archival.

Summary of Changes We have added a detailed computational and storage cost analysis in Appendix B.6, separating sensitivity shaping overhead, checkpoint-averaging operations, and archived checkpoint storage.

Reviewer_aLA7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that EWC’s local Fisher estimate may fail to characterize sensitivity under finite parameter displacement. It proposes Fisher-Shaped EWC (FS-EWC), which (i) augments the current-task EWC objective with the worst-case empirical FIM trace over an ℓ2\ell_2 ball of radius rr around the current parameters, optimized by one-step finite-difference ascent, and (ii) averages checkpoints along the task’s optimization trajectory to construct the retained anchor before FIM estimation and consolidation. It proves a finite-displacement bound (Proposition 1), gives a Bayesian interpretation, and evaluates on vision and language benchmarks, reporting gains over EWC/EWC-LoRA and diagnostics suggesting that shaped solutions tolerate larger drift with smaller loss increases.

Critical strengths
  1. This paper offers a clear and useful reframing of EWC’s limitation: a local FIM at the retained solution does not characterize loss sensitivity after finite drift. The motivation is well founded. The idea of shaping the solution for future retention rather than only protecting it during later tasks is reasonable.

  2. The evaluation spans four standard vision benchmarks, five fine-grained/heterogeneous datasets, and two Llama scales across three NLP task orders. The diagnostics in Fig. 3–4 and Table 4 provides direct evidence for the paper’s central mechanism.

Critical weaknesses
  1. The core objective of this paper is very close to C-Flat/C-Flat++ [1]. Both belong to the family of flatness/sharpness-aware regularization for continual learning. In its first-order component, C-Flat penalizes the worst-case gradient norm over a parameter neighborhood, whereas FS-EWC penalizes the worst-case empirical Fisher trace. Since the theoretical empirical Fisher trace is the average squared per-example gradient norm, FS-EWC’s penalty is a close squared analogue of C-Flat’s first-order flatness term; in the batch approximation used by FS-EWC, it reduces to the squared batch-gradient norm, which is also the quantity C-Flat++ uses as its sharpness proxy (though C-Flat++ uses it as a trigger for selective updates rather than as the penalty itself). Both methods approximate the inner maximization with a one-step perturbation and a Hessian–vector product (finite difference in FS-EWC), add the flatness term to the task loss, and motivate it with analogous neighborhood loss bounds. FS-EWC can therefore be viewed as a close specialization of C-Flat’s first-order flatness idea to EWC. The remaining differences are mainly scope and additional components and these differences do not amount to a fundamentally different mechanism.

  2. The evidence does not support the two-component story, and several empirical claims are overstated. (1) Table 6 reports endpoint 59.23/70.63 vs. trajectory 59.34/70.84 (FAA/AAA); Table 13 shows overlapping intervals with standard deviations of 0.3–0.9, and “late-stage” has a higher FAA than “trajectory.” Moreover, Table 2 explicitly disables multi-checkpoint averaging, so the large fine-grained gains come from shaping alone. The merging component is therefore not validated as a contribution. (2) Some gains are marginal and inconsistent across metrics (CIFAR-100 EWC-LoRA: FAA 88.03 vs. 87.91 but AAA 92.00 vs. 92.27; DomainNet AAA 79.50 vs. 79.50). (3) The shaping term in Eq. (12) is added with unit weight and only rr is tuned. The scale balance with the EWC term is not analyzed.

[1] Li W, Yuan H, Zhao Z, et al. C-flat++: Towards a more efficient and powerful framework for continual learning[J]. International Journal of Computer Vision, 2026, 134(10): 435.

Questions for authors

Please refer to Critical Weaknesses.

4 comments
AuthorsOpenReview ↗

Response 1/3

We sincerely thank the Reviewer aLa7 for taking the time to carefully evaluate our work and provide constructive suggestions. Below, we address each point raised by the reviewer and describe the corresponding revisions made to the manuscript, which are highlighted in blue. We hope that our explanations can address the Reviewer's concerns, and we sincerely welcome any further comments or suggestions.

(Weaknesses 1) Clarification the relation between flatness-aware continual learning methods and FS-EWC.

We thank the reviewer for pointing out the connection to flatness-aware methods like C-Flat [1] and C-Flat++[2]. We have uploaded a revised manuscript and added a paragraph entitled Flatness-Aware Continual Learning to the Related Works section. This addition summarizes prior advances in flatness-aware continual learning and clarifies our specific research focus: analyzing finite-displacement sensitivity beyond EWC's pointwise historical importance estimates and investigating how reference sensitivity shaping affects subsequent Fisher-based consolidation.

FS-EWC starts from a limitation of EWC: importance estimated at a retained solution does not fully characterize the sensitivity encountered under finite future parameter displacement. Proposition~1 relates finite task loss changes to Fisher sensitivity along displacement paths. Since these future paths are unknown during current-task learning, the analysis motivates controlling worst-case neighborhood Fisher sensitivity while preserving inherited EWC protection. We instantiate this criterion using the empirical Fisher, whose trace equals the average squared norm of per-example gradients.

We acknowledge the resulting close relationship to gradient norm regularization, particularly under the batch-gradient approximation for computational efficiency. However, this methodological overlap does not fully characterize our contribution. We explicitly investigate how shaping the learned reference during current-task training affects subsequent retention under historical Fisher protection. The controlled studies in Figs.3--4 show that improved retention can coexist with greater parameter drift and reduced sensitivity under matched perturbations, while Table 4 distinguishes the effects of shaping during reference formation from those during subsequent task adaptation. These analyses provide evidence for the role of reference sensitivity in Fisher-based consolidation.

We have also added comparisons with C-Flat and C-Flat++, each combined with EWC, under the same training protocol. Hyperparameters are configured according to the recommendations in the original papers, with both perturbation radii set to ρ=r=0.2,λCF=0.2\rho=r=0.2,\lambda_{CF}=0.2. Each entry below reports FAA /AAA(%). We also update this result in Table 2 in the manuscript.

MethodFGVC-AircraftStanford CarsCUB-2005Datasets
C-Flat + EWC44.14 / 36.2251.59 / 55.3067.94 / 71.3979.28 / 76.18
C-Flat++ + EWC45.22 / 40.4339.46 / 46.9367.24 / 70.5977.81 / 75.39
FS-EWCLoRA46.66 / 43.4153.04 / 56.2673.42 / 76.2683.24 / 81.19

Across the four evaluated benchmarks, FS-EWCLoRA achieves higher FAA and AAA than both C-Flat + EWC and C-Flat++ + EWC. These results support our design, showing gains over both baselines even when they include EWC.

Summary of Changes. We have expanded the flatness-aware continual learning discussion in Section 2 and added comparisons with C-Flat and C-Flat++, both combined with EWC, in Table 2.

[1] Ang Bian, Wei Li, Hangjie Yuan, et al. Make Continual Learning Stronger via C-Flat. NIPS, 37, 2024.

[2] Li W, Yuan H, Zhao Z, et al. C-flat++: Towards a more efficient and powerful framework for continual learning[J]. International Journal of Computer Vision, 2026, 134(10): 435.

AuthorsOpenReview ↗

Response 2/3

(Weakness 2.1) Insufficient empirical validation of the merging component

We agree that checkpoint averaging does not consistently outperform endpoint retention and that its effectiveness depends on the dataset and checkpoint selection. In FS-EWC, averaging provides an alternative way to contruct the retained reference for subsequent EWC consolidation, with endpoint retention as its single-checkpoint special case (Eq. 14). Current-task Fisher importance is then estimated at the resulting reference.

While averaging is theoretically motivated (see Appendix A.4 and our response to Reviewer VLv3), its empirical gains are conditional. Tables 6 and 13 examine different checkpoint-selection strategies. Table 6 shows modest differences relative to endpoint retention, while Table 13 identifies Late-stage as the best configuration for mean FAA and Full-stage for AAA. These results further highlight the importance of checkpoint selection.

The improvements in Table 2 are independent of averaging and instead demonstrate the effectiveness of Fisher sensitivity shaping combined with historical EWC protection. To further assess the effect of averaging, we extend the Endpoint/Merge comparison to all five benchmarks in Table 2. For both Vanilla LoRA and FS-EWCLoRA, the variants share the same training configuration and random seed, differing only in retained-reference construction. We report FAA / AAA changes (percentage points) relative to each Endpoint baseline.

MethodAircraftCars196CUB-200Flowers5Datasets
SGD+0.69 / −0.06+0.30 / −0.14−0.15 / −0.22−0.29 / −0.44−0.28 / −0.20
FS-EWC−0.21 / +0.21+1.52 / +1.11−0.17 / −0.12−0.47 / −0.22−0.08 / −0.08

For FS-EWCLoRA, averaging improves FAA and AAA on Stanford Cars by 1.52 and 1.11 percentage points, respectively, while producing small or negative changes on the remaining benchmarks. Taken together, the theoretical analyses provide a conditional rationale for checkpoint averaging, while the empirical results show that its benefits depend on the dataset and checkpoint selection. Accordingly, we clarify that checkpoint averaging serves as an auxiliary retained-reference strategy, while Fisher sensitivity shaping remains the central contribution of FS-EWC.

Summary of Changes. We have revised the Abstract, Fig. 1 caption, Introduction to clarify that Fisher sensitivity shaping, integrated with historical EWC protection, is the central contribution of FS-EWC, while checkpoint averaging serves as an auxiliary strategy for retained-reference construction. We have also revised Section 3.3.2 and the description of Eq. (14) to clarify the theoretical motivation for checkpoint averaging and its conditional empirical benefits, emphasizing that averaging is optional and its effectiveness depends on checkpoint selection and the dataset.

(Weakness 2.2) Some gains are marginal

We agree that the magnitude of improvement varies across datasets and metrics. Nevertheless, our evaluations across four standard vision benchmarks, five fine-grained and heterogeneous settings, and two Llama model scales under three NLP task orders collectively support the effectiveness of FS-EWC across diverse settings. The controlled diagnostics in Figs. 3–4 and Table 4 further provide direct evidence for the proposed solution-shaping mechanism.

AuthorsOpenReview ↗

Response 3/3

(Weakness 2.3) The scale balance with the EWC term

We agree that Eq. (12) assigns a default unit weight to the shaping term and that its balance with historical EWC protection was insufficiently analyzed.

To examine the practical balance, we denote the effective coefficient of the implemented shaping correction by α\alpha, while λ\lambda scales historical EWC protection. The original formulation corresponds to α=1\alpha=1. Consistent with Eq. (12), the practical update direction is dt=gt(θ)+αh^t(θ;r,ρ)+λΛt−1(θ−θt−1),d_t=g_t(\theta)+\alpha\widehat h_t(\theta;r,\rho)+\lambda\Lambda_{t-1}(\theta-\theta_{t-1}), where h^t\widehat h_t denotes the finite-difference correction associated with the gradient of the emperical Fisher trace hth_t. The neighborhood radius rr and finite-difference probe scale ρ\rho determine the queried gradients and are kept fixed in this analysis. Setting α=0\alpha=0 removes shaping, while λ=0\lambda=0 removes historical protection. The joint sweep uses ImageNet-A with same seed, and weight averaging of checkpoints at epochs 30, 40, and 50. All other experimental settings remain fixed.

α\λ\alpha \backslash \lambda002k2k20k20k100k100k200k200k
0048.2649.1149.4450.4951.22
0.50.557.0157.0857.8056.9557.47
1159.6459.5859.4560.1759.58
5561.4262.0861.4961.7561.42
101057.3459.5159.9160.1160.37

Effects of shaping and protection strengths on FAA. With shaping disabled (α=0\alpha=0), increasing λ\lambda from 00 to 200000200000 improves FAA from 48.2648.26% to 51.2251.22%. In contrast, with EWC disabled (λ=0\lambda=0), increasing α\alpha from 00 to 55 yields a larger improvement, from 48.2648.26% to 61.4261.42%. Combining α=5\alpha=5 with λ=2000\lambda=2000 further increases FAA to 62.0862.08%, the highest among the evaluated configurations. However, the improvement is non-monotonic: at λ=2000\lambda=2000, increasing α\alpha from 11 to 55 improves FAA from 59.5859.58% to 62.0862.08%, whereas further increasing it to 1010 reduces FAA to 59.5159.51%. These results show that shaping strength matters and that tuning EWC alone cannot achieve the same gains within the tested range.

Mechanistic interpretation through plasticity and stability. The plasticity - stability decomposition in Fig. 6 further explains these differences. Increasing EWC protection without shaping primarily improves BWT. In contrast, Fisher shaping improves both AAnewAA_{new} and BWT. This is consistent with Figs.3 - 4 and Table 4, which show that shaping reduces sensitivity to parameter displacement, allowing improved retention despite greater subsequent movement, whereas EWC protection primarily constrains subsequent updates. At α=5\alpha=5, adding EWC with λ=2000\lambda=2000 further improves BWT by 0.790.79 percentage points, producing an additional FAA gain of 0.660.66 percentage points. Meanwhile, α=5\alpha=5 achieves stronger new-task acquisition than α=1\alpha=1, despite a more negative BWT. These findings show that shaping and historical protection provide complementary benefits, with their relative strengths determining the resulting plasticity-stability balance.

Summary of Changes. We have added a controlled joint coefficient study to Section 4.3 of the revised manuscript. Figure 6 presents the FAA heatmap and the corresponding new-task acquisition versus backward-transfer analysis.

Thanks for your detailed response, which adequately addresses my concerns. I have therefore raised my score. Good luck!

Reviewer_VLv3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the limitations of EWC in rehearsal-free continual learning, noting that conventional local importance estimation fails to effectively capture a model's sensitivity to future parameter shifts. To overcome this, the authors propose Fisher-Shaping EWC (FS-EWC), which minimizes the trace of the empirical Fisher Information Matrix, considered within a defined neighborhood, during task learning. This approach proactively shapes solutions that are robust to future parameter displacements. Furthermore, the method employs a trajectory-based model merging technique upon task completion to construct a preserved solution. Empirical evaluations across various vision and language benchmarks demonstrate that FS-EWC outperforms traditional EWC.

Critical strengths

The research is well-motivated, astutely identifying the shortcomings of traditional EWC, specifically its reliance on local parameter importance. Meanwhile, it shifts the protection mechanism from passive post-hoc constraints to proactive solution shaping.

The authors provide a theoretical foundation, deriving bounds for sensitivity to finite parameter displacements and rigorously demonstrating that controlling the Fisher trace within a local neighborhood effectively limits the increase in task loss.

The experimental evaluation validates the effectiveness of FS-EWC—under both full-parameter and LoRA fine-tuning—across diverse vision and text benchmarks, while ablation studies clearly highlight its advantages in balancing stability and plasticity.

Critical weaknesses

While estimating sensitivity gradients via finite-difference probes is computationally feasible, the additional neighborhood searches and gradient evaluations required for each training batch impose a non-negligible computational overhead. Moreover, only adapting the first-order ascent methods to approximate maximization within a local neighborhood is often insufficient for complex loss landscapes characterized by high non-convexity or significant curvature.

Although trajectory-based model fusion techniques can improve the quality of the retained solutions, there is a lack of solid theoretical analysis to explain why the fused state should outperform a solution derived from a single final state.

Questions for authors

refer to weakness

3 comments
AuthorsOpenReview ↗

Response 1/2

We sincerely thank the Reviewer VLv3 for taking the time to carefully evaluate our work and provide valuable feedback. Below, we address each point raised by the reviewer and describe the corresponding revisions made to the manuscript, which are highlighted in blue.

(Weakness 1) The first-order ascent methods to approximate maximization

We thank the reviewer for highlighting the limitations of the inner maximization. Our one-step neighborhood search maximizes a first-order approximation of the fisher trace.

We agree that this approximation can become inaccurate when curvature effects are substantial and does not exactly solve the original maximization. However,a full second-order expansion of the fisher trace would generally involve third-order derivatives of the training loss, adding further computational cost to a search repeated at every training batch.

Multiple ascent steps offer a practical way to refine the search[1][2]. Following the projected-ascent strategy examined in Appendix of SAM [1], we vary the number of inner maximization steps. Starting from ϵ0=0\epsilon_0=0, we perform KK projected ascent steps within a fixed ℓ2\ell_2-ball centered at θ0\theta_0. At each step, we recompute and normalize the approximate sensitivity-ascent direction at θ0+ϵt\theta_0+\epsilon_t. The terminal perturbation is used to update parameter.

We use ImageNet-A with 10 tasks and a pretrained ViT-B/16 with rank-10 LoRA. The neighborhood radius, finite-difference probe radius, and outer finite-difference coefficient are fixed at r=0.5r=0.5, δ=0.05\delta=0.05.

FS-EWC with different numbers of inner ascent steps on ImageNet-A

Ascent steps KKCE gradient evaluations per updateFAA ↑BWT ↑Optimization time (h:mm)Relative time
1 (default)461.42−7.572:10:011.00×
3860.70−8.344:39:392.15×
51260.50−8.616:15:132.89×

Each CE gradient evaluation comprises one forward pass and one first-order backward pass for the current-task cross-entropy loss; the separate EWC gradient calculation is excluded from this count. Optimization time measures the CUDA-synchronized update loop, including EWC, finite-difference searches, and optimizer updates. It excludes data loading, evaluation, task-end Fisher estimation, model merging, and checkpoint saving.

Compared with K=3K=3 and K=5K=5, the default one-step search achieves 0.72 and 0.92 percentage points higher FAA while reducing optimization time by 53.51% and 65.35%, respectively. Thus, with the number of outer updates held fixed, additional ascent steps substantially increase computational cost without improving accuracy. These results support the one-step search used in our paper.

Summary of Changes. We have added a multi-step neighborhood-search ablation in Appendix B.7, comparing one, three, and five inner ascent steps in terms of final accuracy, backward transfer, and optimization time.

[1] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-Aware Minimization for Efficiently Improving Generalization. ICLR, 2021.

[2] Hoki Kim, Jinseong Park, Woojin Lee, Jaewook Lee, and Yujin Choi. Exploring the Effect of Multi-Step Ascent in Sharpness-Aware Minimization. IEEE Access, 14:82166–82179, 2026.

AuthorsOpenReview ↗

Response 2/2

(Weakness 2) Theoretical rationale for model averaging.

We thank the reviewer for requesting a clearer theoretical comparison between the averaged reference and the final checkpoint.

From the Bayesian perspective underlying EWC, Eq. (35) in Appendix A.4 provides a Gaussian fitting interpretation of this retained anchor. For selected checkpoints θ1,…,θK\theta_1,\ldots,\theta_K, their arithmetic average θˉ=K−1∑k=1Kθk\bar\theta=K^{-1}\sum_{k=1}^{K}\theta_k is the center minimizing the mean squared Mahalanobis distance to these states under a shared positive-definite precision matrix. A related Gaussian interpretation of parameter averaging is presented in~[1]. This interpretation motivates the reference construction but does not guarantee improvement over endpont.

Let θE\theta_E denote the final checkpoint. Suppose that a fixed task loss admits a common local quadratic approximation qq, with constant Hessian H⪰0H\succeq 0, over the selected checkpoints, their average, and the endpoint. Under this local model, we have q(θˉ)−q(θE)=D−V,q(\bar\theta)-q(\theta_E)=D-V,where

\frac{1}{K}\sum_{k=1}^{K} \left[q(\theta_k)-q(\theta_E)\right], V= \frac{1}{2K}\sum_{k=1}^{K} (\theta_k-\bar\theta)^\top H(\theta_k-\bar\theta).$$ Here, $D$ is the average checkpoint surrogate-loss gap relative to the endpoint. The nonnegative term $V$ measures the reduction obtained by parameter averaging relative to the average checkpoint surrogate loss. The selected checkpoints and the endpoint depend on the stochastic training trajectory. Taking expectations over this optimization randomness gives $$\mathbb E \left[q(\bar\theta)-q(\theta_E)\right] =\mathbb E[D]-\mathbb E[V].$$ Thus, averaging improves the expected local surrogate when $\mathbb E[V]>\mathbb E[D].$ This condition depends on both checkpoint selection and the quality of the endpoint under the same loss. Checkpoints collected while the model is still adapting may have higher losses than a well-converged endpoint, increasing $D$ and potentially offsetting the averaging benefit. When selected checkpoints occupy a stable local region and their average surrogate loss is close to that of the endpoint, their curvature-weighted dispersion can allow $V$ to exceed $D$. When the selected checkpoints and the endpoint are already close, both terms may be small, leaving limited scope for a difference. In summary, the benefit depends on the selected training window, local training dynamics, and endpoint quality. This provides a conditional local rationale for averaging, rather than a general guarantee, while its practical effect should be assessed through a direct comparison with the endpoint under the same training protocol. **Summary of Changes.** We have added a conditional local quadratic analysis comparing the averaged reference with the final checkpoint immediately after Eq. (35) in Appendix A.4 (pages 17–18, lines 917–938). We have also revised the retained-reference construction in Section 3.3.2 and its empirical discussion in Section 4.3 (Table 6) and Appendix B.5. [1] Marouf I E, Roy S, Tartaglione E, et al. Weighted Ensemble Models Are Strong Continual Learners[C]//ECCV. 2024: 306–324.

Thanks for your detailed response. I will maintain my score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response

We sincerely thank all three reviewers for their careful evaluations and constructive feedback. We appreciate their recognition of our motivation to address EWC's limitations under finite parameter displacement, the theoretical analysis, and the controlled experimental evidence. We also value their positive assessments of our extensive evaluations across vision and language benchmarks under both full-parameter and LoRA settings.

We have carefully addressed the reviewers' concerns and revised the manuscript accordingly, with changes highlighted in blue. We have also conducted additional controlled experiments, presented in the individual responses below. The key updates are summarized as follows:

  • Clarification of the contribution and relationship to prior work. We expanded the Related Works with a dedicated Flatness-Aware Continual Learning discussion (Section 2) and added comparisons with C-Flat and C-Flat++, both combined with EWC (Table 2). We also revised the Abstract, Fig. 1, Introduction, and contribution summary to emphasize Fisher sensitivity shaping as the central contribution, integrated with historical EWC protection. (aLA7)

  • Refined retained-reference construction and theoretical analysis. We clarified checkpoint averaging as an optional strategy in Section 3.3.2, now entitled Retained Reference Construction. We extended the theoretical analysis in Appendix A.4 to characterize when averaging can improve upon the final checkpoint, and expanded the empirical analysis in Section 4.3 (Table 6) and Appendix B.5 (Tables 13–14) to examine its conditional benefits across checkpoint selections and datasets. (VLv3, aLA7)

  • Deeper analysis of sensitivity shaping and historical protection. We added a joint analysis of shaping strength α\alpha and EWC protection strength λ\lambda (Section 4.3, Fig. 6), including an FAA heatmap and a new-task acquisition versus BWT analysis. These results clarify the complementary effects of shaping and historical protection on the plasticity-stability trade-off. (aLA7)

  • Additional validation and computational analysis. We examine the effect of multiple inner ascent steps, and compare batch-gradient and per-example Fisher shaping. We also clarify the LoRA parameter used for shaping and historical protection and report training-time, GPU-memory, and checkpoint-averaging costs in Appendix B.6 (Tables 15-16) (VLv3, 9HoL). These additional results and implementation clarifications are detailed in the corresponding reviewer responses.

We thank the reviewers again for their suggestions, which have helped us clarify the scope of our contribution, and strengthen its empirical support.

Reviewer_9HoLOpenReview ↗

Beyond Parameter Protection

Thank you for the detailed rebuttal and additional experiments, and keep my original rating unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.