acceptodds
Under review as a conference paper at ICLR 2027

Restoring without Forgetting: Filter-Level Continual Image Restoration via Parameter-Space Integrated Gradients

Abstract

Adapting image restoration models to a stream of new tasks without revisiting past data remains challenging due to catastrophic forgetting. In this work, we propose Restoring without Forgetting (RwF), a filter-level continual adaptation framework for image restoration built upon a critical observation: task-specific knowledge is centered in a small subset of filters and can be separated from those reconstructing general content. RwF first performs parameter-space integrated gradients attribution to localize degradation-critical filters in a coarse-to-fine manner. It then adapts to new tasks by generating task-specific filters from a filter bank using compact factorized low-rank transformations, further augmented with cross-task attention and prototypical contrastive learning, and lastly assembles them back only at localized positions. Experiments on six restoration tasks show that RwF effectively avoids forgetting and achieves competitive restoration quality against all-in-one methods that have full data access, and outperforms LoRA-style adaptation with 10 fewer additional parameters.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual image restoration, Representation learning, Integrated gradient, Parameter-Efficient Finetuning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qow5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RwF for continual image restoration. It localizes degradation-critical filters via parameter-space Integrated Gradients and regenerates only those filters from a filter bank using low-rank transforms, leaving the rest of the backbone frozen. The method is evaluated on synthetic and public multi-task benchmarks across CNN, Transformer and diffusion backbones.

Critical strengths

The filter-reset study shows that ~2–3% of filters carry each task’s restoration ability, clearly more than random filters.

Data-free, no inference overhead, and applicable to CNN, Transformer and diffusion backbones.

Critical weaknesses

Freedom from forgetting comes from per-task parameter storage plus task identity. Comparisons with single-model CL baselines and task-agnostic all-in-one methods are therefore not like-for-like, and the main tables don’t state whether oracle task IDs are used.

The parameter-efficiency claim is inconsistent: the abstract and Tab. 3 claim ~10× fewer parameters than LoRA, but Tab. 6 reports per-task storage comparable to rank-8 LoRA (3.7% vs. 3.5%).

The IG baseline does not isolate the current task. Cross-task attention uses untrained random projections, ablation gains are small and from single runs, and several hyperparameters are inconsistent.

Questions for authors

Do the main results use oracle task IDs or the router? Please report end-to-end results with the router.

Please give consistent parameter accounting, including the stored per-task filter values.

What are the configurations of LoRA-Similar and LoRA-Full, and do they use the same task-ID access?

Were the all-in-one baselines retrained on the same splits? Please add joint-training and per-task fine-tuning references.

Please report multi-seed variance for the main results and the ablations.

1 comment

The authors do not provide a rebuttal. I will keep my original rating.

Reviewer_2p6EOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Restoring without Forgetting (RwF), a filter-level continual image restoration framework. It employs parameter-space Integrated Gradients to identify degradation-critical filters, generates task-specific filters through a historical filter bank and low-rank transformations, and incorporates cross-task attention and prototypical contrastive learning. Promising results are reported across multiple restoration tasks and backbones.

Critical strengths
  1. The paper addresses continual image restoration without access to historical training data, a practically relevant problem. Filter-level adaptation offers a meaningful approach to mitigating catastrophic forgetting.
  2. The experiments cover multiple restoration tasks and three distinct backbones (MPRNet, HINT, and DiffUIR), supported by component ablations and parameter-overhead analysis. These results provide useful empirical evidence for the proposed approach.
Critical weaknesses
  1. The method mainly integrates established filter attribution, low-rank parameter generation, task-specific parameter preservation, and contrastive learning. Its novelty beyond Xie et al. (2021) and existing continual learning methods appears incremental.
  2. RwF largely prevents forgetting by storing and reassembling task-specific filters. This resembles parameter isolation rather than demonstrating knowledge retention within a continuously adapted shared model.
  3. The claimed approximately 10-fold parameter reduction is questionable, as Table 6 reports around 3.7% storage per task, comparable to 3.5% for rank-8 LoRA. A consistent parameter-accounting protocol is needed.
  4. Resetting selected filters demonstrates their importance but does not establish degradation-specific knowledge localization. Comparisons with gradient-magnitude or Fisher-based selection are needed.
  5. The main experiments use fixed task orders without multi-seed analysis. The ten-task extension primarily reports parameter consumption rather than long-horizon restoration performance.
Questions for authors
  1. Does Table 3 assume ground-truth task identities for filter selection? Do the reported performance and forgetting-prevention conclusions still hold when using the automatic task router?
  2. How do the authors reconcile the claimed 10-fold parameter reduction with the LoRA-comparable storage overhead in Table 6? Are all task-specific filters, filter banks, and routing parameters included?
  3. What evidence establishes that IG identifies degradation-specific knowledge rather than generally sensitive parameters? Could simpler importance-based selection achieve similar performance under matched parameter budgets?
1 comment

As no rebuttal was submitted by the authors, the original rating will remain unchanged.

Reviewer_spAjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses continual image restoration without access to earlier task data. The authors argue that degradation knowledge is concentrated in a few filters and locate them with parameter-space integrated gradients. After a warm-up, only these filters are adapted, generated from a filter bank via low-rank transforms and cross-task attention; the rest of the backbone is fixed. A prototypical contrastive loss and an FFT loss are added, and a prototype router identifies the task at test time. Experiments cover a synthetic benchmark with MPRNet and a four-task public benchmark with MPRNet, HINT, and DiffUIR. On the latter, RwF-HINT reaches 29.28 dB average PSNR with 0.17M extra parameters, vs. 29.14 dB with 1.73M for LoRA-Full. Ablations and filter-reset experiments are also included.

Critical strengths

1.The filter-reset experiments in App. B are the most convincing evidence to me that restoration ability is concentrated in a few filters. Resetting the localized filters, about 2–3% of the backbone's filters, removes nearly all restoration gain and hurts more than resetting random filters. Later tasks also mostly reuse earlier filters, reaching 88.3% at the tenth task. These results are the main reason I see value in the work. 2.The design in Sec. 3.3 keeps most backbone weights fixed during filter adaptation and stores each task's filters separately, so earlier tasks can be recovered without past data. Generated filters are written into the backbone, adding no adapter computation at inference. This is useful when old data cannot be kept, though it depends on correct task identification.

Critical weaknesses
  1. IG is central to Sec. 3.2, but Tab. 2 does not compare it with simpler filter selection criteria under the same generator and parameter budget. The filter-reset experiments in App. B show that selected filters matter, not that they specifically encode degradation knowledge or require IG. Also, past-task retention is largely ensured by storing separate task-specific filters. Without stronger controls, I cannot tell what IG adds beyond parameter isolation.
  2. The main comparisons and efficiency claims are difficult to assess. Tab. 3 does not clearly state whether ground-truth task identities are used, while the router in App. C reduces average PSNR by 0.32 dB, more than the 0.14 dB gain over LoRA-Full. A comparison under the same task-identification setting is needed. Tab. 3 also reports 0.17M extra parameters without clarifying whether stored filters are included, whereas App. D reports about 3.7% storage per task, comparable to rank-8 LoRA at 3.5%. With the extra warm-up and localization steps, I am not convinced RwF is more efficient than LoRA in practice.
Questions for authors

1.Could the authors compare IG-based filter selection with random selection, gradient magnitude and weight change under the same generator and parameter budget? I would like to see whether IG actually improves over simpler selection methods. 2.Do Tab. 1 and 3 use ground-truth task identities? Could the authors report results with predicted task identities and compare RwF with LoRA under the same routing conditions? 3. Does the 0.17M parameter count in Tab. 3 include stored task-specific filters? App. D reports about 3.7% storage per task, comparable to rank-8 LoRA at 3.5%. Could the authors clarify the total storage and training costs, including warm-up and IG localization? 4.Which loss weights were used in the experiments? Eq. 18 and Alg. 1 specify different relative weights from App. C for the pixel, prototype and FFT losses, which affects reproducibility.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.