acceptodds
Withdrawn from ICLR 2027

Learning to Restore More: Continual Capability Expansion for Pretrained Image Restoration Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Image restoration models are typically trained with a fixed set of capabilities. When new restoration requirements emerge, existing solutions usually train additional models or jointly retrain the original model with both new and historical data. Instead of designing another restoration backbone, we investigate how a trained restorer can continually acquire new capabilities without forgetting those learned previously. We propose RestoreMore, a continual capability-expansion framework that preserves the pretrained restoration model as a frozen capability anchor and learns residual expansion modules for newly arriving degradations. RestoreMore introduces a capability-oriented bi-level routing mechanism at multiple feature stages. The first routing level identifies restoration capabilities relevant to the current input, while the second selects and combines a sparse set of complementary degradation experts. This design enables newly introduced tasks to selectively reuse historical restoration knowledge and progressively enriches the expert bank available for subsequent restoration tasks. Extensive experiments on a wide range of restoration benchmarks demonstrate that RestoreMore consistently acquires new restoration abilities while preserving and improving previously learned capabilities.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Image restoration, Continual learning, Capability expansion

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TERzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies continual capability expansion for image restoration: given a pretrained image restoration model, enable it to sequentially learn new degradation types without accessing historical training data, while preserving its existing restoration performance as much as possible.

The authors propose RestoreMore. The method freezes the original pretrained restoration model and inserts lightweight residual expansion modules after several feature stages. At each stage, the model first extracts spatial and frequency statistics from the current feature to construct a degradation descriptor qlq_l. It then selects the Top-MM capabilities based on the similarity between qlq_l and historical capability prototypes pi,lp_{i,l}. Each selected capability corresponds to a router Ri,lR_{i,l} that maps qlq_l into a routing query ri,lr_{i,l}. Next, ri,lr_{i,l} is matched against the keys kj,lk_{j,l} of experts in an expert bank to select the Top-KK experts. Finally, the model applies a two-level weighting scheme to the selected expert outputs and adds the resulting residual feature back to the original backbone feature.

When a new degradation task arrives, the authors add a new capability prototype, router, residual expert, and routing key, keeping the pretrained backbone and past components frozen. Experiments are conducted across CNN, Transformer, and Mamba backbones on tasks including deraining, desnowing, dehazing, deblurring, and denoising.

Overall, the strengths of the paper lie in its practical problem formulation, its ability to easily extend existing restoration models, and its solid continual learning results across multiple backbones and tasks. The main weakness is that the necessity of bi-level routing over simpler expert routing lacks convincing validation. In addition, the methodology section and mathematical notations could be organized more clearly to reduce reading effort, and the main architecture diagram and experiment figures are presented in raster format rather than vector format, leaving clear room for visual improvement.

Critical strengths

1. The problem formulation is practical, and the method directly applies to existing image restoration models.

The paper focuses on continuously adding new restoration capabilities to an already trained restoration model without retraining from scratch. RestoreMore keeps the original model as a frozen capability anchor and only adds residual expansion modules at specific feature stages. This interface places minimal constraints on the internal backbone architecture, and the authors validate it across CNN, Transformer, and Mamba designs. Compared to continual learning methods designed for one specific restoration architecture, this approach provides broader model compatibility, which is one of the most valuable aspects of this work.

2. The experiments thoroughly evaluate the core objective: learning new capabilities while preserving old ones.

The paper tracks performance changes across incremental stages for both old and new tasks, rather than just reporting final average metrics. For example, when expanding ALGNet from a single deblurring model to five restoration tasks, the initial deblurring capability shows minimal drop after multiple expansions, while newly added tasks achieve strong restoration performance. The authors also investigate task orderings, real-world scenes, unseen degradation types, and unseen degradation severities. Overall, these experiments demonstrate that RestoreMore achieves a solid empirical balance between stability and plasticity.

Critical weaknesses

1. The novel algorithmic contribution is not sufficiently positioned against existing continual image restoration works.

Several key ingredients of RestoreMore are already present in published continual image restoration methods. DELNet / DELNet+ [1, 2] use a dynamic expert library, freeze historical experts, add new experts for newly arriving tasks, and reuse historical experts through similarity-based routing. LDP [3] also uses query-key matching to retrieve accumulated knowledge.

Therefore, the main technical contribution of RestoreMore appears to lie in its particular combination of a frozen pretrained restorer, stage-wise residual expansion, and the bi-level capability-to-expert routing mechanism. However, the paper does not clearly explain how this design differs from, or improves upon, these closely related approaches.

A more direct comparison with DELNet / DELNet+ and LDP is needed to clarify the actual methodological contribution of RestoreMore.

2. Bi-level routing is a core design choice in this paper, but the current experiments do not demonstrate that this extra level is necessary.

For each selected capability, RestoreMore first computes a weighted combination of expert outputs and then performs another weighted combination across the selected capabilities. The final residual can therefore still be viewed as an input-dependent mixture of experts.

A natural simplified baseline would directly use the descriptor qlq_l to generate a routing query, match it against the expert keys, and select the Top-KK experts, without the intermediate capability-selection stage. Such a single-level query-key router would still support continual expansion of the expert bank.

This issue is especially apparent during the first incremental stage. At this point, the expert bank contains only the null expert and one non-zero expert E1E_1. Even if multiple capabilities are selected, all capability-specific routers ultimately operate over the same single non-zero expert. The bi-level mechanism therefore mainly controls the weight of E1(ul)E_1(u_l) at this stage, while genuine multi-expert composition only becomes possible after more tasks are accumulated.

I therefore think a direct comparison with a single-level query-key expert router is necessary to justify the bi-level design.

[1] Shihong Liu, Kun Zuo, and Hanguang Xiao. DELNet: Continuous All-in-One Weather Removal via Dynamic Expert Library. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2026.

[2] Shihong Liu, Kun Zuo, and Hanguang Xiao. DELNet+: Dynamic Expert Library Image Continual Restoration Network. Image and Vision Computing, 2026.

[3] Peijun Zhao, Tongjun Wang, Wei Wang, Zhaoxu Feng, and Songnan Chen. Learning Discriminative Prompts for Continual Image Restoration. IEEE Access, 2026.

Questions for authors

1. [Evaluation-critical experimental question] Can the authors provide a single-level query-key expert routing baseline matched in parameter count and computational cost?

Specifically, the model could map the current degradation descriptor qlq_l through a single router:

rl=Rl(ql),r_l = R_l(q_l),

compute

sim⁡(rl,kj,l),\operatorname{sim}(r_l, k_{j,l}),

and directly select the Top-KK experts, bypassing the intermediate capability prototype selection and capability-specific routers.

This comparison would address the central design question: given an extensible query-key expert bank, does the extra hierarchy

ql→capability→Ri,l(ql)→expertq_l \rightarrow \text{capability} \rightarrow R_{i,l}(q_l) \rightarrow \text{expert}

consistently outperform the direct formulation

ql→Rl(ql)→expert?q_l \rightarrow R_l(q_l) \rightarrow \text{expert}?

Because bi-level routing is the main component distinguishing this paper from prior modular continual restoration methods, this result will heavily influence my assessment of the technical contribution.

2. [Evaluation-critical related-work / experimental question] How do the authors distinguish RestoreMore from DELNet / DELNet+ [1, 2] and LDP [3]?

DELNet and DELNet+ already use a dynamic expert library, adding new experts for new tasks, freezing historical experts, and reusing past knowledge via similarity-based selection. LDP already uses query-key retrieval in continual image restoration to select relevant prompts from an expanding pool.

Based on the current manuscript, the primary distinctions appear to be: using an arbitrary pretrained restorer as a frozen anchor, applying residual expansion across multiple feature stages, and employing capability-to-expert hierarchical routing. I encourage the authors to articulate these differences more explicitly, and ideally provide direct experimental comparisons or controlled ablations on these key designs.

Demonstrating that stage-wise expansion or bi-level routing provides consistent and notable gains over existing expert/prompt retrieval mechanisms would substantially strengthen my evaluation of the paper's novelty.


[1] Shihong Liu, Kun Zuo, and Hanguang Xiao. DELNet: Continuous All-in-One Weather Removal via Dynamic Expert Library. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2026.

[2] Shihong Liu, Kun Zuo, and Hanguang Xiao. DELNet+: Dynamic Expert Library Image Continual Restoration Network. Image and Vision Computing, 2026.

[3] Peijun Zhao, Tongjun Wang, Wei Wang, Zhaoxu Feng, and Songnan Chen. Learning Discriminative Prompts for Continual Image Restoration. IEEE Access, 2026.

Reviewer_aNpGOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RestoreMore is a continual learning framework that expands pretrained image restoration models to handle new degradations, such as rain, snow, haze, blur, and noise, without retraining the original model or revisiting historical training data. It freezes the pretrained backbone and adds lightweight residual experts for each new task. At inference, a two-stage routing mechanism first selects relevant restoration capabilities and then combines a sparse set of complementary experts. Experiments across CNN-, Transformer-, and Mamba-based models report that RestoreMore learns new restoration tasks while largely preserving, and sometimes improving, existing capabilities compared with joint retraining. Its main contribution is applying capability-guided expert reuse to continual image restoration; its evidence is limited to the evaluated tasks, datasets, and model configurations.

Critical strengths
  1. Preserves existing restoration capabilities.: The method freezes the pretrained model’s weights while learning new tasks. In the reported experiment, ALGNet’s score decreased only slightly, from 33.49 dB to 33.18 dB, after adding four restoration tasks to a model that initially handled only motion deblurring.

  2. Selectively reuses related restoration knowledge.: The method first identifies the restoration capabilities relevant to an input, then selects a subset of the expert modules associated with those capabilities. The authors report higher average PSNR than joint retraining across three types of foundation models.

Critical weaknesses
  1. The evaluation scope is limited, and some reported gains appear marginal.: The experiments focus primarily on deraining, desnowing, dehazing, deblurring, and denoising. It remains unclear whether the results generalize to other degradation types or real-world deployment settings. Moreover, many SSIM improvements appear to be only around 0.001–0.005. Are these gains meaningful relative to the added complexity and training effort?

  2. The novelty of the proposed bi-level routing is insufficiently established.: RestoreMore first selects relevant restoration capabilities and then uses the selected capability routers to retrieve and combine degradation experts. This hierarchy closely resembles CaRE [1] ’s router-selection followed by expert routing, which the paper cites but does not directly compare against or distinguish in sufficient detail. In image restoration, DELNet/DELNet+ [2] also study continual expansion with a dynamic expert library, selecting prior experts for a newly arriving degradation and adding new experts. CoLoRA [3] explores task-specific parameter-efficient adaptation for diverse restoration degradations, although it does not appear to use the same two-stage routing structure. The paper should clarify whether its novelty lies in the capability prototypes, capability-conditioned expert keys, stage-wise routing, or their combination, and provide direct comparisons or controlled ablations against the closest routing and adapter baselines. As written, the paper demonstrates a useful application of hierarchical routing to restoration, but does not convincingly establish that the routing principle itself is new.

[1] Lou, Meng, Yunxiang Fu, and Yizhou Yu. "Scaling Continual Learning to 300+ Tasks with Bi-Level Routing Mixture-of-Experts." arXiv preprint arXiv:2602.03473 (2026).

[2] Liu, Shihong, Kun Zuo, and Hanguang Xiao. "DELNet: Continuous all-in-one weather removal via dynamic expert library." Image and Vision Computing (2026).

[3] Park, Dongwon, Hayeon Kim, and Se Young Chun. "Contribution-based low-rank adaptation with pre-training model for real image restoration." ECCV (2024).

Questions for authors
  1. How does performance change as new degradation tasks are added sequentially? For each degradation type, could the authors report PSNR as the number of sequentially added adapters increases (for example, after adding 2, 4, 6, 8, and 16 adapters) ? This would show how performance evolves as the expert library grows.

  2. How does the router handle images with multiple simultaneous degradations? Real-world images may contain more than one degradation (for example, rain and noise or multiple mixed degradations) even if the model was trained on each degradation separately. Can the current method identify and combine the appropriate experts in such cases, and how well does it perform?

  3. Does the method remain robust to task order, and how does performance evolve during continual learning? If the method is intended for continual learning, please evaluate whether its performance is consistent across different orders in which degradation tasks are introduced. Please also report how the computational cost scales as additional tasks are introduced.

Reviewer_o3YTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RestoreMore, a continual capability expansion framework for pretrained image restoration models. It freezes the original restoration backbone and incrementally introduces residual experts for new degradation types. A bi-level routing mechanism selects relevant restoration capabilities and combines complementary experts to reuse previously learned knowledge. Experiments across different restoration architectures demonstrate improvements in both new task acquisition and retention of existing capabilities.

Critical strengths
  1. The paper addresses a meaningful problem of extending pretrained restoration models to new degradation types without retraining on historical data. The proposed framework is flexible and applicable to different restoration architectures, including CNNs, Transformers, and Mamba-based models.
  2. The experimental results are promising, showing consistent improvements over joint retraining. The ablation studies and generalization experiments further support the effectiveness of the proposed framework.
Critical weaknesses
  1. The methodological novelty is somewhat limited, as the proposed bi-level routing mechanism is closely related to existing continual learning approaches such as CaRE (ICML 2026). The technical differences are not sufficiently clarified, and direct comparisons with established continual restoration methods such as KR, ILAWR, and MINI are missing.
  2. The experimental evaluation is limited to one to four incremental tasks, leaving the scalability of the framework unclear. Moreover, RestoreMore introduces additional experts and routing modules compared with joint retraining, making it difficult to distinguish the benefits of the proposed mechanism from those of increased model capacity.
Questions for authors
  1. Could the authors clarify the key technical differences between RestoreMore and CaRE, particularly regarding the bi-level routing mechanism?
  2. How does RestoreMore compare with existing continual image restoration methods under the same backbone and training settings?
  3. How do performance and computational cost change as more restoration capabilities are incrementally introduced?
  4. Could the authors provide comparisons with simpler adapter-based expansion methods under a similar parameter budget to better isolate the contribution of the proposed routing strategy?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.