acceptodds
Under review as a conference paper at ICLR 2027

IR-Agent: Experience-Driven Self-Evolving Image Restoration Agentic System

Abstract

Multimodal Large Language Model (MLLM)-driven image restoration agent demonstrates effectiveness in degradation coupling scenarios by flexibly selecting tools and determining removal orders. However, its zero-shot planning often fails without experience, necessitating severe trial-and-error overhead to achieve satisfactory outcomes. Currently, two paradigms are employed to address this issue, yet a dilemma persists: Training-based methods embed intrinsic experience into parameters, achieving high inference efficiency but lacking compatibility with new tools or degradations. In contrast, training-free methods utilize explicit experience storage for compatibility but still incur trial-and-error overhead due to naive experience. To resolve the dilemma, we propose E2IR-Agent, which first systematically formulates the experience components of a training-free image restoration agent. Subsequently, a hierarchical experience pool is constructed, which enables coarse-to-fine guidance for diverse tools and removal orders. Furthermore, a self-evolving mechanism is introduced to update the pool from scratch using accumulated records, thereby greatly improving performance and efficiency. Extensive experiments reveal that E2IR-Agent achieves a significant lead in the full reference metrics and yields a Pareto-optimal balance, which outperforms state-of-the-art methods in performance and is 61% more efficient.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Image Restoration, Agentic Systems, Experience Mechanism

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2oLnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents an image restoration agent that uses accumulated experience to guide tool selection and degradation-removal order. Its experience pool contains insight-level, coarse-grained, and fine-grained guidance, updated from restoration outcomes, metric-based rankings, and degradation-pattern profiles. Experiments evaluate restoration quality, tool invocation counts, and adaptation to changes in tools, degradations, and data sources.

Critical strengths
  1. The improvements in PSNR and SSIM under Setting I are substantial, and the real-world evaluation also shows useful gains in quality and invocation count. These results suggest that the experience pool improves actual tool use rather than only the agent’s planning descriptions.
  2. Appendix C explicitly separates the experience-evolution and evaluation datasets. The paper also provides experiments on experience granularity and evolution frequency, which help explain when the mechanism is useful.
Critical weaknesses
  1. The efficiency comparison does not account for the cost of acquiring experience. Section 3.4 evaluates all candidate tools or removal orders to construct experience; for multiple degradations, this involves ∣D∣!|D|! trajectories. The reported 61% reduction follows from the invocation counts in Table 2, from 10.89 to 4.23. However, different tools have different runtimes, and experience construction, retrieval, and MLLM calls introduce additional costs. The partial API accounting in Appendix C does not establish how many requests are needed to recover the initial cost or whether the full system is faster overall.
  2. The justification for decoupling tool selection and removal order is limited, and the adaptation results are more mixed than the discussion suggests. Appendix A.1 examines three two-degradation settings and reports 80% agreement with the modal order. This also shows that changing tools can change the preferred order, but the quality loss from ignoring this interaction is not measured. In Appendix B.1, Table 4, S4 reaches 18.12 dB compared with 20.66 dB for S5, while perception-mode invocations increase from 3.41 at S3 to 5.42 at S4. These results demonstrate some adaptation, but do not clearly establish that the evolved system approaches the reference configuration in both quality and efficiency.
Questions for authors
  1. Could you report total runtime or GPU/API cost including experience generation, updates, retrieval, and inference, together with the number of requests needed to amortize experience construction?
  2. In a small setting where joint search is feasible, how much quality is lost by decoupling tool selection and removal order? Do the order changes observed in Appendix A.1 lead to meaningful restoration differences?
  3. Do the Fidelity and Perception columns in Table 1 correspond to separate restored outputs? Given the mixed LPIPS and no-reference results in Setting II, which quality–efficiency trade-off should be regarded as the main supported advantage?
Reviewer_kLsfOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes E²IR-Agent, an experience-driven self-evolving framework for image restoration agents, aiming to improve restoration quality and inference efficiency in scenarios involving multiple coupled degradations. The authors formulate restoration experience along three dimensions: tool selection, degradation removal order, and visual quality preference. Based on this formulation, they introduce a hierarchical experience pool that organizes restoration strategies at different granularities, enabling degradation-pattern-aware planning. Furthermore, a self-evolving mechanism is developed to continuously update the experience pool through quality-based ranking and pattern profile learning. Experiments on synthetic and real-world datasets demonstrate improvements in restoration quality and reduced tool invocations compared with existing image restoration agents.

Critical strengths

1. Well-motivated problem with practical significance.

The paper addresses an important limitation of existing image restoration agents, namely the substantial trial-and-error overhead associated with experience-free planning. Leveraging reusable experience to improve restoration quality and inference efficiency is well motivated and practically relevant, particularly for complex scenarios involving multiple degradations.

2. A systematic and well-designed experience management framework.

The proposed framework provides a structured formulation of restoration experience by considering tool selection, degradation removal order, and visual quality preference. The hierarchical experience pool further distinguishes between degradation types and fine-grained degradation patterns, enabling more targeted experience retrieval. The integration of experience acquisition, ranking, retrieval, and evolution forms a coherent and reasonably effective framework.

3. Promising experimental results and useful ablation studies.

The experiments demonstrate competitive restoration performance across synthetic and real-world degradation scenarios. The improvements in full-reference metrics, particularly PSNR and SSIM, are encouraging. Moreover, the reduction in tool invocations suggests that the proposed experience mechanism can improve inference efficiency. The ablation studies further support the effectiveness of different experience granularities and the evolving mechanism.

Critical weaknesses

1. Limited analysis of the computational overhead of experience construction.

Although the proposed framework improves inference efficiency by reducing tool invocations, constructing the experience pool involves exhaustive exploration of restoration strategies and additional LLM-based processing. The computational cost of this offline stage is not comprehensively evaluated. Reporting the total experience construction cost and discussing the trade-off between offline overhead and inference savings would provide a more complete understanding of the framework's overall efficiency.

2. Insufficient comparisons with simpler experience retrieval strategies.

While the ablation studies demonstrate the benefits of different experience granularities, the paper lacks direct comparisons with simpler experience management alternatives, such as flat memory retrieval or embedding-based nearest-neighbor retrieval. Such comparisons would help clarify the necessity of the proposed hierarchical experience pool and better isolate the contribution of its design.

3. Limited analysis of the stability of self-evolution.

The paper demonstrates performance improvements through iterative experience updates, but the results are not consistently monotonic across evolution iterations. In particular, performance can decrease after additional evolution rounds. Further analysis of the stability, convergence behavior, and potential limitations of experience evolution would help substantiate the claims regarding continuous adaptability.

Questions for authors

Please see weaknesses.

Reviewer_rKMeOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces E2IR-Agent, a training-free framework for image restoration that improves tool selection and degradation removal ordering through accumulated experience. It organizes experience into three levels: Insight, Coarse, and Fine, providing guidance conditioned on restoration preferences, degradation types, and visual patterns. An experience acquisition process evaluates candidate restoration trajectories using image quality metrics, while statistical ranking and multi-agent pattern learning update the experience pool. During inference, the agent integrates perception, planning, execution, reflection, and rollback.

Critical strengths
  1. When there is experience, this method can indeed reduce the number of tool calls significantly.
  2. The paper incorporates tool selection, the order of degradation removal, and restoration quality preferences into its empirical modeling, while considering that different degradation patterns may require different execution strategies. These factors directly affect the results of handling compound degradation, so the task is well motivated.
Critical weaknesses
  1. The evaluation omits strong baselines such as SUPIR [1] and unified restoration methods discussed in the Related Work, including UniRestore and Defusion. Limited coverage of degradation types does not itself establish limited generalization or extensibility.
  2. The method proposed in the paper does not show obvious advantages compared with other agent methods, especially when human preference is Perception in Setting II, the method proposed in the paper is obviously inferior to TIR-agent.
  3. Experience construction and final evaluation rely on largely overlapping IQA metrics. Improvements on these metrics are therefore expected, but do not independently establish better alignment with users’ subjective preferences.
  4. Experience ranking and evolution depend heavily on IQA, whose reliability for restoration assessment remains limited [2]. In particular, Fidelity-oriented updates require reference images, which means that experience can only be acquired from synthetic data. After deployment, existing experience can only be reused, not updated. When new images without ground truth arrive, it seems impossible to continue updating the experience.
  5. The ablations do not sufficiently establish the complementarity of Insight, Coarse, and Fine experience or the necessity of retaining all three.
  6. Hierarchical experience pools and experience evolution have already been explored in general-purpose agents. The paper should clarify which designs are specifically motivated by the distinctive requirements of image restoration.
  7. The motivation for agent-driven pattern profile learning and multi-agent debate remains unclear.
  8. Unclear benefits over a fixed, highest-ranked strategy. The full agent should be compared with a fixed toolchain selected for each degradation combination and preference using experience or validation data.

[1] Yu F, Gu J, Li Z, et al. Scaling up to excellence: Practicing model scaling for photo-realistic image restoration in the wild[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024: 25669-25680.

[2] Hu J, Yu F, You Z, et al. Position: Evaluation of visual processing should be human-centered, not metric-centered[J]. arXiv preprint arXiv:2603.00643, 2026.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.