acceptodds
Withdrawn from ICLR 2027

Towards Autonomous Image Restoration with Fine-grained Agentic Control and History-grounded Restoration

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Most generative image restoration methods perform restoration in a single round, without explicitly reassessing residual degradations, newly introduced artifacts, or reconstruction errors in the restored results. Recent agentic approaches introduce iterative assessment and replanning, but their control largely remains at the level of tool selection, with visual history primarily used for decision-making rather than restoration execution. To build an autonomous restoration framework, we propose AutoRestore, which tightly integrates VLM-based assessment with generative restoration through fine-grained agentic control and history-grounded restoration. At each round, AutoRestore uses a VLM to identify restoration requirements and generate region-specific instructions. These instructions are further translated into spatially varying uncertainty and uncertainty-driven modulation, enabling fine-grained restoration control. Meanwhile, we introduce history-grounded restoration, which preserves reference representations from the original input and intermediate results as persistent visual memory across rounds. Extensive experiments on multiple image restoration benchmarks demonstrate that AutoRestore achieves superior restoration quality and a better trade-off between fidelity and perceptual quality.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Image Restoration, Generative Models, Uncertainty Modeling, VLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PNuNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors proposed AutoRestore, which focuses on alleviating the problems in iterative image restoration. The method combines VLM-based assessment with instruction-guided spatial modulation and history-grounded restoration. The proposed regional instructions and historical KV cache enable targeted refinement while preserving visual evidence across iterations Comprehensive evaluation, including real-world and synthetic benchmarks, ablation studies, efficiency comparisons, and a user study, shows competitive fidelity-perceptual quality trade-offs and better performance compared to existing related methods.

Critical strengths
  1. The proposed spatial modulation links semantic assessment and the restoration process. It allows VLM to influence spatially varying restoration behavior. This is a useful design for handling spatially variant degradation and restoration requirements within an image.
  2. The historical cache alleviates a practical limitation in iterative restoration.
  3. Comprehensive evaluation experiments are conducted, covering real-world and synthetic benchmarks, separate degradation settings, full-reference and no-reference metrics, and a user study.
Critical weaknesses
  1. The current experiments do not sufficiently isolate the contribution of instruction-guided spatial control. Table 1(b) evaluates residual-scale prediction, but does not establish whether local instructions produce spatially accurate corrections or preserve untargeted regions. Comparisons with scene descriptions, global restoration instructions, and spatially uniform modulation are missing.

  2. The ablation studies of historical memory do not separate original-input anchoring from accumulated intermediate evidence. The improvement may largely arise from retaining the original image. Without an original-plus-current-reference baseline, the necessity of storing all intermediate results remains unclear. The paper also does not examine whether incorrect details in cached results can reinforce errors in later rounds.

  3. The contribution of adaptive iteration and stopping is not sufficiently isolated. Table 1(a) shows that the full model maintains similar PSNR and slightly higher LPIPS than the single-round variants while improving several no-reference metrics. Although Fig. 4 provides qualitative examples, comparisons with fixed-round and fixed-instruction variants are needed to distinguish the benefits of adaptive decisions from those of additional restoration rounds. Fig. 12 provides a pairwise comparison prompt, but the maximum iteration count and the rules for translating its judgments into continuation, termination, and output selection remain insufficiently specified.

  4. The impact of the training–inference instruction gap is unquantified. Although the intermediate results used for training are generated with LQ-only instructions, the instructions used for subsequent optimization remain GT-assisted. The effect of this discrepancy on uncertainty estimation and restoration quality is not analyzed. It is unclear whether the validation instructions in Table 1(b) are GT-assisted or LQ-only. Evaluating performance with LQ-only instructions would help clarify the impact of this gap.

Questions for authors

Several experiments and evaluations would help to improve this paper, the authors could have a small-scale validation.

  1. Evaluate spatial controllability by varying regional instructions, and observe the changes on performance. It would help isolate the contribution of instruction-guided spatial modulation.
  2. Compare the impact of current-only, original-plus-current, and full-history conditioning. This would clarify whether retaining intermediate results provides benefits beyond preserving the original input.
  3. A fixed-instruction variant, with per-round quality and runtime measurements, would help distinguish the contributions of instruction updates and stopping decisions.
3 comments
AuthorsOpenReview ↗

[Part 1] Official Comment by Authors

We thank the reviewer for the thoughtful suggestions on isolating the contributions of our design choices. Below, we clarify the roles of spatial instructions, reference caching, and adaptive iteration, and describe additional experiments to examine spatial controllability, historical conditioning, and the training–inference instruction gap.

W.1 Validation of Instruction-Guided Spatial Control

We thank the reviewer for highlighting this limitation. We agree that evaluating residual-scale prediction in Table 1(b) alone does not sufficiently establish whether regional instructions produce spatially accurate corrections while preserving untargeted regions.

In the revised manuscript, we will add comparisons using scene descriptions, global restoration instructions, regional restoration instructions, and spatially uniform modulation. For a fixed degraded input, we will vary the regional instructions and examine the corresponding uncertainty maps and restoration results. We will evaluate changes in both targeted and untargeted regions to better isolate the contribution of instruction-guided spatial control.

W.2 Original-Input Anchoring and Reference Caching

We thank the reviewer for raising this point. Our original design retains the original LQ input together with intermediate restoration results to preserve reference information across rounds. However, we observe that subsequent rounds typically introduce smaller changes than the first restoration round, and retaining only the original LQ input and the most recent restoration result achieves comparable quality to retaining the full history. We will therefore adopt this simpler reference strategy and revise the relevant sections to distinguish the role of original-input anchoring from that of accumulated intermediate references. In the manuscript, Table 1(a) shows the importance of retaining reference information for fidelity preservation during multi-round restoration. In addition, we benchmarked the inference time and GPU memory usage of full joint attention, reference caching, and LQ-only caching. For subsequent rounds, the caching mechanism reduces inference time from 6.24 s to 2.50 s, approximately a 60% reduction compared with full joint attention.

VariantRound i=1i=1: Time (s)Round i=1i=1: Memory (GiB)Round i>1i>1: Time (s)Round i>1i>1: Memory (GiB)
Full attention3.8711.496.2412.67
+ historical reference cache2.2512.652.5013.83

We will revise the relevant sections to distinguish original-input anchoring from accumulated historical evidence and avoid implying that retaining all intermediate results is necessary. We will also supplement the conditioning comparisons described in Q2.

W.3 Instruction Updates and Adaptive Stopping

We thank the reviewer for identifying this important issue. We have examined fixed-instruction variants and observed that updating instructions can help correct errors in scenes containing complex details, such as text.

For example, when the initial degradation is severe, the VLM may be unable to identify the exact text and provide only a generic instruction such as “recover the text.” Reusing this instruction in subsequent rounds can leave the text incorrectly restored. When instructions are regenerated, the VLM can jointly examine the original LQ input and the intermediate restoration result. In the examples we examined, this enabled it to provide an updated instruction specifying the correct text, which subsequently guided the restoration model to correct the output. We will include these visual comparisons in the revised manuscript.

Our stopping strategy combines the VLM’s adaptive decisions with explicit iteration bounds: a minimum of two and a maximum of four restoration rounds. The observed iteration distributions are reported below.

Dataset2 rounds3 rounds4 roundsAverage rounds
NTIRE2020-Track160%31%9%2.49
DIV2K-Val65%29%6%2.40

These datasets share the same GT images but use real-world and synthetic degradations, respectively. Their iteration distributions and average numbers of rounds are similar. We also observe that severely degraded samples do not consistently require more rounds. We believe this may be because severe degradation can erase substantial image detail, leaving limited information that can be reliably recovered.

We will include these results and clarify how the VLM’s judgments determine continuation, termination, and output selection. We will also supplement the controlled per-round comparisons described in Q3.

AuthorsOpenReview ↗

[Part 2] Official Comment by Authors

W.4 Training–Inference Instruction Gap

We use GT-assisted restoration instructions during training to teach the restoration model the correct correspondence between image semantics and restoration actions. This is consistent with existing image restoration methods that use LLaVA to generate descriptions from GT images when preparing training data.

To mitigate the instruction gap, we discourage the VLM from describing overly specific details that would be difficult to infer from the LQ image alone. During inference, we actively encourage the VLM to make reasonable inferences about the underlying clean image content, guided by the visual cues and scene context available in the LQ image.

GT images are not provided to the VLM or restoration model during inference. We will explicitly clarify the instruction source used for validation in Table 1(b). We acknowledge that the above design rationale alone does not quantify the remaining gap, and we will add comparisons involving LQ-only training instructions to assess its effects on both uncertainty estimation and final restoration quality.

The authors are recommended to conduct small experiments and report the corresponding results of global and regional instructions on the same locally degraded images.

Reviewer_N2RsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose AutoRestore, which utilizes frozen VLM for flow-based image restoration task. The restoration task is formulated into multiple rounds of assessment and decision-making, and the VLM is used for fine-grained, region-specific agentic control over the image. The restorer also retains historic representations from the intermediate restoration rounds, allowing later restoration to attend to accumulated visual information. The authors report results on real and synthetic restoration benchmarks, showing some benefits of their proposed scheme.

Critical strengths
  • The proposed method shows competitive results across several benchmarks, when compared to other single-round and agentic restoration schemes.
  • The authors performed ablation experiments for the proposed components, showing the effectiveness of their proposed restoration scheme.
  • Paper is well-written and easy to follow, with practical implications.
Critical weaknesses
  • In Table 1(a), performance gains of each component are minor and indistinguishable when compared to the baseline, with lower numbers for the metrics of PSNR and LPIPS.
  • Also, given the baseline results on NTIRE2020-Track1 in Table 1(a), it already surpasses other baselines on multiple metrics. This implies that the performance mainly originates from the backbone and data, not from the proposed components.
  • No multi-seed ablation is a critical weakness, since diffusion restorers are known to vary noticeably with the seed randomness.
  • Although the authors claimed that their scheme performs fine-grained, region-specific control, this is not validated on test images with locally degraded inputs.
  • Regarding the autonomous closed loop, the authors do not report the average or maximum number of rounds.
  • The synthetic benchmarks use blur, noise and JPEG degradations that overlap the proposed works' own training degradations, while there are other methods mainly trained on Real-ESRGAN-style datasets.
Questions for authors

Please refer to the weaknesses section.

2 comments
AuthorsOpenReview ↗

[Part 1] Official Comment by Authors

We thank the reviewer for the detailed feedback and constructive suggestions. Below, we discuss the contributions of our components relative to the backbone and training data, address random-seed robustness and adaptive iteration, and present additional comparisons under Real-ESRGAN-style degradations.

W.1 & W.2 Contributions of the Proposed Components

We thank the reviewer for raising the question of how much of the improvement comes from the proposed components. Our primary comparisons include DreamClear and LucidFlux, which also construct their own large-scale, high-quality training datasets. Specifically, DreamClear generates one million training images, while LucidFlux constructs 1.36 million training pairs from approximately 342K clean images. Meanwhile, our restoration backbone, FLUX.2 Klein 4B, is also substantially smaller than the FLUX.1-dev 12B backbone used by LucidFlux. The component ablation in Table 1(a) shows clear improvements in perceptual quality: CLIPIQA+ increases by more than 0.2, and A-FINE-NR decreases by approximately 0.06. These results provide evidence of the proposed components’ contribution beyond the backbone and training data.

Another important benefit of our multi-round restoration is its ability to correct errors remaining in the first-round output. Some corrections, such as correcting incorrectly restored text (Fig. 4 for example), may not be fully reflected in no-reference image-quality metrics. We will improve the manuscript and provide additional visual comparisons to illustrate these corrections and make the benefits of the proposed method clearer.

W.3 Robustness to Random Seeds

We thank the reviewer for pointing out the need for multi-seed evaluation. Averaging results over hundreds of validation images can reduce the influence of per-image sampling fluctuations, but it does not replace an explicit assessment across random seeds. To strengthen the experimental rigor, we will add multi-seed ablation experiments and report the mean performance and variability.

W.4 VLM-Based Stopping Strategy

We apologize for omitting these important implementation details and experimental results. Our stopping strategy combines the VLM’s adaptive decisions with explicit iteration bounds: a minimum of two and a maximum of four restoration rounds.

The table below reports the distribution and average number of restoration rounds across the two datasets.

Dataset2 rounds3 rounds4 roundsAverage rounds
NTIRE2020-Track160%31%9%2.49
DIV2K-Val65%29%6%2.40

NTIRE2020-Track1 and DIV2K-Val share the same GT images but differ in their degradation settings: real-world and synthetic degradations, respectively. Despite this difference, their iteration distributions and average numbers of rounds are similar.

We also observe that severely degraded samples do not consistently require more restoration rounds. We believe this may be because severe degradation can erase substantial image detail, leaving limited information that can be reliably recovered. Additional rounds may therefore offer little benefit, so greater degradation severity does not necessarily imply a need for more iterations. We will include these results and clarify the iteration bounds and stopping protocol in the manuscript.

AuthorsOpenReview ↗

[Part 2] Official Comment by Authors

W.5 Evaluation on Real-ESRGAN-Style Degradations

We thank the reviewer for raising this concern. We have added an evaluation on DIV2K-Val with complex degradations generated using a Real-ESRGAN-style pipeline. In this setting, we compare AutoRestore with SUPIR, DreamClear, FaithDiff, and LucidFlux, which only use Real-ESRGAN-style degradations during training.

MethodPSNR ↑SSIM ↑LPIPS ↓DISTS ↓TOPIQ-FR ↑CLIPIQA+ ↑MUSIQ ↑NIQE ↓A-FINE-NR ↓
SUPIR21.660.53770.43570.20110.33850.626062.824.143-0.822
DreamClear20.860.52050.34360.1562‾\underline{0.1562}0.38610.693868.653.550‾\underline{3.550}-0.844
FaithDiff21.62‾\underline{21.62}0.5289‾\underline{0.5289}0.34360.17600.40800.688171.564.124-0.933
LucidFlux18.360.48060.40540.19400.35420.756674.743.921-0.993
AutoRestore21.080.52500.3497‾\underline{0.3497}0.15160.4063‾\underline{0.4063}0.7326‾\underline{0.7326}73.07‾\underline{73.07}3.277−0.976‾\underline{-0.976}

Best results are bolded, and second-best results are underlined among the methods listed. Tied results receive the same formatting. Higher is better for metrics marked with ↑, and lower is better for those marked with ↓.

As shown in the table, AutoRestore achieves the best DISTS and NIQE and the second-best results on several other perceptual metrics among these methods. These results demonstrate that our method remains competitive when evaluated on Real-ESRGAN-style degradations against methods trained with this degradation framework, providing evidence that its advantages extend beyond the single-degradation benchmarks.

We will include this comparison and describe the degradation-generation protocol in the revised manuscript.

Reviewer_5uiAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AutoRestore is an autonomous image restoration framework that performs iterative assess–restore–reassess refinement. A frozen Qwen3.8-27B VLM generates scene descriptions, region-level restoration instructions, and quality assessments. The restoration system contains two FLUX.2-Klein-based branches:

  • An uncertainty predictor FuF_u, which estimates the spatially varying uncertainty of the latent restoration residual.
  • A restoration model FrF_r, which uses the predicted uncertainty to modulate the restoration process and introduces historical KV caches of visual features from previous rounds.

The model is trained in multiple stages. First, FuF_u is trained on LQ–HQ image pairs and instruction using a Gaussian negative log-likelihood objective. Then FuF_u is frozen while FrF_r is trained. Later stages use offline-generated intermediate restorations and VLM-generated refinement instructions to adapt the model to multi-round restoration. During inference, the VLM generates restoration instructions from the degraded image in the first round and from the current restoration result in subsequent rounds. A historical KV cache preserves visual features from previous rounds, allowing later restoration steps to access earlier visual evidence.

Critical strengths
  1. The paper is generally easy to follow. The motivation, overall framework, and experimental results are presented in a clear and coherent manner.

  2. The work demonstrates substantial experimental effort. This includes the curation and construction of the training data, the design of the multi-stage training pipeline, and comparisons under multiple settings across both real-world and synthetic benchmarks.

  3. The method contains several reasonable and interesting design choices. In particular, the training of the uncertainty predictor, the modulation of timestep embeddings using the predicted uncertainty map, and the use of a KV-cache-based memory mechanism to retain information from previous restoration results are well motivated and technically appealing.

Critical weaknesses
  1. Mismatch between training-time and inference-time restoration instructions.
    During training, the VLM generates restoration instructions after observing both the degraded LQ image and the corresponding GT image. During inference, however, the VLM only observes the LQ image. Since the VLM itself is frozen and is not trained for this task, it is unclear whether the off-the-shelf VLM can generate sufficiently accurate diagnostic instructions under the inference-time setting. If the instructions generated at inference are less accurate than those generated during training, how much does this train–test mismatch affect the final restoration performance? The paper does not provide an experiment that directly quantifies this effect.

  2. Necessity of instruction conditioning for uncertainty estimation.
    Existing image restoration methods have explored estimating degradation severity or spatial degradation maps directly from the degraded image. This raises the question of whether the restoration instruction is necessary as an input to the uncertainty predictor. Table 1(b) briefly reports an ablation comparing uncertainty estimation with and without instructions, but the experimental protocol is not sufficiently specified. In particular, it is unclear whether the comparison concerns only evaluation or involves retraining the uncertainty predictor.

    If the models are retrained, the source of the training instructions becomes important. As described above, training-time instructions are generated by a VLM that has access to both the LQ and GT images. Therefore, these instructions may contain information derived from the target image, creating a potential information leakage or privileged-information issue. It would be helpful to compare the following settings under the same training and evaluation protocol:

    • training without restoration instructions;
    • training with instructions generated from the LQ image alone;
    • training with instructions generated from both LQ and GT.

    The paper should also report how much the uncertainty estimation error affects the final restoration quality. Improvements in NLL, SRCC, or AUPRC do not necessarily imply improvements in the final restored images.

  3. Significant fidelity degradation in the multi-round ablation.
    In the ablation study, multi-round inference substantially decreases fidelity-related metrics, while improving some perceptual metrics. Could the authors provide a more detailed analysis of the cause? In particular, is this degradation related to the mismatch between training-time and inference-time instructions? If so, how would the results change if the model were trained using instructions generated from LQ images alone, matching the inference setting?

    It would also be useful to include per-round performance curves and to analyze whether the degradation comes primarily from structural drift, hallucinated textures, color changes, or errors in the VLM-generated instructions.

  4. Reliability of the stopping decision.
    The method relies on the VLM to decide whether another restoration round is necessary. However, the paper does not report the accuracy or reliability of this stopping decision. How many restoration rounds are typically used for the reported test results? What is the distribution of the number of rounds across different datasets and degradation types? It would also be useful to compare the VLM-based stopping strategy with fixed numbers of restoration rounds and to report whether additional rounds improve or degrade fidelity and perceptual quality.

Possible Typographical or Descriptive Inconsistency

There appears to be a possible inconsistency in the description of how restoration instructions are generated in the appendix. Around lines 939 and 941, one statement seems to suggest that the instructions used during training are generated from the LQ image, whereas another statement indicates that the VLM observes both the LQ image and the corresponding GT image during training. Could the authors clarify the exact source of the instructions used in each training stage and distinguish it explicitly from the instruction-generation process used during inference?

Questions for authors

Please refer to above weakness. Every weakness includes one paricular question.

2 comments
AuthorsOpenReview ↗

[Part 1] Official Comment by Authors

We thank the reviewer for the careful assessment and constructive suggestions. Below, we clarify the instruction-generation and uncertainty-estimation protocols and several missing details in the manuscript, and explain the multi-round ablation and stopping strategy.

W.1 Training–Inference Instruction Mismatch

We use GT-assisted restoration instructions during training to teach the restoration model the correct correspondence between image semantics and restoration actions. This is consistent with existing image restoration methods that use LLaVA to generate descriptions from GT images when preparing training data.

To mitigate the instruction gap, we discourage the VLM from describing overly specific details that would be difficult to infer from the LQ image alone. During inference, we actively encourage the VLM to make reasonable inferences about the underlying clean image content, guided by the visual cues and scene context available in the LQ image. These choices are intended to improve consistency between training and inference. We acknowledge that this rationale alone does not quantify the remaining gap; the additional LQ-only training comparison described in W2 will help assess its practical impact.

W.2 Instruction Conditioning for Uncertainty Estimation

We thank the reviewer for pointing out the insufficient description of this ablation. Table 1(b) compares two separately trained uncertainty predictors: one trained with restoration instructions generated from both LQ and GT images, and another trained without restoration instructions. We will explicitly clarify this training protocol in the manuscript. At inference, GT images and GT-derived instructions are not provided.

Following the reviewer’s suggestion, we will add a third setting trained with instructions generated from LQ images alone, using a matched training and evaluation protocol. We will also evaluate how these alternatives affect the final restoration quality, alongside their uncertainty estimation performance, to clarify whether improvements in uncertainty prediction translate into better restored images.

W.3 Fidelity Degradation in the Multi-Round Ablation

We apologize for the ambiguity in the description of this experiment. In the ablation table, the multi-round variant receives only the latest restoration result at each subsequent round. The original LQ image and earlier restoration results are not retained as historical references.

Although this configuration can improve perceptual quality, repeatedly restoring the latest output without the original input or historical references can accumulate errors and cause substantial drift, leading to significant fidelity degradation. This variant was included for comparison with the configuration using our historical reference cache, highlighting the importance of retaining reference information for fidelity preservation during multi-round restoration. We will clarify this distinction in the table and its accompanying discussion.

AuthorsOpenReview ↗

[Part 2] Official Comment by Authors

W.4 VLM-Based Stopping Strategy

We apologize for missing these important implementation details and experimental results. Our stopping strategy combines the VLM’s adaptive decisions with explicit iteration bounds: a minimum of two and a maximum of four restoration rounds.

The table below reports the distribution and average number of restoration rounds across datasets and degradation settings.

Dataset2 rounds3 rounds4 roundsAverage rounds
NTIRE2020-Track160%31%9%2.49
DIV2K-Val65%29%6%2.40

NTIRE2020-Track1 and DIV2K-Val share the same GT images but differ in their degradation settings: real-world and synthetic degradations, respectively. Despite this difference, their iteration distributions and average numbers of rounds are similar. We also observe that severely degraded samples do not consistently require more restoration rounds. We believe this may be because severe degradation can erase substantial image detail, leaving limited information that can be reliably recovered. Additional rounds may therefore offer little benefit, so greater degradation severity does not necessarily imply a need for more iterations. We will include these results and clarify the iteration bounds and stopping protocol in the manuscript.

W.5 Instruction Generation Across Training Stages

We apologize for the unclear description around lines 939–941. The two instruction-generation strategies refer to different parts of the training-data preparation process.

For first-stage training, we use GT-assisted instructions for supervised optimization. During inference, instructions are generated from the LQ image without access to GT.

When preparing subsequent-stage training data, we first generate instructions from the LQ image alone and use them to produce an offline first-round restoration, simulating the actual inference process. We then provide the GT image together with this restoration result to the VLM to generate the instruction used for second-stage training. Since the initial LQ-only instruction may contain errors, this procedure produces training examples with corrective instructions for imperfect first-round restorations.

Thus, the LQ-only instructions are used to generate intermediate training inputs, while the GT-assisted instructions provide supervision for subsequent restoration. We will revise the description to distinguish these roles and their relationship to inference explicitly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.