acceptodds
Withdrawn from ICLR 2027

HarnessIR: Harnessing Multimodal Foundation Models for Universal Real-World Image Restoration

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Real-world low-quality images suffer from complex mixed degradations, including but not limited to noise, blur, atmospheric effects, etc. Recent agentic methods usually model real-world image restoration (Real-IR) as a sequential tool calling problem over task-specific single-degradation restoration models: a vision-language model (VLM) is used to perceive the image content and degradations and schedule specialized restoration models into an execution sequence. This paradigm, however, is fundamentally limited because complex real-world degradations cannot be cleanly undone degradation by degradation, and the tool use of task-specific models caps the capability of the agent system. In this work, we present HarnessIR, an agentic framework for Real-IR by harnessing a multimodal foundation model (MFM) as the executor. HarnessIR consists of five stages: perception and diagnosis, on-demand tool invocation, prompt composition, execution, and verification-driven refinement. Unlike prior agentic Real-IR methods that rely on tool chains assembled from task-specific models, HarnessIR feeds the restoration requirements, the perceptual diagnosis, and the evidence into a powerful MFM that performs the restoration in a single pass, followed by the verification stage to judge whether the result warrants further processing. Under our harness, off-the-shelf MFMs handle restoration tasks remarkably well, achieving SOTA results on the widely used MiO100 synthetic benchmark. More importantly, by exploiting the strong generalization ability of MFMs, HarnessIR delivers compelling restoration quality on real-world scenes where previous agentic IR systems fail. Codes will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Real-World Image Restoration, Universal Image Restoration, Image Restoration Agent, Harness Multimodal Foundation Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_reJFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents HarnessIR, an inference-only agentic framework that uses a frozen multimodal foundation model as the image restoration executor. A vision-language model diagnoses degradations, selectively gathers evidence from auxiliary tools, and composes an image-specific restoration prompt. A verification stage assesses the output and, when necessary, requests another restoration attempt from the original low-quality input. Experiments with two executors on synthetic and real-world benchmarks report improvements over direct prompting, supported by component ablations, cost analysis, and human preference studies.

Critical strengths
  1. The framework addresses the practical problem of restoring mixed degradations while limiting unwanted content changes.
  2. Improvements are demonstrated with two executors across synthetic and real-world datasets.
Critical weaknesses
  1. The contributions of individual components are not sufficiently isolated. Table E.4 moves from a single VLM instruction to a two-stage pipeline with auxiliary tools, simultaneously changing the reasoning structure, number of calls, and available evidence. A two-stage variant without tools and individual tool ablations are needed to establish which parts of the harness provide gains beyond stronger prompting.

  2. The refinement evaluation lacks a sampling-budget control. The final system averages 1.45 executor calls per image, whereas direct prompting uses one call (Table F.7). Although the single-pass experiments establish benefits from prompt construction, the additional benefit of verification-driven refinement remains difficult to separate from generating more candidates. Repeated sampling with a fixed prompt and the same candidate selector would provide a stronger control.

  3. The evaluation does not fully establish content fidelity. Matching the ground truth’s scalar NR-IQA score does not imply that the restored image preserves the correct content. Appendix C’s human alignment and evaluator-family comparisons are reassuring, but high evaluator correlations do not establish sensitivity to small yet important textual or structural changes. Explicit OCR accuracy, landmark preservation, and localized hallucination measurements would better support the central fidelity claim.

  4. Robustness to erroneous auxiliary evidence is underexplored. OCR, depth estimation, and segmentation can be unreliable on severely degraded inputs. Although the composer may override tool evidence, the paper does not evaluate how incorrect observations affect restoration or whether the system recognizes such failures. This leaves uncertainty about whether evidence collection consistently constrains hallucination or sometimes reinforces it.

  5. Stochastic reliability remains insufficiently quantified. The reproducibility statement acknowledges stochastic decoding and provider-side changes, but the reported results do not quantify variation across repeated runs.

Questions for authors
  1. How much improvement remains over repeated sampling with a strong fixed prompt under the same executor-call budget and candidate-selection procedure?

  2. Can you provide a two-stage, tool-free baseline and separate ablations for OCR, face detection, depth estimation, and segmentation?

  3. How does HarnessIR behave when auxiliary evidence is incorrect, and can you quantify text and structural preservation independently of VLM judgments?

1 comment

We thank the reviewer for the detailed and constructive review. We have decided to withdraw this submission, but we would like to briefly answer the questions. Our answers here are qualitative, and we will consider incorporating some of these experiments in a future version.

W1 / Q2: two-stage tool-free baseline and tool ablations. We ran a two-stage variant without tools. The first VLM also describes in text what the tools would provide. It performs similarly to the single-VLM setting, because no new information enters the second call. Among the tools, the gains on the metrics of the current test sets come mainly from depth estimation and semantic segmentation. OCR and face detection have little effect on these metrics, yet they clearly improve the visual quality of text and faces, which we will show through case analysis. This is largely a limitation of the metrics: many images contain no text or faces, and if they do, these regions often occupy only a small fraction of the pixels.

W2 / Q1: sampling-budget control. We regenerated every image that triggered a redo using the first-round prompt, with the same verifier and the same budget. Most of the gain remains. It mainly comes from drawing a new candidate from the LQ input and selecting the better one, and the revised prompt contributes secondarily. Asking the verifier to write a revised instruction also serves the verifier itself: to revise, it must identify concretely what is wrong with the current result, which grounds its accept-or-redo decision in specific failures.

W3 / Q3: content fidelity. We agree that NR-IQA matching and evaluator correlation do not by themselves establish preservation of small textual or structural details. In this paper, fidelity is assessed through full-reference metrics on paired data, the VLM-based F-Score, and the user studies. Prior generative restoration and agentic IR works typically rely only on full-reference and NR-IQA metrics. Our VLM-based D/F evaluation already goes one step further by explicitly scoring content preservation, including in the no-GT setting. Dedicated measurements of text, landmark, and localized hallucination would require building a systematic evaluation framework for generative restoration, which, to our knowledge, no existing generative restoration or agentic IR work provides. We consider this beyond the scope of this paper.

W4 / Q3: auxiliary evidence. The composer is instructed to weigh each piece of evidence against the LQ image rather than transcribe it. The executor receives only the composer's text, never raw tool outputs. In our observations, when a tool occasionally errs, the VLM usually overrides the wrong evidence. Systematic evaluation is limited by the lack of depth and segmentation annotations on real-world sets.

W5: stochastic reliability. The VLM and MFM services decode stochastically, so individual runs vary. We will consider presenting this run-to-run variation in an appropriate form. What we can state with confidence is that, despite this randomness, the improvement of HarnessIR over direct MFM use is clear and consistent, and well beyond the range of run-to-run variation.

Reviewer_ZCThOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes HarnessIR, a training-free agent for real-world image restoration. A VLM first diagnoses the content and degradations of the input. It optionally calls perception tools (for example: OCR, depth, segmentation) and writes an image-specific instruction for restoration. After the editing model has run, a VLM verifier checks the result together with a few simple measurements. If the result is rejected, the pipeline starts again from the original input with a revised prompt, for up to three rounds. The method is evaluated on MiO100 and on two new real-world sets assembled from public data (Real-Paired-200, Real-NoGT-200).

Critical strengths
  • The problem is relevant and practical.
  • The method is simple and training free.
Critical weaknesses
  • The paper has limited to no novelty. It's is essence a training free promoting technique that uses a fairly standard pipeline of: asses -> instruct -> generate -> verify and correct.

  • Expanding on the novelty issues noted above:

    a) the verify-and-redo loop was widely used before: (1) "EditThinker: Unlocking Iterative Reasoning for Any Image Editor", analyses the edited image with an MLLM, rewrites the instruction and re-runs a frozen editor on the original source image. It also works training-free with GPT-4.1 or Gemini as the critic. (2) "Restore, Assess, Repeat: A Unified Framework for Iterative Image Restoration" follows also the same looping structure.

    b) Same for prompting and tool evidence, with VLM-written prompts for generative restoration going all the way back to "Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild" and "PromptFix: You Prompt and We Fix the Photo" or "SPIRE: Semantic Prompt-Driven Image Restoration"

    c) Similarly, for the paradigm itself, where "Beyond Sequential Tools: A Unified VLM Agent System for Photographic Post-Processing via Dynamic Multi-Expert Fusion" for example replaces sequential tool calls with a VLM that analyses the degradations and a single generative backbone. "GGT-100K: Generative Ground Truth for Generalizable Real-World Image Restoration" also looks very close to this work.

  • The main real-world metric appears to use the pipeline's own model. If I got this right by cross checking the tables, then the same model that diagnoses, writes the prompt and verifies and keeps the best candidate. HarnessIR is therefore selected by essentially the same judge that later will also give it a score.

  • Missing comparisons with the state-of-the-art. Some of this methods outperform the current one: "EvoIR-Agent: Self-Evolving Image Restoration Agentic System via Experience-Driven Learning" and "Restore, Assess, Repeat: A Unified Framework for Iterative Image Restoration"

  • Missing baselines and controls on real data: insufficient comparisons with Restorers with obvious competitors missing. Also no comparisons with open models, making this hard to asses at a similar compute budget.

  • The paper is overall poorly written, dense, with unusual phrasing, word ordering, naming etc that make reading the paper time consuming and difficult.

Questions for authors

Please see the weaknesses section, as my questions revolve directly on addressing the concerns raised.

1 comment

We thank the reviewer for the review. We have decided to withdraw this submission, but we would like to briefly clarify several points.

Novelty. At the level of "assess → instruct → generate → verify", the description applies to essentially every agentic system, including all agentic IR methods since AgenticIR. We do not claim this loop as our contribution. Our contribution is to change the action space of agentic IR. Prior agentic IR methods order task-specific restoration networks. HarnessIR instead writes an evidence-grounded instruction for a single frozen MFM, and its tools collect evidence rather than restore.

(a) Verify-and-redo. Verification with feedback is a general agent design, and we do not claim it as new. EditThinker trains an 8B reasoning model for general instruction-based editing and does not address restoration. RAR is a trained model whose loop updates the previous latent step by step. That is in-place iterative refinement, the strategy Table E.5 compares against, not a redo from the input.

(b) Prompts in SUPIR, PromptFix, and SPIRE. Sharing the name "prompt" does not make them the same thing; the prompts differ in both form and mechanism. These methods train or fine-tune the generator to consume prompts that mainly describe content or target quality, which elicits the generative prior. In HarnessIR the executor is frozen. The instruction mainly constrains it: it specifies what to treat, what to preserve, and where, grounded in tool evidence.

(c) Xiong et al. and GGT-100K. Xiong et al. train per-degradation LoRA experts and a fusion module on a generative backbone. It uses no evidence tools and no verification. GGT-100K is a dataset-construction pipeline. It uses category-specific instructions, offline quality control with metric filtering and manual verification, and may discard samples. HarnessIR is an inference-time method that must restore every input without human involvement. We will appropriately cite some of these works.

The "same judge" model. Using one VLM for different roles with different briefs is standard in agent design. In our AI-use statement, "separate" means that the evaluator is a separate role with its own prompt. It never sees the harness's prompts or verifier outputs. In addition, the full-reference metrics on the paired sets involve no VLM, Appendix C reports evaluators from other model families, and the human study is model-independent.

EvoIR-Agent and RAR. EvoIR-Agent is a training-free agent that still chains task-specific tools. Its experience pool is fitted on images produced by the same synthetic degradation pipeline as MiO100. It has not been peer-reviewed, and its code is not publicly available. RAR is trained on single-degradation datasets covering the same degradation types. Both are mainly developed for the synthetic MiO100 setting, whereas our focus is real-world images.

Baselines and open models. Table E.6 already includes a locally deployed open VLM (Qwen3-VL-32B). For executors, current open-weight editing models lack the capability to follow detailed restoration instructions, which leaves little room for a harness to steer them.

Reviewer_29aHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

HarnessIR is a training-free framework for real-world image restoration. A VLM diagnoses the input, selects auxiliary tools, and uses their evidence to compose a prompt for a multimodal image generator. A verifier can request revised prompts and new restorations from the original input, with up to three execution rounds. Experiments cover MiO100 and two real-world test sets. On Real-Paired-200, HarnessIR-NB2 improves PSNR from 23.94 to 26.63 dB over direct use of the same executor.

Critical strengths
  1. Using auxiliary tools as evidence for prompt composition is well motivated, particularly for preserving text and scene structure. Restarting from the original input also has empirical support. In Table E.5, it reaches 26.63 dB PSNR, compared with 26.01 dB for a single pass and 25.64 dB for refinement of the previous output.

  2. Comparisons against direct use of the same executors show gains across both executors and all three test sets. The blinded user study adds useful evidence: HarnessIR-NB2 and HarnessIR-GPT receive 48.3% and 28.1% of the votes, respectively. The detailed prompts and cost analysis also help assess the system's practical value.

Critical weaknesses
  1. Table E.4 includes detailed-request and single-VLM baselines, but moving to the full single-pass pipeline changes stage separation, the number of VLM calls, and tool use together. A two-stage variant without tools is needed to measure the contribution of tool evidence. Table E.5 leaves a related question open: would repeated sampling with a fixed prompt and the same verifier achieve similar gains at the same generation budget? The current comparison does not separate the value of revised prompts from that of additional candidates and selection.

  2. The results support better restoration quality more clearly than they establish content fidelity. The reference DF-Score evaluator uses Gemini-3.7-Flash, also used in the harness, and the prompts share many task criteria. Appendix C's human agreement and cross-model correlations are reassuring, but do not show whether closely competing methods retain their ranking. With only the degraded input as a reference, plausible but incorrect detail may escape detection. Appendix G's preference study does not show ground truth either. Targeted checks against ground truth would strengthen the fidelity claim. NR-IQA distance to ground truth cannot establish content accuracy: different content can receive the same quality score.

Questions for authors
  1. Could you compare the full pipeline with two-stage prompting without tools, and compare feedback-driven retries with fixed-prompt repeated sampling using the same verifier and generation budget? How much of the gain remains, with uncertainty estimated across repeated runs?

  2. On paired examples containing text or fine structures, how often does HarnessIR recover the correct content, and how often does it introduce errors? Please evaluate against ground truth. Also, how are D_degradation and D_clarity combined into D-Score, including cases where the former is null?

  3. Were the prompts developed on images separate from the test sets? Table B.1 explicitly assigns 20 paired examples to the mixed-degradation category. Could you report results for that subset and for the mixed-degradation portion of Real-NoGT-200 separately? This would help assess the central claim about real-world mixed degradations without averaging them together with other categories.

1 comment

We thank the reviewer for the detailed and constructive review. We have decided to withdraw this submission, but we would like to briefly answer the questions. Our answers here are qualitative, and we will consider incorporating some of these experiments in a future version.

Q1(a): two-stage prompting without tools. We ran this variant. The first VLM outputs the content and degradation analysis. It also describes in text what the tools would otherwise provide: text, face locations, and depth and semantic layout as far as language allows. The second VLM then composes the instruction. The results are similar to the single-VLM setting, because the second call receives no new information. This suggests that the benefit of the two-stage pipeline comes from the tool evidence rather than from splitting the reasoning.

Q1(b): retries vs. fixed-prompt resampling. We regenerated every image that triggered a redo using the first-round prompt, with the same verifier and the same generation budget. Most of the gain remains. The gain therefore mainly comes from drawing a new candidate from the LQ input and selecting the better one, and the revised prompt contributes secondarily. Asking the verifier to write a revised instruction also serves the verifier itself: to revise, it must identify concretely what is wrong with the current result, which grounds its accept-or-redo decision in specific failures rather than an overall impression.

W2: fidelity. The evaluator is based on the same VLM as the harness but uses an independent prompt. It never sees the harness's diagnoses, instructions, or verifier outputs. On the paired sets, the full-reference metrics, which do not involve any VLM, agree with DF-Score. Appendix C also reports evaluators from other model families and a human study. We agree that an LQ-only reference may miss details that are plausible but wrong. We also agree that matching the GT's NR-IQA score does not establish content accuracy. However, a higher NR-IQA score does not indicate a correct restoration either, and can even reward hallucinated details (Fig. 3). We cannot resolve these limitations of NR-IQA, but measuring the deviation from the GT score is an improvement over the common practice of treating a higher score as better.

Q2: text and fine structures; D-Score. Quantifying content errors in text and fine structures against the GT is still an open evaluation problem for generative restoration, because it is not well defined which deviations count as errors. To our knowledge, neither existing generative restoration methods nor agentic IR works report such a measurement. We consider it beyond the scope of this paper, though it is an important direction.

On the D-Score: we found that the evaluator prompt in the supplementary material is an earlier version, which scored D_degradation and D_clarity separately and averaged them. The reported scores come from a later version in which the evaluator directly outputs a single D-Score that considers both, so the null case does not arise. We apologize for the mismatch.

Q3: development images; mixed subsets. The prompts were not developed on the test sets. The role briefs encode general restoration requirements and the requirements of each degradation type, not properties of any particular dataset. They were checked on a small set of selected images, about ten per category. We will report the mixed-degradation subsets of both real-world sets separately.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.