- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents HarnessIR, an inference-only agentic framework that uses a frozen multimodal foundation model as the image restoration executor. A vision-language model diagnoses degradations, selectively gathers evidence from auxiliary tools, and composes an image-specific restoration prompt. A verification stage assesses the output and, when necessary, requests another restoration attempt from the original low-quality input. Experiments with two executors on synthetic and real-world benchmarks report improvements over direct prompting, supported by component ablations, cost analysis, and human preference studies.
Critical strengths
- The framework addresses the practical problem of restoring mixed degradations while limiting unwanted content changes.
- Improvements are demonstrated with two executors across synthetic and real-world datasets.
Critical weaknesses
-
The contributions of individual components are not sufficiently isolated. Table E.4 moves from a single VLM instruction to a two-stage pipeline with auxiliary tools, simultaneously changing the reasoning structure, number of calls, and available evidence. A two-stage variant without tools and individual tool ablations are needed to establish which parts of the harness provide gains beyond stronger prompting.
-
The refinement evaluation lacks a sampling-budget control. The final system averages 1.45 executor calls per image, whereas direct prompting uses one call (Table F.7). Although the single-pass experiments establish benefits from prompt construction, the additional benefit of verification-driven refinement remains difficult to separate from generating more candidates. Repeated sampling with a fixed prompt and the same candidate selector would provide a stronger control.
-
The evaluation does not fully establish content fidelity. Matching the ground truth’s scalar NR-IQA score does not imply that the restored image preserves the correct content. Appendix C’s human alignment and evaluator-family comparisons are reassuring, but high evaluator correlations do not establish sensitivity to small yet important textual or structural changes. Explicit OCR accuracy, landmark preservation, and localized hallucination measurements would better support the central fidelity claim.
-
Robustness to erroneous auxiliary evidence is underexplored. OCR, depth estimation, and segmentation can be unreliable on severely degraded inputs. Although the composer may override tool evidence, the paper does not evaluate how incorrect observations affect restoration or whether the system recognizes such failures. This leaves uncertainty about whether evidence collection consistently constrains hallucination or sometimes reinforces it.
-
Stochastic reliability remains insufficiently quantified. The reproducibility statement acknowledges stochastic decoding and provider-side changes, but the reported results do not quantify variation across repeated runs.
Questions for authors
-
How much improvement remains over repeated sampling with a strong fixed prompt under the same executor-call budget and candidate-selection procedure?
-
Can you provide a two-stage, tool-free baseline and separate ablations for OCR, face detection, depth estimation, and segmentation?
-
How does HarnessIR behave when auxiliary evidence is incorrect, and can you quantify text and structural preservation independently of VLM judgments?
We thank the reviewer for the detailed and constructive review. We have decided to withdraw this submission, but we would like to briefly answer the questions. Our answers here are qualitative, and we will consider incorporating some of these experiments in a future version.
W1 / Q2: two-stage tool-free baseline and tool ablations. We ran a two-stage variant without tools. The first VLM also describes in text what the tools would provide. It performs similarly to the single-VLM setting, because no new information enters the second call. Among the tools, the gains on the metrics of the current test sets come mainly from depth estimation and semantic segmentation. OCR and face detection have little effect on these metrics, yet they clearly improve the visual quality of text and faces, which we will show through case analysis. This is largely a limitation of the metrics: many images contain no text or faces, and if they do, these regions often occupy only a small fraction of the pixels.
W2 / Q1: sampling-budget control. We regenerated every image that triggered a redo using the first-round prompt, with the same verifier and the same budget. Most of the gain remains. It mainly comes from drawing a new candidate from the LQ input and selecting the better one, and the revised prompt contributes secondarily. Asking the verifier to write a revised instruction also serves the verifier itself: to revise, it must identify concretely what is wrong with the current result, which grounds its accept-or-redo decision in specific failures.
W3 / Q3: content fidelity. We agree that NR-IQA matching and evaluator correlation do not by themselves establish preservation of small textual or structural details. In this paper, fidelity is assessed through full-reference metrics on paired data, the VLM-based F-Score, and the user studies. Prior generative restoration and agentic IR works typically rely only on full-reference and NR-IQA metrics. Our VLM-based D/F evaluation already goes one step further by explicitly scoring content preservation, including in the no-GT setting. Dedicated measurements of text, landmark, and localized hallucination would require building a systematic evaluation framework for generative restoration, which, to our knowledge, no existing generative restoration or agentic IR work provides. We consider this beyond the scope of this paper.
W4 / Q3: auxiliary evidence. The composer is instructed to weigh each piece of evidence against the LQ image rather than transcribe it. The executor receives only the composer's text, never raw tool outputs. In our observations, when a tool occasionally errs, the VLM usually overrides the wrong evidence. Systematic evaluation is limited by the lack of depth and segmentation annotations on real-world sets.
W5: stochastic reliability. The VLM and MFM services decode stochastically, so individual runs vary. We will consider presenting this run-to-run variation in an appropriate form. What we can state with confidence is that, despite this randomness, the improvement of HarnessIR over direct MFM use is clear and consistent, and well beyond the range of run-to-run variation.