acceptodds
Under review as a conference paper at ICLR 2027

Fixing the Last Mile: Failure-Aware Image Restoration with Region-Specific Guidance

Abstract

Recent advances in image restoration have greatly improved global image fidelity, producing visually plausible results across most regions of an image. Nevertheless, even high-quality restorations often exhibit localized failures, including artifacts in semantically important regions such as faces and text, structural inconsistencies, and texture mismatches. Although these errors typically affect only a small portion of the image, they often dominate the overall perceptual quality of the result. In this work, we propose **Failure-Aware Image Restoration (FAIR)** to address the **“last-mile”** problem in image restoration. Rather than applying one uniform generative prior over the whole image, we condition a single-pass restoration model on both **where** local failures are likely to occur and **what** corrective guidance each region requires. This region-bound conditioning enables fine-grained and controllable restoration that improves semantic accuracy and stylistic consistency in critical regions while maintaining faithful, high-quality restoration elsewhere. Our method complements existing restoration pipelines by shifting the emphasis from global fidelity to local perceptual acceptability. Extensive experiments show that the proposed approach resolves common failure cases and improves perceptual quality, bridging the gap between technical accuracy and practical usability.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Image Restoration, Diffusion Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5e8YOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose Failure-Aware Image Restoration (FAIR) to address the “lastmile” problem in image restoration, which is to correct localized restoration failures using region-specific guidance. The method constructs a dataset of approximately 67K annotated failure regions, categorizes failures into four types, and incorporates textual instructions, reference images, or geometric contours to guide restoration. Based on FLUX.2-klein-9B, FAIR employs a two-stage training curriculum and a region-weighted reconstruction objective. Extensive experiments show that the proposed approach resolves common failure cases and improves perceptual quality, bridging the gap between technical accuracy and practical usability.

Critical strengths
  1. The problem framing is useful and the system is complete. Localized failures in otherwise good restorations are a real deployment issue, and the paper delivers an end-to-end pipeline (annotation, region control, multi-modal guidance, single-step inference) rather than a single component.

  2. The paper does many of experiments. An equal-information comparison with all conditions removed (Table 10), per-tier breakdowns (Table 9, 11), a cascade baseline (Table 12), run-to-run variance (Table 17), and robustness to smaller condition models and mismatched references (Tables 15, 16).

Critical weaknesses
  1. Limited methodological novelty.

Most of the proposed framework's components closely resemble existing methods:

  • Region-specific restoration and mask-based control: InstructRestore [1] already introduces region-customized restoration using textual instructions, region masks, and spatially differentiated enhancement.
  • Reference-guided restoration: ReFIR [2] uses retrieved high-quality reference images to mitigate hallucinations and improve restoration fidelity.
  • Region-adaptive generative priors and one-step restoration: CODSR [3] introduces region-adaptive prior activation, text-matching guidance, and a one-step diffusion framework.
  • Adaptive reference conditioning: AdaRefSR [4] explicitly regulates reference information based on its relevance to the degraded input.
  • Pretrained generative backbone with adversarial fine-tuning: HYPIR [5] already employs a pretrained diffusion model with adversarial training, without requiring a diffusion loss.

Although FAIR combines these components with failure-type prediction and guidance selection, the individual techniques are largely established. More importantly, the paper does not adequately demonstrate whether its failure-aware selection mechanism offers substantial improvements over simpler, failure-type-agnostic guidance strategies.

  1. Excessive system complexity and questionable practical value.

FAIR combines multiple large pretrained models, including Qwen3-VL, SAM3, DINOv3, CLIP, and FLUX.2-klein-9B, together with a large reference database and a multi-stage condition-generation pipeline. While each component serves a purpose, the overall system appears heavily dependent on assembling existing models rather than introducing a compact and efficient restoration mechanism.

This complexity raises concerns about practical deployment, particularly in terms of computational cost, memory requirements, implementation overhead, and inference latency. Although the paper emphasizes single-step restoration, this does not necessarily translate into an efficient end-to-end pipeline when failure detection, segmentation, and reference retrieval are included.

  1. Unclear contribution of the proposed method versus the substantially larger pretrained backbone.

FAIR employs FLUX.2-klein-9B as its core restoration backbone, substantially larger than the approximately 0.87B SD2.1-based backbones and 2.6B SDXL-based backbone used by several competing methods. Furthermore, FAIR incorporates additional large models for text encoding, failure detection, segmentation, and reference retrieval, making its overall system considerably larger.

This raises a fundamental question: Are the reported restoration improvements primarily attributable to the proposed failure-aware framework, or to the substantially stronger pretrained generative backbone and larger overall system?

Although the LQ-only ablation demonstrates additional improvements from region and reference guidance, it does not isolate the effect of backbone capacity. The comparisons therefore do not sufficiently control for model size or computational budget.

Moreover, despite this considerable model capacity, FAIR still underperforms AdaRefSR on two important full-reference restoration metrics in Table 1: PSNR (20.91 vs. 21.23) and SSIM (0.558 vs. 0.561). These results further weaken the claim of consistently superior restoration quality.

References

[1] Liu, S., et al. "InstructRestore: Region-Customized Image Restoration with Human Instructions." NeurIPS, 2025.

[2] Guo, H., et al. "ReFIR: Grounding Large Restoration Models with Retrieval Augmentation." NeurIPS, 2024.

[3] Chen, H., et al. "Bridging Fidelity-Reality with Controllable One-Step Diffusion for Image Super-Resolution." CVPR, 2026.

[4] Wang, Y., et al. "Trust but Verify: Adaptive Conditioning for Reference-Based Diffusion Super-Resolution via Implicit Reference Correlation Modeling." ICLR, 2026.

[5] Lin, X., et al. "Harnessing Diffusion-Yielded Score Priors for Image Restoration." ACM Transactions on Graphics, 2025.

Questions for authors
  1. Given the substantial overlap with existing methods such as InstructRestore, ReFIR, CODSR, AdaRefSR, and HYPIR, what is the key technical innovation that distinguishes FAIR beyond integrating existing components?

  2. Considering that FAIR integrates multiple large pretrained models (Qwen3-VL, SAM3, DINOv3/CLIP, and FLUX.2-klein-9B), what are the total parameter count, GPU memory requirements, and end-to-end inference latency, including failure detection, segmentation, and reference retrieval? How do these costs compare with existing restoration methods, and are the performance improvements sufficient to justify the additional complexity?

  3. To what extent are FAIR's improvements attributable to its substantially larger FLUX.2-klein-9B backbone rather than the proposed failure-aware mechanism? Could the authors provide backbone-matched comparisons or evaluate FAIR with a smaller restoration backbone?

  4. Despite using a larger backbone, FAIR achieves lower PSNR and SSIM than AdaRefSR in Table 1. How do the authors interpret this results, particularly in relation to the increased computational cost?

7 comments
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for your careful review and for highlighting the importance of local restoration failures. Your feedback helped us clarify FAIR's contribution, inference costs, and evaluation.

Q1. Contribution and failure-aware selection

We thank you and Reviewer pVYz for recognizing the value of studying local restoration failures. FAIR's core contribution is to identify which local restoration failures need an external reference and use that distinction to build the training data and design a single-step restoration model. We collected approximately 67,000 failure regions from existing restorers' outputs (Sec. 3.1). At inference, FAIR predicts the failure type, selects a reference crop, contour, or text instruction, and applies that guidance to the corresponding region in a single restoration pass (Secs. 3.2 and 4.3).

ReFIR [1] retrieves reference images, and AdaRefSR [2] regulates their use. InstructRestore [3] uses instructions to locate regions and adjust enhancement strength. CODSR [4] adapts generative priors by region, and HYPIR [5] is a single-step restoration baseline. FAIR addresses which regions need a reference in the first place. Table 2 evaluates the training design. Table 11 shows that omitting visual guidance raises in-region LPIPS by 68% for OCR/ID and 1.6% for Subjective Feeling, demonstrating that the benefit varies by failure type.

In Fig. 7, several baselines leave the seal blurred or generate an unrelated emblem. Fig. 9 shows FAIR applying distinct references to two regions in one pass and restricting a reference-driven change to the selected object. The HYPIR-to-FLUX.2-klein-9B [6] cascade in Fig. 15 loses local cues in the first pass and copies incorrect plate characters in the second. Simply chaining the two models left last-mile failures unresolved in this test. Relative to the cascade, FAIR lowers in-region LPIPS from 0.0482 to 0.0330 and raises PSNR from 17.28 to 20.93 dB (Table 12). In the blinded study, FAIR ranks in the top two for overall quality, target correction, and preservation outside the target in 79.5%, 77.5%, and 78.5% of evaluations, respectively (Table 3). Its pairwise overall-quality win rates against the cited methods, including ReFIR+SeeSR [7] and ReFIR+SUPIR [8], range from 84.5% to 99.4% (Table 13).

Q2. End-to-end cost

FAIR restores in one pass. Users can supply the region, instruction, and reference to bypass the automatic guidance stages.

StageModel parametersRecorded peak GPU memoryTime per image
Failure detection8.77B (Qwen3-VL-8B)20.5 GB0.76 s
Region segmentation0.86B (SAM3)—0.07 s
Retrieval and verification0.45B (DINOv3 and CLIP)21.5 GB≈0.25 s
FAIR restoration≈17B (DiT and text encoder)41 GB3.16 s

Detection uses merged LoRA and a static KV cache. At one GPU query per image, automatic guidance adds 1.08 s to the 3.16 s measured on WR-SR [9]. The stage-summed estimate is 4.24 s/image, about 26% from guidance. Restoration peaks at 41 GB.

GPU retrieval uses DINOv3-L/16, FAISS (4.1M tiles, search_k = 64), CLIP-B/32 (top eight), SAM3, and Qwen3-VL-4B. Per query, search takes 3 ms, loading 0.11 s, recropping and verification 0.06 s each, and embedding and reranking 0.02 s.

MethodTime/imageLPIPSSSIM
AdaRefSR2.30 s0.2980.581
FAIR3.16 s0.2620.591

On WR-SR [9], FAIR trades speed for better LPIPS and SSIM (Table 8). With identical references, FAIR repairs a face and recovers "CAFE" lettering missed by AdaRefSR (Figs. 11-12). Blinded pairwise win rates are 94.6%, 93.2%, and 93.4% for overall quality, target repair, and preservation outside the target (Table 13). These gains justify the optional cost for users who prioritize local repair. We will report the timing assumptions and trade-off.

References

[1] ReFIR: Grounding Large Restoration Models with Retrieval Augmentation. Guo H, Dai T, Ouyang Z, et al. NeurIPS 2024.

[2] Trust but Verify: Adaptive Conditioning for Reference-Based Diffusion Super-Resolution via Implicit Reference Correlation Modeling. Wang Y, Wan Y, Zheng S, et al. ICLR 2026.

[3] InstructRestore: Region-Customized Image Restoration with Human Instructions. Liu S, Ma J, Sun L, et al. NeurIPS 2025.

[4] Bridging Fidelity-Reality with Controllable One-Step Diffusion for Image Super-Resolution. Chen H, Chen J, Pan J, Dong J. CVPR 2026.

[5] Harnessing Diffusion-Yielded Score Priors for Image Restoration. Lin X, Yu F, Hu J, et al. ACM Transactions on Graphics, 2025.

[6] FLUX.2 [klein] 9B Base. Black Forest Labs. Hugging Face 2026.

[7] SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution. Wu R, Yang T, Sun L, et al. CVPR 2024.

[8] Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild. Yu F, Gu J, Li Z, et al. CVPR 2024.

[9] Robust Reference-Based Super-Resolution via C2-Matching. Jiang Y, Chan KCK, Wang X, et al. CVPR 2021.

AuthorsOpenReview ↗

Official Comment by Authors

Q3. Backbone capacity

With model capacity fixed, failure-aware guidance improves restoration. The last three rows compare inference settings with the same trained FAIR checkpoint and the same FLUX.2-klein-9B [6] backbone. The first uses only the degraded image and the generic prompt “Restore,” with no region mask or visual reference. The second predicts the region mask and repair instruction from the degraded image and uses a crop from that image to retrieve the visual reference. The third still predicts the region mask from the degraded image, but uses the annotated instruction and a clean ground-truth crop for reference retrieval.

ConfigurationTraining and inference settingFID ↓In-region LPIPS ↓
Stage-1 restoration (Table 2a)Same 9B backbone, trained for general restoration80.38—
Final FAIR, generic instruction (Table 10)Final checkpoint, LQ input and “Restore” prompt71.790.0353
Final FAIR, fully LQ-driven (response to 529s)Same checkpoint, region, text, and retrieval query derived from LQ67.970.0351
Final FAIR, annotation-assisted (Table 1)Same checkpoint, LQ region prediction, annotated text, and GT-based retrieval query63.280.0319

Compared with the generic prompt, the fully LQ-driven setting reduces FID from 71.79 to 67.97, in-region LPIPS from 0.0353 to 0.0351, and out-of-region LPIPS from 0.2512 to 0.2486, while MUSIQ rises from 71.04 to 71.21. With the checkpoint fixed, the gain comes from the change in guidance.

FAIR targets last-mile restoration. As shown in Q1, existing restorers and the tested restorer-editor cascade can leave local failures unresolved, such as blurred seals or incorrect license-plate characters. If a smaller backbone struggles to restore the overall image, that comparison cannot answer whether FAIR improves local failures in otherwise successful restorations. All ablations in the paper hold the FLUX.2-klein-9B backbone fixed. This comparison measures the benefit of failure-aware guidance at fixed model capacity.

Q4. Fidelity relative to AdaRefSR

PSNR and SSIM cannot fully measure perceptual quality in generative restoration. Both reward agreement with one GT image, so a plausible reconstruction of ambiguous details can score lower [10]. We include both for completeness. We assess whether local repairs look better and preserve their surroundings using regional perceptual distances, visual comparisons, and blinded human judgments, following the human-centered approach of Yu et al. [11].

Synthetic benchmark (Table 1)PSNR (dB) ↑SSIM ↑In-region LPIPS ↓Out-of-region LPIPS ↓FID ↓
AdaRefSR21.230.5610.03680.27781.93
FAIR20.910.5580.03190.24363.28

FAIR lowers both regional LPIPS scores and FID. With the same references, it restores a coherent face in the pink-hat example and recovers the “CAFE” lettering, while AdaRefSR leaves these local failures unresolved (Figs. 11–12). FAIR's pairwise win rates over AdaRefSR, derived from blinded rankings, are 94.6% for overall quality, 93.2% for target correction, and 93.4% for preservation outside the target (Table 13). Each type of metric captures only part of restoration quality and can rank methods differently from human observers. Regional perceptual metrics, visual comparisons, and the blinded study together show a clear overall advantage for FAIR over AdaRefSR in the cases tested.

References

[10] The Perception-Distortion Tradeoff. Blau Y, Michaeli T. CVPR 2018.

[11] Evaluation of Visual Processing Should Be Human-Centered, Not Metric-Centered. Yu F, Hu J, You Z, et al. NeurIPS 2026.

Thank you for the detailed rebuttal and additional experiments. These results address some of my concerns. However, several important issues remain.

1. Methodological Novelty. The rebuttal clarifies FAIR's distinction from existing methods, but does not fully establish the contribution of failure-aware guidance selection. Table 11 demonstrates the benefit of visual guidance, rather than the necessity of failure-type-specific selection. Furthermore, the new ablation responding to Reviewer 529s shows that removing explicit region control while retaining the instructions and references produces nearly identical results (in-region LPIPS: 0.0316 vs. 0.0319).

2. Benchmark and Evaluation Fairness. The authors clarify in their response to Reviewer 529s that Table 1 uses GT crops for reference retrieval and annotated instructions. Although the reference-based baselines receive the same references, this setting does not represent fully automatic restoration. Under the fully LQ-driven setting, FID increases from 63.28 to 67.97, and in-region LPIPS from 0.0319 to 0.0351. This distinction should be emphasized when interpreting the reported improvements.

3. System Complexity and Backbone Capacity. The complete pipeline remains substantially more complicated than most competing methods. The fixed-backbone ablations demonstrate that additional guidance improves FAIR, but do not resolve whether the advantages over competing methods arise primarily from the larger pretrained backbone. The lower PSNR and SSIM relative to AdaRefSR in Table 1 further highlight the importance of evaluating the performance and complexity trade-off.

Overall, my primary concerns regarding methodological novelty, benchmark fairness, and backbone capacity remain insufficiently resolved. I therefore maintain my original rating of 2 (Weak Rejection).

Reviewer_529sOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The key to generative image restoration is to recognize the local failures. Within this context, this paper advance the practice by conditioning a one-step restoration model on region control and textual guidance, with optional reference images or ruled contours. To train the model, named FAIR, the paper construct failure-region annotations using coarse restorations with segmentation and reference retrieval. Training first learns general restoration and then adds region guidance with a larger reconstruction weight outside the target region. Evaluation covers synthetic benchmark and different settings on WR-SR and real-world scenarios. Ablations and a blinded preference study with participants are included.

Critical strengths
  1. The problem and data representation are concrete. Clear motivations on this. Details regarding the construction are fully given.
  2. Evaluations are performed across several settings, while, the restoration gains also beyond such. Confirm the rationales behind this paper.
  3. The paper is good to follow, presentations are good and accessible to readers.
Critical weaknesses
  1. The reviewer finds that, the synthetic test procedure needs further clarifications, e.g., what are being retrieved? References? Crops? Since this setting only changes only what the model sees. Moreover, are the text and retrieval queries in Tab. 1 obtained entirely from LQ inputs?
  2. Region-conditioned experiments need more studies, i.e., quantitative test that removes only region control while keeping text and references fixed.
  3. Lower LPIPS while with higher reference similarity, the conflicts here do not establish the claims on correct fidelity. Please specify this observations.
  4. Also, the reviewer finds it hard to understand the setting, please clarify the failure-prediction evaluation and whether the pipeline correctly abstains on images without notable restoration failures.
Questions for authors

See Weaknesses. The reviewer's rating is of 3 but still willing to see, and being contingent on a satisfactory rebuttal.

4 comments
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for your time and careful review. Your technical feedback helped us clarify the evaluation protocol and strengthen the evidence for region control.

Q1. Retrieval and the use of ground truth in Table 1

In Table 1, we predict region control R from the low-quality input (LQ), use annotated instruction t to represent the user’s request, and query the reference gallery with a clean ground-truth (GT) crop to retrieve local patches (Appendices B.3 and D.2). We retrieve them with DINOv3, rerank them by visual and text similarity, and recrop them with SAM3 when applicable (Appendix B.3). The selected patch is visual guidance v. FAIR and the reference-based baselines receive the same reference images. This setting reflects the goal of user-guided restoration. The annotated instruction stands in for the user’s explicit repair request, and the selected reference stands in for the visual guidance the user would supply.

Table 14 of the submitted manuscript already reports a related comparison: failure prediction from LQ alone versus LQ with access to GT. For failure-region and type prediction, Qwen receives only the degraded input in the LQ-only setting and the degraded input and the clean ground-truth image in the GT-assisted setting. Both settings use the same retrieval and restoration procedures. In a separate experiment on the same 120 images, we derive region control R, instruction t, and the retrieval query crop from the degraded input. FAIR retains much of its performance with all guidance derived from LQ, as the table shows. The real-world evaluation also derives these conditions from LQ (Table 7). We will highlight this fully LQ-driven setting and its results in the revised main text.

MetricTable 1 (annotation-assisted)Fully LQ-driven
FID ↓63.2867.97
In-region LPIPS ↓0.03190.0351
Out-of-region LPIPS ↓0.24290.2486
MUSIQ ↑71.4771.21

Q2. Roles of region control and ablation of the mask input

Thank you for suggesting this ablation. We used the Table 1 protocol on the 120 synthetic test images and removed only the explicit region mask R, keeping the model weights, degraded input, text instruction t, and visual reference v fixed.

MetricWith RWithout R
In-region LPIPS ↓0.03160.0319
Out-of-region PSNR (dB) ↑21.2021.18
MUSIQ ↑71.5471.50
DINO reference similarity ↑0.6590.655
CLIP reference similarity ↑0.6840.683
Out-of-region LPIPS ↓0.24310.2430

The table shows that retaining R improves several metrics, including out-of-region PSNR. In straightforward cases, the semantic description often identifies the target. Image- or region-level scores can obscure gains on individual objects. When several similar objects fit the same description, FAIR’s strict-region inference mode uses R to specify which object to restore. FAIR uses reference-guided restoration inside R and reference-free restoration outside it. In Fig. 9 (bottom), only the selected boy receives the plaid pattern from the reference.

During Stage 2 training, the mask weights pixels outside the target region up to five times as heavily as those inside in the image-space L₂ loss (Sec. 3.2.3 and App. C.2). This weighting improves overall perceptual fidelity, reducing FID from 64.63 to 63.22 and full-image LPIPS from 0.274 to 0.270 (Table 2(d–e)). We will add the mask-input ablation to Appendix D in the revision.

AuthorsOpenReview ↗

Official Comment by Authors

Q3. Reference similarity and ground-truth fidelity

A well-matched reference can supply local structure or texture cues consistent with the clean ground truth (GT). Using these cues while preserving content visible in the degraded input can bring the restored region closer to both the reference and the GT. DINO and CLIP similarities measure how closely the output follows the reference. In-region LPIPS measures perceptual distance to the GT. In Table 16, shuffling references within the same failure category lowers both reference similarities and raises in-region LPIPS from 0.0320 to 0.0353. This suggests that matching the reference to the target matters even within the same failure category.

Our goal is for the target region to follow restoration-relevant cues from the reference while preserving the input content. To see whether FAIR copies unrelated content, we replace the matched reference on all 120 images with either a reference from another failure category or an unrelated crop. Both mismatched settings yield PSNR, SSIM, FID, and regional LPIPS close to the no-reference setting, with much lower reference similarity than with Table 1 references. The same two-step Qwen3-VL-4B judge checks whether each of the 120 regions preserves GT content (objects and structure, text, identity, logos, or material patterns). Both mismatched settings yield content-match rates close to and below the no-reference rate. Table 1 references give the highest match rate. The low similarities suggest that FAIR follows mismatched references only weakly in this test. In the revision, we will add the stress test and clarify how we measure reference adherence and GT content fidelity.

Visual guidance settingPSNR ↑SSIM ↑FID ↓In-region LPIPS ↓Out-of-region LPIPS ↓DINO-Sim ↑CLIP-Sim ↑Two-step VLM content matches ↑
Table 1 reference20.910.55863.280.03190.24290.6500.680107/120
Reference from another failure category20.820.55368.060.03490.24880.0200.26686/120
Unrelated crop from another image20.820.55368.370.03510.24900.0300.35192/120
Visual guidance omitted20.840.55467.760.03500.2484——93/120

Q4. Failure prediction and abstention

We evaluate failure prediction on a held-out split with human-verified annotations, including images that annotators marked as having no notable failures, to check for false alarms (Appendix F). Given only the degraded input, the fine-tuned Qwen3-VL-8B predicts the failure category and describes its location. Category accuracy is 84.03%. SAM3 converts the predicted location description into a mask, and 90.66% of the resulting masks reach IoU ≥ 0.5 against the annotated masks.

Every degraded input is restored. The predictor decides whether to flag a local failure and where to apply guidance. FAIR takes the degraded image directly as input and uses the prediction to guide a single restoration pass. This allows FAIR to address likely local failures during restoration. If the predictor flags no local failure, FAIR can still restore the degraded image using only the generic prompt "Restore" (Table 10). Failure-aware guidance can help even on inputs that are easier to restore. We ran the full automatic pipeline on the same 120 images at three lighter degradation levels. Across all three levels, FAIR outperforms HYPIR [1] and AdaRefSR [2] on perceptual metrics (see the table below).

Input PSNRMethodPSNR ↑LPIPS ↓DISTS ↓FID ↓LPIPS in / out ↓
31.0 dBFAIR24.850.1100.07023.60.0127 / 0.100
HYPIR23.220.1660.10544.10.0201 / 0.151
AdaRefSR24.130.1560.09327.40.0180 / 0.140
27.0 dBFAIR24.100.1400.08328.80.0163 / 0.126
HYPIR22.990.1700.10443.70.0201 / 0.154
AdaRefSR23.750.1830.10334.40.0212 / 0.164
24.2 dBFAIR22.600.1960.10441.20.0230 / 0.176
HYPIR22.110.2130.11753.40.0247 / 0.193
AdaRefSR23.080.2290.12344.20.0263 / 0.206

References

[1] Harnessing Diffusion-Yielded Score Priors for Image Restoration. Lin X, Yu F, Hu J, et al. ACM Transactions on Graphics, 2025.

[2] Trust but Verify: Adaptive Conditioning for Reference-Based Diffusion Super-Resolution via Implicit Reference Correlation Modeling. Wang Y, Wan Y, Zheng S, et al. ICLR 2026.

Thanks for the reply. Points are marked and responses are fully acked by the reviewer, and the reviewer decided to leave the rating unchanged.

The reviewer encourages the authors to put efforts into subsequent studies, expand the benchmark, and validate more scenes and settings.

AuthorsOpenReview ↗

Official Comment by Authors

Thank you for your time and for acknowledging our responses. We will publicly release all training and test data, expand the benchmark, and validate FAIR on more datasets and scenarios. We welcome further discussion and suggestions.

Reviewer_pVYzOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses an under-appreciated problem: localized failures in faces, text, and structure that dominate perceived quality despite high global fidelity. The core idea—predicting where failures are likely and binding region-specific corrective guidance—is novel and well-motivated against the uniform-prior limitation of existing methods. The 67K failure-aware dataset with four-tier guidance is a meaningful contribution, and the two-stage curriculum with region-aware reweighting is principled. Quantitative gains (FID 81.93→63.28, LPIPSin improvements) and qualitative results are convincing. Concerns include dependence on automatic failure annotation whose reliability is only partially validated, and the added complexity of condition prediction at inference. The evaluation is thorough (13 metrics, multiple baselines), though real-world generalizability remains to be shown. Overall, a solid and impactful contribution.

Critical strengths

1.The "last-mile" problem is real and well motivated: global metrics are largely saturated, yet localized failures in faces, text, and logos occupy a tiny area while dominating perceived quality and practical acceptability. Shifting the objective from global fidelity to local perceptual acceptability is a meaningful reframing rather than an incremental metric gain.

2.Evaluation spans three test sets with 13 metrics against five non-reference and three reference-based baselines, and the ablation cleanly ablates the two-stage curriculum, LoRA sharing, joint conditioning, and region-aware reweighting. The authors additionally validate robustness to LQ-only versus GT-assisted conditions, smaller condition models, shuffled references, and include a 20-observer blinded study. The experimental results remain competitive among all previous state-of-the-art methods.

Critical weaknesses

1.The synthetic benchmark contains only 120 triplets (30 per tier) and is constructed by the same automatic pipeline that produced the training data, risking distributional and annotation-bias leakage. More importantly, the headline FID improvement (81.93 → 63.28) is computed on just 120 images where FID is known to be unreliable; the reported variance (±0.33 across five seeds) captures sampler noise on a fixed test set, not test-set sampling uncertainty.

2.FAIR is not uniformly best: it trails AdaRefSR on PSNR and SSIM, is clearly behind SeeSR and ReFIR+SeeSR on MANIQA (0.4356 vs about 0.511), and loses MUSIQ, CLIP-IQA, and Q-Align to CODSR or InstructRestore on the real-world set. Since the paper's own thesis is that pixel fidelity does not equal perceptual quality, it should state explicitly which metrics are expected to regress and whether those regressions are acceptable, rather than foregrounding only favorable IQA scores.

3.The largest gain occurs precisely on the OCR/ID tier (40% lower LPIPS_in), meaning the method actively rewrites appearance using retrieved exemplars in the regions where faithfulness matters most; Figure 8 shows that swapping the reference changes the recovered hair and eye color of the same subject. Beyond an ethics statement declaring outputs are reconstructions rather than evidence, the paper offers no rejection mechanism, no uncertainty estimate, and no analysis of behavior when the retrieved reference is irrelevant or wrong.

4.Stage 1 is of model trained at 1024 while stage 2 fine-tunes at 512, yet source images are cropped to 1024 and the paper never states the test resolution or provides any resolution-generalization ablation. Training regions are also heavily imbalanced (Pattern/Texture 63.9%, OCR/ID only 10.1%, etc.), so the hardest tier has the fewest samples, and its headline 40% gain rests on just 30 test regions per tier.

Questions for authors

LPIPS_in is a perceptual distance and is not appropriate to semantic or ID correctness. Please report OCR-level CER/WER on text regions or ArcFace identity similarity on faces, and a human- or VLM-rated semantic correctness rate, since the 40% OCR/ID claim currently rests entirely on a perceptual metric. When content is irretrievable from x_LQ , borrowing from a reference necessarily writes incorrect semantics. Please add a stress test with irrelevant or wrong references, quantify the resulting semantic error rate, and describe any abstention mechanism (e.g., disabling visual guidance when retrieval similarity is low) rather than relying only on an ethics disclaimer.

4 comments
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for taking the time to review our work and for sharing your expertise. Your constructive suggestions helped us strengthen the evaluation and clarify what the results support.

Q1. Synthetic benchmark and FID uncertainty

We agree that FID estimates for FAIR and AdaRefSR [1] can vary with sampling on the 120-image synthetic test set. Other metrics and our blinded user study offer additional evidence. The test set follows the training set’s construction pipeline, with no overlap (Appendix D.2). Both splits target the same “last-mile” failures that state-of-the-art restorers leave unresolved, including distorted faces, unreadable text, and inconsistent textures, so training and testing address the same task. ReFIR [2] and AdaRefSR [1] report FID on small test sets, including WR-SR (80 images) [3] for ReFIR. Compared with AdaRefSR, FAIR lowers in-region LPIPS from 0.0368 to 0.0319 and out-of-region LPIPS from 0.277 to 0.243. FAIR is preferred over AdaRefSR in 94.6% of pairwise overall-quality comparisons derived from the blinded rankings (Table 13). Together, these results support better local restoration quality for FAIR than for AdaRefSR on this benchmark.

To test generalization, we evaluated WR-SR [3], a public reference-based super-resolution benchmark with 80 test images and paired references, and a real-world set of 207 degraded images, for a total of 407 images across three datasets. For the real-world set, all guidance comes from the degraded inputs. FAIR compares favorably overall with the baselines on both external datasets (Tables 7 and 8). The real-world results help address concerns about shared synthetic data distributions and annotation bias. We will clarify what the small-sample FID and external results show in the revision.

Q2. Metric trade-offs and visual preference

Metrics measure different aspects of image quality, so FAIR does not rank first on every metric. As you pointed out, defects in faces, text, or logos can dominate perceived quality even when they occupy a small region. We therefore use regional perceptual metrics to assess local restoration quality. FAIR leads on 8 of 13 metrics in Table 1, including in- and out-of-region LPIPS, FID, DISTS, Q-Align, TOPIQ, and DINO and CLIP reference similarity. PSNR and SSIM measure pixel and structural agreement with GT, so a visually plausible local correction can score lower than the unrestored image if it increases pixel or structural discrepancies from GT. MUSIQ, CLIP-IQA, Q-Align, MANIQA, and TOPIQ score the whole image without a reference and can miss small but perceptually important defects such as distorted faces or unreadable text. Pixel fidelity, whole-image quality scores, and regional perceptual agreement capture different aspects of quality and can favor different methods. Local restoration quality therefore requires more than one measure (Yu et al., 2026 [4]). For this local restoration task, lower scores on some pixel-fidelity and no-reference metrics are acceptable when regional perceptual metrics and human judgments show better local repairs.

In our blinded study, 20 observers selected their top two from nine anonymized outputs for each of ten cases (Table 3). FAIR placed in the top two in 79.5% of evaluations for overall quality, 77.5% for target correction, and 78.5% for preservation outside the target. Pairwise comparisons derived from these rankings favored FAIR over AdaRefSR in 94.6% of comparisons for overall quality and 93.2% for target correction (Table 13). The blinded study lets observers judge the target repair and the surrounding content directly. Observers clearly preferred FAIR to AdaRefSR for overall quality and target-region repair in the cases tested. This preference agrees with the regional LPIPS results. We will discuss these metric trade-offs in more detail in the revision.

References

[1] Trust but Verify: Adaptive Conditioning for Reference-Based Diffusion Super-Resolution via Implicit Reference Correlation Modeling. Wang Y, Wan Y, Zheng S, et al. ICLR 2026.

[2] ReFIR: Grounding Large Restoration Models with Retrieval Augmentation. Guo H, Dai T, Ouyang Z, et al. NeurIPS 2024.

[3] Robust Reference-Based Super-Resolution via C2-Matching. Jiang Y, Chan KCK, Wang X, et al. CVPR 2021.

[4] Evaluation of Visual Processing Should Be Human-Centered, Not Metric-Centered. Yu F, Hu J, You Z, et al. NeurIPS 2026.

AuthorsOpenReview ↗

Official Comment by Authors

Q3. OCR, identity, and unreliable references

FAIR improves text transcription, identity similarity, and content agreement, remains robust to the tested reference mismatches, and uses visual-guidance abstention when no suitable reference is found. For text, Qwen3-VL-4B transcribes clean GT crops. We retain crops with CER of 0.35 or less against the annotated text and use the same reader on each method’s output crops. FAIR’s CER is 0.260, compared with 0.278 for HYPIR [5] and at least 0.777 for other baselines. Character recall is 81.4%, 75.0%, and at most 52.4%, respectively.

For identity, FAIR reaches 0.605 ArcFace similarity on face crops versus 0.440 for AdaRefSR [1]. Across all 120 test regions, Qwen3-VL-4B first identifies the GT content (structure, text, identity, logos, or material patterns), then checks whether the output keeps it. Less sharp outputs count as matches when the content is preserved. FAIR matches 107/120 regions, versus 93/120 for HYPIR.

Wrong-reference stress test. We replaced the reference on all 120 images with one from another failure category or an unrelated crop. DINO similarity to the supplied reference is 0.02–0.03, suggesting that FAIR copies little from the wrong reference. Matched references perform best overall, and both mismatch settings score close to the no-reference setting. FAIR therefore remains robust to the tested reference mismatches, maintaining restoration quality close to its no-reference performance.

Visual guidance settingPSNR ↑FID ↓Full-image LPIPS ↓In-region LPIPS ↓Out-of-region LPIPS ↓DISTS ↓MUSIQ ↑DINO-Sim ↑CLIP-Sim ↑Two-step VLM content matches ↑
Table 1 reference20.9163.280.27030.03190.24290.138771.470.6500.680107/120
Reference from another failure category20.8268.060.27880.03490.24880.145571.410.0200.26686/120
Unrelated crop from another image20.8268.370.27910.03510.24900.146271.450.0300.35192/120
Visual guidance omitted20.8467.760.27820.03500.24840.145871.16——93/120

Visual-guidance abstention. We omit the reference if no candidate reaches 0.50 visual similarity and passes Qwen3-VL semantic verification (App. B.3). Black-image dropout covers this setting (§3.2). With region control and the task-specific instruction retained, the no-reference setting gives FID 67.76 and in/out LPIPS 0.0350/0.2484 on all 120 images, ahead of the best baseline values (81.93, 0.0367/0.272). Removing region control and using “Restore” worsens these scores to 71.79 and 0.0353/0.2512 (Table 10). We will add the semantic evaluations and stress test to the revision.

Q4. Inference resolution and tier balance

At inference, we cap the long side at 1024 px on the synthetic and real-world sets and use the native 480–1024 px WR-SR [3] inputs. To address your question about inference resolution, we evaluated two settings using the same FAIR model, with the image long side capped at 512 and 1024 px, respectively. The change in inference resolution had little effect on target-region perceptual fidelity. As shown in the table below, in-region LPIPS against the ground truth is 0.0235 at the 512 px cap and 0.0237 at the 1024 px cap.

Metric512 px inference1024 px inference
In-region LPIPS ↓0.02350.0237
Out-of-region LPIPS ↓0.16710.1629
Out-of-region PSNR ↑22.3922.64
DINO-Sim-Ref ↑0.6060.622
CLIP-Sim-Ref ↑0.6500.643
MUSIQ ↑69.971.8
CLIP-IQA ↑0.6880.735

The class imbalance in training reflects how often each failure appeared in the restoration outputs used to build the dataset. In the reported experiments, we kept the observed training proportions to match the distribution of failures encountered in practice. We balanced the test set across failure tiers and report evaluation metrics separately for each tier (Table 9) to compare how different models perform on restoration tasks of different difficulty. In the revised manuscript, we will specify the inference and scoring resolutions, include the resolution comparison, and explain the class distributions of the training and test sets in the dataset description.

References

[5] Harnessing Diffusion-Yielded Score Priors for Image Restoration. Lin X, Yu F, Hu J, et al. ACM Transactions on Graphics, 2025.

Thanks for the replies from authors, some of my concerns are solved. However, after considering the replies and the reviews from other reviewers, I still keep my rating. The reason lies in that, in text restoration experiments, most of the shown cases in the manuscript are not readable. And how does the model reject a wrong text content in the reference image in such case? Since the model totally borrows the content for the restoration. If the reference is in failure, the failure will of course present in the restoration. The same problem could be happened on the face restoration. The paper did not discuss such failure cases, and the proposed solution will still far from solving the last-mile restoration problem.

AuthorsOpenReview ↗

Official Comment by Authors

Thank you for the follow-up. We would like to clarify the last-mile task. When the degraded input no longer provides reliable local detail, FAIR lets users specify the desired content and restore it at the intended location while maintaining consistency with the input. Our introduction analyzes the source of this problem. When the degraded input contains insufficient information for reliable local restoration, correcting these residual defects requires additional information (Sec. 1 and Fig. 2). Critical local defects can remain even in an otherwise well-restored image. FAIR gives users a way to correct them and make the overall result acceptable for practical use.

In this task, user-supplied information defines the intended restoration. We believe the model should remain faithful to the user’s intent. A user who knows the lettering on a sign, or wants it restored to specific text, can express that requirement through an instruction and reference. FAIR uses this information together with the degraded input to recover the specified content. The same principle applies to faces, where a user-supplied identity reference constrains the recovery of facial details. When the degraded input cannot resolve the exact text or identity, the supplied guidance determines the intended details.

The paper discusses when references provide useful guidance. When a reference is compatible with the target region’s content and structure and supplies missing text, identity cues, logos, or textures that match the user’s request, faithfully recovering that information is the goal of local restoration (Sec. 1, Fig. 2, and Sec. 3). In Table 16, shuffling references within the same failure category raises in-region LPIPS from 0.0320 to 0.0353 and lowers both reference-similarity scores, showing that matched references improve both reference adherence and GT fidelity. With references from another failure category or unrelated crops, our previous stress tests yield DINO similarity of only 0.02–0.03 and FID of 68.06 and 68.37, close to 67.76 without visual references. These results support FAIR’s use of relevant reference cues and limited transfer of unrelated content.

When the original text has been lost and the user cannot supply relevant information, no method can guarantee recovery of the true lettering. FAIR’s practical value is to compensate for that missing information through user guidance and correct the details that prevent an image from being usable. We define the last-mile problem as local content in an otherwise restored image remaining inconsistent with the user’s intent, and FAIR provides an interface for expressing and fulfilling that intent. We therefore consider this a direct, practical approach to addressing the last-mile problem, with the goal of producing local repairs that meet the user’s expectations. Agreement with GT is measured separately in our benchmark experiments. In real use, when users specify the intended text or appearance, fulfilling that request is an important criterion for controllable restoration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of the Revision

We thank all reviewers for their constructive feedback. We have uploaded a revised PDF; additions are marked with blue wavy underlines and deletions with red strikethrough. The new Sec. 4.5 brings the key results of the discussion into the main text.

  1. The gains come from failure-aware training and guidance, not from the backbone (5e8Y). On FAIR's own FLUX.2-klein-9B backbone, stage-1 restoration reaches FID 80.38, on par with HYPIR (84.12) and CODSR (84.45). With the final checkpoint fixed, guidance lowers FID from 71.79 to 63.28. HYPIR uses the larger 12B FLUX.1-dev, yet FAIR with only the generic "Restore" instruction achieves lower FID and LPIPS.

  2. FAIR leads all baselines even when every condition comes from the degraded input (529s, 5e8Y). With fully LQ-driven guidance, FAIR obtains FID 67.97 and in-region LPIPS 0.0351, against the best baseline values of 81.93 and 0.0367. The revision now states how R, t, and v are obtained in each setting.

  3. FAIR restores the correct text, identity, and content (pVYz). FAIR has the lowest OCR character error rate (0.260), the highest character recall (81.4%) and ArcFace identity similarity (0.605, against 0.440 for AdaRefSR), and keeps the GT content in 107/120 regions, against 93/120 for HYPIR.

  4. FAIR does not copy wrong references and stays strong without one (pVYz, 529s). With references from another failure category or unrelated crops, DINO similarity to the supplied reference falls from 0.650 to 0.02–0.03, and the results stay close to the no-reference setting. When no suitable reference is retrieved, FAIR omits it and still leads all baselines on FID and regional LPIPS.

The appendix adds the per-stage cost of the full pipeline, inference and scoring resolution with a 512 vs. 1024 px comparison, the training class distribution, results at three lighter degradation levels, where FAIR leads on every perceptual metric, visual-guidance abstention and region control, and a note on small-sample FID. We will release the training data, the test benchmark, and the inference code.

We hope the revision addresses your concerns and welcome further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.