acceptodds
Withdrawn from ICLR 2027

Refine What Remains: Base-Relative Selective Refinement for Reference-Guided Face Restoration

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reference-guided restoration can recover most facial content from a degraded image while leaving localized errors. A second full-image pass may correct these errors but can also change regions that the first pass restored well. We study Base-relative selective refinement: improving a fixed first-pass result (the Base) while limiting regressions in reliable regions. Our refiner conditions on the Base latent, degraded image, and same-identity reference. First, Base-conditioned adaptation projects repair-oriented updates against selected preservation gradients. These constraints provide a local, first-order safeguard rather than a guarantee for every output. Second, stochastic Flow-Map optimization combines global candidate credit with repair and preservation preferences. Differentiable trajectory replay provides action-sensitivity weights that transfer these preferences to position-level policy updates. Target-derived evidence guides training; at inference, an evidence predictor uses only available inputs. On CelebA-Test-Ref, the deployable Pred-H model improves paired reconstruction and identity measures over the Base, with a remaining distributional trade-off. Under oracle evidence, spatial credit improves Base-relative repair and preservation diagnostics.

Primary area:
generative models
Keywords:
reference-guided face restoration, selective refinement, flow matching, reinforcement learning, GRPO

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Y5q3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Well, the paper adds a selective second pass on a frozen reference-guided face restoration model. A GEM-style projected adaptation (Stage A) is followed by Flow-Map policy optimization with repair/preservation rewards and replay-based spatial credit (Stage B). At test time a predicted evidence map (Pred-H) replaces the oracle one.

Critical strengths
  1. The problem is real. A second pass can raise image scores while breaking details the first pass got right, and the fixed-Base diagnostics make this measurable.

  2. Pred-H improves on the Base in PSNR , LPIPS and both ID scores without the target, and the author are upfront about the worse FID-GT.

Critical weaknesses
  1. Selective repair and preservation are only measured for Oracle-H (Table 2). Pred-H only has whole-image metrics, so we can't tell whether correct regions survive in the deployable model.

  2. Most of the gain comes from Stage A (Oracle-H PSNR 17.20 to 18.43, full model 18.47), yet Stage A has no ablation, not even without the projection or with plain GEM.

  3. Table 2 uses the same BAD/GOOD maps and nearly the same Base-relative error terms that drive Stage B, and Oracle-H takes these maps as input. That is close to testing on the training objective.

Questions for authors
  1. How well does H_pred match H_oracle, e.g. IoU of the BAD regions, and how is the predictor trained?

  2. Fig. 1 says simply rerunning restoration doesn't help, but gives no numbers. How does a plain fine-tuned second pass with the same budget compare?

  3. J already has BAD-region repair and GOOD-region preservation terms. With those rewards fixed, does replay-based weighting beat uniform weights?

Reviewer_uD96OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In reference-guided face restoration, a second restoration pass may correct residual errors while also damaging details already recovered well in the first pass. The paper addresses this problem through selective refinement relative to a fixed first-pass result, combining gradient projection under preservation constraints with trajectory-replay-based spatial credit assignment. On CelebA-Test-Ref, Pred-H, which uses predicted evidence maps, improves reconstruction and identity metrics, although its FID remains worse than that of the first-pass result. Regional repair and content preservation are evaluated under the Oracle-H setting.

Critical strengths
  1. Evaluating both repair and degradation relative to a fixed Base reveals issues that whole-image averages can obscure. This evaluation approach is well aligned with the paper's objective.

  2. Without access to target images at inference, Pred-H improves PSNR from 17.2043 to 18.4371 and reduces LPIPS from 0.3405 to 0.3112, demonstrating useful reconstruction gains.

Critical weaknesses
  1. The key evidence for selective refinement remains limited to Oracle-H. Table 2 does not report regional repair and preservation metrics for Pred-H, so the whole-image improvements in Table 1 do not establish local preservation under the deployable setting. Moreover, Table 2 introduces spatial preference terms and sensitivity weighting together, leaving the specific benefit of trajectory-replay-based weighting unclear.

  2. Several settings that directly affect the method's behavior are insufficiently specified, including the evidence predictor's architecture and training procedure, the definitions of the Stage A losses and preservation boundaries, the reward weights, and the value or schedule of αk\alpha_k in Eq. (12). These choices affect the repair-preservation trade-off, and the current description is insufficient for independent reproduction.

Questions for authors
  1. Could you report the Table 2 metrics for Pred-H, along with the corresponding Stage-A results, to clarify the repair and preservation effects of each stage under the deployable setting?

  2. Under the same budget, how do the results change when sensitivity weights are replaced with uniform weights or weights taken directly from the evidence maps?

  3. Please provide the evidence predictor's training details, reward weights, preservation boundary definitions, and the setting of αk\alpha_k.

Reviewer_w25oOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper tackles the problem of refining the output of a reference-guided face restoration model. The authors consider a setup where a first-pass reference-guided restoration output, the "Base," is already available. They argue that a second pass should fix the errors that remain in the Base without damaging the regions that the Base has already restored well. To this end, the authors propose a two-stage training framework. In Stage A, it adapts the Base-conditioned refiner under preservation boundaries, and in Stage B, it leverages the Flow-Map GRPO with separate repair and preservation preferences. During training, the method first derives an “oracle version” that is allowed to use an error map computed from the groundtruth target image. At inference, a separate predictor needs to be used to estimate this map. The authors also propose new Base-relative diagnostic metrics, such as Repair Success, reliable-region failure rates and patch degradation. The manuscript provides results on CelebA-Test-Ref and shows that the deployable model (Pred-H) improves reconstruction metrics such as PSNR, SSIM, LPIPS and identity similarity score over the Base.

Critical strengths
  • The problem of iterative refinement on a fixed first-pass output of reference guided restoration is valid and relevant.
  • The proposed Base-relative diagnostics seems to be useful in reference-guided restoration, since they are useful in revealing the trade-offs between image quality / fidelity / identity preservation that are not captured easily by the image-level metrics such as PSNR, LPIPS etc.
Critical weaknesses

[1] Stage B is the most complex and expensive part of the method, but it adds very little to the final result. When using the groundtruth (oracle), Stage A accounts for about ~95+% of the total PSNR gain while stage B adds only +0.04 dB PSNR and −0.005 LPIPS on top of Stage A. The deployable Pred-H model is also almost the same as the Stage A oracle row (+0.009 dB PSNR, −0.001 LPIPS). The effectiveness of the stage B is highly unclear.

[2] The projection step is the core idea of Stage A, but no ablation is provided on that. At least some experiments should be provided in which stage A is trained with the plain update instead of projected one. Since most of the gain comes from Stage A (see [1]), it is unclear whether the gain comes from the projection or simply from additional training on Base outputs. Without this ablation, contribution 2 is not supported.

[3] The role of the evidence map is unclear. Pred-H performs almost identical to Oracle-H. This could mean the predictor is very accurate, or it could mean the model largely ignores the map. I reckon the second case is quite possible, given how small the Stage B gains are. The authors should report results with no map, a constant map and a random map. They should also report the accuracy of the evidence predictor.

[4] The obvious simple baselines are missing. For example: running the Base a second time, fine-tuning the Base for longer, or training a Base-conditioned refiner with a standard loss. Figure 1 claims that repeated restoration fails, but this is never measured. There is also no comparison against plain RL fine-tuning of the same Base. Without these, it is not possible to conclude on whether the proposed machinery is needed/useful.

[5] The presentation of the oracle results is misleading. The oracle map is computed from the test ground-truth image, so the oracle rows use target information at test time. Even so, Table 1 bolds the Oracle-H values as the best results "across all rows." This makes the method look stronger inadequately and should be avoided for authentic presentation.

[6] The training reward and the evaluation diagnostics are nearly the same. Training rewards repair in high-error regions and preservation in low-error regions, using pixel and LPIPS errors. Table 2 then scores the model with the same kinds of quantities, using region weights from the same oracle map. So the gains in Table 2 are expected by construction. Independent evaluation is needed, for example a user study, regions based on face parsing, or other perceptual metrics.

[7] The qualitative results are very limited and underwhelming. Only two examples in main paper are not enough, the figure does not point out the regions where one could notice improvements if any, and the outputs of the baselines seem to have much lower brightness/contrast compared to that of the proposed method and groundtruth – potentially a bug in generating / displaying the outputs?

[8] The improvements are small and come without any statistical support. In Table 2, Repair Success rises by 1.7 points, Pix-Fail falls by 1.4 points, and patch degradation falls by about 0.01. There are no multiple seeds or confidence intervals.

[9] The experimental scope is limited: Only one benchmark is used (CelebA-Test-Ref with synthetic degradations). There is no evaluation on real-world degraded images. Also, the refiner is applied only to the authors' own Base. The paper frames the problem as a general post-restoration setting, but it never refines outputs from other competing methods.

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.