Refine What Remains: Base-Relative Selective Refinement for Reference-Guided Face Restoration

Reference-guided restoration can recover most facial content from a degraded image while leaving localized errors. A second full-image pass may correct these errors but can also change regions that the first pass restored well. We study Base-relative selective refinement: improving a fixed first-pass result (the Base) while limiting regressions in reliable regions. Our refiner conditions on the Base latent, degraded image, and same-identity reference. First, Base-conditioned adaptation projects repair-oriented updates against selected preservation gradients. These constraints provide a local, first-order safeguard rather than a guarantee for every output. Second, stochastic Flow-Map optimization combines global candidate credit with repair and preservation preferences. Differentiable trajectory replay provides action-sensitivity weights that transfer these preferences to position-level policy updates. Target-derived evidence guides training; at inference, an evidence predictor uses only available inputs. On CelebA-Test-Ref, the deployable Pred-H model improves paired reconstruction and identity measures over the Base, with a remaining distributional trade-off. Under oracle evidence, spatial credit improves Base-relative repair and preservation diagnostics.