acceptodds
Under review as a conference paper at ICLR 2027

RAEFace: Rethinking Latent Representations for Blind Face Restoration

Abstract

Most state-of-the-art blind face restoration (BFR) methods, whether codebook- or diffusion-based, rely on VAE-style latent representations. Representation autoencoders (RAEs), built on pretrained visual encoders, have shown strong performance in image generation but remain largely unexplored in restoration-oriented low-level vision. We show that RAE latents better preserve facial semantics under severe degradation and exhibit stronger semantic compatibility between high-quality (HQ) and low-quality (LQ) observations than VAE latents. In light of this, we propose RAEFace, a framework that unifies HQ target modeling and LQ condition injection in a shared RAE space to produce semantically faithful base restorations. To seek perceptually stronger yet faithful solutions, we introduce Perceptual Improvement via Extrapolation (PiE), a training-free inference strategy that performs stepwise extrapolation during flow-matching sampling. Using the base prediction as a fidelity anchor, PiE extrapolates beyond it along the direction from the degraded latent to its restored counterpart. Our analysis links the effectiveness of this direction to HQ–LQ semantic compatibility, which helps preserve facial content during extrapolation. Experiments on real-world and synthetic benchmarks demonstrate that RAEFace achieves state-of-the-art perceptual quality with competitive fidelity. Results on inpainting and colorization further suggest the broader potential of RAE representations for related low-level vision tasks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Blind Face Restoration, Representation AutoEncoder, Flow Matching

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_HotuOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces a restoration-aware VAE mechanism in diffusion-based face restoration task, named RAEFace. Compared to current latent representation, RAEFace leverages information on degraded LQ images, and reaches better performance on simulated and real-world benchmarks.

Critical strengths
  • The novelty sounds good. To better achieve the face resotration task, RAEFace introduce latent alignment from REPA to give better guidance for diffusion process.

  • The experiments are convincing. RARFace achieve better performance on CelebA-Test, LFW-Test, WebPhoto-Test and Wider-Test.

Critical weaknesses
  • A detaied comparison is needed for related latent space alignment works, like FaithDiff[1] and other related similar works.

  • I have some concerns on results shown in Figure 9 (smiling woman at top-right) and Figure 11. (1) How could RAEFace know that the woman shows her tooth when smiling, as this area is inpainted? (2) Why in Figure 11, results provided by CodeFormer has wrong color?

  • The training target is not presented well in Method section. The authors may provide a detailed algorithm, showing how the loss is calculated, and if there are additional training stage for the VAE encoder.

Questions for authors

Please see "Critical Weaknesses" part for my questions for authors.

4 comments

We thank Reviewer Hotu for the positive assessment and insightful feedback. We have added experiments and analyses, revised the manuscript, and provide point-by-point responses below. We welcome further questions and discussion.

W1: Comparison with FaithDiff

A detailed comparison is needed for related latent space alignment works, like FaithDiff[1] and other related similar works.

We thank the reviewer for highlighting FaithDiff and related work. We have added an end-to-end BFR comparison with FaithDiff using the same test inputs and evaluation metrics. RAEFace consistently outperforms FaithDiff on all four no-reference quality metrics—CLIPIQA, MUSIQ, MANIQA, and TOPIQ—across LFW, WebPhoto, Wider, and CelebA.

At the level of latent alignment, FaithDiff’s alignment module and our REPA objective serve different roles. FaithDiff uses a learned module to align and integrate LQ encoder features with the current noisy latent to construct denoising inputs. In RAEFace, REPA, adopted from RAEv2, instead supervises intermediate DDT features using paired HQ encoder representations to regularize their spatial token-to-token similarity structure.

LFW

MethodFID ↓CLIPIQA ↑MUSIQ ↑MANIQA ↑TOPIQ ↑
FaithDiff41.370.67875.770.4950.889
Ours46.670.74376.650.5750.945

WebPhoto

MethodFID ↓CLIPIQA ↑MUSIQ ↑MANIQA ↑TOPIQ ↑
FaithDiff87.680.67974.970.4880.833
Ours77.540.74176.500.5660.906

Wider

MethodFID ↓CLIPIQA ↑MUSIQ ↑MANIQA ↑TOPIQ ↑
FaithDiff35.700.71076.070.5120.900
Ours34.900.76976.580.5910.949

CelebA

MethodPSNR ↑LPIPS ↓IDS ↑FID(HQ) ↓FID(FFHQ) ↓CLIPIQA ↑MUSIQ ↑MANIQA ↑TOPIQ ↑
FaithDiff24.910.2430.87217.2242.800.65376.100.5140.905
Ours24.340.2260.86110.7138.590.70176.440.5730.943

W2: Interpretation of inpainting and colorization results

I have some concerns on results shown in Figure 9 (smiling woman at top-right) and Figure 11. (1) How could RAEFace know that the woman shows her tooth when smiling, as this area is inpainted? (2) Why in Figure 11, results provided by CodeFormer has wrong color?

1) Inpainting: inferring missing details from informative facial attributes. RAEFace cannot determine with certainty whether teeth are present in the masked region. The teeth in Fig. 9 are a plausible completion conditioned on the visible facial context and learned priors, rather than guaranteed recovery of the hidden ground truth. Our masked-face linear probes provide supporting evidence that useful facial attributes remain accessible under occlusion:

Train setTest setDINOv3-LSigLIP2-LMAE-LSD-VAE
HQHQ85.22%86.87%78.90%49.97%
Rectangle-maskedRectangle-masked81.03%80.86%74.59%49.61%
HQRectangle-masked77.60%71.12%65.39%47.15%
Brush-maskedBrush-masked81.18%81.13%74.14%49.84%
HQBrush-masked76.53%71.95%66.07%47.10%

HQ denotes unmasked input; Mask denotes inputs with the specific mask.

These results show that facial attributes remain more accessible in the semantic encoders (DINOv3, SigLIP and MAE) than in SD-VAE under both mask types. They support context-informed completion, but do not establish the correctness of a particular hidden detail.

2) Colorization: color ambiguity and preservation of visible details. A color difference from the reference alone does not necessarily indicate an error, because grayscale inputs admit multiple plausible colorizations. Our objective is to generate natural colors while preserving the facial structure and details visible in the input. The concern with CodeFormer in Fig. 11 of the original submission (Fig. 15 in the revised manuscript) is that it also modifies visible facial details, going beyond color assignment and conflicting with this preservation goal. Appendix F.1 now clarifies the color ambiguity and the importance of preserving visible facial structure and details.

W3: Training objectives, loss computation, and encoder training

The training target is not presented well in Method section. The authors may provide a detailed algorithm, showing how the loss is calculated, and if there are additional training stage for the VAE encoder.

We have included a detailed training algorithm in the revised manuscript, clarifying the training targets, loss computation, and which components are trained or frozen (Sec. 3.2 and Appendix A, Algorithms 1–2).

Thanks for the authors' feedback. I have no more concerns and will raise my score to 4.

Thank you very much for your kind words and for revising the score. We are greatly encouraged by your positive feedback and sincerely appreciate your time and guidance throughout the review process.

Reviewer_EoQXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RAEFace, a framework that models the blind face restoration (BFR) task based on Representation Autoencoders (RAE). In general, RAEFace largely reuses the architecture of RAEv2, adapting its Internal Guidance interface into a new mechanism tailored for BFR called Perceptual Improvement via Extrapolation (PiE). During inference, low-quality (LQ) features replace the shallow features used in RAEv2’s DDT for Internal Guidance to perform extrapolation, thereby achieving a better perception-fidelity tradeoff. Statistical analyses are also provided to validate the effectiveness of PiE. Ultimately, RAEFace reports state-of-the-art performance on both real and synthetic datasets and demonstrates promising potential for downstream applications such as image inpainting and colorization.

Critical strengths

This work explores BFR using RAE, analyses the differences between RAE and VAE for this task, and builds a well-performing RAEFace framework based on RAEv2. An inspiring extrapolation mechanism is proposed for perceptual improvement during inference. The overall writing and experiments are quite complete, providing a relatively thorough investigation into the potential benefits of utilizing RAE for BFR.

Critical weaknesses
  1. RAEFace lacks techniques specific to BFR that distinguish it from general image restoration (IR). It appears to be a general framework trained on face datasets for BFR, and the approach could seemingly be applied directly to general IR tasks with different training data.
  2. Appendix A.1 states that RAEFace is built upon RAEv2 [R1], implying that Equation (8) for PiE is directly derived from the Internal Guidance [R2] interface in RAEv2. However, the main text in Section 3.3 introduces PiE without citing RAEv2 or discussing Internal Guidance. These connections and distinctions should be explicitly addressed in the main text.

Overall, my current feeling is that RAEFace adopts a straightforward approach to build a BFR framework based on RAEv2 and its Internal Guidance interface. The experiments and analyses comparing RAE and VAE for this task are commendable, and the reported results are complete and promising. However, the method itself lacks significant technical innovation relative to RAEv2 or the BFR field, and does not exceed expectations. Therefore, I lean toward an initial rating slightly above the borderline, but unlikely to score a higher one at present.

Reference:
[R1] Singh J, Zheng B, Wu Z, et al. Improved baselines with representation autoencoders. arXiv preprint arXiv:2605.18324, 2026.
[R2] Zhou X, Li Q, Hu X, et al. Guiding a diffusion transformer with the internal dynamics of itself. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026: 11536-11545.

Questions for authors
  1. Are there any BFR-specific techniques in RAEFace that distinguish it from general image restoration? If possible, how RAEFace performs on the general IR domain?
  2. Could you discuss the relationship and differences between PiE and the Internal Guidance mechanism in RAEv2, and incorporate this discussion into Section 3.3?
  3. Could you expand the strength range of PiE in Figure 8, as some of the metrics (e.g., TOPIQ, MANIQA) have not reached the peak at strength = 0.2. It might be help to present the complete variation of metrics relative to the strength.
  4. Figure 3 lacks sufficient explanation and experimental data: Fig.3 (a) is difficult to understand at first glance as it is not mentioned in the text, and its symbols lack direct definitions. Fig.3 (b) only displays w=0w = 0 and w=0.1w = 0.1, lacking visualization for larger values. For Fig.3 (c), the specific ww value corresponding to each point on the curve is unknown, and the range of ww should be further expanded to observe the subsequent perception-fidelity trade-off changes in RAE.
7 comments

We thank Reviewer EoQX for recognizing the value of our work and for the constructive feedback. We have made every effort to address concerns (W1–W2) and four questions (Q1–Q4) through additional experiments and analyses, manuscript revisions, and detailed, point-by-point responses below. We hope the additional evidence and clarifications address your concerns, and we welcome further questions and discussion.

W1 & Q1: BFR-specific design and general applicability

RAEFace lacks techniques specific to BFR that distinguish it from general image restoration (IR). It appears to be a general framework trained on face datasets for BFR, and the approach could seemingly be applied directly to general IR tasks with different training data.

Are there any BFR-specific techniques in RAEFace that distinguish it from general image restoration? If possible, how RAEFace performs on the general IR domain?

1) BFR-specific design.

RAEFace is simple yet effective. It does not rely on BFR-specific architectural modules, yet achieves state-of-the-art perceptual results across four benchmarks and competitive pixel and identity fidelity on CelebA-Test. To our knowledge, it provides the first systematic exploration of RAE representations for BFR, moving beyond the compact VAE spaces used by the compared latent generative baselines.

We choose BFR because it supports quantitative representation analysis while posing demanding semantic challenges. The 14 facial attributes annotated on FFHQ enable linear probing under degradation: DINOv3 achieves approximately 73.92% attribute-prediction accuracy versus 48.53% for the evaluated VAE, a gap of 25.39 percentage points (Fig. 2, right). Preserving expressions, age, pose, and spatial relationships while handling occlusions and old-photograph artifacts further tests whether these representation advantages translate into faithful restoration (Figs. 5 and 6).

Vanilla RAE alone still leaves a perceptual gap (Fig. 1). PiE addresses this through training-free refinement along the input-specific weak-to-strong direction z^HQ−zLQ\hat z_{\mathrm{HQ}}-z_{\mathrm{LQ}}, without a separate refinement network. The oracle and semantic-compatibility analyses (Figs. 3(c) and 4) provide empirical support for this simple yet effective design.

2) General image restoration.

Yes. The colorization and inpainting results demonstrate direct adaptations of the same formulation to grayscale and masked inputs, respectively (Sec. 4.4 and Appendices F.1–F.2, Figs. 9, 15, and 16). These are standalone restoration tasks, without a preceding BFR or super-resolution stage.

As suggested, we have additionally trained the model for general-image super-resolution on LSDIR using the same framework and training configuration and evaluated it on RealSR, DRealSR, and DIV2K-Val, achieving the best NIQE and TOPIQ scores on all three benchmarks, the best LPIPS on RealSR and DIV2K-Val, and the highest MUSIQ on DRealSR compared with StableSR, DiT4SR, OSEDiff, and VOSR. These results further support its applicability beyond faces and demonstrate the practical value of a simple, unified restoration framework.

RealSR

MethodPSNR ↑SSIM ↑LPIPS ↓DISTS ↓NIQE ↓MUSIQ ↑TOPIQ ↑MANIQA ↑
StableSR24.200.6980.3040.2175.80865.230.5670.621
DiT4SR22.850.6430.3270.2276.25567.980.6020.664
OSEDiff24.630.7190.2970.2145.70569.090.6260.632
VOSR24.700.7050.2850.2115.90267.500.5900.614
Ours23.300.6540.2820.2185.13868.940.6420.658

DRealSR

MethodPSNR ↑SSIM ↑LPIPS ↓DISTS ↓NIQE ↓MUSIQ ↑TOPIQ ↑MANIQA ↑
StableSR28.080.7510.3310.2306.59258.680.5380.555
DiT4SR25.580.6680.3820.2516.82065.070.5910.634
OSEDiff27.910.7830.2970.2166.44464.700.6000.590
VOSR27.500.7320.3750.2497.37357.260.5100.528
Ours26.690.7210.3120.2305.89665.380.6220.623

DIV2K-Val

MethodPSNR ↑SSIM ↑LPIPS ↓DISTS ↓NIQE ↓MUSIQ ↑TOPIQ ↑MANIQA ↑
StableSR23.280.5680.3110.2044.74365.830.5960.617
DiT4SR21.700.5440.3460.2124.95468.470.6310.647
OSEDiff23.720.6110.2940.1984.71267.970.6190.613
VOSR23.730.6010.3040.1885.47170.550.6360.654
Ours22.760.5660.2910.1924.51469.810.6740.651

W2 & Q2: Relationship between PiE and Internal Guidance

Appendix A.1 states that RAEFace is built upon RAEv2 [R1], implying that Equation (8) for PiE is directly derived from the Internal Guidance [R2] interface in RAEv2. However, the main text in Section 3.3 introduces PiE without citing RAEv2 or discussing Internal Guidance. These connections and distinctions should be explicitly addressed in the main text.

Could you discuss the relationship and differences between PiE and the Internal Guidance mechanism in RAEv2, and incorporate this discussion into Section 3.3?

PiE and Internal Guidance share the weak-to-strong extrapolation principle, but differ in how the weak reference is constructed. In RAEv2, Internal Guidance uses an intermediate prediction from the early-layer self-REPA head as the weak reference and the full model output as the strong endpoint. PiE instead uses the fixed encoded LQ observation zLQ=E(xLQ)z_{\mathrm{LQ}}=E(x_{\mathrm{LQ}}) as the weak reference and the restored prediction z^HQ\hat z_{\mathrm{HQ}} as the strong endpoint. Consequently, d=z^HQ−zLQd=\hat z_{\mathrm{HQ}}-z_{\mathrm{LQ}} represents the degraded-to-restored direction in the shared RAE space. Equation (8) thus instantiates the existing extrapolation construction with a restoration-specific reference. We have revised Sec. 3.3 to explicitly cite RAEv2 and Internal Guidance and clarify this relationship and distinction.

Q3: PiE strength range

Could you expand the strength range of PiE in Figure 8, as some of the metrics (e.g., TOPIQ, MANIQA) have not reached the peak at strength = 0.2. It might be help to present the complete variation of metrics relative to the strength.

Thank you for the suggestion. We have extended the PiE sweep to w=0.4w=0.4 and added a user preference study to examine the perception–fidelity trade-off (Appendix B, Fig. 11 and Appendix C, Fig. 13(b), revised manuscript). The questionnaire design and instructions for evaluating texture quality and facial-feature consistency with the HQ reference are provided in Appendix H, Fig. 20.

At w=0.065w=0.065 and w=0.1w=0.1, approximately 66–69% of judgments favored PiE for texture quality, and 58–67% favored it for facial-feature consistency. At w=0.2w=0.2 and w=0.3w=0.3, both preference rates fell below 50%. Together with the metric curves, these observations suggest that moderate extrapolation offers a useful balance on the evaluated examples. At larger strengths, IDS decreases further and artifacts become more noticeable, as illustrated in Appendix D, Fig. 14.

Q4: Clarification of Figure 3

Figure 3 lacks sufficient explanation and experimental data: Fig.3 (a) is difficult to understand at first glance as it is not mentioned in the text, and its symbols lack direct definitions. Fig.3 (b) only displays w=0 and w=0.1, lacking visualization for larger values. For Fig.3 (c), the specific w value corresponding to each point on the curve is unknown, and the range of w should be further expanded to observe the subsequent perception-fidelity trade-off changes in RAE.

Thank you for these suggestions. We have revised Fig. 3 and added detailed explanations and symbol definitions to the manuscript. We clarify its three panels below.

Fig. 3(a): Conceptual illustration of PiE.

Here, M0\mathcal M_0 and Mt\mathcal M_t denote the schematic clean and time-tt latent manifolds, respectively. GT marks the encoded HQ target, and the dashed GT neighborhood illustrates plausible solutions that may improve perceptual quality over the base prediction while remaining faithful to the observed input. PiE extrapolates from the predicted clean endpoint using d=z^HQ−zLQd=\hat z_{\mathrm{HQ}}-z_{\mathrm{LQ}} and z^PiE=z^HQ+wd\hat z_{\mathrm{PiE}}=\hat z_{\mathrm{HQ}}+wd. The modified endpoint changes the sampling velocity and hence the next flow state. The dashed paths ℓHQ\ell_{\mathrm{HQ}} and ℓPiE\ell_{\mathrm{PiE}} indicate the local directions toward the base and PiE endpoints, respectively, while the solid arrows show the corresponding sampling updates.

Fig. 3(b): Base-to-PiE detail enhancement.

We have added visualizations for w=0,0.05,0.1,0.15,0.2,0.25,0.3w=0,0.05,0.1,0.15,0.2,0.25,0.3 in Appendix B, Fig. 12 of the revised manuscript, together with failure cases at larger strengths in Appendix D, Fig. 14.

Fig. 3(c): Oracle perception–fidelity trade-off.

The oracle study replaces the predicted HQ endpoint with the ground-truth HQ latent to exclude base-prediction errors. The curves cover w∈[0,1]w\in[0,1] in increments of 0.010.01, with six markers at w=0.0,0.2,0.4,0.6,0.8,1.0w=0.0,0.2,0.4,0.6,0.8,1.0. The range and marker interval are specified in the revised Fig. 3 caption, while Appendix G.1 gives the full sweep settings, including the 0.010.01 increment and 101101 evaluated strengths.

Reviewer_C9KtOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RAEFace, a blind face restoration (BFR) framework built on Representation Autoencoders (RAEs) instead of the usual VAE latents. The key idea is to map both the HQ target and the LQ condition into a shared RAE space derived from a frozen pretrained vision encoder (default DINOv3), then perform conditional flow matching with LQ conditioning via cross-attention. This gives a semantically faithful “base restoration.” On top of that, the authors introduce Perceptual Improvement via Extrapolation (PiE), a training-free inference strategy that extrapolates the predicted HQ latent beyond the base anchor along the direction z^HQ−zLQ\hat z_{HQ} - z_{LQ}. The paper argues—and supports with an oracle study—that RAE latents have better HQ–LQ semantic compatibility than VAE latents, making such extrapolation perceptually beneficial. Experiments on CelebA-Test and three real-world benchmarks show competitive fidelity and strong no-reference perceptual scores; additional colorization/inpainting results are provided.

Critical strengths
  1. Timely reframing of BFR.​ Most BFR papers treat the latent space as fixed and focus on conditioning/guidance. This work correctly identifies that the representation itself matters for degraded inputs, which is a useful perspective shift.
  2. Solid use of RAE properties.​ Putting LQ condition and HQ target in the same semantic space is coherent with recent RAE/DiT literature, where frozen encoders preserve richer semantics than compressed VAE latents .
  3. PiE is simple and pragmatic.​ Training-free extrapolation is cheap to apply, easy to control with one scalar ww, and avoids introducing a second guidance network.
  4. Oracle correlation analysis is a genuine plus.​ The linear-probe accuracy ↔ perceptual-gain correlation (Fig. 4) gives a more principled explanation than pure metric chasing, and helps justify why RAE > VAE for this setting.
  5. Reasonable empirical coverage.​ Comparisons across GFPGAN / CodeFormer / InterLCM / OSDFace / GeoMAR, plus ablation on encoder choice, target/condition space, and PiE vs CFG, make the paper more than a pure method claim
Critical weaknesses
  1. Novelty is incremental.​ RAE decoder training, DDT/flow-matching backbone, REPA alignment, cross-attention conditioning, and weak-to-strong extrapolation are all inherited from recent works. The core delta is “apply RAE to BFR + add LQ→HQ extrapolation.” The paper should state more honestly what is new at the mechanism level versus adaptation.
  2. PiE is under-justified as a restoration mechanism.​ Extrapolating along z^HQ−zLQ\hat z_{HQ}-z_{LQ} can easily be viewed as pushing the output toward “more restored-looking” latents that score higher on MANIQA/MUSIQ/CLIP-IQA without necessarily recovering ground-truth detail. The oracle study supports plausibility, but there is no evidence that PiE improves identity correctness or reduces hallucination on hard degradations. Reporting only small PSNR drops hides this trade-off.
  3. Benchmark tables are perception-heavy.​ On CelebA-Test (Table 2), RAEFace is not the best on PSNR (25.14 CodeFormer vs 24.34 yours) and only marginally better on LPIPS. Real-world tables rely entirely on no-reference metrics + FID. For a fidelity-claiming method, this is weak; at least IDS, identity retrieval, or user-study pairwise preference against the strongest baselines should be emphasized more carefully.
  4. VAE vs RAE ablation is confounded.​ In Table 3, “VAE target + RAE condition” mixes latent spaces and decoders; gains may come from the RAE decoder rather than the semantic compatibility claim. A cleaner ablation would fix the decoder and only swap the latent/encoder, or fix the encoder and swap only the decoder.
  5. Condition-injection details are thin.​ The paper says zLQ=E(xLQ)z_{LQ}=E(x_{LQ}), but the RAE formulation in Sec. 2 uses noise-augmented latents z~0\tilde z_0. It is unclear whether the LQ condition is also noise-augmented, how this interacts with flow states ztz_t, and whether training/inference condition statistics match. This matters because PiE explicitly uses zLQz_{LQ} at test time.
  6. PiE strength is very small and dataset-dependent.​ w=0.065w=0.065 on CelebA, w=0.1w=0.1 on real sets. There is no sensitivity curve in the main paper, no failure-case analysis when ww grows, and no check on whether PiE helps or hurts low-quality inputs where zLQz_{LQ} is unreliable.
  7. Application extensions are weakly controlled.​ Colorization/inpainting comparisons are qualitative vs CodeFormer only. CodeFormer is a codebook method not tuned for inpainting; a stronger check would include a diffusion inpainting baseline or masked-ground-truth metrics.
  8. Writing/presentation issues.​ Duplicate sentences in conclusion/appx, missing values in some tables, “EI2Det”-style typos carried from related works, and Figure references occasionally vague. Acceptable for review but should be cleaned.
Questions for authors
  1. What exactly is the novel contribution beyond “RAE + DDT + weak-to-strong extrapolation”? Is the claim a new representation analysis for BFR, a new sampling trick, or a new unified conditioning framework?
  2. In PiE, if z^HQ\hat z_{HQ} is already a good base, why does extrapolating further improve perceptual metrics instead of amplifying errors? Do you have any case where PiE hurts identity or introduces artifacts? Please show failure examples.
  3. How is zLQz_{LQ} computed during sampling—clean encoder latent, noise-augmented latent, or projected tokens only? Does this match training?
  4. Can you disentangle “RAE encoder semantics” from “RAE decoder capacity”? In Table 3, a VAE-latent target with RAE decoder may already explain part of the gain.
  5. Why not report PSNR/LPIPS/IDS curves vs PiE strength ww in the main paper? At what ww does fidelity collapse?
  6. On real-world benchmarks, can you add a human preference study or ArcFace ID retrieval, rather than relying only on CLIP-IQA / MUSIQ / MANIQA?
  7. How sensitive is RAEFace to very strong degradation (heavy blur, tiny faces, occlusion) where LQ latent semantics may be wrong? Does PiE then extrapolate in a bad direction?
  8. For the inpainting/colorization results, were baselines re-run with the same LQ construction? CodeFormer is not an inpainting-specialized model, so the comparison may be unfair.
  9. The REPA loss uses HQ representation alignment—does this indirectly leak HQ semantics into LQ conditioning? If so, is the “LQ-only” semantic-access claim slightly overstated?
  10. Will code/configs for the exact PiE+CFG sampling schedule be released? The combination of CFG first then PiE replacement is non-standard and hard to reproduce from the text alone.
7 comments
AuthorsOpenReview ↗

Response to C9Kt

W1: Novelty beyond inherited components

RAE decoder training, DDT/flow-matching backbone, REPA alignment, cross-attention conditioning, and weak-to-strong extrapolation are all inherited from recent works. The core delta is “apply RAE to BFR + add LQ→HQ extrapolation.” The paper should state more honestly what is new at the mechanism level versus adaptation.

While acknowledging that RAEFace is deliberately simple and builds on established components, we would like to clarify that its research contribution goes beyond their combination. Our contribution includes the representation analysis, the unified framework it motivates, and its applicability across representations and multiple restoration tasks.

First, the technical framework is only one part of our contribution and equally important is our systematic analysis of restoration representations. Fig. 2 (right) shows that RAE representations retain more accessible facial semantics across HQ and LQ inputs. The oracle study across eight RAE and five VAE variants (Fig. 3(c)) excludes base-prediction errors and shows that the same extrapolation operation behaves differently across latent spaces. Fig. 4 further links perceptual gains more strongly to HQ–LQ semantic compatibility than to spatial similarity or interpolation quality. These findings provide empirical guidance for representation selection and perceptual refinement.

Second, these insights inform a unified framework for faithful restoration and perceptual enhancement. Fig. 1 shows that substituting RAE for VAE alone preserves facial semantics but leaves a perceptual gap. We therefore model degraded conditions and HQ targets in a shared semantic space, combining a faithful base prediction with perceptual refinement. PiE adapts the existing weak-to-strong construction by treating the degraded latent as the weak reference and the base prediction as the strong endpoint. The contribution lies in this formulation and its justification through representation analysis, rather than a claim that the individual operators are new.

Third, the framework’s simplicity supports its applicability beyond a single backbone or BFR alone. We evaluate different pretrained representations in Sec. 4.3 and apply the same formulation to face colorization and inpainting in Sec. 4.4 and Fig. 9, without adding a separate task-specific refinement network. These results illustrate the practical value of a simple framework built around the semantic relationship between degraded and target representations.

W2: Identity fidelity and hallucination risks of PiE

Extrapolating along z^HQ−zLQ can easily be viewed as pushing the output toward “more restored-looking” latents that score higher on MANIQA/MUSIQ/CLIP-IQA without necessarily recovering ground-truth detail. The oracle study supports plausibility, but there is no evidence that PiE improves identity correctness or reduces hallucination on hard degradations. Reporting only small PSNR drops hides this trade-off.

High-quality BFR requires both fidelity and perceptual quality. A faithful base can still lack realistic detail. PiE therefore aims to enhance perceptual detail while limiting fidelity loss. It follows an intuitive weak-to-strong direction, z^HQ−zLQ\hat z_{\mathrm{HQ}}-z_{\mathrm{LQ}}, from the degraded latent to the restored prediction. We support this training-free choice empirically through the oracle study across eight RAE and five VAE variants (Fig. 3(c)) and the semantic-compatibility analysis (Fig. 4). To assess the identity cost and perceived quality beyond no-reference metrics, we have added an IDS sweep and a user preference study. Selected results are reported below, and extended sweeps for the three RAE encoders are provided in Appendix B, Fig. 11 of the revised version.

IDS (↑\uparrow) across PiE strengths.

PiE strength (ww)DINOv3SigLIP2MAEVAE
00.8680.8300.8710.870
0.050.8620.8240.8660.854
0.10.8530.8150.8570.822
0.150.8420.8060.8430.784
0.20.8290.7940.8250.740

At w=0.1w=0.1, DINOv3 improves MANIQA by 0.1687, with an IDS decrease of only 0.015 (0.868 → 0.853). At the same strength, the evaluated VAE improves MANIQA by 0.0044, while incurring a larger IDS decrease of 0.048 (0.870 → 0.822). Similar trends are observed in other no-reference metrics.

We further investigate whether human preference for PiE-enhanced textures is accompanied by greater consistency with the reference facial features. Given the LQ input, the GT reference, and two anonymized restoration results, participants made separate forced-choice judgments of texture richness and naturalness, and the consistency of visible facial features with the GT. The table below reports the proportion of judgments favoring PiE over w=0w=0.

PiE strength (ww)Texture-quality preferenceFacial-feature consistency preference
0.06565.63%58.33%
0.168.75%66.67%
0.243.75%35.42%
0.336.46%19.79%

Our HQ-referenced user study further shows that, at w=0.065w=0.065 and w=0.1w=0.1, participants preferred PiE over the base restoration in both texture quality and visible facial-feature consistency with the GT. These findings provide human evidence beyond no-reference quality metrics, supporting improvements in perceived detail alongside favorable judgments of reference facial-feature consistency under moderate PiE enhancement. The results are provided in Appendix C, Fig. 13(b) of the revised paper, with the questionnaire design and instructions in Appendix H, Fig. 20.

The same refinement formulation also extends to inpainting (see Appendix F.3, Fig. 17).

W3: Evidence beyond no-reference quality metrics

On CelebA-Test (Table 2), RAEFace is not the best on PSNR (25.14 CodeFormer vs 24.34 yours) and only marginally better on LPIPS. Real-world tables rely entirely on no-reference metrics + FID. For a fidelity-claiming method, this is weak; at least IDS, identity retrieval, or user-study pairwise preference against the strongest baselines should be emphasized more carefully.

RAEFace ranks third in both PSNR and IDS, while placing first or second on perceptual metrics (Table 2), offering competitive pixel and identity fidelity alongside strong perceptual quality. Recent BFR methods published in 2025 and 2026, including InterLCM, OSDFace, and GeoMAR, also do not lead in PSNR, reflecting their emphasis on visual quality and preservation of facial semantics and identity rather than pixelwise agreement alone. Our trade-off is adjustable: CFG alone reaches 25.11 dB PSNR (Table 4). To provide stronger evidence beyond no-reference metrics, we have added face recognition experiments on the challenging SCface surveillance dataset and a human preference study comparing restoration results.

As reported in Appendix E, Table 7, on SCface (distance 3, cameras 1–5), we evaluate all methods under the same recognition protocol, matching restored surveillance images against an HQ gallery containing the same identities. RAEFace achieves the highest Rank-1 and Rank-5 recognition accuracy among the compared methods.

MethodRank-1 ↑Rank-5 ↑
Ours94.62%98.92%
CodeFormer90.00%97.23%
InterLCM89.54%98.62%
OSDFace88.77%97.23%
GFPGAN86.15%97.23%
DAEFR63.85%85.38%
GeoMAR55.23%81.08%

Our human preference study on images from the evaluated benchmarks also shows that RAEFace receives the highest Rank-3 preference rate (77.84%) among the compared methods, providing complementary human evidence for overall restoration quality. The results are provided in Appendix C, Fig. 13(a) of the revised paper, with the questionnaire design and instructions in Appendix H, Fig. 19.

Here, Rank-kk denotes the cumulative proportion of judgments placing a method within the top kk.

MethodRank-1 ↑Rank-2 ↑Rank-3 ↑
Ours43.73%63.73%77.84%
OSDFace17.84%36.47%54.31%
InterLCM12.94%30.20%46.08%
CodeFormer8.24%24.71%38.82%
GFPGAN9.80%20.20%32.94%
GeoMAR5.49%14.71%24.12%
DAEFR1.96%10.00%25.88%

W4: Separating representation gains from decoder capacity

In Table 3, “VAE target + RAE condition” mixes latent spaces and decoders; gains may come from the RAE decoder rather than the semantic compatibility claim. A cleaner ablation would fix the decoder and only swap the latent/encoder, or fix the encoder and swap only the decoder.

To directly examine whether the gains are mainly caused by decoder capacity, we use the oracle sweep in Fig. 3(c). It compares the PiE gains under a matched 0.4 PSNR budget across different representation–decoder pairs:

RepresentationDecoder parametersΔCLIPIQAΔMUSIQΔMANIQAΔTOPIQ
SD-VAE49.49M0.00500.2482−0.0047−0.0079
VA-VAE41.42M0.01941.24130.01710.0150
E2E-VA-VAE41.42M0.02031.19210.02020.0185

Within the VAE family, VA-VAE and E2E-VA-VAE obtain larger perceptual gains than SD-VAE despite having smaller decoders. This comparison helps separate decoder capacity from latent semantic organization: the stronger PiE response is consistently associated with the semantic organization of the latent space, rather than simply with a larger decoder.

W5: Noise augmentation and training–inference consistency

The paper says zlq=E(xlq)z_{lq} = E(x_{lq}), but the RAE formulation in Sec. 2 uses noise-augmented latents z~0. It is unclear whether the LQ condition is also noise-augmented, how this interacts with flow states zt, and whether training/inference condition statistics match. This matters because PiE explicitly uses zlqz_{lq} at test time.

The LQ condition is not noise-augmented during either restoration training or inference, and we use zLQ=E(xLQ)z_{\mathrm{LQ}}=E(x_{\mathrm{LQ}}) in both stages. The noise-augmented latent in Sec. 2 is used only for RAE decoder training. Along each flow trajectory, zLQz_{\mathrm{LQ}} remains fixed while the flow state ztz_t evolves. PiE uses this same unaugmented condition at test time. Thus, the conditioning protocol is consistent between training and inference, with no mismatch introduced by condition noise augmentation.

Comments on the paper

3 comments

Dear Reviewer,

With fewer than three days remaining in the discussion period, we would appreciate it if you could let us know whether our responses have adequately addressed your concerns. If so, we hope you will take these clarifications into account when revisiting your overall assessment of our work.

If any concerns remain or additional questions have arisen, please let us know. We will do our utmost to address them thoroughly within the remaining time.

Thank you again for your time, thoughtful feedback, and effort in reviewing our work.

Best regards,
The Authors

Dear Reviewers,

As the discussion period is nearing its end, we would like to take this opportunity to ensure that we have adequately addressed all your concerns. If you have any remaining questions, concerns, or suggestions, please feel free to let us know. We sincerely appreciate your valuable feedback and would be happy to clarify any outstanding issues to further improve our work.

Thank you again for your time and effort in reviewing our paper and engaging in the discussion.

Best regards,

The Authors

AuthorsOpenReview ↗

Summary of Rebuttal Updates and Core Contributions

Dear Area Chair,

We thank you and Reviewers EoQX, Hotu, and C9Kt for your careful evaluation of RAEFace. We revised the main text to reflect our rebuttal. First-round revisions are marked in red and second-round revisions in blue, and third-round revisions in green.

Below, we summarize the added experiments, analysis of PiE strength, and the role of latent representations in restoration.

  1. Strong restoration performance, generalization across restoration tasks, and robustness in downstream face recognition.
  • In a new end-to-end comparison, RAEFace outperforms FaithDiff on all four evaluated no-reference quality metrics across LFW, WebPhoto, Wider, and CelebA. The same restoration framework also extends to face colorization and inpainting without a task-specific refinement network. Additional experiments on real-world super-resolution provide evidence that its applicability is not limited to faces.
  • To assess whether restored faces remain useful for recognition, we reported results on the SCface surveillance benchmark. RAEFace achieves 94.62% Rank-1 retrieval accuracy, compared with 90.00% for CodeFormer under the same protocol. Unlike perceptual scores, retrieval tests whether facial identity is preserved.
  1. Deeper understanding of PiE strength and its practical use.
  • Following Reviewer EoQX’s suggestion, we extended the quantitative sweep to w=1.5w=1.5 and the visual comparisons to w=5.0w=5.0. We also added a reference-based user study (Appendix C, Fig. 13(b)). At w=0.1w=0.1, participants preferred PiE over the base restoration in 68.75% of judgments for texture quality and 66.67% for consistency of visible facial features with the reference. At w=0.2w=0.2, preference for facial-feature consistency fell to 35.42%, even as some no-reference scores continued to improve. This gives a practical reason to favor moderate refinement rather than the strength that maximizes a no-reference score.
  • The appropriate strength also depends on the representation. At the same w=0.1w=0.1, MANIQA improved by 0.1687 in the DINOv3 latent space, with a 0.015 decrease in identity similarity. In the evaluated VAE space, MANIQA improved by only 0.0044, while identity similarity decreased by 0.048. The same refinement operation can therefore yield substantially different trade-offs depending on the latent space in which it is applied.
  1. Core contribution: a systematic study of RAE latent spaces yields a simple method that works across restoration tasks.
  • To our knowledge, RAEFace is the first systematic study of representation autoencoders (RAEs) for low-level image restoration. Our oracle study compares eight RAEs and five VAEs while removing base-prediction error from the comparison. Together with Fig. 4, it shows that refinement gains are more closely associated with semantic compatibility between degraded and clean representations than with spatial similarity alone.
  • This finding motivates a simple design: RAEFace represents degraded inputs and clean targets in a shared semantic space, makes a faithful base prediction, and uses the degraded representation itself as the reference for refinement. The contribution is not a new extrapolation operator, but an account of when an existing operator works, why the representation matters, and how to use it across restoration tasks.

[Reviewer reassessment.]

Both reviewers who participated in the follow-up discussion initially recommended weak acceptance (3) and remained positive. After two rounds of discussion, Reviewer EoQX confirmed that the concern about the PiE strength range had been addressed. Although the reviewer retained a reservation about technical novelty relative to RAEv2, they specifically valued the RAE–VAE comparison, the expanded PiE analysis, and the extension to other restoration tasks, and remained open to further discussion. Reviewer Hotu reported no remaining concerns and raised their score from 3 to 4 (clear acceptance).

Reviewer C9Kt assigned a score of 1 and listed eight weaknesses and ten questions. Grouped by primary focus, eight items concerned clarification, presentation, or reproducibility, and ten concerned experiments or analysis. We responded to all 18 individually. Although we did not receive a follow-up, we believe the reference-based user study and SCface retrieval results directly address the concern about relying on no-reference metrics, while the decoder-capacity comparison and expanded failure analysis address the questions about the source and limits of PiE’s gains.

We believe that the methodological contribution, strengthened experimental evidence, and deeper analysis together make a strong case for acceptance. We are grateful to the reviewers for recognizing the value of the work and for helping us present a clearer account of where refinement helps and where it comes at the expense of fidelity. Thank you for your consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.