acceptodds
Under review as a conference paper at ICLR 2027

Super-Resolution in the Right Latent Space: A Frozen Vision-Foundation Substrate

10 views

Abstract

In an image latent space, the embeddings of high-resolution, natural, and sharp images form a manifold. Degradation of high-resolution images pushes their embeddings off this manifold. Real-world super-resolution (SR) then becomes the task of mapping the degraded embedding back onto this manifold — not anywhere on the manifold, but to the point that preserves what the input still carries, both its semantics and pixel details. Every published method implements this mapping in a reconstruction-oriented latent space or pixel space. We claim these spaces are the wrong substrates for SR. Low-resolution and degraded images are embedded far from the manifold, making the mapping difficult and expensive. The lack of semantic information in these substrates also makes it difficult to navigate to the faithful point on the manifold, causing severe hallucination when degradation is heavy. Thus, restoring in a suitable latent space is crucial to the SR task. We show that the latent space of fused layers of a frozen DINOv3-L is one such space that makes the SR task easier. Degraded images are embedded near the manifold. Moreover, this substrate contains a hierarchy of information, from pixel record to degradation robust semantics, guiding the model to find the faithful point on the manifold. On this substrate, a M decoder is trained under reconstruction and adversarial objectives to map the degraded embeddings back and decode to pixel space in one pass. The resulting model, RAESR, attains the best fidelity–perception trade-off among state-of-the-art adversarial and diffusion-based restorers on RealSR, DRealSR, LSDIR and DIV2K-Val, at ms per image on a single H20 GPU. Swapping the substrate for a VAE latent under an identical recipe loses on every metric.

Primary area:
generative models
Keywords:
Real-World Image Super-Resolution, Vision Foundation Models, GANs, DINOv3, RAE, Perceptual Quality

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_zKNROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents RAESR, a one-pass framework for real-world SR built upon frozen DINOv3-L representations. Motivated by the limitations of pixel-space and VAE-based restoration, the authors adopt a 23-layer fused representation as the restoration substrate and train a 415M-parameter Transformer decoder using reconstruction, perceptual, and adversarial objectives. A multi-level DINOv3-based discriminator is employed to improve perceptual quality.Experiments on RealSR, DRealSR, LSDIR, and DIV2K-Val demonstrate competitive fidelity–perception trade-offs and inference efficiency. The paper further investigates the proposed substrate through latent interpolation, layer-wise ablations, VAE substitution, and human preference evaluations.

Critical strengths
  1. RAESR achieves competitive perceptual quality across four benchmarks, outperforming the evaluated baselines on DISTS, NIQE, and MANIQA while requiring only 37.2 ms per 512×512 image. These results demonstrate the practical potential of frozen vision-foundation representations for efficient real-world SR.

  2. Beyond benchmark comparisons, the paper investigates latent interpolation, layer-wise information recovery, feature ablations, and substrate substitution. In particular, the distinct effects of shallow and deep features provide meaningful empirical evidence for their complementary roles in fidelity and perceptual quality. The detailed experimental protocols further strengthen the paper.

Critical weaknesses
  1. The frozen representation encoder, multi-layer feature aggregation, and pretrained decoder initialization largely build upon RAE/RAEv2. Moreover, prior works such as VOSR and F2IDiff have explored vision-foundation features for SR guidance. While RAESR differs by using these features as the restoration substrate, the paper does not establish whether this design offers advantages over using comparable features as conditioning within pixel- or VAE-space restoration. Consequently, the independent contribution of the proposed formulation remains unclear.

  2. The DINOv3-versus-SD3-VAE comparison does not fully control for representation capacity or initialization: the two substrates differ by a factor of 16 in latent dimensionality, and the VAE input projection is reinitialized. Furthermore, improved latent interpolation curves do not directly establish shorter manifold distances or lower restoration complexity, particularly since RAESR predicts images directly rather than explicitly optimizing a latent restoration trajectory. These limitations weaken the central claim that the substrate itself makes SR inherently easier.

Questions for authors
  1. Can the authors provide more controlled comparisons that account for latent dimensionality and decoder initialization? In particular, does using DINO features as the restoration substrate offer measurable advantages over using the same features as conditioning under comparable architectures and training budgets?

  2. Is there direct evidence that the proposed substrate simplifies the restoration mapping, beyond improved latent interpolation curves? Comparisons of training convergence, decoder capacity requirements, or computational budgets across substrates would help substantiate this central claim.

5 comments
AuthorsOpenReview ↗

Response to Reviewer zKNR (Part 1)

We appreciate the reviewer's careful and constructive review, and for recognizing the complementary roles of the shallow and deep features and the detail of our methods. The revised PDF adds the two controls requested in Question 1 (Fig. 8, Sec. 5.3). The reviewer also asks for more direct evidence that the restoration mapping is easier to learn on this substrate, for example through training convergence, decoder capacity or compute. We are currently studying this and running experiments on it now, and we will update the results in this thread as soon as possible. We answer each concern below:

1. Our Contributions are Beyond Reusing RAE/RAEv2 (W1)

"The frozen representation encoder, multi-layer feature aggregation, and pretrained decoder initialization largely build upon RAE/RAEv2."

We did reuse RAEv2's simple framework but our contribution lies elsewhere. RAE and RAEv2 build this latent for generation and clean reconstruction. We show that it is a better substrate for restoration than reconstruction-oriented latents (Sec. 5, and points 2 and 3 below), and we turn it into a restorer with a new training design, a one-pass decoder trained against a dense, position-paired critic on DINOv3-B (Sec. 3.3, 3.4). The architecture stays as simple as RAEv2's. But we see this simple design as a strength. It is what makes RAESR efficient, one pass at 37 ms per image with 719M parameters (Table 1).

2. Latent width and decoder initialization (W2, Q1)

"the two substrates differ by a factor of 16 in latent dimensionality, and the VAE input projection is reinitialized"

VAE based reconstruction still trails behind RAESR even if the latent dimension and input projection are matched. We widened the SD3-VAE latent from 16 to 256 channels to increase its capacity. Only two layers change shape: the last convolution of the VAE encoder and the first convolution of its own decoder. All other weights are copied. The decoder weights that read the 240 new channels start at zero, so the widened VAE initially reproduces the original. We then fine-tuned the VAE for 5,000 steps on reconstruction (full protocol in App. H). After this change, the VAE latent has the same size as the DINOv3-L latent, and no layer of the VAE encoder is narrower than it.

On 128 held-out images, the widened VAE reconstructs better than the original (PSNR 32.10 vs. 30.86 dB, LPIPS 0.053 vs. 0.067). The original 16-channel VAE fine-tuned with the same budget reaches only 30.99 dB and 0.073, so the gain comes from the width, not the extra training. Zeroing out the 240 new channels raises LPIPS to 0.087, so the extra capacity does carry more information.

VAE (128 held-out images)PSNR$\uparrow$LPIPS$\downarrow$
SD3-VAE, 16 channels30.860.0673
SD3-VAE, 16 channels, fine-tuned 5,000 steps30.990.0733
Widened VAE, 256 channels, fine-tuned 5,000 steps32.100.0527
Widened VAE, 240 new channels zeroed–0.0868

Pixel-unshuffling turns the widened latent into 1,024 tokens of width 1,024, exactly the DINOv3-L shape. The SR decoder therefore keeps its input projection, and no weight is re-initialized. Trained with the same initialization and three-stage recipe, this substrate is no better than the original VAE and trails DINOv3-L on 11 of the 12 metrics (table below, Sec. 5.3, Fig. 8, App. H). A higher-capacity VAE does not make a better restoration substrate, even if the input projection is matched.

SubstratePSNR ↑SSIM ↑LPIPS ↓DISTS ↓NIQE ↓MUSIQ ↑MANIQA ↑TOPIQ ↑CLIP-IQA ↑DeQA ↑Q-Align ↑Q-Insight ↑
DINOv3-L (RAESR)23.720.62650.30120.19714.32070.450.67170.68200.70593.9374.0663.791
SD3-VAE (64)23.610.60330.37340.24954.34469.760.63000.66320.70113.7403.8013.687
SD3-VAE (1024)23.530.59840.37970.25544.32169.820.62690.65900.72923.6793.7473.676

The sample images in Fig. 8 show the failure directly. Both VAE widths repaint the harbor boats as rocks and reduce the handwritten price sign to something unrecognizable, while RAESR keeps both. Fig. 11 adds two more examples, where the VAE substrates replace the moss between roof tiles with glossy tile surface and add spurs and kinks to letter strokes.

AuthorsOpenReview ↗

Response to Reviewer zKNR (Part 2)

This part continues our response from Part 1.

3. Advantage of DINOv3-L as Substrate over as Conditioning (W1, Q1)

"does using DINO features as the restoration substrate offer measurable advantages over using the same features as conditioning under comparable architectures and training budgets?"

Yes. We thank the reviewer for this constructive question. Starting from the width-matched VAE outlined in the previous comment, we let the same decoder read the same frozen DINOv3-L latent through cross-attention after blocks 4, 11, 18 and 25 of its 28 layers. Queries come from the decoder tokens and keys and values from the DINOv3-L tokens. Each block is gated by tanh(g) with g initialized at zero, so training starts from the plain VAE model. These blocks add 20.1M parameters. Both encoders stay frozen and run at inference, and initialization, data and recipe match the other models. This gives the features the role they play in VOSR and F2IDiff, inside our architecture and budget (Sec. 5.3, full architecture and details in App. H).

Conditioning improves the VAE substrate as is shown in the table below. Yet the conditioned model still underperforms the DINOv3-L substrate on 11 of the 12 metrics.

SubstratePSNR ↑SSIM ↑LPIPS ↓DISTS ↓NIQE ↓MUSIQ ↑MANIQA ↑TOPIQ ↑CLIP-IQA ↑DeQA ↑Q-Align ↑Q-Insight ↑
DINOv3-L (RAESR)23.720.62650.30120.19714.32070.450.67170.68200.70593.9374.0663.791
SD3-VAE (64)23.610.60330.37340.24954.34469.760.63000.66320.70113.7403.8013.687
SD3-VAE (1024)23.530.59840.37970.25544.32169.820.62690.65900.72923.6793.7473.676
SD3-VAE (1024) + DINO23.680.61390.32360.21474.14769.820.64390.65600.67933.8053.8943.728

The same latents are worth more as substrate than as a side condition. The images agree. In Fig. 8, conditioning brings the harbor boats back but distorts them, and the price sign loses its form. Table 1 points the same way. VOSR-1.4B, which conditions its generator on vision-foundation features, underperforms RAESR on all twelve metrics in both its multi-step and one-step versions, at 2.4× the parameters. This is the independent contribution of the substrate formulation.

4. Direct evidence that restoration is easier on this substrate (W2, Q2)

"improved latent interpolation curves do not directly establish shorter manifold distances or lower restoration complexity"

The new controls provide this evidence at equal capacity and compute. The interpolation curves show where degraded embeddings sit, while the controls above test the mapping itself. All four models share the same 415M decoder, initialization and three-stage training budget, and with capacity and compute held fixed, the DINOv3-L substrate restores best, on 10 of the 12 metrics. For training convergence, we are running a controlled comparison across substrates and will post the results in this thread as soon as they are available.

We hope these comparisons address the reviewer's concerns. If anything remains unclear, we would be glad to clarify or provide further results during the discussion period. If our responses resolve the concerns, we would appreciate it if the reviewer would reconsider the evaluation in light of the new results.

AuthorsOpenReview ↗

Response to Reviewer zKNR (Part 3)

This part completes point 4 of Part 2 with the training-convergence comparison. The revised PDF now includes it (Sec. 5.1, Fig. 6 bottom row, App. J).

5. Training convergence at equal decoder capacity (W2, Q2)

"Comparisons of training convergence, decoder capacity requirements, or computational budgets across substrates would help substantiate this central claim."

When decoder capacity and training budget are held constant, test quality improves much faster on the DINOv3-L substrate. We train two decoders of the same size, one on each substrate, and score both on the same test set during training.

  • Decoders: On DINOv3-L, we use the official RAEv2 decoder, which has 415M parameters. The VAE decoder must match this size. The SD3 VAE decoder in Sec. 5.3 has only 49.5M parameters, so we use the VAE of Qwen-Image-2.1 instead, whose decoder has 259M parameters. We widen its latent to 1,024 channels, the token width of DINOv3-L, and enlarge its decoder to the same 415M parameters. Every added block is initialized as the identity, and neither step costs reconstruction quality. (App. J.1).
  • Training: Each decoder starts from the reconstruction weights pretrained for its own latent. Both are then trained in the same way, with frozen encoders and the same data, degradations, and optimizer settings. The losses are L1 and LPIPS only. A run takes 30,000 steps at batch size 16 and is repeated with ten seeds (App. J.2).
  • Evaluation: We score 16 checkpoints of every run on 393 held-out images from the four benchmarks.
  • Resolution: Training and evaluation use 256 px crops. The official RAEv2 decoder is released for 256 px inputs, so we use it unchanged. The RAESR decoder in the paper is adapted to 512 px, and running at 256 px keeps this adaptation out of the comparison. We repeated the same comparison at 512 px and reproduced the same phenomenon. The curves are in App. J.4 (Fig. 13).

The curves are in Fig. 6 (bottom row) for LPIPS, DISTS, Q-Align and MANIQA, and in Fig. 12 for the other six metrics. Both figures show the mean and the standard deviation over the ten seeds. Table 12 lists the final scores.

The two substrates start from almost the same scores, because at step 0 both decoders still reconstruct the degraded input. DINOv3-L then reaches the final (30,000-step) score of the VAE by step 4,000 on LPIPS, by step 3,000 on DISTS and by step 2,000 on the other eight metrics (Table 12). It remains better than that score for the rest of training.

With decoder size, training signal and schedule matched, the DINOv3-L substrate reaches the final test quality of the VAE substrate in far fewer steps. This is direct evidence that the restoration mapping is easier to learn on this substrate.

We hope this answers Question 2. If anything remains unclear, we would be glad to clarify during the discussion period.

Reviewer_6cJROpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes RAESR, a real-world image super-resolution model that operates in a fused latent space formed from 23 layers of a frozen DINOv3-L encoder. A 415M Transformer decoder maps degraded embeddings to high-resolution images in one pass. The paper analyzes why this substrate is suitable, compares it against VAE latents under a matched training recipe, and reports strong fidelity–perception and efficiency–perception trade-offs on RealSR, DRealSR, LSDIR, and DIV2K-Val.

Critical strengths
  1. The paper presents a clear and compelling view that the choice of latent substrate is central to real-world super-resolution. The analysis of the fused DINOv3 representation, including its hierarchy from low-level detail to robust semantics, provides a useful conceptual contribution beyond proposing another restoration architecture.

  2. The experimental evaluation is comprehensive and well controlled. The matched comparison with a VAE latent, multi-benchmark results, one-pass inference design, and explicit latency and parameter reporting make the practical benefits of RAESR convincing.

  3. It is particularly impressive that the method does not rely on a pretrained diffusion prior. With the DINOv3 latent substrate and an RAEv2-initialized decoder trained with reconstruction and adversarial objectives, it achieves strong super-resolution performance with a comparatively simple one-pass design.

Critical weaknesses

The no-reference quality evaluation relies mainly on conventional metrics such as NIQE, MUSIQ, and MANIQA. Adding recent LLM-based quality metrics, such as DeQA, QAlign, or Q-Insight, would provide a more semantic and human-aligned assessment of restoration quality and could make the conclusions more robust.

Questions for authors

I am very curious why replacing the latent substrate and initializing the decoder from RAEv2, followed by reconstruction and adversarial training without a pretrained diffusion prior, is sufficient to achieve such strong results. A deeper explanation of this phenomenon would make the paper more insightful.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 6cJR

We appreciate the reviewer's careful and constructive review, and for recognizing the substrate view, the matched comparison with a VAE latent, and the one-pass design without a pre-trained diffusion prior. The revised PDF adds the three LLM-based quality metrics for all methods (Table 1, Tables 4 to 7). We answer each point below:

1. LLM-based quality metrics (W1)

"Adding recent LLM-based quality metrics, such as DeQA, QAlign, or Q-Insight, would provide a more semantic and human-aligned assessment of restoration quality and could make the conclusions more robust."

We added all three metrics and RAESR ranks first on each. We scored all 14 methods in Table 1 on the same 3,443 images. We use their public checkpoints with the official prompts and preprocessing (App. B.1). Scores range from 1 to 5, and the table reports the mean over the four benchmarks.

MethodDeQA ↑Q-Align ↑Q-Insight ↑
RAESR3.93204.05433.7874
RAESR-D (distilled)3.91234.04013.7698
Real-ESRGAN3.13423.20253.3169
Real-HAT-GAN2.84482.87383.1129
ResShift3.18573.07973.2896
VOSR-ms3.37673.51823.5752
VOSR-os3.48193.66553.6567
AdcSR3.71463.80993.7095
StableSR3.29363.36253.4552
DiffBIR3.92873.99553.7539
OSEDiff3.63613.75523.6856
TSD-SR3.68243.79493.6982
DiT4SR3.56403.62503.6696
SUPIR3.60053.59753.5042

Per benchmark statistics are displayed in Tables 4 to 7. In the substrate comparison, the DINOv3-L substrate also scores highest on all three (Fig. 8).

2. Why no pre-trained diffusion prior is needed (Q1)

"I am very curious why replacing the latent substrate and initializing the decoder from RAEv2, followed by reconstruction and adversarial training without a pre-trained diffusion prior, is sufficient to achieve such strong results."

RAESR is not entirely prior-free. Pre-trained diffusion models provide restorers with three crucial elements, all of which have counterparts in RAESR. The first two are semantic and natural-image priors. In RAESR, the semantic prior comes from the frozen pre-trained encoder, and the natural-image prior from the pre-trained decoder it starts from. The third is iterative sampling, which RAESR replaces with one pass of an adversarially trained decoder.

  • Semantic Prior: DINOv3 is trained on 1.7 billion images. Its student network takes masked patches along with small local crops of an image, and is trained to predict the representation that a teacher network gives to larger, unmasked crops of the very same picture. Like denoising, this objective recovers complete content from a corrupted or partial input, but in representation space. Because the encoder is kept as-is, it contains this crucial semantic prior and supplies it after just one forward pass, without sampling. REPA (Yu et al., ICLR 2025) notes that denoising produces meaningful representations inside diffusion models, but these still lag behind self-supervised ones, and argues that learning them is a main bottleneck of diffusion training. RAESR, alternatively, directly accesses the representation through its pre-trained encoder.

  • Natural-image Prior: This prior comes from RAEv2's own decoder, which RAESR starts from. It is pre-trained to reconstruct images out of similar latent representations, so it already renders a clean latent into a natural image. RAESR's training then only has to learn the map from the degraded latent back to the manifold. The critic guiding this map works on the features of a frozen DINOv3-B encoder, and Projected GANs (Sauer et al., NeurIPS 2021) demonstrate that a discriminator built on pre-trained features makes adversarial training faster and more sample-efficient.

  • Iterative Sampling: Restoration is one-to-many, and iterative sampling narrows the plausible outputs step by step until it selects one of them. In RAESR, the adversarial loss makes one decoder pass commit to a sharp output, and the semantic tokens make that output the correct one. Fig. 8 separates the two (Sec. 5.3). On the original and width-matched VAE latents, the same pass gives nearly the same NIQE (4.344 and 4.321 against 4.320) but a much worse LPIPS (0.3734 and 0.3797 against 0.3012). Giving the width-matched VAE model the DINOv3-L latent as conditioning recovers part of this gap (LPIPS 0.3236).

In addition to these priors, the restoration task itself is easier on this substrate, as analyzed in the paper. The map back to the manifold is shorter and better guided (Sec. 5.1 and 5.2).

We hope these additions address your points. If anything remains unclear, please let us know during the discussion period, and we will be glad to clarify or provide further results. If our responses resolve your concerns, we would appreciate it if you would reconsider your evaluation in light of the new results.

Thank you for the authors' response. My concerns have been addressed, and I will increase my score to 4.

I think the observation in this work is somewhat similar to that in VQFR, which finds that more compressed latent representations tend to produce reconstructions closer to HQ images. Similarly, this work shows that semantic-rich latent representations can bring degraded-image embeddings closer to the HQ manifold. Both suggest that the choice of latent space plays an important role in image restoration.

Overall, I think this is a good paper with an interesting perspective.

AuthorsOpenReview ↗

Follow-up Response to Reviewer 6cJR

Thank you for reading our response and for raising your score. We are glad that it addressed your concerns.

Thank you also for pointing out the connection to VQFR (Gu et al., ECCV 2022). We agree. VQFR finds that a codebook with a larger compression patch size reconstructs a degraded face more closely to a high-quality face, at a cost in fidelity. We find that the fused DINOv3-L latent embeds degraded images near the manifold of clean images (Sec. 5.1). Both results show that the choice of latent space matters for restoration.

The two differ in where this closeness comes from. In VQFR, it comes from compressing the face into codes learned from high-quality faces. In our work, it comes from the self-supervised pre-training of the encoder, and the latent still retains the low-level pixel information (Sec. 3.1).

The revised PDF now cites VQFR and discusses this connection (App. D). Thank you again for your time and your constructive review.

Reviewer_63UKOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RAESR, a one-pass method for real-world SR. It uses fused DINOv3-L features as the input representation and trains a Transformer decoder to produce the high-resolution image. The authors support that the fused DINOv3 representation provides a more suitable starting point and richer guidance.

Critical strengths
  1. The paper compares multiple methods across four benchmarks, reports runtime, and performs the sufficient ablation studies.

  2. The paper shifts attention from the choice of generator to the representation in which restoration is performed. This is an interesting and useful research perspective.

Critical weaknesses
  1. The DINOv3 and SD3-VAE representations differ substantially in dimensionality. In specific, the VAE latent is rearranged into tokens of width 64, whereas DINOv3 tokens have width 1024. The input projections also have different shapes. Thus, the observed difference may reflect representation capacity or optimization difficulty, not only the semantic hierarchy emphasized by the paper.

  2. The paper does not report the number of human participants and the number of valid judgments per comparison.

  3. The paper states that the Figure 7 ablations on the right are performed at inference time with the decoder frozen. So the model is not retrained for each ablation setting. The decoder is trained on the full feature set. Is it fair to compare ablated features without retraining the decoder for each setting?

Questions for authors
  1. Could you provide stronger controls for the DINOv3 compared to SD3-VAE?

  2. Could you provide results with the decoder retrained or fine-tuned for each ablation, or explain why inference-only ablations are sufficient to support the claims?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 63UK (Part 1)

We appreciate the reviewer's careful and constructive review, and for recognizing the shift of attention from the generator to the representation in which restoration is performed as an interesting and useful research perspective. The revised PDF adds a stronger control for the DINOv3-L versus SD3-VAE comparison (Sec. 5.3, Fig. 8, App. H) and reports the human study counts (Table 2, App. A). We answer each concern below:

1. Stronger controls for DINOv3-L versus SD3-VAE (W1, Q1)

"the observed difference may reflect representation capacity or optimization difficulty, not only the semantic hierarchy emphasized by the paper"

VAE based reconstruction still trails behind RAESR even if the latent dimension and input projection are matched. We widened the SD3-VAE latent from 16 to 256 channels to increase its capacity. Only two layers change shape: the last convolution of the VAE encoder and the first convolution of its own decoder. All other weights are copied. The decoder weights that read the 240 new channels start at zero, so the widened VAE initially reproduces the original. We then fine-tuned the VAE for 5,000 steps on reconstruction (full protocol in App. H). After this change, the VAE latent has the same size as the DINOv3-L latent, and no layer of the VAE encoder is narrower than it.

On 128 held-out images, the widened VAE reconstructs better than the original (PSNR 32.10 vs. 30.86 dB, LPIPS 0.053 vs. 0.067). The original 16-channel VAE fine-tuned with the same budget reaches only 30.99 dB and 0.073, so the gain comes from the width, not the extra training. Zeroing out the 240 new channels raises LPIPS to 0.087, so the extra capacity does carry more information.

VAE (128 held-out images)PSNR ↑LPIPS ↓
SD3-VAE, 16 channels30.860.0673
SD3-VAE, 16 channels, fine-tuned 5,000 steps30.990.0733
Widened VAE, 256 channels, fine-tuned 5,000 steps32.100.0527
Widened VAE, 240 new channels zeroed–0.0868

Pixel-unshuffling turns the widened latent into 1,024 tokens of width 1,024, exactly the DINOv3-L shape. The SR decoder therefore keeps its input projection, and no weight is re-initialized. Trained with the same initialization and three-stage recipe, this substrate is no better than the original VAE and trails DINOv3-L on 11 of the 12 metrics (table below, Sec. 5.3, Fig. 8, App. H). A higher-capacity VAE does not make a better restoration substrate, even if the input projection is matched.

SubstratePSNR ↑SSIM ↑LPIPS ↓DISTS ↓NIQE ↓MUSIQ ↑MANIQA ↑TOPIQ ↑CLIP-IQA ↑DeQA ↑Q-Align ↑Q-Insight ↑
DINOv3-L (RAESR)23.720.62650.30120.19714.32070.450.67170.68200.70593.9374.0663.791
SD3-VAE (64)23.610.60330.37340.24954.34469.760.63000.66320.70113.7403.8013.687
SD3-VAE (1024)23.530.59840.37970.25544.32169.820.62690.65900.72923.6793.7473.676
SD3-VAE (1024) + DINO23.680.61390.32360.21474.14769.820.64390.65600.67933.8053.8943.728

When we further provide this widened VAE model the DINOv3-L latent, injected via cross-attention, it improves (Sec. 5.3, Fig. 8 and full architecture and details in App. H ). Thus, what the VAE substrate lacks is the information in the DINOv3-L latent instead of capacity.

The sample images in Fig. 8 show the failure directly. Both VAE widths repaint the harbor boats as rocks and reduce the handwritten price sign to something unrecognizable, while RAESR keeps both. Fig. 11 adds two more examples, where the VAE substrates replace the moss between roof tiles with glossy tile surface and add spurs and kinks to letter strokes.

AuthorsOpenReview ↗

Response to Reviewer 63UK (Part 2)

This part continues our response from Part 1.

2. Human study: participants and valid judgments (W2)

"The paper does not report the number of human participants and the number of valid judgments per comparison."

Table 2 now reports both, and App. A describes the full protocol. 40 participants contribute to each of the four comparisons.›

ComparisonParticipantsValid judgmentsRAESRBaselineTieRAESR preference
RAESR vs. Real-HAT-GAN40215163262686.2%
RAESR vs. DiffBIR40218133553070.7%
RAESR vs. SUPIR40211109802257.7%
RAESR vs. TSD-SR40219101912752.6%

Two quality controls act on participants instead of on each judgement. For each participant, the session has 25 trials, three of which are catch trials that compare the ground truth with the bicubically upsampled input. A participant who fails more than one catch trial, or whose median response time is under 3 seconds, is excluded together with all his/her responses. We intentionally apply the time rule to a participant's median and do not filter single fast trials. Some comparisons are obvious and can be judged within two seconds, so removing each fast trial would discard exactly the clearest comparisons and bias the preference rates. Three participants were excluded, two by the time rule and one by the catch trials. Of the remaining 40, 38 completed all 25 trials and 2 stopped early after passing both rules. Each trial draws its image and its baseline at random, so the number of valid judgments differs slightly between comparisons (863 in total, catch trials excluded). The preference rate excludes ties.

3. Fig. 7 ablations with a frozen decoder (W3, Q2)

"Is it fair to compare ablated features without retraining the decoder for each setting?"

The ablation in Fig. 7 studies how the trained decoder uses each level of its input, so the decoder should stay fixed. The purpose of this ablation is to see how the trained SR model uses the information from different layer bands. We replace one band of layers, the shallow one or the deep one, with its training-set mean. This removes the information of that band, and we observe how the output changes. The comparison is relative. For each band, we look at whether fidelity or perception changes relative to the unmodified model.

Replacing a band does shift the input distribution, but every ablation setting has this shift, so it is uniform across the settings we compare. We also reduce the shift. We do not delete the shallow or the deep layers but replace them with their training-set mean, which matches the statistics the decoder saw during training (App. E). Random eight-layer bands replaced in the same way show no such pattern (App. E). Thus, the effect comes from which layers are removed instead of the distribution shift.

Fine-tuning the decoder for each setting would introduce more uncertainty. Once the input changes, the same training settings will lead to different training outcomes. Thus, we could not tell whether a difference comes from the removed layers or from the training itself. This holds especially for adversarial training, which is hard to tune and whose effect under a different input is hard to quantify. Fine-tuning would also answer a different question, namely how well a new model performs after adapting to a reduced input.

We hope these additions address the reviewer's concerns. If anything remains unclear, we would be glad to clarify or provide further results during the discussion period. We would also be grateful for any pointers on where the presentation can be improved. If our responses resolve the concerns, we would appreciate it if the reviewer would reconsider the evaluation in light of the new results.

I thank the authors for their response. All my concerns have been satisfactorily addressed. I will maintain my weak accept recommendation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

We thank all reviewers for their careful and constructive reviews and advice. We are encouraged that the reviewers found the substrate perspective valuable and the results strong across four benchmarks at a low inference cost. We have uploaded a revised PDF. This comment concisely lists what we added and where to find it.

New experiments

  1. MLLM-based quality metrics We additionally evaluate all 14 methods of Table 1 with DeQA-Score, Q-Align and Q-Insight. RAESR ranks first on all three. The results are in Table 1, the per-benchmark results in Tables 4 to 7, and the evaluation protocol in App. B.1.
  2. New Ablation Study on Widened SD3-VAE We widened the SD3-VAE latent to the shape of the DINOv3-L latent and trained the SR decoder on it with the same recipe (Sec. 5.3, Fig. 8, App. H) and compared it against the RAESR model.
  3. DINOv3-L as conditioning. We kept this VAE as the substrate and gave the decoder the DINOv3-L latent through cross-attention, trained with the same recipe (Sec. 5.3, Fig. 8, App. H) and compared it against the RAESR model.

With these two experiments, Fig. 8 now compares the DINOv3-L substrate against three VAE-based models on twelve metrics: the original SD3-VAE, the widened SD3-VAE, and the widened SD3-VAE with DINOv3-L conditioning. The DINOv3-L substrate is best on 10 of the 12.

  1. Training convergence on the two substrates (new in the latest revision). We train two decoders of the same size in the same way, one on the DINOv3-L substrate and one on a VAE substrate, each with ten seeds, and compare their test metrics during training (Sec. 5.1, Fig. 6 bottom row, App. J). The DINOv3-L substrate improves much faster.

Other additions

  1. Human study. Table 2 now reports the number of participants and valid judgments for each comparison, and App. A describes the full protocol and quality control.
  2. Sample images. Fig. 8 and Fig. 11 (App. H) show new samples for all four models, each with a zoomed region.
  3. Experiment details. App. H documents the two substrate experiments in full, including the reconstruction quality of the widened VAE (Table 10) and the complete scores (Table 11). App. J documents the convergence experiment, including the final scores (Table 12) and the curves of the remaining metrics (Fig. 12).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.