- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents RAESR, a one-pass framework for real-world SR built upon frozen DINOv3-L representations. Motivated by the limitations of pixel-space and VAE-based restoration, the authors adopt a 23-layer fused representation as the restoration substrate and train a 415M-parameter Transformer decoder using reconstruction, perceptual, and adversarial objectives. A multi-level DINOv3-based discriminator is employed to improve perceptual quality.Experiments on RealSR, DRealSR, LSDIR, and DIV2K-Val demonstrate competitive fidelity–perception trade-offs and inference efficiency. The paper further investigates the proposed substrate through latent interpolation, layer-wise ablations, VAE substitution, and human preference evaluations.
Critical strengths
-
RAESR achieves competitive perceptual quality across four benchmarks, outperforming the evaluated baselines on DISTS, NIQE, and MANIQA while requiring only 37.2 ms per 512×512 image. These results demonstrate the practical potential of frozen vision-foundation representations for efficient real-world SR.
-
Beyond benchmark comparisons, the paper investigates latent interpolation, layer-wise information recovery, feature ablations, and substrate substitution. In particular, the distinct effects of shallow and deep features provide meaningful empirical evidence for their complementary roles in fidelity and perceptual quality. The detailed experimental protocols further strengthen the paper.
Critical weaknesses
-
The frozen representation encoder, multi-layer feature aggregation, and pretrained decoder initialization largely build upon RAE/RAEv2. Moreover, prior works such as VOSR and F2IDiff have explored vision-foundation features for SR guidance. While RAESR differs by using these features as the restoration substrate, the paper does not establish whether this design offers advantages over using comparable features as conditioning within pixel- or VAE-space restoration. Consequently, the independent contribution of the proposed formulation remains unclear.
-
The DINOv3-versus-SD3-VAE comparison does not fully control for representation capacity or initialization: the two substrates differ by a factor of 16 in latent dimensionality, and the VAE input projection is reinitialized. Furthermore, improved latent interpolation curves do not directly establish shorter manifold distances or lower restoration complexity, particularly since RAESR predicts images directly rather than explicitly optimizing a latent restoration trajectory. These limitations weaken the central claim that the substrate itself makes SR inherently easier.
Questions for authors
-
Can the authors provide more controlled comparisons that account for latent dimensionality and decoder initialization? In particular, does using DINO features as the restoration substrate offer measurable advantages over using the same features as conditioning under comparable architectures and training budgets?
-
Is there direct evidence that the proposed substrate simplifies the restoration mapping, beyond improved latent interpolation curves? Comparisons of training convergence, decoder capacity requirements, or computational budgets across substrates would help substantiate this central claim.
5 comments
Response to Reviewer zKNR (Part 2)
This part continues our response from Part 1.
3. Advantage of DINOv3-L as Substrate over as Conditioning (W1, Q1)
"does using DINO features as the restoration substrate offer measurable advantages over using the same features as conditioning under comparable architectures and training budgets?"
Yes. We thank the reviewer for this constructive question. Starting from the width-matched VAE outlined in the previous comment, we let the same decoder read the same frozen DINOv3-L latent through cross-attention after blocks 4, 11, 18 and 25 of its 28 layers. Queries come from the decoder tokens and keys and values from the DINOv3-L tokens. Each block is gated by tanh(g) with g initialized at zero, so training starts from the plain VAE model. These blocks add 20.1M parameters. Both encoders stay frozen and run at inference, and initialization, data and recipe match the other models. This gives the features the role they play in VOSR and F2IDiff, inside our architecture and budget (Sec. 5.3, full architecture and details in App. H).
Conditioning improves the VAE substrate as is shown in the table below. Yet the conditioned model still underperforms the DINOv3-L substrate on 11 of the 12 metrics.
| Substrate | PSNR ↑ | SSIM ↑ | LPIPS ↓ | DISTS ↓ | NIQE ↓ | MUSIQ ↑ | MANIQA ↑ | TOPIQ ↑ | CLIP-IQA ↑ | DeQA ↑ | Q-Align ↑ | Q-Insight ↑ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| DINOv3-L (RAESR) | 23.72 | 0.6265 | 0.3012 | 0.1971 | 4.320 | 70.45 | 0.6717 | 0.6820 | 0.7059 | 3.937 | 4.066 | 3.791 |
| SD3-VAE (64) | 23.61 | 0.6033 | 0.3734 | 0.2495 | 4.344 | 69.76 | 0.6300 | 0.6632 | 0.7011 | 3.740 | 3.801 | 3.687 |
| SD3-VAE (1024) | 23.53 | 0.5984 | 0.3797 | 0.2554 | 4.321 | 69.82 | 0.6269 | 0.6590 | 0.7292 | 3.679 | 3.747 | 3.676 |
| SD3-VAE (1024) + DINO | 23.68 | 0.6139 | 0.3236 | 0.2147 | 4.147 | 69.82 | 0.6439 | 0.6560 | 0.6793 | 3.805 | 3.894 | 3.728 |
The same latents are worth more as substrate than as a side condition. The images agree. In Fig. 8, conditioning brings the harbor boats back but distorts them, and the price sign loses its form. Table 1 points the same way. VOSR-1.4B, which conditions its generator on vision-foundation features, underperforms RAESR on all twelve metrics in both its multi-step and one-step versions, at 2.4× the parameters. This is the independent contribution of the substrate formulation.
4. Direct evidence that restoration is easier on this substrate (W2, Q2)
"improved latent interpolation curves do not directly establish shorter manifold distances or lower restoration complexity"
The new controls provide this evidence at equal capacity and compute. The interpolation curves show where degraded embeddings sit, while the controls above test the mapping itself. All four models share the same 415M decoder, initialization and three-stage training budget, and with capacity and compute held fixed, the DINOv3-L substrate restores best, on 10 of the 12 metrics. For training convergence, we are running a controlled comparison across substrates and will post the results in this thread as soon as they are available.
We hope these comparisons address the reviewer's concerns. If anything remains unclear, we would be glad to clarify or provide further results during the discussion period. If our responses resolve the concerns, we would appreciate it if the reviewer would reconsider the evaluation in light of the new results.
Response to Reviewer zKNR (Part 3)
This part completes point 4 of Part 2 with the training-convergence comparison. The revised PDF now includes it (Sec. 5.1, Fig. 6 bottom row, App. J).
5. Training convergence at equal decoder capacity (W2, Q2)
"Comparisons of training convergence, decoder capacity requirements, or computational budgets across substrates would help substantiate this central claim."
When decoder capacity and training budget are held constant, test quality improves much faster on the DINOv3-L substrate. We train two decoders of the same size, one on each substrate, and score both on the same test set during training.
- Decoders: On DINOv3-L, we use the official RAEv2 decoder, which has 415M parameters. The VAE decoder must match this size. The SD3 VAE decoder in Sec. 5.3 has only 49.5M parameters, so we use the VAE of Qwen-Image-2.1 instead, whose decoder has 259M parameters. We widen its latent to 1,024 channels, the token width of DINOv3-L, and enlarge its decoder to the same 415M parameters. Every added block is initialized as the identity, and neither step costs reconstruction quality. (App. J.1).
- Training: Each decoder starts from the reconstruction weights pretrained for its own latent. Both are then trained in the same way, with frozen encoders and the same data, degradations, and optimizer settings. The losses are L1 and LPIPS only. A run takes 30,000 steps at batch size 16 and is repeated with ten seeds (App. J.2).
- Evaluation: We score 16 checkpoints of every run on 393 held-out images from the four benchmarks.
- Resolution: Training and evaluation use 256 px crops. The official RAEv2 decoder is released for 256 px inputs, so we use it unchanged. The RAESR decoder in the paper is adapted to 512 px, and running at 256 px keeps this adaptation out of the comparison. We repeated the same comparison at 512 px and reproduced the same phenomenon. The curves are in App. J.4 (Fig. 13).
The curves are in Fig. 6 (bottom row) for LPIPS, DISTS, Q-Align and MANIQA, and in Fig. 12 for the other six metrics. Both figures show the mean and the standard deviation over the ten seeds. Table 12 lists the final scores.
The two substrates start from almost the same scores, because at step 0 both decoders still reconstruct the degraded input. DINOv3-L then reaches the final (30,000-step) score of the VAE by step 4,000 on LPIPS, by step 3,000 on DISTS and by step 2,000 on the other eight metrics (Table 12). It remains better than that score for the rest of training.
With decoder size, training signal and schedule matched, the DINOv3-L substrate reaches the final test quality of the VAE substrate in far fewer steps. This is direct evidence that the restoration mapping is easier to learn on this substrate.
We hope this answers Question 2. If anything remains unclear, we would be glad to clarify during the discussion period.
Response to Reviewer zKNR (Part 1)
We appreciate the reviewer's careful and constructive review, and for recognizing the complementary roles of the shallow and deep features and the detail of our methods. The revised PDF adds the two controls requested in Question 1 (Fig. 8, Sec. 5.3). The reviewer also asks for more direct evidence that the restoration mapping is easier to learn on this substrate, for example through training convergence, decoder capacity or compute. We are currently studying this and running experiments on it now, and we will update the results in this thread as soon as possible. We answer each concern below:
1. Our Contributions are Beyond Reusing RAE/RAEv2 (W1)
We did reuse RAEv2's simple framework but our contribution lies elsewhere. RAE and RAEv2 build this latent for generation and clean reconstruction. We show that it is a better substrate for restoration than reconstruction-oriented latents (Sec. 5, and points 2 and 3 below), and we turn it into a restorer with a new training design, a one-pass decoder trained against a dense, position-paired critic on DINOv3-B (Sec. 3.3, 3.4). The architecture stays as simple as RAEv2's. But we see this simple design as a strength. It is what makes RAESR efficient, one pass at 37 ms per image with 719M parameters (Table 1).
2. Latent width and decoder initialization (W2, Q1)
VAE based reconstruction still trails behind RAESR even if the latent dimension and input projection are matched. We widened the SD3-VAE latent from 16 to 256 channels to increase its capacity. Only two layers change shape: the last convolution of the VAE encoder and the first convolution of its own decoder. All other weights are copied. The decoder weights that read the 240 new channels start at zero, so the widened VAE initially reproduces the original. We then fine-tuned the VAE for 5,000 steps on reconstruction (full protocol in App. H). After this change, the VAE latent has the same size as the DINOv3-L latent, and no layer of the VAE encoder is narrower than it.
On 128 held-out images, the widened VAE reconstructs better than the original (PSNR 32.10 vs. 30.86 dB, LPIPS 0.053 vs. 0.067). The original 16-channel VAE fine-tuned with the same budget reaches only 30.99 dB and 0.073, so the gain comes from the width, not the extra training. Zeroing out the 240 new channels raises LPIPS to 0.087, so the extra capacity does carry more information.
Pixel-unshuffling turns the widened latent into 1,024 tokens of width 1,024, exactly the DINOv3-L shape. The SR decoder therefore keeps its input projection, and no weight is re-initialized. Trained with the same initialization and three-stage recipe, this substrate is no better than the original VAE and trails DINOv3-L on 11 of the 12 metrics (table below, Sec. 5.3, Fig. 8, App. H). A higher-capacity VAE does not make a better restoration substrate, even if the input projection is matched.
The sample images in Fig. 8 show the failure directly. Both VAE widths repaint the harbor boats as rocks and reduce the handwritten price sign to something unrecognizable, while RAESR keeps both. Fig. 11 adds two more examples, where the VAE substrates replace the moss between roof tiles with glossy tile surface and add spurs and kinks to letter strokes.