- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a real-image super-resolution framework, HiCoSR, which constructs a high-resolution latent branch by fusing intermediate features from the encoder with upsampled coarse-grained latent variables, thus mitigating the loss of detail caused by VAE space compression. The model employs a two-stage recovery process from coarse to fine, performing one step of denoising at each stage, and promotes compatibility between the new representation and the pre-trained latent space through shared decoder supervision. On three benchmarks, it achieves improvements in PSNR and SSIM, and its inference speed is competitive, but its perceptual quality metrics show varying performance.
Critical strengths
The high-resolution latent characterization design is effective. After removing the HiCo feature, the PSNR on RealSR decreased from 26.48 to 24.61 dB, while the DISTS increased from 0.1817 to 0.2142.
It achieves best PSNR and SSIM on all three benchmarks, and also best LPIPS and DISTS on RealSR; evaluation on both real and synthetic data enhances the reliability of the results.
Reusing SD-Turbo, employing LoRA, and omitting image description generation results in low inference costs. Processing a 512×512 image on an RTX 5090 takes 0.13 seconds, close to OSEDiff and faster than TVT.
Critical weaknesses
Reconstruction supervision under a shared frozen decoder is insufficient to prove the latent distribution is "strictly aligned." Direct distribution analysis and ablation experiments distinguishing the contributions of shared decoder supervision from the increase in spatial capacity are lacking.
The coarse stage improves PSNR by 0.41 dB, but LPIPS and FID decrease slightly. The complete model actually performs two sequential denoising steps, but the table labels it as one step; a single-scale baseline matching training conditions and computational cost is lacking, making it difficult to determine whether the gains come from hierarchical design or additional computation.
Questions for authors
What exactly does "strict alignment" mean, and what evidence supports this? Can you compare high-resolution encoders with and without shared decoder supervision, all other things being equal, to verify their contribution to the compatibility and recovery performance of the pre-trained denoiser?
How does the performance compare to a single-scale double-denoising baseline using the same decoder, discriminator, and loss settings? Please also report fidelity, perceptual quality, end-to-end time, and actual number of denoising iterations.
4 comments
Response to Reviewer xsFk Comments 2
Q2 & W2 — Single-Step Hierarchical Denoising and Ablation Results
HiCoSR is a single-step hierarchical diffusion framework, with one denoising step at each resolution. The coarse denoiser establishes global structure at f8, while the fine denoiser refines fused f4 features with additional LoRA modules and adapted effective weights. Each of these distinct, resolution-specific modules predicts a clean latent in one forward pass. "Single-step" describes denoising within each stage; NFE counts the total denoiser evaluations across the cascade. We will clarify this convention in Table 2 as one step per stage with a total NFE of 2. We compare against f8→f8 and f4→f4 two-pass baselines using the same discriminator and matched training settings. Final f8 latents are decoded by Decoder×8, and final f4 latents by Decoder×4. All timings are measured end-to-end at 512×512 on an RTX 5090.
| Configuration | NFE | Time (s) | PSNR↑ | SSIM↑ | DISTS↓ | MANIQA↑ |
|---|---|---|---|---|---|---|
| Single pass, 4× (w/o D_f8) | 1 | 0.11 | 26.07 | 0.7664 | 0.1875 | 0.6407 |
| Two passes, 8× | 2 | 0.10 | 25.49 | 0.7386 | 0.1961 | 0.6437 |
| Two passes, 4× | 2 | 0.20 | 26.11 | 0.7665 | 0.1871 | 0.6413 |
| HiCoSR (ours) | 2 | 0.13 | 26.48 | 0.7695 | 0.1817 | 0.6538 |
With NFE fixed at 2, HiCoSR leads both same-scale baselines on all four quality metrics, gaining 0.99 dB PSNR over f8→f8 and 0.37 dB over f4→f4. Adding a second f4 pass yields only 0.04 dB over the fine-only variant while increasing latency by 0.09 s (0.11→0.20 s). The f8→f4 cascade gains 0.41 dB with only 0.02 s additional latency (0.11→0.13 s). Thus, HiCoSR achieves higher quality and 35% lower latency than f4→f4 at the same NFE.
Across all eight metrics in Table 3, adding the coarse stage improves PSNR, SSIM, DISTS, MANIQA, MUSIQ and Q-Align over the fine-only variant. LPIPS increases from 0.2445 to 0.2476 and FID from 112.98 to 115.37. This operating point prioritizes structural fidelity while improving reference-based perceptual similarity measured by DISTS and all three no-reference scores, consistent with our restoration objective.
Denoising entirely at f4 requires the model to recover global structure and fine detail jointly on a larger latent grid. Repeating this operation retains that coupled task. HiCoSR separates these responsibilities: the f8 stage establishes global structure on a compact 64×64 grid, providing guidance for the f4 stage to refine local detail on a 128×128 grid. This also reduces computation compared with two passes at f4. The controlled comparisons demonstrate that the gains come from the coarse-to-fine, multi-resolution design, beyond simply increasing NFE to 2.
Thank you for the clarifications and additional experiments; after considering the rebuttal, I maintain my original score of 3.
Thank You for Your Time and Positive Feedback
Thank you for carefully considering our clarifications and additional experiments. We sincerely appreciate your time and constructive feedback. If there are any remaining concerns you would like us to address, we would be happy to provide further clarification.
Response to Reviewer xsFk Comments 1
We thank the reviewer for recognizing the contribution of our higher-resolution latent representation, the improvements in fidelity and reference-based perceptual metrics, and the practical inference efficiency. We also appreciate the detailed questions on latent compatibility and the hierarchical design, which we address point by point below with additional analyses and controlled comparisons.
Q1 & W1 — Latent Compatibility and Shared-Decoder Supervision
Here, alignment refers to compatibility with the frozen pretrained decoder: the f4 latent preserves finer detail while remaining decodable by the original Decoder×8. We will use "pretrained-decoder compatibility constraint" in the revision to state this property precisely. With the decoder unchanged, the f4 representation achieves 39.46 dB PSNR and 0.0197 LPIPS, versus 33.83 dB and 0.0601 for the original VAE (Table 1).
We isolate the effect of this compatibility constraint by comparing the same f4 HiCo-Encoder trained with either the frozen pretrained Decoder×8 or a jointly trained decoder under matched training settings. We denote the resulting latents as f4 locked and f4 unlocked, respectively. The latter removes the shared frozen-decoder supervision. We evaluate both variants in the same restoration framework on RealSR, with results below.
At identical f4 spatial capacity, the frozen-decoder constraint improves all seven reported metrics, with a 0.85 dB PSNR gain, lower LPIPS (0.2687→0.2476) and lower FID (121.86→115.37), demonstrating its benefit beyond increased latent resolution.
Latent visualization. Figure R1 compares the original VAE's f8 latent, HiCoVAE's f4 locked latent, and f4 unlocked using 3,000 crops of size 512×512 from the open-source Aesthetic-4K dataset. For each crop, we use a bicubically upsampled 1024×1024 input for the original f8 VAE and the original 512×512 input for both f4 encoders. This gives 4×128×128 latents in all three cases for the t-SNE visualization.
In the joint t-SNE plot, the original f8 and HiCoVAE f4 locked latents strongly overlap without visibly separate clusters, whereas f4 unlocked exhibits a clearly different distribution. The three individual distribution plots on the right show the same contrast. The remaining differences between the locked f8 and f4 views may partly reflect the bicubic upsampling applied to the f8 inputs. Together with the restoration ablation, these observations support the role of shared-decoder supervision in maintaining compatibility with the pretrained latent domain.
t-SNE visualization of original VAE f8, HiCoVAE f4 locked, and f4 unlocked latents.
Figure R1. Latent distributions with and without shared frozen-decoder supervision. Left: joint t-SNE visualization. Right: individual views of f8 locked, f4 locked, and f4 unlocked, respectively.
Shared-decoder supervision supports both cross-scale fusion and reuse of pretrained priors. Without this constraint, the f4 latent can drift from the original f8 latent domain. First, the upsampler and fusion module must then reconcile this distribution mismatch while transferring the denoised f8 structure to f4, making it harder to combine coarse guidance with fine source details. Second, the fine denoiser inherits SD-Turbo weights learned in the original VAE latent domain. A shifted f4 distribution weakens transfer of their semantic priors and increases the adaptation burden on LoRA. Keeping the two scales compatible therefore facilitates feature fusion and lets the fine stage use pretrained semantic knowledge more effectively.