PReS-SR: Prior-Preserving Residual Specialization for Efficient Real-World Image Super-Resolution

Real-world image super-resolution (Real-ISR) requires faithful recovery of observation-supported structures and plausible reconstruction of under-determined details. Pretrained full-image diffusion models provide rich natural-image priors, but their VAE reconstruction bottleneck may degrade spatially precise content. Existing generative SR methods mitigate this issue with LR-derived guidance during denoising or decoding, while recent VAE-free approaches move diffusion directly into pixel space. Despite these advances, the diffusion pathway still reconstructs the entire HQ image, coupling structural recovery with perceptual detail synthesis. Residual diffusion SR methods avoid such redundancy by predicting only a residual over a deterministic pixel-space SR reconstruction, but retarget the diffusion process itself from full-image generation to residual prediction. To retain the pretrained generative prior while leveraging the advantages of residual diffusion SR, we propose PReS-SR, a prior-preserving residual specialization framework that moves the residual-specialization boundary after diffusion prediction. Specifically, a discriminative restoration is carried directly to the output, while the pretrained diffusion backbone retains its full-image representation. A base-injected latent residual refinement block and a feature-injected residual decoder convert this representation into base-relative residual completion. Moreover, fidelity-perception control is localized to the lightweight post-backbone residual decoding, allowing multiple operating points to share the same diffusion computation. Experiments show that PReS-SR is particularly effective under constrained generative capacity, delivering competitive restoration quality with 56% fewer parameters and 1.8$\times$ faster inference than FiDeSR, while producing five controlled outputs up to 3.7$\times$ faster than PiSA-SR.