acceptodds
Withdrawn from ICLR 2027

Spectral-Aware Adaptation for Fidelity-Preserving Diffusion Super-Resolution

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Diffusion-based methods have recently advanced real-world image super-resolution (SR) by leveraging powerful generative priors, typically adapting frozen backbones via low-rank updates such as LoRA. Existing approaches assign LoRA rank uniformly across layers, implicitly assuming homogeneous structural importance. We show that diffusion UNets instead exhibit pronounced intrinsic rank heterogeneity: effective rank is lowest at the input and output extremities of the network and rises toward the bottleneck. Motivated by this observation, we propose SAFA (Spectral-Aware Fidelity Adaptation), a parameter-efficient adaptation framework that estimates the effective rank of every candidate module via singular value decomposition, keeps the most spectrally expressive ones, and distributes a fixed trainable-parameter budget among them in proportion to effective rank, without adding any inference-time cost. Compared to uniform allocation on the same backbone, SAFA concentrates capacity in structurally expressive layers and produces 2.27x stronger total update energy on RealSR while training 6.1x fewer LoRA parameters; against prior diffusion-based SR methods it improves fidelity metrics (PSNR, SSIM) and reference-based perceptual similarity (LPIPS, DISTS, FID) on DIV2K, RealSR, and DRealSR. Qualitatively, SAFA holds structure together in regions that are easy to hallucinate text, repetitive patterns and fine textures where prior diffusion-based SR methods introduce geometrically inconsistent detail. We find that this fidelity-oriented allocation trades off some no-reference perceptual scores relative to methods tuned for aesthetic realism, and we report this trade-off explicitly rather than claiming uniform improvement. We further verify that the heterogeneity is not checkpoint-specific: the same macro-pattern holds across SD1.5, SD2.1 and SDXL, where decoder layers average 1.8x the effective rank of encoder layers, although the precise spatial profile is architecture-dependent. Our results suggest that pre-adaptation spectral structure is a lightweight, gradient-free signal for guiding parameter-efficient adaptation, and that it transfers across diffusion UNet backbones.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
diffusion models, low-rank adaptation, image super-resolution, spectral analysis, parameter-efficient fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_SafaOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

SAFA uses the singular values of pretrained UNet weights to select LoRA modules and adjust their ranks before training for image super-resolution.

Critical strengths

This paper introduces Spectral-Aware Fidelity Adaptation (SAFA) to estimate effective LoRA ranks for different modules, reducing LoRA parameters from 8.11M to 1.32M while achieving competitive PSNR and SSIM performance.

Critical weaknesses

(1) The main idea provides limited novelty. Similar non-uniform rank allocation strategies have already been studied in methods such as AdaLoRA. SAFA mainly replaces the training-time importance estimation with a fixed statistic derived from pretrained weights. Furthermore, the practical benefit of LoRA parameter reduction is unclear. The experiments are only conducted on UNet-based Stable Diffusion backbones, where the LoRA parameters are already very small (1.32M compared to 866M parameters in SD 2.1). Therefore, reducing LoRA parameters from 8.11M to 1.32M may have limited impact on training speed, memory usage, or inference efficiency. The effectiveness of such aggressive LoRA compression seems unclear.

(2) The ablation studies in Tab. 2 and 3 do not include no-reference metrics. Only reporting PSNR and LPIPS is insufficient to demonstrate the effectiveness of the proposed method for diffusion-based image super-resolution. Meanwhile, SAFA achieves almost the worst CLIPIQA and MUSIQ scores across all benchmarks, indicating that the proposed rank allocation strategy may have difficulty preserving perceptual quality.

Questions for authors

(1) What values of r_min and r_max are used in Eq. (3)? How does the performance change when directly increasing or decreasing the LoRA ranks of all modules?

(2) Are the illustrations in Fig. 2 AI-generated? Some details appear distorted, and several components are not well aligned, especially in the UNet architecture. Please clarify how this figure is generated and whether it accurately represents the actual model architecture.

Reviewer_rKmuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to improve the reconstruction fidelity of real-world image super-resolution through parameter-efficient adaptation of pretrained diffusion models. The authors focus on the uniform allocation of LoRA ranks and use the effective rank of pretrained weights to guide module selection and adapter capacity. The proposed SAFA method selects high effective ranks for adaptation and proportionally assigns LoRA ranks under a fixed parameter budget. The allocation is determined before training and remains fixed while reconstruction and perceptual objectives are optimized in two stages. The authors also analyze spectral patterns across different diffusion UNets and examine how the learned updates are distributed within the network. Experiments on synthetic and real-world benchmarks show improved reconstruction fidelity and perceptual similarity to reference images compared with existing diffusion SR methods. The proposed SAFA method achieves these results with fewer trainable LoRA parameters while retaining single-step inference.

Critical strengths

The authors aim to improve the reconstruction fidelity of real-world image super-resolution through parameter-efficient adaptation of pretrained diffusion models. This paper shows some advantages in the following aspects.

(1) The authors address the allocation of adaptation capacity across a diffusion UNet, where different stages have different roles in reconstruction. They analyze the spectral structure of pretrained weights and observe variations in effective rank across network depth and different backbones. The authors then use these observations to guide module selection and rank allocation before training. This connects the analysis of pretrained model structure with the design of SR adaptation and provides a basis for studying how adaptation capacity should be distributed within a generative restoration model.

(2) The authors address content preservation in generative real-world SR, where plausible detail may still depart from the observed structure. Such errors affect the reliability of restoration even when the output appears sharp. The proposed framework combines selective adaptation with reconstruction-fidelity training to recover detail while maintaining reconstruction accuracy. The experimental results show improved fidelity and perceptual similarity to reference images across synthetic and real-world benchmarks, with fewer trainable parameters.

Critical weaknesses

I appreciate the authors' efforts to improve reconstruction fidelity by using diffusion-based model adaptation. However, I have the following concerns.

(1) The authors propose to determine LoRA allocation from the effective rank of pretrained weights. However, SR-LoRA (ICCV 2025) already uses the stable rank of pretrained weights to assign layerwise LoRA ranks in visual transfer learning. This work is not discussed in the paper, and the methodological contribution beyond existing spectral allocation remains unclear. The authors motivate SAFA through differences in effective rank among UNet layers. However, this analysis does not establish why the proposed nuclear norm threshold is appropriate for the rank needed by weight updates for SR. The authors could further explain what information this criterion captures beyond stable rank and why selecting modules with high effective ranks under a parameter budget provides a substantive advance for restoration using diffusion models.

(2) The authors compare SAFA with uniform LoRA allocation to support the effectiveness of the proposed strategy. However, the uniform results in Tables 2 and 3 are taken from PiSA-SR, and this baseline is not retrained with SAFA's training procedure. PiSA-SR uses a different semantic training objective that includes classifier score distillation. The comparison does not separate the benefit of allocation from the effect of training changes. Although Table 3 compares effective rank with the Frobenius norm at a matched budget, changing the signal can also change which modules are selected. It is unclear whether proportional ranks are better than uniform ranks on the same selected modules. The authors could provide a uniform baseline trained under the parameter budget and compare the rank assignments with fixed module selection.

Questions for authors

Here are some questions for the authors.

(1) My major concern is whether the reported gains come from the proposed allocation. The authors could train a baseline with uniform ranks under the same SR training procedure and parameter budget as SAFA. A further comparison using uniform ranks within SAFA's selected modules would help separate the effect of module selection from that of proportional allocation. The authors could also specify how the training iterations are divided between the two stages.

(2) The allocation operates on modules that may contain several weight matrices, but Algorithm 1 defines an effective rank for a single matrix. The procedure for combining these values into a module score is unclear. The authors could explain this aggregation and identify the candidate modules. It would also be helpful to provide the names and assigned ranks of the 12 selected modules, together with the rank limits used in Equation 3.

(3) My concern about novelty relates to the unaddressed connection with SR-LoRA. This prior work assigns layerwise LoRA ranks from the stable rank of pretrained weights. The authors could discuss this connection and explain the additional methodological value of SAFA for diffusion SR. In particular, the rationale for replacing stable rank with thresholds and restricting adaptation to selected modules could be better discussed.

(4) The paper argues that the spectral allocation signal transfers across diffusion UNet backbones. However, the reported SR experiments use SD2.1, while the analysis on other backbones concerns their pretrained weight spectra. Similar spectral patterns do not establish that the allocation improves restoration on another backbone. The authors could provide an SR evaluation on another backbone and clarify whether the same allocation settings are used without additional tuning.

Reviewer_4EG8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SAFA, which uses the effective rank of pretrained diffusion UNet weights to select adaptation modules and allocate LoRA ranks under a parameter budget. Allocation is performed before training, and the learned updates are merged into the backbone afterward. The main paper and supplementary material report reconstruction quality, parameter efficiency, training resource usage, and spectral analyses, showing a trade-off between fidelity and perceptual quality relative to some diffusion baselines.

Critical strengths
  1. The allocation procedure requires neither gradient statistics during training nor dynamic budget adjustment. The design is simple, and merging the updates introduces no additional inference components into the backbone.
  2. Performance on reconstruction and full-reference perceptual metrics is relatively consistent across datasets. Comparisons with diffusion and GAN baselines support the method’s practical value.
  3. The supplementary material improves experimental completeness through threshold sensitivity analysis under a fixed budget, relative weight-update analysis, training resource measurements, and additional visual results.
Critical weaknesses
  1. Nonuniform LoRA rank allocation has been studied previously[1][2]. The main contribution here is using pretrained weight spectra for static module selection and capacity allocation. While practically useful, the method lacks sufficient comparisons with existing adaptive allocation approaches, and the effects of matrix dimensions and module locations have not been ruled out. Its distinct contribution therefore remains insufficiently substantiated.
  2. The main uniform-allocation baseline was not retrained under the same training protocol, and the existing ablations do not separate the contributions of module selection and nonuniform rank allocation. An ablation of SAFA’s selection ratio under a fixed parameter budget is also missing. Evidence across backbones mainly consists of spectral distribution observations; the transferability of downstream adaptation gains remains unverified.
  3. The allocation box at the bottom right of Figure 2 contains mottled backgrounds, inconsistent local coloring raising the possibility of AI-generated imagery. The current AI usage statement does not explicitly clarify whether generative AI was involved in creating the figure. Clarification and improvements to the figure’s clarity are needed.
  4. Table colors do not consistently follow the captions. Table 1 omits highlighting for some best and second-best results. Supplementary Table 5 also marks some best results in blue, including SAFA’s CLIPIQA, MUSIQ, and MANIQA scores on DIV2K. These presentation and proofreading issues should be systematically corrected.

[1] Qingru Zhang et al. AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning. ICLR, 2023.

[2] Fabian Paischer et al. Parameter Efficient Fine-tuning via Explained Variance Adaptation. NeurIPS, 2025.

Questions for authors
  1. Could you compare uniform and proportional rank allocation over the same selected modules under identical training configurations and parameter budgets, and provide an ablation of SAFA’s selection ratio?
  2. Does spectral allocation retain a consistent independent advantage after controlling for matrix dimensions and module locations? Are there downstream experiments on other backbones supporting the transferability claim?
  3. Was generative AI used to assist in creating Figure 2? If so, please specify its role and explain or remove visual elements whose meanings are unclear.
  4. Please check and correct the highlighting of best and second-best results in Tables 1 and 5 according to their respective captions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.