acceptodds
Under review as a conference paper at ICLR 2027

PNR-SR: Policy-Native Reward-Policy Co-Training for One-Step Real-World Image Super-Resolution

12 views

Abstract

One-step generative models have substantially improved the inference efficiency of real-world image super-resolution (Real-ISR), while reward-based post-training provides a promising paradigm for improving restoration quality beyond supervised learning. However, existing approaches typically rely on an offline-trained reward model that remains fixed during policy optimization. As the SR policy evolves, its output distribution can progressively diverge from the reward model's learned preference distribution, weakening the reward model's ability to reliably distinguish restoration quality on newly generated outputs. In this work, we formulate reward-based post-training for Real-ISR as a closed-loop optimization problem and propose PNR-SR, a Policy-Native Reward-Policy Co-Training framework for one-step Real-ISR. To this end, we introduce HR-exposure ordering to derive policy-native preference supervision from the current SR policy, enabling online reward model optimization without human annotations or fixed external metrics. The reward model and SR policy are subsequently optimized in an online closed loop, where the reward model is continuously updated with policy-generated preferences to track the evolving SR policy distribution and provide dynamically updated reward guidance. To stabilize online reward model optimization, we further introduce dual-anchor calibration and a historical reward baseline to mitigate reward score drift. Extensive experiments demonstrate that our framework effectively improves one-step Real-ISR performance and outperforms existing state-of-the-art methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Super-Resolution, Generative models, Reward Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_jKurOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PNR-SR, an online reward-policy co-training framework for one-step real-world image super-resolution. The method introduces HR-exposure ordering to construct preference pairs from the current SR policy and continuously updates the reward model alongside the SR policy. Dual-anchor calibration and a historical reward baseline are further introduced to stabilize training. Experiments on several SR benchmarks demonstrate competitive perceptual quality, supported by ablation studies and human evaluations.

Critical strengths
  1. Interesting preference supervision strategy. HR-exposure ordering offers a practical way to construct preference pairs from the evolving SR policy without repeatedly collecting human annotations.
  2. Competitive perceptual performance. The proposed method achieves strong results on multiple SR benchmarks. The ablation studies and human evaluations provide additional support for its effectiveness.
Critical weaknesses
  1. Incomplete comparisons with closely related reward-learning methods. The paper compares several SR backbones but does not directly evaluate recent reward-learning approaches such as FinPercep-RM and RefReward-SR under comparable settings. Furthermore, the method mainly improves no-reference perceptual metrics, while simpler CSD-based training achieves better results on several full-reference metrics under the same backbone.
  2. Limited generalizability and efficiency analysis. The experiments are primarily conducted on a single Z-Image-based architecture. Additional evaluations on different SR backbones, together with an analysis of the extra training cost introduced by online reward-model updates, would better establish the generality and practical benefits of the framework.
Questions for authors
  1. Can the authors compare PNR-SR with recent reward-based SR methods under consistent settings and further analyze the fidelity–perception trade-off?
  2. Can the authors validate the framework on additional SR backbones and report the computational overhead of online reward-policy co-training?
2 comments
AuthorsOpenReview ↗

Response to Reviewer jKur(1/2)

Dear Reviewer jKur,

We thank you for your valuable comments and constructive suggestions. We have carefully considered each comment and revised the manuscript accordingly. For brevity, we use W# to denote weaknesses and Q# to denote questions (e.g., W1, Q1) throughout our response.


[W1 & Q1]: Additional Comparisons and Trade-off Analysis

A1:

Regarding recent related reward-learning methods, since FinPercep-RM has not yet released its model weights, we add a direct comparison with RefReward-SR, whose reward model is trained on large-scale human preference annotations. The results are reported below. PNR-SR outperforms RefReward-SR on LPIPS, DISTS, and the distribution consistency metric FID across both datasets. Meanwhile, PNR-SR uses a smaller generative backbone (6.31B vs. 13.28B) and requires only single-step inference (1 step vs. 25 steps).

The differences in no-reference (NR) perceptual metrics should also be considered in light of the differences between the reward models used by the two methods. During post-training, RefReward-SR does not rely solely on its own reward model, but uses a hybrid reward function combining RefReward-SR, LPIPS, and DeQA-Score. DeQA-Score is trained on the human-annotated KonIQ-10k, SPAQ, and KADID-10k datasets. The NR metrics used in our evaluation include TOPIQ, CLIPIQA+, MUSIQ, and MANIQA, trained on KonIQ-10k, as well as Q-Align, trained on KonIQ-10k, SPAQ, KADID-10k, AVA, and LSVQ. Their training data all overlap with those of DeQA-Score. Therefore, directly optimizing DeQA-Score as a reinforcement learning reward inherently introduces a strong metric-alignment bias toward these NR evaluation metrics trained on overlapping datasets. To complement the comparison with a different evaluation basis, we additionally include NIQE, which does not require training on the aforementioned human-annotated image quality datasets. PNR-SR achieves better NIQE results on both datasets.

Efficiency Analysis

MethodParameters↓Inference Steps↓Text Conditioning
Ours6.31B1Fixed text prompt
RefReward-SR13.28B25Requires additional text extraction model

Metrics

DatasetMethodPSNR↑LPIPS↓DISTS↓FID↓MUSIQ↑TOPIQ↑MANIQA↑CLIPIQA+↑Q-Align↑NIQE↓
RealSRRefReward-SR24.700.32270.2362122.671.700.73310.68040.73154.1355.304
RealSROurs24.220.29430.2140116.069.860.69020.66070.69373.8305.128
DIV2KRefReward-SR22.870.32600.207729.8171.820.73950.66830.74894.4594.817
DIV2KOurs22.590.29050.181723.9769.990.70140.65030.70774.1054.530

We further provide qualitative comparisons in Fig. 13 of the Appendix, demonstrating that our method achieves superior text restoration and detail generation.

Fidelity–Perception Trade-off Compared with CSD

More pronounced advantages in no-reference metrics. Under the same backbone, PNR-SR performs slightly worse than CSD on LPIPS (0.2943 vs. 0.2898) and DISTS (0.2140 vs. 0.2073), and also has a lower PSNR (24.22 vs. 24.89). However, it achieves better results across all five no-reference perceptual metrics, including improvements of 0.0266 in TOPIQ, 0.0275 in CLIPIQA+, and 1.24 in MUSIQ. Its advantages in no-reference metrics are particularly clear.

Fidelity–perception trade-off analysis. In real-world super-resolution, If a model tends to generate smooth details, it can also achieve higher PSNR and better LPIPS scores. Whereas realistic, sharp high-frequency details are often visually preferred but may come at the expense of full-reference metrics. As shown in the comparison between CSD and our method in Fig.12 of the appendix, PNR-SR’s higher no-reference scores correspond to more realistic high-frequency textures and fewer artifacts.

CSD fails to suppress periodic artifacts. As noted in StrSR[1], one-step SR models based on the DiT architecture are prone to periodic artifacts caused by high-frequency spectral leakage. We observe periodic artifacts in the CSD model’s outputs (see Fig. 12 in the appendix). By updating the policy-native reward model online, PNR-SR effectively suppresses these artifacts, producing cleaner and more realistic restoration results.

[1] Jingkai Wang, Yixin Tang, Jue Gong, Jiatong Li, Shu Li, Libo Liu, Jianliang Lan, Yutong Liu, and Yulun Zhang. Spectral and trajectory regularization for diffusion transformer super-resolution. In ECCV, pp. 94–113. Springer, 2026.

AuthorsOpenReview ↗

Response to Reviewer jKur(2/2)

[W2 & Q2]: Validate the framework on additional SR backbones and report the computational overhead of online reward-policy co-training?

A2:

We additionally validate our framework on two backbones, FLUX.2-klein and HiDream-O1. For FLUX.2-klein, we compare against the publicly released FLUX.2-klein version of StrSR; for HiDream-O1, we compare against a CSD baseline that we train ourselves.

Compared with StrSR (FLUX.2-klein), PNR-SR achieves PSNR improvements of 1.48 dB and 0.62 dB on RealSR and DIV2K-Val, respectively, while outperforming it on three out of five and all five no-reference perceptual metrics, respectively. Specifically, on RealSR, PNR-SR improves MUSIQ by 1.37 and TOPIQ by 0.0402, while on DIV2K-Val, the corresponding improvements are 2.53 and 0.0800.

FLUX.2-klein

DatasetMethodPSNR↑LPIPS↓DISTS↓FID↓MUSIQ↑TOPIQ↑MANIQA↑CLIPIQA+↑Q-Align↑
RealSRPNR-SR (FLUX.2-klein)24.390.28890.2186121.769.820.66370.65770.67863.708
RealSRStrSR (FLUX.2-klein)22.910.26740.2115109.568.450.62350.64990.68043.731
DIV2K-ValPNR-SR (FLUX.2-klein)22.530.28470.191027.0669.690.67110.65000.68353.895
DIV2K-ValStrSR (FLUX.2-klein)21.910.27210.178821.5867.160.59110.62970.67893.817

On the HiDream-O1 backbone, PNR-SR outperforms CSD on four no-reference perceptual metrics on both datasets. It also achieves better LPIPS and DISTS on RealSR, and better DISTS and FID on DIV2K-Val.

HiDream-O1

DatasetMethodPSNR↑LPIPS↓DISTS↓FID↓MUSIQ↑TOPIQ↑MANIQA↑CLIPIQA+↑Q-Align↑
RealSRPNR-SR (Hi-Dream)24.340.28410.2034116.069.520.61780.63780.64893.533
RealSRCSD (Hi-Dream)24.720.28890.2047113.768.700.61900.62160.63793.414
DIV2K-ValPNR-SR (Hi-Dream)22.500.29370.178723.9469.930.63510.61960.65893.785
DIV2K-ValCSD (Hi-Dream)22.470.28710.186526.3669.210.64380.60950.65753.666

These results demonstrate that the perceptual quality benefits of PNR-SR are not limited to Z-Image but extend to other generative backbones.

We measure the computational cost of the complete training pipeline on A100 GPUs. The three stages require 156, 17, and 29 GPU hours, respectively, totaling 202 GPU hours. In comparison, single-stage CSD training requires 188 GPU hours.

Reviewer_H1qxOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes PNR-SR, an online reward-policy co-training framework for one-step Real-ISR. The authors introduce HR-exposure ordering to construct policy-native preference pairs, enabling continuous updates to the reward model. To further stabilize training, the framework incorporates dual-anchor calibration alongside a historical reward baseline. Experimental results demonstrate competitive performance across multiple Real-ISR benchmarks.

Critical strengths

1.The proposed framework directly addresses the identified limitations of fixed reward models. The overall design is intuitive and effective, with reasonably comprehensive ablation studies supporting its key components.

2.The proposed method achieves competitive results across multiple Real-ISR benchmarks, demonstrating its effectiveness in improving restoration quality.

Critical weaknesses

1.Reward-policy co-training has been explored in both the LLM and computer vision communities. This work mainly extends the established paradigm with a task-specific preference construction strategy and training stabilization techniques, leaving its methodological novelty and significance somewhat limited.

2.While constructing preference pairs through different levels of HR exposure is intuitive and interesting, its effectiveness and generalization remain insufficiently validated. Further investigation is needed, as detailed in the questions below.

Questions for authors

1.Could the authors further clarify the key contributions of PNR-SR compared with existing reward-policy co-training methods, and discuss whether the proposed framework offers broader insights beyond simply adapting an established paradigm to the super-resolution task?

2.The authors claim that higher HR exposure leads to better restoration quality. However, I am concerned that this assumption may not always hold for nonlinear SR policies. For instance, higher exposure could introduce artifacts in fine textures despite bringing the input latent closer to the ground truth. Could the authors further investigate such cases?

3.It would be interesting to further investigate what the reward model actually learns. How can the authors verify that it captures genuine restoration quality rather than exposure-dependent statistical cues, such as frequency distributions, sharpness, or noise patterns?

4.Furthermore, the paper lacks a thorough analysis of the reward model's generalization ability and whether it learns robust IQA capabilities. Could the authors provide additional qualitative and quantitative evaluations to better understand what the reward model actually learns?

5 comments
AuthorsOpenReview ↗

Response to Reviewer H1qx(1/3)

Dear Reviewer H1qx,

We thank you for your valuable comments and constructive suggestions. We have carefully considered each comment and revised the manuscript accordingly. For brevity, we use W# to denote weaknesses and Q# to denote questions (e.g., W1, Q1) throughout our response.


[W1 & Q1]: Novelty

A1:

HR-exposure ordering provides a distinct, low-cost preference-construction strategy to support reward-policy co-training. We acknowledge that the concept of reward-policy co-training itself is not first introduced in this work. The core contribution of PNR-SR lies in proposing and validating HR-exposure ordering, a low-cost and effective preference-construction mechanism for reward-policy co-training in generative super-resolution. This mechanism continually generates preference supervision from the current policy, thereby driving closed-loop optimization of the reward model and policy.

This mechanism differs clearly from existing methods. For example, Mutual-Taught [1] in the LLM community constructs pseudo-preferences from outputs generated before and after policy updates; Adv-GRPO [2] in computer vision treats reference images as positive samples and current generations as negative samples, training the reward model online as a binary discriminator. In contrast, our preference pairs are generated by the same current policy for the same LR input, with their ordering determined by controlled differences in HR exposure. Thus, our mechanism does not require assuming that the updated policy always outperforms the previous policy, and provides finer-grained quality preference supervision than directly distinguishing reference images from generated images. More broadly, this HR-exposure-driven preference data construction mechanism has strong potential to extend to other generative tasks and can also be used to construct quality preference data for training general-purpose image quality assessment (IQA) models. We have conducted preliminary experiments on the latter, which will be further discussed in our response to Q4.

We have added the corresponding discussion to the additional related work section in Appendix C of the paper.

[1] Shi, Tianyuan and Huang, Canbin and Wan, Fanqi and Zhong, Longguang and Yang, Ziyi and Shen, Weizhou and Quan, Xiaojun and Yan, Ming. “Mutual-taught for co-adapting policy and reward models.” In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 16285–16298, 2025.

[2] Mao, Weijia and Chen, Hao and Yang, Zhenheng and Shou, Mike Zheng. “The image as its own reward: Reinforcement learning with adversarial reward for image generation.” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5999–6009, 2026.


[W2 & Q2]: Does Higher HR Exposure Introduce New Artifacts

A2:

First, as shown in Fig. 2 of the main paper, our human evaluation shows 89.1% agreement between human preferences and HR-exposure ordering, with agreement tending to increase as the exposure gap grows. This indicates that the mechanism provides generally reliable preference supervision. Fig. 10 in the appendix presents paired outputs with exposure gaps Δτ\Delta\tau of 0.06, 0.16, 0.26, and 0.32. In these examples, outputs with higher exposure exhibit more natural and structurally coherent fine textures, as well as improved text restoration.

To investigate the concern that higher exposure may introduce new artifacts, we further examine the 41 cases in which HR-exposure ordering disagrees with human preferences and identify three typical situations. First, when HR-exposure levels are close, the paired outputs exhibit small perceptual quality differences, making it difficult to establish clear and consistent human preferences. The average exposure gap Δτ\Delta\tau across these 41 cases is only 0.115. Second, some HR reference images themselves contain artifacts, and higher exposure may cause the outputs to retain these defects. Third, when the HR reference images are relatively blurry, outputs with lower exposure can sometimes produce richer visual details by drawing on generative priors.

Among these 41 examined cases, we observe no clear new artifacts that appear exclusively in higher-exposure outputs and cannot be traced to defects already present in the HR references. For preference disagreements caused by defects in the HR reference images themselves, filtering out low-quality HR references beforehand could further reduce this source of preference supervision noise in future work. The analysis of failure cases of HR-exposure ordering and corresponding examples are provided in Appendix D.2 and Fig. 11.

AuthorsOpenReview ↗

Response to Reviewer H1qx(2/3)

[W2 & Q3]: Does the Reward Model Learn Genuine Restoration Quality

A3:

We design multiple experiments to verify that the reward model learns genuine restoration quality.

(1) Improvements in the model’s outputs provide empirical evidence. After reward-policy co-training, LPIPS and multiple perceptual quality metrics improve. Moreover, as shown in Fig. 6 of the appendix, our constructed preference data include pairs with and without periodic artifacts, as well as pairs with distorted and undistorted text. After co-training, periodic artifacts in the outputs are mitigated, and text distortions are also reduced. These results support that the reward model learns cues related to restoration quality from our constructed preference data and provides effective quality optimization signals for the policy.

(2) Reward distributions at different HR-exposure levels. If the reward model assigns scores solely based on the exposure level τ\tau, reward scores for different samples at the same τ\tau should be relatively similar. To investigate this, we randomly select 200 samples from DIV2K-Val, collect their corresponding restoration outputs at exposure levels τ∈{0.5,0.6,0.7,0.8}\tau \in \{0.5, 0.6, 0.7, 0.8\}, and examine their reward distributions. As shown in the table below and the distribution visualizations in the Fig. 7 of appendix, the mean reward increases with the exposure level. However, within each fixed τ\tau, reward scores do not concentrate around a single value but exhibit clear variation across samples (the standard deviations are very large). This indicates that the reward model can distinguish image quality differences within the same exposure level.

Exposure level τ\tauMean rewardReward standard deviation
0.5−0.65180.8969
0.60.62221.0951
0.71.50540.9994
0.82.18050.8622

(3) Reward Model Agreement with Human Preferences on LR-Only Outputs. To verify whether the reward model can assess restoration quality without relying on differences in HR exposure, we design an independent preference evaluation protocol based on human annotations. Specifically, we randomly select 100 images from DIV2K-Val and apply two independent random degradations to each image, obtaining two slightly different LR inputs of the same HR. At training steps 500, 1000, and 1500, we use the current policy to generate paired SR outputs under LR-only conditions. Each output pair is manually annotated according to visual quality preferences, excluding pairs without a clear preference. We then use the online reward model at the corresponding training step to score the respective LR–SR pairs and compute pairwise agreement between model-predicted and human preferences over valid preference pairs.

Training stepstep-500step-1000step-1500
Agreement80.7%75.0%74.6%

The reward model maintains high agreement with human preferences across all three stages, supporting its ability to distinguish restoration quality consistently with human perception under LR-only conditions, rather than relying on exposure-level differences for ranking.

The corresponding analysis and Figure are provided in Appendix B.2.

AuthorsOpenReview ↗

Response to Reviewer H1qx(3/3)

[W2 & Q4]: Reward Model Generalization and IQA Capabilities

A4:

We provide additional quantitative evaluations from two perspectives: the reward model’s cross-model generalization ability and the general applicability and scalability of the HR-exposure preference construction method.

(1) The reward model achieves higher agreement with human preferences in cross-model evaluation. We use the raw-in-domain test split of RefSR-18K [1], publicly released by RefReward-SR. This test split contains 200 LR images, each associated with four candidate outputs with human ratings, with candidate outputs drawn from eight SR models. Pairing the candidate outputs corresponding to the same LR input yields 1,200 pairs. After excluding 11 pairs with identical human ratings, we obtain a human preference test set containing 1,189 pairs. We directly evaluate the reward model obtained through co-training on this test set. It achieves 69.77% pairwise agreement with human preferences, outperforming PSNR, LPIPS, MUSIQ, and CLIPIQA+. This result demonstrates that the learned reward model, trained only on Z-Image outputs, generalizes to outputs from other SR models and can distinguish image quality across models, providing evidence that it learns cues related to image quality.

(2) Expanding the preference data further improves agreement with human preferences. To further examine the general applicability and feasibility of the HR-exposure preference construction method, we perform warm-up training on FLUX.2-klein, FLUX.1-Kontext, Z-Image, and Iris-3B, and collect preference data offline. We also extend this preference data collection mechanism to two multi-step models, SeeSR and DiT4SR. Their explicitly accessible sampling trajectories allow us to directly construct outputs at different HR-exposure levels without additional retraining of the generative models. We collect a total of 482,102 preference pairs and use the same reward model architecture as in the paper to train an IQA model offline. On the same test set, this model achieves 77.22% pairwise agreement with human preferences. These results indicate that the HR-exposure preference construction mechanism is not limited to the setting studied in this work, but can be applied to different generative models and has the potential to support the development of new image quality assessment models.

MethodPSNRLPIPSMUSIQCLIPIQA+Reward(only Z-image)Reward (Multi-Source)
agreement0.40900.63840.61560.56730.69770.7722

[1] Song, Yushuai and Quan, Weize and Wang, Weining and Sun, Jiahui and Liu, Jing and Yu, Pengbin and Chen, Zhentao and Li, Meng and Shen, Wei and Yuan, Lunxi and others. "RefReward-SR: LR-Conditioned Reward Modeling for Preference-Aligned Super-Resolution" In European Conference on Computer Vision. pp. 94--112. 2026

Reviewer_5WMzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PNR-SR, an online reward-policy co-training framework for one-step real-world image super-resolution (Real-ISR), aiming to address the distribution mismatch between fixed reward models and evolving SR policies. The proposed HR-exposure ordering constructs preference pairs by varying the amount of ground-truth HR information injected into the initial latent states, enabling continuous updates of an LR-conditioned reward model using samples from the current policy. The updated reward model guides SR policy optimization through differentiable reward feedback, while dual-anchor calibration and a historical reward baseline stabilize co-training. Experiments on DIV2K-Val, RealSR, ScreenSR, RealLQ250, and PhoneSR30, together with ablation studies, reward dynamics analysis, and human evaluations, demonstrate competitive restoration performance.

Critical strengths
  1. Policy-native preference learning. The paper addresses the potential distribution mismatch between fixed reward models and evolving generative policies by introducing HR-exposure ordering to construct preference supervision from current policy outputs without additional human annotations. This online reward adaptation strategy is a meaningful contribution beyond conventional fixed-reward post-training. A human study involving 500 restoration pairs reports 88.91% agreement with the proposed ordering (excluding ties), providing empirical support for the preference construction.

  2. Comprehensive experimental evaluation. The method achieves competitive performance across multiple Real-ISR benchmarks, including controlled comparisons with CSD- and GAN-based training using the same Z-Image backbone. Ablations on online reward adaptation, dual-anchor calibration, and the historical reward baseline, together with reward dynamics analysis and human evaluations, provide evidence of perceptual improvements and improved reward-scale stability.

Critical weaknesses
  1. HR-exposure ordering constructs preferences by injecting varying amounts of ground-truth HR information into the initial states. Consequently, the resulting supervision may primarily capture quality differences induced by privileged HR information rather than fine-grained preferences among LR-only restorations. Moreover, the reward model is trained on HR-privileged outputs, whereas inference uses LR-only inputs. The reward dynamics analysis in Appendix B.2 demonstrates the preservation of reward margins on HR-exposure pairs but does not establish improved preference prediction over a fixed reward model on the actual deployment distribution. This leaves the central claim of online reward adaptation insufficiently validated.

  2. In Table 5, the full model provides only marginal improvements over the offline reward baseline: LPIPS decreases from 0.2959 to 0.2943, and CLIPIQA+ increases from 0.6863 to 0.6937, while PSNR slightly decreases. Furthermore, the experiments do not sufficiently disentangle the benefits of online reward updates from those of privileged HR supervision or implicit distillation. The necessity of the more complex co-training framework therefore remains unclear.

Questions for authors
  1. Could the authors compare the pairwise preference prediction accuracy of online and fixed reward models on an independent, human-annotated dataset of LR-only SR outputs, and analyze how this accuracy evolves during policy training? This would directly test the central distribution-mismatch hypothesis.

  2. Could the authors provide controlled baselines using the same HR-exposure samples without online reward model updates, such as direct privileged distillation or conventional supervised training with HR-exposure augmentation? This would help isolate the benefits of online preference learning from those of privileged HR supervision.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 5WMz(1/2)

Dear Reviewer 5WMz,

We thank you for your valuable comments and constructive suggestions. We have carefully considered each comment and revised the manuscript accordingly. For brevity, we use W# to denote weaknesses and Q# to denote questions (e.g., W1, Q1) throughout our response.


[W1 & Q1]: Preference Prediction Evaluation under LR-Only SR outputs and Validation of the Distribution-Mismatch Hypothesis

A1:

The online reward model maintains high agreement with human preferences under LR-only SR outputs. To directly test the distribution-mismatch hypothesis and evaluate the reward model’s preference prediction ability on the LR-only deployment distribution,We design an independent preference evaluation protocol based on human annotations. Specifically, we randomly select 100 images from DIV2K-Val and apply two independent random degradations to each image to obtain two slightly different LR inputs. The current policy then generates restoration outputs under LR-only conditions. Each pair of restored images is manually annotated according to visual quality preferences, excluding pairs without a clear preference.

We construct these evaluation samples at training steps 500, 1000, and 1500 for both the offline and online training strategies. The offline group uses the fixed reward model, while the online group uses the online reward model at the corresponding training step. Each reward model scores the LR–SR pairs generated by its respective policy, and pairwise agreement with human preferences is computed over valid preference pairs. As shown below, the online group maintains high agreement at all three stages, whereas the offline group exhibits relatively low agreement. These results demonstrate our method’s preference prediction ability under LR-only SR outputs and validate the distribution-mismatch hypothesis.

Variantstep-500step-1000step-1500
Online80.7%75.0%74.6%
Offline40.4%41.8%53.6%
AuthorsOpenReview ↗

Response to Reviewer 5WMz(2/2)

[W2]: Limited Gains from Online Reward

A2:

In Table 5, the full online model and the offline reward baseline achieve similar LPIPS and CLIPIQA+ scores because the offline baseline also learns from HR-exposure preference supervision and can provide useful guidance during early post-training. However, despite their similar global metrics, the two methods exhibit substantial differences in visual quality, as analyzed below.

(1) The two methods differ substantially in visual quality, and artifacts can mislead LPIPS. In our visual evaluation, we observe numerous spurious high-frequency artifacts (resembling high-frequency noise) in the outputs of the offline reward baseline, whereas the online model effectively avoids these defects. Such local abnormalities are not fully reflected by global metrics and can even mislead LPIPS into assigning artificially favorable scores. We present 5 representative examples in Fig.8 of the appendix, with the corresponding LPIPS scores annotated in the upper-right corner of each image: although the offline model achieves better LPIPS scores on these samples, its outputs contain no additional beneficial high-frequency details and instead exhibit artifacts that noticeably degrade visual quality.

(2) The online model produces outputs with frequency distributions more consistent with HR images. To quantify the abnormal high-frequency information introduced by artifacts, we introduce FSDS [1], which measures the overall frequency distribution consistency between SR and HR images and reflects the extent to which abnormal frequency components are introduced; higher values indicate better performance. The comparison across three datasets is presented below.

VariantRealSR (FSDS ↑)DIV2K (FSDS ↑)ScreenSR (FSDS ↑)
Offline10.3112.1913.07
Online12.5014.5415.60

(3) A larger reward weight further exposes reward hacking. We increase the reward loss weight to 0.5. With the larger reward weight, local color blotches and artifacts become more pronounced in the offline reward baseline (see Fig. 9 of the appendix), whereas the online model again effectively avoids this issue. This further indicates that a fixed reward model can lead to reward hacking and degraded visual quality.

Taken together, these observations suggest that the offline reward model incorrectly treats spurious high-frequency artifacts and abnormal colors as scoring cues, allowing the SR policy to quickly exploit these weaknesses through reward hacking and consequently degrade visual quality. In contrast, online reward-policy co-training effectively avoids this issue by continually adapting to the evolving policy distribution. The substantial difference in visual quality was also the main motivation for moving from a fixed reward model to online co-training.

[1] Deng, Haoyu and Xu, Zijing and Duan, Yule and Wu, Xiao and Shu, Wenjie and Deng, Liang-Jian. "Exploring the low-pass filtering behavior in image super-resolution." In Proceedings of the 41st International Conference on Machine Learning, vol. 235, pp. 10553–10573. 2024.

[Q2]: Benefits of Online Preference Learning

A3:

We add two controlled baselines: (1) Direct privileged distillation: we replace the reward loss with a distillation loss that encourages the output at a lower HR-exposure level to match the stop-gradient output generated at a higher HR-exposure level using the same noise; (2) Supervised training with HR-exposure augmentation: we initialize the policy from the Warm-up model and follow the training configuration of the Co-training stage, using two different exposure levels in each training iteration. The policy is optimized solely with the supervised loss, without any reward model. The results on RealSR are reported below: our method achieves comparable or better performance in PSNR, LPIPS, DISTS, and FID, and outperforms both baselines across all no-reference perceptual metrics.

MethodPSNR↑LPIPS↓DISTS↓FID↓MUSIQ↑TOPIQ↑MANIQA↑CLIPIQA+↑Q-Align↑
Full model (Ours)24.220.29430.2140116.0069.860.69020.66070.69373.830
Direct privileged distillation24.370.30030.2137114.8169.250.67700.64120.66723.742
Supervised training with HR-exposure augmentation24.390.29900.2127114.4569.240.67100.64020.66573.717

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response Summary to Reviewers and Area Chairs

Dear Area Chair and Reviewers,

We sincerely thank you for your thoughtful feedback and efforts in helping us improve this work. We are encouraged that all three reviewers recognize the effectiveness and competitive performance of our method. Reviewers 5WMz and jKur highlight the value of policy-native preference supervision, with Reviewer 5WMz describing online reward adaptation as "a meaningful contribution beyond conventional fixed-reward post-training." Reviewer H1qx also acknowledges that our framework directly addresses the limitations of fixed reward models.

We have responded in detail to each reviewer and summarize the key points of our responses below:

  • Necessity and effectiveness of online reward adaptation. Our qualitative and quantitative analyses show that the reward model learns cues related to genuine restoration quality and maintains high agreement with human preferences under LR-only conditions without HR exposure, clearly outperforming the fixed reward baseline.

  • Improvements in visual quality. We added more qualitative analyses to the Appendix, demonstrating our method's advantages over other variants and competing methods in artifact suppression and realistic detail generation, beyond metrics.

  • Novelty and reliability of HR-exposure ordering. We clarified its distinctions from existing reward-policy co-training methods. Human evaluations and failure-case analyses further demonstrate the reliability of this preference-construction mechanism and identify factors affecting the ordering.

  • Additional baseline comparisons and framework generalization. We added comparisons with controlled baselines and recent reward-learning methods, as well as experiments on additional generative backbones, further validating our method's advantages and generality. We also reported the full training cost.

  • Generalization of the reward model. The learned reward model achieves higher agreement with human preferences than commonly used evaluation metrics in cross-model evaluation. An IQA model trained on multi-source preference data further demonstrates the scalability and broader application potential of our preference-construction mechanism.

Through this work, we hope to provide an effective framework for reward-policy co-training in generative super-resolution. Thank you again for your time and thoughtful consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.