- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes HePO-SR: it uses paired GT as positive samples, randomly samples negative samples from the outputs of a heterogeneous pool of pretrained SR models, and constructs a DPO preference objective in a multi-layer DINOv2 feature space, without requiring human preference labels or an explicit reward model.
Critical strengths
- The method is clear, easy to understand, and well motivated.
- The ablation studies are relatively comprehensive. The paper ablates negative pool expansion, adaptive layer weights, decoupled temperature coefficients, etc., supporting the main design choices.
Critical weaknesses
- Insufficient discussion of the drop in full-reference metrics. In the experiments, full-reference metrics such as PSNR decrease while no-reference perceptual metrics increase. The paper provides little discussion on this and does not deeply analyze the phenomenon of “perceptual improvement but fidelity degradation,” nor does it explain its relationship with DINOv2 feature preferences, heterogeneous negative samples, or the decoupled coefficients.
- No separate ablation on the positive-sample strategy. The paper emphasizes using GT as positive samples, but does not compare GT positive samples with other positive-sample construction methods (e.g., the best candidate selected by IQA ranking, or positive samples selected by an external reward model). Therefore, the contribution of the GT positive-sample design cannot be independently verified.
- Lack of direct experimental comparison with preference optimization SR methods such as DP²O / GDPO. Although the authors explain architectural incompatibility and insufficient reproducibility conditions, readers still cannot judge the actual advantage of HePO relative to existing preference optimization SR methods.
Questions for authors
- The paper provides insufficient discussion on the drop in full-reference metrics. Why do full-reference metrics decrease while no-reference metrics increase compared with before fine-tuning? Intuitively, using DINOv2 features for sample weighting computation should optimize the corresponding LPIPS?
- Can you provide a specific explanation of the heterogeneous pool ablation, such as analyzing the contribution of each SR model?
6 comments
Response to Weakness2: No separate ablation on the positive-sample strategy.
We appreciate the reviewer raising this question. This is a good suggestion. We agree that the contribution of the GT-positive design should be verified independently, and we have run exactly the comparison the reviewer suggests. In the alternative strategy, the GT positive is replaced as follows: for each training input, we rank the four pool outputs with a weighted ensemble of full-reference and no-reference metrics (PSNR, SSIM, LPIPS, DISTS, CLIP-IQA, MUSIQ, MANIQA) and take the best-ranked candidate as the positive sample; the negative is then sampled at random from the remaining three candidates. Everything else, including the backbone (OMGSR), the training configuration, and the evaluation, is identical to our main setting.
Results on RealSet65 (bold: best in each column; underline: second best):
| Variant | CLIP-IQA ↑ | MUSIQ ↑ | MANIQA ↑ | TOPIQ ↑ | AFINE-NR ↓ |
|---|---|---|---|---|---|
| OMGSR baseline | 0.7400 | ||||
| IQA-ranked positive | 69.91 | 0.6293 | 0.6844 | -1.0187 | |
| HePO-SR (GT positive) | 0.7556 | 70.54 | 0.6370 | 0.7124 | -1.0353 |
The IQA-ranked-positive variant removes nearly all of the gain: CLIP-IQA rises by only 0.0010, and the other four metrics are worse than the baseline (MUSIQ 69.91 vs 70.29, MANIQA 0.6293 vs 0.6360, TOPIQ 0.6844 vs 0.6911, AFINE-NR -1.0187 vs -1.0230). GT-positive training, in contrast, improves all five metrics.
Two further analyses support this conclusion.
(1) Relative to the negatives, the GT acts mainly as an anchor rather than as a source of perceptual gain. As shown in our response to Reviewer 3, Q1, the GT attraction term alone already raises CLIP-IQA from 0.7400 to 0.7550, close to the full objective (0.7556), but MUSIQ and MANIQA fall below the baseline and TOPIQ gains only 0.0011. Replacing a fixed anchor with a candidate that changes with each input makes the anchor unstable, which in turn affects training.
(2) We also scored the GT and the existing negatives with IQA metrics: in most cases the GT's overall score is higher, and it strictly outscores the pool negatives in 84.88% of the sampled pairs (our response to Reviewer ucq3, Q3).
This experiment therefore independently verifies the contribution of the GT-positive design. It also shows why the role of the positive and the negative should not be conflated: the positive needs a single fixed, unbiased anchor, whereas the negative benefits from preference diversity. We will add it to the appendix of the revised paper.
Response to Q2: Can you provide a specific explanation of the heterogeneous pool ablation, such as analyzing the contribution of each SR model?
We appreciate this request. In Table 3, the heterogeneous pool ablation (Variant 1 vs. Full) compares the two-model pool {SwinIR, StableSR} with the four-model pool, which adds Real-ESRGAN and DiffBIR; all other settings are the same. Adding the two sources improves CLIP-IQA (0.7513 → 0.7556), TOPIQ (0.7040 → 0.7124) and AFINE-NR (−1.0323 → −1.0353), while MUSIQ (70.59 → 70.54) and MANIQA (0.6407 → 0.6370) are lower. Because this comparison adds two sources at once, it changes both the size and the composition of the pool, and it cannot show what each source contributes.
To analyze each source and the effect of pool composition, we therefore explored this systematically: on both OSEDiff and OMGSR, we trained every negative pool that can be built from the four sources (4 single-source, 6 two-model and 4 three-model pools, plus the full pool), 30 runs in total. Each backbone uses its protocol in Appendix B, and only the negative pool changes. The table averages RealSet65 results over all pools of the same size (bold: best pool size per backbone and column; underline: second best; baselines are listed for reference):
| Backbone | Pool size (pools averaged) | CLIP-IQA ↑ | MUSIQ ↑ | MANIQA ↑ | TOPIQ ↑ | AFINE-NR ↓ |
|---|---|---|---|---|---|---|
| OSEDiff | baseline | 0.6871 | 68.91 | 0.6013 | 0.5957 | -0.9725 |
| OSEDiff | 1 (4) | 0.6994 | 69.57 | 0.6204 | 0.6153 | -0.9638 |
| OSEDiff | 2 (6) | 0.7026 | 69.73 | 0.6210 | -0.9662 | |
| OSEDiff | 3 (4) | 0.6229 | 0.6196 | |||
| OSEDiff | 4 (1) | 0.7082 | 69.88 | 0.6248 | -0.9710 | |
| OMGSR | baseline | 0.7400 | 70.29 | 0.6360 | 0.6911 | -1.0230 |
| OMGSR | 1 (4) | 0.7475 | 70.31 | 0.6370 | 0.6977 | -1.0220 |
| OMGSR | 2 (6) | 0.6368 | -1.0254 | |||
| OMGSR | 3 (4) | 0.7501 | 70.40 | 0.7044 | ||
| OMGSR | 4 (1) | 0.7556 | 70.54 | 0.6370 | 0.7124 | -1.0353 |
On average, more sources generally work better. On both backbones, every multi-model average is above the single-source average on at least four of the five metrics, and the four-model pool has the best average on four of the five. The exception is MANIQA, where the differences are small: on OSEDiff the three-model average is higher by 0.0003, and on OMGSR the four-model pool ties with the single-source average for the highest value (0.6370).
Per-source contribution, measured by removing each source from the full pool. StableSR contributes most on both backbones: removing it causes the largest drops on CLIP-IQA, MUSIQ, TOPIQ and AFINE-NR (OMGSR: −0.0187, −0.51, −0.0269, and 0.0272 worse; OSEDiff: −0.0116, −0.31, −0.0158, and 0.0086 worse), and on OMGSR it brings four metrics below the baseline. On OMGSR, the other sources play smaller roles: SwinIR mainly supports MUSIQ, MANIQA and AFINE-NR, while Real-ESRGAN and DiffBIR mainly support CLIP-IQA and TOPIQ. The role of a source can also differ between backbones: the contribution of SwinIR to CLIP-IQA and TOPIQ is slightly negative on OMGSR (−0.0029 and −0.0006) but positive on OSEDiff (+0.0050 and +0.0079), and the effect of Real-ESRGAN on CLIP-IQA is negligible on OSEDiff (0.0001, versus 0.0041 on OMGSR).
The averaged table also shows two further advantages of the pool.
Robustness. The four-model pool is the only pool size that ranks first or second on every metric on both backbones. On OMGSR, every multi-model average improves all five metrics over the baseline, whereas the single-source average is slightly below the baseline on AFINE-NR (-1.0220 versus -1.0230). On OSEDiff, AFINE-NR improves steadily as sources are added (from -0.9638 with one source to -0.9710 with four).
Training cost. All pool sizes are trained with the same protocol (same number of steps, same batch size, one negative per training sample), so the gain of the four-model pool over the single-source average comes at no extra training cost (OSEDiff: CLIP-IQA +0.0088, MUSIQ +0.31, MANIQA +0.0022, TOPIQ +0.0095, AFINE-NR better by 0.0072; OMGSR: CLIP-IQA +0.0081, MUSIQ +0.23, MANIQA equal, TOPIQ +0.0147, AFINE-NR better by 0.0133). The only extra cost is producing the cached negatives of the additional SR models, which is inference only and done once (Appendix B). In addition, finding the best combination of k sources with cached outputs would require training all combinations: 15 runs per backbone for our four sources (the search reported above), with the count roughly doubling for every added source and the search repeated for every new backbone. This cost is hard to justify in practice, while the fixed pool needs one run per backbone.
We will add the averaged table and the remaining results to the appendix of the revised paper.
Response to Q1&&Weakness1:Why do full-reference metrics decrease while no-reference metrics increase compared with before fine-tuning? Intuitively, using DINOv2 features for sample weighting computation should optimize the corresponding LPIPS?
We appreciate the reviewer raising this question. This is a good question. We agree that the paper should discuss the full-reference (FR) drop more explicitly, and we will add such a discussion.
discussion on the drop in full-reference metrics. This is the well-known perception-distortion trade-off. HePO-SR is a preference objective that targets perceptual quality, so a moderate decrease in distortion-type metrics is expected rather than anomalous. The distortion-side change in our results is small: on DRealSR with OMGSR, PSNR moves from 27.56 to 27.50 dB and SSIM improves (0.7465 to 0.7518), while the main changes appear in LPIPS (0.3266 to 0.3378) and DISTS (0.2335 to 0.2512). The repulsion term is what moves the output away from the over-smoothed negatives, and the gains on most no-reference metrics come from that same term (see our response to Reviewer ucq3, Q1). The change in FR metrics is the cost of that term, and the ratio of β1 to β2 decides where the model sits on this trade-off. In addition, a blinded user study on DRealSR is reported in our response to Reviewer LHEm, Q4: raters prefer our output on fidelity in 63.4% of images and on naturalness in 73.1%, so the FR drop is not perceived as a loss of structural fidelity. We will add this discussion of the perception-distortion trade-off to the revised paper.
DINOv2-based sample weighting and LPIPS. We separate two questions: what the weighting does, and whether it should reduce LPIPS.
(1) What the weighting does. The adaptive weights enter Eq. (12): for each sample they combine the five layer-wise preference logits, and each logit is −logσ(β/(2s2)(A+−A−)), a comparison between the model's and the reference's distance to the same target. The weights therefore decide how much each layer contributes to the preference comparison for that sample; they do not reweight channels of a feature distance to the GT. Because the layers encode different levels of content, we let the weights depend on the sample instead of fixing them, and Table 3 measures the effect: with the same layers and base weights, adaptive weighting raises CLIP-IQA (0.7408 to 0.7556), MUSIQ (70.13 to 70.55), TOPIQ (0.6868 to 0.7124) and AFINE-NR (-1.0095 to -1.0353), with MANIQA slightly lower (0.6376 to 0.6370).
(2) Whether this should reduce LPIPS. We have no evidence for such a relation, so we do not claim it. LPIPS is a fixed weighted distance in a pretrained VGG feature space, whereas our weights act on preference comparisons in DINOv2 space. We have not established a correspondence between the two, nor any result showing that a weighting in one space controls a distance in the other. What we can report is that the FR metrics move with the β1/β2 ratio: on DRealSR, moving from (β1,β2)=(5,3) to (4,4) and (3,5) changes PSNR from 27.50 to 27.16 and 27.03 dB.