acceptodds
Withdrawn from ICLR 2027

Downstream Task Scores in Super-Resolution Measure the Pipeline, Not the Pixels Alone

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Super-resolution (SR) is increasingly evaluated by downstream task scores, but these scores depend on the reconstructed images, the downstream model, and its decision rule. We audit how decision rules affect the performance and rankings attributed to SR. In remote-sensing segmentation, we hold reconstruction pixels, model weights, and continuous score maps fixed while comparing native decisions with a global threshold selected on validation data. Several SR pipelines score zero under the native rule even though their score maps retain discrimination above chance, but below that on matched high-resolution (HR) inputs. With HR-trained models, threshold selection nearly matches the best performance among the tested global thresholds, yet leaves a substantial gap to HR under matched conditions. Threshold gains persist when downstream models are fine-tuned on SR images. Large utility gains need not change method rankings, while changing the decision rule can alter which method ranks first. Controlled photography and microscopy analyses show that decision policies and target definitions can reverse comparisons, and audits of published downstream benchmarks show the same decision sensitivity, while the gap to matched references remains larger than the threshold gain from the default operating point. These findings motivate four reporting requirements: the decision budget, the matched reference, the target and cohort definition, and the claim scope. Together, they make clear what a downstream score measures and which comparisons it supports.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Super-resolution, downstream evaluation, evaluation protocols, decision thresholds, remote sensing, model ranking stability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gM3tOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how decision rules affect downstream task scores in super-resolution evaluation. By fixing SR outputs, observer weights, and score maps while varying the decision rule, the authors show that threshold selection can substantially improve downstream F1 scores and sometimes change method rankings. Based on experiments across remote sensing and several additional domains, the paper proposes four reporting requirements for downstream SR evaluation.

Critical strengths
  1. Controlled analysis. The paper cleanly isolates the effect of the decision rule by keeping reconstructions and observers fixed. This clearly demonstrates that poor hard-decision scores may partly reflect threshold choice rather than complete loss of discriminative information.
  2. Broad empirical coverage. The analysis is repeated across multiple SR pipelines, observers, adapted models, and additional domains, showing that decision sensitivity is not limited to a single setting.
Critical weaknesses
  1. Limited novelty. The main techniques—threshold selection, continuous-score diagnostics, matched references, and bootstrap comparisons—are largely standard. The proposed reporting requirements are useful, but appear more like a consolidation of existing evaluation principles than a substantial new methodology.
  2. Poor presentation. The paper lacks a clear method/framework overview figure or other essential visualizations that would help readers understand the overall evaluation pipeline. The section organization is also unconventional and fragmented. This is a major factor behind my rejection recommendation.
Questions for authors
  1. What is the main methodological novelty beyond existing work on threshold selection, calibration, and evaluation sensitivity?
  2. Given that the experiments span heterogeneous tasks and protocols, what evidence supports the generality of the proposed framework across downstream SR evaluation?
Reviewer_44RDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper audits how downstream super-resolution (SR) scores depend on reconstructions, observers, and decision rules. Its main experiment fixes images, observer weights, and score maps while selecting one threshold per input arm using validation labels. Across five GeoSR-Bench settings, six-pipeline mean F1 rises from 0.003–0.042 to 0.195–0.467, while matched HR reaches 0.821–0.858 (pp. 4–6, §§3–4.2, Table 2). Four SegFormer/USBuildings arms make no positive predictions despite within-image AUROC of 0.804–0.824. Further audits address adapted observers, published benchmarks, controlled policies and targets, and spatial cohorts. Four reporting requirements specify decision budgets, matched references, target/cohort definitions, and claim scope (pp. 2–3, Table 1).

Critical strengths
  • The intervention identifies the contribution of the decision rule. Holding score maps fixed isolates its effect on the evaluated sample. The paper distinguishes retained discrimination, recovered hard-decision utility, and remaining HR deficits. Validation selection and the test-label finite-grid optimum are clearly separated; the latter is diagnostic (p. 4, §3.1, Eq. 1). Per-arm AP, four alternative decision rules, and matched input dimensions strengthen the analysis and its interpretation of the residual gap (pp. 15–17, Tables S1–S3).
Critical weaknesses
  1. The empirical breadth is uneven. The largest effects, including those after observer adaptation, come from 16× cross-sensor remote sensing. From a 0.5 cut, EDTR's SR arms gain 0.0170–0.0229 micro-F1; SR-COD has at most 0.0047 IoU grid headroom and retains its leader (p. 22, Tables S8–S9). The practical impact across other SR settings is therefore less clear. Since the underlying decision principles are established, the contribution rests mainly on the empirical audit and reporting framework (pp. 4, 9).

  2. Uncertainty and geographic scope remain limited. The adapted-observer gains and small leader margins of 0.0042 and 0.0006 lack paired confidence intervals (p. 7; Table S10). USBuildings covers only two regions, while the RoadDetection held-out-source check uses 48 tiles and reports no intervals (pp. 14, 18–19). The 39 source groups shared by validation and test are resampled independently within each split; clarifying how this shared-source dependence affects the interpretation of uncertainty would help. The existing checks are useful, but provide limited evidence for broader geographic transfer.

  3. The cause of the extreme score shift in the RGT example remains unclear. The fixed-score-map intervention establishes an effect of the decision rule, but does not explain why all foreground–background margins are negative in the RGT–SegFormer–USBuildings example (p. 5, Section 4.1). Existing input-domain or observer diagnostics could help interpret this specific case.

  4. The controlled examples need clearer context. City100 uses 20 test images and a diagnostic policy that accesses test HR, alongside a calibration-only policy that does not; BioSR uses 12 test cells (pp. 19–20). These are informative but small sensitivity studies. BioSR compares fixed rules across target definitions, yet the target labels and selection objective used to calibrate INPUT_CAL remain unclear (pp. 20–21).

  5. Presentation. The main text is very dense, with many quantities, acronyms and intervals per paragraph. The central argument is hard to follow on a first read. Moving the RS-specific numerical detail to the appendix, and giving one clear figure for each of the four separated quantities, would help a broad ICLR audience.

Questions for authors
  1. Can you provide paired confidence intervals for the adapted-observer gains and the two reported leader differences, stating whether thresholds are held fixed or refitted?
  2. Which target definition generated BioSR's INPUT_CAL calibration labels, and what objective selected its global threshold before the fitted rule was evaluated unchanged against all four target definitions?
  3. Do existing input-domain or score-distribution diagnostics help explain the negative margins in the RGT–SegFormer–USBuildings example in §4.1? Please distinguish evidence about their causes from the utility gains obtained by changing the threshold.
Reviewer_nPUJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper audits how downstream SR scores depend on the decision rule, rather than on reconstruction quality alone. By fixing reconstructions and observers while changing thresholds, it shows that validation-based threshold selection can substantially improve F1, while a large gap to matched HR remains and method rankings may change.

Critical strengths
  1. The main intervention is well controlled: reconstruction pixels, observer weights, and score maps are fixed, so score changes can be attributed to the decision rule.

  2. The paper highlights an important failure mode: zero native F1 can coexist with substantial continuous discrimination.

  3. The threshold analysis is repeated with observers fine-tuned on SR outputs, and the gains remain positive across all tested pipeline-observer pairs.

Critical weaknesses
  1. The main evidence comes from one remote-sensing benchmark, two tasks, and an unusually large 16x setting. The conclusions should be tested on a broader suite of SR-plus-downstream benchmarks covering natural images, multiple scales and degradations, and different downstream tasks.

  2. The City100 and BioSR studies are useful controlled demonstrations, but their targets are derived from HR image operators rather than independent task annotations. More statements and discussion should be clarified.

  3. The adapted-observer experiment reports point estimates for rank changes but no paired confidence intervals. Please quantify uncertainty for the reported margins and ranking changes.

  4. Theoretical analysis is good for your statement, and the paper would benefit from a more abstract explanation of when threshold changes should be large, for example, in terms of score shifts, calibration, foreground prevalence, or the F1-optimal threshold.

  5. The splits are not fully independent: the paper reports source-group overlap in RoadDetection and spatial overlap in USBuildings. More discussions are demetermined.

Questions for authors

Please refer to "Critical Weaknesses"

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.