acceptodds
Withdrawn from ICLR 2027

Reference-Dependent Scores, Margin-Stable Decisions: Calibrating MRI Reconstruction Benchmarks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reference-based MRI benchmarks conflate reconstruction quality with the statistical compatibility between an input and its evaluation target. We introduce a calibration framework for this dependence in repeated-acquisition MRI. When a reference contains the same measured repetition used to form an accelerated input, realization-specific reconstruction error can correlate with reference noise and shift method scores even when repeat count and averaging are matched. We separate this method-level score distortion from the downstream decision instability it may induce. A margin-normalized certificate, ρ, makes the distinction explicit: the disjoint-reference winner is guaranteed to remain the winner when ρ < 1, whereas near-tied candidates are vulnerable to much smaller differential score shifts. On M4Raw 0.3T data, an all-slice T1/T2 study over 540 subject–contrast–slice units yields positive independent-validator loss for shared-reference selection (0.00916 NRMSE on V1 and 0.01389 on V2), and the direction transfers to 270 FLAIR units. Adding the official PI-VarNet nearly removes selection harm because the learned model creates a large candidate margin, even though its shared–disjoint score shift is the largest in the pool. A second competitive learned candidate restores 70/540 per-unit selector changes while independent-validator loss remains small. Within the classical pool, a runner-up margin approximation explains 205 of 220 observed selector changes with no false positives. These results motivate a practical reporting protocol for repeated-acquisition benchmarks: pair reference provenance and score sensitivity with candidate-margin-based decision stability.

Primary area:
datasets and benchmarks
Keywords:
MRI reconstruction, evaluation benchmarks, reference dependence, reconstruction metrics, decision stability, repeated acquisitions

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4nFQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how the construction of reference impacts performance when a restoration method's output is scored against pseudo-labels generated by multi-repetition averaging, particularly in MRI datasets. When the noisy observation used as the restoration input also contributes to the target being scored, the score rewards carrying that noise over and inflates methods' apparent performance. The analysis identifies this effect in both classical and deep learning-based approaches under NRMSE, and recommends scoring against repetition-averaged references that exclude the input repetition. A selection margin is used to analyze when reference overlap changes the selected method, and experiments on a real-world MRI dataset show discrepancies in method selection depending on whether the reference includes the input repetition.

Critical strengths
  1. The authors have identified an important issue in repetition-average-based benchmarks that reward noise carrying, and the analysis, formalized using selection loss D(V)D(V), shows a positive trend across reported groups of methods, justifying the practice of reporting this metric along with the traditional ones for robustness.
Critical weaknesses
  1. The experiment setup and interpretation are confounded. In an RSS magnitude averaging operation, a positive noise floor is left on the images; thus, the reference and validators could have been rewarding methods that reproduce this bias. Further, D(V) is defined over a hand-crafted pool of methods, and its value depends strongly on the pool's composition. Moreover, the reported selection losses are very small; along with the limited method pool, it is unclear whether reference overlap would change practical assessments when stronger baselines are included.

  2. The scope and clarity of the paper need improvement. First, the authors' proposed selection loss was only tested and validated on one multi-repetition averaged MRI dataset, whereas another experiment using a noiseless dataset with synthetic injected noise could separate the contribution much more cleanly due to known noise distributions as well as no motion/magnetic field drifts. Second, several notations are used interchangeably and hinder the readability of the paper. For example, reference shift Δref(m)\Delta_{ref}(m) seems to be the same quantity as δm\delta_m.

Questions for authors
  1. Since references and validators are built using RSS magnitude averaging, a positive noise floor is left on all references and validators. Could you use complex averaging and show whether the conclusion changes at all?

  2. Could you include visualizations of the reconstructed images selected using shared versus disjoint references, along with error maps to demonstrate noise carry-through?

  3. Would it be possible to test on data with known ground truth (i.e., simulated white Gaussian noise with independence across repetitions) so that the effect of D(V)>0D(V)>0 selecting methods that are further from the true image can be demonstrated?

  4. Has there been any attempt to model or address motion or magnetic field drift in the dataset since you used different repetitions for different purposes? Appendix 5 only provided a one-line explanation.

  5. Could you clarify: Is Δref(m)\Delta_{ref}(m) equivalent to δm\delta_m? It would be helpful if all new quantities defined in the paper shared the same notation and naming.

Reviewer_UiQbOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies how the construction of the reference image affects MRI reconstruction benchmarks when repeated acquisitions are available. The central argument is that when the evaluation reference contains the same measured repetition used to form the undersampled input (“reference overlap”), reconstruction error can correlate with reference noise. The authors conclude that reference overlap consistently distorts method-level scores, but changes benchmark decisions only when candidate methods are near-tied, and they recommend reporting reference provenance and disjoint-reference sensitivity as part of benchmark design.

Critical strengths
  1. A relevant evaluation question. Retrospective undersampling routinely scores a reconstruction against the same acquisition it was derived from. The point that a shared noise realization can reward methods that preserve that noise rather than recover the anatomy applies broadly to reconstruction benchmarks.

  2. Separating score distortion from decision change. Distinguishing per-method score shifts (Δ_ref), selection changes, and validator loss is a useful observation. In particular, a dominant method can leave selections stable while its score remains reference-dependent.

Critical weaknesses
  1. Limited novelty. The new element is an empirical test of overlap with matched repeat counts on one dataset. That is useful, but the conceptual contribution is modest for ICLR. The decomposition in Eq. 1, where the cross term E⟨e_m, n_R⟩ vanishes for an independent reference, is a standard result, and the same independence principle underlies Noise2Noise/Noise2Self and measurement-splitting approaches such as SSDU. Wang et al. (2024) already showed that the noise in conventional references (“hidden noise”) can change which reconstruction is preferred. The margin-stability criterion (Sec. 6.1) is an algebraic restatement that a winner remains the winner when its differential score shift is smaller than its margin. It is also retrospective, since computing it requires both references.

  2. The practical effect is small and largely disappears with modern methods. The large effects come from the classical pool (D ≈ 0.009/0.014 NRMSE). With PI-VarNet or M4c2 added alone, selections change on 1/540 and 0/540 units. With both learned models, D = 0.00047/0.00062 NRMSE. The aggregate ranking is unchanged under both references: M4c2 leads in each case (p. 5; Supp. Sec. 8). For the comparisons a benchmark is mainly used for, which are among competitive learned methods, the paper therefore shows a negligible decision-level effect.

  3. Presentation. A simple message is obscured by heavy notation and terminology: R_sh, R_dis, V₁, V₂, D, D_sel, Δ_ref, δ_m, γ, γ_m, ρ, C3, C5, M4c2, and Cohorts A and B. The basic variables in Eq. 1 (x, e_m, n_R) are introduced tersely, and the introduction frames the issue in a way that is easy to confuse with ordinary input fidelity. It does not state plainly that the problem is a shared noise realization between input and reference. The supplement also refers to earlier experimental iterations (“Cohort A/B,” an earlier invalid “M4c” run, a broadcasting defect), which reads more like a revision log than supporting material. A shorter paper with one clear schematic, one main table, and the aggregate-selection result up front would communicate the finding much more effectively.

Questions for authors
  1. How large are the effects with stronger and better-tuned baselines, and at other acceleration factors?
  2. Do noise-aware estimators (e.g., SURE variants or the approach of Wang et al., 2024) or input-splitting schemes remove the selection effect without requiring extra repetitions?
Reviewer_ccJdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper discussed the reference effect for MRI evaluaiton. The paper found that if the reference contains the same measured repetition the same as the input, the best algorithm selection may be biased. And the paper suggest to report reference provenance together with decision stability for the future evaluation.

Critical strengths

The studied problem in this paper is less studied and very important for MRI evaluation. And the experiment design is clean which use the same number of repetitions for 2 references and differ only in whether they contain the input repetition. The paper is good written, and the Figure 1 is clearly present the overall design.

Critical weaknesses

W1 The paper interpret the differences in noise. This could be further discussed. For example, the different sample of reference may also lead to motion. If different references have different motion, then the ranking differences are also reasonable.

W2 The model selection change could also from the performance noise, if 2 algorithms have similar performance,the selection may change just due to the performance noise. Authors should discuss more to avoid this concern.

Questions for authors

Q1 In Figure 2, the selection changes are counted per slice. However, the usual evaluation is based on the whole dataset, will this lead to bias?

Q2 If the repeated acquisitions were registered to each other before constructing the references, would the model selection changes still be large? This could help to solve the concern on W1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.