- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper audits the reference images of M4Raw, the public benchmark for low-field MRI reconstruction. The authors find sub-pixel mis-alignment of the raw ground-truth repeated acquisition. They then build a corrected reference which corrects the mis-alignment through k-space phase-ramping (a well established method). The authors then further examine the influence of the new reference to ranking the reconstruction methods. They conclude that the published reference cannot reliably order these methods, and they release the corrected reference and the code.
Critical strengths
- An important and practical problem. Reference quality on a public benchmark that people use for leaderboards directly affects which methods the field believes are better. Auditing it is a useful service to the community, especially when published gaps between methods are only 0.05–0.5 dB.
- Rigorous and transparent methodology. The audit includes null controls, three agreeing estimators, synthetic-truth recovery, and a leave-one-repetition-out validation. The decision rule was frozen before unblinding, and deviations and exploratory analyses are disclosed in detail.
- A useful release. The corrected reference, per-volume error budgets, and code are released under open licenses.
Critical weaknesses
- Limited aded value over existing work. The misalignment and its likely cause were already reported by the M4Raw authors. Prior work has also shown that hidden reference properties bias rankings: hidden noise (Wang et al., 2024), implicit data crimes (Shimron et al., 2022), and NEX-16 vs NEX-256 references reversing low-field rankings (Shimron et al., 2024). Wang et al. (ICLR 2026) noted that imperfect or misaligned references can flip rankings for mild motion. The present paper quantifies and corrects a known issue on one dataset. That is valuable, but the conceptual contribution beyond prior work is modest.
- The ranking change attributable to misalignment alone is small. With registration as the only change (R1_official, Table 4), only one pair flips, and NAFNet stays ahead of BM3D in every contrast. The paper’s framing around “hidden misalignment” partly attributes an effect of the normalization convention to misalignment.
- Narrow scope. The confirmatory family contains only three methods (U-Net, NAFNet, BM3D). The more recent methods (Corruption2Self, ImT-MRD) are exploratory only. It is hard to extract a general lesson for learned inverse problems beyond “check alignment of multi-repetition references.”
- Limited evidence on training impact. If averaged references are used as training labels, correcting them is clearly important. However, the retraining experiment (Table 8) shows that models trained on the registered label score within about 0.1 dB of their published-label twins in PSNR, even though their spectral anisotropy changes. The practical benefit of the corrected label for training, as opposed to evaluation, is therefore not established.
- Presentation. The paper is very dense. It uses many construction labels (R0, R0_official, R1, R1_official, R2, R3), several criteria and variants, and long qualifications. A clearer main text that leads with that decomposition would improve readability.
Questions for authors
- What are the results with leave-input-out references, i.e., scoring each method only against the registered average of the other repetitions?
- Does the registration-only effect persist for more recent methods (e.g., Corruption2Self, ImT-MRD, diffusion-based denoisers) if they are included in a confirmatory analysis?
Response to Reviewer kPC4: added value, attribution, training benefit, leave-input-out
Thank you for the careful reading. We accept W2, W4 and W5, and the revision is reorganised accordingly. Terms are defined in the general response.
W1. Limited added value over existing work. We agree on what was known: the dataset paper measured the displacement, attributed it to field drift and left it uncorrected, and earlier audits showed that hidden properties of a reference bias scores and rankings (the related work now states the relation to PMoC3D). We see three additions. First, a controlled experiment on what networks take from the defect and on how the leaderboard rewards it (W4 below). Second, the effect described by the NEX study, which the leave-input-out references reproduce on M4Raw (Q1 below). Third, a mechanism that says where the problem occurs: permanent magnets without frequency tracking, and not at 3 T (the fastMRI prostate audit of App. K.11 finds no drift signature).
W2. Misalignment alone changes little, and the framing credits it with an effect of the normalisation convention. We agree; this is the main change of the revision. Sec. 5 opens with the decomposition (Table 2; general response, item 1), and the abstract and the contribution list no longer credit misalignment with the sum. Under registration alone (the comparison of Table 4 of the submission, now Table 5) the order changes in 1 of the 33 network-vs-BM3D cells of Table 2. Gaps among the six published-label networks move by a median of 0.07 dB (maximum 0.31 dB). The gap to BM3D moves for every method: by −0.17 to −0.53 dB for the six published-label networks, 21–78 % of the published gap for the five that lead BM3D. The noisy input moves too, so not all of this is learned; the twin contrast gives the learned part, 0.14–0.41 dB in 27 of 27 cells. The title is unchanged during the discussion period.
W3. A narrow confirmatory family; what is the general lesson? The family was fixed before unblinding and held the only M4Raw weights public at that time; enlarging it afterwards would defeat the pre-registration. The revision applies the same paired statistics, descriptively, to 11 networks and 12 further seed runs. The lesson is item 4 of the general response: when labels and yardstick share a defect, supervised models can take it over and the leaderboard rewards it.
W4. The benefit of the corrected label for training is not established. This is a fair reading of Table 8 of the submission (now Table 10), and the revision says so. With three seeds (Table 11) the twins are at parity in PSNR against R1_official (±0.09 dB, less than the spread between seeds), which is consistent with the measured attenuation: it costs 0.01–0.05 dB (App. K.8). What the corrected label changes is measurable: the output of a published-label twin is attenuated along the readout axis (ratio 0.93–0.97, against 0.98–1.00 for its registered-label twin), and under LPIPS and DISTS each twin is closer to the reference built like its own label (81 of 81 cells against R1_official, 74 of 81 against the published label), by modest margins. Which output is better depends on which target is better, and the registered average predicts held-out repetitions better by 1.1–2.0 dB. The practical point is the asymmetry: training on corrected labels costs nothing measurable against the registered reference and 0.10–0.36 dB against the published one. We claim no diagnostic benefit (Limitations).
W5. Presentation. We agree. Sec. 5 now opens with a notation table (Table 1) and the decomposition table (Table 2).
Q1. Results with leave-input-out references. Now run (App. K.3, Table 12). With the input left out, registration alone changes the gap to BM3D by −0.19 to −0.43 dB for the six published-label networks (18 of 18 cells significant), within about 0.1 dB of the value with the input included, and the twin contrast remains (0.09–0.30 dB, 27 of 27 cells). Leaving the input out is a factor of its own, the one the NEX study describes: it costs eight of the nine label-trained networks a further 0.18–0.55 dB against BM3D. Against the registered reference without the input, 8 of the 11 networks lead BM3D in T1w and 7 in T2w, and BM3D has the highest mean PSNR in FLAIR.
Q2. Does the registration-only effect persist for more recent methods? With the authors' weights, registration alone changes the gap to BM3D by −0.68 / −0.12 / −0.37 dB for C2S and by −0.11 / −0.07 / −0.00 dB for ImT-MRD (all significant except the last); neither changes order against BM3D. Neither was trained on the averaged labels, and the noisy input moves in the same direction, so this is not evidence of a learned defect. The submission's statement that registration displaces C2S upwards is corrected: that move belongs to the normalisation order. These rows are descriptive (Table 2). We found no public weights of a diffusion denoiser trained on M4Raw; whether the finding holds for diffusion models is untested (Limitations).