acceptodds
Under review as a conference paper at ICLR 2027

Hidden Misalignment in Low-Field MRI Benchmark References: The Leaderboard Depends on How the Reference Is Built

Abstract

Deep-learning enhancement is the most promising route to clinical-quality low-field MRI. Its progress is measured on M4Raw, the only public low-field (0.3 T) raw k-space benchmark. There, every method is scored against the average of repeated acquisitions. We show that this reference is systematically misaligned, and that the leaderboard built on it depends on how the reference is constructed. An audit of all 240 subject-contrast groups finds sub-pixel displacements between repetitions. Registering them lowers the reference's own noise level by 6-14% (validation) and 15-25% (test). The displacement lies along the readout axis, has the same sign in every validation group and 99% of test groups, and grows linearly with acquisition order (about 2.3 ppm per repetition). This is the signature of an uncompensated frequency drift of the permanent magnet, with head motion on top in the labelled motion subset. We build a corrected reference by exact k-space phase-ramp registration and phase-aligned complex averaging, with a per-volume error budget. It is validated by synthetic-truth recovery and leave-one-repetition-out prediction. We also measure the reference's own resolution. Two disjoint halves of the published reference agree only to 28.8-31.7 dB, with a per-subject spread of 1.0-1.7 dB. Registration raises the agreement by 1.1-1.8 dB and shrinks the spread two- to three-fold. We re-score the publicly released baselines under a decision rule frozen before unblinding. For every network-vs-BM3D pair the reference x method interaction is Holm-significant and same-signed under every reference construction we test. Registering the reference under the published protocol, and changing nothing else, removes about half of NAFNet's published lead over BM3D (+0.94/+0.87/+0.37 -> +0.50/+0.49/+0.20 dB). Whether the ordering then inverts depends on the reference construction (one, four or five pairs under the three constructions we test) and on the metric and region. The published reference cannot order these methods. We release the corrected reference (CC BY), the error budgets and the code.

Primary area:
datasets and benchmarks
Keywords:
low-field MRI, benchmark audit, reference quality, image denoising, sub-pixel registration, B0 drift, pre-registered analysis, ranking stability, error budget, M4Raw

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kPC4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper audits the reference images of M4Raw, the public benchmark for low-field MRI reconstruction. The authors find sub-pixel mis-alignment of the raw ground-truth repeated acquisition. They then build a corrected reference which corrects the mis-alignment through k-space phase-ramping (a well established method). The authors then further examine the influence of the new reference to ranking the reconstruction methods. They conclude that the published reference cannot reliably order these methods, and they release the corrected reference and the code.

Critical strengths
  1. An important and practical problem. Reference quality on a public benchmark that people use for leaderboards directly affects which methods the field believes are better. Auditing it is a useful service to the community, especially when published gaps between methods are only 0.05–0.5 dB.
  2. Rigorous and transparent methodology. The audit includes null controls, three agreeing estimators, synthetic-truth recovery, and a leave-one-repetition-out validation. The decision rule was frozen before unblinding, and deviations and exploratory analyses are disclosed in detail.
  3. A useful release. The corrected reference, per-volume error budgets, and code are released under open licenses.
Critical weaknesses
  1. Limited aded value over existing work. The misalignment and its likely cause were already reported by the M4Raw authors. Prior work has also shown that hidden reference properties bias rankings: hidden noise (Wang et al., 2024), implicit data crimes (Shimron et al., 2022), and NEX-16 vs NEX-256 references reversing low-field rankings (Shimron et al., 2024). Wang et al. (ICLR 2026) noted that imperfect or misaligned references can flip rankings for mild motion. The present paper quantifies and corrects a known issue on one dataset. That is valuable, but the conceptual contribution beyond prior work is modest.
  2. The ranking change attributable to misalignment alone is small. With registration as the only change (R1_official, Table 4), only one pair flips, and NAFNet stays ahead of BM3D in every contrast. The paper’s framing around “hidden misalignment” partly attributes an effect of the normalization convention to misalignment.
  3. Narrow scope. The confirmatory family contains only three methods (U-Net, NAFNet, BM3D). The more recent methods (Corruption2Self, ImT-MRD) are exploratory only. It is hard to extract a general lesson for learned inverse problems beyond “check alignment of multi-repetition references.”
  4. Limited evidence on training impact. If averaged references are used as training labels, correcting them is clearly important. However, the retraining experiment (Table 8) shows that models trained on the registered label score within about 0.1 dB of their published-label twins in PSNR, even though their spectral anisotropy changes. The practical benefit of the corrected label for training, as opposed to evaluation, is therefore not established.
  5. Presentation. The paper is very dense. It uses many construction labels (R0, R0_official, R1, R1_official, R2, R3), several criteria and variants, and long qualifications. A clearer main text that leads with that decomposition would improve readability.
Questions for authors
  1. What are the results with leave-input-out references, i.e., scoring each method only against the registered average of the other repetitions?
  2. Does the registration-only effect persist for more recent methods (e.g., Corruption2Self, ImT-MRD, diffusion-based denoisers) if they are included in a confirmatory analysis?
1 comment
AuthorsOpenReview ↗

Response to Reviewer kPC4: added value, attribution, training benefit, leave-input-out

Thank you for the careful reading. We accept W2, W4 and W5, and the revision is reorganised accordingly. Terms are defined in the general response.

W1. Limited added value over existing work. We agree on what was known: the dataset paper measured the displacement, attributed it to field drift and left it uncorrected, and earlier audits showed that hidden properties of a reference bias scores and rankings (the related work now states the relation to PMoC3D). We see three additions. First, a controlled experiment on what networks take from the defect and on how the leaderboard rewards it (W4 below). Second, the effect described by the NEX study, which the leave-input-out references reproduce on M4Raw (Q1 below). Third, a mechanism that says where the problem occurs: permanent magnets without frequency tracking, and not at 3 T (the fastMRI prostate audit of App. K.11 finds no drift signature).

W2. Misalignment alone changes little, and the framing credits it with an effect of the normalisation convention. We agree; this is the main change of the revision. Sec. 5 opens with the decomposition (Table 2; general response, item 1), and the abstract and the contribution list no longer credit misalignment with the sum. Under registration alone (the comparison of Table 4 of the submission, now Table 5) the order changes in 1 of the 33 network-vs-BM3D cells of Table 2. Gaps among the six published-label networks move by a median of 0.07 dB (maximum 0.31 dB). The gap to BM3D moves for every method: by −0.17 to −0.53 dB for the six published-label networks, 21–78 % of the published gap for the five that lead BM3D. The noisy input moves too, so not all of this is learned; the twin contrast gives the learned part, 0.14–0.41 dB in 27 of 27 cells. The title is unchanged during the discussion period.

W3. A narrow confirmatory family; what is the general lesson? The family was fixed before unblinding and held the only M4Raw weights public at that time; enlarging it afterwards would defeat the pre-registration. The revision applies the same paired statistics, descriptively, to 11 networks and 12 further seed runs. The lesson is item 4 of the general response: when labels and yardstick share a defect, supervised models can take it over and the leaderboard rewards it.

W4. The benefit of the corrected label for training is not established. This is a fair reading of Table 8 of the submission (now Table 10), and the revision says so. With three seeds (Table 11) the twins are at parity in PSNR against R1_official (±0.09 dB, less than the spread between seeds), which is consistent with the measured attenuation: it costs 0.01–0.05 dB (App. K.8). What the corrected label changes is measurable: the output of a published-label twin is attenuated along the readout axis (ratio 0.93–0.97, against 0.98–1.00 for its registered-label twin), and under LPIPS and DISTS each twin is closer to the reference built like its own label (81 of 81 cells against R1_official, 74 of 81 against the published label), by modest margins. Which output is better depends on which target is better, and the registered average predicts held-out repetitions better by 1.1–2.0 dB. The practical point is the asymmetry: training on corrected labels costs nothing measurable against the registered reference and 0.10–0.36 dB against the published one. We claim no diagnostic benefit (Limitations).

W5. Presentation. We agree. Sec. 5 now opens with a notation table (Table 1) and the decomposition table (Table 2).

Q1. Results with leave-input-out references. Now run (App. K.3, Table 12). With the input left out, registration alone changes the gap to BM3D by −0.19 to −0.43 dB for the six published-label networks (18 of 18 cells significant), within about 0.1 dB of the value with the input included, and the twin contrast remains (0.09–0.30 dB, 27 of 27 cells). Leaving the input out is a factor of its own, the one the NEX study describes: it costs eight of the nine label-trained networks a further 0.18–0.55 dB against BM3D. Against the registered reference without the input, 8 of the 11 networks lead BM3D in T1w and 7 in T2w, and BM3D has the highest mean PSNR in FLAIR.

Q2. Does the registration-only effect persist for more recent methods? With the authors' weights, registration alone changes the gap to BM3D by −0.68 / −0.12 / −0.37 dB for C2S and by −0.11 / −0.07 / −0.00 dB for ImT-MRD (all significant except the last); neither changes order against BM3D. Neither was trained on the averaged labels, and the noisy input moves in the same direction, so this is not evidence of a learned defect. The submission's statement that registration displaces C2S upwards is corrected: that move belongs to the normalisation order. These rows are descriptive (Table 2). We found no public weights of a diffusion denoiser trained on M4Raw; whether the finding holds for diffusion models is untested (Limitations).

Reviewer_HbxbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work addresses the test image misalignment issues for M4Raw low-field MRI benchmark. It identified the systematic subpixel displacement between repeated scan, which may be influenced by potential B0 drift from low-field MRI. The proposed study proposes scan alignment and registration methods to improve the reference dataset.

Overall, although it addresses an important issues regarding benchmark reliability, it only addresses a single dataset and the experiment results do not clearly indicate whether claimed misalignment is mainly responsible for the suggested benchmark issue. It is not clear whether this is considered as contribution sufficient to ICLR.

Critical strengths
  1. It addresses an important problem regarding reference construction in MRI experiment benchmarks

  2. The authors provide a detailed analysis of the reference misalignment between repeated scans

  3. A detailed comparison results are included with respect to the reference construction.

Critical weaknesses
  1. It only addresses a single low-field MRI dataset using one scanner and acquisition configuration. It is unclear whether this can be generalized to other MRI datasets (different acquisition configuration and protocols). Also, the proposed displacement correction methods employs a simple existing method. I am not sure whether this is a significant contribution enough to ICLR. Further justification is necessary to demonstrate the generalizability.

  2. There is no comparative experiments between the proposed misalignment and existing B0 drift correction or registration methods. It unclear the true contribution beyond applying existing method.

  3. Although the main motivation of the proposed method is the misalignment between repeated scans which reduces the benchmark reference quality, the experiment results seems to reveal that it is not the most important component. In Section 5.3, the comparison changes the order between registration and normalization procedures, and the results show that alignment correction alone is not the main cause of the ranking reversal. Section 5.4 show the results with registration alone (normalization remain unchanged), and it does not change the ranking while NAFNet’s advantages over BM3D has been reduced substantially (across T1w, T2w, FLAIR). The normalization convention itself seems to be more influential, based on the results. It is necessary to analyze the effect of each component independently (misalignment, normalization and other reference construction parameters).

  4. A more crucial concern is that the ranking instability is a more fundamental issue caused by the limitation of the evaluation method itself. PSNR/SSIM measures pixelwise distance which does not convey true task-specific or true clinical importance of the test images. The ranking by small difference of PSNR/SSIM does not provide statistically / clinically meaningful information. Medical imaging essentially require task-specific evalutions. As an example, fastMRI challenge has included the radiologist evaluations. I think addressing and improving the evaluation metric itself is much more important direction to explore.

  5. The statement in the abstract “the published reference cannot order these methods” is too strong compare to its experiment results. I recommend to adjust expression to sensitivity and limited robustness of reference based evaluation method.

Questions for authors
  1. Can this be generalized to other dataset? Please also provide the justification.

  2. How does the proposed method differ from existing B0 drift correction or registration methods?

  3. Based on the results in 5.3 & 5.4, how can authors interpret the observed ranking reversal with the paper’s primary claim of misalignment?

  4. Is there an evidence that PSNR/SSIM conveys meaningful information regarding clinical and diagnostic quality? How can the small difference in PSNR be interpreted?

  5. As the suggested ranking depends on reference construction, normalization, and evaluation metric, how is the claim “the published reference cannot order these methods” justified?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Hbxb: misalignment vs normalisation, conclusion, scope, image quality

Thank you for the detailed report. W3 and W5 are correct, and the revision is reorganised around them, so we answer them first. Terms are defined in the general response.

W3 / Q3. Alignment and normalisation order change together; how important is each? Table 2 separates them for 11 networks. For the official NAFNet against BM3D the pre-registered change (−1.34 / −1.10 / −0.67 dB) is the sum of a registration component (−0.44 / −0.39 / −0.17) and a normalisation-order component (−0.91 / −0.71 / −0.50), the larger of the two. The revision attributes to misalignment only the registration component, and within it only what the controls support. Under registration alone every published-label network moves against BM3D (17 of 18 cells significant), but so does the noisy input (−0.19 to −0.55 dB), so the sign alone is not evidence of a learned defect. The twins isolate the learned part: a published-label network loses 0.14–0.41 dB more than its twin (27 of 27 cells over three seeds, a median of 74 % of its loss). The normalisation-order component has no retraining control. The revision reports it as a dependence on a convention of the protocol, not as a learned effect and not as misalignment (general response, item 1). Under registration alone (published normalisation) one pair inverts, official U-Net against BM3D in T2w, and NAFNet stays ahead of BM3D, as you note. Under the average-then-normalise order its unregistered lead is only +0.09 / +0.05 / +0.01 dB, and registration inverts it (App. G.1).

W5 / Q5. "Cannot order these methods" is stronger than the evidence. We agree and removed the sentence from the abstract and from Sec. 5. The new closing sentences are quoted in the general response (item 2). Sec. 5 adds three limits, among them that the better self-consistency of the registered reference does not show that its ranking reflects clinical image quality better.

W1 / Q1. One dataset and one scanner. The lessons we expect to carry over are in the general response (item 4). The misalignment itself is expected where four conditions meet: a permanent magnet, no frequency tracking between repetitions, a low readout bandwidth, and repetitions averaged as acquired. App. K.9 tabulates the expected displacement for documented drift rates: 0.16 px per repetition for a 50 mT Halbach magnet in an M4Raw-like protocol unless the frequency is tracked, less than 0.01 px at 3 T. As a second dataset we audited the fastMRI prostate DWI averages (3 T, another vendor, organ and sequence; App. K.11): the shifts are slice-dependent with no dominant axis, the systematic component is 14 to 34 times smaller per acquisition than on M4Raw, and the audit finds no drift signature. We did not re-score methods there. The consequence for rankings is measured on one benchmark, because to our knowledge M4Raw is the only public low-field dataset with raw repeated acquisitions.

W2 / Q2. Relation to B0 correction and to established registration methods. For a Cartesian spin-echo readout, a centre-frequency offset between two repetitions appears in the image plane as a shift along the readout axis plus a global phase. The k-space phase ramp we apply is the exact inverse of that in-plane part, so it is the retrospective frequency-drift correction for this acquisition (App. K.7). Field-map unwarping and navigator-based tracking need data that the release does not contain. The revision compares four translation estimators on the test split (Table 15): the registration of SimpleITK, an established package, differs from ours by a median of 0.03 px per slice. The operator is classical. The contribution is the chain around it: a mechanism with a testable signature, a validated reference with error budgets, a paired test of the gaps between methods, and a retraining control.

W4 / Q4. Do the ranking changes correspond to meaningful differences in image quality? We agree with the premise. We have no reader study and claim no diagnostic benefit, and the Limitations name radiologist evaluation as the appropriate endpoint. Sec. 5 says that PSNR gaps of this size should not be read as differences in image quality: the gap of a published-label network to BM3D is about 1 dB or less, and it moves by 0.17–0.53 dB with registration, by up to 0.91 dB with the normalisation order and by 0.18–0.77 dB with the input repetition. Under LPIPS and DISTS (App. K.6) the label-trained networks stay far closer to the reference than BM3D under every construction, so the PSNR results should not be read as BM3D being as good as the networks. These metrics are reference-based too, so part of this may be agreement with the label. The registered reference is the better estimate by a criterion that needs no ground truth and no denoiser: it predicts held-out repetitions better by 1.1–2.0 dB. Our contribution to the metric problem you raise is one measured mechanism by which a reference-based metric rewards a defect.

Thank you for the new analyses and clarification. However, my main concerns are only partially resolved.

  1. It is helpful to separate the effect of the normalization and misalignment registration. However, the new results still indicate the ranking reversal depend more on normalization convention than misalignment registration. Specifically, registration alone reverses one of 33, whereas the combination of registration & normalization show 14 of 33 changes.

  2. I appreciate the additional results with LPIPS and DISTS evaluations. The results show that trained networks always outperform BM3D with these measures regardless of the reference construction. This may imply that PSNR itself is not a reliable metric, where small difference in PSNR may not convey the true (task-specific or clinical) image quality differences.

  3. I also appreciate the comparative analysis with existing registration methods. The theoretical analysis of expected displacement under different scanner and the fastMRI prostate DWI experiment are helpful to identify the condition where such misalignment may occur. However, the experiments are still conducted only with MR4raw. Comparisons with SimpleITK is also helpful, but I remain somewhat uncertain about the generalizability and broader impact of this work.

AuthorsOpenReview ↗

Reply to Reviewer Hbxb: what the paper claims on misalignment, PSNR reversals and scope

Thank you for reading the revision and our responses so closely. We agree with the limits you name, and we use this reply to state what the paper claims within them.

1. The central finding. We agree as far as the re-scoring goes. Its finding is that the leaderboard is sensitive to how the reference is built, and under the published normalisation registration alone reverses one comparison. The audit of Secs. 2 to 4 concerns the misalignment itself: registration lowers the noise level of the reference by 15–25 % on the test split. Two points on how we read the re-scoring. First, a count of reversals depends on which of the two changes is made first: with the normalisation changed first, registration reverses 8 of the 33 comparisons and the normalisation 6 (App. K.2). The revision therefore reports the change of the gap for every network and treats a reversal as an outcome that depends on the construction. In gap size the two components are comparable. In Table 2 the registration component is as large as the normalisation-order component, or larger, for four of the six published-label networks in every contrast; the official NAFNet is the main exception. Under registration alone the gap of the six networks to BM3D changes by −0.17 to −0.53 dB (17 of 18 cells significant), 21–78 % of the published lead of the five that lead BM3D. Second, misalignment is the one construction factor for which the paper has a physical mechanism, a repair validated on held-out repetitions, and a retraining control. That control shows that part of the registration effect is learned from the labels: a published-label network loses 0.14–0.41 dB more than its twin (27 of 27 cells over three seeds). For the normalisation order the paper shows a dependence and claims nothing more.

2. Perceptual metrics and practical significance. We agree, and we do not claim that the PSNR reversals reflect image quality. Sec. 5 says that PSNR gaps of this size should not be read as differences in image quality, and App. K.6 that the PSNR results should not be read as BM3D being as good as the networks. The Limitations state that no downstream or reader-study endpoint is presented. A reader study or a task-based evaluation is the right next step; we cannot add one in this period. What the paper does claim is narrower. The gap of a published-label network to BM3D is about 1 dB or less on the benchmark's own metric. It moves by 0.17–0.53 dB with registration, by up to 0.91 dB with the normalisation order and by 0.18–0.77 dB with the input repetition. The dependence also has a practical side. A network trained on registered labels scores 0.10–0.36 dB lower against the published label than its twin, while the two agree to ±0.09 dB against R1_official. A group that registers its labels therefore scores lower on the current leaderboard than with the published labels. The perceptual metrics also follow the training label: each twin is closer to the reference built like its own label (81 of 81 cells against R1_official, 74 of 81 against the published label), by modest margins. They are reference-based too, so this does not show which output is better.

3. Scope and novelty. We agree that the consequence for rankings is demonstrated on one benchmark. The Limitations say so: one site and one scanner, and an effect size that will depend on drift rate, bandwidth, number of averages and training labels. On novelty beyond this benchmark we would point to the method of the audit, not to the registration operator, which is classical. Three elements do not depend on M4Raw. First, a four-step audit for any dataset with repeated acquisitions (App. K.10). Its last step is the paired change of the gap between two methods when the reference changes, which the pre-registered rule tests, with a descriptive decomposition by construction factor. On the fastMRI prostate data the first two steps find no drift signature, so the audit can also return a negative result; methods were not re-scored there. Second, the retraining control. Under registration alone the noisy input and two methods never trained on these labels also move against BM3D, so a re-scoring study alone cannot attribute the effect to learning. Twins trained on the published and on registered labels can. We are not aware of an earlier audit of an MRI benchmark that isolates a learned effect of a geometric defect with such a control. Third, a mechanism that says where the displacement is expected: permanent magnets without frequency tracking, and not at 3 T (App. K.9). We see the paper as an audit of the one public benchmark of its kind, with a method that transfers, and the revision words its lessons accordingly ("shown here on one benchmark").

Thank you for the response. I think, rather than focusing on the misalignment alone, a broader discussion of evaluation metrics and protocols in MRI benchmarks (e.g. reference construction, limitations of error metrics, perceptual / clinical evaluations) would be a promising direction and make the contribution much stronger. Although this work includes interesting results, its scope is somewhat limited. I keep my original score.

Reviewer_h9BwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper discussed the evaluation problem in the low-field MRI dataset, i.e. M4Raw. The paper report how sub-pixel displacements on the M4Raw references will effect the evaluation, and the current method rankings may be biased. The authors build a corrected reference by registering the repeated acquisitions, and show that the published rankings are sensitive to how the reference is constructed.

Critical strengths

S1 The paper discussed the misalignment problem in the M4Raw dataset and discussed how imperfect references can effect the evaluation, which is very important for the MRI evaluation.

S2 The paper is well written, and all figures and table are clearly presented.

Critical weaknesses

W1 The applied registration method limited to 2D in-plane translation. However, the rotation and through-plane motion may also effect the final results. Therefore, the corrected reference may still be imperfect, and the ranking may still be biased.

W2 The main finding of the paper is based the method rankings. However, the evidence on the ranking reverse is limited, which may not be sufficient to support the conclusion. Considering more methods for the ranking comparison may make the conclusion more convincing.

Questions for authors

Q1 Is this problem only happens for the M4Raw dataset, or is this problem a general problem for low-field MRI denoising?

Q2 The paper mainly use the pixel-based metrics to evaluate the denoising, and lots of recent works also use the perceptual metrics like DISTS[1] and LPIPS[2] to evaluate. Will the ranking change if we use the perceptual metrics?

[1] Ding, Keyan, et al. "Image quality assessment: Unifying structure and texture similarity." IEEE transactions on pattern analysis and machine intelligence 44.5 (2020): 2567-2581.

[2] Zhang, Richard, et al. "The unreasonable effectiveness of deep features as a perceptual metric." 2018 IEEE/CVF conference on computer vision and pattern recognition. IEEE, 2018.

2 comments
AuthorsOpenReview ↗

Response to Reviewer h9Bw: rotation, more methods, generality, perceptual metrics

Thank you for the positive assessment and four concrete points. Each led to a new analysis. Terms are defined in the general response.

W1. Registration is limited to in-plane translation; rotation and through-plane motion may leave the corrected reference imperfect. We agree, and the revision now says so. Rotation. A reference that also applies the estimated in-plane rotation (median 0.07–0.20°; App. K.4) changes E by at most 0.08 dB for the nine networks trained on M4Raw labels. For the official NAFNet, registration alone gives −0.52 / −0.41 / −0.19 dB with the rotation and −0.44 / −0.39 / −0.17 dB without it. Through-plane. It cannot be corrected from 2-D data with 5 mm slices. For an assumed slice-selection bandwidth of 1–2 kHz the drift moves the excited slice by 0.08–0.15 mm per repetition. It enters the error budget of each volume, and its effect on the gaps between methods cannot be measured (Limitations). Why the registered reference is still the better one. The registered average of the other repetitions predicts a held-out repetition better than the unregistered average does, by 1.98 / 1.09 / 1.52 dB. This check needs no ground truth and no denoiser. We no longer suggest that the corrected reference gives the final ranking.

W2. The evidence on ranking reversal is limited; more methods would be more convincing. We agree. The revision reports E for every network and treats a reversal as an outcome that depends on the construction. Among the 33 network-vs-BM3D cells the order changes in 1 cell under registration alone and in 14 cells under the pre-registered comparison, which also changes the normalisation order. The paired analysis now covers 11 networks (Table 2): the two official checkpoints; NAFNet-w32, SCUNet, Restormer and U-Net-64, trained by us on the published labels; three of these also trained on registered labels; and C2S and ImT-MRD with their authors' weights. Under registration alone each of the six published-label networks moves against BM3D, by −0.17 to −0.53 dB (significant in 17 of 18 cells). The noisy input moves too (−0.19 to −0.55 dB), so the sign alone does not show a learned effect. The twin experiment does: a published-label network loses 0.14–0.41 dB more than its registered-label twin, in 27 of 27 cells over three seeds. The input position matters as well (App. K.5): against R1_official the official NAFNet leads BM3D by +0.42 to +0.50 dB (T1w) at every position, but against the published label by +0.94 to +0.97 dB at the two middle repetitions and by +0.48 to +0.51 dB at the first and last.

Q1. A problem of M4Raw only, or of low-field MRI denoising in general? The mechanism applies where four conditions meet: a permanent magnet, no frequency tracking between repetitions, a low readout bandwidth, and repetitions averaged as acquired. A drift of r ppm per minute displaces a repetition acquired T minutes later by d = r · T · f0 / BW pixels along the readout axis (App. K.9, Table 16). On M4Raw the measured 0.24 px per repetition corresponds to between 1 and 2 ppm per minute. A 50 mT Halbach scanner drifting by about 6 ppm/min (O'Reilly et al., 2021) gives 0.16 px per repetition in an M4Raw-like protocol, unless the frequency is tracked, as in that study. Superconducting 3 T magnets (Hui et al., 2021) give less than 0.01 px. As a negative control we audited the fastMRI prostate DWI averages (3 T, 269 patients; App. K.11): the shifts differ from slice to slice and have no dominant axis, the systematic component is 14 to 34 times smaller per acquisition than on M4Raw, and the audit finds no drift signature. We did not re-score methods there. To our knowledge M4Raw is the only public low-field benchmark with raw repeated acquisitions, so the consequence for rankings can be measured only there.

Q2. Will the ranking change under perceptual metrics such as DISTS and LPIPS? We computed LPIPS (AlexNet, VGG) and DISTS against 7 reference constructions (App. K.6, Table 14). The nine label-trained networks and ImT-MRD are far closer to the reference than BM3D under every construction: the LPIPS-AlexNet distance to the published label is 0.022 / 0.024 / 0.020 for the official NAFNet and 0.078 / 0.116 / 0.128 for BM3D. Their order against BM3D never changes, and registration alone takes at most 14 % of the LPIPS lead of the label-trained networks. C2S is the exception: it is behind BM3D in T1w and behind or level in FLAIR. Between twins the order follows the training label: against R1_official the registered-label twin is the closer one in 81 of 81 cells, and against the published label the published-label twin in 74 of 81. Among the label-trained networks in general the order is not stable: under LPIPS-AlexNet the two official networks swap in T1w between these two references. So the fragile ordering against BM3D is the PSNR one. These metrics are reference-based too, so part of the networks' lead may be agreement with the label.

Thanks for the detailed response and additional evaluation. I will maintain my positive assessment and keep my rating at Weak acceptance.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: two effects separated, conclusion moderated, new analyses

We thank the reviewers for careful reports. Three requests are common: to separate the effect of misalignment from that of the normalisation order (Hbxb-W3, kPC4-W2), to moderate "the published reference cannot order these methods" (Hbxb-W5), and to broaden the evidence and the lesson. We accept all three. Changes are blue in the revised PDF and listed at the start of App. K.

Terms. R0_official is the published label: each repetition is normalised, then the repetitions are averaged. R1_official is the same with the repetitions registered before they are averaged, so registration is the only change ("registration alone", under the published normalisation). R1 is registered, averaged, then normalisedl. It is the reference of the pre-registered rule. D is the PSNR gap of a method to BM3D, and E is the change of D between two references. Six published-label networks were trained on the published labels (two official checkpoints, four trained by us). Three registered-label networks are three of these architectures, trained by us on registered labels. These nine are the label-trained networks, and twins share architecture and seed. Values are test-split means over 25 subjects for T1w / T2w / FLAIR, in dB.

1. Two components, kept apart. The pre-registered comparison changed registration and the normalisation order together. Sec. 5 now opens with a notation table (Table 1) and this decomposition (Table 2, 11 networks; every E is Holm-significant unless marked n.s.).

against BM3Dpublished gap Dregistration alonenormalisation order aloneboth
official NAFNet+0.94 / +0.87 / +0.37−0.44 / −0.39 / −0.17−0.91 / −0.71 / −0.50−1.34 / −1.10 / −0.67
official U-Net−0.18 / +0.05 / −0.38−0.53 / −0.34 / −0.32−0.13 / +0.14 / −0.09 (n.s.)−0.66 / −0.20 / −0.41
noisy input−2.63 / −3.11 / −2.38−0.55 / −0.19 / −0.28+0.96 / +1.10 / +0.63+0.40 / +0.92 / +0.35

Registration (the misalignment). Under registration alone every method moves against BM3D, the noisy input included, so the sign alone is no evidence that a network learned anything. The learned part is isolated by twins over three seeds. A published-label network loses 0.14–0.41 dB more than its registered-label twin, in 27 of 27 cells (3 architectures, 3 seeds, 3 contrasts), a median of 74 % of its loss. Equivalently, the registered-label twin scores 0.10–0.36 dB lower against the published label, while the twins agree to ±0.09 dB against R1_official. Repairing the labels costs nothing against the registered reference, yet the published leaderboard ranks the repaired model lower.

Normalisation order (a convention). Eight of the nine label-trained networks lose 0.13–0.91 dB, and methods not trained on such labels gain 0.3–2.4 dB. This is consistent with an intensity scale taken from the label, but we have no retraining control for it, and part of it is arithmetic: a common change of reference moves a method with a larger error less. The revision reports it as a dependence on a convention of the protocol, not as a learned effect and not as misalignment.

2. Conclusion moderated. "The published reference cannot order these methods" is removed. The abstract now states: "The gaps between these methods, and under some constructions their order, therefore depend on how the reference is built. We do not claim that any alternative ordering is the clinically correct one."

3. New analyses (App. K). All are descriptive. The pre-registered rule and its table are unchanged, and the test inference of each new checkpoint was run once. Three seeds for the twins (K.1). Leave-input-out references, under which the registration effect remains (K.3). A rotation-inclusive reference: E changes by at most 0.08 dB for the label-trained networks (K.4). Every repetition as the input (K.5). LPIPS and DISTS (K.6). SimpleITK reproduces our shifts to a median of 0.03 px per slice (K.7). The PSNR cost of the readout attenuation, 0.01–0.05 dB (K.8). A drift formula with a table for other scanners (K.9), the audit as a four-step protocol (K.10), and an audit of the fastMRI prostate DWI averages at 3 T that finds no drift signature (K.11).

4. What we expect to carry over (shown on one benchmark). A systematic inconsistency between averaged repetitions becomes a learned property of supervised models. A benchmark whose labels and yardstick share a construction rewards that property. Gaps on such a leaderboard should be reported with a sensitivity analysis over the reference construction. Comparisons across training paradigms are the most sensitive ones.

Where to find the changes. Sec. 5 ("Two components", "The learned part: twins", "What the results do and do not show") and Sec. 6. The pre-registered outcome is now Table 3. Tables 1 to 8 of the submission are Tables 3, 6, 4, 5, 7, 8, 9 and 10 of the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.