- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies how the construction of reference impacts performance when a restoration method's output is scored against pseudo-labels generated by multi-repetition averaging, particularly in MRI datasets. When the noisy observation used as the restoration input also contributes to the target being scored, the score rewards carrying that noise over and inflates methods' apparent performance. The analysis identifies this effect in both classical and deep learning-based approaches under NRMSE, and recommends scoring against repetition-averaged references that exclude the input repetition. A selection margin is used to analyze when reference overlap changes the selected method, and experiments on a real-world MRI dataset show discrepancies in method selection depending on whether the reference includes the input repetition.
Critical strengths
- The authors have identified an important issue in repetition-average-based benchmarks that reward noise carrying, and the analysis, formalized using selection loss , shows a positive trend across reported groups of methods, justifying the practice of reporting this metric along with the traditional ones for robustness.
Critical weaknesses
-
The experiment setup and interpretation are confounded. In an RSS magnitude averaging operation, a positive noise floor is left on the images; thus, the reference and validators could have been rewarding methods that reproduce this bias. Further, D(V) is defined over a hand-crafted pool of methods, and its value depends strongly on the pool's composition. Moreover, the reported selection losses are very small; along with the limited method pool, it is unclear whether reference overlap would change practical assessments when stronger baselines are included.
-
The scope and clarity of the paper need improvement. First, the authors' proposed selection loss was only tested and validated on one multi-repetition averaged MRI dataset, whereas another experiment using a noiseless dataset with synthetic injected noise could separate the contribution much more cleanly due to known noise distributions as well as no motion/magnetic field drifts. Second, several notations are used interchangeably and hinder the readability of the paper. For example, reference shift seems to be the same quantity as .
Questions for authors
-
Since references and validators are built using RSS magnitude averaging, a positive noise floor is left on all references and validators. Could you use complex averaging and show whether the conclusion changes at all?
-
Could you include visualizations of the reconstructed images selected using shared versus disjoint references, along with error maps to demonstrate noise carry-through?
-
Would it be possible to test on data with known ground truth (i.e., simulated white Gaussian noise with independence across repetitions) so that the effect of selecting methods that are further from the true image can be demonstrated?
-
Has there been any attempt to model or address motion or magnetic field drift in the dataset since you used different repetitions for different purposes? Appendix 5 only provided a one-line explanation.
-
Could you clarify: Is equivalent to ? It would be helpful if all new quantities defined in the paper shared the same notation and naming.