- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates whether generative super-resolution models can genuinely recover subject-specific anatomy from ultra-low-field (64 mT) MRI, or if fine details are hallucinated. Using real paired scans from public datasets, the authors measure an empirical information ceiling via cross-modal Fourier shell correlation, reporting that real 64 mT scans retain reliable subject-specific structure only down to approximately 3–4 mm half-pitch in-plane. They further demonstrate that standard synthetic degradations preserve finer information than real acquisitions. By auditing trained models with a stranger-controlled test, they find that fine output details are no more correlated with the subject’s own 3T scan than with a stranger’s, indicating fabrication at fine scales. Since PSNR and SSIM score resemblance to a reference rather than subject-specific fidelity, benchmarks relying on them cannot distinguish recovery from hallucination. The validation employs known-answer tests, registration controls, synthetic and oracle baselines across multiple architectures, including GANs and diffusion models.
Critical strengths
-
I find the central question of this paper highly important: are the fine details in model outputs genuinely recovered from the input, or are they generated by learned priors, potentially amounting to hallucinations? This question is more clinically meaningful than simply pursuing higher PSNR/SSIM scores. Indeed, it extends beyond ultra-low-field MRI super-resolution to medical image super-resolution more broadly: can information that has been lost or smoothed away during acquisition truly be created from nothing? Rather than merely criticizing existing models, the authors attempt to measure how much subject-specific structural information remains in real 64 mT scans. This shifts the discussion from subjective visual assessment to a quantifiable measurement. Framing the problem in terms of an “information ceiling of recoverability” is both direct and insightful.
-
The experiments are carefully designed. The paper is grounded in real paired 64 mT/3T scans rather than relying solely on synthetic degradation, which makes its conclusions more convincing than those based purely on synthetic experiments. In addition, the authors employ known-answer tests, registration-error controls, intensity-normalization checks, stranger controls, synthetic-degradation controls, raw-input baselines, and oracle controls, demonstrating an awareness that frequency-domain measurements are not inherently reliable. Both the main text and the appendix clearly discuss the boundaries of the proposed methodology and describe failure cases, which is commendably transparent.
Critical weaknesses
-
The paper’s central model-level claim relies primarily on direct audits of only four held-out subjects, of which only two are fully certified. This evidence base is too small to support broad conclusions across models and datasets, and the results may be sensitive to individual registration quality, contrast differences, or data anomalies. Because faithfulness is not computable for most of the 24 audited runs, these four cases carry nearly all of the direct evidence. If any one of them contains an undetected bias, the main conclusion would be substantially weakened. Furthermore, the ceiling estimates themselves are computed on small cohorts (n = 8, 10, and 11 across the three datasets), the standard error of the ceiling estimates is substantial, revealing that the result is highly sensitive to the specific subjects included in the sample.The authors’ disclosure of this limitation is appreciated, but it does not mitigate the serious impact this small evidence base has on the paper’s central support.
-
The paper’s significance lies mainly in raising an important evaluation question rather than in methodological novelty. It shows that low-field MRI super-resolution may fail to distinguish genuine recovery from plausible hallucination, which is clinically relevant. However, for the ICLR main track, the technical novelty is limited: the paper does not propose a new model, learning algorithm, training objective, or broadly reusable benchmark/metric. It mainly applies existing frequency-domain measurements and control-based comparisons to a specific low-field MRI setting. Moreover, the limitations of PSNR and SSIM are widely recognized in the low-level vision community. Even without strong methodological novelty, the contribution could be much stronger if the scope were broader. Similar recoverability-versus-hallucination issues may exist in many other medical imaging tasks, such as low-dose CT to standard-dose CT, or non-contrast CT to contrast-enhanced CT. Demonstrating that the proposed audit generalizes across such settings would make the paper more compelling for a machine-learning venue. As it stands, the work is valuable but may be insufficient for a strong acceptance at the ICLR main track.
Questions for authors
-
The central model-level claim relies on only four held-out subjects, of which only two are fully certified. Could the authors expand the number of auditable cases? Many runs are not computable due to registration, anchor, frame-consistency, or data-quality issues. Could the authors provide a complete accounting of all attempted subjects/runs and failure reasons, and discuss whether the excluded cases could systematically affect the conclusion? This is my primary concern. Given the insightful perspective of this paper, I would not want to assign a low score for this reason alone. However, the current experimental evidence is simply too limited. If the authors can expand the auditable cases and strengthen the empirical support, I would be willing to raise my score.
-
The direct evidence mainly comes from single-contrast T1w diffusion models and a small number of subjects. To what extent do the authors believe the conclusion generalizes to other modalities, and clinical data containing lesions or abnormal anatomy? Similar hallucination issues may arise in other medical imaging tasks, such as low-dose CT to standard-dose CT, or non-contrast CT to contrast-enhanced CT. Have the authors considered extending the proposed audit to these settings? If not, what are the main limitations?
-
Since PSNR, SSIM, and sample-variance uncertainty do not distinguish genuine recovery from fabrication, what evaluation protocol would the authors consider that may be suitable for low-field MRI super-resolution work in the future? Is there a reusable audit procedure that worth to explore?
Response to Reviewer CScL
We thank Reviewer CScL for the constructive framing of what would raise confidence.
Primary concern, and Q1: expand the auditable cases and provide a complete accounting. This is what we did. We audited the external tool on a pre-registration frozen before scoring: 22 attempted subjects, which decompose as 19 auditable (16 fabrication, 3 coarse recovery; 17 new subjects plus the two Zenodo pilots sub-0048 and sub-0064), 1 gated out (sub-HYPE16, NC-ANCHOR), and 2 OpenNeuro pilot consistency reruns (sub-HYPE21, sub-HYPE22) that reproduce the submitted values and are not counted again; 0 exceedances. The four pilot values reproduce exactly. A complete accounting of every attempted run-subject combination, with gate outcomes and reason codes, is posted as a separate comment in this thread. The recalibrated ledger is stated plainly: Tier 1 rests on one fully certified trained-model subject and six certified external-tool subjects (sub-0048 among them), and the frame-internal invariant holds on 19 of 19 external subjects and 8 of 8 final-epoch runs. We also tested the reviewer's point that the ceilings may be sensitive to which subjects the cohorts contain. A leave-one-out jackknife over the three ceiling cohorts shows they are not: every leave-one-out median stays inside the published confidence interval, and removing any single subject shifts a cohort median by at most 0.10 mm half-pitch. The reviewer wrote that expanded auditable cases would be grounds to reconsider; that expansion is above.
Q2. Does the protocol generalize to other modalities, for example CT or lesion cohorts? The protocol requires paired acquisitions of the same subject and nothing modality-specific: gates with known answers, a per-subject anchor, a raw-input benchmark, and the stranger control. It transfers wherever such pairs exist. Lesion cohorts and CT are future work that the released protocol enables, and we do not claim them here.
Q3. Is the evaluation protocol reusable? Yes. We release the measurement protocol as a reusable audit comprising the gate battery with known answers, per-subject anchors, the raw-input benchmark, the stranger control, the reason-code taxonomy, and the frozen pre-registration manifests used in this work. It is designed to run on any paired-acquisition super-resolution system.