- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents EXEC-MRI, a protocol for testing whether rankings of MRI reconstruction methods stay stable when kspace trajectory deviates from design. The deviation (GED) is simualted and eight methods are compard.
Critical strengths
- Systematic and thorough experiments. The study covers eight methods, six severity levels, oracle conditions, two external datasets, five image metrics, a radial trajectory, a nonlinear saturation model, a larger training configuration, and density-weight controls. That is a comprehensive picture of how reconstruction pipelines behave under trajectory mismatch.
- Sound statistics and reproducibility. Bootstrap intervals are paired at the volume level, and the paper separates descriptive rank changes from significance claims. The release includes hashed manifests, frozen trajectory and density-weight arrays, and a regeneration script. This is a higher standard than most reconstruction benchmarks meet.
Critical weaknesses
- The core phenomenon is largely expected. The reversals happen where every method has already failed. The reported rank changes therefore compare reconstructions that would all be unusable in practice. This undercuts the paper’s central claim that nominal rankings are unreliable in a meaningful way. Forward-model mismatch is a known failure mode of learned reconstruction (Antun et al., 2020; Darestani et al., 2021). Gradient-response errors and their correction are well established (Vannesjo et al., 2013, 2016; Campbell-Washburn et al., 2016). Methods that rely on data consistency to different degrees will plausibly degrade at different rates and change rank. The paper’s value is in measuring this systematically rather than in revealing a new effect.
- Limited general insight for the ICLR audience. The topic itself is in scope. However, the paper does not draw a general lesson about learned inverse solvers under operator mismatch, such as which architectural or training choices predict robustness, or why fixed unrolled models fail to benefit from the correct operator. The oracle result in particular is reported but not explained. Without such insight, the contribution is mostly a set of conditional observations specific to this setup, and its relevance beyond MRI reconstruction is limited.
Questions for authors
- Do reversals occur at severities where the leading methods still produce clinically acceptable image quality?
- Which of the findings do you expect to be stable across simulators and density-compensation implementations?