acceptodds
Withdrawn from ICLR 2027

EXEC-MRI: Gradient Execution Deviations Reverse MRI Reconstruction Rankings

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

MRI reconstruction relies on knowing where signals were sampled in Fourier space. Hardware response can shift these locations, making the actual acquisition differ from the reconstruction model. We ask whether method rankings remain reliable under this mismatch. EXEC-MRI provides a controlled evaluation protocol for MRI with non-Cartesian sampling paths, such as spirals. It pairs simulated gradient execution errors with tests of trajectory knowledge, numerical settings, and training. On fastMRI Brain, 22/28 method pairs change mean-score order from matched acquisition to moderate mismatch; reversals also occur on two external datasets. Numerical choices and model adaptation can themselves change the ordering, showing that robustness comparisons depend on the complete reconstruction pipeline rather than architecture alone. These retrospective simulations show why nominal performance alone does not establish relative performance under acquisition error. EXEC-MRI makes the conditions of such comparisons explicit and reproducible.

Primary area:
datasets and benchmarks
Keywords:
MRI reconstruction, robustness evaluation, benchmark protocol, non-Cartesian sampling, gradient impulse response function, acquisition-model mismatch, ranking stability, inverse problems

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VrSaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents EXEC-MRI, a protocol for testing whether rankings of MRI reconstruction methods stay stable when kspace trajectory deviates from design. The deviation (GED) is simualted and eight methods are compard.

Critical strengths
  1. Systematic and thorough experiments. The study covers eight methods, six severity levels, oracle conditions, two external datasets, five image metrics, a radial trajectory, a nonlinear saturation model, a larger training configuration, and density-weight controls. That is a comprehensive picture of how reconstruction pipelines behave under trajectory mismatch.
  2. Sound statistics and reproducibility. Bootstrap intervals are paired at the volume level, and the paper separates descriptive rank changes from significance claims. The release includes hashed manifests, frozen trajectory and density-weight arrays, and a regeneration script. This is a higher standard than most reconstruction benchmarks meet.
Critical weaknesses
  1. The core phenomenon is largely expected. The reversals happen where every method has already failed. The reported rank changes therefore compare reconstructions that would all be unusable in practice. This undercuts the paper’s central claim that nominal rankings are unreliable in a meaningful way. Forward-model mismatch is a known failure mode of learned reconstruction (Antun et al., 2020; Darestani et al., 2021). Gradient-response errors and their correction are well established (Vannesjo et al., 2013, 2016; Campbell-Washburn et al., 2016). Methods that rely on data consistency to different degrees will plausibly degrade at different rates and change rank. The paper’s value is in measuring this systematically rather than in revealing a new effect.
  2. Limited general insight for the ICLR audience. The topic itself is in scope. However, the paper does not draw a general lesson about learned inverse solvers under operator mismatch, such as which architectural or training choices predict robustness, or why fixed unrolled models fail to benefit from the correct operator. The oracle result in particular is reported but not explained. Without such insight, the contribution is mostly a set of conditional observations specific to this setup, and its relevance beyond MRI reconstruction is limited.
Questions for authors
  1. Do reversals occur at severities where the leading methods still produce clinically acceptable image quality?
  2. Which of the findings do you expect to be stable across simulators and density-compensation implementations?
Reviewer_aH7dOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes EXEC-MRI, an evaluation protocol for MRI that characterizes reconstruction methods' robustness to the acquisition model's coordinate mismatch. It simulates gradient execution errors and measures methods' performance. The authors investigate a set of methods, and reveal that the majority of method pairs' performances reverse under the induced coordinate mismatch. Aside from reconstruction methods, reversals depend on mismatch strength, trajectory knowledge, density compensation numerical weights, and training intervention.

Critical strengths

EXEC-MRI lets us control the mismatch severity and observe the methods’ mean order change. The paper also shows that standard density compensation weights remove the reversal at mild mismatch. This way, the authors show that an ordering determined for a specific architecture is incomplete until the full pipeline is fixed, which can help assess the methods' relative robustness more accurately.

Critical weaknesses
  1. The paper's main focus is the relative order of the methods' performances, but reversals can be caused by small differences in metrics, e.g. CG-SENSE and E2E-VarNet in Table 8, Moderate. Once these scores compress and drop toward a floor, a higher mean rank does not mean one method remained more robust in any practical sense. Under mild mismatch, the reversals are more pronounced, but this table's data was collected using "Historical" – numerically sensitive – density compensation weights. We see from section 4.4 that using standard density compensation on frozen checkpoints removes the reversal, and post-training them for 6 epochs restores the reversals but with reduced differences. This raises doubts about whether training with standard weights from scratch would produce reversals too.
  2. For reconstructions, the paper only uses hand-specified GIRFs with strengths from minimal to extreme. Appendix F.1 additionally passes the waveform through two measured GIRFs (3T London, 7T Zurich). Their spectral shapes differ from the presets, so a matched deviation in percent of k-max is a different distortion. Understanding whether reversals also occur under those measured GIRFs is central to the paper's general claim.
Questions for authors
  1. Will the more pronounced mild severity reversals from Table 8 still hold up under standard density training from scratch?
  2. Can you provide the nominal and mismatch reconstructions with the 2 measured GIRFs you plot on Figure 10 and report whether the reversals still take place?
Reviewer_atagOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focused on evaluating and comparing the MRI reconstruction methods under the trajectory error. It suggests to compare different MRI reconstruction methods under different trajectory conditions like nominal and mismatch. The experiment on MRI shows that most method pairs change order under trajectory mismatch.

Critical strengths

S1 The studied problem is important, as reliable evaluation itself becomes a new and important problem. And the paper shows the interesting model rank inconsistency.

S2 The results are clearly presented, and we can easily understand the rank inconsistency of different methods under different GED severity.

S3 Limitations are clearly discussed, which make the paper's contribution more clear.

Critical weaknesses

W1 A main concern of this paper is that all experiments are simulated, and the data will giveup the phase information, which make the data format different from the real-world data. Therefore, its still unclear whether the discussed situation also holds in the real-world data.

W2 In figure 2, the main relative performance inconsistency happens are for the MoDL-Aug, other methods show a clear rank for nominal case, and equally bad for the GED case. So its still unclear whether the rank inconsistency is obvious or not.

Questions for authors

Q1 In table 5, the relative reconstruction performance are consistent. If the all methods are tained on the data with GED, will the rank inconsistency still hold?

Q2 Could authors explain the reason why Figure 2 and Table 2, all deep learning methods show worse performance than the classical methods under norminal case, could this be the models are undertraining?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.