acceptodds
Under review as a conference paper at ICLR 2027

SIREM: Speech-Informed MRI Reconstruction with Learned Sampling

Abstract

Reconstructing speech rtMRI from highly undersampled acquisitions remains challenging because the rapid articulatory dynamics of speech demand both high spatial and temporal resolution. Yet speech rtMRI is acquired together with an information source that conventional reconstruction methods largely ignore: the synchronized acoustic signal. We introduce SIREM, a speech-informed MRI reconstruction framework that exploits this cross-modal relationship as an auxiliary prior. Because vocal-tract configurations are closely coupled to the acoustics they produce, part of the image content can be inferred from speech while the acquired k-space data provide complementary anatomical information. SIREM therefore decomposes each reconstructed frame into audio-driven and MRI-driven components that are combined through an anatomically derived spatial weighting map. The audio branch predicts articulator-related structure from synchronized speech, whereas the MRI branch recovers complementary content from the measured k-space samples. We further introduce a learnable soft weighting profile over spiral arms, allowing the contribution of individual k-space arms to be optimized jointly with the multimodal reconstruction. This provides a unified formulation for audio-guided prediction, MRI reconstruction, and k-space arm reweighting. We evaluate SIREM on the USC 75-Speaker speech rtMRI benchmark against conventional reconstruction approaches, including wavelet-based compressed sensing, total variation, and a deep learning method. SIREM achieves 44–53 higher reconstruction throughput than the iterative methods while preserving anatomically plausible vocal-tract structure, and achieves real-time computational throughput, with per-frame compute within a 30-fps budget. Our results provide an initial benchmark for speech-informed rtMRI reconstruction and demonstrate the potential of synchronized acoustics as an auxiliary prior for accelerating dynamic speech MRI.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Real-time MRI reconstruction, multimodal learning, compressed sensing, inverse problems, medical imaging

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_VD26OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes SIREM, rtMRI reconstruction methods that combine image prediction branch as prior with rt speech MRI reconstruction. Overall, it seems to be premature in its current form.

Critical strengths
  1. The idea that incorporate speech acoustics as prior is interesting and promising direction to rt MRI reconstruction.

  2. The experiment results improves over the conventional compressed sensing methods

Critical weaknesses
  1. It seems that the proposed method does not utilize temporal correlation between MRI frames, which is common and essential in existing rtMRI reconstruction frameworks (although audio encoder may implicitly employ temporal information, the explicit correlation between MRI frames are generally exploited and much more powerful). There should be an explanation to such design choices. Also, the proposed method is trained and reconstructs in lower temporal resolution (78ms/frame) compared to the original resolution the dataset provides (12ms/frame) to exploit full 13 arm rotations. It is necessary to describe that higher resolution reconstruction would be available with the proposed methods.

  2. The comparison is only with the conventional compressed sensing method and E2E-VN (2020). SIREM only show the processing time advantages over the compressed sensing methods, not over E2E-VN (16.68 vs. 15.65ms). It is necessary to compare with recent dynamic MRI methods (E2E-VN is a general MRI reconstruction method not specialized for dynamic or rTMRI)

  3. The weighting map (W_EbA) is constructed only from anatomical segmentation. Also, it is generated only from the initial frame and then fixed over the entire frames, which does not consider the changes between frames (e.g. object movement) It is also unclear that w_EbA is binary or continuous values. There is no justification of such design choices, and they do not seem optimal. For example, certain articular regions are more informative.

  4. “Learnable sampling” seems to be misleading. It seems to apply continuous weight to 13 acquired spiral arms, which means it cannot actually reduce the number of acquisition (i.e. there is not acceleration in data acquisition time). Also, the two regularization terms operates with the opposite goals, the PSF loss term induces equal weights, L_budget applies differentiated weights.

Questions for authors
  1. How does SIREM compare with the existing SOTA dynamic MRI methods? (reconstruction quality, computation time, etc)?

  2. Can SIREM achieve higher temporal resolution that the dataset provides?

  3. Please provide the justification to the choise of the weighting map (W_EbA)? (the choice of anatomical segmentation, choice of initial frame, etc)

  4. Please provide further justification and advantages of the proposed “Learnable weights” and the two regularization terms. Also, what is the relationship with the PSF loss and actual point-spread function?

  5. What is the benefit of constraining arm weights to K=2, when all the 13 arms are actually necessary? (no acceleration available)

2 comments

We thank the reviewer for the detailed assessment and for highlighting the concerns.

Comparison with recent dynamic MRI methods: Our experiments include E2E-VN as a learned reconstruction baseline. SIREM achieves competitive reconstruction results, is 7.3% slower than E2E-VN, and is substantially faster than the two iterative baselines. We are also evaluating an additional learned reconstruction method under the same experimental setting and will report the results once the experiment is complete.

Frame rate and temporal modeling: The current experiments use 13-arm inputs at 12.81 fps, and the MRI branch processes each frame independently. Therefore, the present results demonstrate speech-informed reconstruction at this acquisition rate rather than increased acquisition temporal resolution. We will distinguish inference speed from acquisition frame rate more clearly and discuss explicit MRI temporal modeling as future work.

Why anatomical masks: They provide a simple and interpretable restriction on where audio may contribute. The fixed map is not a reliability estimate and does not track motion; this is a stated limitation, not a hidden assumption. We will state this limitation more clearly and discuss learned or temporally adaptive weighting as future work.

Learned spiral-arm weights: We agree that “learned spiral-arm reweighting” is more accurate than “learned sampling”. All 13 spiral arms are acquired and retained and the learned weights are applied retrospectively. In the revised manuscript (on preparation), we will use the more precise terminology and remove any implication that the current method learns a prospective sampling trajectory or reduces acquisition time

Sampling-budget interpretation: The K=2 constraint acts as a soft regularizer on the arm weights; it does not impose a physical acquisition budget or reduce acquisition time because all 13 arms are retained. We will revise the terminology and discussion to make this distinction explicit.

I thank the authors for the responses and clarifications. However, the most crucial concerns still remain.

  1. Current response does not include new experimental evidence for insufficient experiment results and ablation studies. Particularly, E2E-VN is the only deep learning based baseline which is presented in 2020 and targeted for static MRI (not specialized for dynamic MRI). As the proposed method addresses speech rtMRI, it is necessary to include comparisons with more recent SOTA methods, especially with dynamic MRI methods incorporating temporal correlation.

  2. The justification regarding spatial fusion still remain. Particularly, the relative weighting between audio-based branch and MRI reconstruction branch is determined by a fixed anatomical mask through the segmentation map, which requires further justification. Also, final result may not guarantee the data consistency, which is not yet discussed in the manuscript.

  3. The advantage of “learned sampling” is still unclear, as it does not provide practical scan time reduction. Also, the theoretical justification of PSF regularization term and L_budget is not clear yet, and related ablation study is not provided as well (e.g. proposed vs. equal weighting)

  4. Authors acknowledged that the method does not improve acquisition temporal resolution, nor incorporate temporal correlation explicitly. This remains my overall concerns regarding the practical advantages of the proposed methods.

Reviewer_sLiaOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The work is about rtMRI reconstruction. The authors introduced a speech-informed MRI reconstruction framework called SIREM.

Critical strengths
  • A new strategy of doing rtMRI reconstruction by combining synchronized speech with undersampled multi-coil spiral k-space.
  • The paper is easy to read and many experimental results are conducted.
Critical weaknesses
  • Using speech info to facilitate the rtMRI reconstruction sounds reasonable; however, no strong evidence and results are obtained in the manuscript to support this idea well.
  • It’s worth mentioning that the speech info has been used in rtMRI tasks like synthesis and generation.
  • It’s not clear how to define a_t, the synchronized speech segment. If a slightly different synchronized speech segment is obtained/used, whether it will affect the reconstruction results significantly or not is not clear.
  • In Eqn (3), it’s not clear how to define w_EbA, and if the results are robust to w_EbA.
  • The data used in experimental results is small.
  • The quantitative results in e.g. Table 1 lacks standard deviation.
  • It’s not clear if the introduced method is robust and general or not. It’s faster than traditional methods, but traditional methods are robust and general (do not depend on training data).
  • Some areas of the manuscript are repetitive, e.g. Eqn (3) and (15) are the same.
Questions for authors

See above the weaknesses.

3 comments

We thank the reviewer for the detailed comments. We would first like to clarify an important distinction between speech-to-image generation and MRI reconstruction. SIREM reconstructs images from measured raw multi-coil k-space and uses synchronized speech only as an auxiliary prior, thus from what it was sequence from the MRI directly. It does not generate MRI images from audio alone.

Evidence for using speech: The paper provides a direct controlled ablation between SIREM and the same reconstruction model without audio. Speech improves all seven metrics under both evaluation targets. Against the CG-SENSE targets, it improves SSIM by 0.036 and PSNR by 1.01 dB, while reducing NRMSE by 0.029, HFEN by 0.071, LPIPS by 0.017, and FID by 6.32. Appendix K.2 further shows improvements within the articulator regions. Therefore, the contribution of speech is supported by quantitative evidence, not only qualitative examples.

Previous speech-based rtMRI work: These studies are cited and discussed in Sec. 2. They address speech-to-rtMRI synthesis, generation, or inversion in the image or video domain. SIREM addresses a different task: reconstruction from acquired non-Cartesian multi-coil k-space, with speech used as auxiliary information. We will make this distinction more prominent in the revised manuscript.

Definition of ata_t: This is explicitly defined in Sec. 3.2, Eq. (5). For the tt-th 13-arm window, ata_t is the 16-kHz waveform centered at the acquisition time ctc_t with the context window. It contains the current and preceding audio samples only, with no future context. We will repeat this definition near the audio-branch description for better visibility. We will also state sensitivity to audio–MRI misalignment as a limitation of the present evaluation.

Dataset size: We would like to clarify that although the number of subjects is relatively limited, reconstruction is performed at the frame level, resulting in a substantially larger number of reconstruction samples. We use the full USC 75-Speaker corpus, comprising 72 usable subjects and approximately 2,200 utterances, with a subject-disjoint 50/7/15 split. The test set alone contains 463 utterances and 5,556 frames from 15 unseen speakers. To our knowledge, this is one of the largest publicly available speech rtMRI datasets providing raw multi-coil k-space. We nevertheless acknowledge that the data originate from a single site and scanner, and we state this limitation in the paper.

Standard deviations: We now report mean ± standard deviation for SSIM, PSNR, NRMSE, HFEN, LPIPS, and VIF, and FID below. We will include the updated table in the revised manuscript.

MethodSSIM ↑PSNR ↑NRMSE ↓HFEN ↓LPIPS ↓VIF ↑FID ↓
Target: CG-SENSE reference
Zero-filled0.223 ± 0.03713.349 ± 1.110.787 ± 0.0100.936 ± 0.0170.444 ± 0.0370.043 ± 0.006310.963
Wavelet0.612 ± 0.03123.050 ± 1.520.258 ± 0.0180.590 ± 0.0640.277 ± 0.0200.234 ± 0.039172.992
TV0.635 ± 0.03023.032 ± 1.560.258 ± 0.0190.574 ± 0.0400.287 ± 0.0170.079 ± 0.01198.002
E2E-VN0.599 ± 0.04723.752 ± 1.450.239 ± 0.0260.620 ± 0.0590.173 ± 0.0260.223 ± 0.02783.148
SIREM w/o audio0.570 ± 0.05322.899 ± 1.480.264 ± 0.0300.683 ± 0.0680.180 ± 0.0230.197 ± 0.02980.476
SIREM0.606 ± 0.04923.914 ± 1.560.235 ± 0.0310.612 ± 0.0620.163 ± 0.0220.222 ± 0.02974.156
Target: Total Variation
Zero-filled0.254 ± 0.04614.326 ± 1.120.788 ± 0.0090.944 ± 0.0150.455 ± 0.0360.038 ± 0.005297.2
Wavelet0.659 ± 0.04324.488 ± 1.220.245 ± 0.0140.601 ± 0.0300.199 ± 0.0360.226 ± 0.018160.1
E2E-VN0.573 ± 0.06222.274 ± 1.520.317 ± 0.0250.732 ± 0.0710.151 ± 0.0240.197 ± 0.024169.1
SIREM w/o audio0.649 ± 0.03323.821 ± 1.610.269 ± 0.0220.536 ± 0.0420.077 ± 0.0120.293 ± 0.01791.6
SIREM0.677 ± 0.03123.824 ± 1.560.264 ± 0.0210.535 ± 0.0420.075 ± 0.0110.297 ± 0.01991.3

Repeated equations: We apologize for our copy-and-paste error where Eqs. (3) and (15) are identical. We will remove the repeated equation in the revised manuscript and replace it with the appropriate cross-reference.

Missing response about generalization: Our evaluation uses a subject-disjoint split, and the test set contains 15 speakers who were not seen during training. This evaluates generalization across speakers. However, the data come from a single site and scanner, so cross-site and cross-scanner generalization remains to be evaluated. We will state this limitation more clearly in the revised manuscript.

Thanks for the response. The main concerns like "... no strong evidence and results are obtained in the manuscript to support this idea well." are still there. For example, for the latest results with the standard deviation, the proposed method doesn't produce the best results in terms of many metrics than other existing strategies.

Reviewer_oCTYOpenReview ↗

Keep my score

Thank you for the clarifications. The response is textual only, and I still have some concerns: (1) It is still unclear where the causal audio window is specified, (2) the gains over E2E-VN are small. Perhaps, a significance test, and comparison with at least one further unrolled method. (3) As authors acknowledged that fusion rule is not optimal, a simple alternative fusion baseline would help show that the speech prior, rather than the specific design. Due to these remainng concerns, I would like to keep my rating.

Reviewer_oCTYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a real time MRI reconstruction framework with an auxiliary prior of speech. Specifically, a pre-trained HuBERT audio model is introduced to help the reconstruction of adjoint NUFFT+SENSE. Besides, a learnable spiral-arm weighting profile is adopted to perform a soft weighting on the sampled k-space measurement. Experiments demonstrate comparable performance compared to the existing methods with faster computational time.

Critical strengths
  1. The motivation to incoporate the speech information to help MRI reconstruction is reasonable and may have practical application.
  2. The two branch design is simple but clear. The fusion strategy make it easy to identify what part of the speech information contribute to.
Critical weaknesses
  1. The authors claim the proposed method as "learned sampling". However, it only learns a soft weighting of known fixed sampling patterns, instead of learning what the pattern should be.
  2. The authors also claim that their method is real-time. But during the reconstruction, the audio branch use the information from future. The authors should clarify on this part and test the performance of using a causal window of audio information.
  3. The improvement on performance compared to the existing method are quite limited. More importantly, learning-based iterative reconstruction methods are not included for comparison in the experiments.
Questions for authors

Why the design of using speech information through a spatial reweighting is the optimal?

1 comment

We thank the reviewer for the careful assessment and for highlighting the concerns regarding sampling terminology, real-time inference and experimental evaluation.

Learned sampling: We agree that “learned spiral-arm reweighting” is more accurate than “learned sampling”. All 13 spiral arms are acquired and retained and the learned weights are applied retrospectively. In the revised manuscript (on preparation), we will use the more precise terminology and remove any implication that the current method learns a prospective sampling trajectory or reduces acquisition time

Real-time reconstruction: By real-time, we refer to inference throughput. For reconstruction at time t, the audio input contains the current audio segment together with a limited amount of preceding context; no future audio samples are used. We will make this causal window definition explicit in the revised manuscript to avoid confusion.

Comparison with learned reconstruction methods: Our results include comparisons with E2E-VN, an unrolled learned MRI reconstruction method, as well as a controlled ablation comparing SIREM with the identical architecture without audio. The ablation shows that incorporating speech improves all seven metrics under both evaluation targets. While we do not claim uniform superiority over all reconstruction approaches, our results demonstrate that synchronized speech provides complementary information while maintaining real-time computational throughput.

Fusion strategy: We do not claim that the proposed fusion rule is optimal. It is a simple and interpretable inductive bias that restricts the contribution of audio to articulator regions while preserving MRI-driven reconstruction elsewhere. This design directly tests the paper’s central hypothesis without allowing speech to replace the measured MRI information. Learned, confidence-based, and temporally adaptive fusion strategies are promising directions that we discuss as future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.