acceptodds
Under review as a conference paper at ICLR 2027

Foundation Model Alignment Regularizes Transformer-Based Unrolled MRI Reconstruction

Abstract

Deep unrolled reconstruction networks alternate data consistency steps with learned refinement networks and are typically supervised only at the final reconstruction. More expressive refinement networks have produced limited improvements over the convolutional E2E-VarNet on fastMRI, partly, we hypothesize, because their training converges to suboptimal solutions. Across single-coil knee and multi-coil brain reconstruction, we find that the bottlenecks of unregularized transformer cascades collapse early to a small fraction of their available rank and do not recover, whereas the convolutional model partially recovers during training. We address this problem by aligning the refinement network's bottleneck of every cascade with dense representations extracted from the fully sampled ground truth by a frozen diffusion or self-supervised foundation model. A learned convolution in each cascade maps the bottleneck to the corresponding foundation model features. The foundation model and the alignment convolutions are removed after training, leaving the deployed reconstruction network and its inference cost unchanged. We evaluate this approach on single- and multi-coil datasets using multiple foundation models and convolutional and transformer-based reconstruction architectures. At longer training budgets, the aligned transformers surpass aligned E2E-VarNet, whereas their unregularized counterparts remain below unregularized E2E-VarNet. On knee data, the gains increase with acceleration and are largest for the transformer architectures. Alignment maintains high-rank bottlenecks across cascades and improves transformer training when unaided optimization stalls or deteriorates. These gains arise from foundation model alignment rather than increased rank alone: directly regularizing the bottleneck rank raises it beyond that of every aligned model without producing a lasting improvement in reconstruction. Where tested, the improvement over unregularized training is significant at in one-sided paired Wilcoxon signed-rank tests over evaluation slices, with Holm correction across the seven reported tests.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mri reconstruction, unrolled networks, foundation models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uPTyOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper hypothesizes that the saturating performance of unrolled networks on the fastMRI benchmark is partly due to training converging to suboptimal solutions. The authors propose aligning the bottleneck representations of each cascade with features from a frozen vision foundation model during training. The method improves reconstruction performance, particularly for transformer-based unrolled networks, without increasing inference costs.

Critical strengths
  • The proposed method improves the performance of transformer-based unrolled networks.
  • The approach is relatively simple.
  • The authors evaluate multiple foundation models and reconstruction architectures.
Critical weaknesses
  • E2E-VarNet already performs very well for 2D accelerated MRI reconstruction in the in-distribution setting. The gains from foundation model alignment are marginal for E2E-VarNet at 4× and 8× acceleration. Based on the reported numbers E2E-VarNet also performs better than HUMUS-Net. Hence, I see limited practical benefit.

  • Based on my experience with fastMRI and similar VarNet architectures and optimizer settings, I am concerned that the models are not trained to convergence. The fastMRI knee dataset contains approximately 35k training slices, so 300k iterations with batch size one correspond to fewer than 10 epochs. I would suggest training for at least 20–30 epochs and using a validation set for model selection.

  • The reported PSNR and SSIM values appear unusually low. For E2E-VarNet on multi-coil brain data at 8× acceleration, I would expect approximately 38 dB PSNR, consistent with results reported on the fastMRI leaderboard. This raises concerns about the training and evaluation setup.

  • The gap between HUMUS-Net and E2E-VarNet is unexpectedly large. On multi-coil brain data, the unregularized HUMUS-Net performs approximately 3.6 dB worse than E2E-VarNet, whereas results on the fastMRI benchmark suggest that the two architectures should perform similarly well. This makes it difficult to assess how much of the improvement comes from the proposed method rather than suboptimal baseline training.

  • The paper focuses a lot on the rank-collapse observation but also show that high rank does not provide better performance.

Questions for authors
  • How does the proposed method scale with dataset size? Does it consistently improve reconstruction performance across different data regimes, or is it primarily a regularization technique that benefits low-data settings?

  • What makes a vision foundation model a suitable teacher for improving MRI reconstruction accuracy? Which properties of its learned representations are expected to be beneficial?

  • The proposed method aligns each cascade's bottleneck with teacher representations extracted from the fully sampled reference. How does this compare with directly supervising each cascade output using the fully sampled image?

  • Dataset size and convergence: How many training slices are used in the experiments? Have the authors verified that the baseline models are sufficiently trained and that their performance has converged?

  • Can the authors review or clarify the evaluation setup, particularly for multi-coil brain reconstruction? The reported PSNR and SSIM values are lower than I would expect from the fastMRI benchmark.

  • In Table 1, HUMUS-Net-L experiences a drop from approximately 29.0 to 25.2 dB PSNR between 180k and 300k iterations. Can the authors explain this behavior? Is this instability common across different training runs?

  • Why does HUMUS-Net perform substantially worse than E2E-VarNet in the multi-coil brain experiments? Can the authors elaborate on this gap, given that fastMRI benchmark results suggest similar reconstruction performance?

  • What motivates reporting results at intermediate, potentially undertrained checkpoints? The computational costs of training these reconstruction models are not prohibitive, so I do not see a clear reason to emphasize intermediate iterations rather than converged performance.

  • Why do the authors consider the single-coil fastMRI knee setting? Since the fastMRI single-coil data are simulated and have limited practical relevance, I would consider focusing on multi-coil reconstruction a more meaningful setting for evaluating the proposed method.

  • The MRI inverse problem should be introduced more clearly. Equation (1) presents one possible variational formulation rather than the general MRI reconstruction problem. Moreover, effective rank should be defined when it is first introduced, rather than later in the experimental section.

3 comments

We thank the reviewer for their careful reading of the paper and for raising important questions.

Q1. Scaling with dataset size.

We have not performed a dataset size sweep and therefore cannot determine whether alignment provides larger relative gains in low-data regimes. We view the method as a representation regularizer rather than specifically a low-data technique. Evaluating whether this regularization becomes more beneficial as the training set decreases is an interesting direction, which we now leave for future work.

Q2. What makes a foundation model a suitable teacher?

Our initial framing did not clearly distinguish the roles of rank regularization, deep supervision, and teacher alignment. We now introduce these three regularization techniques to address the underutilized bottleneck in different ways.

  1. Rank regularization promotes diverse, uncorrelated feature directions, but higher rank does not ensure that these features contain useful spatial information, and its long training gains are inconsistent.
  2. Deep supervision provides additional ground truth supervision at every cascade output, but also does not yield consistent improvements.
  3. Foundation models instead provide a dense, informed, and spatially structured target directly at the bottleneck.

We hypothesize that this spatial structure is the key property that produces the sustained gains, while the increase in effective rank is a byproduct. The broadly similar improvements across four teachers with different architectures, objectives, and training domains further suggest that the benefit does not depend on a specific teacher. We have revised the paper to clarify this interpretation.

Q3. Supervising each cascade output with the fully sampled image.

We thank the reviewer for this suggestion and extend the deep supervision baseline for all architectures in the single-coil knee experiments. Specifically, we add

LDS=∑k=18wkℓk\mathcal{L}_{\mathrm{DS}} = \sum_{k=1}^{8} w_k \ell_k, with wk∝exp⁡(−τ(8−k))w_k \propto \exp(-\tau(8-k)),

where ℓk\ell_k is the SSIM loss between the kk-th cascade output and the fully sampled target. The weights are normalized to sum to one, and τ\tau increases from zero by 1/31/3 every 6000 iterations, gradually putting more weight on the later cascades.

Deep supervision does not consistently improve reconstruction across architectures and seeds. Moreover, the transformer bottlenecks remain low rank despite the additional supervision at every cascade output. Thus, supervising intermediate reconstructions does not reproduce the gains of foundation model alignment, suggesting that the benefit may depend on providing a dense, spatially structured feature target directly at the bottleneck.

Q4. Dataset size and convergence.

We use the whole fastMRI training split for both knee and brain (973 knee volumes and 4469 brain volumes, yielding 34742 knee slices and 70748 brain slices, respectively). We agree that our previously reported numbers were not converged. We extended the current setup to 1.02 million iterations and hope this addresses the concern.

Q5. Evaluation setup.

We thank the reviewer for identifying this discrepancy. The submitted version reported slice-wise metrics, whereas the fastMRI benchmark reports volume-wise metrics. We now use volume-wise evaluation in the main tables and move the slice-wise results to the appendix. The unaligned E2E-VarNet now achieves up to 36.64 dB PSNR.

The remaining difference from the approximately 38 dB leaderboard result may reflect differences in the experimental setup. We use randomized fastMRI undersampling masks, an 8-cascade E2E-VarNet, and only the official training split. In contrast, the referenced leaderboard model uses uniform undersampling, 12 cascades, and the combined training and validation data.

Q6. HUMUS-Net-L instability.

We now report two seeds in all three main tables. The revised results confirm that the stability of the HUMUS-Net family depends strongly on initialization.

We believe that the smaller batch size likely contributes further to the observed instability. We train with batch size 1, whereas the original HUMUS-Net recipe uses effective batch sizes of 8 and 4 for HUMUS-Net and HUMUS-Net-L, respectively. We therefore expect that using a larger batch would improve stability, but we could not perform a controlled batch size ablation in this revision and cannot establish it as the sole cause.

An additional contributing factor may be the different input fed to the original HUMUS-Net compared to our implementation. The original HUMUS-Net operates on center-cropped images and uses adjacent slice information, whereas we use the full field of view without adjacent slice information to ensure a fair comparison with E2E-VarNet.

Q7. HUMUS-Net performance gap.

The large 7.35 dB gap (volume-wise) was caused by an unstable HUMUS-Net run. We now report a second seed, for which the unregularized HUMUS-Net is approximately 0.71 dB below the unregularized E2E-VarNet. Thus, the previously reported gap is not representative of typical HUMUS-Net performance, but reflects its sensitivity to initialization under our training setup.

Table 1. Multi-coil brain at 8×8 \times, volume-wise PSNR (dB) after 1.02M iterations, 150 test volumes, two seeds per student (first / second).

StudentNo teacherSDDINOv3DiT
E2E-VarNet36.52 / 36.6436.78 / 36.7636.79 / 36.8436.79 / 36.89
HUMUS-Net29.17 / 35.9336.74 / 36.8637.18 / 37.1937.00 / 37.06

Q8. Intermediate checkpoints reporting.

We agree that the costs of training these architectures are smaller relative to other models, especially for E2E-VarNet. The reason we report intermediate checkpoints is to disentangle whether our alignment mechanism is purely speeding up convergence and/or whether it gives lasting gains.

Q9. Reporting single-coil knee

Our main motivation was to show whether this approach works in both the single-coil and multi-coil setting. We understand the reviewer's concerns with this dataset, however at this time we cannot provide multi-coil knee numbers.

Q10. Introducing the MRI problem and defining effective rank early.

Thanks for the suggestion. We fixed this in this draft.

Thank you for the additional experiments and clarifications. Overall, the response improves the submission, but does not sufficiently resolve my concerns about the baseline comparison and the practical and scientific significance of the proposed approach. I therefore maintain my score.

The practical advantage over a strong E2E-VarNet baseline also remains insufficiently compelling to me. In the revised brain results, the best aligned HUMUS-Net improves over the best unaligned E2E-VarNet by 0.55 dB, while alignment improves E2E-VarNet itself by 0.12–0.27 dB. However, the method introduces additional training machinery and tuning, while the strongest results also involve the larger HUMUS-Net backbone. Given the competitive performance of plain E2E-VarNet, the tradeoff between these improvements and the additional training cost and complexity needs a stronger justification.

The concern about the baseline comparison remains only partly resolved. The updated results demonstrate an improvement even for the better-performing HUMUS-Net seed. Nevertheless, the large variation between the two unregularized runs raises questions about the stability of the chosen training setup. The response identifies batch size and differences from the original training recipe as possible contributors, but does not test whether addressing these issues reduces the advantage of alignment. It therefore remains unclear how much improvement would persist under a more consistently stable training setup.

The contribution of the foundation-model teacher also remains insufficiently understood. The proposed importance of spatially structured teacher features remains a hypothesis. The dependence on training-set size also remains unexplored, limiting our understanding of when this regularization is most useful.

Reviewer_7394OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a transformer-based unrolled MRI reconstruction method. It aligns the bottleneck feature of the cascade with the frozen foundation model feature from fully sampled image. The main motivation arises from the observation of effective rank in the bottleneck features. The experiments with fastMRI knee/brain show that the proposed method improves the reconstruction results. Overall, the results seems promising, but the main motivation (rank collapse) requires further theoretical justification and the authors need to differentiate the proposed method with exisiting feature-level knowledge distillation methods.

Critical strengths
  1. A simple unrolled MRI reconstruction framework without increasing the inference time

  2. The experiment considers multiple student and teacher (diffusion, self-supervised) architectures.

  3. Authors included analysis and experiments on bottleneck effective rank and deep supervision baselines.

Critical weaknesses
  1. The main motivation of this study is the effective rank collapse in the bottleneck feature. However the relation between low effective rank and final reconstruction quality should be justified first. For example, there exist many counterexamples that low-rank representation is helpful for MRI reconstruction (as a form of Casorati or Hankel matrix regularization), where such low-rank constraint greatly help improve the final reconstruction quality, although they are not exactly equivalent form to bottle representation shown in this work. There are other works that maintain or improve the final task performance through low-rankness in feature layers (e.g. LoRA). Also, the own experiment by the authors suggests that effective rank is not the main contributor to the final reconstruction quality, where direct rank regularization could increase the effective rank but did not improve the reconstruction performance.

  2. It is unclear how the proposed method differentiate from the existing feature-level knowledge distillation. It is necessary to analyze and compare with existing feature distillation, perceptual feature supervision, or even simpler pretrained extractor.

  3. It seems some design choices need to be justified. The teacher representations are extracted from different layers depending on different foundation models (last 4 blocks of DINOv3, internal transformer block of DIT, decoder activation of stable diffusion). Student alignment is applied only to each refinement network. Such choices seem quite arbitrary without appropriate justifications. They require further justification with ablation studies.

  4. The proposed method aligns student’s bottle features with the same teacher representation. However, different levels of cascades may learn different level of representation, and applying the same targets to them can restrict the diversity and expressiveness in multiple level feature representation. This necessitates further justification and related ablations studies.

5.It is difficult to examine the statistical power because the limited number of experiments are conducted (only two seeds with the single coil experiments / one seed for multi-coil) Also, as the performance improvement seems not significant, it may be achieved from reducing the overfitting in the training. It would be helpful to compare with existing overfitting-prevention methods (additional regularization, data augmentation, pretraining, etc)

Minor points

  1. It can greatly help the readability to include the overview figure.

  2. The preprocessing & feature standardization procedures differ across teacher model, which requires further justification & analysis.

  3. Please provide information about the initialization of the student networks whether they were randomly initialized and employed pretrained weights.

  4. As both diffusion-based teacher and self-supervised teacher improve the performance, it would be useful to analyze which properties of teacher representation is essential to the successful alignment.

  5. Please provide the k-space undersampling patterns used in the experiements in the figure.

Questions for authors
  1. What justifies effective rank collapse is the main cause of the performance degradation, as the experiment with rank regularization did not improve the PSNR even after raising the effective rank of the feature?

  2. Is the proposed method more advantages over simpler feature-level knowledge distillation or perceptual feature supervision? The effect of using foundation models and feature-level regularization can be distinguished?

  3. How does the results changes when teacher feature layers or student alignment locations changes?

  4. Does the results maintain if the number of repeated seeds increases, given some baseline have large performance variation between seeds (e.g. HUMUS-Net-L) ?

3 comments

We thank the reviewer for the questions and suggestions.

Q1. Is effective rank collapse a cause of degraded reconstruction performance?

We agree that the original framing may have suggested a stronger causal claim than intended, and we have revised the paper accordingly. We do not claim that low effective rank causes poorer reconstruction or that higher rank necessarily improves it. Rather, the exceptionally low effective rank of the HUMUS-Net bottlenecks, together with the limited anatomical structure visible in their PCA visualizations, suggests that these representations are highly redundant and underutilized beyond what would be expected from the intrinsic spatial redundancy of medical images. The relationship with reconstruction performance remains correlational.

This observation is consistent with the insight of Dong et al. that global token mixing can promote representational redundancy. Although their theoretical setting differs from ours, HUMUS-Net also performs global spatial mixing across image patches, and we observe highly redundant bottleneck features both quantitatively and qualitatively.

To clarify what drives the improvement, we compare three different intervention strategies.

  1. Rank regularization promotes feature diversity and increases effective rank, but does not consistently improve long training performance, showing that higher rank alone is insufficient.
  2. Deep supervision adds a ground-truth reconstruction target after every cascade, but provides no consistent gains; moreover, the bottlenecks remain low rank, indicating that intermediate reconstruction supervision does not resolve their underutilization.
  3. Foundation model alignment instead provides a dense, informed, and spatially structured target directly at every bottleneck, consistently improving the transformer-based students while increasing effective rank as a byproduct.

We therefore interpret low effective rank as a correlate of bottleneck redundancy rather than an established cause of poorer reconstruction. Among the mechanisms tested, the sustained gains are most consistent with the spatially structured feature supervision provided by the foundation models, rather than with increased rank or additional reconstruction supervision alone.

Q2. Difference from feature distillation and generic feature regularization.

We agree that our approach belongs to the broader family of feature-based knowledge distillation, and we now clarify this connection in the paper.

To test whether the improvement can be obtained without a pretrained teacher, we add two teacher-free alternatives. Deep supervision applies an additional ground truth reconstruction loss to the output of every cascade, testing whether more direct supervision throughout the unrolled network is sufficient. Rank regularization instead acts directly on the bottleneck features, promoting feature diversity without prescribing their spatial content. Rank regularization yields an early gain but does not provide consistent long training gains. Deep supervision also does not consistently improve reconstruction quality in the end. Thus, neither additional ground truth supervision nor teacher-free feature diversification reproduces the gains of foundation model alignment.

Foundation model alignment provides a different signal: a dense, informed, and spatially structured target directly at every bottleneck. The broadly similar gains across four teachers with different architectures, objectives, and training domains suggest that the benefit does not depend on medical semantics or knowledge from a particular teacher. Rather, dense spatial structure may be the relevant shared property.

We acknowledge that alternative forms of spatially structured supervision may provide a similar signal. However, constructing such alternatives is nontrivial and would require designing and validating the target representation and alignment objective. Foundation models provide a simple, effective, and off-the-shelf solution, but we do not claim that they are the only one. We leave the design of alternative targets to future work and would welcome a specific comparison that the reviewer considers particularly informative.

AuthorsOpenReview ↗

Q3, Q4, Minor comments

Q3. Teacher layers, alignment location, and cascade-specific representations.

We select the teacher layers based on established practice: the final four blocks for DINOv3 and MedDINOv3, the first decoder upsampling block for Stable Diffusion following DIFT, and block 8 of 28 for DiT following REPA. We align the student bottleneck following U-REPA and because our analysis identifies feature redundancy at this deepest representation.

To test whether a shared target makes the cascades redundant, we measure the linear CKA at three places: between the bottleneck of each cascade and the teacher features, between the last decoder feature of each cascade and that of the last cascade, and between the teacher features of the image after each cascade and those of the final image, each for aligned runs and no-teacher controls (new appendix figure). As expected, the aligned bottlenecks are similar to the teacher features in every cascade. The decoder features and the intermediate images, on the other hand, are not more similar to those of the last cascade than in the unaligned controls, which suggests that the cascade outputs do not converge to the same representation.

Q4. Seeds, conventional regularization, pretraining.

We extend two seed reporting to all three main tables. Additional seeds are currently infeasible given the cost of training every configuration to 1.02M iterations, and we now state this limitation explicitly. We could not identify a straightforward data augmentation or student pretraining baseline for this setting. Nevertheless, these approaches are complementary to our alignment approach and we did not combine them in this version. If the reviewer has a specific strategy in mind, we would appreciate the reference or suggestion and consider it where feasible.

Minor comments

  • We added an overview figure for readability.
  • We clarify that all student backbones are trained from random initialization.
  • We expand the discussion of why different teachers provide broadly similar gains. This observation is compatible with the Platonic Representation Hypothesis, which suggests that independently trained models may converge toward shared representations. However, we do not test this hypothesis directly.
  • We provide additional details and justification for the teacher-specific preprocessing and feature standardization procedures in the appendix.

Thank you for your responses and revisions. However, several concerns still remain.

  1. There still exist the gap between the motivation and the results of the manuscript. The main contribution of the performance improvement are seemingly from feature alignment with the foundation models, so authors need to reorganize the main motivation and contribution of the proposed work.

  2. Also, the concerns regarding the reliability of experiments still remain (only 1 or 2 seeds). Given the relatively modest performance gain, current results are insufficient to demonstrate its reproducibility. More importantly, given the modest performance gain of the proposed method, it can be simply achieved by reduced overfitting, but related analysis or theoretical justifications are insufficient. It is also uncertain that the proposed method has performance advantages of the most recent SOTA MRI reconstruction methods.

  3. Theoretical analysis / empirical analysis / Ablations studies on design choices (e.g. teacher / student alignment locations) are still missing.

Therefore, I will maintain my original score.

Reviewer_9CpyOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an improvement to end-to-end MRI reconstruction techniques: they align the bottleneck representations with a vision foundation models features, effectively regularizing the latent space of the model and thus eliminating rank collapse issues. They observe consistent improvement in reconstruction metrics across anatomies and both single- and multi-coil settings.

Critical strengths
  • The paper is interesting and ties in findings from representation alignment literature well into the medical image reconstruction domain.
  • The improvements are promising, the proposed method seems to be effective at improving reconstruction fidelity.
  • The experiments are extensive and the baselines are sensible.
Critical weaknesses
  • Overall, it is somewhat unclear what is the exact mechanism that enables the improvements, and whether the rank collapse is a major issue. We don't observe rank collapse with convolutional models, yet the alignment helps. At the same time, out-of-the-box Transformer-based models have collapsed rank yet they outperform convolutional architectures. Having a more concrete understanding of the underlying mechanism would further strengthen this paper.
  • Some details about experimental methodology, metrics and design choices need further clarification (detailed in Questions).
  • It is unclear if the improvements have diagnostic value in medical practice. Evaluation by medical professionals would provide strong supporting evidence for the merits of the paper.
Questions for authors

High-level:

  • Can the authors discuss further on the mechanism of representation alignment that improves reconstructions? It is not directly rank-related as the ablations in the experiments showed. It also helps convolutional models, so cannot be purely Transformer-related. If the benefit is coming from a strong visual prior that restructures the latent space, it is unclear why medical image trained teachers are not better than general domain ones.
  • Can the same effect be achieved by applying supervision at the U-Net outputs in each cascade (using the ground truth) instead of leveraging a foundation model's latent space? This would be a cheaper and more straightforward alternative.

Experiments:

  • Numbers are reported on a validation split for fastMRI knee dataset. Is this the same split the hyperparameters have been tuned for training? This would over-estimate performance on knee data.
  • It appears that most improvements are in PSNR, which favors smoother/blurrier reconstructions. The improvements in SSIM are more modest. Does the technique have a blurring effect?
  • Have authors ablated the effect of projection head size/architecture?

Suggestions:

  • It would be interesting to adapt the teacher features to the unrolling depth. I expect features in unrolled bottlenecks to become more refined, and thus matching it with the teacher can further help (e.g. distill different layers from the same teacher).
  • Have authors ablated the effect of alignment loss weighting between cascades? It could be beneficial to match the weights to the observed trends in rank collapse across cascade depth.

Minor:

  • What does "less distributed" refer to in line 425?
  • Typo in line 472.
3 comments
AuthorsOpenReview ↗

Q1. Mechanism explaining the improvements

We thank the reviewer for their constructive feedback and for raising important questions.

The original framing may have suggested a stronger causal claim than intended, and we have revised the paper accordingly. Our initial observation is that the transformer-based students exhibit very low bottleneck effective rank, indicating that their available feature capacity is underutilized (also illustrated in Figure 3). We use effective rank as a proxy for this underutilization. This work views this association as correlational, not directly causal.

To better disentangle the role of bottleneck rank from other factors, we target bottleneck underutilization in three ways: (i) rank regularization directly promotes feature diversity by encouraging independent feature directions; (ii) deep supervision gives every cascade an additional ground truth reconstruction target; and (iii) foundation model alignment supplies every bottleneck with a dense, spatially organized feature target.

Rank regularization increases effective rank and yields an early gain but no consistent long training gain. Similarly, deep supervision also does not yield consistent gains. Moreover, our effective rank analysis in the appendix shows that the bottlenecks remain low rank under deep supervision, despite the additional supervision applied to every cascade output. Thus, neither increased feature diversity alone nor additional reconstruction supervision is sufficient to produce consistent gains.

Foundation model alignment, by contrast, consistently improves the transformer-based students while increasing their effective rank as a byproduct. We hypothesize that the additional benefit of foundation model alignment comes from encouraging intermediate representations to encode meaningful spatial and structural information, rather than merely increasing their dimensionality. Previous studies show that pretrained foundation models capture rich structural information, including in medical images [1]. By aligning each cascade's bottleneck with dense features extracted from the fully sampled ground truth, our method encourages the reconstruction network to preserve information relevant to the underlying image structures. Unlike direct rank regularization, which promotes feature diversity without specifying what information should be represented, foundation model alignment provides a spatially structured target that guides the learned representations. We expect that richer representations combined with structurally informative supervision account for the reconstruction improvements.

Although our investigation was motivated by the particularly severe rank collapse observed in transformer-based architectures, the proposed regularization is not transformer-specific. Convolutional architectures such as E2E-VarNet already incorporate stronger local spatial inductive biases, which may partly explain their less pronounced rank collapse and smaller improvements from alignment. Nevertheless, foundation model alignment introduces additional supervision from pretrained representations of the fully sampled ground truth, which can also benefit convolutional refinement networks. This is consistent with our experiments, where E2E-VarNet benefits from alignment, although the improvements are substantially larger for transformer-based architectures.

We agree that medical domain foundation models might intuitively be expected to provide more relevant representations for MRI reconstruction. However, the usefulness of pretrained representations depends not only on their training domain but also on the pretraining objective, data diversity, and the downstream task. MedDINOv3 was pretrained on CT images rather than MRI, so its medical domain adaptation does not necessarily translate into better representations for MRI reconstruction. Conversely, general domain foundation models are trained on large and diverse datasets and may learn transferable representations of spatial structures that are also useful for medical images. Our results suggest that such general structural information can serve as effective supervision without medical domain pretraining, and that the benefit of foundation model alignment extends beyond increasing representation rank and is likely related to the additional structural information contained in pretrained teacher representations.

We acknowledge, however, that our experiments do not fully isolate spatial organization from the other properties of the teacher representations. Foundation models offer a simple, effective, and off-the-shelf source of dense spatial supervision, but there are likely other ways to provide such a signal. Exploring these alternatives is a promising direction for future work, but constitutes a separate design problem beyond the scope of the present study.

[1] Cekmeceli et al. "Do vision foundation models enhance domain generalization in medical image segmentation?", 2024. https://arxiv.org/pdf/2409.07960

AuthorsOpenReview ↗

Q2 - Q5, Suggestions and Minor Comments

Q2. Supervising each cascade output with the fully sampled image.

We thank the reviewer for this suggestion and implement it now as a deep supervision baseline across all three architectures. We supervise every cascade output using the fully sampled target:

LDS=∑k=18wkℓk\mathcal{L}_{\mathrm{DS}} = \sum_{k=1}^{8} w_k \ell_k, with wk∝exp⁡(−τ(8−k))w_k \propto \exp(-\tau(8-k))

where the weights are normalized to sum to one and τ\tau increases from zero by 1/31/3 every 6000 iterations, so all cascades are initially weighted equally and later cascades are gradually emphasized. Deep supervision does not consistently improve reconstruction across architectures and seeds, and the transformer bottlenecks remain low rank despite the additional supervision. Thus, supervising intermediate reconstructions does not reproduce the gains of foundation model alignment. This suggests that the improvement is not merely due to additional supervision, but may depend on a dense, spatially structured feature target directly at the bottleneck.

Q3. Calibration and evaluation splits.

This is a fair point. In the previous version, calibration and evaluation slices were disjoint, but slices from the same volume could still appear in both sets. In the revised knee tables, we split by volume: 40 volumes for calibration and 159 disjoint volumes for evaluation. We recalibrate the alignment weights on the calibration set and report revised results on the evaluation set.

Q4. Does alignment produce blurrier reconstructions?

We thank the reviewer for raising this concern. We add qualitative comparisons and quantify edge preservation using

Redge=20log⁡10meanp∈E∣∇r(p)∣meanp∈E∣∇g(p)∣R_{\mathrm{edge}} = 20 \log_{10} \frac{\mathrm{mean}_{p \in \mathcal{E}} \lvert \nabla r(p) \rvert}{\mathrm{mean}_{p \in \mathcal{E}} \lvert \nabla g(p) \rvert},

where rr is the reconstruction, gg is the ground truth, and E\mathcal{E} contains the strongest 5 percent of ground-truth edge locations, obtained by applying a Sobel filter to the Gaussian-smoothed ground truth. We then compare each aligned model with its seed-matched no-teacher control:

dRedge=Redgealigned−RedgecontroldR_{\mathrm{edge}} = R_{\mathrm{edge}}^{\mathrm{aligned}} - R_{\mathrm{edge}}^{\mathrm{control}}.

A negative value indicates that alignment weakens the selected edges relative to the control, whereas a positive value indicates stronger edge preservation. The table reports dRedgedR_{\mathrm{edge}} over all 5706 slices from the 159 single-coil knee evaluation volumes at 1.02M iterations. Each cell reports the two seeds.

Table 1. dRedgedR_{\mathrm{edge}} (dB), first / second seed.

ArchitectureSDDINOv3DiT
E2E-VarNet+0.03 / +0.02+0.15 / -0.21+0.08 / +0.13
HUMUS-Net+0.21 / -0.00+0.16 / +0.20+0.19 / +0.10
HUMUS-Net-L+0.45 / -0.01+0.52 / +0.05+0.47 / +0.01

Alignment gives sharper edges than the matched control in 14 of the 18 pairs and no measurable difference in 3. Thus, we find no evidence that alignment systematically weakens strong edges. We acknowledge however the limitation that this metric measures edge preservation rather than clinical diagnostic value, which would require evaluation by medical experts.

Q5. Projection head.

The projection head architecture is an important design choice and an unablated limitation of our work as other heads may outperform our 3×33 \times 3 convolution. We use an independent 3×33 \times 3 convolution at each cascade bottleneck, as it preserves the spatial layout of the teacher features while permitting local spatial mixing. This is motivated by our hypothesis that spatially structured teacher features contribute to the alignment gains, consistent with iREPA, which emphasizes spatial structure over global semantics. We cannot provide sufficiently trained experiments comparing other heads and leave this to future work.

Suggestion 1. Matching teacher layers to cascade depth.

We agree that for example earlier teacher layers may suit earlier cascades, but evaluating this requires independently selecting and calibrating the layer per cascade, which we are unable to test in this revision. In the current experiments, all cascades use the same representation: the concatenated outputs of the final four blocks for DINOv3 and MedDINOv3, following the DINOv2 linear segmentation protocol; the first decoder upsampling block for Stable Diffusion, following DIFT; and block 8 of 28 for DiT, following REPA. We now clarify these choices and identify cascade-dependent teacher-layer selection as future work.

Suggestion 2. Weighting between cascades.

We weigh the alignment losses of all eight cascades equally. Cascade-dependent weighting may better reflect differences across unrolling depth, but finding a schedule requires an additional calibrated search. We now state equal weighting as an unablated design choice and leave adaptive weighting to future work.

Minor Comments

We replace “less distributed” with “lower effective rank” and fix the typo.

Reviewer_9CpyOpenReview ↗

Response to authors

I thank the authors for the detailed response and additional experiments. My concerns are mostly addressed. I maintain my recommendation of weak acceptance of the paper.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their constructive feedback. We have since revised the paper and added new experiments to address some of it:

  • We extend all main experiments from the originally submitted training budget to 1.02M iterations on both single-coil knee and multi-coil brain data, reporting 30k, 750k, and 1.02M checkpoints.
  • We adopt volume-wise fastMRI evaluation in the main text, move slice-wise results to the appendix, and report two seeds in all three main tables.
  • We add the teacher-free rank regularizer and deep supervision to all three architectures (Table 1). These baselines separate the effects of higher rank, local ground truth supervision and alignment with dense foundation model features.
  • At 1.02M iterations, the best aligned HUMUS-Net runs outperform the best aligned E2E-VarNet on both datasets.
  • We test the main claims with 14 one-sided paired Wilcoxon signed-rank tests over evaluation volumes, Holm-corrected jointly.
  • We revise the writing to better explain the relation between low effective rank and underused bottleneck representations. We also clarify the method, discuss rank regularization and deep supervision in more depth, expand the discussion, and state the limitations explicitly.
  • We add PCA visualizations of the bottleneck to the main text as qualitative support for our claims and a method overview figure to improve readability.

We summarize our main results as follows.

Table 1. Volume-wise PSNR gain (dB) over the no-teacher baseline of the same seed, single-coil knee at 8×8\times, 159 evaluation volumes. Entries are first / second seed; the last column gives the range over the four teachers and both seeds.

StudentIterationsRank regularizerDeep supervisionFour teachers (range)
E2E-VarNet30k+0.17+0.17 / −0.04-0.04+0.05+0.05 / −0.14-0.14+0.17+0.17 to +0.54+0.54
E2E-VarNet1.02M−0.03-0.03 / −0.15-0.15+0.06+0.06 / +0.12+0.12−0.05-0.05 to +0.24+0.24
HUMUS-Net30k+0.58+0.58 / +0.21+0.21−0.34-0.34 / −0.44-0.44+0.17+0.17 to +0.73+0.73
HUMUS-Net1.02M−0.04-0.04 / −0.31-0.31−0.17-0.17 / +0.04+0.04+0.15+0.15 to +0.31+0.31
HUMUS-Net-L30k+0.60+0.60 / +0.43+0.43−0.02-0.02 / −0.12-0.12+0.38+0.38 to +0.85+0.85
HUMUS-Net-L1.02M+0.41+0.41 / +0.36+0.36+0.18+0.18 / −0.07-0.07+0.33+0.33 to +0.51+0.51

Table 2. Volume-wise PSNR (dB) after 1.02M iterations at 8×8\times, first / second seed: the no-teacher baseline and the best of the four teachers (knee) or three teachers (brain).

StudentKnee: no teacherKnee: best teacherBrain: no teacherBrain: best teacher
E2E-VarNet30.2130.21 / 30.0830.0830.3030.30 / 30.3230.3236.5236.52 / 36.6436.6436.7936.79 / 36.8936.89
HUMUS-Net30.2430.24 / 30.0430.0430.4830.48 / 30.3430.3429.1729.17 / 35.9335.9337.1837.18 / 37.1937.19
HUMUS-Net-L29.9729.97 / 29.8329.8330.4130.41 / 30.3430.34n/an/a
Area_Chair_hyjqOpenReview ↗

Reminder: Please Review the Author Rebuttal

Dear Reviewers,

The authors have submitted their rebuttal and addressed the concerns raised in the reviews. Please take some time to carefully examine their responses and participate in the discussion. We would appreciate your updated assessment, including any potential rating adjustments, where appropriate.

Thank you for your time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.