- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MoDeFold, an all-in-one medical image restoration framework that jointly models modality-specific content and degradation information. It uses a contrastively trained modal-degradation embedder and a multi-scale VQVAE with clean-code alignment to guide a shared unfolding network. Experiments on MRI, CT, and PET restoration tasks show competitive results, with ablation studies supporting the proposed design.
Critical strengths
- Reasonable motivation and design.
The idea of jointly considering anatomical content and degradation is well motivated for medical image restoration. Using the degradation embedding to modulate content features before unfolding is an interesting design choice.
- Good experimental results.
MoDeFold performs well across multiple benchmarks. On the main benchmark, it improves average PSNR by about 1.13 dB over R2R. The ablation studies are also fairly complete and show that the proposed components contribute to the final performance.
Critical weaknesses
Critical Weaknesses
- Limited evaluation of generalization.
The main benchmark relies on synthetic degradations, and each dataset uses a relatively fixed degradation setting. It is unclear how well the method handles unseen degradation levels or different acquisition conditions. Also, the improvement on the AMIR benchmark is quite small (0.05 dB in average PSNR over TAT), making the generalization claims less convincing.
- The contribution of the proposed design is not fully isolated.
MoDeFold has 49M parameters, noticeably more than several baselines, and also uses additional pretraining data and stages. Although the ablation results are useful, it is still difficult to tell how much of the gain comes from the proposed joint conditioning rather than increased capacity or pretraining. A more controlled comparison would make the contribution clearer.
Questions for authors
-
Modality–degradation representation. In the main benchmark, degradation types are largely tied to specific datasets. How can we tell whether the learned embedding captures modality–degradation relationships rather than simply dataset identity? Would it generalize to unseen modality–degradation combinations?
-
Necessity of unfolding. Since the learned back-projection (B_k) is not constrained to be the adjoint of (\Phi_k), the updates are not true gradient steps of Eq. (5). What is the actual benefit of the unfolding formulation over a standard restoration network with the same conditioning mechanism?
-
Role of vector quantization. The unfolding network uses continuous content features (c) rather than quantized features (Q(c)). Is the VQVAE codebook actually necessary, or could direct alignment with clean continuous features provide similar benefits?
9 comments
W2: Question about model size and pretraining
Response
We thank the reviewer for the question about model size and pretraining. The increase to 49M parameters mainly comes from processing the low and high frequency components with separately parameterized unfolding branches after the Haar transform. The transform itself adds no trainable parameters. We evaluated a variant without frequency splitting that retains the proposed prior conditioning and uses 28M parameters. Table 1 compares it with the full model and R2R under the same four dataset benchmark.
Table 1. Model size and restoration results. Each result is PSNR in dB followed by SSIM.
| Method | Params | ACDC | COVID19CTscans | HCC TACE Seg | BraTS2021 | Average |
|---|---|---|---|---|---|---|
| R2R | 30M | 26.2389 / 0.7008 | 34.8370 / 0.9499 | 43.1277 / 0.9673 | 30.5578 / 0.9058 | 33.6904 / 0.8810 |
| MoDeFold without frequency split | 28M | 27.3298 / 0.7802 | 35.7321 / 0.9698 | 43.9812 / 0.9810 | 31.0012 / 0.9089 | 34.5111 / 0.9100 |
| MoDeFold full | 49M | 27.6824 / 0.7831 | 36.0758 / 0.9700 | 44.3118 / 0.9829 | 31.1950 / 0.9112 | 34.8163 / 0.9118 |
The 28M variant exceeds the 30M R2R by 0.8207 dB in average PSNR, while the full frequency split adds 0.3052 dB. We will add the new variant and parameter counts to the paper's ablation table.
The ablations further distinguish model size from the proposed guidance. With the unfolding backbone and frequency branches held fixed, removing both priors reduces average PSNR from 34.8163 to 31.0908 dB. Retaining the joint token only at the final fusion stage, without using it to modulate content features, yields 32.3316 dB. Removing contrastive learning or clean code alignment yields 34.1298 and 34.0982 dB, respectively. These comparisons show the contribution of prior learning and content modulation within the same architecture. The additional four datasets are used only for prior pretraining, while end to end unfolding training uses the four datasets in the quantitative benchmark.
Q1: Whether the embedding captures modality and degradation information beyond dataset identity
Response
We thank the reviewer for this question. The two contrastive losses in Eq. 7 use the same embedding but different positive sets. For an anchor , collects images of the same imaging modality, and collects images with the same degradation. These sets are defined by modality and degradation labels, not by dataset identity. Images from different datasets are therefore pulled together when they share the relevant label. When a batch contains a pair from different modalities with the same degradation, that pair is positive for the degradation loss and negative for the modality loss. A representation that merely assigns a separate cluster to each dataset would not satisfy both relations for such cross dataset pairs.
As specified in Appendices A and D, we train the embedder on eight datasets with four degradation categories, rather than using the four dataset restoration benchmark alone. The losses do not create a separate class or token for each observed modality and degradation combination. This allows the shared token to encode the two label relations for combinations outside the benchmark.
W1: Generalization to degradation and acquisition conditions
Response
We thank the reviewer for the thoughtful comments. The AMIR benchmark evaluates MRI super resolution, CT denoising, and PET synthesis, with one restoration task associated with each modality. We therefore also evaluate on a four dataset benchmark comprising cardiac and brain MRI from ACDC and BraTS2021 and thoracic and abdominal CT from COVID19CTscans and HCC TACE Seg. The two CT datasets have different Poisson Gaussian noise settings, and HCC TACE Seg additionally contains directional streaks. This setting evaluates a shared model across distinct anatomy and more than one CT corruption condition. MoDeFold improves average PSNR over R2R by 1.126 dB on this benchmark. On AMIR, it achieves 34.44 dB average PSNR compared with 34.39 dB for TAT, while also improving average SSIM from 0.9358 to 0.9373 and reducing average RMSE from 12.45 to 11.47.
To improve the breadth of prior learning, we pretrain both the Modal Degradation Embedder and content VQVAE on eight datasets covering Gaussian noise, Poisson Gaussian noise, Poisson Gaussian noise with streaks, and low resolution. The final unfolding model is trained on the original four datasets. The embedder learns modality and degradation relations through two contrastive objectives, while clean code alignment encourages the content features to retain structures shared with clean images. These complementary priors provide image dependent guidance when the unfolding model encounters different anatomy and corruption patterns. Consistent with their contribution, removing contrastive learning lowers average PSNR from 34.8163 to 34.1298 dB, and removing clean code alignment lowers it to 34.0982 dB in Table 4.
The eight dataset prior pretraining and four dataset unfolding training are splited all based on the IMIS-Bench partitioning for training or testing, so there’s no data leakage.