acceptodds
Under review as a conference paper at ICLR 2027

Where to Refine and What to Recover: Disentangled Cross-modal Semantic Alignment for CT-to-PET Generation

Abstract

Positron Emission Tomography (PET) and Computed Tomography (CT) play complementary and essential roles in advanced healthcare, particularly for cancer diagnosis. However, CT is far more widely available than PET, primarily due to its substantially lower acquisition and operational costs. To bridge this gap, many CT-to-PET generation methods are developed based on structural alignment while overlooking the intrinsic characteristics of PET, such as metabolic activity, which limits the clinical utility of the synthesized PET images. To address this challenge, we propose a disentangled alignment framework that formulates CT-to-PET generation as a controlled residual generation problem, explicitly guiding both the spatial location ("where") and semantic content ("what") of PET signals. Specifically, a frozen backbone first generates a stable PET prediction, while a Scalable Interpolant Transformer (SiT) models the remaining residual refinement. Then, to characterize what should be recovered, intermediate SiT representations are adopted to align the semantic residual via deep supervision, thereby reinforcing the learning of missing semantics. Furthermore, an activation gate is introduced to identify regions requiring refinement while suppressing redundant residual noise in well-reconstructed areas. Extensive experiments on the popular Lung-PET-CT-Dx and RIDER-Lung-PET-CT datasets demonstrate that our method significantly outperforms state-of-the-art baselines, highlighting its strong potential for CT-to-PET image generation.

Primary area:
generative models
Keywords:
Cross-model Image Synthesis, Resisual Generative Modeling, Representation Alignment, Spatial Localization.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VuXdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a CT-to-PET synthesis framework that refines a frozen ResUNet prediction using a residual SiT. PET-DINO feature differences supervise what to recover, while an activation gate controls where residual corrections are applied.

Critical strengths
  1. Well-motivated refinement strategy. The idea of improving an initial PET prediction by selectively correcting its errors is reasonable. PET-DINO is only used during training, adding no extra cost at inference.

  2. Comprehensive experimental evaluation. The experiments cover external evaluation, high-SUV reconstruction errors, and downstream classification. The improvements in high-SUV regions are particularly encouraging and support the motivation of the proposed method.

Critical weaknesses
  1. Limited methodological novelty. The proposed method mainly combines existing techniques, including residual generation, SiT, representation alignment (REPA), and spatial gating. While the motivation is reasonable, the overall contribution appears relatively incremental, with limited new methodological insights beyond integrating these components.

  2. Insufficient validation of the core claims. Although the ablation study demonstrates performance improvements, it provides limited insight into how the proposed "what to recover" and "where to refine" mechanisms actually work. More in-depth analysis of the learned residual representations and activation maps is needed to directly validate these central claims, rather than relying mainly on performance gains.

Questions for authors
  1. Section 4.4 mainly reports performance improvements without sufficiently explaining why the proposed "what to recover" and "where to refine" mechanisms work. More in-depth analysis would strengthen the core claims, such as comparing residual alignment with full-target REPA-style alignment and PET-DINO perceptual loss, and spatial gating with a learned global scalar. Additional quantitative results on global and high-SUV errors, together with visualizations of predicted residuals, activation maps, and refinement regions, would provide more direct evidence of the effectiveness of these mechanisms.

  2. Please report the training wall-clock time and inference latency, including the cost of ResUNet pretraining and residual refinement, with comparisons to representative baselines.

  3. Please provide more details on the ResUNet architecture and training configuration. Its standalone CT-to-PET generation performance should also be clearly reported on both datasets.

  4. How was PET-DINO obtained? Please clarify its pretraining dataset, training procedure, and whether it was trained from scratch or fine-tuned from an existing model.

  5. Could the authors provide a sensitivity analysis of λrep\lambda_{\mathrm{rep}} in Eq. (29) to further assess the robustness of the proposed residual alignment?

4 comments

Q1: We thank the reviewer for this constructive suggestion. We agree that understanding the mechanisms behind what to recover and where to refine requires more than global reconstruction metrics.

(1) Mechanistic explanation.

As discussed in Section 3.3, Residual Alignment combines absolute feature alignment with residual direction and magnitude alignment. The absolute term preserves global target semantics, while the residual-specific terms guide the generator toward missing semantic corrections relative to the anchor PET. Unlike output-level perceptual supervision, it directly guides the generator's internal representations. Furthermore, the spatially varying Activation Gate (Section 3.4) selectively modulates residual corrections, whereas a global scalar would apply uniform modulation across locations.

(2) Additional quantitative evidence.

We have extended Table 4 and Section 4.6 (p. 10) with component-wise reconstruction errors at SUV>1SUV>1 and SUV>2.5SUV>2.5. Compared with the anchor baseline, Full Fusion reduces MAE/RMSE by 28.3%/33.5% at SUV>1SUV>1 and 37.7%/43.6% at SUV>2.5SUV>2.5. Individual components show different trade-offs, while their combination achieves the lowest errors across all four regional metrics. Figure 5 additionally illustrates the qualitative progression of residual refinement.

We acknowledge that direct comparisons against full-target REPA, PET-DINO perceptual loss, and learned global-scalar gating, together with explicit residual and activation-map visualizations, would provide stronger evidence isolating the individual mechanisms. These comparisons are not included in the current results, and we limit our conclusions accordingly.

Q2: Thank you for the suggestion. We have added a computational complexity and runtime analysis in the Appendix, reporting model parameters, GFLOPs, and inference latency for representative baselines. Our method requires 61.91461.914M parameters and 1,531.151,531.15 GFLOPs, with an inference latency of 264.737264.737 ms. It achieves 4.76×4.76\times and 1.84×1.84\times faster inference than BBDM and FastDDPM, respectively. We also clarify the training cost of both stages: approximately 2 hours for ResUNet anchor pretraining and 5 hours for Residual SiT refinement, resulting in a total training time of approximately 7 hours. These additions provide a more comprehensive assessment of the computational cost of our framework.

Q3: Thank you for this suggestion. We have expanded Appendix B to provide a detailed description of the ResUNet architecture and its independent training configuration. Specifically, we clarify its five-stage residual encoder, four-stage decoder, channel dimensions (64/128/256/512/51264/128/256/512/512), residual blocks, normalization and activation layers, and skip-connected upsampling strategy. We also provide the training details, including the input resolution, batch size, learning rate, weight decay, loss composition and coefficients, mixed-precision setting, and EMA configuration. These additions clarify how the deterministic CT-to-PET backbone is constructed and trained before being frozen as the anchor generator for subsequent residual refinement.

Q4: We thank the reviewer for this important question. We have clarified the pretraining dataset and architecture of PET-DINO in the revised manuscript.

Specifically, PET-DINO is pretrained on PET images from 168 lung cancer patients in the AutoPET dataset. It adopts a ViT-B/16 architecture with 12 Transformer blocks, a 768-dimensional embedding space, 12 attention heads, and a patch size of 16×1616\times16.

During downstream CT-to-PET training, PET-DINO is kept completely frozen and serves exclusively as a feature teacher for Residual Alignment. Specifically, it extracts representations from the ground-truth PET and the generated anchor PET, and their feature difference provides the semantic residual supervision for Residual SiT. Its parameters are never updated during downstream optimization, and the entire PET-DINO pathway is removed during inference.

We have added these details to the revised manuscript to improve the transparency and reproducibility of our approach.

Q5: Thank you for the comment. We adopt λREPA=0.5\lambda_{\mathrm{REPA}}=0.5 to balance semantic representation supervision and the primary flow-matching objective. The three internal REPA components are assigned fixed weights of 0.45, 0.45, and 0.10 to capture complementary aspects of semantic residual alignment. We emphasize that the effectiveness of this design is already verified through the controlled ablation study in Table 4, where removing Residual Alignment while keeping the other components unchanged leads to consistent performance degradation. Specifically, PSNR decreases from 32.562 to 28.247 dB, SSIM from 0.8742 to 0.7935, and PCC from 0.8964 to 0.7859. These results provide direct evidence that Residual Alignment makes a substantial contribution to the proposed framework. Thus, λREPA=0.5\lambda_{\mathrm{REPA}}=0.5 is used as a fixed auxiliary supervision weight, rather than as an independently optimized parameter.

Thank you for the comprehensive and complete rebuttal, which has addressed some of my concerns. However, the paper still suffers from weak methodological novelty and insufficient mechanism analysis. Thus, I will keep my original rating.

We appreciate the reviewer's candid assessment. We would like to clarify that our contribution does not lie in introducing a new standalone backbone, but in formulating an anchor-conditioned residual refinement framework with two explicitly disentangled objectives.

(1) Anchor-relative semantic residual learning. Unlike conventional full-target generation or standard feature alignment, we explicitly model the semantic discrepancy between the ground-truth PET and the frozen anchor PET, defined as ΔZT=ZGT−Zanchor\Delta Z_T=Z_{\mathrm{GT}}-Z_{\mathrm{anchor}}. By combining absolute feature alignment with residual direction and magnitude alignment, our method preserves global target semantics while specifically guiding the generator to recover information missing from the anchor prediction. This provides a task-specific semantic supervision strategy for residual generation.

(2) Spatially adaptive correction localization. Our Activation Gate is not designed to identify high-metabolic regions, but to estimate where the anchor prediction requires correction. Conditioned on both CT and anchor PET, it predicts a spatially varying modulation map G=σ(gϕ(CT,B))G=\sigma(g_\phi(CT,B)), which selectively controls the application of generated residuals. This distinguishes correction localization from conventional attention-based metabolic region identification.

(3) Complementary integration. These mechanisms address two distinct but interdependent questions: Residual Alignment determines what semantic information to recover, while Activation Gating determines where to apply the corrections. Their integration produces the final prediction Y^=B+G⊙ΔY\hat{Y}=B+G\odot\Delta Y, enabling selective refinement while preserving reliable anchor predictions.

Therefore, the methodological contribution lies in the anchor-relative semantic supervision and spatially selective correction formulation, rather than the individual use of established network components. We believe this distinction is central to understanding the novelty of our framework.

Reviewer_dKuLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed a CT-to-PET translation framework that focuses on semantic characteristics of PET instead of just structural similarities between the two modalities. Specifically, they proposed a residual modelling framework that addresses what to recover and where to refine. The model uses a pretrained backbone to produce a generated anchor PET image, then to learn what to recover from the PET, the framework models the residual correction through a semantic residual alignment strategy with the ground truth image. The model then learns an Activation Gate that decides where to refine the generated results with the residual correction. The method achieved state-of-the-art results on multiple CT-to-PET translation tasks, and downstream prediction accuracy.

Critical strengths

Technically sound framework with clear motivations: The paper cleverly motivates CT-to-PET problem as a body of two questions: what to reconstruct and where to do that on top of a stable anchor prediction in order to preserve spatial structures and represent semantics that are weakly coupled with structural appearance. The Residual Alignment component encourages the model to recover lost semantic information by the original anchor.

Extensive experimental results: The method not only achieved state-of-the art results on two popular CT-to-PET translation datasets, but also evaluated in localized regions with more cross-modal discrepancy and difficulty, demonstrating superior performance on harder regions. The framework also exhibits strong downstream predictive performance too.

Critical weaknesses

Many key method components are underspecified, and the headline comparison depends on them. The "Fusion" component from Table 4 is undefined in the paper, which is separate from the Activation Gate, so it is unclear what component this row is describing. This component adds significant results so need to be specified accurately ( +0.63 dB and +0.017 SSIM). The objectives to train the gates in Equation 8 were also not elaborated on. Table 5 lists a gate-only stage, a representation-adaptation stage, alignment weights (0.45/0.45/0.10), and a 0.25 residual scale. None of these training stages are explained in the manuscript. Meanwhile, Section 4.1 and Appendix B say the SiT, alignment head, and gate are "optimized jointly", which conflicts with this table. Finally, the model PET-DINO the paper used is not cited or described anywhere, which is critical for reproducibility purposes.

Missing experiments on certain design choices: The paper made design choices that were not validated by experimental results. In particular, does supervision on the residual have empirical gains over complete target representation (Equation 1 and 2)? Why use generative modelling when you can use MSE regression directly to learn the residual?

Questions for authors

Q1 Clarify the unexplained components in the methodology, as stated in Weaknesses. The authors need to clarify in detail what the Fusion components referred to in Table 4 are, explain the detail of the training stages, and their PET-DINO encoder.

Q2 Additional ablation studies, as stated in Weaknesses. I would like to know the effect of residual learning over typical ground truth supervision, and generative modelling over regression.

Q3 Ablation studies in hypermetabolic regions Table 4 showed the effects of each component on general performance. I would like to see the ablation on hypermetabolic areas like in Table 3, which is the core purpose of the method. Specifically, I want to know how each of these components help the model improve its prediction on areas with high cross-modal discrepancy.

Q4 Robustness to other backbones All experiments utilized a pretrained ResUnet to generate the anchor PET. Does the framework work well for other backbones like Pix2Pix, BBDM...? Since the method itself is quite architecture agnostic, we should expect performance improvements over multiple translation backbones.

Q5 Additional comparison with similar baselines There is a previous existing work with a similar approach to the "where" question. Specifically, CPDM tried to guide the diffusion process through the Attention Map and Attenuation Map. How does the method compare with CPDM [1] in performance? What are the advantages of the authors' learnable activation gates over CPDM's approach? How do the importance maps differ between these two methods in detail?

[1] Nguyen, Dac Thai, et al. "Ct to pet translation: a large-scale dataset and domain-knowledge-guided diffusion approach." 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE, 2025.

2 comments

Q1: We thank the reviewer for the valuable suggestions. We have clarified the three points as follows.

(1) Fusion components (Table 4; Sections 3.4 and 4.6).

Gate refers to the trainable spatial prediction branch, while Gated Fusion denotes the parameter-free operation Y^=Yanchor+G⊙ΔY\hat{Y}=Y_{\mathrm{anchor}}+G\odot\Delta Y. The former learns where to refine, whereas the latter applies the predicted modulation during reconstruction.

(2) Training stages (Appendix B, p. 15).

We employ three stages over 3,000 steps: Stage One (0–499) trains only the Gate; Stage Two (500–999) updates SiT Block 4, the REPA projector, and the Gate; and Stage Three (1000–2999) jointly optimizes all SiT blocks, the projector, and the Gate. ResUNet and PET-DINO remain frozen throughout. Stage-specific learning rates are also detailed in Appendix B.

(3) PET-DINO (Section 3.3; Appendix B, pp. 14–15).

PET-DINO is pretrained on PET images from 168 lung cancer patients in AutoPET. It uses a frozen ViT-B/16 encoder with 768-dimensional embeddings. Teacher features are spatially aligned with SiT tokens through bilinear interpolation, and a lightweight projector enables semantic residual supervision. PET-DINO is used only during training.

Q2: We thank the reviewer for this valuable suggestion. We have added a controlled 2×22\times2 ablation study in Appendix C.5 (Table 10) to investigate the individual contributions of residual learning and generative modeling.

Residual learning consistently improves reconstruction accuracy, reducing MAE by 14.95%14.95\% and 21.11%21.11\% under regression and generative modeling, respectively. Meanwhile, generative modeling improves PSNR and SSIM under both prediction targets. In particular, Residual Generation improves PSNR by 0.1540.154 dB and SSIM by 0.01100.0110 over Residual Regression, with marginal increases in MAE and RMSE.

These results demonstrate the complementary benefits of residual learning and generative modeling.

Q3: We thank the reviewer for this valuable suggestion. We have extended Table 4 with MAE and RMSE evaluations in GT-defined hypermetabolic regions (SUV>1SUV>1 and SUV>2.5SUV>2.5), and added corresponding analysis in Section 4.6.

The results show that Residual SiT alone improves high-uptake reconstruction but increases errors in broader uptake regions. Residual alignment and localization gating provide complementary benefits, addressing semantic refinement and spatial selectivity, respectively. Their combination achieves the best performance across all four metrics.

Compared with the baseline, Full Fusion reduces MAE/RMSE by 28.3%/33.5%28.3\%/33.5\% for SUV>1SUV>1 and 37.7%/43.6%37.7\%/43.6\% for SUV>2.5SUV>2.5. These results demonstrate the importance of jointly modeling what and where to refine.

Q4: We thank the reviewer for this valuable suggestion. To verify the architecture generality of our framework, we conducted additional experiments using two alternative anchor generators: Pix2Pix (GAN-based) and BBDM (diffusion-based). Each pretrained backbone is frozen, and our residual refinement framework is trained independently.

For Pix2Pix, PSNR increases by 0.8850.885 dB, while MAE and RMSE decrease by 7.60%7.60\% and 24.05%24.05\%, respectively. For BBDM, PSNR improves by 0.3550.355 dB, with slight reductions of 0.09%0.09\% in MAE and 0.94%0.94\% in RMSE.

These results demonstrate that our framework generalizes beyond ResUNet and provides consistent improvements across different translation architectures. We have added the corresponding results and analysis to the Appendix.

Q5: We thank the reviewer for this insightful comment. We have added CPDM to Table 1 and clarified the distinctions between the two approaches.

(1) Quantitative comparison.

Our method consistently outperforms CPDM on both datasets. On Lung-PET-CT-Dx, PSNR increases from 32.21432.214 to 34.31634.316 dB, while MAE and RMSE decrease by 7.25%7.25\% and 19.64%19.64\%, respectively. On RIDER-Lung-PET-CT, PSNR improves from 30.01230.012 to 31.37031.370 dB, with MAE and RMSE reductions of 16.39%16.39\% and 14.43%14.43\%. SSIM and PCC also improve on both datasets.

(2) Differences in importance maps.

CPDM employs two domain-knowledge-guided maps: an Attention Map trained to identify potential high-metabolic-activity regions and a CT-derived Attenuation Map encoding tissue attenuation information. These maps are incorporated as additional conditions throughout diffusion. In contrast, our Activation Gate explicitly learns where the generated anchor PET requires correction, rather than simply highlighting potential metabolic activity. Importantly, high metabolic activity does not necessarily imply inaccurate prediction.

(3) Advantages of our Activation Gate.

Our gate jointly considers CT and anchor PET information and directly modulates residual updates, allowing selective correction while preserving reliable predictions. Furthermore, PET-DINO representation alignment independently guides what semantic information to recover, explicitly separating correction localization from residual generation.

Therefore, unlike CPDM's domain-knowledge-guided generation, our framework disentangles where to refine and what to recover, offering anchor-adaptive refinement with improved reconstruction accuracy.

Reviewer_NPWPOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a residual framework for CT-to-PET generation. A frozen ResUNet produces an anchor PET image, while a SiT models residual corrections. PET-DINO feature differences provide representation supervision, and a spatial gate controls where corrections are applied. Validation includes two lung datasets, an external cohort, high-uptake region analysis, and downstream classification.

Critical strengths
  1. The residual refinement strategy has a clear motivation: preserve the anchor prediction while limiting unnecessary modifications.

  2. Evaluation extends beyond global reconstruction metrics to high-uptake regions, an external cohort, and downstream classification.

Critical weaknesses
  1. The reported metrics contain unexplained inconsistencies. Table 1 reports MAE = 0.141200 and RMSE = 0.067900 for Palette; Table 2 reports MAE = 0.125730 and RMSE = 0.035750 for Fast-DDPM. Under standard definitions with the same scale and aggregation, MAE cannot exceed RMSE. The paper specifies no alternative convention explaining these values. These discrepancies undermine confidence in the baseline comparisons.

  2. Essential methodological and evaluation details are missing. The appendix names LgenL_{\mathrm{gen}} without defining the generative loss, interpolation path, or prediction target. The source and pretraining data of PET-DINO are unspecified. Patient counts, split proportions, and downstream classification labels are also missing. Furthermore, Table 4 treats Activation Gate and Fusion as separate components without clearly explaining their distinction, making the ablation difficult to interpret.

Questions for authors
  1. What explains the MAE values exceeding RMSE in Tables 1–2? Please specify the normalization and aggregation procedures and correct any reporting errors.

  2. Please define the implemented generative objective, identify PET-DINO’s pretraining data, report patient counts for each split, and clarify the distinction between Gate and Fusion in Table 4.

1 comment

Q1: We thank the reviewer for identifying the MAE/RMSE inconsistencies in Tables 1–2. We agree that MAE cannot exceed RMSE when both metrics are computed over identical samples with consistent aggregation. The affected entries were reporting errors, and we apologize for these inaccuracies.

Normalization and aggregation. As specified in Appendix A (Eqs. 19–22), PET intensities are represented as SUVbw, clipped to [0,10][0,10], and normalized to [0,1][0,1], without patient-specific or slice-wise min–max normalization. For global evaluation, MAE is averaged over per-slice absolute errors, while RMSE is the square root of the mean per-slice MSE. Since all slices have the same resolution (256×256256\times256), this is equivalent to pooled-pixel aggregation. Region-level metrics are separately defined in Appendix A (Eqs. 25–27).

Corrections. We re-ran the evaluation and corrected the following entries:

  • Table 1, Lung-PET-CT-Dx (Palette): RMSE 0.067900→0.1679000.067900 \rightarrow \mathbf{0.167900}.
  • Table 1, RIDER-Lung-PET-CT (Palette): RMSE 0.091913→0.1919130.091913 \rightarrow \mathbf{0.191913}.
  • Table 2, Small Cell (Fast-DDPM): MAE 0.125730→0.0257300.125730 \rightarrow \mathbf{0.025730}.

All corrected entries satisfy MAE≤RMSE\mathrm{MAE}\leq\mathrm{RMSE}. These changes are reflected in Table 1 (p. 7) and Table 2 (p. 8) of the revised manuscript.

Q2: We thank the reviewer for these helpful suggestions. We clarify the four points below.

(1) Generative objective (Section 3.2, Eqs. 1–3).

We employ hotspot-weighted conditional flow matching in the residual PET space. Given the ground-truth PET image YY and the frozen anchor prediction BB, the target residual is defined as R=(Y−B)/sR=(Y-B)/s. We construct the interpolation path as xt=(1−t)x0+tRx_t=(1-t)x_0+tR and optimize the predicted velocity against the target velocity v∗=R−x0v^*=R-x_0. Additional weighting is applied to regions with SUV thresholds of 1, 2.5, and 5. The overall objective combines flow matching, PET-REPA (λ=0.5\lambda=0.5), and auxiliary regularization (Eq. 11 and Appendix B).

(2) PET-DINO pretraining (Section 3.3).

PET-DINO was pretrained on PET images from 168 lung cancer patients in the AutoPET dataset. The encoder remains frozen during downstream training and provides semantic residual supervision only during training.

(3) Patient splits (Section 4.1 and Appendix A).

The Adenocarcinoma cohort contains 51/10/14 patients for training, validation, and testing, respectively, corresponding to 17,892/3,614/5,550 slices. Cross-histology evaluation is conducted on one held-out Small Cell Carcinoma patient (381 slices), without fine-tuning. RIDER is an independent external evaluation cohort comprising 47 patients and 16,901 paired CT–PET slices, which are excluded from generator training and validation.

(4) Gate versus Gated Fusion (Section 4.6, Table 4).

Gate denotes the trainable spatial map prediction branch, defined as G=σ(gϕ(X,B))G=\sigma(g_\phi(X,B)), whereas Gated Fusion denotes the parameter-free operation Y^=B+G⊙ΔY\hat{Y}=B+G\odot\Delta Y. This distinction separates learning where to refine from applying the predicted spatial modulation during reconstruction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.