acceptodds
Under review as a conference paper at ICLR 2027

TUG: Tumor-Guided Longitudinal MRI Forecasting with Trajectory Selection

Abstract

Forecasting longitudinal MRI requires modeling tumor evolution while preserving anatomical realism, but these objectives are not always aligned. Image-oriented generative models may produce high-fidelity scans while underrepresenting tumor changes, whereas stronger tumor supervision may compromise image quality. We introduce TUG, a tumor-guided forecasting framework that bridges this appearance–evolution gap during training and inference. During training, we introduce an evolution-aware residual modulator that injects tumor structure, follow-up interval, and treatment information into the generative residuals. We further supervise the lesion expressed by one-step clean-image estimates and preserve future-tumor information in intermediate diffusion features. During inference, we introduce tumor-aware candidate scoring to progressively prune low-ranked trajectories and softly weight the completed predictions using terminal tumor evidence. Across three longitudinal glioma MRI cohorts, our approach consistently improves tumor-evolution fidelity while retaining competitive image quality, demonstrating complementary benefits from training and inference time.

Primary area:
generative models
Keywords:
Longitudinal MRI Forecasting, Glioblastoma, Diffusion Models, Medical Image Generation, Tumor Dynamics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ARMKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents TUG, a BrLP-based conditional 3D latent diffusion model designed to align whole-image and tumor-specific objectives. TUG incorporates three key designs: (1) an Evolution-Aware Residual Modulator (ERM) conditioning ControlNet residuals on baseline tumor masks, intervals, and treatments; (2) two auxiliary losses (ECL and ERA) that supervise future lesions using frozen segmentation networks; and (3) a test-time trajectory pruning and blending strategy via a learned ranker. On 228 validation pairs across three cohorts (LUMIERE, MU-Glioma-Post, UCSF-ALPTDG), TUG achieves the highest Dice scores while matching BrLP in PSNR and SSIM.

Critical strengths
  • The problem is well motivated: Fig. 1 and the intro make a convincing case that PSNR/SSIM alone hide clinically relevant lesion errors, and reporting image and lesion fidelity jointly (Table 1, Fig. 4) is a useful practice for this field.
  • The method is well designed: ERM is initialized to preserve the pretrained pathway, the future mask is used only as supervision, and the trajectory scorers are trained with patient-grouped out-of-fold predictions so that no future information is used at inference.
  • The comparison is compute-matched (Eq. 14), and the ablations cleanly separate the contribution of training-time guidance (Table 2a) from inference-time refinement (Table 2b). The reported numbers are internally consistent.
Critical weaknesses
  • The tumor-fidelity metric is the same quantity that TUG optimizes.

    • The nnU-Net segmenter used for evaluation (L356, L705-706) is the one used in the ECL loss, in the prospective scorer's inputs (L305-306), and to build the scorer's training targets (L724-725). TUG is therefore trained and selected against the exact function used to score it, while the baselines are not. Part of the Dice gain may come from fitting this specific segmenter rather than from better tumor evolution.
    • The limitations section acknowledges this (L483-484), but it is central to the main claim. At least one independent segmenter (different architecture or training data), ideally complemented by expert reading on a subset, should be used for evaluation.
  • The evaluation measures the final lesion state rather than tumor evolution.

    • Dice against the future mask can be high even if nothing changes. A copy-current-scan baseline (X̂f = Xs) is missing and is essential in longitudinal forecasting, especially since MU and UCSF masks include the resection cavity (L672-677), which is largely static.
    • Change-specific metrics would directly support the paper's claim: e.g. error in lesion volume change, Dice restricted to the growth/regression regions, or agreement in response category (progression/stable/regression).
  • The inference-time gain is not isolated from the effect of averaging fewer samples.

    • Table 2b compares Mean20 with "Pruning + Mean8". Averaging fewer samples by itself produces a sharper lesion (the paper itself argues that averaging suppresses spatially varying structure, L50-53), so Dice can increase without a good scorer. A random-pruning + Mean8 control is needed, together with a simple heuristic selector (e.g. consensus/medoid) and an oracle best-of-N upper bound to show how much of the available gap the learned scorer closes.
    • The final output is still an (weighted) average image, and Dice is computed on that average. Results on individual samples (e.g. mean Dice of the top-1 trajectory) would clarify whether the method improves forecasts or mainly improves aggregation.
  • The evaluation protocol needs clarification and statistical support.

    • Results are reported on a "held-out validation set" (L325), and the TaDiff-3D checkpoint is selected "from validation performance" (L805). It is unclear which split was used to choose TUG's hyperparameters (loss weights, timestep ranges, the 24→20→8 schedule, τ). If the same 228 pairs were used, the results are not test results.
    • No standard deviations, confidence intervals or significance tests are reported. Several key gaps are small (e.g. 0.5217 vs 0.5155 case-avg. Dice against TaDiff-3D). A paired test over the 228 cases and results over multiple sampling seeds are needed.
    • Results are pooled across cohorts with different lesion definitions (L672-677). Per-cohort results should be reported.
  • Baselines and the role of ERM.

    • It is not stated whether BrLP and LDM-GBM were trained on the same 949 pairs, nor what the "pretrained forecasting model" used for initialization (L215-216) was trained on.
    • SADM (PSNR 15.51, SSIM 0.196, Dice 0.06) and TaDiff-3D (SSIM 0.52) are author adaptations whose results suggest they may be undertrained or poorly suited to this setting. This weakens the comparison, as acknowledged in L484-485.
    • ERM alone does not appear to help much: "No extra loss" (ERM + Mean20, Table 2a) vs. BrLP (Table 1) gives +0.012 case-avg. Dice but −0.011 SSIM. ERM is never ablated against ECL+ERA without ERM.
  • Novelty is mainly in the combination. Each component closely follows an existing technique: FiLM-style residual modulation (ERM), reward feedback on one-step clean estimates as in ControlNet++ (ECL), representation readout/alignment as in Readout Guidance/REPA (ERA), and progressive seed pruning/TTSnap (trajectory pruning). The application to longitudinal forecasting is reasonable, but the paper should state more explicitly what is new beyond adapting these techniques.

Questions for authors
  • Please report Dice with an independent segmenter, a copy-current-scan baseline, and at least one change-specific metric.
  • Please add the random-pruning + Mean8 control and an oracle best-of-N upper bound for Table 2b.
  • Please clarify which data were used for hyperparameter selection, and report per-cohort results with confidence intervals or paired significance tests.
9 comments

W1: Independent Segmentation Evaluation and Evaluator Bias

The tumor-fidelity metric is the same quantity that TUG optimizes.

The nnU-Net segmenter used for evaluation (L356, L705-706) is the one used in the ECL loss, in the prospective scorer's inputs (L305-306), and to build the scorer's training targets (L724-725). TUG is therefore trained and selected against the exact function used to score it, while the baselines are not. Part of the Dice gain may come from fitting this specific segmenter rather than from better tumor evolution.

The limitations section acknowledges this (L483-484), but it is central to the main claim. At least one independent segmenter (different architecture or training data), ideally complemented by expert reading on a subset, should be used for evaluation.

We thank the reviewer for this important observation. To address potential evaluator bias, we conducted additional experiments using an independently trained 3D SegResNet segmenter (S2), which differs architecturally from the original nnU-Net evaluator and was not involved in ECL supervision, prospective scorer training, or trajectory selection. To account for random initialization variability, S2 was trained with three independent random initializations, and the reported results were averaged across the three runs.

MethodFuture Dice (S2)
BrLP0.3224
TaDiff-3D0.4189
TUG (Ours)0.4702

TUG achieves a paired improvement of +0.0513 (95% CI [0.0187, 0.0791]) over TaDiff-3D. These results demonstrate that TUG's improvement in future lesion-state agreement is not confined to the original evaluation segmenter, mitigating the concern that the observed gains arise solely from fitting the segmentation function used during training.

W2: Future Lesion-State Fidelity vs. Evolution-Specific Accuracy

The evaluation measures the final lesion state rather than tumor evolution.

Dice against the future mask can be high even if nothing changes. A copy-current-scan baseline (X^f=Xs\hat{X}_f = X_s) is missing and is essential in longitudinal forecasting, especially since MU and UCSF masks include the resection cavity (L672-677), which is largely static.

Change-specific metrics would directly support the paper's claim: e.g. error in lesion volume change, Dice restricted to the growth/regression regions, or agreement in response category (progression/stable/regression).

We thank the reviewer for this insightful comment. We agree that future-tumor Dice evaluates agreement with the observed future lesion state rather than explicitly measuring the spatial localization of growth or regression. However, we would like to clarify that future lesion-state fidelity remains a meaningful and relevant objective in longitudinal MRI forecasting.

In our pairwise forecasting setting, the initial state is fixed and known, making the observed future lesion geometry a clinically relevant prediction target. This endpoint-based evaluation is also consistent with established longitudinal glioma forecasting studies, such as TaDiff (Liu et al., IEEE TMI 2025). Accordingly, our objective is to generate future MRI volumes whose expressed lesion structures spatially agree with the observed future state, rather than to predict lesion masks independently. Future-tumor Dice therefore measures an important aspect of image-based forecasting fidelity, although it does not fully characterize the underlying evolution.

We therefore regard change-specific metrics as important complementary evaluations, rather than replacements for future-state fidelity. Our contribution is to introduce lesion-aware supervision and trajectory selection into full-volume MRI forecasting to improve the agreement between generated and observed future lesion states.

W3: Effectiveness of Trajectory Pruning and Aggregation

The inference-time gain is not isolated from the effect of averaging fewer samples.

Table 2b compares Mean20 with "Pruning + Mean8". Averaging fewer samples by itself produces a sharper lesion (the paper itself argues that averaging suppresses spatially varying structure, L50-53), so Dice can increase without a good scorer. A random-pruning + Mean8 control is needed, together with a simple heuristic selector (e.g. consensus/medoid) and an oracle best-of-N upper bound to show how much of the available gap the learned scorer closes.

The final output is still an (weighted) average image, and Dice is computed on that average. Results on individual samples (e.g. mean Dice of the top-1 trajectory) would clarify whether the method improves forecasts or mainly improves aggregation.

We thank the reviewer for this valuable suggestion. We agree that reducing the number of averaged trajectories may itself affect lesion fidelity, independently of the learned scorer. To isolate these effects, we conducted additional ablations involving random pruning, oracle selection, and different aggregation strategies.

(1) Learned pruning vs. random pruning. We introduced a Random Pruning + Mean8 baseline, which retains the same number of trajectories as our learned pruning strategy but selects them randomly.

Selection / AggregationFuture Dice
Mean20 (without pruning)0.4871
Random Pruning + Mean80.4847
Learned Pruning + Mean80.5089
Learned Pruning + Soft Weighting (τ=1)0.5217
Oracle Top-8 + Mean80.5288
Oracle Top-10.5505

With the number of averaged trajectories held constant, learned pruning improves Dice over random pruning by +0.0242 (95% CI [0.0168, 0.0325]). Learned pruning also outperforms the unpruned Mean20 baseline, whereas random pruning does not. This indicates that the improvement cannot be attributed solely to reducing the number of averaged samples.

(2) Oracle selection. We further evaluated ground-truth-guided oracle selection to estimate the remaining potential of candidate ranking.

Selection StrategyFuture Dice
Learned Pruning + Mean80.5089
Oracle Top-8 + Mean80.5288
Oracle Top-10.5505

Oracle Top-8 + Mean8 achieves a Dice of 0.5288, compared with 0.5089 for learned pruning under the same eight-candidate averaging protocol. This leaves a gap of 0.0199, suggesting that the learned scorer captures a substantial portion of the available improvement while leaving room for better candidate selection. Oracle Top-1 reaches 0.5505, additionally indicating the existence of high-quality individual trajectories within the candidate pool.

(3) Selection vs. aggregation. The comparison between learned pruning with uniform averaging (0.5089) and subsequent soft weighting (0.5217) suggests that both candidate selection and aggregation contribute to the final prediction. Importantly, our random-pruning control establishes that the benefit of learned selection persists even when the aggregation operation and number of retained candidates are identical.

Overall, these ablations support the effectiveness of the learned trajectory scorer beyond the confounding effect of averaging fewer samples, while highlighting the complementary roles of candidate selection and aggregation.

Reviewer_7BmvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TUG forecasts a follow-up glioma T1C scan from the current scan, tumour mask, time interval and treatment. It builds on a pretrained BrLP-style 3D latent diffusion model with ControlNet and adds three training-time components: FiLM-style modulation of ControlNet residuals (ERM), a segmentation-consistency loss on one-step clean estimates through a frozen segmenter (ECL), and a representation-alignment readout that predicts future lesion logits (ERA). At inference, 24 trajectories are progressively pruned to 8 by learned rankers and then soft-averaged, under a budget matched to 20 full trajectories. On 228 validation pairs pooled from LUMIERE, MU-Glioma-Post and UCSF-ALPTDG, TUG reports a case-averaged Dice of 0.5217, against 0.5155 for TaDiff-3D and 0.3628 for BrLP, with slightly lower PSNR/SSIM than BrLP.

Critical strengths
  1. Whole-volume PSNR/SSIM is dominated by stable anatomy and misses tumour change. For me, it is really targting a meaningful evaluation gap. The paper shows the BrLP vs. TaDiff trade-off clearly and scores lesion fidelity by segmenting every method's output with one fixed segmenter. This matters to my assessment because it reframes what longitudinal forecasting should be judged on, which is useful to the community regardless of the method.

  2. The test-time selection is set up carefully. The comparison is compute-matched in denoiser evaluations, and the scorers are trained on patient-grouped out-of-fold data and never see future images at inference. Data are split by patient across three public cohorts, and the ablation separates training-time from inference-time effects. This is fairer than most test-time-scaling work and is the part of the paper I find most convincing.

Critical weaknesses
  1. The segmenter that produces the reported Dice is the same checkpoint used inside ECL, and the trajectory scorers are trained on Dice from that same segmenter. It is somehow curlar from my point of view. TUG therefore optimises its evaluation metric at both training and inference time, while the baselines do not. Part of the gain may be images this particular segmenter likes rather than more accurate tumours. This is decisive for me because the headline result cannot be separated from the metric. Dice from an independent segmenter, lesion-ROI intensity metrics, or a blinded reader study would address it.

  2. The margin over TaDiff-3D is 0.006 Dice on a single split of 228 pairs, with no confidence intervals, seeds or significance tests. Therefore, the improvement is small and not shown to be reliable. The same split was used for selection (TaDiff-3D's checkpoint was chosen on it, and it is not stated how TUG's λs, timestep ranges, pruning schedule or τ were chosen). Results are pooled across cohorts whose lesion definitions differ (MU and UCSF include the resection cavity, LUMIERE does not). There is also no no-change baseline (X_s, or M_s as the future mask), so 0.52 Dice is hard to interpret. Together these mean I cannot tell whether TUG beats TaDiff-3D at all. Furthermore, the components closely follow existing methods (ControlNet++, REPA/Readout Guidance, FiLM, Progressive Seed Pruning, BrLP latent averaging), so the contribution rests mainly on this empirical result.

Questions for authors
  1. How do the method rankings change when Dice comes from a segmenter that is not used in training or scoring?
  2. What Dice does the identity (no-change) forecast get, and what does an oracle best-of-N get?
  3. Which hyperparameters were tuned, and on which data? Can you report results on a test split held out from all selection, with patient-level bootstrap CIs and paired tests against TaDiff-3D?
  4. What are the per-cohort and leave-one-cohort-out results, and how much does including the resection cavity raise Dice?
  5. How does random pruning to 8 trajectories compare with learned pruning? The current Pruning+Mean8 vs. Mean20 comparison changes both selection and the number of averaged samples.
  6. What is the effect of removing ERM, which is never ablated?
  7. What is the wall-clock time and memory compared with Mean20, given that each pruning stage decodes and segments every surviving 3D latent?
14 comments

W1: Evaluator Bias and Circular Optimization

The segmenter that produces the reported Dice is the same checkpoint used inside ECL, and the trajectory scorers are trained on Dice from that same segmenter. It is somehow curlar from my point of view. TUG therefore optimises its evaluation metric at both training and inference time, while the baselines do not. Part of the gain may be images this particular segmenter likes rather than more accurate tumours. This is decisive for me because the headline result cannot be separated from the metric. Dice from an independent segmenter, lesion-ROI intensity metrics, or a blinded reader study would address it.

We thank the reviewer for highlighting this important concern. We agree that using the same segmenter for ECL supervision, trajectory scoring, and final evaluation may introduce evaluator-specific bias, and that improvements under this evaluator alone would be insufficient to establish generalizable lesion fidelity.

To address this concern, we conducted additional experiments using an independently trained 3D SegResNet (S2) with three independent random initializations. S2 was not involved in ECL supervision, trajectory scoring, or inference-time selection.

As detailed in our response to Q1, the method ranking remains unchanged under S2: TUG achieves a Future Dice of 0.4702, compared with 0.4189 for TaDiff-3D and 0.3224 for BrLP. Notably, the TUG–TaDiff-3D improvement increases from +0.0062 under the original evaluator (S1) to +0.0513 under S2 (95% CI [0.0187, 0.0791]), using transition-weighted evaluation with patient-clustered bootstrap confidence intervals.

These results demonstrate that TUG's improvement is not confined to the segmenter used during training and selection, directly addressing the concern that our method merely produces images favored by the original evaluator.

We acknowledge that independent automated segmentation does not fully replace blinded clinical assessment or establish accurate spatial evolution in every case. Nevertheless, the consistent ranking under an independently trained evaluator provides substantially stronger evidence for the effectiveness of our lesion-aware forecasting framework.

W2: Statistical Reliability, Evaluation Protocol, and Methodological Novelty

The margin over TaDiff-3D is 0.006 Dice on a single split of 228 pairs, with no confidence intervals, seeds or significance tests. Therefore, the improvement is small and not shown to be reliable. The same split was used for selection (TaDiff-3D's checkpoint was chosen on it, and it is not stated how TUG's λs, timestep ranges, pruning schedule or τ were chosen). Results are pooled across cohorts whose lesion definitions differ (MU and UCSF include the resection cavity, LUMIERE does not). There is also no no-change baseline (X_s, or M_s as the future mask), so 0.52 Dice is hard to interpret. Together these mean I cannot tell whether TUG beats TaDiff-3D at all. Furthermore, the components closely follow existing methods (ControlNet++, REPA/Readout Guidance, FiLM, Progressive Seed Pruning, BrLP latent averaging), so the contribution rests mainly on this empirical result.

We agree that the original +0.0062 Dice improvement over TaDiff-3D is insufficient on its own to establish reliable superiority, and that validation reuse, annotation heterogeneity, and strong identity baselines require careful consideration. We have conducted additional experiments to address these concerns.

Statistical reliability and independent evaluation. As detailed in Q1 and Q2, patient-clustered bootstrap analysis shows that the original S1 transition-weighted improvement of +0.0062 has a 95% CI of [−0.0178, +0.0294], which includes zero. The S1 patient-weighted difference is −0.0200 [−0.0356, −0.0036]. However, under the independently trained S2 evaluator, TUG achieves 0.4702 Dice versus 0.4189 for TaDiff-3D, an improvement of +0.0513 (95% CI [0.0187, 0.0791]). The S2 patient-weighted improvement is +0.0129 [−0.0139, +0.0393], indicating that the advantage remains sensitive to aggregation. We therefore acknowledge the uncertainty in the original result rather than claiming consistent statistical superiority.

Validation protocol and baseline interpretation. As discussed in Q1–Q3, we additionally evaluated Raw MRI Identity, AE Identity, and Mask Persistence, and conducted cohort-stratified and StrictWT evaluations. Raw Identity achieves 0.5841/0.5023 Dice under S1/S2, exceeding TUG's 0.5217/0.4702, while Mask Persistence achieves 0.6398. We acknowledge that persistence is a strong baseline for endpoint lesion overlap.

However, our methodological contributions operate in the latent forecasting stage, using the frozen autoencoder inherited from BrLP. AE Identity, which reconstructs the current MRI through this autoencoder without temporal forecasting, achieves 0.4468/0.3987 under S1/S2. TUG improves upon this reconstruction-constrained baseline by +0.0749 under S1 and +0.0715 under S2, supporting the effectiveness of our latent forecasting framework while revealing a substantial reconstruction bottleneck. The StrictWT experiments further show competitive performance against TaDiff-3D under a stricter tumor definition, although they do not establish consistent superiority. We also acknowledge the heterogeneity across cohorts and annotation conventions.

As discussed in Q2, validation was used for checkpoint and inference-strategy selection, and an untouched independent test set is not currently available. We will clarify the hyperparameter-selection procedure and distinguish development-validation findings from independent generalization claims.

Methodological contribution beyond the original performance margin. We recognize that individual techniques such as FiLM modulation, representation alignment, and progressive pruning have precedents. We do not claim these operations as standalone inventions. Rather, our contribution lies in their lesion-aware formulation and coordinated application to 3D longitudinal latent forecasting, where global appearance fidelity and lesion-specific structural fidelity must both be considered.

Importantly, additional controlled ablations provide evidence beyond the original +0.0062 comparison. Removing ERM reduces S2 Dice by 0.0594 in the controlled E100 comparison, while learned pruning improves over random pruning by +0.0242 Dice under S1 and +0.0169 under S2, using matched candidate counts and aggregation protocols (see Q4 and Q5).

These results provide component-level evidence for the contributions of lesion-aware residual modulation and learned trajectory selection, rather than relying solely on the marginal original comparison against TaDiff-3D.

Overall, we have refined our interpretation to avoid overstating universal superiority. Our central contribution is not solely a marginal increase in Dice, but a lesion-aware forecasting framework that explicitly addresses the complementary requirements of image fidelity and structural lesion fidelity, supported by independent evaluation and controlled component-level evidence.

Q1: Independent Segmenter, Identity Baselines, and Oracle Selection

How do the method rankings change when Dice comes from a segmenter that is not used in training or scoring? What Dice does the identity (no-change) forecast get, and what does an oracle best-of-N get?

We thank the reviewer for raising these important questions regarding independent evaluation, identity baselines, and oracle selection. We conducted additional experiments to address each concern.

1. Independent Segmenter Evaluation.

To assess potential evaluator bias, we independently trained a 3D SegResNet (S2) with three random initializations. Unlike the original segmenter (S1), S2 was not involved in ECL supervision, trajectory scoring, or inference-time selection.

MethodDice (S1)Dice (S2)
BrLP0.36280.3224
TaDiff-3D0.51550.4189
TUG (Ours)0.52170.4702

Importantly, the method ranking remains unchanged under S2. TUG outperforms TaDiff-3D by +0.0513 Dice (95% CI [0.0187, 0.0791]) and BrLP by +0.1478 Dice under independent evaluation. These results indicate that TUG's improvements are not restricted to the segmenter used during training and trajectory selection.

2. Identity Baselines and the Latent Reconstruction Bottleneck.

We additionally evaluated three persistence-based references:

MethodDice (S1)Dice (S2)
Mask Persistence0.63980.6398
Raw MRI Identity0.58410.5023
AE Identity0.44680.3987
TUG (Ours)0.52170.4702

Mask Persistence directly uses the current ground-truth lesion mask as the future prediction, without requiring segmentation of a generated image.

We acknowledge that Raw MRI Identity outperforms TUG on endpoint Dice. This highlights both the strength of persistence in longitudinal glioma imaging and the limitations of endpoint overlap for evaluating spatial evolution.

However, an important distinction is that our methodological contributions operate in the latent forecasting stage, using the pretrained autoencoder adopted from BrLP. We do not redesign or optimize this autoencoder.

To isolate the effect of the latent reconstruction pathway, AE Identity reconstructs the current MRI through the same frozen autoencoder without forecasting any temporal changes.

The substantial degradation from Raw Identity to AE Identity (0.5841→0.4468 under S1; 0.5023→0.3987 under S2) demonstrates that the inherited autoencoder introduces considerable lesion-information loss even before future-state prediction.

Despite this limitation, TUG improves over AE Identity by +0.0749 Dice under S1 and +0.0715 under S2, demonstrating that our latent-space forecasting and selection framework provides meaningful improvements over the reconstruction-constrained persistence reference.

These findings support the effectiveness of our contributions within the adopted latent representation, while also identifying reconstruction fidelity as an important limitation of the overall pipeline. Improving the underlying autoencoder could potentially enable further gains, although this remains to be experimentally verified.

3. Oracle Best-of-N Selection.

We additionally evaluated ground-truth-guided oracle selection from the generated candidate trajectories:

Selection StrategyDice (S1)
Learned Pruning + Mean80.5089
Oracle Top-8 + Mean80.5288
Oracle Top-10.5505

Oracle Top-1 selects the individual candidate with the highest ground-truth Dice, while Oracle Top-8 selects the eight highest-ranked candidates before averaging.

The oracle results demonstrate that higher-quality lesion predictions are present among the sampled trajectories, but current learned selection does not always identify them. This indicates remaining opportunities to improve candidate ranking and aggregation.

We emphasize that oracle selection uses future ground truth and serves only as a diagnostic reference, not as a deployable inference strategy.

Overall, these additional experiments establish that TUG's improvements over generative baselines persist under independent segmentation evaluation, that its latent forecasting mechanism improves upon the AE-constrained identity reference, and that further improvements may be achievable through better reconstruction and trajectory selection.

Reviewer_MM8QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposed a tumor-guided longitudinal MRI forecasting framework (TUG), which focuses on the balancing between producing high-fidelity scans and accurately depicting the important local changes of the disease. During the training phase, TUG introduces Evolution-Aware Residual Modulator (ERM), which applies tumor supervision in the image space and representation space through Evolution Consistency Loss (ECL) and Evolution Representation Alignment (ERA), respectively, to jointly optimize the overall image fidelity and the quality of local lesion prediction. In the inference phase, TUG further introduces tumor-guided trajectory selection and aggregation to improve the utilization efficiency of random diffusion trajectories through prospective trajectory pruning and terminal soft weighting.

The experimental results conducted on three longitudinal glioma MRI datasets show that TUG maintains competitive similarity (PSNR, SSIM) and tumor prediction quality (Dice). The ablation experiments further validate the contributions of ECL, ERA, and the two inference phases to the final performance. However, the current manuscript still lacks verification of key modules and mechanisms.

Critical strengths
  1. This paper focuses on the inconsistency between the fidelity of the overall image and the fidelity of the local disease evolution. This problem is in line with the actual needs of longitudinal medical image prediction.

  2. The experiments demonstrate clear performance improvements compared to multiple longitudinal MRI prediction methods, maintaining good PSNR and SSIM while improving the tumor Dice. The ablation experiments provide preliminary evidence of the effectiveness of different components.

Critical weaknesses
  1. The core claim of tumor-evolution fidelity lacks direct verification. The paper mainly uses the Dice of the future tumor area to evaluate the quality of evolution prediction, but the static segmentation overlap cannot fully reflect the actual changes of the tumor over time. Besides, this work used the same frozen tumor segmentationer for ECL supervision and Dice evaluation.

  2. The ablation and mechanism verification of key components are insufficient. The results lack independent ablation of ERM, and important controls such as ERA-only, random pruning, and weighting-only. Besides, this work did not directly analyze the correlation between intermediate trajectory scores and the final prediction quality, thus the core mechanism for Prospective Trajectory Pruning is not verified.

Besides, the results are mainly reported on the merged validation set, lacking independent performance of each cohort and cross-cohort generalization verification. Moreover, the comparison of SOTA is insufficient (e.g., Brain-WM, TumorFlow). The paper also lacks statistical results (e.g., multiple random seeds and segmentationers) and detailed efficiency analysis of different methods and components

Questions for authors

Why is the Dice metric for future lesions mainly used to measure tumor-evolution fidelity? Can other methods be added to directly verify the predictive ability for disease changes? Is TUG sensitive to random noise? Why is ERM only applied to a sparse subset of residual scales?

6 comments

W1: Validation of Tumor-Evolution Fidelity

The core claim of tumor-evolution fidelity lacks direct verification. The paper mainly uses the Dice of the future tumor area to evaluate the quality of evolution prediction, but the static segmentation overlap cannot fully reflect the actual changes of the tumor over time.

We thank the reviewer for this important observation. In our pairwise longitudinal forecasting setting, the initial state is known and fixed. Therefore, a more accurate prediction of future lesion geometry provides a more faithful representation of the observed evolution outcome relative to that initial state. This endpoint-based evaluation is consistent with established longitudinal glioma forecasting studies, such as TaDiff (Liu et al., IEEE TMI, 2025), which evaluates predicted tumor progression through agreement with the observed future tumor state. Accordingly, we use future-tumor Dice to assess whether the generated MRI expresses recognizable tumor structures that spatially agree with the observed future lesion. Importantly, this evaluation is performed on tumor regions extracted from the generated MRI, rather than on separately predicted masks, ensuring that the measured agreement reflects the lesion structures expressed in the forecasted image.

Besides, this work used the same frozen tumor segmentationer for ECL supervision and Dice evaluation.

To further address potential evaluator bias, we conducted additional experiments using an independently trained segmenter (S2), which was not involved in ECL supervision. To account for initialization variability, S2 was trained with three independent random initializations, and the reported results were averaged across the three runs.

MethodFuture Dice (S2)
BrLP0.3224
TaDiff-3D0.4189
TUG (Ours)0.4702

TUG achieves a paired improvement of +0.0513 (95% CI [0.0187, 0.0791]) over TaDiff-3D, supporting the generalizability of the observed improvement beyond the original evaluation segmenter.

W2: Component Ablations and Mechanistic Validation

We thank the reviewer for highlighting the need for more comprehensive ablations and mechanism validation. We conducted additional experiments to assess the individual contributions of ERM, ECL, ERA, prospective pruning, and terminal weighting.

The ablation and mechanism verification of key components are insufficient. The results lack independent ablation of ERM, and important controls such as ERA-only, random pruning, and weighting-only.

Training component ablations. We completed the missing ERA-only experiment, yielding the following results:

ConfigurationFuture Dice (S1)
ERM-only0.3746
ERM + ECL0.4549
ERM + ERA (w/o ECL)0.3576
ERM + ECL + ERA (Full)0.4871

Although ERA alone does not improve upon ERM-only, adding ERA to ERM + ECL yields an additional +0.0322 Dice, suggesting that ERA provides complementary benefits when combined with image-space supervision. We also conducted an independent ERM ablation under matched training conditions. Removing ERM reduces future Dice by 0.0520 (S1) and 0.0594 (S2), with the latter yielding a 95% CI of [0.0455, 0.0764].

Pruning and weighting ablations. With matched candidate counts, learned pruning outperforms random pruning by +0.0242 Dice (S1) and +0.0169 Dice (S2). Holding the retained candidates fixed, terminal soft weighting further improves Dice over uniform averaging by +0.0128 (S1) and +0.0035 (S2), although the latter improvement is not statistically conclusive.

Besides, this work did not directly analyze the correlation between intermediate trajectory scores and the final prediction quality, thus the core mechanism for Prospective Trajectory Pruning is not verified.

Prospective scoring mechanism validation. We directly examined the relationship between intermediate trajectory scores and final candidate quality. The mean within-case Spearman correlations are 0.478 and 0.578 at the two pruning stages, respectively. The corresponding top-candidate retention rates are 89.4% and 66.1%, compared with random-selection baselines of 83.3% and 40.0%. Together with the random-pruning control, these results demonstrate that intermediate scores contain useful predictive information for identifying promising trajectories before generation is complete.

Besides, the results are mainly reported on the merged validation set, lacking independent performance of each cohort and cross-cohort generalization verification.

Cohort-wise evaluation. To examine performance across heterogeneous datasets, we further report the improvements over TaDiff-3D separately for each cohort under the independent segmenter S2:

CohortTUG − TaDiff-3D (S2 Dice)
LUMIERE+0.0993
MU-Glioma-Post+0.0452
UCSF-ALPTDG−0.0072

TUG achieves improvements on LUMIERE and MU-Glioma-Post, while showing a small negative difference on UCSF-ALPTDG. These results provide a more transparent assessment of cohort-level variability and indicate that the improvement is not uniform across datasets.

Moreover, the comparison of SOTA is insufficient (e.g., Brain-WM, TumorFlow).

Additional SOTA comparisons. We acknowledge the relevance of Brain-WM and TumorFlow. Brain-WM addresses joint treatment prediction and future MRI generation, and its publicly released codebase has undergone ongoing refinement, making a fully matched reproduction under our evaluation protocol challenging. TumorFlow, in contrast, focuses on biophysically guided MRI synthesis conditioned on tumor-concentration fields propagated through a growth model. Although its original task setting differs from our pairwise longitudinal forecasting framework, we are actively investigating whether it can be adapted to our evaluation protocol. We will report any additional results as soon as they become available.

The paper also lacks statistical results (e.g., multiple random seeds and segmentationers) and detailed efficiency analysis of different methods and components.

Statistical robustness and computational efficiency. As reported in W1, we additionally evaluated TUG using an independent segmenter (S2) trained with three random initializations, and report paired improvements with 95% confidence intervals. Regarding computational efficiency, Mean20 inference requires 14.13 s per prediction, compared with 56.11 s for learned pruning with Mean8 aggregation and 66.16 s for the complete refinement pipeline with terminal soft weighting. The measured peak GPU memory usage is 9.66 GiB.

Q1: Why Primarily Use Future-Tumor Dice?

Why is the Dice metric for future lesions mainly used to measure tumor-evolution fidelity?

As explained in W1, our task focuses on predicting the future tumor state from a known initial state. Since the current tumor mask is given, the spatial agreement between the predicted and observed future lesions provides a direct measure of the fidelity of the forecasted evolution endpoint. Future-tumor Dice is therefore a natural primary metric for this pairwise forecasting setting. Moreover, unlike global image-fidelity metrics such as PSNR and SSIM, Dice explicitly evaluates the lesion structures expressed in the generated MRI, which are central to our tumor-guided forecasting objective.

Q2: Direct Evaluation of Disease Evolution

Can other methods be added to directly verify the predictive ability for disease changes?

We agree that more direct assessment of disease evolution would further strengthen the evaluation. One promising approach is to employ a clinically validated classifier to assess tumor evolution based on the current and predicted future MRIs. We plan to incorporate such clinically informed evaluation methods in future work.

Q3: Sensitivity to Random Noise

Is TUG sensitive to random noise?

TUG explicitly accounts for stochastic variation through multi-trajectory generation and refinement. Rather than relying on a single noise initialization, we generate 24 candidate trajectories from different random seeds, progressively prune less promising trajectories, and softly aggregate the retained predictions. This design aims to reduce reliance on any individual stochastic realization while exploiting diversity among candidate futures. Empirically, multi-candidate averaging improves future Dice from 0.436 (single trajectory) to 0.487 (Mean24), while learned pruning and terminal weighting further improve it to 0.522. These results demonstrate the practical benefits of stochastic aggregation and trajectory refinement over single-trajectory prediction.

Q4: Sparse Residual-Scale Modulation in ERM

Why is ERM only applied to a sparse subset of residual scales?

ERM is applied to a selected subset of ControlNet residual scales to introduce tumor-specific spatial and evolution-related conditioning while limiting perturbations to the pretrained generative pathway. This design maintains the original residual connections at initialization and avoids introducing additional modulation parameters at every 3D feature scale. The intention is to balance targeted conditioning and preservation of the pretrained image prior. We acknowledge that this is a design rationale rather than evidence that sparse modulation is optimal.

Comments on the paper

1 comment

Summary of Rebuttal and Reviewer Discussion

We sincerely thank all reviewers for their constructive feedback and thoughtful discussions. Throughout the rebuttal, we conducted extensive additional experiments to address the major concerns and strengthen the evidence for TUG's methodological contributions. We summarize the key findings and reviewer discussions below as a reference for the subsequent deliberation.

1. Summary of Additional Experimental Evidence

  • Independent evaluation: An independently trained 3D SegResNet (S2), evaluated across three random initializations, confirms TUG's improvement over TaDiff-3D (0.4702 vs. 0.4189 Dice, paired difference +0.0513, 95% CI [0.0187, 0.0791]), addressing concerns about evaluator-specific optimization.
  • Component contributions: Removing ERM reduces S2 Dice by 0.0594, while learned pruning consistently outperforms random pruning (+0.0242 under S1; +0.0169 under S2). Additional ERA-only and weighting-only experiments further clarify the contributions of individual components.
  • Mechanism validation: Intermediate trajectory scores correlate with final candidate quality (Spearman 0.478/0.578), providing direct evidence for prospective trajectory selection.
  • Evaluation completeness: We additionally reported identity and oracle baselines, change-specific metrics, patient-clustered confidence intervals, cohort-wise results, and computational costs.

Together, these experiments provide substantially stronger evidence that the proposed tumor-guided training and trajectory selection mechanisms contribute meaningfully beyond the original baseline comparisons.

2. Summary of Individual Reviewer Discussions

  • Reviewer ARMK (3 → 3): We addressed the concerns regarding independent evaluation, identity baselines, change-specific metrics, component ablations, statistical reliability, and methodological novelty. The reviewer explicitly confirmed that their main concerns were resolved and maintained a positive assessment, expressing willingness to discuss the paper with other reviewers.
  • Reviewer 7Bmv (2 → 3): The reviewer positively recognized the additional component ablations and independent S2 evaluation. Following a further question regarding S2's performance on real follow-up MRIs and its training independence, we provided detailed clarifications on the architecture, patient-level data split, and evaluation protocol. The reviewer subsequently raised their score from 2 to 3.
  • Reviewer MM8Q (2 → 2): The reviewer acknowledged that our additional experiments addressed many concerns, while continuing to question evolution fidelity and robustness. We further clarified that tumor-aware future-image fidelity is itself a meaningful forecasting objective. Global image similarity alone cannot adequately characterize localized pathological structures. Prior work such as TaDiff (Liu et al., IEEE TMI 2025) has likewise emphasized the importance of modeling and evaluating future tumor states beyond global image similarity.

Given a known initial state, the spatial agreement between tumor-related structures in generated future MRIs and those observed in real follow-up scans provides meaningful evidence of future-state prediction. Importantly, TUG improves this agreement under independent evaluation while achieving substantially higher whole-image fidelity than TaDiff-3D (25.61 vs. 19.56 dB PSNR). Furthermore, the independent ERM ablation, learned-versus-random pruning controls, prospective scoring analysis, and multi-trajectory refinement provide concrete evidence for the effectiveness of our framework.

3. Closing Remarks

We are encouraged that two reviewers now support the paper with scores of 3, including one score increase following the additional experiments and clarifications.

We would like to emphasize that TUG's contribution lies in jointly improving tumor-aware structural fidelity and maintaining whole-image quality through coordinated training- and inference-time mechanisms. The additional independent evaluations and controlled ablations substantiate this contribution beyond the original performance comparisons.

We sincerely appreciate the reviewers' careful consideration and hope these findings will be useful in the final reviewer–AC discussion. We will incorporate the additional experimental results, methodological clarifications, and insights gained from the rebuttal discussions into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.