- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces TumorCast, a model to predict longitudinal CT scans in latent space. The paper adopts the JEPA framework and incorporates clinical information to guide the future state generation. The authors validates the effectiveness of generated latent states with an LLM as the backbone on lesion diameter, size, and metastasis status prediction tasks on a private longitudinal CT dataset.
Critical strengths
The topic of predicting longitudinal CT in latent space is novel and of interest to the research community.
Critical weaknesses
- The technical contributions are limited because the authors follow the standard JEPA framework and do not extend beyond this technique.
- The downstream evaluation is problematic with an LLM as the backbone. LLMs are powerful and might easily neglect the impact of input latent states.
Questions for authors
-
The technical contributions are limited. This work used the standard JEPA framework consisting of a teacher-student architecture and a predictor. Cross attention between text and vision and AdaLN are incorporated. The training paradigm separates the update of the encoder and predictor. Autoregressive training is simplified. Details such as KV cache, mask attention, and impact of supervising the endpoints only need to be added. The authors should consider extending beyond these techniques to strengthen this work scientific contributions.
-
The downstream evaluation is problematic. On downstream tasks, the authors finetuned an LLM (Qwen3.5) to process latent states and output predictions including lesion diameter, lesion size change, and liver metastasis status. However, an LLM is powerful and can easily demolish or ignore the impact of the latent state. Hence, this setting cannot determine the contributions of the latent states. To verify their effectiveness, the authors should design a setting where the latent states are the dominant factor, such as linear probing.
-
In addition, the downstream tasks do not expand the scope of conventional methods, and the discovery is not novel. Predicting diameter, size, and metastasis status are regression or classification tasks that can be done by supervised learning conditioned on the current CT scan. The authors should configure downstream tasks to improve the impact of this study, such as counterfactual response, intermediate state interpretation, etc.
-
The interpretability of the proposed model is weak. Since the major claim of this paper is that the progression process is important, the authors should consider provide interpretation of the learned intermediate states to demonstrate its relevance to final prediction.
-
The claim in the introduction section that JEPA-type models discard the predictor does not hold. In model-based planning, the JEPA’s predictors were trained and applied.
-
The use of terminology lacks rigidity, such as “chained training” -> “autoregressive training”. The word “curriculum” is often used to describe training with increasing levels of difficulty, but this work uses it to describe different stages of model development.
-
The clarity of equations needs to be improved, such as equations (4) and (7). Adding description of the symbols could help.
In sum, the technical contributions and downstream application need to be improved.
3 comments
Author Response to Reviewer wNBh (Part 2)
(Q2-1) Can You Design an Evaluation Where Latent States Are the Primary Factor?
We thank the reviewer for this suggestion. We added two evaluations that use the predicted latent states without the bridge or LLM reader.
1. Latent trajectory alignment. With the encoder and predictor frozen, we measured the cosine similarity between predicted and observed latent changes, , on the 171 size test cases.
| Pairing configuration | Mean cosine similarity [95% CI] |
|---|---|
| Another patient’s predicted change vs. true change | 0.0112 [−0.0058, 0.0361] |
| Training-set mean organ change vs. true change | 0.0357 [−0.0023, 0.0768] |
| Patient’s own predicted change vs. true change | 0.3282 [0.2789, 0.3814] |
The patient-matched prediction achieved substantially higher alignment than the two near-zero baselines, supporting that the predictor captures patient-specific latent trajectories.
2. Linear probing. With the encoder and predictor frozen, we trained organ-specific ridge probes on the full predicted latent states to estimate future lesion diameter, without the bridge or LLM reader. Regularisation was selected using only training patients. On the 171 size test cases, linear probing achieved 43.12% BAcc, outperforming the best fine-tuned open-weight VLM baseline (36.25%) by 6.87%.
Together, these reader-free evaluations show that the predicted latent states capture patient-specific changes and contain task-relevant information recoverable through linear probing.
(Q2-2) Does the LLM Ignore the Latent State?
We thank the reviewer for this question. We performed a fixed-reader latent-replacement test on TumorCast-27B, replacing each case’s latent with predicted latents from three same-organ patients while keeping the question and history unchanged. Results are averaged over the three replacements on 171 size samples and 71 liver-metastasis samples.
| Metric | Correctly matched latent | Random latent replacement (mean of 3) |
|---|---|---|
| Size BAcc ↑ | 48.66% | 43.68% |
| Liver-metastasis AUPRC ↑ | 71.43% | 65.39% |
| Liver-metastasis BAcc ↑ | 78.68% | 74.84% |
Random replacement reduced the three metrics by 4.98%, 6.04%, and 3.84%, showing that the LLM uses patient-specific latent information.
(Q3-1) Can you conduct other downstream task like treatment-conditioned response analysis?
We thank the reviewer for this suggestion. Because individual counterfactual outcomes are unobservable, we conducted a counterfactual sensitivity analysis on the 127 size-forecasting cases with recorded treatment plans. We replaced the original treatment with “No anti-tumour treatment” in both the latent-dynamics module and the text-based question-answering stage, while keeping all other inputs unchanged.
| Metric | Result |
|---|---|
| Changed growth status | 56 / 127 (44.09%) |
| Worsened growth status | 49 / 127 (38.58%) |
| Unchanged / Regressed growth status | 71 / 127 (55.91%) / 7 / 127 (5.51%) |
| Mean tumor diameter increase (No Treatment vs. Original) | +4.38 mm |
These results confirm that our model follows clinical logic, predicting +4.38 mm larger tumor growth and 38.58% disease progression when treatment is removed.
(Q3-2 & Q4) Are the Learned Intermediate States Interpretable?
-
Clinical interpretability (Q2-1). Trajectory alignment and linear probing show that the predicted latent states preserve patient-specific evolution and contain lesion-size-related signals recoverable by a linear probe.
-
Relevance to final predictions (Q2-2). Replacing the patient-matched latent states reduced model performance, indicating that the Reader uses patient-specific latent information when generating its predictions.
These results show that the learned intermediate states encode patient-specific disease evolution and clinical attributes and contribute to the final predictions, rather than being opaque features ignored by the Reader.
Author Response to Reviewer wNBh (Part 3)
(Q5 & Q6 & Q7) Terminology Standardization and Equation Clarity
We thank the reviewer for these helpful suggestions. In the revised manuscript, we will standardize our terminology and clarify our mathematical formulations as follows:
- Predictor retention in JEPA. We acknowledge that V-JEPA 2-AC retains its predictor for action-conditioned planning. We will correct our statement and instead emphasise TumorCast’s predictor design for discrete treatment events and variable time intervals.
- Chained Latent Supervision (Q1-3): We clarify this term to denote multi-step latent rollout under sparse, irregular CT sampling, distinguishing it from standard dense autoregressive training.
- Staged Training: We agree that "Curriculum Learning" could be misleading since samples are not ordered by difficulty, and we will replace it with Staged Training throughout the manuscript.
- Equations and Symbols: We will add complete symbol definitions and detailed explanations for all equations to improve readability.
Author Response to Reviewer wNBh (Part 1)
(Q1-1) Technical Contributions and Relationship to the JEPA Framework
We appreciate the opportunity to clarify how TumorCast advances beyond a direct application of standard JEPA. Standard JEPA relies on dense, regular observations—an assumption that breaks down in clinical oncology, where CT scans are sparse, irregularly timed, and driven by multimodal interventions (EHR and treatments). To overcome these barriers, we introduce three non-trivial, clinically grounded adaptations:
(Q1-2) Clarification on the comment: “training separates the update of the encoder and predictor”
Rather than updating the encoder and Predictor in isolation, we adopt a progressive strategy to balance domain adaptation with training stability:
1. Joint Adaptation: Initially, the 3D CT encoder and Predictor are updated jointly to adapt the generic static visual encoder to a domain-specific representation space tailored for longitudinal lesion prediction. 2. Frozen-Encoder Predictor Training: Once adapted, we freeze the encoder to prevent the follow-up supervision targets (St+1) from drifting during optimization, allowing the Predictor to stably learn treatment- and interval-driven disease dynamics within a fixed latent space.
(Q1-3) Why Name It “Chained Latent Supervision” Instead of “Autoregressive”?
We thank the reviewer for noting this connection. While multi-step rollout shares a recursive form with autoregressive inference, we adopt Chained Latent Supervision to accurately define our training objective and avoid terminological ambiguity.
In the revision, we will clarify its conceptual link to autoregressive rollout while consistently using Recursive Latent Rollout for multi-step state prediction at inference.