acceptodds
Under review as a conference paper at ICLR 2027

GBM-WAM: Anatomy-guied Latent World-Action Model for Brain Glioblastoma Treatment Planning

Abstract

Planning next-step treatment for glioblastoma (GBM) requires current magnetic resonance imaging (MRI) and clinical history, while follow-up scans provide longitudinal supervision for modeling treatment-relevant progression beyond direct observation-to-action prediction. World-action models (WAMs) provide a natural paradigm for exploiting such longitudinal information, yet common imagine-thenact designs generate explicit future observations before predicting the action. For volumetric MRI, explicit future generation requires costly reconstruction of dense 3D anatomy and appearance details that are not uniformly relevant to treatment, while treatment-relevant progression is distributed across complementary MRI sequences that may not all be available in clinical practice. We therefore propose GBM-WAM, an anatomy-guided latent WAM that recasts explicit future volume generation as partial-to-complete latent world learning within a causal multimodal large language model (MLLM), enabling training-rich, inference-light planning from variable MRI subsets and clinical context. To learn GBM progression dynamics from partial MRI inputs, a latent world expert combines a mixture-oftransformer architecture with flow matching to predict complete multi-sequence future representations, shaping world query states with shared and sequencespecific progression cues. Concurrently, a temporal mask expert performs statemodulated spatial decoding to supervise localized tumor morphology and longitudinal volumetric changes, providing complementary anatomical guidance. While both auxiliary experts supervise intermediate states during training, they are bypassed at inference to enable direct treatment planning in a single MLLM pass. Experiments on internal and external multicenter cohorts show that GBM-WAM outperforms state-of-the-art methods, achieving up to 93.2% accuracy and up to 91.6% F1 score, while accommodating variable combinations of available MRI sequences and requiring only 0.375 s per case for direct treatment inference.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Brain Glioblastoma, World-Action Model, Treatment Planning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qotwOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes GBM-WAM, an anatomy-guided latent World-Action Model that learns future MRI representations through a Latent World Expert and supervises tumor segmentation and longitudinal changes through a Temporal Mask Expert to improve next-step treatment prediction for glioblastoma. Overall, the paper is clearly written and demonstrates strong experimental performance.

Critical strengths
  1. This work presents a World Action Model for glioblastoma treatment planning, which carries strong clinical value. Generating costly volumetric MRI also represents a practical challenge in real-world deployment.
  2. The experimental design is relatively comprehensive with competitive results. The paper includes evaluations on internal and external datasets, experiments across different modality combinations, module ablation studies, and inference efficiency comparisons.
  3. The model flexibly handles incomplete MRI modalities.
  4. The manuscript is well-written with clear figures and tables, making it easy to follow.
Critical weaknesses
  1. Here is a concern regarding the “world action model”. The model does not seem to produce next observations. Instead, it leverages future MRI prediction during training to optimize latent states, while directly outputting treatment plans via the Action Head at inference time. Given that future tumor simulation is optional at inference and appears to serve primarily as an auxiliary supervision objective for the VLA model during training, what fundamentally distinguishes the proposed approach from conventional VLA methods augmented with future prediction losses?

  2. Where is the anatomy-guided component manifested? Is it via segmentation loss, or other mechanisms? Or does it refer to the 3D representations extracted from NeuroVFM? How does this anatomy-guided design specifically affect the action output?

  3. What is the distribution of the five treatment categories across the internal and external test sets? Per-class classification performance should be reported, especially the accuracy for the tail categories.

  4. The claim about modeling “longitudinal disease progression dynamics” is not well supported by metrics computed in the latent space. Latent representations may blur fine-grained details and primarily reflect static spatial information, the dynamic progression of tumor regions cannot be accurately quantified in this manner.

  5. The overall approach appears to be an incremental combination of existing techniques, and the core novelty is not sufficiently prominent.

Questions for authors

The authors need to more clearly define Joint World Action Modeling with VLA and provide more convincing evidence. Please also respond to other questions regarding the methodology and datasets. I would be very happy to raise the score if the rebuttal is reasonable.

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

We sincerely thank you for finding the paper clearly written, and for recognizing its strong clinical value, the practical challenge of generating volumetric MRI, the flexible handling of incomplete MRI modalities, and the relatively comprehensive experimental design.

[W1 & W5] The distinction from a VLA and the technical novelty

Thank you for your careful reading and for identifying the training–inference distinction. GBM-WAM is an early query-based exploration of world-action modeling for GBM treatment planning, connecting anatomy-guided future MRI representation learning with direct treatment planning. While WAMs have been extensively explored for embodied control, our work investigates this paradigm for partially observed, high-dimensional 3D MRI and limited longitudinal treatment records.

During training, future MRI latents and tumor masks supervise intermediate world and anatomical states. At inference, the action query retains access to these supervised states for direct treatment planning, while auxiliary decoding and flow integration are bypassed. Table C1 summarizes how this relates to VLA and WAM approaches.

Table C1. Comparison of action-prediction designs.

DesignFormulationStrengthTrade-off
Action-only VLAObservation + language → actionDirect predictionNo explicit future-state objective
Explicit-generation WAMObservation → future observation → actionPredictive world learningIterative generation at inference
GBM-WAMMask queries → world queries → actionPredictive learning + single-pass inferenceLongitudinal MRI + mask targets for training

An action-only VLA predicts directly from observation and language, giving an efficient decision path without a future-state objective. An explicit-generation WAM additionally models future observations, but its generation step adds inference cost. GBM-WAM combines these strengths: it learns from future MRI and anatomical targets during training, then uses the supervised query states for direct prediction. This matters for 3D MRI, where explicitly generating a future volume is expensive (27.7 s vs. 0.375 s per case in our latency measurements).

This training–inference split follows a direction that recent WAMs are also taking. Fast-WAM [1] and GigaWorld-Policy-0.5 [2] keep future visual supervision during training and omit future-video generation at inference. Both, however, start from a pretrained video world model. No comparable pretrained world model is available for volumetric MRI, where the data are 3D, multi-sequence, often incomplete, and longitudinally scarce. Bringing the WAM idea to GBM therefore requires building the world learning itself, and each component of GBM-WAM answers one obstacle in doing so.

First, generating a 3D volume is costly, so we predict the future in the latent space of an MRI-specific encoder (NeuroVFM [3]). Second, patients rarely have every sequence, so flow matching with sequence-specific MoT branches predicts the complete multi-sequence future from whichever sequences are available. Third, the tumor occupies a small part of the volume, so current-mask, future-mask, and change supervision direct the states toward the anatomy that matters. Finally, these supervised states are not discarded after training. Under the mask → world → action order, the action query reads them at inference. This is what distinguishes GBM-WAM from a VLA trained with auxiliary future-prediction losses, where such losses only shape the shared backbone. We tested this distinction directly. The parallel-query control in the following Table C2 is such a model: it keeps the same backbone and the same world and mask losses, but the action query cannot read the mask and world states. It reaches 88.5% F1 and 90.8% accuracy, versus 91.6% and 93.2% for GBM-WAM, so 3.1 F1 points come from the action query's access to the supervised states rather than from the auxiliary losses alone. We also note that GBM-WAM can still produce the next observation when needed: future MRI latents and masks can be decoded on demand (27.7 s per case), and only the default treatment planning path skips this step. Manuscript Table 3 shows that each step adds to treatment planning: F1 rises from 85.9% with direct latent regression to 87.8% with flow matching, 89.2% with MoT branches, and 91.6% with temporal mask supervision.

In the revised manuscript, we have added a new Appendix F, which presents this comparison (Table C1) and distinguishes the query states retained at inference from auxiliary decoding and flow integration.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

[W2] How anatomical guidance reaches the action query

Thank you for asking how anatomical guidance influences treatment planning. In GBM-WAM, anatomical guidance is provided by the temporal mask expert, which complements the latent world objective by supervising tumor anatomy at the current and follow-up visits. To answer your question directly: "anatomy-guided" refers to the segmentation and change losses applied through the mask queries, not to the NeuroVFM representations. The frozen NeuroVFM encoder supplies general 3D MRI features in every configuration, including the variant without mask supervision.

During training, the current and future mask query states modulate a shared decoder through adaptive group normalization, using current MRI features for both time points. The mask and change losses push these states to encode anatomical information. Because the queries follow the mask → world → action order, the world queries read the anatomy-supervised mask states, and the action query attends to both groups, so anatomical supervision informs treatment planning through the intermediate query states.

To test these connections, Table C2 compares the submitted order with two matched controls. Reversing the first two groups lets the mask queries read world states while preserving the action query's access to both groups. The parallel control gives each group the same MRI and clinical inputs while blocking information flow between groups.

Table C2. Query order and connectivity on the internal test set.

Query configurationAccuracyF1SpecificityPrecision
Mask → World → Action93.2%91.6%97.5%92.0%
World → Mask → Action92.0%90.1%96.9%90.7%
Parallel query groups90.8%88.5%96.2%89.2%

The two controls measure the direction of mask–world information flow and the contribution of connections between query groups, respectively. Reversing the order costs 1.5 F1 points. Removing inter-group connections costs 3.1 points, because the auxiliary losses still shape the shared backbone during training but the action query loses direct access to the mask and world states at inference.

Separately, with the world expert fixed, the mask ablations show that full temporal mask supervision improves internal accuracy from 91.3% to 93.2% and F1 from 89.2% to 91.6%. Together, the mask ablations and the query controls show both that anatomical supervision contributes and how its information reaches the action predictor.

In the revised manuscript, we have clarified in Section 3.4 how mask supervision shapes the query states and how those states are accessible to the action query, and added the query-order controls in Section 4.4.

[W3] Class support and tail performance

Thank you for requesting the treatment-category distribution and per-class performance. The distribution of each source cohort and of both test sets is reported in Appendix A of the original manuscript. For per-class performance, because Reviewer yXax [W3] asked for the same analysis, we answer both in Shared Response 2, which reports the row-normalized confusion matrices for all five categories, including the tail categories (Tables S3 and S4). These matrices have been added to Appendix D of the revised manuscript.

[W4] Evidence for localized disease progression

Thank you for highlighting the distinction between latent agreement and localized anatomical progression. We agree it is central to our progression claim. Because Reviewer yXax [W2] raised the same point, we answer both in Shared Response 1. It restricts the latent comparison to the tumor region and stratifies it by treatment category and follow-up interval (Tables S1 and S2).

In the revised manuscript, we have clarified the scope of the tumor-region analysis and its relationship to the progression claim in Appendix D.

[1] Yuan et al. Fast-WAM: Do World Action Models Need Test-time Future Imagination? 2026.

[2] GigaWorld Team et al. GigaWorld-Policy-0.5: A Faster and Stronger WAM Empowered by AutoResearch. arXiv:2607.13960, 2026.

[3] Kondepudi et al. Health system learning enables generalist neuroimaging models. 2026.

Dear Reviewer qotw,

Thank you again for your constructive suggestions!

With two days remaining in the discussion period, we would be grateful if you could review our updated response and share your feedback. We have clarified GBM-WAM’s distinction from a VLA with auxiliary losses and added query-connectivity controls to demonstrate how anatomical and world states contribute to treatment prediction. We have also highlighted the technical novelty and provided tumor-region evaluation and per-class confusion analyses.

Please let us know if any points need further clarification. Thank you for your time and consideration!

Best regards,
Authors of Paper 38881

Thank you for the authors’ response. I have considered adjusting my score.

Reviewer_yXaxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GBM-WAM predicts the next-time treatment for glioblastoma from MRI scans and clinical history. Future MRI features and tumor masks provide extra supervision during training, while treatment prediction runs without these extra modules at test time. The paper reports 93.2% internal accuracy and 81.9% external accuracy, with support for missing MRI sequences. However, key design choices need stronger support, and the experiments do not clearly establish how much prediction depends on patient-specific MRI and tumor changes. These gaps weaken the main claims, so I do not support acceptance.

Critical strengths
  1. The method supports all seven non-empty combinations of three MRI sequences. Its reported inference time of 0.375 seconds per case makes the approach practical for further testing.
  2. On the external cohort of 49 patients, accuracy improves from 78.3% for Brain-WM to 81.9%. Component ablations and patient-level confidence intervals provide further support for the reported gains.
Critical weaknesses
  1. The overall idea is clear, but the need for the mask–world–action query order is not fully explained. Table 3 does not compare parallel prediction heads or remove the change loss while keeping both current and future mask supervision. Why is this query order needed, and how much does the change loss add when both current and future masks are already supervised?

  2. Appendix D reports 94.74% similarity between predicted and true future features, while copying current features already gives about 92.7%. This score over the whole volume may mainly reflect unchanged brain anatomy. Scores within tumor regions, grouped by treatment and follow-up time, would better show whether the model learns useful tumor changes.

  3. Mask quality and treatment errors need further analysis. Appendix E reports Dice scores of 71.02% for current masks and 51.20% for future masks. The paper does not test sensitivity to noisy training masks or examine whether poor mask predictions are linked to treatment errors. A treatment confusion matrix and examples of failed predictions would help explain where the full model fails.

Questions for authors

See Weaknesses.

5 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

We sincerely thank you for highlighting GBM-WAM's support for all seven non-empty combinations of three MRI sequences, its 0.375 s per-case inference time, the improvement over Brain-WM on the external cohort (78.3% → 81.9% accuracy), and the supporting component ablations and patient-level confidence intervals.

Your summary also asks how much the prediction depends on patient-specific MRI. We tested this directly with inference-time input controls on the submitted checkpoint (Table A1 in our reply to Reviewer Gakt): withholding MRI lowers internal accuracy from 93.2% to 74.4%, and exchanging MRI across patients lowers it to 68.8%. Dependence on tumor changes is addressed under W2 and W3 below.

[W1] Query ordering and the isolated change loss

Thank you for asking us to separate the effects of query ordering and change supervision. The mask queries learn to represent tumor anatomy through supervision at the current and follow-up visits. Placing them first lets the world queries use localized anatomical cues when forming states for future MRI prediction. The action query then reads both groups in the same causal pass, combining anatomical and prognostic cues for treatment planning.

Table B1 compares the submitted order with two controls that keep the backbone, query counts, training objectives, and patient split fixed. Reversing the first two groups lets the mask queries read the world states while preserving the action query's access to both groups. Following your suggestion, the parallel control gives each group the same MRI and clinical inputs while blocking information flow between groups.

Table B1. Query order and connectivity on the internal test set.

Query configurationAccuracyF1SpecificityPrecision
Mask → World → Action93.2%91.6%97.5%92.0%
World → Mask → Action92.0%90.1%96.9%90.7%
Parallel query groups90.8%88.5%96.2%89.2%

Reversing the order reduced performance, consistent with the world queries losing access to the anatomy-supervised states even though the action query still reads both groups. The drop was larger with parallel queries: the auxiliary losses still shape the shared backbone during training, but the action query has no direct access to the mask and world states at inference. The two controls therefore isolate the effect of the mask–world direction and of inter-query connections, respectively.

The current and future mask losses provide complementary anatomical targets at the two visits. The change loss in Eq. 8 adds a cross-visit target by supervising which voxels change segmentation class: change is scored as half the L1 distance between the two predicted class-probability vectors and compared with the binary ground-truth change map using BCE and Dice. In Table B2, the final two rows retain both segmentation targets and differ only in the change-loss weight.

Table B2. Temporal mask supervision on the internal test set.

Mask supervisionAccuracyF1SpecificityPrecision
No mask supervision91.3%89.2%96.6%89.8%
Current mask only92.3%90.4%97.1%90.9%
Future mask only92.5%90.7%97.2%91.2%
Current + future masks, without change loss92.8%91.0%97.2%91.4%
Current + future masks, with change loss (Ours)93.2%91.6%97.5%92.0%

With the MoT world objective fixed, supervising either mask improved F1 over the 89.2% reference. Supervising both improved F1 over either single-mask variant, consistent with complementary anatomical information across visits.

Adding the change loss further increased F1 from 91.0% to 91.6% (accuracy from 92.8% to 93.2%) under the same training setup, with specificity and precision also improving (97.2% → 97.5% and 91.4% → 92.0%). We regard this as a modest gain and describe it as such in the revised manuscript. The role of Lchange\mathcal{L}_{\mathrm{change}} is to add a cross-visit target that the two per-visit segmentation losses do not provide: it supervises, voxel by voxel, whether the predicted class distribution changes between the current and follow-up masks (Eq. 8).

In the revised manuscript, we have added the change-loss ablation (manuscript Table 3) and the query-order and parallel-query controls in Section 4.4.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

[W2] Whole-volume and tumor-region latent fidelity

Thank you for suggesting tumor-region evaluation stratified by treatment and follow-up interval. Because Reviewer qotw raised the same concern [W4], we report the full analysis in Shared Response 1 (Localized Tumor Progression) above (Tables S1–S2). In brief, restricting the cosine similarity to the union of baseline and follow-up tumor masks lowers absolute agreement for both GBM-WAM and static copy but widens the gap from +2.04 to +7.09 points. The gain is largest for pairs followed by surgery (+15.10) and for follow-up intervals longer than 90 days (+9.05), i.e., where the tumor region changes most.

[W3] Mask quality and treatment errors

Thank you for asking how mask quality affects treatment planning. The temporal mask expert uses a shared lightweight decoder with 32.3M parameters, modulated by one current-state query and one future-state query. The mask and world objectives supervise the anatomical and prognostic states that the action query reads. At inference, these states are retained while the auxiliary decoders are bypassed.

Appendix E reports 71.02% current Dice and 51.20% future Dice. Forecasting future anatomy from baseline MRI under treatment-and-time conditioning is harder than segmenting the observed scan. Meanwhile, Table B2 shows that the full temporal mask objective improves treatment F1 from 89.2% to 91.6% on the fixed world-supervised configuration, so imperfect masks still provide useful supervision.

To test whether poorer mask predictions accompany treatment errors, we performed two complementary analyses. First, we compared mean current- and future-mask Dice between correctly and incorrectly predicted pairs (Table B3). Second, we ranked the same MRI pairs by current-mask Dice, divided them into four approximately equal groups, and reported mask quality and treatment performance within each group (Table B4). Each MRI pair contributes equally to the Dice averages. When target lesions are completely resected or absent at follow-up, Dice is evaluated as 100% if the predicted mask is also empty, and 0% otherwise.

Table B3. Mask Dice by treatment planning correctness (internal test set). Values are mean Dice (%).

Treatment planningCurrent DiceFuture Dice
Correct71.4952.13
Incorrect64.5438.36
Overall71.0251.20

Table B4. Treatment performance across current-mask Dice quartiles. Dice values are mean [IQR: 25th–75th percentile], in %.

Current Dice groupCurrent DiceFuture DiceAccuracyF1SpecificityPrecision
Q1, lowest58.74 [51.42–64.25]30.84 [16.25–46.70]86.5%81.6%95.6%80.3%
Q269.63 [67.81–71.36]47.56 [37.12–58.45]91.3%90.8%96.2%92.5%
Q375.28 [73.54–76.82]59.37 [50.83–69.24]96.1%95.5%98.8%95.3%
Q4, highest80.46 [78.65–83.21]67.07 [58.91–76.43]99.0%98.7%99.4%99.7%
Pooled71.02 [66.55–77.25]51.20 [35.62–68.15]93.2%91.6%97.5%92.0%

Incorrect treatment plans have lower mean Dice at both time points, with a larger gap for the future mask, and treatment accuracy rises steadily from the lowest to the highest Dice quartile. Together, the two analyses show an association between mask quality and treatment planning accuracy, reflecting shared anatomical complexity rather than a strict causal dependency.

The treatment confusion matrices are reported in Shared Response 2 (Tables S3 and S4). Patient-specific heterogeneity limits generalization to atypical disease trajectories. For example, UCSD-PTGBM-0066, a 62-year-old man with prior surgery followed by monitoring, whose recorded next treatment was a re-operation for recurrent tumor, was predicted as AM instead of SUR, which highlights the need to better recognize individual changes in disease course.

In the revised manuscript, we have added the error analysis and a representative failure example, and documented the decoder and the mask-quality analysis, in Appendix D.

Dear Reviewer yXax,

Thank you again for your constructive suggestions!

With two days remaining in the discussion period, we would be grateful if you could review our updated response and share your feedback. We have added query-order controls and an isolated change-loss ablation, together with tumor-region evaluation stratified by treatment and follow-up interval. We also examined the association between mask quality and treatment errors and provided confusion matrices.

Please let us know if any points need further clarification. Thank you for your time and consideration!

Best regards,
Authors of Paper 38881

Reviewer_GaktOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes GBM-WAM to predict the next treatment category for patients with glioblastoma using current brain MRI and clinical history, without requiring future-image generation at inference. The treatment predictor connects a frozen NeuroVFM MRI encoder to a Qwen-based multimodal model. A latent world expert learns to predict future MRI representations rather than reconstruct complete images, while a temporal mask expert learns to segment current and future tumors and capture changes in tumor extent. Experiments on internal and external cohorts across five treatment classes support its effectiveness for retrospective treatment prediction.

Critical strengths
  1. The paper is well motivated and clearly structured.
  2. The combination of latent world modeling and temporal mask supervision is a novel design for this treatment-prediction task.
Critical weaknesses
  1. The model predicts a treatment category without providing a clinical explanation or showing which image regions inform its decision, for example through a generated rationale or attribution maps. This makes it difficult to assess whether predictions rely on relevant MRI findings or shortcuts in the clinical text.
  2. The model's practical usefulness remains unclear given the narrow task setting: it uses a single MRI time point and limited clinical history to predict one of five treatment categories. This simplifies real treatment planning and leaves its value for longitudinal, individualized decisions untested.
Questions for authors
  1. Could the authors include a clinical-text-only baseline or shuffled-MRI control to show how much MRI contributes beyond prior treatment history?
  2. What voxel spacing and resampling settings are used after registration to SRI24 space? Are they consistent across MRI sequences, time points, and cohorts before cropping to 140 x 192 x 192?
  3. Beyond the reported RTX 4090 inference timings, could the authors share the GPU type and count used for training, total GPU-hours, and peak memory usage during training and inference? Does the reported latency include MRI encoding and preprocessing, and is it measured the same way for Brain-WM?
  4. Does the clinical history include treatment dates or elapsed time between prior interventions and the current MRI? If not, how does the model distinguish similar treatment histories at different stages of care?
4 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/3)

Thank you for finding the paper well motivated and clearly structured and for recognizing the combination of latent world modeling and temporal mask supervision as a novel design for this treatment prediction task. All of your weaknesses and questions are answered in this reply.


[W1 & Q1] On the contribution of MRI and the visual evidence

Thank you for suggesting these controls. The SNO–EANO consensus review describes surgery followed by concurrent chemoradiotherapy and then adjuvant temozolomide (SUR → CRT → TMZ) as a standard first-line pathway for eligible patients with newly diagnosed GBM [1]. Prior treatment history is clinically informative because it places the current visit in the course of care. However, patients with similar histories receive different next treatments depending on their age, functional status, and tumor findings on the current MRI. This motivates using both MRI and clinical context to predict the next recorded treatment.

To measure what each input contributes, we evaluated three inference-time controls using the same checkpoint, test pairs, and labels, without retraining. The first withholds all MRI. The second exchanges baseline MRI examinations across patients within the same site and sequence-availability group. The third withholds prior treatment history while retaining the matched MRI and demographics.

Table A1. Full-input reference and input controls (internal test set).

Test inputAccuracyF1SpecificityPrecision
Clinical text only74.4%67.2%90.9%67.8%
Clinical text + shuffled MRI68.8%61.7%89.6%63.3%
MRI + demographics, without prior treatment history61.4%52.3%86.5%54.8%
Matched MRI + full clinical prompt93.2%91.6%97.5%92.0%

We do not dispute that the clinical text is the strongest single input. Prior treatment history places the visit on the first-line pathway, and the next step often follows from it. The text-only control remains informative (67.2% F1, 74.4% accuracy), and withholding the treatment history causes the largest drop (to 52.3% F1, 61.4% accuracy). The role of MRI is to individualize the decision where the history alone is ambiguous.

Adding the matched MRI to the full clinical prompt raises F1 from 67.2% to 91.6% (accuracy from 74.4% to 93.2%), whereas adding another patient's MRI lowers it (61.7% F1, 68.8% accuracy). Because shuffling preserves MRI availability while breaking the patient–scan correspondence, this gap shows that the gain comes from patient-specific MRI information rather than from the mere presence of an image.

With the clinical prompt held fixed, performance also varies with the MRI sequences provided (manuscript Table 2): accuracy rises from 87.4% with T2 alone to 93.2% with all three sequences, and F1 from 84.6% to 91.6%, supporting the complementary value of the sequences.

Regarding which image regions inform the decision, we added a feature-level region-occlusion test. For each test pair, we replaced the latent spatial features corresponding to the baseline tumor mask with the mean brain feature representation. As a control, we occluded the features of a size-matched region mirrored to the contralateral hemisphere.

Table A2. Region occlusion at inference (internal test set).

Occluded regionAccuracyF1SpecificityPrecision
Tumor region75.8%68.8%91.5%69.6%
Size-matched contralateral region92.5%90.8%97.3%91.3%
None (full input)93.2%91.6%97.5%92.0%

Occluding the tumor features lowers F1 from 91.6% to 68.8% (−22.8 points), whereas occluding a size-matched contralateral region changes it only to 90.8% (92.5% accuracy). With the tumor features occluded, performance falls close to the level of the text-only control (67.2% F1, 74.4% accuracy; Table A1), demonstrating that occluding the tumor region removes almost all of the MRI contribution. The MRI contribution therefore comes primarily from the tumor region of the patient’s own scan rather than from background brain features. Together with Table A1, this indicates that the clinical text places the visit on the treatment pathway, while the tumor region of the matched MRI accounts for most of the remaining gain. GBM-WAM does not generate a textual rationale, but the temporal mask expert can be decoded at inference to show the tumor anatomy encoded in the query states that the action query reads, as visualized in 3D in Figure 3 of the manuscript.

In the revised manuscript, we have added the inference-time controls and the region-occlusion test in Appendix D.

AuthorsOpenReview ↗

Rebuttal by Authors (2/3)

[W2 & Q4] On task simplification and prior treatment timing

Thank you for raising the scope of the task and the role of treatment timing. Despite extensive efforts to harmonize available longitudinal GBM imaging and treatment records, the three internal cohorts yielded 527 patients and 1,659 visits, with each MRI sequence represented by a 140 × 192 × 192 volume (about 5.16 million voxels). Given the cohort size and high-dimensional inputs, we froze the state-of-the-art pretrained MRI encoder (NeuroVFM), leveraged the vision-language prior of Qwen3.5, and fine-tuned it with LoRA to limit overfitting. Dense pairing provided more supervision per patient across follow-up intervals while keeping training and test patients disjoint. You are right that the model takes a single MRI visit as input: earlier visits enter only through the recorded treatment history, and longitudinal imaging is used at training time, where the follow-up MRI and masks supervise the world and mask queries. This keeps the input fixed across cohorts with very different follow-up coverage, but it means that multi-visit imaging input and individualized longitudinal decisions are untested here, which we state in Appendix E. We agree that modeling individualized treatment trajectories is an important next step. We view this study as an initial evaluation of world-action modeling in neuro-oncology and plan to extend it with richer longitudinal records and outcome data.

The submitted model receives the sequence of prior treatment categories without dates or elapsed times. When that sequence is identical for two patients, the current MRI carries each patient's disease state at the decision visit and demographics provide patient-specific context, giving GBM-WAM a basis for different next-treatment plans under identical recorded histories.

Following your comment, we added the number of days from each prior treatment event to the current MRI to the clinical prompt, retaining repeated treatments as separate entries, and retrained the model with the same architecture and patient splits. Adding timing information improved performance on the internal test set (Table A3), showing that elapsed times complement the recorded treatment sequence. We thank the reviewer for highlighting a clinically important aspect of treatment history that the original prompt did not capture.

Table A3. Effect of per-event elapsed time in the clinical prompt (internal test set).

Prior treatment inputAccuracyF1SpecificityPrecision
Treatment history without elapsed times93.2%91.6%97.5%92.0%
Treatment history with elapsed times94.2%92.2%97.8%92.7%

In the revised manuscript, we have distinguished elapsed time since prior treatments from the baseline-to-follow-up interval used in auxiliary training and added Table A3 in Appendix D. The scope of the five-category prediction task is discussed in Appendices B and E.


[Q2] On voxel spacing, registration, and resampling

Thank you for asking us to clarify the spatial preprocessing. Before cropping, all MRI sequences, time points, and cohorts shared the SRI24 reference grid with 1.0 × 1.0 × 1.0 mm isotropic voxel spacing. We selected SRI24 because the public MU-Glioma Post and UCSF-ALPTDG releases already use this space, and used ANTs to register scans that required atlas alignment, resampling images with trilinear interpolation and tumor masks with nearest-neighbor interpolation.

After alignment, we center-cropped the volumes to 140 × 192 × 192 voxels to remove peripheral background and standardize the input size. The crop preserves voxel spacing. All imaging-based comparisons used the same spatial preprocessing before model-specific input preparation.

In the revised manuscript, we have stated the reference grid, voxel spacing, and cropping in Section 4.1 and Appendix C.

AuthorsOpenReview ↗

Rebuttal by Authors (3/3)

[Q3] Training compute, peak GPU memory, and inference latency

Thank you for asking about GBM-WAM's compute requirements and how we measured latency. Training took approximately 75 hours on two NVIDIA RTX 4090 GPUs (about 150 GPU-hours), with a peak of about 19.0 GiB per GPU. At inference, direct treatment planning peaks at about 6.5 GiB of GPU memory on a single GPU.

We benchmarked at batch size 1 on a single RTX 4090 and report the mean over five cases. Each measurement covers the forward pass for that execution mode, including 3D MRI encoding, on volumes already loaded in memory. To answer directly: the reported latency includes center-cropping and MRI encoding and excludes offline preprocessing. Offline preprocessing (skull-stripping and registration to SRI24) is performed once per scan before inference and is shared by both models, so it is outside the timed window. The measurement also excludes checkpoint loading, reading the .nii.gz files from disk, metric computation, and result serialization. Brain-WM was timed under the same protocol. Both direct treatment planning modes bypass future-state generation.

Table A4. Per-case latency for direct treatment planning and optional decoding.

Model and execution modeLatency per case
GBM-WAM · Direct treatment planning0.375 s
GBM-WAM · With temporal mask decoding0.563 s
GBM-WAM · With 50-step Heun latent generation27.69 s
Brain-WM · Direct treatment planning2.580 s

In the revised manuscript, we have stated the benchmarking setup in Section 4.5 and added the timing window, training configuration, total GPU-hours, and peak memory during training and inference in Appendix C.

[1] Wen et al. Glioblastoma in adults: a Society for Neuro-Oncology (SNO) and European Society of Neuro-Oncology (EANO) consensus review. Neuro-Oncology, 2020.

AuthorsOpenReview ↗

We are looking forward to your feedback

Dear Reviewer Gakt,

Thank you again for your constructive suggestions, which helped us improve the quality and clarity of our paper.

Before the author–reviewer discussion period closes, we would appreciate any feedback on our follow-up response, particularly on the MRI input controls and tumor-region occlusion experiments.

Please let us know if any points would benefit from further clarification. If our response addresses your main concerns, we would be grateful if you would consider raising your score.

Thank you for your time and consideration.

Best regards, Authors of Paper 38881

Comments on the paper

3 comments
AuthorsOpenReview ↗

Shared Response 2 on Per-Class Confusion Analysis

This comment is a shared response to Reviewer yXax [W3] and Reviewer qotw [W3], who both asked where the treatment errors of GBM-WAM occur. Reviewer yXax requested a treatment confusion matrix to explain where the full model fails, and Reviewer qotw requested per-class performance, especially on the tail categories. We thank both reviewers for these requests. Below, we describe how the internal and external test sets differ, report the row-normalized confusion matrices for both, and discuss the error patterns.

The internal and external cohorts differ markedly. Our internal data were split 80:20 at the patient level to prevent leakage, and the real-world records we collected are inherently class-imbalanced. The internal set is dominated by active monitoring (AM), with radiotherapy (RT) the smallest class. The external set is more balanced across surgery (SUR), concurrent chemoradiotherapy (CRT), and AM, and contains many surgeries for recurrent tumor. RT and temozolomide (TMZ) together make up only about one tenth of the external pairs. These two external classes are therefore the minority classes whose metrics are least stable.

Table S3. Internal confusion matrix. Rows are normalized by true class.

True \ predictedSURCRTRTTMZAM
SUR87.1%0.0%0.0%3.2%9.7%
CRT0.0%93.5%0.0%0.0%6.5%
RT0.0%0.0%90.0%0.0%10.0%
TMZ1.6%0.0%0.0%92.1%6.3%
AM1.1%0.7%0.0%3.9%94.3%

Table S4. External confusion matrix. Rows are normalized by true class.

True \ predictedSURCRTRTTMZAM
SUR71.4%10.7%0.0%3.6%14.3%
CRT4.8%90.5%0.0%0.0%4.8%
RT0.0%25.0%75.0%0.0%0.0%
TMZ0.0%20.0%20.0%60.0%0.0%
AM4.0%4.0%0.0%0.0%92.0%

Internally, more than 87% of the pairs in every category are predicted correctly. Errors on the four active-treatment classes flow mostly into AM, the majority class, while AM itself is most often confused with TMZ (3.9%). Externally, CRT and AM remain above 90%, and three patterns emerge. (i) SUR recall drops to 71.4%: many external surgeries are re-operations for recurrent tumor, which are rare in our internal data (see the failure example in our reply to Reviewer yXax). (ii) RT and TMZ, the two minority classes, have the lowest values (75.0% and 60.0%). RT errors are all predicted as CRT, a closely related treatment that also delivers radiotherapy, whereas TMZ errors fall on CRT or RT, which differ more from TMZ, a chemotherapy-only treatment. (iii) Confusions between active treatment and AM are the most consequential and reflect the strong heterogeneity of patient disease courses across cohorts. In future work, we will focus on these cases with more diverse longitudinal data and a training design that accounts for the clinical cost of each error.

In the revised manuscript, we have added row-normalized confusion matrices for both test sets and an example of a treatment planning error in Appendix D.

AuthorsOpenReview ↗

Shared Response 1 on Localized Tumor Progression

This comment is a shared response to Reviewer yXax [W2] and Reviewer qotw [W4], who both asked whether our latent-space evidence reflects change inside the tumor rather than unchanged brain anatomy. We thank both reviewers and agree with this concern. Below, we restate the original whole-volume metric and then report a new tumor-region analysis stratified by treatment category and follow-up interval.

Our original metric (Appendix D) compares predicted follow-up latents with a persistence baseline that copies the baseline features, over the whole volume and three MRI sequences:

Table S1. Whole-volume cosine similarity to the observed follow-up latents (internal test set).

MethodT2T1CEFLAIRMean
Copy baseline latents92.94%92.74%92.43%92.70%
GBM-WAM95.40%94.47%94.36%94.74%

Because stable anatomy dominates the flattened features, a high whole-volume score can conceal larger errors inside the tumor. We therefore mapped the union of the baseline and follow-up tumor masks onto the latent spatial grid (35×12×1235 \times 12 \times 12) and restricted the comparison to those features, applying the same ROI to GBM-WAM and the static copy on the same pairs and sequences. We then stratified the results by the next recorded treatment category and by follow-up interval.

Table S2. Whole-volume and tumor-region latent cosine similarity on the internal test set. Δ\Delta is the gain over static copy in percentage points.

MethodWhole volumeROI overallSURCRTRTTMZAM≤90 days>90 days
Static copy92.70%73.87%52.01%62.41%65.91%73.41%77.96%78.86%67.51%
GBM-WAM94.74%80.96%67.11%74.71%76.31%80.91%83.37%84.41%76.56%
Δ+2.04+7.09+15.10+12.30+10.40+7.50+5.41+5.55+9.05

Three observations follow. First, restricting the comparison to the tumor region lowers absolute similarity for both methods but widens the gap between them, from +2.04 to +7.09 points, so the whole-volume score understated the improvement where the anatomy actually changes. Second, the gain is largest where tumor-region change is expected to be largest: pairs followed by surgery (SUR) have the lowest static-copy similarity and the largest improvement (+15.10), followed by CRT (+12.30), whereas active monitoring (AM) and TMZ pairs have higher similarity and smaller gains (+5.41 and +7.50). RT pairs fall between CRT and TMZ in both static-copy similarity (65.91%) and gain (+10.40), following the same ordering, although the estimate for this small group is less stable. Third, longer follow-up intervals (>90 days) show lower similarity for both methods but a larger gap (+9.05 vs. +5.55), so the improvement over persistence persists at longer prediction horizons.

We note the scope of this evidence. Our primary task is treatment planning, a discriminative task. GBM-WAM is not a model dedicated to tumor segmentation or tumor-growth forecasting. The temporal mask expert (a lightweight shared decoder with 32.3M parameters) is an auxiliary module that guides the query states toward treatment-relevant anatomy. Our aim is that these anatomical states, and the feature changes they capture between time points, benefit downstream treatment planning.

In the revised manuscript, we have described the current metric as whole-volume latent agreement, added Table S2, and clarified its relationship to the progression claim in Appendix D.

AuthorsOpenReview ↗

Summary of Core Contributions and New Experiments

We sincerely thank the area chairs and all reviewers for their time, constructive feedback, and recognition of our work. We are encouraged that the reviewers recognized our "well motivated" problem with "strong clinical value" (Reviewers Gakt, qotw), the "novel design" combining latent world modeling and temporal mask supervision (Reviewer Gakt), the flexible handling of incomplete MRI inputs across "all seven non-empty combinations of three MRI sequences" (Reviewers yXax, qotw), the 0.375 s per-case inference that "makes the approach practical for further testing" (Reviewer yXax), the "relatively comprehensive" experiments "with competitive results" on internal and external cohorts (Reviewers yXax, qotw), and the "clearly structured" and "well-written" presentation (Reviewers Gakt, qotw). Below, we summarize our core contributions and technical novelties:

  1. We present an early exploration of world-action modeling for GBM treatment planning. Unlike embodied settings with abundant video, our setting involves partially observed, high-dimensional 3D MRI and limited longitudinal treatment records. GBM-WAM predicts the next recorded treatment from any non-empty subset of T2, T1CE, and FLAIR together with the clinical context.
  2. We propose a latent world expert for partial-to-complete future MRI prediction. Flow matching with sequence-specific MoT branches supervises complete multi-sequence follow-up latents from whichever baseline sequences are available, so the model learns disease evolution without generating volumes in pixel space.
  3. We propose a temporal mask expert that supervises tumor anatomy at the current and follow-up visits through a shared lightweight decoder, together with a temporal change loss that explicitly supervises which voxels change segmentation class between visits.
  4. We design an ordered query interface (mask → world → action) that connects both forms of supervision to the same action predictor. World queries read the anatomy-supervised states and the action query reads both groups in one causal pass, so the auxiliary decoders and flow integration are bypassed at inference (0.375 s per case, versus 27.7 s with explicit future-state generation).
  5. Our experiments show that GBM-WAM reaches 93.2% accuracy and 91.6% F1 on the internal test set, remains robust across all seven MRI subsets, and outperforms Brain-WM on the external cohort (81.9% vs. 78.3% accuracy), with ablations supporting each component of the joint training objective.

Additionally, we have conducted new experiments and analyses during the rebuttal, which have been added to the revised manuscript:

  1. New analysis of training cost, GPU memory, and latency: rebuttal Table A4 → manuscript Section 4.5, Table 5, and Appendix C (suggested by Reviewer Gakt).
  2. New query-order controls and change-loss ablation: rebuttal Tables B1/C2 and B2 → manuscript Section 4.4, Table 4 and Table 3 (Idx. 4–8) (suggested by Reviewers yXax, qotw).
  3. New input controls (MRI withheld, MRI shuffled, treatment history withheld): rebuttal Table A1 → manuscript Appendix D, Table S3 (suggested by Reviewer Gakt).
  4. New experiment adding elapsed time since prior treatments: rebuttal Table A3 → manuscript Appendix D, Table S5 (suggested by Reviewer Gakt).
  5. New tumor-region latent fidelity analysis: rebuttal Table S2 → manuscript Appendix D, Table S2 (suggested by Reviewers yXax, qotw).
  6. New analysis of mask quality versus treatment errors: rebuttal Tables B3 and B4 → manuscript Appendix D, Tables S6 and S7 (suggested by Reviewer yXax).
  7. New confusion-matrix analysis: rebuttal Tables S3 and S4 → manuscript Appendix D, Table S4(a) and Table S4(b) (suggested by Reviewers yXax, qotw).
  8. New region-occlusion test of which image regions inform the prediction: rebuttal Table A2 → manuscript Appendix D, Table S8 (suggested by Reviewer Gakt).
  9. New comparison of action-prediction designs (VLA, explicit-generation WAM, and GBM-WAM): rebuttal Table C1 → manuscript Appendix F, Table S9 (suggested by Reviewer qotw).

All tables above are given in this rebuttal. Tables S1–S4 are in Shared Responses 1 and 2, which answer the two questions raised by both Reviewers yXax and qotw. Tables A1–A4, B1–B4, and C1–C2 are in our replies to Reviewers Gakt, yXax, and qotw, respectively.

We sincerely thank the area chairs and reviewers again for their invaluable time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.