- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies the relationship between the model's predictive capability of the node-level signals (forecasting held-out trajectories) and its capability of predicting post-operative seizure outcome. In a simulated setup, the author proposed an ODE-based Epileptic-brain World to predict node-level signals and demonstrated that it performs better than simple baselines. Later, the author demonstrated that in such a simulated setup, the model with good MSE on the test set cannot outperform a model just predicting surgical outcome.
Critical strengths
Predicting surgical outcome is always an important question to answer in seizure treatment, and it is always very hard to solve.
Critical weaknesses
-
The author introduced many "fancy" terms that make the paper very hard to follow. I strongly suggest the author revise the paper's tone and directly present the task, model performance, and evaluation instead of packing them into forecasting gate, certification, intervention competence, and no-rollout floor, which makes human decode such terms many times in reading the paper.
-
Besides the fancy terminologies, the paper does not establish a sufficiently novel or significant contribution. The simulation using line-length statistics to discriminate intervention outcomes, and use MSE as a training objective (L286–290). The main finding largely restates known objective mismatch, while the additional analysis are within a narrowly constructed synthetic task. The experimentation does not make clear what generalizable insight is gained from such a highly simulated task.
-
in L143 the author claimed “Machine learning for epilepsy has concentrated on localization (Li et al., 2021) and self-supervised iEEG/EEG models (Wang et al., 2023; Jiang et al., 2024); neither predicts outcomes under a specified intervention.” However, there are many literatures combine electrophysiological biomarker localization with resection coverage to predict postoperative seizure outcomes, including [1,2,3].
[1]Fedele, Tommaso, et al. "Resection of high frequency oscillations predicts seizure outcome in the individual patient." Scientific Reports 7.1 (2017): 13836.
[2]Kuroda, Naoto, et al. "Objective interictal electrophysiology biomarkers optimize prediction of epilepsy surgery outcome." Brain communications 3.2 (2021): fcab042.
[3]Matarrese, Margherita AG, et al. "Spike propagation mapping reveals effective connectivity and predicts surgical outcome in epilepsy." Brain 146.9 (2023): 3898-3912.
Questions for authors
My main concern is about the novelty and significance of the contribution, as detailed in the weaknesses above.
5 comments
Response to Reviewer pWU7 (W2)
W2. What the experiments reveal, and why we use simulation
2. Reply to W2 — What is new, and what can we learn from a simple simulated task? The simulation gives us both possible surgical outcomes and the patient state that determines them. That lets us ask a more precise question than whether forecasting and surgery prediction differ: did the model fail to learn the useful information, or did it fail to use that information when scoring the surgeries? Our experiments show that the information can remain in a forecast-trained encoder even when the final score fails. Improving line-length fidelity alone does not fix that score. Three findings support this.
(i) Similar forecasts can lead to very different surgery predictions [Submitted]. We trained EWM ten times on the same 16-node data, with the same model and training objective but different random seeds. Forecasting skill stays between 0.651 and 0.686, yet intervention AUROC ranges from 0.335 to 0.751 (Spearman ; Sec. 4.1). Thus, a forecasting score does not tell us which trained model will predict the surgical outcomes well.
(ii) The model may have useful information but use it incorrectly [Submitted + New]. TGR-WM is a second model trained to forecast neural signals. Its encoder—the part that processes the recorded signals—estimates each node's excitability, denoted . In a new diagnostic (Sec. 4.3; Appendix H.3), we read that estimate at the relay node known to determine the surgical outcome. Using the simulator's true state to set the direction of each model's score, all six trained models achieve 0.94–0.98 AUROC; three require the direction to be reversed. The direction matters because a larger raw score can otherwise mean either greater or lower risk.
The submitted EWM experiments show a related problem: models trained with different seeds can assign opposite score directions to the same relevant signal. Choosing each direction using validation labels and putting the members' scores on a common scale raises EWM from 0.519/0.509 to 0.704/0.710 AUROC (Table 18).
We also tested a small change to the prediction head on TGR-WM's frozen encoder [New]. The submitted head averages node features; the new head uses their mean and maximum. The encoder itself was not retrained. With 25 labelled patients, the new head performs much better than the original frozen-encoder head (Rebuttal Table R3-2). It also correctly orders the two surgeries for 91/95 and 89/91 patients whose outcomes differ. This is direct evidence that useful information remains in the forecasting features.
Rebuttal Table R3-2. A small head on frozen forecasting features (paper Tables 15 and 27; Sec. 4.2). AUROC after training on 25 virtual patients (50 outcome labels).
| Prediction head | Parameters | 12-node AUROC | 16-node AUROC |
|---|---|---|---|
| Frozen TGR-WM encoder + mean head (submitted) | 33 | 0.557 | 0.530 |
| Frozen TGR-WM encoder + mean+max head [New] | 65 | 0.838 | 0.888 |
| End-to-end Temporal head (submitted) | — | 0.638 | 0.573 |
Response to Reviewer pWU7 (W2(iii) and W3)
(iii) Restoring signal variation does not necessarily improve surgery prediction [New]. The intervention score uses line length, the average absolute change in a signal between consecutive time points. You note that outcomes are scored by line length while training uses MSE (L286–290). We therefore tested whether restoring line length in the forecasts improves surgery prediction, by training 12-node TGR-WM with an additional loss designed to preserve line length:
Rebuttal Table R3-3. Line-length loss on 12-node TGR-WM (Sec. 4.3; Appendix H.5).
| Training / scoring | Predicted/true line-length ratio | Intervention AUROC |
|---|---|---|
| Submitted loss, original scoring | 0.010 | 0.793 |
| + Line-length loss, original scoring | 0.851 | 0.502 |
| + Line-length loss, scoring the known surviving relay | 0.851 | 0.886 |
The ratios are measured on intact test trajectories. The last row uses the simulator's knowledge of the relevant node.
With the original score, line-length training reduces intervention AUROC by −0.291 [−0.350, −0.230]. Scoring only the simulator-known relay instead raises it by +0.384 [+0.322, +0.446] (paired 95% CIs). Thus, even when predicted signal variation improves, the node used for scoring still matters. Paper Table 30 reports all four model–setting comparisons.
What we learn. A forecasting test alone cannot tell whether the model has learned the information needed for an intervention or whether its final score uses that information correctly. Our results show why both should be checked: a forecast-trained encoder can preserve useful state information, while the surgery ranking still fails. This gives a practical way to investigate failures of forecast-trained world models beyond this simulated task.
Why simulate the task? For a real patient, we observe the outcome of the surgery actually performed, not the outcome of an alternative surgery on that same patient. Simulation gives us both possible outcomes and the true state behind them, so we can test the model's predictions and investigate errors separately. Here, the true surviving-relay state predicts the outcome almost perfectly (0.998/0.999 AUROC; Figure 4). Both candidate resections leave the same edited network, so this benchmark tests whether models recover and use that state; it does not compare different seizure-propagation paths. Sec. 4 (paragraph "Benchmark, and what it measures") states this scope and now also states that both resections leave the same edited network; Appendix H.4 adds a rollout-graph comparison.
W3. Prior work on predicting epilepsy surgery outcomes
3. Reply to W3 — The Related Work statement (L143 of the submission) overlooks existing outcome-prediction studies. The wording was too broad. Sec. 2 (paragraph "Epilepsy intervention modelling") no longer says that this literature does not predict outcomes, and now cites the three studies you list (blue in the revised PDF):
Clinical studies have predicted postoperative seizure outcomes using electrophysiological biomarkers and information about resected tissue (Fedele et al., 2017; Kuroda et al., 2021; Matarrese et al., 2023).
Our question is one that records of performed surgeries cannot answer: whether a model validated only by forecasting can compare two candidate surgeries for the same patient. That requires both outcomes and the state that governs them, which the simulator provides (W2).
Response to Reviewer pWU7 (W1)
Thank you for the review and for identifying the missing clinical studies. The revised PDF replaces the four terms you list with plain wording throughout (W1) and corrects the Related Work statement (W3).
Our question is simple: if a model learns to predict future neural signals, can we trust it to predict the outcomes of two possible surgeries? We train world models on simulated neural recordings, without showing them surgical outcomes. We then use their predictions to score two candidate resections for each virtual patient, whose outcomes are known from the simulator. EWM passes the forecasting test but predicts surgical outcomes poorly (0.519/0.509 AUROC at 12/16 nodes, versus 0.743/0.701 for the direct Static predictor). We also examine why forecasting success does not translate into accurate surgery predictions.
[New] marks analyses performed after submission. Paired values are 12-node / 16-node; new difference intervals are 95% patient-clustered paired bootstrap CIs. Rebuttal tables are numbered R3-1 to R3-3; "paper Table" refers to the revised PDF.
W1. Clearer writing and terminology
1. Reply to W1 — The paper uses too many unfamiliar terms. The original wording introduced several terms before explaining them clearly. We have replaced them with direct language in the main text, appendix, and Figures 1 and 2 (revised text marked in blue):
Rebuttal Table R3-1. Terminology replacements in the revised manuscript.
The two terms that remain in the title are defined in the first paragraph of the Introduction, and "orientation bit" is explained where it is introduced (Introduction, fourth paragraph). The Introduction's third paragraph now opens with the research question, and the transition at the start of Sec. 4.4 is replaced by a plain description of the comparisons.