acceptodds
Under review as a conference paper at ICLR 2027

FORECASTING SKILL ALONE DOES NOT CERTIFY INTERVENTION COMPETENCE IN WORLD MODELS

Abstract

World models are validated by forecasting held-out trajectories, yet they are also used to predict outcomes after actions that change the system. Held-out forecast error does not directly measure whether a model can distinguish outcomes under candidate actions, although it is often treated as evidence of intervention competence. We introduce forecast-gated intervention evaluation: a world model first demonstrates held-out forecast skill and is then tested on interventions. We ask whether rolling its learned dynamics forward adds intervention value beyond label-supervised predictors that use the same observation and intervention but never roll dynamics forward. The benchmark is two simulated Epileptor cohorts (12 and 16 nodes) whose resection outcome is decided by one latent variable, a deliberately favorable case for certification. An ODE-based Epileptic-brain World Model (EWM) clears the gate but trails the no-rollout baseline in both settings: AUROC is 0.519 vs. 0.743 at 12 nodes and 0.509 vs. 0.701 at 16 nodes. Ten further rollout models pass the gate. At the primary label budget, no evaluated rollout clears the prespecified margin over the label-supervised no-rollout head built on TGR-WM's temporal encoder. Swapping the EWM and TGR-WM encoders reverses their intervention-performance ordering, suggesting that intervention transfer is strongly encoder-dependent in this benchmark. Within each EWM member, the decoded signal tracks the intervention-relevant state, but independently trained members exhibit different orientations in that association, so averaging cancels it. A validation-derived orientation bit per member brings EWM only level with the floor. World models intended for interventions should be evaluated on interventions, against strong no-rollout baselines.

Primary area:
learning on time series and dynamical systems
Keywords:
world models, intervention prediction, evaluation protocol, benchmark, model-based reinforcement learning, representation learning, dynamical systems

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pWU7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the relationship between the model's predictive capability of the node-level signals (forecasting held-out trajectories) and its capability of predicting post-operative seizure outcome. In a simulated setup, the author proposed an ODE-based Epileptic-brain World to predict node-level signals and demonstrated that it performs better than simple baselines. Later, the author demonstrated that in such a simulated setup, the model with good MSE on the test set cannot outperform a model just predicting surgical outcome.

Critical strengths

Predicting surgical outcome is always an important question to answer in seizure treatment, and it is always very hard to solve.

Critical weaknesses
  1. The author introduced many "fancy" terms that make the paper very hard to follow. I strongly suggest the author revise the paper's tone and directly present the task, model performance, and evaluation instead of packing them into forecasting gate, certification, intervention competence, and no-rollout floor, which makes human decode such terms many times in reading the paper.

  2. Besides the fancy terminologies, the paper does not establish a sufficiently novel or significant contribution. The simulation using line-length statistics to discriminate intervention outcomes, and use MSE as a training objective (L286–290). The main finding largely restates known objective mismatch, while the additional analysis are within a narrowly constructed synthetic task. The experimentation does not make clear what generalizable insight is gained from such a highly simulated task.

  3. in L143 the author claimed “Machine learning for epilepsy has concentrated on localization (Li et al., 2021) and self-supervised iEEG/EEG models (Wang et al., 2023; Jiang et al., 2024); neither predicts outcomes under a specified intervention.” However, there are many literatures combine electrophysiological biomarker localization with resection coverage to predict postoperative seizure outcomes, including [1,2,3].

[1]Fedele, Tommaso, et al. "Resection of high frequency oscillations predicts seizure outcome in the individual patient." Scientific Reports 7.1 (2017): 13836.

[2]Kuroda, Naoto, et al. "Objective interictal electrophysiology biomarkers optimize prediction of epilepsy surgery outcome." Brain communications 3.2 (2021): fcab042.

[3]Matarrese, Margherita AG, et al. "Spike propagation mapping reveals effective connectivity and predicts surgical outcome in epilepsy." Brain 146.9 (2023): 3898-3912.

Questions for authors

My main concern is about the novelty and significance of the contribution, as detailed in the weaknesses above.

5 comments
AuthorsOpenReview ↗

Response to Reviewer pWU7 (W1)

Thank you for the review and for identifying the missing clinical studies. The revised PDF replaces the four terms you list with plain wording throughout (W1) and corrects the Related Work statement (W3).

Our question is simple: if a model learns to predict future neural signals, can we trust it to predict the outcomes of two possible surgeries? We train world models on simulated neural recordings, without showing them surgical outcomes. We then use their predictions to score two candidate resections for each virtual patient, whose outcomes are known from the simulator. EWM passes the forecasting test but predicts surgical outcomes poorly (0.519/0.509 AUROC at 12/16 nodes, versus 0.743/0.701 for the direct Static predictor). We also examine why forecasting success does not translate into accurate surgery predictions.

[New] marks analyses performed after submission. Paired values are 12-node / 16-node; new difference intervals are 95% patient-clustered paired bootstrap CIs. Rebuttal tables are numbered R3-1 to R3-3; "paper Table" refers to the revised PDF.

W1. Clearer writing and terminology

1. Reply to W1 — The paper uses too many unfamiliar terms. The original wording introduced several terms before explaining them clearly. We have replaced them with direct language in the main text, appendix, and Figures 1 and 2 (revised text marked in blue):

Rebuttal Table R3-1. Terminology replacements in the revised manuscript.

Term in the submissionWording now used
Forecasting gate; "clears the gate"Forecasting test; "passes the forecasting test"
Forecast-gated intervention evaluationForecast-first intervention evaluation
No-rollout floorNo-rollout baseline (the Static predictor)
Certify / certificationGuarantee; "a favorable case for forecasting to suffice"
Intervention competenceAbility to predict intervention outcomes; accurate intervention prediction

The two terms that remain in the title are defined in the first paragraph of the Introduction, and "orientation bit" is explained where it is introduced (Introduction, fourth paragraph). The Introduction's third paragraph now opens with the research question, and the transition at the start of Sec. 4.4 is replaced by a plain description of the comparisons.

AuthorsOpenReview ↗

Response to Reviewer pWU7 (W2)

W2. What the experiments reveal, and why we use simulation

2. Reply to W2 — What is new, and what can we learn from a simple simulated task? The simulation gives us both possible surgical outcomes and the patient state that determines them. That lets us ask a more precise question than whether forecasting and surgery prediction differ: did the model fail to learn the useful information, or did it fail to use that information when scoring the surgeries? Our experiments show that the information can remain in a forecast-trained encoder even when the final score fails. Improving line-length fidelity alone does not fix that score. Three findings support this.

(i) Similar forecasts can lead to very different surgery predictions [Submitted]. We trained EWM ten times on the same 16-node data, with the same model and training objective but different random seeds. Forecasting skill stays between 0.651 and 0.686, yet intervention AUROC ranges from 0.335 to 0.751 (Spearman ρ=−0.01\rho=-0.01; Sec. 4.1). Thus, a forecasting score does not tell us which trained model will predict the surgical outcomes well.

(ii) The model may have useful information but use it incorrectly [Submitted + New]. TGR-WM is a second model trained to forecast neural signals. Its encoder—the part that processes the recorded signals—estimates each node's excitability, denoted x^0\hat{x}_0. In a new diagnostic (Sec. 4.3; Appendix H.3), we read that estimate at the relay node known to determine the surgical outcome. Using the simulator's true state to set the direction of each model's score, all six trained models achieve 0.94–0.98 AUROC; three require the direction to be reversed. The direction matters because a larger raw score can otherwise mean either greater or lower risk.

The submitted EWM experiments show a related problem: models trained with different seeds can assign opposite score directions to the same relevant signal. Choosing each direction using validation labels and putting the members' scores on a common scale raises EWM from 0.519/0.509 to 0.704/0.710 AUROC (Table 18).

We also tested a small change to the prediction head on TGR-WM's frozen encoder [New]. The submitted head averages node features; the new head uses their mean and maximum. The encoder itself was not retrained. With 25 labelled patients, the new head performs much better than the original frozen-encoder head (Rebuttal Table R3-2). It also correctly orders the two surgeries for 91/95 and 89/91 patients whose outcomes differ. This is direct evidence that useful information remains in the forecasting features.

Rebuttal Table R3-2. A small head on frozen forecasting features (paper Tables 15 and 27; Sec. 4.2). AUROC after training on 25 virtual patients (50 outcome labels).

Prediction headParameters12-node AUROC16-node AUROC
Frozen TGR-WM encoder + mean head (submitted)330.5570.530
Frozen TGR-WM encoder + mean+max head [New]650.8380.888
End-to-end Temporal head (submitted)—0.6380.573
AuthorsOpenReview ↗

Response to Reviewer pWU7 (W2(iii) and W3)

(iii) Restoring signal variation does not necessarily improve surgery prediction [New]. The intervention score uses line length, the average absolute change in a signal between consecutive time points. You note that outcomes are scored by line length while training uses MSE (L286–290). We therefore tested whether restoring line length in the forecasts improves surgery prediction, by training 12-node TGR-WM with an additional loss designed to preserve line length:

Rebuttal Table R3-3. Line-length loss on 12-node TGR-WM (Sec. 4.3; Appendix H.5).

Training / scoringPredicted/true line-length ratioIntervention AUROC
Submitted loss, original scoring0.0100.793
+ Line-length loss, original scoring0.8510.502
+ Line-length loss, scoring the known surviving relay0.8510.886

The ratios are measured on intact test trajectories. The last row uses the simulator's knowledge of the relevant node.

With the original score, line-length training reduces intervention AUROC by −0.291 [−0.350, −0.230]. Scoring only the simulator-known relay instead raises it by +0.384 [+0.322, +0.446] (paired 95% CIs). Thus, even when predicted signal variation improves, the node used for scoring still matters. Paper Table 30 reports all four model–setting comparisons.

What we learn. A forecasting test alone cannot tell whether the model has learned the information needed for an intervention or whether its final score uses that information correctly. Our results show why both should be checked: a forecast-trained encoder can preserve useful state information, while the surgery ranking still fails. This gives a practical way to investigate failures of forecast-trained world models beyond this simulated task.

Why simulate the task? For a real patient, we observe the outcome of the surgery actually performed, not the outcome of an alternative surgery on that same patient. Simulation gives us both possible outcomes and the true state behind them, so we can test the model's predictions and investigate errors separately. Here, the true surviving-relay state predicts the outcome almost perfectly (0.998/0.999 AUROC; Figure 4). Both candidate resections leave the same edited network, so this benchmark tests whether models recover and use that state; it does not compare different seizure-propagation paths. Sec. 4 (paragraph "Benchmark, and what it measures") states this scope and now also states that both resections leave the same edited network; Appendix H.4 adds a rollout-graph comparison.

W3. Prior work on predicting epilepsy surgery outcomes

3. Reply to W3 — The Related Work statement (L143 of the submission) overlooks existing outcome-prediction studies. The wording was too broad. Sec. 2 (paragraph "Epilepsy intervention modelling") no longer says that this literature does not predict outcomes, and now cites the three studies you list (blue in the revised PDF):

Clinical studies have predicted postoperative seizure outcomes using electrophysiological biomarkers and information about resected tissue (Fedele et al., 2017; Kuroda et al., 2021; Matarrese et al., 2023).

Our question is one that records of performed surgeries cannot answer: whether a model validated only by forecasting can compare two candidate surgeries for the same patient. That requires both outcomes and the state that governs them, which the simulator provides (W2).

Reviewer_ys2COpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether a world model that forecasts well can also predict the outcome of interventions. Rollout models must first pass a forecasting gate, and are then compared with no-rollout predictors that see the same inputs. The test uses simulated Epileptor cohorts where each virtual patient has two candidate resections. The main model passes the gate but ranks interventions near chance (AUROC 0.519/0.509 vs. 0.743/0.701 for the static baseline), and none of the other rollout models clearly beats the strongest no-rollout predictor.

Critical strengths
  • Interesting benchmark. The simulator gives the outcome of both resections for every patient, so comparisons can be paired.
  • The evaluation is carefully prespecified, with a paired bootstrap and a fixed margin (Table 1).
  • Extensive controls across architectures, capacity, encoders and label budgets.
  • The authors are upfront that the task only requires recovering one state.
Critical weaknesses
  • The main insight is interesting, but it is expected from prior work. It is already known that predictive accuracy and decision usefulness can diverge (Lambert et al., L4DC 2020; Grimm et al., NeurIPS 2020; Freeman et al., NeurIPS 2019). It is unclear what this paper adds beyond showing this in one more synthetic setting.
  • The novelty of the benchmark is unclear. Virtual resection on Epileptor networks is established (Proix et al., Brain 2017; Olmi et al., PLoS Comput. Biol. 2019). The new part is a simplified connectome where one node decides the outcome, so a graph-free MLP does as well as GIN (Table 3).
  • The writing could be streamlined. Several passages list results without a clear takeaway, and some transitions are hard to follow, e.g. "We answer with a forecast-gated..." (p. 2) with no question before it, and "Four contrasts... ask what the dissociation depends on" (p. 8).
Questions for authors
  • Which result here would the prior work above not have predicted?
  • Appendix C says rollouts on the common surviving nodes are identical for both resections. Does the benchmark test rolling dynamics forward at all, or only the encoder's state estimate?
  • The main comparison changes both rollout and access to intervention labels. Beyond the label-free context baselines (Appendix E), could you add a comparison with supervision held fixed?
3 comments
AuthorsOpenReview ↗

Response to Reviewer ys2C (W1 and Q1)

Thank you for the careful review and the specific questions. Both of your questions about rollout led to new controls in the revised PDF: rollout versus no rollout under matched supervision (Q3; Appendix H.2) and a rollout-graph comparison (Q2; Appendix H.4).

[New] marks post-hoc, exploratory analyses. Values separated by "/" are 12-node / 16-node. New difference intervals are 95% patient-clustered paired bootstrap CIs. Rebuttal tables are numbered R2-1 to R2-5; "paper Table" refers to the revised PDF.

W1 & Q1. Novelty beyond known objective mismatch

1. Reply to W1 & Q1 — Which results would prior work not have predicted? Three results are not predicted by the work you cite (Lambert et al., 2020; Grimm et al., 2020; Freeman et al., 2019). That work does not tell us whether the information needed for an intervention is missing from a forecast-trained model or present but used incorrectly. Our simulator reveals the state that determines each outcome, so we can tell these two cases apart.

(i) A forecast-trained encoder can retain outcome-relevant information [New]. In a simulator-assisted encoder diagnostic, TGR-WM's x^0\hat{x}_0 at the known surviving relay, with signs aligned using simulator ground truth, reaches 0.94–0.98 AUROC in all six members (Sec. 4.3; Appendix H.3). With the encoder frozen, replacing the 33-parameter mean head of the frozen-encoder control (Table 15) with a 65-parameter mean+max head raises AUROC at 25 labelled patients from 0.557/0.530 to 0.838/0.888 (Sec. 4.2; Appendix H.1, Table 27).

(ii) The failure is one of direction and scoring, and it varies across seeds of the same model. Three of the six TGR-WM encoder estimates need sign reversal. Ten 16-node EWM runs differing only in seed have forecast skill 0.651–0.686 but intervention AUROC 0.335–0.751 (Spearman ρ=−0.01\rho=-0.01; Sec. 4.1). Orienting EWM members using validation labels and standardizing scores raises AUROC 0.519/0.509 → 0.704/0.710 (paired gains +0.185 [+0.098, +0.272] / +0.200 [+0.124, +0.276]; Table 18). Encoder swaps change EWM from 0.519/0.509 to 0.875/0.831 and TGR-WM from 0.793/0.912 to 0.488/0.689 (Table 21).

(iii) Aligning the training loss with the scored statistic does not repair the ranking [New]. Adding a line-length loss to 12-node TGR-WM brings the predicted line length close to the true one, yet intervention AUROC falls. Scoring the simulator-known surviving relay restores it, so the choice of scoring node contributes substantially to this failure (Sec. 4.3).

Rebuttal Table R2-1. Line-length loss on 12-node TGR-WM (Sec. 4.3; Appendix H.5). Line-length ratios are measured on intact test trajectories.

Training and scoringPredicted/true line lengthIntervention AUROC
Submitted loss; original scoring (maximum over surviving nodes)0.0100.793
Added line-length loss; original scoring0.8510.502
Added line-length loss; scoring the simulator-known surviving relay0.8510.886

The original scoring loses −0.291 [−0.350, −0.230] after line-length training; scoring the known relay gains +0.384 [+0.322, +0.446] (paired 95% CIs). Paper Table 30 reports all four model–setting comparisons.

These findings identify encoder choice, score direction, and final scoring as important factors affecting intervention prediction.

AuthorsOpenReview ↗

Response to Reviewer ys2C (W2, Q2 and Q3)

W2 & Q2. What the benchmark tests and what rollout adds

2. Reply to W2 — What does the benchmark contribute? The new part is not virtual resection itself (Proix et al., 2017; Olmi et al., 2019) but its use as ground truth for learned world models: the benchmark provides both candidate outcomes for every virtual patient and reveals the state that governs them. This makes it possible to evaluate paired intervention decisions and to identify why a forecast-trained model fails, which no dataset of performed surgeries allows. The true surviving-relay state yields 0.998/0.999 AUROC (Figure 4), whereas EWM reaches 0.519/0.509 despite passing the forecasting test (Table 3). The graph-free MLP is close to GIN (0.801/0.823 vs. 0.810/0.823), consistent with the benchmark's focus on state recovery.

We screened ten propagation-focused variants (Table 13); none jointly met the propagation, outcome-balance, and full-state predictability criteria. Sec. 5 now reports this screen.

3. Reply to Q2 — What does rollout test here? It tests the model's state estimate and how that estimate is scored, not differences in propagation between the two surgeries. The two candidates leave the same edited adjacency matrix and different relays in the scored sets. Appendix C verifies identical rollout latent statistics on common surviving nodes for the frozen 12-node EWM members. Sec. 4 (paragraph "Benchmark, and what it measures") now states this.

A graph-condition control [New] asks whether the rollout graph matters at all. It compares resected-graph with intact-graph rollout features, using the same frozen encoder and labels, with one 17-parameter head trained per condition:

Rebuttal Table R2-2. Resected-graph versus intact-graph rollout features (paper Table 29; Appendix H.4). ΔAUROC is resected minus intact, across patients. The last two columns count outcome-discordant test patients whose two surgeries are ordered correctly.

Labelled patients12-node ΔAUROC [95% CI]16-node ΔAUROC [95% CI]12-node correct: intact → resected16-node correct: intact → resected
100+0.047 [+0.024, +0.072]+0.013 [+0.004, +0.023]91/95 → 92/9588/91 → 88/91
980+0.027 [+0.013, +0.044]+0.000 [−0.004, +0.004]91/95 → 92/9588/91 → 88/91

The resected graph improves across-patient AUROC in three of four comparisons but barely changes which surgery is preferred for the same patient.

Q3. Rollout versus no rollout under matched supervision

4. Reply to Q3 — The main comparison changes both rollout and label access; can supervision be held fixed? Yes, and we ran this control [New]. We freeze the same TGR-WM encoder and train two 19-parameter heads on the same intervention labels. Both receive the estimated excitability x^0\hat{x}_0; one uses pooled initial latents, the other pooled rollout latents.

Rebuttal Table R2-3. Rollout versus no rollout with the same labels (paper Table 28; Appendix H.2). In the paper the two heads are labelled B0X and B1X.

SettingLabelled patientsNo rolloutRolloutPaired ΔAUROC [95% CI]
12-node1000.9280.955+0.027 [+0.009, +0.048]
16-node1000.9560.971+0.015 [+0.001, +0.032]
12-node9800.9510.959+0.008 [−0.003, +0.019]
16-node9800.9770.978+0.001 [−0.004, +0.007]
  • With matched labels, rollout adds a small improvement at 100 patients and little at 980. Sec. 4.2 now reports these differences.
  • With no outcome labels, the encoder-only score orders the two surgeries within a patient better than the original rollout at 12 nodes and similarly at 16 nodes (Rebuttal Table R2-4).

Rebuttal Table R2-4. Label-free ordering of the two surgeries within a patient (Appendix H.2). Share of outcome-discordant test patients ordered correctly; neither scorer uses outcome labels.

CohortDiscordant patientsMax-pooled encoder x^0\hat{x}_0Original rolloutPaired gain [95% CI], percentage points
12-node9594.7%84.2%+10.5 [+4.1, +17.8]
16-node9196.7%93.4%+3.3 [−2.2, +9.0]

Together, these comparisons show that much of the useful information is available before rollout.

AuthorsOpenReview ↗

Response to Reviewer ys2C (W3)

W3. Presentation

5. Reply to W3 — The writing could be streamlined. Thank you for pointing to specific passages. The two transitions you quote were hard to follow as written; both are rewritten, and the label-budget paragraph now leads with its result:

Rebuttal Table R2-5. Presentation changes in the revised paper.

LocationChange
Introduction, third paragraph (formerly "We answer with…")Now opens with the question: "We ask whether a world model that passes a held-out forecasting test can reliably predict intervention outcomes."
Sec. 4.4, opening (formerly "Four contrasts…")"We next test whether intervention performance depends on the dynamics core, the world-model family, model size, or the encoder."
Sec. 4.2, label-budget paragraphTwo result-first sentences added: the 25-patient frozen-head AUROC and the matched-supervision rollout differences (details in Appendix H.1–H.2).
AbstractOne sentence now states the research question as in the Introduction.
Reviewer_hteSOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript evaluates whether forecast-trained time-series world models predict surgical outcomes on simulated Epileptor networks better than label-supervised predictors without rollout. The main world model fails on interventions despite passing the MSE forecasting gate, and no rollout model beats the strongest supervised no-rollout baseline by the prespecified margin.

Critical strengths
  • Extensive diagnostics cover 11 rollout models, encoder swaps, linear probes, label budgets, and seed sensitivity.
  • Paired counterfactual surgical outcomes for each virtual patient allow controlled within-patient comparisons.
Critical weaknesses
  1. Limited novelty and confounded capacity allocation (Autocorrelation vs. Supervision)
  • Unlike discrete language modeling, the premise that continuous time-series MSE forecasting does not guarantee downstream decision competence is already established in RL objective mismatch, physiological driver-blindness, and time-series foundation model benchmarks.
  • The main contrast conflates rollout value with supervision: time-series rollout models spend 100% of their capacity learning pre-intervention autocorrelation via MSE with zero outcome labels.
  • Conversely, no-rollout baselines dedicate all capacity to mapping (Xpre,a)→y(X^{pre}, a) \rightarrow y using 1,960 counterfactual surgical outcome labels.
  • In Table 15, pretraining TGR-WM's encoder on MSE loses 13–16 AUROC points compared to end-to-end label training (Temporal head, 0.907/0.939).
  • When encoder training is matched, label-free rollout actually improves performance, and label-free TGR-WM small beats the supervised Temporal head.
  1. Simplified benchmark where network simulation is unnecessary
  • After either candidate surgery removes the seizure zone, the remaining relay node is completely disconnected from the rest of the brain network.
  • Because seizures cannot spread after surgery, the outcome depends only on that isolated node's initial state, making step-by-step simulation unnecessary.
  1. Mismatch between the MSE forecasting gate and the intervention readout
  • Training and gating continuous time series with MSE smooths out predicted waves (i.e. like a mean-seeking behavior), washing out the sharp fluctuations (line length) that the readout uses to score seizures.
  • Passing a smooth MSE forecast gate therefore cannot guarantee that a model preserves the relative spike ordering needed to rank surgeries.
Questions for authors

Because the intervention readout uses an L1 temporal derivative (line length) while the gate uses pointwise MSE, all MSE-trained models attenuate line length (Table 22). If the training loss and forecasting gate include a temporal-difference (line-length) or spectral term, does forecast skill predict intervention AUROC?

3 comments
AuthorsOpenReview ↗

Response to Reviewer hteS (W1 —- W1.1, 1.2, 1.3)

Thank you for the careful review. Our description of the Table 15 frozen-encoder control did not make its mean-only readout clear. We checked that control and added a stronger readout comparison (W1.4–W1.5).

[New] marks post-hoc, exploratory analyses. AUROC pairs are 12-node / 16-node; difference intervals are 95% patient-clustered paired bootstrap CIs. Rebuttal tables are numbered R1-1 to R1-6; "paper Table" refers to the revised PDF.

W1. Novelty, supervision, and model capacity

1. Reply to W1.1 — What is new beyond the known mismatch? The known mismatch says that forecast accuracy and decision performance can diverge. Our results show something more specific: a forecast-trained encoder can retain strong information about the outcome-relevant state, even when the final intervention score fails to use it correctly. A task with a known outcome-relevant state makes this visible.

  • Outcome-relevant information remains in the encoder [New]. In a simulator-assisted diagnostic, TGR-WM's encoder estimate x^0\hat{x}_0 at the known surviving relay, with signs aligned using simulator ground truth, yields 0.94–0.98 AUROC in all six members (Sec. 4.3; Appendix H.3). With the encoder frozen, a 65-parameter head trained on 25 labelled patients reaches 0.838/0.888 (W1.4–W1.5).
  • Forecast training does not fix its direction. Three of the six TGR-WM encoder estimates need sign reversal; this concerns the encoder output x^0\hat{x}_0, and the decoded line-length readout of these members is positively oriented (Table 19). EWM members trained from different seeds take opposite score directions; aligning them with validation labels and standardizing their scores raises AUROC from 0.519/0.509 to 0.704/0.710 (Table 18).
  • Forecast skill does not reveal which model will work. Ten 16-node EWM runs with the same data, architecture, and objective have similar forecast skill (0.651–0.686) but intervention AUROC from 0.335 to 0.751 (Spearman ρ=−0.01\rho=-0.01; Sec. 4.1).
  • Making the forecast more faithful does not repair the ranking [New]. Restoring line length in the forecasts lowers 12-node TGR-WM AUROC from 0.793 to 0.502 (W3).

These results show how score direction and the final readout contribute to the gap between forecasting and intervention prediction.

2. Reply to W1.2–W1.3 — Does outcome supervision explain the main comparison? The reviewer is right that the submitted comparison differs in access to outcome labels: the supervised baselines use 1,960 outcomes from 980 patients, while the forecast-trained models use none. That comparison tests whether forecasting success leads to accurate intervention prediction; it does not isolate the benefit of rollout. We therefore added a matched-supervision control [New]: two 19-parameter heads use the same frozen TGR-WM encoder, the same outcome labels, and the same estimate x^0\hat{x}_0. One uses initial features; the other uses rollout features.

Rebuttal Table R1-1. Rollout versus no rollout with the same labels (paper Table 28; Appendix H.2).

SettingLabelled patientsNo rolloutRolloutRollout − no rollout [95% CI]
12-node1000.9280.955+0.027 [+0.009, +0.048]
16-node1000.9560.971+0.015 [+0.001, +0.032]
12-node9800.9510.959+0.008 [−0.003, +0.019]
16-node9800.9770.978+0.001 [−0.004, +0.007]

With the same labels, rollout gives small gains at 100 patients and almost no gain at 980. Thus, the large gap in the original comparison cannot be read as the benefit of rollout alone. We also compared two label-free scorers on which one correctly orders the two surgeries for the same patient [New].

Rebuttal Table R1-2. Label-free ordering of the two surgeries within a patient (Appendix H.2). Share of outcome-discordant test patients ordered correctly; gain in percentage points.

CohortPatientsMax-pooled encoder x^0\hat{x}_0Original rolloutGain [95% CI]
12-node9594.7%84.2%+10.5 [+4.1, +17.8]
16-node9196.7%93.4%+3.3 [−2.2, +9.0]

Neither scorer uses outcome labels. The three sign-reversed encoder members (W1.1) do not change this ordering: their maximum lies at a peripheral node kept by both resections, so they give both surgeries the same score.

Changes in the revised manuscript: Sec. 4.2 now reports the matched-supervision differences; Appendix H.2 (paper Table 28) gives both comparisons.

AuthorsOpenReview ↗

Response to Reviewer hteS (W1—- W1.4, 1.5)

3. Reply to W1.4–W1.5 — Does Table 15 show a 13–16-point cost of MSE pretraining? The original frozen-encoder control uses a 33-parameter head that averages node features. We kept the encoder fixed and tested a 65-parameter head that uses both the mean and maximum [New]:

Rebuttal Table R1-3. Frozen-encoder heads across label budgets (paper Tables 15 and 27; Appendix H.1). AUROC, 12-node / 16-node. Each labelled patient supplies two outcome labels.

Predictor25 patients50 patients100 patients980 patients
Frozen encoder, mean head (33 parameters; submitted)0.557 / 0.5300.633 / 0.5980.626 / 0.6350.751 / 0.810
Frozen encoder, mean+max head (65 parameters) [New]0.838 / 0.8880.866 / 0.8940.867 / 0.9010.887 / 0.913
End-to-end Temporal head (submitted)0.638 / 0.5730.764 / 0.6710.773 / 0.7630.907 / 0.939
Label-free TGR-WM rollout (submitted)0.793 / 0.9120.793 / 0.9120.793 / 0.9120.793 / 0.912
  • Against end-to-end training. At 25–100 patients the new frozen head exceeds the end-to-end head in both settings. At 980 patients the end-to-end advantage shrinks to +0.020 [−0.005, +0.045] / +0.025 [+0.007, +0.044]. The 13–16-point gap therefore cannot be attributed to MSE pretraining alone.
  • Against the rollout. The label-free rollout is no longer clearly better than its own encoder: against the 25-patient head, rollout minus head is −0.045 [−0.091, +0.002] / +0.025 [−0.000, +0.049]. That head uses 50 outcome labels and the rollout none. The head also correctly orders 91 of 95 and 89 of 91 outcome-discordant patients.

This compares two readout designs and head sizes, not pooling alone. The positive result for TGR-WM small also remains: it exceeds the end-to-end Temporal head at 16 nodes (+0.026 [+0.010, +0.044]; Table 14).

Changes in the revised manuscript: Sec. 4.2 now reports the 25-patient result; Appendix H.1 (paper Table 27) gives the comparison above; Appendix D specifies that the earlier finding applies to the original mean-only head.

AuthorsOpenReview ↗

Response to Reviewer hteS (W2, W3 and Q1)

W2. Simplified benchmark

4. Reply to W2 — Why use a benchmark that does not require network simulation? The reviewer is right that the two candidate surgeries do not require comparing different propagation patterns. We designed the benchmark to test a different question: can a forecast-trained model recover and correctly use the state that determines intervention outcomes? The simulator provides both candidate outcomes and the relevant state, so we can separate errors in state estimation from errors in the final score. Both surgeries leave the same edited adjacency matrix, but the surviving-node masks retain different relays. Appendix C verifies identical rollout latent statistics on the common surviving nodes for the frozen 12-node EWM members.

The true surviving-relay state predicts outcomes at 0.998/0.999 AUROC (Figure 4), yet EWM reaches only 0.519/0.509 despite passing the forecasting test (Table 3). We also screened ten propagation-focused variants (paper Table 13); none jointly met the propagation, outcome-balance, and full-state predictability criteria.

Changes in the revised manuscript: Sec. 4 (paragraph "Benchmark, and what it measures") states that the task is state recovery rather than propagation, and now also states that both resections leave the same edited adjacency. Sec. 5 now reports the outcome of the propagation screen.

W3 & Q1. MSE training and the line-length readout

5. Reply to W3 — Does the MSE/line-length mismatch explain the failure? Decoded line length is strongly attenuated in the submitted models (Table 22). We added temporal-difference (TD) and line-length-preservation (LL) training losses, retaining the original architectures and intervention readout [New]. All configurations pass the original MSE forecasting test.

Rebuttal Table R1-4. Intervention AUROC under each training loss (paper Table 30; Appendix H.5). 12-node / 16-node. TD uses λ=0.1\lambda=0.1 for EWM and λ=1\lambda=1 for TGR-WM; LL uses λ=0.1\lambda=0.1.

ModelSubmitted+ TD loss+ LL loss
EWM0.519 / 0.5090.588 / 0.7340.372 / 0.689
TGR-WM0.793 / 0.9120.878 / 0.8830.502 / 0.867
  • TD. Derivative forecasting skill (skill of the predicted first temporal difference against a zero-change predictor) changes by only about 10−310^{-3}, so the AUROC changes do not follow a consistent improvement in derivative prediction.
  • LL. Preserving line length does not repair intervention ranking in 12-node TGR-WM. Rebuttal Table R1-5 shows that the choice of scoring node matters even after the signal variation is restored.

Rebuttal Table R1-5. Restoring line length versus choosing the scoring node, 12-node TGR-WM (Sec. 4.3; Appendix H.5). Line-length ratios are measured on intact test trajectories.

Training and scoringPredicted/true line lengthIntervention AUROC
Submitted MSE loss; original scoring (maximum over surviving nodes)0.0100.793
Added LL loss; original scoring0.8510.502
Added LL loss; scoring the simulator-known surviving relay0.8510.886

With the original scoring, AUROC falls by −0.291 [−0.350, −0.230] after LL training. Scoring the known relay instead raises it by +0.384 [+0.322, +0.446] (paired 95% CIs).

6. Reply to Q1 — Would a derivative- or spectrum-aware forecasting test predict intervention outcomes? In the tested range, derivative forecasting skill did not track intervention AUROC. We added TD and LL terms to the training loss and evaluated derivative forecasting skill as the candidate test metric, across 48 members: two model families × two node sizes × four training settings (submitted, TD λ=0.1\lambda=0.1, TD λ=1\lambda=1, LL λ=0.1\lambda=0.1) × three members.

Rebuttal Table R1-6. Spearman correlation with member-level intervention AUROC (Appendix H.6). Within each model family; 24 members per family.

QuantityEWMTGR-WM
Derivative forecasting skill0.110.12
MSE forecasting skill0.090.35
How closely the decoded relay signal tracks the true relay excitability0.940.71

Across these members, agreement with the true relay state correlates much more strongly with intervention AUROC than either forecasting metric. We focused on temporal-difference and line-length objectives because they directly correspond to the intervention readout.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.