acceptodds
Under review as a conference paper at ICLR 2027

MedRoAM: Improving Rollout Robustness and Intervention Sensitivity in Medical Language World Models

Abstract

Medical language world models (LWMs) use large language model (LLM) backbones to predict how a patient's state evolves given the clinical history and an intervention. Despite their promise, medical LWMs remain at an early stage of development, and their limitations are not yet well understood. Through a series of preliminary analyses, we identify two key shortcomings: (1) limited rollout robustness, reflected in a substantial performance drop from teacher-forced to autoregressive simulation, and (2) weak intervention sensitivity, where predictions barely change even when the probing intervention is removed. To address these shortcomings, we propose MedRoAM, which jointly optimizes two simple-yet-effective training objectives: Privileged Self-Distillation (PSD) and Intervention Response Alignment (IRA). PSD uses a frozen teacher conditioned on ground-truth histories to guide a student trained on a mixture of clean and noisy rollout histories, encouraging robustness to accumulated errors while anchoring predictions to the initial model. IRA incorporates pharmacological knowledge to align numerical response estimates across the original intervention, its removal, and a similarly acting substitute. Extensive experiments across several LLM backbones show that MedRoAM improves autoregressive rollout accuracy on in-distribution and out-of-distribution data while largely preserving teacher-forced performance. Evaluations on 14 supervised and 7 held-out intervention-response probing pairs further demonstrate improved intervention sensitivity and pharmacological consistency.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Medical World Model, Large Language Model, Clinical Trajectory Modeling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_YFVxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MedRoAM refines medical language world models using Privileged Self-Distillation (PSD) and Intervention Response Alignment (IRA). PSD aligns a student conditioned on clean or cached rollout histories with a frozen teacher conditioned on observed histories. IRA constrains differentiable numerical surrogates across original, removed, and similarly substituted interventions. The study evaluates teacher-forced and autoregressive prediction on MIMIC-IV and an external eICU-CRD Demo cohort, together with 14 supervised and seven held-out intervention-response relations. Baseline comparisons, ablations, additional backbones, bootstrap intervals, and refinement-seed checks support improved rollout prediction and several pharmacological-consistency metrics. Rollouts remain conditioned on recorded interventions and measurement schedules; the intervention tests do not establish patient-specific causal effects.

Critical strengths

S1. The objectives address clearly diagnosed failure modes.

PSD directly targets accumulated-history error, while IRA supplies a training signal for intervention-dependent responses that ordinary value-token prediction does not explicitly enforce. The EHRWorld (align) comparison uses the same refinement subset and a shared initial checkpoint, helping separate the proposed objectives from merely continuing training. The component ablations are informative: they show a trade-off between rollout accuracy and intervention responsiveness rather than claiming that the joint method dominates every component on every metric.

S2. The rollout improvements have meaningful robustness checks.

Beyond the main Qwen results, the appendix evaluates additional backbone families, patient-clustered paired bootstrap intervals, and three matched 4B refinement seeds. For Qwen3-4B, the ID Full gains over EHRWorld (align) are 1.61 percentage points [1.36, 1.88] in Numer. S@25 and 3.53 [3.22, 3.84] in Stat F1 (Table 15). The fixed-initial-checkpoint seed study and external rollout evaluation add credibility to the predictive improvements. Testing seven relations excluded from IRA supervision is also more informative than evaluating only the supervised probes.

Critical weaknesses

W1. Improved sensitivity is accompanied by unresolved specificity costs.

Table 18 shows that held-out negative-control sensitivity rises from 23.58% to 30.49% at 4B and from 20.60% to 32.70% at 8B relative to EHRWorld (align); held-out USR also worsens at these scales. Figure 12 further shows that absolute responses to similar substitutions increase, even when their size relative to removal effects decreases. The authors acknowledge these results, and the controls are not established biological zero-effect interventions. Nevertheless, they indicate that some gains may coexist with broader responsiveness to intervention edits, rather than consistently improved selectivity. This limits the intervention-specificity contribution, while leaving the rollout-accuracy gains intact.

W2. The numerical alignment objective has only partial validation against free decoding.

IRA fixes the sign, token layout, place values, and teacher-forced prefixes using an observed response (Section 3.3 and Appendix C.1). It therefore does not represent the expected value of unrestricted numerical generation. Table 17 usefully examines this gap, but the reported diagnostic uses training-set instances: at the default lambda = 0.1, surrogate/decoded-effect correlation is 0.426, and 8.25% of margin-satisfied instances still have unchanged decoded values. Held-out decoded probe performance is reported elsewhere, but it does not directly validate the surrogate-decoding relationship on unseen episodes. This leaves uncertainty about how reliably optimizing IRA controls generated numerical responses beyond its supervised formats.

Questions for authors

Q1 On held-out relations, can the positive-probe and negative-control changes be analyzed per probe with paired uncertainty and absolute effect magnitudes? The aim is to distinguish improved target-directed responses from broadly amplified edit sensitivity, using the current controls without assuming they are proven physiological nulls. This would clarify the scope of the intervention improvement.

Q2 Can the Table 17 diagnostic be repeated on held-out episodes with observed response values, separating supervised and held-out relations and cases with different decoded digit/decimal layouts? This targeted evaluation would show whether the training-set surrogate-decoding agreement transfers, without requiring new clinical outcomes or a new model architecture.

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

We sincerely thank the reviewer for the detailed and thoughtful feedback. We address the questions below.

Response to W1 & Q1:

Thank you for this suggestion. In Table 22 of the revised manuscript, we have added per-probe paired uncertainty estimates for the differences between MedRoAM and EHRWorld (align) in sensitivity and normalized absolute effect magnitudes, covering both positive probes and negative controls at the 4B and 8B scales. For convenience, we present the detailed 4B results below. Here, ∣dordec∣\lvert d^{\mathrm{dec}}_{or}\rvert denotes the normalized absolute change in the model's prediction following intervention removal. Entries are differences between MedRoAM and EHRWorld (align), with 95% confidence intervals, and Δ\DeltaSens is measured in percentage points.

GroupProbeΔ\DeltaSens [95% CI]Δ⟨∣dordec∣⟩\Delta\langle\lvert d^{\mathrm{dec}}_{or}\rvert\rangle [95% CI]
PositiveNitroglycerin →\rightarrow HR+5.26 [-0.28, 10.25]+0.048 [-0.048, 0.143]
PositiveFentanyl →\rightarrow HR+0.49 [-3.32, 4.06]+0.042 [-0.010, 0.096]
PositiveMidazolam →\rightarrow MAP+12.07 [4.02, 20.12]+0.162 [0.076, 0.255]
PositiveMagnesium sulfate →\rightarrow MAP+2.44 [-1.48, 6.45]+0.028 [0.003, 0.053]
PositiveNicardipine →\rightarrow HR+15.48 [6.55, 24.40]+0.362 [0.201, 0.507]
PositiveDexmedetomidine →\rightarrow MAP+4.06 [-3.55, 11.17]+0.574 [0.392, 0.761]
PositiveHydralazine →\rightarrow HR+8.37 [3.80, 13.69]+0.462 [0.316, 0.592]
PositiveMacro average+6.88 [4.66, 9.27]+0.239 [0.194, 0.284]
Negative controlPantoprazole →\rightarrow HR+7.95 [3.18, 12.95]+0.032 [-0.031, 0.094]
Negative controlPantoprazole →\rightarrow MAP+13.41 [7.73, 19.32]+0.036 [-0.005, 0.076]
Negative controlPiperacillin →\rightarrow HR+7.42 [1.77, 13.43]-0.002 [-0.082, 0.080]
Negative controlAcetaminophen →\rightarrow HR+2.07 [-2.39, 6.54]+0.001 [-0.081, 0.084]
Negative controlHeparin →\rightarrow HR+8.93 [3.46, 14.41]+0.083 [-0.009, 0.179]
Negative controlCefepime →\rightarrow HR+7.49 [2.27, 12.38]+0.053 [-0.009, 0.114]
Negative controlCeftriaxone →\rightarrow MAP+0.00 [-7.78, 7.39]+0.040 [-0.018, 0.106]
Negative controlLevetiracetam →\rightarrow HR+7.97 [-0.72, 16.67]+0.080 [-0.091, 0.237]
Negative controlMacro average+6.91 [4.69, 9.06]+0.040 [0.010, 0.071]

As seen, although MedRoAM exhibits higher sensitivity to unrelated-intervention removal than EHRWorld (align), the increase in response magnitude is modest. Specifically, the macro-averaged increase in normalized absolute effect magnitude is 0.239 for positive probes, compared with only 0.040 for negative controls. Thus, although removing unrelated interventions can change MedRoAM's predictions, the average increase in response magnitude is much smaller than that for related interventions. Nevertheless, we acknowledge that increased sensitivity to unrelated-intervention removal remains a limitation. We have clarified this in the manuscript and identified explicit constraints on unrelated interventions in IRA as a direction for future improvement.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

Response to W2 & Q2:

Following your suggestion, we repeat the training-set IRA surrogate-decoding agreement analysis on unseen test episodes, separating supervised and held-out probes and cases with different decoded digit/decimal layouts.

First, we compare performance on supervised and held-out probes separately. The 4B results from Table 19 of the revised manuscript are shown below.

ProbesModelSatisfiedUnchanged ∣\mid SatisfiedDirection ∣\mid ChangedSpearman-ρ\rhoChangedDirection
SupervisedEHRWorld-4B (align)4.4348.8445.91-0.07619.518.96
SupervisedMedRoAM-4B -w/o IRA4.6839.5634.700.00442.7214.82
SupervisedMedRoAM-4B18.0616.8171.480.30453.4238.19
Held-outEHRWorld-4B (align)2.6584.8565.450.0984.412.89
Held-outMedRoAM-4B -w/o IRA6.7470.2460.36-0.04926.7316.13
Held-outMedRoAM-4B11.5649.3178.150.29426.0820.39

As seen, although agreement on supervised probes in the test episodes is weaker than in the training-set analysis, IRA still improves surrogate-decoding correlation and conditional directional accuracy over both baselines. Similar improvements are observed on held-out probes.

Second, we further examine performance under different decoded digit and decimal layouts. Table 20 of the revised manuscript reports the following analysis for MedRoAM-4B (λ=0.1\lambda=0.1) on supervised probes from both the training and test sets, with "Matched" indicating an unchanged numeric format and "Deviating" indicating a changed format.

DatasetGroupSatisfiedUnchanged ∣\mid SatisfiedDirection ∣\mid ChangedSpearman-ρ\rhoChangedDirection
TrainMatched17.702.3289.230.52574.3666.35
TrainDeviating33.7512.3278.080.33580.1562.58
TrainAll24.668.2584.190.42676.8764.72
TestMatched12.6412.1473.930.41649.3136.45
TestDeviating31.0121.3566.940.13063.2442.33
TestAll18.0616.8171.480.30453.4238.19

Compared with "Matched" instances, "Deviating" instances exhibit higher Satisfied and Changed rates but lower Spearman-ρ\rho, indicating larger and more variable deviations from the observed ground-truth values and weaker surrogate-decoding agreement. We hypothesize that instances in which predictions differ in numeric format from the ground truth tend to be more challenging to predict, with less stable model behavior.

Overall, these experiments examine whether training-set surrogate-decoding agreement transfers to unseen cases. MedRoAM achieves better agreement than EHRWorld (align) and the "-w/o IRA" baseline on both supervised and held-out test probes. Nevertheless, agreement remains imperfect on unseen held-out probes and instances with changed numeric formats. We have clarified these limitations in the manuscript.


We hope these responses address the reviewer's questions. We have incorporated the additional results and clarifications into the revision. Please refer to the updated paper for more details.

Dear Reviewer YFVx,

We sincerely thank you again for your valuable feedback, which has helped us improve the quality of our work. As the discussion deadline is approaching, we kindly request you to review our rebuttal. We have also uploaded a revised PDF of the paper, addressing your concerns to the best of our ability. We are happy to address any remaining concerns. Thank you again for your time and effort!

Best regards,
Authors

Reviewer_oL33OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents an EHR world model that predicts the evolution of EHR over time and in response to interventions. The work is clearly motivated by two observations of current clinical language world models (EHRWorld). First, there is a pronounced autoregressive decoding drift when using a generated EHR prefix compared to a ground-truth prefix. Second, predictions of future states are not sensitive enough to intervention changes. MedRoAM specifically addresses these two limitations, and on 4B, 8B, and 14B backbones, it shows considerable improvements over the baseline models of the respective sizes.

Critical strengths
  • The organization is very clear. The two preliminary observations directly motivate the two loss terms. MedRoAM is then evaluated in two aspects: state prediction accuracy and intervention sensitivity.

  • The tests cover both in-distribution (MIMIC-IV) and out-of-distribution (eICU) data, which are two major large-scale public EHR datasets. The scope demonstrates robustness.

  • The analysis regarding backbone choice and parameter sensitivity is thorough. The design choices and selection of parameters are overall justified.

Critical weaknesses
  • Table 1 results look saturated, particularly for the teacher-forced setups. It is worthwhile to analyze whether choosing 25% as the threshold of absolute error is too lenient. Specifically, the threshold works for heart rate and mean arterial pressure, but for measurements such as potassium and sodium, it becomes problematic. The EHRWorld paper also reports S@10, S@15, and S@25. Reporting the same metrics would help position the paper more accurately.

  • Incorporating both removal and substitution makes sense clinically, but in practice, it makes the Intervention Response Alignment (IRA) term quite messy. The equivalence loss, where the substitution term is used, does not provide enough improvement in performance based on the ablation (Table 10).

  • The exact number of trajectories, as categorized by intervention-response probing, is not immediately clear. Table 5 lists the counts before filtering.

Questions for authors

as above

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We sincerely thank the reviewer for the constructive feedback and recognition of our work. We address the questions below.

Response to W1

Thank you for this valuable suggestion. We have added S@10, S@15, and S@25 results for MedRoAM and the comparison methods on both ID and OOD data in Table 16 of the revised manuscript. For convenience, we report the ID results for MedRoAM and EHRWorld (align) below, covering teacher-forced prediction (Next) and autoregressive rollout (Full).

ModelNext S@10Next S@15Next S@25Full S@10Full S@15Full S@25
EHRWorld-4B (align)63.6775.2387.9043.7055.3371.75
MedRoAM-4B63.2074.7287.4944.7756.6873.36
EHRWorld-8B (align)63.7075.3487.9443.5455.3071.54
MedRoAM-8B63.4675.1187.8644.8356.8073.37
EHRWorld-14B (align)64.5075.9988.3044.4656.0572.27
MedRoAM-14B64.1475.8088.3145.5157.4173.87

As expected, stricter thresholds yield lower absolute scores. Nevertheless, the trends remain consistent with those observed for S@25, i.e., across model sizes and both ID and OOD settings, MedRoAM improves autoregressive rollout accuracy over the EHRWorld baselines while largely preserving teacher-forced accuracy.

More specifically, for potassium and sodium, we additionally report S@10, S@15, S@25, and Stat F1 under autoregressive rollout on the ID test set. In the table below, K denotes potassium and Na denotes sodium.

ModelK -> S@10K -> S@15K -> S@25K -> Stat F1Na -> S@10Na -> S@15Na -> S@25Na -> Stat F1
EHRWorld-4B (align)57.5675.2092.7647.4996.8499.1099.9549.71
MedRoAM-4B57.5875.9192.9448.0697.2399.1799.9452.99
EHRWorld-8B (align)58.1575.9293.0147.4096.8999.0699.9649.59
MedRoAM-8B57.9576.2893.1047.8697.1799.0599.9654.53
EHRWorld-14B (align)57.3175.0592.5648.0497.4599.2999.9751.36
MedRoAM-14B57.9375.7293.0448.8397.4499.2099.9753.98

These results support the reviewer's concern that S@25 can be overly permissive for these channels: scores are around 93% for potassium and nearly 100% for sodium, with small differences between the models. In contrast, Stat F1, which evaluates normal/abnormal status relative to the reference range, is substantially lower. MedRoAM achieves higher Stat F1 than EHRWorld (align) on both channels at all three model sizes. Thus, S@25 alone provides an incomplete assessment of rollout quality and should be interpreted alongside stricter numerical thresholds and Stat F1.


Response to W2

We would like to clarify that the equivalence loss targets response consistency under similar-drug substitution. Although removing Lequiv\mathcal{L}_{\mathrm{equiv}} has a relatively small effect on rollout accuracy in Table 10, rollout accuracy alone does not directly assess whether predictions remain consistent when an intervention is replaced with a pharmacologically similar drug. Thus, we added a direct comparison of substitution-response magnitudes in Figure 12 of revised manuscript, reproduced below for convenience. The table reports the macro-averaged normalized absolute change in predictions under similar-drug substitution.

Model sizeMedRoAM (full)MedRoAM w/o Lequiv\mathcal{L}_{\mathrm{equiv}}
4B1.283.03
8B1.141.67
14B1.632.21

Removing the equivalence loss increases substitution-response magnitude across all three model sizes. Such discrepancies are undesirable in these probes, where similarly acting interventions are expected to yield broadly comparable responses. These results demonstrate that Lequiv\mathcal{L}_{\mathrm{equiv}} helps limit excessive prediction changes under similar-drug substitutions, providing a benefit that is not fully reflected in rollout accuracy. We have clarified this role and expanded the corresponding analysis in the revised manuscript.


Response to W3

Following your suggestion, we have added the number of retained trajectories for each probe after filtering to Table 5 of the revised manuscript.


We hope these responses address the reviewer's questions. We have incorporated the additional results and clarifications into the revision. Please refer to the updated paper for more details.

Dear Reviewer oL33,

We sincerely thank you again for your valuable feedback, which has helped us improve the quality of our work. As the discussion deadline is approaching, we kindly request you to review our rebuttal. We have also uploaded a revised PDF of the paper, addressing your concerns to the best of our ability. We are happy to address any remaining concerns. Thank you again for your time and effort!

Best regards,
Authors

Thank you for the detailed response and the clarifications. All my concerns are fully addressed. I raised the score accordingly.

Thanks for your time and for confirming that all your concerns have been resolved. We are grateful for your positive evaluation.

Reviewer_XzUjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces the MedRoAM framework which improves the rollout robustness and intervention sensitivity of medical language world models through Privileged Self-Distillation (PSD) and Intervention Response Alignment (IRA). Experiments across multiple LLM backbones demonstrate improved autoregressive clinical trajectory prediction on in-distribution and out-of-distribution data, and more consistent responses to medical interventions.

Critical strengths
  • The paper identifies and adresses two important limitations of medical language world models: poor rollout robustness and weak intervention sensitivity.

  • The authors evaluate their approach across multiple LLMs, in-distribution and out-of-distribution datasets and held-out interventions, and support their experiments by extensive ablation studies.

Critical weaknesses

[W1]: Although IRA increases sensitivity to unrelated interventions (Table 18) and also increasing the IRA loss weight reduces rollout accuracy (Figure 4). These findings raise questions about whether the improvements reflect better modeling of intervention effects or greater responsiveness to intervention changes. Could the authors elaborate on this and how it affects the reliability of the simulated clinical trajectories?

[W2]: Rollout experiments use recorded intervention sequences, while intervention probing mainly evaluates individual treatment changes. Although in the Appendix authors partially address this, the paper does not evaluate multi-step rollouts under modified treatment sequences. Could the authors discuss whether the improvements in both capabilities extend to such scenarios?

Questions for authors

Please refer to the weaknesses.

2 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We sincerely thank the reviewer for the positive assessment and thoughtful questions. We address the questions below.

Response to W1

Thank you for raising this concern. We examine it from three perspectives: the direction of the responses, their magnitude under unrelated interventions, and the trade-off with rollout accuracy.

First, the results in Table 2 show that MedRoAM improves agreement with the expected pharmacological direction. We decompose responses to intervention removal into three categories: direction-consistent responses, direction-opposing responses, and unchanged predictions. Using Sens and Mono as proportions, these correspond to "Sens ×\times Mono", "Sens ×\times (1 - Mono)", and "1 - Sens", respectively. The table below presents the 4B results on held-in probes:

ModelDirection-consistent responses ↑\uparrowDirection-opposing responses ↓\downarrowUnchanged predictions
EHRWorld-4B (align)32.09%16.20%51.71%
MedRoAM-4B -w/o IRA28.43%24.43%47.14%
MedRoAM-4B48.11%12.74%39.15%

Compared with "-w/o IRA", full MedRoAM increases direction-consistent responses from 28.43% to 48.11%, while reducing direction-opposing responses from 24.43% to 12.74%. It also improves both measures relative to EHRWorld (align). These results support the contribution of IRA to pharmacologically consistent responses, beyond an increase in response frequency alone.

Second, we acknowledge that sensitivity to unrelated interventions increases. To assess their magnitude, we have added the mean normalized absolute effects of intervention removal for positive and negative-control probes to Table 21 of the revised manuscript. The 4B results are shown below:

ModelHeld-in: positive probesHeld-in: negative controlsHeld-out: positive probesHeld-out: negative controls
EHRWorld-4B0.560.140.450.18
EHRWorld-4B (align)0.640.140.480.19
MedRoAM-4B2.490.190.720.23

For MedRoAM, the mean response to negative-control removal is substantially smaller than that to positive-intervention removal in both settings: 0.19 vs 2.49 on held-in probes and 0.23 vs 0.72 on held-out probes. Relative to EHRWorld (align), the increases in negative-control response magnitude are also smaller than those for positive probes. Nevertheless, responses to unrelated interventions remain a limitation, particularly for held-out specificity, and we explicitly acknowledge this in the revised manuscript.

Finally, we agree that a large IRA weight can impair rollout accuracy, as shown in Figure 4. Despite that, at our default setting of λ=0.1\lambda = 0.1, Numer. S@25 decreases by only 0.15% relative to the model without IRA (73.51% →\rightarrow 73.36%), while the direction-consistent response score increases by 19.68% (28.43% →\rightarrow 48.11%). These results support our choice of λ=0.1\lambda = 0.1 as a practical balance between rollout accuracy and intervention alignment.


Response to W2

We agree that evaluating multi-step rollouts under modified treatment sequences is important for downstream clinical decision-making.

Table 12 provides evidence that MedRoAM retains intervention responsiveness when conditioned on its own generated histories. Specifically, for MedRoAM-4B, the direction-consistent response score remains close to that obtained with ground-truth histories (47.92% vs 48.11%), and is substantially higher than that of EHRWorld-4B (align) under its own rollouts (28.88%). Moreover, conditioning EHRWorld (align) on MedRoAM-generated histories improves its score from 28.88% to 29.63%. Together with the rollout evaluations, these results support the complementary benefits of PSD and IRA: improved rollout robustness and stronger intervention responsiveness under generated histories.

However, this experiment does not directly establish the accuracy of multi-step trajectories following changes to the treatment sequence. In observational EHR data such as MIMIC-IV, each patient trajectory records outcomes under the treatments actually administered, while the corresponding outcomes under alternative treatment sequences are unobserved. Although the model can generate predictions for modified treatment sequences, the available data do not provide patient-specific counterfactual ground-truth against which to directly validate those predictions.

We have clarified this distinction in Appendix D.3 and Limitations section (Appendix E). The revised manuscript explicitly recognizes the importance of evaluating multi-step trajectories under modified treatment sequences and identifies suitable counterfactual evaluation settings and downstream clinical utility as directions for future work.


We hope these responses address the reviewer's questions. We have incorporated all new results and clarifications into the revision.

Dear Reviewer XzUj,

We sincerely thank you again for your valuable feedback, which has helped us improve the quality of our work. As the discussion deadline is approaching, we kindly request you to review our rebuttal. We have also uploaded a revised PDF of the paper, addressing your concerns to the best of our ability. We are happy to address any remaining concerns. Thank you again for your time and effort!

Best regards,
Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Update of Manuscript

Dear reviewers,

We sincerely thank you for your constructive feedback and helpful suggestions. In response, we have revised the manuscript and highlighted the key additions in blue for ease of review. Below is a summary of the main updates.

Additional Experiments and Analyses

  • Rollout evaluation under stricter error thresholds:
    [Appendix D.3, Table 16] We added S@10 and S@15 alongside S@25 for all comparison methods under teacher-forced prediction and autoregressive rollout on both ID and OOD data, addressing concerns about the permissiveness of the 25% threshold. [Reviewer oL33, W1]

  • Further analysis of the equivalence loss:
    [Appendices D.2 and D.4, Figure 12] We added comparisons with the variant without Lequiv\mathcal{L}_{\mathrm{equiv}} across model sizes, showing its role in limiting excessive prediction changes under similar-drug substitutions. [Reviewer oL33, W2]

  • Response magnitudes for positive probes and negative controls:
    [Appendix D.4, Table 21] We supplemented sensitivity scores with normalized absolute response magnitudes under intervention removal for both held-in and held-out probes, providing a clearer comparison of responses to related and unrelated interventions. [Reviewer XzUj, W1]

  • Per-probe comparisons with paired uncertainty estimates:
    [Appendix D.4, Table 22] We added per-probe differences between MedRoAM and EHRWorld (align) in sensitivity and normalized absolute response magnitudes, with paired 95% confidence intervals, for positive probes and negative controls at the 4B and 8B scales. [Reviewer YFVx, W1&Q1]

  • Surrogate-decoding agreement on unseen episodes:
    [Appendix D.4, Tables 19 and 20] We extended the agreement analysis to test episodes, separating supervised and held-out probes. We also added comparisons across decoded numeric-format groups on the training and test sets and discussed the remaining agreement gap. [Reviewer YFVx, W2&Q2]

Data Reporting and Discussion

  • Trajectory counts after filtering:
    [Appendix B.4, Table 5] We added the number of retained trajectories for each intervention-response probe after filtering. [Reviewer oL33, W3]

  • Multi-step counterfactual rollouts and evaluation limitations:
    [Appendix D.3 and Appendix E] We expanded the discussion of rollouts under modified treatment sequences, clarified the distinction between intervention responsiveness under generated histories and counterfactual trajectory accuracy, and explained the lack of patient-specific counterfactual outcomes for direct validation. We also identified counterfactual evaluation and downstream clinical utility as directions for future work. [Reviewer XzUj, W2]

We provide further details in our point-by-point responses and thank the reviewers again for helping us improve the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.