- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
MedRoAM refines medical language world models using Privileged Self-Distillation (PSD) and Intervention Response Alignment (IRA). PSD aligns a student conditioned on clean or cached rollout histories with a frozen teacher conditioned on observed histories. IRA constrains differentiable numerical surrogates across original, removed, and similarly substituted interventions. The study evaluates teacher-forced and autoregressive prediction on MIMIC-IV and an external eICU-CRD Demo cohort, together with 14 supervised and seven held-out intervention-response relations. Baseline comparisons, ablations, additional backbones, bootstrap intervals, and refinement-seed checks support improved rollout prediction and several pharmacological-consistency metrics. Rollouts remain conditioned on recorded interventions and measurement schedules; the intervention tests do not establish patient-specific causal effects.
Critical strengths
S1. The objectives address clearly diagnosed failure modes.
PSD directly targets accumulated-history error, while IRA supplies a training signal for intervention-dependent responses that ordinary value-token prediction does not explicitly enforce. The EHRWorld (align) comparison uses the same refinement subset and a shared initial checkpoint, helping separate the proposed objectives from merely continuing training. The component ablations are informative: they show a trade-off between rollout accuracy and intervention responsiveness rather than claiming that the joint method dominates every component on every metric.
S2. The rollout improvements have meaningful robustness checks.
Beyond the main Qwen results, the appendix evaluates additional backbone families, patient-clustered paired bootstrap intervals, and three matched 4B refinement seeds. For Qwen3-4B, the ID Full gains over EHRWorld (align) are 1.61 percentage points [1.36, 1.88] in Numer. S@25 and 3.53 [3.22, 3.84] in Stat F1 (Table 15). The fixed-initial-checkpoint seed study and external rollout evaluation add credibility to the predictive improvements. Testing seven relations excluded from IRA supervision is also more informative than evaluating only the supervised probes.
Critical weaknesses
W1. Improved sensitivity is accompanied by unresolved specificity costs.
Table 18 shows that held-out negative-control sensitivity rises from 23.58% to 30.49% at 4B and from 20.60% to 32.70% at 8B relative to EHRWorld (align); held-out USR also worsens at these scales. Figure 12 further shows that absolute responses to similar substitutions increase, even when their size relative to removal effects decreases. The authors acknowledge these results, and the controls are not established biological zero-effect interventions. Nevertheless, they indicate that some gains may coexist with broader responsiveness to intervention edits, rather than consistently improved selectivity. This limits the intervention-specificity contribution, while leaving the rollout-accuracy gains intact.
W2. The numerical alignment objective has only partial validation against free decoding.
IRA fixes the sign, token layout, place values, and teacher-forced prefixes using an observed response (Section 3.3 and Appendix C.1). It therefore does not represent the expected value of unrestricted numerical generation. Table 17 usefully examines this gap, but the reported diagnostic uses training-set instances: at the default lambda = 0.1, surrogate/decoded-effect correlation is 0.426, and 8.25% of margin-satisfied instances still have unchanged decoded values. Held-out decoded probe performance is reported elsewhere, but it does not directly validate the surrogate-decoding relationship on unseen episodes. This leaves uncertainty about how reliably optimizing IRA controls generated numerical responses beyond its supervised formats.
Questions for authors
Q1 On held-out relations, can the positive-probe and negative-control changes be analyzed per probe with paired uncertainty and absolute effect magnitudes? The aim is to distinguish improved target-directed responses from broadly amplified edit sensitivity, using the current controls without assuming they are proven physiological nulls. This would clarify the scope of the intervention improvement.
Q2 Can the Table 17 diagnostic be repeated on held-out episodes with observed response values, separating supervised and held-out relations and cases with different decoded digit/decimal layouts? This targeted evaluation would show whether the training-set surrogate-decoding agreement transfers, without requiring new clinical outcomes or a new model architecture.
3 comments
Rebuttal by Authors (2/2)
Response to W2 & Q2:
Following your suggestion, we repeat the training-set IRA surrogate-decoding agreement analysis on unseen test episodes, separating supervised and held-out probes and cases with different decoded digit/decimal layouts.
First, we compare performance on supervised and held-out probes separately. The 4B results from Table 19 of the revised manuscript are shown below.
| Probes | Model | Satisfied | Unchanged Satisfied | Direction Changed | Spearman- | Changed | Direction |
|---|---|---|---|---|---|---|---|
| Supervised | EHRWorld-4B (align) | 4.43 | 48.84 | 45.91 | -0.076 | 19.51 | 8.96 |
| Supervised | MedRoAM-4B -w/o IRA | 4.68 | 39.56 | 34.70 | 0.004 | 42.72 | 14.82 |
| Supervised | MedRoAM-4B | 18.06 | 16.81 | 71.48 | 0.304 | 53.42 | 38.19 |
| Held-out | EHRWorld-4B (align) | 2.65 | 84.85 | 65.45 | 0.098 | 4.41 | 2.89 |
| Held-out | MedRoAM-4B -w/o IRA | 6.74 | 70.24 | 60.36 | -0.049 | 26.73 | 16.13 |
| Held-out | MedRoAM-4B | 11.56 | 49.31 | 78.15 | 0.294 | 26.08 | 20.39 |
As seen, although agreement on supervised probes in the test episodes is weaker than in the training-set analysis, IRA still improves surrogate-decoding correlation and conditional directional accuracy over both baselines. Similar improvements are observed on held-out probes.
Second, we further examine performance under different decoded digit and decimal layouts. Table 20 of the revised manuscript reports the following analysis for MedRoAM-4B () on supervised probes from both the training and test sets, with "Matched" indicating an unchanged numeric format and "Deviating" indicating a changed format.
| Dataset | Group | Satisfied | Unchanged Satisfied | Direction Changed | Spearman- | Changed | Direction |
|---|---|---|---|---|---|---|---|
| Train | Matched | 17.70 | 2.32 | 89.23 | 0.525 | 74.36 | 66.35 |
| Train | Deviating | 33.75 | 12.32 | 78.08 | 0.335 | 80.15 | 62.58 |
| Train | All | 24.66 | 8.25 | 84.19 | 0.426 | 76.87 | 64.72 |
| Test | Matched | 12.64 | 12.14 | 73.93 | 0.416 | 49.31 | 36.45 |
| Test | Deviating | 31.01 | 21.35 | 66.94 | 0.130 | 63.24 | 42.33 |
| Test | All | 18.06 | 16.81 | 71.48 | 0.304 | 53.42 | 38.19 |
Compared with "Matched" instances, "Deviating" instances exhibit higher Satisfied and Changed rates but lower Spearman-, indicating larger and more variable deviations from the observed ground-truth values and weaker surrogate-decoding agreement. We hypothesize that instances in which predictions differ in numeric format from the ground truth tend to be more challenging to predict, with less stable model behavior.
Overall, these experiments examine whether training-set surrogate-decoding agreement transfers to unseen cases. MedRoAM achieves better agreement than EHRWorld (align) and the "-w/o IRA" baseline on both supervised and held-out test probes. Nevertheless, agreement remains imperfect on unseen held-out probes and instances with changed numeric formats. We have clarified these limitations in the manuscript.
We hope these responses address the reviewer's questions. We have incorporated the additional results and clarifications into the revision. Please refer to the updated paper for more details.
Dear Reviewer YFVx,
We sincerely thank you again for your valuable feedback, which has helped us improve the quality of our work. As the discussion deadline is approaching, we kindly request you to review our rebuttal. We have also uploaded a revised PDF of the paper, addressing your concerns to the best of our ability. We are happy to address any remaining concerns. Thank you again for your time and effort!
Best regards,
Authors
Rebuttal by Authors (1/2)
We sincerely thank the reviewer for the detailed and thoughtful feedback. We address the questions below.
Response to W1 & Q1:
Thank you for this suggestion. In Table 22 of the revised manuscript, we have added per-probe paired uncertainty estimates for the differences between MedRoAM and EHRWorld (align) in sensitivity and normalized absolute effect magnitudes, covering both positive probes and negative controls at the 4B and 8B scales. For convenience, we present the detailed 4B results below. Here, ∣dordec∣ denotes the normalized absolute change in the model's prediction following intervention removal. Entries are differences between MedRoAM and EHRWorld (align), with 95% confidence intervals, and ΔSens is measured in percentage points.
As seen, although MedRoAM exhibits higher sensitivity to unrelated-intervention removal than EHRWorld (align), the increase in response magnitude is modest. Specifically, the macro-averaged increase in normalized absolute effect magnitude is 0.239 for positive probes, compared with only 0.040 for negative controls. Thus, although removing unrelated interventions can change MedRoAM's predictions, the average increase in response magnitude is much smaller than that for related interventions. Nevertheless, we acknowledge that increased sensitivity to unrelated-intervention removal remains a limitation. We have clarified this in the manuscript and identified explicit constraints on unrelated interventions in IRA as a direction for future improvement.