- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents a hierarchical (two-stage) model for actions in sequential data, e.g. treatment actions in clinical patient trajectories. Each level in the hierarchy is some kind of continuous time HMM, i.e. a latent variable model where the latent state can have discrete states that change continuously in time. The paper learns two such models on top of each other, such that the lower level is fixed while learning the higher level. The lower level is interpreted as a “shared action-value anchor” and the higher level as “behavioral regimes” deviating from the shared anchor.
Critical strengths
None.
Critical weaknesses
The modeling approach seems sensible although it is difficult to understand the model very well by reading the main text. However, there are very many existing hierarchical modeling approaches, some more sophisticated than the presented method (e.g. end-to-end), that have been applied to clinical and other kinds of sequential data, and hence I am missing methodological novelty in this work. The analysis of the real-world data is reasonable but not very thorough, and hence I don’t find that either convincing enough to support acceptance at a top ML conference.
Questions for authors
Suggestion: it would be useful to better define terms used in the main text, e.g. what are “continuous outcome-proxy variables” and how are they different from “measurements”?
2 comments
Thank you for your rebuttal. I have read it and we will discuss with the other reviewers. I still feel the methdological contribution is a bit limited for ICLR. For now, I keep my original score.
Response to Reviewer yX9p
Thank you for your review. We clarify the method, its novelty, and the key terms below.
The method explains why patients in similar conditions receive different treatments, measured against one common clinical reference.
The novelty is this separation, not the hierarchy: a fixed, outcome-based reference plus regime costs on the same scale, so regimes are directly comparable. Prior work differs:
Fitting with the anchor is a real restriction: the anchored class q−G equals G only if q∈G (Theorem 3), and anchoring during fitting, not post hoc, cuts cost RMSE from 0.547 to 0.125 (Table 7). A more flexible end-to-end model loses this:
Takeaway: end-to-end training matches action NLL within 0.005 but raises cost RMSE by 68%; the action-only reference matches observed actions in 67.4% / 82.5% of decisions vs. 3.2% / 34.4% for ours, absorbing habits.
The analysis shows what changes in physiology and in treatment preference, and how this alters individual decisions:
Quantitative checks support this reliability: Stage I and II give the lowest forecasting and action NLL vs. six baselines and neural action models (Figures 5 and 8), robust to fixed classes, regime counts, anchor variants, features, and 40 refits per case (Tables 10, 16, 19–20, Appendix G.1).
A measurement is a sparse lab snapshot; the outcome proxy is the model's continuous estimate that carries the trend between snapshots (Appendices C and F).
Removing the proxy worsens forecasting NLL in all four settings (Appendix J.2).
In the revised manuscript, Section 2 points to Table 2 (Appendix C); Appendix A extends related work, and Appendices J.2 and J.5 add ablations (Tables 14 and 18).
[1] Ashwood et al., "Mice alternate between discrete strategies during perceptual decision-making," Nat. Neurosci., 2022.
[2] Ke et al., "Inverse reinforcement learning with switching rewards and history dependency for characterizing animal behaviors," ICML, 2025.
[3] Qiao et al., "Multi-modal inverse constrained reinforcement learning from a mixture of demonstrations," NeurIPS, 2023.
[4] Severson et al., "Personalized input-output hidden Markov models for disease progression modeling," MLHC, 2020.