- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes RC-DQN, an offline reinforcement learning method for titrating dexmedetomidine in postoperative ICU patients, using six-hour decision windows and 16 ordered dose levels. A shared network has three heads: a Q-function for long-term clinical benefit, a separate critic that scores physiological side-effect cost (bradycardia and hypotension), and a truncated-Poisson model of clinician dosing that respects dose ordering. At decision time, a conformal filter calibrated on held-out data removes doses that are poorly supported by clinician behavior or have high estimated cost. The method then picks the remaining dose with the best benefit-minus-cost score, with rule-based fallback to no infusion when no dose qualifies or a top dose looks unsafe. Evaluated with fitted Q evaluation on MIMIC-IV and eICU-CRD, RC-DQN achieves the highest estimated rate of survival without delirium and the highest composite return.
Critical strengths
None
Critical weaknesses
The method piles on many parts, but i0.26 percentage points on MIMIC (50.49% vs 50.23%) and 0.28 points on eICU, and the confidence intervals overlap. I'm not convinced the extra complexity pays off.
Adding cost to the conformal score is presented as the key novelty, but turning it off changes almost nothing, and one score even goes up. So it’s unclear which parts are actually doing the work.
I’m not sure the evaluation can tell good policies from bad ones. ConformalDQN almost never agrees with clinicians, yet it scores second. On eICU, the evaluator misses the real clinician outcome by about 6 points, which is bigger than any difference between methods.
Questions for authors
see weaknesses
1 comment
- Model complexity and contribution RC-DQN addresses offline dose selection by jointly considering treatment utility, historical action support, and predicted physiological cost. Its ordinal behavior model characterizes support across dose levels; a separate cost critic scores candidate actions; and conformal filtering, a cost threshold, and fallback rules determine the final action. In the in-database evaluation, RC-DQN had the numerically highest reported point estimates for both endpoint and total-reward FQE in each database. Endpoint estimates for RC-DQN versus ConformalDQN were 50.49% versus 50.23% on MIMIC-IV and 78.43% versus 78.15% on eICU. Action distributions under low heart rate and low blood pressure additionally describe how the policy responds in physiologically risky states. The contribution is the integration of ordinal support modeling, separate utility and cost scores, and constrained action selection; the offline estimates and action analyses characterize different aspects of this framework.
- Contribution of the risk-modulated conformal score The risk-modulated conformal score combines behavior probability and the cost-critic score to filter actions using historical support and predicted cost. To assess this component, we compared full RC-DQN with Ordinal Cost-Shielded Conformal (OCSC). OCSC reuses the full-model weights, removes the cost term from the conformal score, and recalibrates a behavior-only threshold on validation data; the cost threshold, cost-adjusted ranking, and fallback rules remain in place. Endpoint FQE for full RC-DQN versus OCSC was 50.49% versus 49.97% on MIMIC-IV and 78.43% versus 78.21% on eICU. On MIMIC-IV, total-reward FQE was 10.2554 versus 11.7565, and action-match rate was 50.17% versus 24.02%. Thus, adding cost to the conformal score changes policy behavior and produces different results across metrics. It isolates the score change and associated recalibration within a framework where the critic, cost threshold, ranking, and fallback remain active.
- Reliability of offline evaluation Action match and FQE measure different quantities: the former measures agreement with recorded clinician actions, while the latter estimates a target policy’s endpoint value over test initial states. Low agreement and a high FQE estimate are therefore not inherently contradictory. However, if a policy selects actions with insufficient support in the corresponding states, FQE relies more heavily on function approximation and extrapolation; neither action match nor FQE alone establishes true policy quality. We therefore interpret FQE as model-dependent estimates under the prespecified offline evaluation. RC-DQN has the numerically highest endpoint FQE point estimate in both databases. Action match and action distributions in risk states describe policy behavior; they are not direct evidence of clinical efficacy or safety.