acceptodds
Withdrawn from ICLR 2027

RC-DQN: A RISK-AWARE DUAL-CRITIC CONFORMAL DEEP Q-NETWORK FOR DEXMEDETOMIDINE DOSE TITRATION IN POSTOPERATIVE ICU PATIENTS

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Dexmedetomidine dose titration in postoperative intensive care unit patients is a sequential decision-making problem requiring a balance among prespecified treatment objectives, physiological tolerance, and action support. Applying offline reinforcement learning to this setting is complicated by the ordinal dose space, the scalarization of treatment utility and physiological cost, and extrapolation to actions that are poorly supported by retrospective data. We developed a Risk-Aware Dual-Critic Conformal Deep Q-Network (RC-DQN) using retrospective trajectories from two critical-care databases. Patient records were organized into six-hour decision windows, and dexmedetomidine exposure was represented by 16 ordered actions. RC-DQN combines an ordinal behavior-policy model, separate utility and physiological-cost-score critics, risk-modulated conformal calibration, an absolute cost-score shield, and rule-based fallback. The primary offline evaluation used terminal-outcome fitted Q evaluation (FQE) to estimate the mean initial-state value for the alive-without-delirium terminal outcome. An independent FQE evaluated returns under the prespecified composite reward, while action matching, ablation analysis, and bidirectional cross-database sensitivity analysis provided complementary assessments. Among the evaluated policies, RC-DQN produced the numerically highest within-database point estimates for both the alive-without-delirium endpoint and composite-reward return across both databases. Its endpoint estimates were higher than the corresponding observed test-cohort proportions, while action-matching analysis showed partial agreement with documented clinical decisions. Cross-database evaluation produced asymmetric results when source-trained policies and evaluators were applied to target-database states, indicating sensitivity to database-specific distributions. Overall, RC-DQN provides a structured approach for retrospectively learning and evaluating ordinal medication-titration policies under behavioral-support and physiological-cost constraints.

Primary area:
reinforcement learning
Keywords:
Offline reinforcement learning, Healthcare, Dexmedetomidine

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_UjbPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RC-DQN, an offline reinforcement learning method for titrating dexmedetomidine in postoperative ICU patients, using six-hour decision windows and 16 ordered dose levels. A shared network has three heads: a Q-function for long-term clinical benefit, a separate critic that scores physiological side-effect cost (bradycardia and hypotension), and a truncated-Poisson model of clinician dosing that respects dose ordering. At decision time, a conformal filter calibrated on held-out data removes doses that are poorly supported by clinician behavior or have high estimated cost. The method then picks the remaining dose with the best benefit-minus-cost score, with rule-based fallback to no infusion when no dose qualifies or a top dose looks unsafe. Evaluated with fitted Q evaluation on MIMIC-IV and eICU-CRD, RC-DQN achieves the highest estimated rate of survival without delirium and the highest composite return.

Critical strengths

None

Critical weaknesses

The method piles on many parts, but i0.26 percentage points on MIMIC (50.49% vs 50.23%) and 0.28 points on eICU, and the confidence intervals overlap. I'm not convinced the extra complexity pays off.

Adding cost to the conformal score is presented as the key novelty, but turning it off changes almost nothing, and one score even goes up. So it’s unclear which parts are actually doing the work.

I’m not sure the evaluation can tell good policies from bad ones. ConformalDQN almost never agrees with clinicians, yet it scores second. On eICU, the evaluator misses the real clinician outcome by about 6 points, which is bigger than any difference between methods.

Questions for authors

see weaknesses

1 comment
  1. Model complexity and contribution RC-DQN addresses offline dose selection by jointly considering treatment utility, historical action support, and predicted physiological cost. Its ordinal behavior model characterizes support across dose levels; a separate cost critic scores candidate actions; and conformal filtering, a cost threshold, and fallback rules determine the final action. In the in-database evaluation, RC-DQN had the numerically highest reported point estimates for both endpoint and total-reward FQE in each database. Endpoint estimates for RC-DQN versus ConformalDQN were 50.49% versus 50.23% on MIMIC-IV and 78.43% versus 78.15% on eICU. Action distributions under low heart rate and low blood pressure additionally describe how the policy responds in physiologically risky states. The contribution is the integration of ordinal support modeling, separate utility and cost scores, and constrained action selection; the offline estimates and action analyses characterize different aspects of this framework.
  2. Contribution of the risk-modulated conformal score The risk-modulated conformal score combines behavior probability and the cost-critic score to filter actions using historical support and predicted cost. To assess this component, we compared full RC-DQN with Ordinal Cost-Shielded Conformal (OCSC). OCSC reuses the full-model weights, removes the cost term from the conformal score, and recalibrates a behavior-only threshold on validation data; the cost threshold, cost-adjusted ranking, and fallback rules remain in place. Endpoint FQE for full RC-DQN versus OCSC was 50.49% versus 49.97% on MIMIC-IV and 78.43% versus 78.21% on eICU. On MIMIC-IV, total-reward FQE was 10.2554 versus 11.7565, and action-match rate was 50.17% versus 24.02%. Thus, adding cost to the conformal score changes policy behavior and produces different results across metrics. It isolates the score change and associated recalibration within a framework where the critic, cost threshold, ranking, and fallback remain active.
  3. Reliability of offline evaluation Action match and FQE measure different quantities: the former measures agreement with recorded clinician actions, while the latter estimates a target policy’s endpoint value over test initial states. Low agreement and a high FQE estimate are therefore not inherently contradictory. However, if a policy selects actions with insufficient support in the corresponding states, FQE relies more heavily on function approximation and extrapolation; neither action match nor FQE alone establishes true policy quality. We therefore interpret FQE as model-dependent estimates under the prespecified offline evaluation. RC-DQN has the numerically highest endpoint FQE point estimate in both databases. Action match and action distributions in risk states describe policy behavior; they are not direct evidence of clinical efficacy or safety.
Reviewer_5pQ9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RC-DQN to titrate dexmedetomidine doses for patients recovering in the ICU after surgery. It has three parts: doses are modeled as ordered levels; each dose is scored on both clinical benefit and physiological cost ; and safe fallback rules apply when needed. authors tested it on MIMIC-IV and eICU-CRD using FQE, baseline comparisons, ablations and policy behavior analysis. Within each database, RC-DQN scored slightly higher on FQE. Across databases, the results reverse direction .

Critical strengths

For postoperative ICU patients, dexmedetomidine dosing must be adjusted as the condition changes. This balance weighs therapeutic goals such as adequate sedation and delirium control, against risks like hypotension and bradycardia. The author models this as an offline sequential decision making problem, balancing long term treatment benefit, physiological cost and how closely care follows historical practice. The motivation is clear and the work has obvious clinical value.

Critical weaknesses

1.The paper demonstrates risk-responsive behavior through dose distributions and changes in action selection in states with low heart 1.This paper shows risk responsive behaviour: dose distributions and action choices shift when heart rate or blood pressure is low. But it does not measure physiological cost directly and it does not report how often constraints are violated. The present results therefore cannot show that the risk control mechanism really lowers physiological cost. 2.The proposed method beats the strongest baselines by only 0.26 and 0.17 percentage points on the two datasets. Its confidence intervals ignore uncertainty from policy training and from the evaluation estimate. This is weak evidence that the method is consistently better.

Questions for authors
  1. An independent offline evaluation of the predefined physiological cost is needed. Report two rates: how often the final actions including fallbacks, violate the constraints and how often the fallback is triggered. These rates show the effect of the risk control mechanism.
  2. After repeated policy training and FQE refitting, how large are the paired differences in estimated policy value between RC-DQN and the strongest baselines and how uncertain are they? Does the ranking stay stable? The current confidence intervals cannot show whether variation in policy training or evaluator fitting could change the ranking.
  3. How much extra benefit does the cost critic add compared with hard thresholds and fallback rules used alone?
  4. The estimand of the terminal outcome FQE is not specified. Do the reported percentages mean the undiscounted probability of being alive without delirium, or a discounted terminal return?
  5. The cross database endpoint ranges in Section 4.3 do not cover all models in Tables 3 and 4. State which subset of models is summarized or widen the ranges to cover all tabulated results.
1 comment
  1. Physiological cost and constraint violations We prespecified physiological costs associated with bradycardia and hypotension and incorporated predicted cost into action selection through the cost critic, threshold, and fallback rules. For the evaluated RC-DQN checkpoints, among 10,454 MIMIC-IV test windows, empty-set fallback occurred 0 times (0.00%), high-dose veto 3,274 times (31.32%), and final-action predicted cost exceeded the threshold 103 times (0.99%). Among 2,583 eICU windows, the corresponding counts were 92 (3.56%), 5 (0.19%), and 97 (3.76%). All rates use test windows as the denominator; threshold exceedance is measured after fallback. These are model-internal diagnostics: threshold exceedance uses the policy’s own cost critic.
  2. Policy comparison and uncertainty The primary endpoint is the initial-state FQE estimate of survival without delirium. In the original in-database evaluation, RC-DQN and the highest-scoring learned baseline achieved 50.49% versus 50.23% (ConformalDQN) on MIMIC-IV and 78.43% versus 78.26% (BCQ) on eICU: differences of 0.26 and approximately 0.17 percentage points. The reported episode-bootstrap intervals hold the policy and FQE evaluator fixed, capturing conditional test-episode sampling variability. To assess evaluator variability, we refitted FQE with eight seeds for each fixed policy. On MIMIC-IV, using another set of RC-DQN and ConformalDQN weights, mean estimates were 51.29% and 48.73%, with population standard deviations of 0.75 and 0.73 percentage points across FQE seeds. The paired difference was positive for every FQE seed and averaged 2.56 points. On eICU, the corresponding means for RC-DQN and ConformalDQN were 78.43% and 78.36%, a 0.07-point difference.
  3. Contribution of the cost critic The cost critic supplies state-dependent physiological-cost scores for candidate actions. These scores influence bootstrap action selection during training and enter the cost threshold, risk-modulated conformal score, utility–cost ranking, and high-dose fallback condition at inference. Thresholds and fallback rules then use the scores to govern selection. The existing ablation assesses a combined cost-aware configuration: endpoint FQE for Ordinal Conformal (OC) and Ordinal Cost-Shielded Conformal (OCSC) was 43.33% versus 49.97% on MIMIC-IV and 77.82% versus 78.21% on eICU. Because the comparison changes multiple mechanisms, including the critic, threshold, ranking, and fallback, it does not isolate the critic’s incremental contribution over hard rules and fallback without cost scores.
  4. Endpoint FQE estimand The endpoint FQE estimand is the expected undiscounted terminal indicator under a fixed policy. We use γ_FQE=1, terminal reward 1 for survival without delirium, and 0 for the other mutually exclusive outcomes and nonterminal steps. Episodes with delirium in the first window are excluded. Subsequent episodes terminate at first recorded delirium; those without recorded delirium continue to their original terminal window and are classified as death before delirium or alive without delirium. The reported percentage averages initial-state FQE values across eligible test episodes after clipping each to [0,1]. It is a model-based cohort-level estimate, not a calibrated individual probability or an observed causal effect.
  5. Cross-database result ranges We thank the reviewer for identifying the mismatch between the text and tables. The tables include all eight learned policies, but the original ranges did not cover every result. Using the full tables, endpoint FQE estimates range from 33.27% to 44.96% for MIMIC-IV to eICU (DDQN is the lower bound) and from 77.01% to 85.88% for eICU to MIMIC-IV (ACTIVE-IQL is the lower bound). In both directions, the source-fitted policy and FQE evaluator are applied together to target-database test states.
Reviewer_9xNtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes RC-DQN for offline dexmedetomidine dose selection in postoperative ICU patients. It combines a truncated-Poisson behavior model, separate utility and physiological-cost critics, conformal action filtering, and deterministic shielding and fallback rules.

Critical strengths
  • The method distinguishes behavioral support from physiological cost.

  • The paper reports results on two databases and acknowledges important limitations: conformal inclusion does not establish clinical safety, the bootstrap intervals condition on fixed evaluators, and cross-database experiments transport both the policy and evaluator.

Critical weaknesses
  • The validity of the decision process and policy-value estimates remains uncertain. Appendix A.2 aggregates clinical measurements within each six-hour window, while the action represents infusion during that window. It is unclear whether every state variable precedes the action or includes treatment responses. The construction of mortality and delirium risk features is also unspecified. These ambiguities matter because contemporaneous or future information could invalidate the intended decision process.

  • The contribution of the proposed constraints is not convincingly established. Equation (7) penalizes predicted costs above the threshold, thereby pushing those predictions downward; the same predictions subsequently determine admissibility. Although the authors explicitly describe this as score regularization, its interaction with the asymmetric cost loss needs validation. Conformal calibration pools dependent windows and lacks reported held-out coverage or action-set-size diagnostics.

Questions for authors
  1. At what exact time is each action selected? Do all state measurements precede the corresponding infusion window, and how are mortality and delirium risk features generated? Were imputation models fitted exclusively on training data?

  2. What discount factor and horizon are used for endpoint FQE? What explains the eICU clinician-policy discrepancy, and do the small policy differences persist across training seeds, evaluator refits, and paired comparisons?

  3. How does removing Equation (7) affect cost prediction, constraint violations, and policy value? How frequently do shielding and fallback activate, and is the cost critic evaluating the same policy ultimately deployed?

2 comments
  1. Decision timing and preprocessing We divided records into consecutive six-hour windows from ICU admission. The state aggregates clinical observations within each window, and the action is the dexmedetomidine infusion-rate category aggregated over the same window, potentially from multiple administrations. No individual within-window decision time is specified, and some observations may follow administration. The results therefore represent retrospective evaluation using contemporaneous records, not prospective decisions at window start. The hospital-provided death_risk and pnd_risk fields measure mortality and pain risk, respectively; we trained no separate risk models. Imputation preceded the data split: low-missingness variables used KNN, while some variables used within-stay forward fill and full-data column means. Thus, imputation was not fitted on training data alone. State and total-reward standardizers were fitted on the training split and applied to validation and test data.
  2. Endpoint FQE and policy comparison The primary endpoint is alive without delirium. Endpoint FQE uses γ_FQE=1, with terminal reward 1 for this outcome and 0 for other mutually exclusive outcomes and nonterminal steps. Episodes with delirium in the first window are excluded; subsequent episodes terminate at first recorded delirium or, if none is recorded, at their original terminal window, distinguishing death before delirium from survival without delirium. The horizon follows each ICU trajectory, not a fixed 30 days. The reported percentage averages eligible test initial-state FQE values, each clipped to [0,1], and estimates the undiscounted terminal indicator under a fixed policy. For the clinician policy, the paper reports 70.01% FQE versus 76.11% observed in the same eligible eICU cohort (−6.10 percentage points). Across ten FQE seeds , the estimate was 68.82% ± 0.62 percentage points versus 76.11% observed; on MIMIC-IV, it was 48.10% ± 0.35 versus 47.03% observed. For fixed policies and eight FQE seeds, MIMIC-IV estimates for RC-DQN and ConformalDQN were 51.29% ± 0.75 and 48.73% ± 0.73, respectively; the paired difference was positive for every FQE seed, averaging 2.56 percentage points. On eICU, the corresponding means were 78.43% and 78.36%, a 0.07-point difference.
  3. Cost regularizer, masking, and fallback Equation (7) penalizes predicted costs above threshold for recorded actions, alongside the asymmetric cost TD loss. To isolate its effect, we retrained full and no-Equation-(7) models, retaining the cost critic, bootstrap action rule, threshold, and inference rules; each configuration was recalibrated on validation data. On MIMIC-IV, with five policy seeds and three FQE seeds per policy, full versus ablated models had mean endpoint FQE of 49.76% versus 46.24% and total-reward FQE of 11.3421 versus 8.9058. The endpoint difference was not positive for every policy seed. Mean final-action predicted-cost violation rates were 2.18% versus 51.74%, and empty-set rates were 0.01% versus 49.84%. These differences show that removing Equation (7) changes model-internal cost scores and action filtering. As each model uses its own critic, they do not establish a difference in actual physiological cost. For the fixed checkpoints, conformal and cost masks excluded 16.40% and 3.21% of MIMIC-IV state–action candidates. Of 10,454 test windows, empty-set fallback occurred 0 times, high-dose veto 3,274 times (31.32%), and final-action threshold exceedance 103 times (0.99%). In eICU, the masks excluded 87.44% and 6.48% of candidates; among 2,583 windows, the corresponding counts were 92 (3.56%), 5 (0.19%), and 97 (3.76%). Mask fractions use candidates as the denominator; fallback and violation rates use windows, with violations measured after fallback. The cost critic bootstraps with actions selected by Q−λC, whereas the final policy additionally applies conformal and cost masks, ranking, and fallback. It supplies scores used to construct the final policy; it is not an independently fitted evaluator of that complete policy. Threshold violations use the model’s own scores.

Thank you for your response and the additional experiments, but I am still unsure whether this method can maintain its advantages if it only utilizes the information available before selecting the dosage. And there is no explanation for the gap between the clinician policy estimate and the observed rate in eICU. I tend to maintain the original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.