acceptodds
Under review as a conference paper at ICLR 2027

VasoDPO: Aligning Treatment Actions with Patient States through State-Fixed Preference Construction

Abstract

Clinical treatment generation aims to tailor actions to patients, yet retrospective electronic health records provide limited preference supervision over alternative actions under the same patient state. We study this problem through retrospective single-step generation of normalized norepinephrine actions for patients with sepsis in intensive care units. We propose VasoDPO, a state-fixed Direct Preference Optimization (DPO) framework that anchors chosen responses to recorded actions and uses mean arterial pressure (MAP) to construct rejected actions under the same patient state. After minimum action-gap filtering, VasoDPO upweights retained pairs contrasting a nonzero recorded action with a zero rejected action, motivated by near-zero concentration observed after supervised fine-tuning. Because favorable action-replacement train-on-synthetic, test-on-real (TSTR) scores can coexist with near-zero action concentration when state features are retained, we jointly evaluate TSTR with direct diagnostics of near-zero concentration, directional violations, and MAP sensitivity. Across five training seeds on a fixed split, VasoDPO achieves higher mean TSTR AUROC and AUPRC and lower mean near-zero and directional-violation rates than a matched DPO baseline using the same preference pairs with uniform weights. Matched ablations support the contributions of MAP-guided construction and omission weighting, while state–pair shuffling degrades both TSTR and action diagnostics. Controlled MAP sweeps show that the mean generated action varies with MAP while other inputs remain fixed.

Primary area:
foundation or frontier models, including LLMs
Keywords:
clinical treatment generation, direct preference optimization, large language models, electronic health records, critical care, sepsis

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_u4UxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper is written in the context of clinical treatment generation from retrospective EHR data, specifically for sepsis patients in ICUs. To apply LLMs in this setting, we need preference data of preferred and rejected actions but retrospective records only provide the administered action for each state, so there are no natural preference pairs to compare alternative actions under the same state. This work proposes VasoDPO, a DPO framework that uses the recorded action as the chosen response and constructs a MAP-guided rejected action under the same patient state. The paper also shows empirical results on ICU Sepsis dataset, showcasing that VasoDPO outperforms Standard-DPO and other LLM baselines (SimPO, KTO, SafeDPO) in train-on-synthetic, test-on-real mortality prediction and direct action diagnostics .

Critical strengths

The paper is well written and easy to follow. Experiments are thoughtfully designed to explore the behaviour of the generated actions beyond downstream predictive scores, through near-zero concentration, directional violations and a controlled MAP sweep.

Critical weaknesses

The paper doesn’t explain why LLMs are a reasonable modeling choice for predicting actions from states in this medical setting. The state only has four numeric features, which are serialized into text, and simple tabular models like Ridge and XGBoost perform comparably on several metrics

Questions for authors

Can you clarify how exactly mean arterial pressure (MAP) helps generate alternate rejected trajectories? Can you also give more evidence for why LLMs are suited for this task?

Reviewer_dveMOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper tries to predict norepinephrine doses for ICU sepsis patients with a fine-tuned LLM, using only four inputs. The authors find that simply training the model to copy clinicians makes it output zero almost every time. Their fix is DPO, where the clinician’s dose is the preferred answer and the rejected answer is a fake dose made from simple MAP rules, like zero when blood pressure is low or a very high dose when it’s high. They evaluate with a mortality prediction task that uses the generated doses, rates of near-zero and MAP-inconsistent doses, and a test where they vary MAP and watch how the dose changes. Their method does better than standard DPO and the other baselines on these measures.

Critical strengths

None

Critical weaknesses

1.Overall the novelty is limited. DPO, pair weighting and rule-based negatives all exist. This is a fix tuned for one dataset. 2. I’m also not sure why DPO and an 8B LLM are the right tools here. In practice this is supervised training on fixed pairs made by rules. There’s no reward and no environment. It’s predicting one number from four numbers. I’d want to see a simple tabular model with a loss that penalizes zeros, and some offline RL baseline, since that’s the standard approach for sepsis and the paper even cites those works. 3.The MAP sweep confused me. The dose goes up as MAP goes up, which seems backwards given the rules the method is built on. The paper doesn’t comment on this at all. 4.I’d really like to see NZR, LMW and CGVR computed on the real clinician doses. Right now there’s nothing to compare against. If doctors often give little or no drug at low MAP, then 1% near-zero isn’t “better”, it’s just different from practice. 5. The TSTR results didn’t convince me. The authors’ own control shows that giving everyone a dose of zero gets basically the same AUROC 0.676 (Table 4) as real data 0.675, Table 1. So the dose barely matters to the predictor, and a 0.02 gain doesn’t say much. Beating the real clinician doses on this metric actually makes me more suspicious, not less. There’s also no uncertainty over test patients, only over training seeds. 6. The state is very thin, only APACHE, time, heart rate and MAP. The model doesn’t even see the dose the patient is already on, which is probably the most important thing when titrating norepinephrine. Without it, MAP is hard to interpret, since a normal MAP might just mean the patient is on a high dose. There are no fluids or lactate either. Earlier sepsis papers the authors cite used far richer states. 7. I also am concerned anout treating every clinician dose as the right answer in section 3.3 (a⁺ = aᵢ). Some of those patients died, and outcomes are never used in training. So the model is just imitating clinicians, mistakes included, and it can’t be better than them. That doesn’t really match the “aligning treatment with patient states” framing. 8.VasoDPO’s strong NZR and CGVR results (Tables 1 and 2) are not surprising, since the model is trained to avoid exactly what these metrics count. The authors acknowledge the shared thresholds in Section 6, but most of the claimed improvement still depends on these numbers.

Questions for authors

see weaknesses

Reviewer_B1W6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes VasoDPO, which uses recorded norepinephrine doses and mean arterial pressure (MAP)-guided alternatives to construct preference pairs for training a clinical language model. It combines DPO with extra weighting of selected pairs and evaluates the generated doses through downstream mortality prediction and action diagnostics.

Critical strengths

My expertise is in causal inference. I am less familiar with preference optimization for clinical language models, so my assessment focuses primarily on the methodological motivation and interpretation of the evaluation.

  • The paper produces an interesting application of preference optimization to treatment actions when explicit preference labels are unavailable.
  • The paper shows useful matched comparisons that examine the contributions of mean arterial pressure (MAP)-guided construction and pair weighting, including a comparison with weighted supervised fine-tuning (SFT).
  • The paper clearly shows that favorable downstream prediction scores can coexist with nearly constant generated actions. Including direct action diagnostics is a useful contribution.
Critical weaknesses
  • The justification for the preference labels is not fully convincing to me. The recorded action is always treated as preferred, but the paper provides limited evidence that it is actually preferable to the constructed alternative for that patient. I think this can create a potential disadvantage to reinforcement learning (RL)-based recommendations (such as optimal dynamic treatment regime), where the optimal treatment recommendation does not necessarily match the optimal action.
  • The connection to causal treatment recommendation remains unclear, or at least unexplored. The paper does not establish causal identification or explain sufficiently how learning these constructed preferences relates to outcome-based policy learning and dynamic treatment regimes.
  • As consequence of the two points above, the evaluation does not establish better treatment decisions. The evaluation supports improvements in predictive performance and agreement with the specified action rules, but it remains unclear what these improvements imply for treatment quality. test-on-real (TSTR) scores do not evaluate patient outcomes under the generated doses, and the directional diagnostics partly reflect the rules used to construct training preferences. An independent assessment of the generated doses would strengthen the clinical motivation.
  • mean arterial pressure (MAP) is a reasonable variable to consider, but its relevance alone does not justify the specific rejection rules, thresholds, and sampling intervals. These choices need stronger support given that other patient characteristics do not directly inform the rejection rule.
  • The motivation for using a language model to predict a numerical dose from four clinical inputs remains underdeveloped. The comparisons are useful, but the practical benefit over simpler approaches needs clearer explanation.
Questions for authors
  • What evidence supports treating the recorded dose as preferable to each constructed alternative?
  • Is the intended final goal to reproduce clinician actions, enforce selected treatment rules, or improve patient outcomes? How should the contribution be understood relative to outcome-based policy learning and dynamic treatment regimes?
  • Could you evaluate the generated actions using criteria specified independently of the MAP-based preference construction?
  • How were the rejection thresholds and sampling intervals chosen, and how sensitive are the results to these choices and to incorporating other patient characteristics?
  • What is the main practical advantage of the language-model formulation over direct numerical dose prediction, given the small set of inputs avaliable here?
Reviewer_seqEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VasoDPO, a preference optimization framework for generating patient-specific norepinephrine actions. It constructs rejected actions using mean arterial pressure (MAP) while keeping patient states fixed, and upweights preference pairs involving omitted nonzero treatments. Experiments on an ICU sepsis dataset show improvements over standard DPO in downstream mortality prediction and action-level diagnostics.

Critical strengths
  1. The paper addresses a meaningful problem in clinical treatment generation. Constructing preference pairs under the same patient state is a reasonable approach to learning from retrospective records.

  2. The experiments include several relevant baselines and well-controlled ablations. The results consistently support the contributions of MAP-guided preference construction and omission-based weighting.

Critical weaknesses
  1. The methodological novelty is limited. The main contributions are heuristic rejected-action construction and pair reweighting within standard DPO.

  2. The evaluation does not establish whether the generated treatment actions are clinically appropriate. Downstream mortality prediction and threshold-based diagnostics are indirect measures of treatment quality.

  3. The experiments use only one dataset and a single data split. The robustness of the proposed approach across different patient populations and clinical settings remains unclear.

Questions for authors
  1. How sensitive are the results to the MAP thresholds and the heuristic rules used to construct rejected actions?

  2. Could the authors evaluate treatment quality more directly, for example through off-policy evaluation or comparisons with clinician actions?

  3. How does the proposed approach generalize to other clinical datasets or treatment types beyond norepinephrine dosing?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.