acceptodds
Under review as a conference paper at ICLR 2027

Equal Credit or Disease Priority? Dynamic Credit Routing in Offline Multi-Agent Reinforcement Learning for Comorbid Critical Care

Abstract

Clinical reinforcement learning (RL) has largely optimized treatment for a single disease, leaving cross-disease coordination in critically ill patients underexplored. Sepsis–acute kidney injury (AKI) exemplifies this challenge, where the same intervention may benefit one disease while worsening the other, and the clinically dominant disease may shift over time. This requires a policy to decide, at each moment, whose priority weight (i.e., Sepsis's or AKI's) should dominate learning, and how their competing treatment preferences are reconciled into a single action. We provide the first offline multi-agent reinforcement learning (MARL) formulation of comorbid care and introduce our model BIFOCAL, which coordinates disease-conditioned value branches through dynamic credit routing: a telescope (future deterioration risk) sets which disease deserves priority, a magnifying glass (present treatment conflict) determines when that priority takes effect, and their coupling decides which disease dominates learning and reconciles preferences into a single action. Across 4 diverse ICU datasets from U.S. and China comprising 87,566 trajectories with 1,285,977 decision points, BIFOCAL consistently achieves higher estimated policy value than clinician behavior. On MIMIC-IV, it further outperforms baselines in the comorbid cohort and two disease-specific methods in disease-only cohorts under 5 off-policy and 2 action-consistency metrics. Its lead on single-disease cohorts suggests that BIFOCAL's utility extends beyond comorbidity to single-disease care. Furthermore, a blinded physician evaluation of 30 comorbid cases over six dimensions supports its clinical potential. These results establish BIFOCAL as the first large-scale offline MARL framework for cross-disease treatment coordination in critical care.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Application to Healthcare, Treatment Planning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.80 over 5 reviews, above 27% of the others.
Reviewer_XkdtOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates offline reinforcement learning for managing patients with concurrent sepsis and acute kidney injury (AKI). The authors formulate the problem using two disease-conditioned value branches and propose BIFOCAL, which dynamically allocates learning credit based on two signals: predicted future deterioration risk determines which disease receives greater weight, while current treatment conflict determines when and how strongly asymmetric weighting is applied. At decision time, the same risk–conflict principle is used to fuse disease-specific Q-values along the shared fluid-action dimension.

Critical strengths

Separating direction from activation is a reasonable inductive bias for competing objectives. More importantly, the manuscript carefully delineates what the routing rule does: it modifies branch-specific optimization rates while preserving the forward additive value, recovers equal routing at zero conflict, and does not claim causal credit assignment. This precision makes the contribution substantially easier to assess and trust.

Critical weaknesses
  1. BIFOCAL’s strongest claim concerns improved estimated policy value. However, the policy executed in the paper relies on greedy Q fusion, whereas the OPE target-policy probabilities are derived from softmax-transformed Q-values. For the 125-action joint space, the behavior policy is likewise constructed from disease-specific behavior probabilities rather than explicitly estimated as a joint clinician policy. Given the sensitivity of importance-sampling OPE to the specification of both the target and behavior policies, these choices may alter method rankings. This concern is compounded by the lack of a clearly documented validation split, especially because the risk horizon is selected partly on the basis of downstream OPE performance.

This is especially important in clinical offline RL, where prior work has shown that conclusions can vary substantially with evaluation and MDP design choices. Proceedings of Machine Learning Research I would need an untouched final test evaluation, target-policy/temperature sensitivity, behavior-policy sensitivity, and paired uncertainty on the differences against the strongest baselines before treating the policy-value advantage as robust.

  1. BIFOCAL assigns semantic meaning to the risk-derived disease weights and then applies those weights directly to the AKI and Sepsis Q profiles. Yet the individual branch values of an additive decomposition are not uniquely identified in a way that automatically guarantees comparable scales. The paper's own sensitivity analysis states that correspondence between routing weights and actual Q influence deteriorates as branch scales diverge and explicitly notes that it does not establish globally aligned or identifiable Q-head scales. This matters because a nominal weight of, for example, 0.6 versus 0.4 does not necessarily imply 60/40 disease influence if one Q-head has substantially larger variation. A scale-invariant fusion rule, explicit calibration/normalization, or a convincing invariance analysis would materially strengthen the method.
Questions for authors
  1. What exact patient subsets are used to select the 24-hour risk horizon, conflict threshold c0c_0, routing temperature τr\tau_r, anchor coefficient, checkpoints, and other hyperparameters? Table 6 documents 60/40 train/test splits for the public datasets, while the 24-hour horizon is selected partly using downstream OPE. Was the final test cohort completely untouched during all such selections? Please also confirm that disease-specific expert pretraining, the risk module, and the conflict router never use patients assigned to the corresponding comorbidity test set.

  2. The executable BIFOCAL policy is greedy, whereas Appendix C.5 defines target probabilities through softmax-transformed Q-values. Are the reported WIS/WISt/WISb/WISbt/DR values therefore estimates for a stochastic surrogate policy rather than the deployed greedy policy? Please report sensitivity to the softmax temperature and explain why the chosen stochastic policy is the appropriate estimand.

4 comments
AuthorsOpenReview ↗

[1/4] Overview: greedy policy evaluated directly, joint behavior model tested, splits documented

Code (anonymous): https://anonymous.4open.science/r/BIFOCAL-7DFF/README.md

Thank you for the precise review, and for noting that the paper states what the routing rule does and does not do. We ran the checks you ask for. This overview follows your review point by point.

Where to find what. Part 2/4: target policy, behavior policy and the four checks you ask for (W1, Q2). Part 3/4: data splits and selection (Q1). Part 4/4: Q-head scales (W2).

Critical Weaknesses

  • W1, "the policy executed in the paper relies on greedy Q fusion, whereas the OPE target-policy probabilities are derived from softmax-transformed Q-values". Correct, and the greedy policy is now evaluated directly. Its WISt is 79.747 (95% interval 76.540 to 82.902), the highest of the six methods evaluated. Part 2/4.
  • W1, "the behavior policy is likewise constructed from disease-specific behavior probabilities rather than explicitly estimated as a joint clinician policy". Tested with a clinician model fitted jointly over the 125 actions. BIFOCAL stays first and changes least: 76.443 to 75.017, against drops of 5.1 to 19.1 for the five baselines. Part 2/4.
  • W1, "these choices may alter method rankings". BIFOCAL's rank does not change. It is first on truncated WIS under the softmax and the greedy target policy, and under the original and the joint behavior model. Part 2/4.
  • W1, "the lack of a clearly documented validation split". Now documented: the split unit, the patient overlap, and the patients used for each selection. Part 3/4.
  • W1, "I would need an untouched final test evaluation, target-policy/temperature sensitivity, behavior-policy sensitivity, and paired uncertainty on the differences against the strongest baselines". Each of the four is answered in part 2/4. The paired difference to the strongest baseline is positive in all three settings (+0.619, +0.344, +4.051), with 95% intervals that include zero. New results use a strict test subset on which no component was trained; a fully untouched re-split is not available in this response.
  • W2, "a nominal weight of, for example, 0.6 versus 0.4 does not necessarily imply 60/40 disease influence". Agreed: the weights set a direction of priority, and the paper claims only that (95.62% of changed decisions move toward the higher-weight disease). We ran the scale-normalized fusion you suggest, and it does not improve the policy (WISt 65.700 against 76.443). Part 4/4.

Questions For Authors

  • Q1, "Was the final test cohort completely untouched during all such selections?". Not completely. The split was made by ICU stay, so 1,936 of the 5,884 comorbid test patients have another stay in a training source, and the risk horizon and the checkpoint were selected with test-set metrics. The new greedy-policy results use a strict subset without these patients (4,069 of 6,392 stays), and BIFOCAL is first there. Part 3/4.
  • Q2, "Are the reported [...] values therefore estimates for a stochastic surrogate policy rather than the deployed greedy policy?". Yes. The softmax policy was used because a deterministic policy leaves few stays with non-zero importance weight. The greedy policy is now evaluated directly, and BIFOCAL is first under both. Part 2/4.

What holds after these checks. On the comorbid cohort, BIFOCAL is first on truncated WIS in each of the three settings: the softmax policy of the submission, the joint behavior model, and the greedy policy on the strict subset.

AuthorsOpenReview ↗

[2/4] W1 and Q2: greedy policy evaluated directly; BIFOCAL first under both target policies and both behavior models

W1 and Q2. "the policy executed in the paper relies on greedy Q fusion, whereas the OPE target-policy probabilities are derived from softmax-transformed Q-values" and "Are the reported WIS/WISt/WISb/WISbt/DR values therefore estimates for a stochastic surrogate policy rather than the deployed greedy policy? Please report sensitivity to the softmax temperature and explain why the chosen stochastic policy is the appropriate estimand."

Answer: yes, the values in the submission are for softmax policies. We have now evaluated the deployed greedy policy directly: BIFOCAL has the highest truncated WIS of the six methods evaluated (79.747). It also stays first, and changes least, when the behavior policy is replaced by a jointly fitted clinician model.

  • Why a softmax policy was used. Importance sampling gives zero weight to every stay that departs once from a deterministic policy, so few stays carry the estimate. A softmax over the Q-values keeps more of them (4,174 of 6,392 effective stays after truncation). The temperature is 1 for all six learned methods (App. C.5). The softmax policy is a means of comparing methods with adequate support. The policy that would be deployed is the greedy one, so we now evaluate it directly.
  • The greedy policy, evaluated directly (new). Strict test subset (3,948 patients, 4,069 ICU stays, none used to train any component; part 3/4), a single training seed per method, WISt with cap 100:
MethodWISt, greedy policy
BIFOCAL79.747
dBCQ79.128
Implicit Q-Learning77.691
Dynamic-weights CN77.054
BC76.840
CQL74.164

BIFOCAL's 95% interval is 76.540 to 82.902, with 746 effective stays after truncation (18%).

The four checks you ask for

  • 1. Untouched final test evaluation. The table above uses a strict subset on which no component was trained. A fully untouched re-split is not available in this response (part 3/4).
  • 2. Target-policy and temperature sensitivity. We report the two ends of the temperature range. At temperature 1 (Table 1(a)), BIFOCAL is first among the six learned methods with WISt 76.443. At temperature zero, which is the greedy policy, it is first with 79.747. The greedy estimate is also stable across truncation caps: 78.883, 79.357, 79.747 and 80.763 at caps 1, 10, 100 and 1,000.
  • 3. Behavior-policy sensitivity (new). We replaced the behavior model of the submission (two disease-specific models combined; App. C.5) with a behavior cloning model fitted jointly over the 125 actions, keeping the policies and the test set of Table 1(a) fixed:
MethodWISt, original behavior modelWISt, joint behavior modelChange
BIFOCAL76.44375.017-1.4
CF-CQL76.09970.966-5.1
Independent Q-Learning75.94770.321-5.6
QMIX72.53962.012-10.5
B3C72.60558.699-13.9
VDN65.98846.914-19.1

BIFOCAL stays first and changes least, and the order of the first three methods is unchanged. The weights are more concentrated under this model (BIFOCAL's truncated effective sample size is 204), which is why the corresponding interval below is wider.

  • 4. Paired uncertainty against the strongest baseline (new). Patient-level paired bootstrap, 2,000 resamples; BIFOCAL minus the baseline, with 95% interval:

    • Greedy policy, strict subset, vs dBCQ: +0.619 (-2.035 to 3.386).
    • Table 1(a), vs CF-CQL: +0.344 (-0.568 to 1.282).
    • Joint behavior model, vs CF-CQL: +4.051 (-1.229 to 9.902).

    The difference is positive in all three settings, and the intervals include zero. What we claim is therefore a consistent ordering across target policies and behavior models, not a statistically separated gap.

On the comorbid cohort, in each of the three settings (the softmax policy of the submission, the joint behavior model, and the greedy policy on the strict subset), BIFOCAL is first on truncated WIS among the methods compared.

AuthorsOpenReview ↗

[3/4] Q1: split unit, overlap counts and selection subsets; strict subset used for new results

Q1. "What exact patient subsets are used to select the 24-hour risk horizon, conflict threshold, routing temperature, anchor coefficient, checkpoints, and other hyperparameters? [...] Was the final test cohort completely untouched during all such selections? Please also confirm that disease-specific expert pretraining, the risk module, and the conflict router never use patients assigned to the corresponding comorbidity test set."

Answer: not completely, and we give the exact numbers. The split was made by ICU stay, so some comorbid test patients have another stay in a training source, and two selections used test-set metrics. All new greedy-policy results use a strict subset from which these patients are removed, and BIFOCAL is first there.

  • Split unit and overlap. The 60/40 split of Table 6 was made by ICU-stay ID. A patient with several stays can therefore fall on both sides of the split, or in a single-disease cohort and in the comorbid cohort. Of the 5,884 comorbid test patients (6,392 ICU stays), 1,936 also appear in a training source.
  • The components you name. Comorbid test patients who also appear in the training data of each: AKI expert pretraining 1,126; sepsis expert pretraining 528; risk module 653; conflict router and joint policy 907. The groups overlap. So we cannot give the confirmation you ask for on the original test set.
  • Strict subset. Removing all 1,936 patients leaves 3,948 patients, 4,069 ICU stays and 46,470 decision points, none used to train any component. On this subset BIFOCAL has the highest WISt of the six methods evaluated (79.747; part 2/4).
  • What was selected, and on which patients.
    • 24-hour risk horizon: eight horizons from 4 to 32 hours, compared on the original test set by prediction correlation, NMSE and downstream OPE (App. A.4).
    • Conflict threshold, routing temperature and anchor coefficient: 0.5, 1 and 1, as in the saved configuration. We have no selection record for these three values.
    • Checkpoint: 5 training epochs. Its saved selection statistic is WIS on the original test set.
    • Nothing was re-tuned for this response. The settings and the checkpoint are those of the submission.
  • Untouched final test. The strict subset removes the patient overlap; it does not change how the settings above were chosen. An evaluation on a new patient-level split, with every component retrained and selected inside the training data, is not available in this response.

The revision states the split unit and these counts in Table 6.

AuthorsOpenReview ↗

[4/4] W2: weights set a direction of priority; scale-normalized fusion tested

W2. "a nominal weight of, for example, 0.6 versus 0.4 does not necessarily imply 60/40 disease influence if one Q-head has substantially larger variation. A scale-invariant fusion rule, explicit calibration/normalization, or a convincing invariance analysis would materially strengthen the method."

Answer: we agree on the reading. The weights set a direction of priority, not a 60/40 share of influence, and the paper claims only the former. We ran the scale-normalized fusion you suggest: it does not improve the policy (WISt 65.700, against 76.443 for the fusion rule of the paper), so we keep fusion on the raw Q scale, where both heads are expressed in the same units of return.

  • Why the two heads are on a comparable scale. As you note, the TD loss alone fixes only the sum of the two branch values. Their scales are tied by construction instead: both heads start from disease experts trained on the same reward (terminal reward of plus or minus 100, discount 0.99), and the anchor keeps each head close to its expert, most strongly where conflict is low.
  • What the weights do: set a direction. With both Q-heads held fixed, replacing equal weights by the adaptive weights changes the fluid decision in 3.19% of 7,874 high-conflict decisions, and 95.62% of these changes (95% interval 92.89 to 97.93) move toward the fluid level preferred by the higher-weight disease (App. A.6).
  • What we do not claim: a proportion. The paper's own sensitivity analysis (App. Fig. 11), which you cite, already states that it does not establish aligned or identifiable Q-head scales. Remark 1 makes the same point for learning: equal credit means an equal gradient coefficient, not equal gradient norms.
  • Scale-normalized fusion (new). The model and the routing weights are frozen. In each state, the 25 Q-values of each disease head are standardized (z-score) before fusion. Nothing else changes.
    • Decisions: the joint greedy action changes in 7.051% of states. The greedy action of each head taken alone is unchanged.
    • Estimated value, setting of Table 1(a): WISt falls from 76.443 to 65.700, paired difference -10.743 (95% interval -14.646 to -7.099).
  • Why raw-scale fusion is the better choice here. Both heads estimate the same quantity, the expected discounted 90-day survival return. A wide spread across a head's actions means that the choice matters more for that disease in this state. Standardizing each head in each state removes exactly this information: a head that is almost indifferent between its actions gets the same say as one for which the choice is decisive. Standardization also flattens the softmax policy used in OPE, so we read the result as "normalization does not help", not as a precise effect size.

In short: the weights decide which disease has priority, the shared reward scale and the anchor keep the two heads comparable, and normalizing the heads did not improve on this.

Reviewer_WjTgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose BIFOCAL, an offline multi-agent reinforcement learning algorithm that determines which disease is posing a higher risk for a patient and selects the treatment targeting that disease when the best treatments for the two diseases conflict. Their model first learns an encoder that maps history to a representation that is predictive of future state and risk. Then, it uses the predicted risk from this representation in a weighted sum of the Q-functions when the best action from the two Q-functions conflict. The authors demonstrate their model outperforms multi-agent RL baselines on an off-policy estimate of the reward, treatment discrepancy, and action consistency.

Critical strengths
  1. The paper is well-written. The introduction frames the motivation for the work nicely. A treatment that helps for sepsis may worsen the burden for AKI, so being able to determine which disease takes higher precedence is important when making a treatment decision.
  2. The authors solve this problem with a novel creative solution that uses ML to predict the risks posed by each disease and when the recommended actions would conflict. The encoder-decoder model for predicting future risk from history is interesting. Training a model to predict when the 2 Q-functions for the 2 diseases conflict by supervising with the Jensen-Shannon divergence of the distributions from applying softmax to the Q-values is also interesting.
  3. The experiments compare their model to a large number of multi-agent RL methods. The authors also compare to ablations of the risk and conflict components in their model. The analysis of the policies for the 2 diseases in Figure 5 and the examples in Figures 6 and 7 show the authors put some thought into the clinical implications. It's interesting to see in Figure 6 how conflict is high either when the two diseases prefer opposite fluid directions or aggressive action is paired with unstable patient state!
Critical weaknesses
  1. Why does a separate model need to be computed to predict when there is a conflict? Can't that be determined just by comparing the actions that maximize the Q-values?
  2. The authors explain in the paragraphs after equation 3 that they intentionally only used the weighted sum of the Q-values in the action selection step when there is a conflict but not when computing the TD target in equation 2. Why can't the weighted sum also be used in equation 2? The paragraph also mentions the difference between these 2 sums only comes into play in 0.339% of states. Does it really make a difference then? Is there an ablation evaluating this? Also, I don't quite follow how using the weighted sum when there is no conflict would be an issue. If a=argmaxaQ1(s,a)a = argmax_a Q_1(s, a) and a=argmaxaQ2(s,a)a = argmax_a Q_2(s, a), wouldn't a=argmaxaw1Q1(s,a)+w2Q2(s,a)a = argmax_a w_1 Q_1(s, a) + w_2 Q_2(s, a) for any w1>0,w2>0w_1 > 0, w_2 > 0?
  3. Deterioration targets specific to sepsis and AKI are used to supervise the future risk predictor, but only 90-day mortality is used in the reward when learning the Q-function for both sepsis and AKI. If the disease-specific deterioration targets were used to define disease-specific reward functions, I imagine the Q-functions would better reflect the individual diseases and there may be more conflicts. This would likely make their method more useful. What do the results look like with this kind of reward function?
  4. How does BIFOCAL compare to single-agent RL? Since the two diseases share the same reward function, the experiments should also compare to learning a single Q-function for the combined action space as in prior RL work that uses the MIMIC data.
Questions for authors

In addition to the questions in the weaknesses section above:

  1. Can you show an example trajectory where we can see multiple points where a conflict was encountered, BIFOCAL picked different actions compared to a model that doesn't consider risk or conflict, and that led to different outcomes?

  2. In Table 1, why is minimizing treatment discrepancy and maximizing action consistency with the observed data good? Given that the evaluated reward for the clinician policy is lower, wouldn't you want some discrepancy from the observed data where the BIFOCAL policy is genuinely better? I understand that agreeing with clinical practice could indicate that the recommended action is safe, but perhaps different metrics to evaluate safety would be better for capturing that?

  3. Is training an encoder for the history to predict future risk needed or is supervised learning of a model that predicts future risk directly from history sufficient? Can the future state encoder-decoder and hidden state decoder be ablated?

6 comments
AuthorsOpenReview ↗

[1/6] Overview: all seven points answered, with new single-agent baselines and code

Code (anonymous): https://anonymous.4open.science/r/BIFOCAL-7DFF/README.md

Thank you for the careful reading and for your positive assessment of the motivation and the mechanism. We answer all seven of your points. W1 to W4 are your Critical Weaknesses 1 to 4, and Q5 to Q7 are your Questions 5 to 7. In brief:

  • W4. How does BIFOCAL compare to single-agent RL? We added three single-model methods, and BIFOCAL ranks first among the four methods compared here on both estimators (FQE 72.464 vs 69.222 for the best of the three; WISt 79.747 vs 77.691). See part 2/6.
  • W1. Can conflict be found by comparing the two best actions? No. That comparison only tells whether the two top choices differ. It cannot tell how serious the disagreement is, and it misses contested patients whose two top choices happen to coincide: in Fig. 6 Case 4 both branches pick fluid level 4, yet the conflict score is 0.823. See part 3/6.
  • W2. Why not put the weighted sum into the TD target? Because the model would then learn the wrong thing. The target teaches "how likely is this patient to survive". With our weights inside it, the model would learn a number bent by our own risk estimates, and every error in those estimates would be copied into the values and accumulate step after step. Your formula is correct and consistent with the method. See part 3/6.
  • W3. Why not disease-specific rewards? Because each branch would then be rewarded for making its own disease's numbers look better, even when that is worse for the patient. For example, fast fluid removal by ultrafiltration improves the AKI fluid balance but can drop blood pressure in a patient with septic shock, and faster net ultrafiltration is associated with lower survival (Murugan et al., JAMA Netw Open 2019). An AKI-only reward would pay for it. With one shared reward, both branches are rewarded only if the patient survives. When we tried disease proxy rewards in a single model, FQE fell from 68.266 to 60.369. See part 4/6.
  • Q5. Can we show a trajectory where a different action led to a different outcome? Not with retrospective data. Each patient received only the clinicians' treatment, so what would have happened to the same patient under BIFOCAL's choice cannot be observed, and no clinically validated simulator of sepsis-AKI exists to predict it. We show where BIFOCAL chooses differently, how estimated value changes across patient strata, and how six ICU physicians, the intended users, judged its recommendations on 30 real cases. See part 5/6.
  • Q6. Why is agreement with clinicians good? It is not a goal in itself, and we agree with you. Clinician actions are a reference, not ground truth. See part 5/6.
  • Q7. Is the future-prediction encoder needed, or is direct supervised risk prediction enough? We have not run that direct ablation. Table 3 compares routing with true future labels against routing with predicted risk (WISt 71.412 vs 76.443), but the future-label variant has low policy support. This comparison is exploratory and does not establish whether the future-prediction encoder is needed. See part 6/6.

The design in two sentences. BIFOCAL keeps one value branch per disease, sharing the fluid action and one outcome (90-day survival), and its core mechanism, conflict-gated credit routing, splits each shared learning error between the branches: predicted 24-hour risk sets which disease receives more, and the current disagreement over fluid sets how much. It suits high-risk decisions because it changes a decision only when the two diseases truly conflict, moves it in a predictable direction (toward the disease at higher predicted risk), and shows the clinician the reason.

AuthorsOpenReview ↗

[2/6] W4 Single-agent RL: BIFOCAL ranks first vs three single-model methods (new results)

W4 (your Weakness 4). "How does BIFOCAL compare to single-agent RL? [...] the experiments should also compare to learning a single Q-function for the combined action space"

Answer: against three single-model methods, two of which learn exactly what you describe, a single Q-function over the 125 combined actions, BIFOCAL ranks first on both estimators. This addresses your concern that the paper lacked a comparison with single-agent RL over the combined action space.

Setup: all methods were trained on the comorbid cohort and evaluated as greedy policies on the same strict test subset: the MIMIC-IV comorbid test set after excluding every patient used to train any component (3,948 patients, 4,069 ICU stays, 46,470 decision points). This subset had been inspected in earlier analyses, so we do not describe it as untouched. Each method is shown for a single training seed and a single FQE evaluator seed. CQL and Implicit Q-Learning are centralized single-agent models over the 125 joint actions (fluid x ultrafiltration x vasopressor, 5 x 5 x 5), with no disease branches; Implicit Q-Learning uses twin critics, both over the full joint action. CN is an offline adaptation of the dynamic-weights network of Abels et al.: one joint-action Q-network that receives dynamic priority weights. Implicit Q-Learning is not the "IQL" of our Table 1, which is Independent Q-Learning.

A. BIFOCAL vs single-model methods (new results)

MethodFQE (new)WISt, cap 100Parameters
BIFOCAL72.46479.747427K
CQL (new baseline)69.22274.164251K
Implicit Q-Learning (new baseline)59.22777.691251K
Dynamic-weights CN (new baseline)68.26677.054424K

FQE uses no importance weights. WISt is truncated weighted importance sampling. BIFOCAL's WISt has a 95% interval of 76.540 to 82.902.

B. BIFOCAL vs multi-agent baselines (Table 1(a) of the submission, 6,392 stays, with one new column)

MethodWISt, cap 100ESStESS before truncation (new)
BIFOCAL76.4434,1741.06
CF-CQL76.0994,1251.02
Independent Q-Learning75.9474,3061.26
B3C72.6053,1771.02
QMIX72.5391,4921.07
VDN65.988821.00

What the numbers say:

  • Against the best of these three single models on each estimator: +3.2 FQE (vs CQL) and +2.1 WISt (vs Implicit Q-Learning).
  • The closest test of "one joint model instead of two branches": the CN has the same size (424K vs 427K parameters) and also receives priority weights, but has no disease branches. It is lower on both estimators (-4.2 FQE, -2.7 WISt).
  • The two estimators order the baselines differently (Implicit Q-Learning is second on WISt and last on FQE), which is why we report both. BIFOCAL is first on each.

This suggests that sharing one reward does not make a single centralized model sufficient: at the same model size, removing the disease branches lowers estimated value.

Scope: BIFOCAL's branches start from disease-expert pretraining (Sec. 2.1), which the single models do not use, so training budgets are not matched. All values are offline estimates.

AuthorsOpenReview ↗

[3/6] W1 and W2: why conflict is graded, and why weights stay out of the TD target

W1 (your Weakness 1). "Why does a separate model need to be computed to predict when there is a conflict? Can't that be determined just by comparing the actions that maximize the Q-values?"

Answer: no. Comparing the two best actions only tells whether the top choices differ. It cannot tell how serious the disagreement is, and it misses contested patients whose two top choices happen to coincide.

  • We need the size of the disagreement, not a yes or no. The conflict score sets how much extra learning weight the higher-risk disease receives (Eq. 1): none when the diseases agree, more as they disagree more. A comparison of best actions has only two states, so routing could only be fully on or fully off. Def. 1 therefore measures conflict over the whole preference profile across the five fluid levels (Jensen-Shannon divergence, from 0 to 1).
  • The same best action can still be contested. In Fig. 6 Case 4 both branches pick fluid level 4, yet the conflict score is 0.823: an aggressive fluid choice in a patient with creatinine 9.6 and BUN 63. A comparison of best actions would report no conflict. This is the pattern you point out in Fig. 6 yourself.
  • Why a small learned model. The router reads the patient-state representations of the two pretrained encoders together with their proposals, so its score is smoothed across similar patients. A number computed from one state's raw Q-values is noisy in offline learning. The router is trained once on labels from the frozen disease experts (25.1 s) and then frozen.
  • Evidence that it measures what it should. Predicted conflict deciles track the observed divergence between the branches' fluid preferences (Fig. 2b), and the score correlates 0.77 with the disagreement of the final trained branches (App. A.3).

W2 (your Weakness 2). "Why can't the weighted sum also be used in equation 2? [...] the difference between these 2 sums only comes into play in 0.339% of states. Does it really make a difference then? Is there an ablation evaluating this?" and your formula on the no-conflict case.

Answer: because the model would then learn the wrong thing. The TD target teaches "how likely is this patient to survive". With the weights inside it, the model would learn a number bent by our own risk estimates.

  • What goes wrong if the weights enter the target. First, the values stop meaning survival: a weighted sum with weights adding to 1 is a different quantity (at equal weights it is half of the joint value). Second, errors accumulate: the weights come from a frozen risk predictor and conflict router, and inside the target every error they make is copied into the value and passed back to all earlier steps through bootstrapping. Third, the property of Prop. 1 is lost: routing as in Eq. 2 changes how fast each branch learns without adding or removing zero-error solutions of the TD term.
  • What the weights do instead. One job only: when the two diseases disagree on the shared fluid action, they decide which one has the final say.
  • Your formula is correct. If one action maximizes both Q1 and Q2, it also maximizes w1 Q1 + w2 Q2 for any positive weights. The method agrees: in such states the weights do not change the chosen action. The low-conflict gate (equal weights when the conflict score is below 0.5) matters only where the best actions differ but the disagreement is mild. There it stops a risk difference alone from tipping a near-consensus.
  • Risk-only routing during learning. Risk-only routing reweights the gradients in every state, including states where the branches agree. Its estimated WISt is 66.802, compared with 76.443 with conflict gating (Table 3). Given Risk-only’s low policy support, we treat this comparison as exploratory and do not interpret the gap as an effect size.
  • Does 0.339% matter? It is small on purpose: fusion touches only contested decisions. The fused and additive rules pick the same action in 99.661% of test states. Among 7,874 contested fluid decisions fusion changes 3.19%, and 95.62% of those move toward the prioritized disease (App. A.6). Its effect on aggregate value is correspondingly small (DR 76.230 for gated fusion, 75.767 to 75.969 for the other Q-level rules; Table 5). Table 3 also compares routing during learning with equal-weight learning (WISt 76.443 vs 62.570). Given the equal-weight variant’s low policy support, this comparison is exploratory and does not reliably quantify the benefit of routing during learning.
  • Is there an ablation? Table 5 varies the decision rule with learning held fixed. The paper has no ablation that changes the TD target. A milder variant would keep the unweighted sum as the value and only pick the next action with the fused rule; since the two rules agree in 99.661% of states, we expect a small effect, and we have not tested it.
Reviewer_tw5dOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

First of all RL is not my primary research area, and my opinions may need to be further cross-validated with other reviewers' comments.

After reading the paper, my biggest concern is whether the proposed method actually makes a meaningful contribution or unnecessarily complicates the problem, given that simpler and more conventional methods exist to address it. The baselines chosen by the authors are mostly based on multi-agent formulations, while conventional alternatives are largely overlooked. As a result, the question of whether multi-agent RL is actually necessary remains unaddressed, which raises concerns about the motivation for the proposed method and its effectiveness, especially given the currently reported results (the current results and evaluations raise further concerns about the reliability of the proposed method).

Critical strengths

Overall writing is clear.

Critical weaknesses

CW1.

One concern arising directly from the results reported in Tables 1 and 2 is that the truncated estimators show a dramatic drop in estimated policy values. This naturally raises questions about the reliability of the proposed estimates and their sensitivity to importance-sampling weights. Since the authors do not report uncertainty estimates and truncation may introduce additional bias because the weights are modified, some sensitivity analysis should be conduced for this.

CW2.

See Question below.

Questions for authors

Q1. Please explain why multi-agent RL is necessary in this research setting

Why not just use one model to consider both diseases simultaneously? Dealing with competing treatment priorities does not necessarily imply that separated disease-specifics agents are needed. See these papers:

(1) Kumar, Aviral, et al. "Conservative q-learning for offline reinforcement learning." Advances in neural information processing systems 33 (2020): 1179-1191.

(2) Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. "Offline reinforcement learning with implicit q-learning." arXiv preprint arXiv:2110.06169 (2021).

(3) Abels, Axel, et al. "Dynamic weights in multi-objective deep reinforcement learning." International conference on machine learning. PMLR, 2019.

Given the potential additional computational costs and implementation complexity, if multi-agent could not provide practical advantages over these simpler methods, it is quite meaningless for clinical settings which are of high-risk decision makings. If the authors could provide sufficient evidence to justify that multi agent formulation does have advantages in such setups, I'm happy to further discuss and change my score.

8 comments
AuthorsOpenReview ↗

Q1 Why multi-agent? Direct advantage over single-model methods (overview and code)

Code (anonymous): https://anonymous.4open.science/r/BIFOCAL-7DFF/README.md

Thank you for the suggestion. We have added direct comparisons with all three single-model methods you named, and the results support our disease-branch formulation.

Short answer. BIFOCAL has a direct advantage over the single-model methods you listed, in three respects: (1) higher estimated policy value than CQL, Implicit Q-Learning and a dynamic-weights network, the comparisons you requested; (2) a design suited to high-risk decisions; (3) a mechanism that can extend to diverse patient populations. These benefits come from our specific design, conflict-gated credit routing, and not from the multi-agent form itself.

Cross-check with the other reviews, as you suggested. On the points you were unsure about, the other reviewers write the following.

  • Is the method a meaningful contribution or an unnecessary complication? Reviewer btWu lists as a strength: "A meaningful coordination problem and an intuitive mechanism". Reviewer WjTg calls the method "a novel creative solution". Reviewer Xkdt: "Separating direction from activation is a reasonable inductive bias for competing objectives."
  • Is it credible for high-risk clinical use? Reviewer S6Ln writes that the blinded clinician evaluation "goes above and beyond for the venue and should be considered a serious point in the favour of the paper". Reviewer btWu notes that the physicians' ratings "inform plausibility and interpretability".
  • Single-model baselines. Reviewers S6Ln and WjTg ask the same question you do. The comparison in part 2/5 answers all three.
  • Reliability of the estimates. Reviewer S6Ln notes that evaluation “is always the hardest task in offline, off-policy RL evaluation for clinical work”. Several of the diagnostics these reviewers request (ESS, cap sensitivity, intervals, FQE) are the ones we report in parts 2/5 and 5/5.

Motivation and design. Our motivation is to solve a hard decision-making problem in clinical care; the mechanism can also generalize to other settings where conflicting objectives act on one subject (controlled non-clinical study: App. A.5). In sepsis-AKI, fluid restores septic perfusion but adds to AKI fluid burden, the more urgent disease shifts over time, and guidelines leave the trade-off open: SSC 2021 gives no recommendation between restrictive and liberal fluids after initial resuscitation (Evans et al., 2021; see also Zarbock et al., 2023). BIFOCAL keeps one value branch per disease, sharing the fluid action and one outcome (90-day survival). Its core mechanism, conflict-gated credit routing, splits each shared learning error between the branches: predicted 24-hour risk sets which disease receives more, and the current disagreement over fluid sets how much. The branches are the means; routing is the core contribution.

Parts 2/5 to 4/5 give the evidence for (1) to (3). Part 5/5 answers CW1.

AuthorsOpenReview ↗

Q1 Why not one model? BIFOCAL ranks first vs CQL, Implicit Q-Learning and dynamic-weights network

Result: BIFOCAL ranks first on both estimators against CQL, Implicit Q-Learning and the dynamic-weights network.

Setup: all methods were trained on the comorbid cohort and evaluated as greedy policies on the same strict test subset: the MIMIC-IV comorbid test set after excluding every patient used to train any component (3,948 patients, 4,069 ICU stays, 46,470 decision points). This subset had been inspected in earlier analyses, so we do not describe it as untouched. Implicit Q-Learning (Kostrikov et al.) is not the "IQL" of our Table 1, which is Independent Q-Learning. CN is an offline adaptation of Abels et al.: one joint-action Q-network that receives dynamic priority weights. Seeds: each method is shown for a single training seed and a single FQE evaluator seed.

Estimated value (FQE uses no importance weights; WISt is truncated WIS with cap 100):

  • BIFOCAL: FQE 72.464; WISt 79.747 (95% CI 76.540-82.902)
  • CQL: FQE 69.222; WISt 74.164
  • Implicit Q-Learning: FQE 59.227; WISt 77.691
  • Dynamic-weights CN: FQE 68.266; WISt 77.054

What the numbers say:

  • Against the best of these three single models on each estimator: +3.2 FQE (vs CQL) and +2.1 WISt (vs Implicit Q-Learning).
  • The closest test of "why not one model?": the CN has the same size (424K vs 427K parameters) and also receives priority weights, but has no disease branches. It is lower on both estimators (-4.2 FQE, -2.7 WISt).
  • On the original comorbid test set, BIFOCAL's WISt (76.443) and WIS (87.700) are also higher than all five baselines (see part 5/5 for why untruncated WIS is a reference only).

This suggests that the disease branches are not an unnecessary complication: a single model of the same size, given a priority signal as input, does not reach the same value.

AuthorsOpenReview ↗

Q1 High-risk clinical use: conservative, stable in the sickest patients, inspectable

A decision-support tool for high-risk care should be conservative, dependable in the sickest patients, and open to clinician scrutiny. The routing design provides all three.

  • Conservative: it changes decisions only where the two diseases conflict, and in a predictable direction. A tool that overrides clinical practice broadly is hard to trust in high-risk care. Routing departs from equal weights only under conflict. Relative to plain additive action selection, the fused rule changes the action in 0.339% of test states. Among 7,874 high-conflict decisions it changes the fluid choice in 3.19%, and 95.62% of those changes move toward the prioritized disease (App. A.6).
  • Dependable where the stakes are highest: performance holds in the most severe patients. A method that is good on average but fails in severe or shocked patients would not be acceptable. Truncated bootstrap WIS stays between 74.3 (high shock) and 81.7 (mild) across strata of severity, shock, disease dominance and conflict (App. Fig. 8). The gain over VDN (the same two branches without routing) is largest in the hardest strata: 7.28 severe vs 0.84 mild; 5.59 high-shock vs 0.47 low-shock; 6.51 balanced vs 4.19 single-disease-dominant.
  • Open to scrutiny: the clinician can see why a recommendation is made. A high-risk recommendation has to be checkable before it is acted on. Every recommendation exposes each disease's fluid preference, the conflict score and the priority weights; a single joint Q-network over 125 actions does not contain these quantities. Six ICU physicians from four hospitals, the intended users, judged disease priority for 30 comorbid cases before seeing the model. BIFOCAL's priorities were consistent with their judgments, and they rated priority appropriateness and interpretability at 5.5-6.3 out of 7 (App. C.6). This assessment is independent of OPE. It concerns plausibility and usefulness as decision support, and it is not a prospective efficacy study.

Additional evidence on the routing mechanism:

  • With the same branches, Table 3 reports WISt 76.443 with routing and 62.570 with equal weights. Given the equal-weight variant’s low policy support, we treat this comparison as exploratory and do not interpret the gap as an effect size.
  • Standard multi-agent baselines without routing are below the clinician reference of 76.047 (VDN 65.988; QMIX 72.539).
  • Independent of OPE: one-step TD-error reduction in high-conflict states is 1.34 with routing vs 1.25 with equal weights (Fig. 2c).
Reviewer_S6LnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors develop a multi-agent reinforcement learning (MARL) framework for multi-disease treatment policy estimation in the ICU. They evaluate their learned policies in the MIMIC-IV and other datasets, leveraging two cohorts (along with their intersection): patients with sepsis, and patients with acute kidney injury. Their learned policies, when evaluated offline, tend to outperform both clinicians and other RL frameworks.

Critical strengths

RL in the clinic is an extremely challenging setting. I think the authors have done some substantially positive work here:

  • The plausibility (blinded clinician) evaluation is very helpful for establishing credibility of the policy. In my view, this goes above and beyond for the venue and should be considered a serious point in the favour of the paper.
  • The authors have put significant effort into constructing fair comparisons, with a number of models and evaluation measures. This reflects the fundamental difficulty of off-policy evaluation for offline RL, and should also be a serious point in favour of the paper.
Critical weaknesses

My central concern is evaluation, which is always the hardest task in offline, off-policy RL evaluation for clinical work.

I would consider there's a few distinct claims being made in this paper: (a) a hypothetical RL agent has some benefit over the existing clinical policy & implementation, (b) multi-agent RL offers something over standard RL in this setting given the multi-disease nature, and (c) BIFOCAL offers something over existing MARL techniques in this setting. (a) seems somewhat hampered by the lack of a behavior cloning baseline, while (b) seems hampered by the lack of a strong single-agent baseline trained in the joint cohort and evaluated there. (c) is perhaps the best-supported claim in the paper.

In terms of selected baselines, I think CQL, dBCQ, and behaviour cloning trained on the joint cohort are missing (this relates to (a) and (b)). These baselines are clearly feasible given the WD3QNE experiment, which establishes that a joint action and joint state space are both possible. Furthermore, evaluations of WD3QNE in the AKI cohort (which it was trained on!) and HRRT in the sepsis cohort are missing (this relates to (b)). Finally, the drop in policy value from WIS to WISt/DR seems to require some discussion. All the WIS measures seem to rely on trajectory weights from a somewhat under-specified model. I'd like to see how sensitive they are to this model, for example, along with whatever best practices are for reporting WIS estimates (my knowledge is out of date, but even a few years ago, it would be conventional to e.g. also assess using trajectory weights estimated from a behavior cloning model).

Regarding presentation:

  • In both Tables 1 & 2: I would recommend colour-coding models (or find some other way of differentiating them) based on which cohort they were trained in (joint, or one or the other).
  • The figures should be more professional in appearance, in my opinion.

Smaller concerns:

  • Additional experimental details to confirm now backward-in-time data leakage into the predictors would be useful. Binning can be deceptive if not employed carefully.
Questions for authors
  1. Can you add CQL/BCQ/BC baselines trained on the joint cohort and evaluated on each of the three cohorts? Can you add the missing WD3QNE eval on the Sepsis and joint cohorts?
  2. Can you strengthen the WIS measures by using alternative trajectory weight estimates and studying the robustness to these weights a bit more?
4 comments
AuthorsOpenReview ↗

[1/4] Overview: BC, CQL and dBCQ added on three cohorts, WIS robustness checks, and code

Code (anonymous): https://anonymous.4open.science/r/BIFOCAL-7DFF/README.md

Thank you for the constructive review, and for recognizing the blinded clinician evaluation and the effort on fair comparisons. This overview follows the structure of your review, point by point, and says where each answer is.

Where to find what. Part 2/4: baselines. Part 3/4: WIS estimates. Part 4/4: leakage and presentation.

Critical Weaknesses

  • "My central concern is evaluation". We now rest the comparison on three kinds of evidence with different assumptions: importance-sampling estimates, FQE, and the blinded assessment by six ICU physicians, which does not depend on OPE.
  • Claim (a), "hampered by the lack of a behavior cloning baseline". BC added. On the joint cohort BIFOCAL is above BC by 7.1 on FQE (72.464 vs 65.373) and by 2.9 on WISt (79.747 vs 76.840). Part 2/4.
  • Claim (b), "lack of a strong single-agent baseline trained in the joint cohort and evaluated there". CQL and dBCQ added, and BIFOCAL ranks first on both estimators in the joint cohort (FQE 72.464 vs 69.222 for CQL; WISt 79.747 vs 79.128 for dBCQ). Part 2/4.
  • Claim (c), "perhaps the best-supported claim". We agree. The new baselines are meant to bring (a) and (b) to the same standard. Part 4/4.
  • "CQL, dBCQ, and behaviour cloning trained on the joint cohort are missing". All three added, each trained on the joint cohort. Part 2/4.
  • "evaluations of WD3QNE in the AKI cohort [...] and HRRT in the sepsis cohort are missing". Clarified, and both methods are now also evaluated on the joint cohort. WD3QNE is the sepsis method and HRRT the renal method; Table 2 already evaluates each on its own disease cohort. On the joint cohort BIFOCAL is higher than HRRT (FQE 72.464 vs 64.826) and than WD3QNE (72.464 vs 50.208). Part 2/4.
  • "the drop in policy value from WIS to WISt/DR seems to require some discussion". Explained and checked. Untruncated weights are heavy-tailed for every learned method (effective sample size 1.00 to 1.26), so untruncated WIS is a reference only. Part 3/4.
  • "I'd like to see how sensitive they are to this model". Done, with the check you name. With weights re-estimated from a jointly fitted behavior cloning model, BIFOCAL remains first in point estimate (75.017 vs 70.966 for the best baseline) and changes least (by 1.4, against 5.1 to 19.1). Support under these weights is limited (truncated effective sample size 204), so we read this as directional. FQE, weight diagnostics and cap sensitivity are also added. Part 3/4.
  • "whatever best practices are for reporting WIS estimates". Our reporting is broader than each published clinical RL study we checked (Nature Medicine 2018; npj Digital Medicine 2021, 2022 and 2023; JAMA 2025). Each reports one value estimator; we report six. Part 3/4.

Regarding presentation

  • Colour-coding models by training cohort in Tables 1 and 2. Adopted. Part 4/4.
  • "The figures should be more professional in appearance". Adopted; the figures will be redrawn. Part 4/4.

Smaller concerns

  • "confirm no backward-in-time data leakage into the predictors [...] Binning can be deceptive". The predictors read only the past at decision time, and every fitted quantity comes from the training split only. Part 4/4.

Questions For Authors

  • Q1, first half: "add CQL/BCQ/BC baselines trained on the joint cohort and evaluated on each of the three cohorts". Done. Across the six cohort-by-estimator columns BIFOCAL is first in four and second in two, and it is the only method in the top two everywhere. Part 2/4.
  • Q1, second half: "add the missing WD3QNE eval on the Sepsis and joint cohorts". The sepsis evaluation is already in Table 2 (WISt 2.683 vs 56.277 for BIFOCAL); the joint-cohort evaluation is new (FQE 50.208 vs 72.464). Part 2/4.
  • Q2: "strengthen the WIS measures by using alternative trajectory weight estimates and studying the robustness to these weights". Done. Under behavior-cloning weights BIFOCAL remains first in point estimate; support under these weights is limited. Part 3/4.
AuthorsOpenReview ↗

[2/4] Claims (a), (b) and Q1: BC, CQL, dBCQ trained on the joint cohort, evaluated on three cohorts (new results)

Claims (a) and (b), and Q1. "(a) seems somewhat hampered by the lack of a behavior cloning baseline, while (b) seems hampered by the lack of a strong single-agent baseline trained in the joint cohort and evaluated there." "Can you add CQL/BCQ/BC baselines trained on the joint cohort and evaluated on each of the three cohorts?"

Answer: we added BC, CQL and dBCQ, trained on the joint cohort and evaluated on all three cohorts. In the joint cohort BIFOCAL ranks first on both estimators, which addresses (a) and (b).

Setup: all baselines are single models trained on the joint (comorbid) cohort and evaluated as greedy policies, with a single training seed and a single FQE evaluator seed. The comorbid columns use the strict test subset: the MIMIC-IV comorbid test set after excluding every patient used to train any component (3,948 patients, 4,069 ICU stays). The single-disease columns use the test patients of the AKI-only and sepsis-only cohorts. FQE uses no importance weights; WISt is truncated WIS with cap 100. The best value in each column is in bold.

New results on three cohorts

MethodComorbid FQEComorbid WIStAKI-only FQEAKI-only WIStSepsis-only FQESepsis-only WISt
BIFOCAL72.46479.74772.86277.72156.37481.470
BC (new)65.37376.84071.65867.04638.04673.460
CQL (new)69.22274.16474.60761.32038.29182.622
dBCQ (new)65.18679.12869.63561.51736.13176.406

What the numbers say:

  • Claim (a): the gain is more than imitating clinicians. BC is the clinician policy under the same estimators. In the joint cohort BIFOCAL is higher by 7.1 on FQE and by 2.9 on WISt.
  • Claim (b): against strong single-agent models on the joint cohort, BIFOCAL is first on both estimators. FQE 72.464 vs 69.222 for the best single model (CQL); WISt 79.747 vs 79.128 for the best single model (dBCQ). Two further single models from our other responses are also lower: Implicit Q-Learning (59.227; 77.691) and a dynamic-weights network of the same size as BIFOCAL (68.266; 77.054).
  • Across all six columns BIFOCAL is first in four and second in two. CQL is higher on AKI-only FQE (74.607 vs 72.862) and on sepsis-only WISt (82.622 vs 81.470). No baseline is consistent: CQL is last on comorbid WISt (74.164) and on AKI-only WISt (61.320). BIFOCAL is the only method in the top two in every column.
  • A caution on the sepsis-only cohort. There the two estimators disagree strongly for every method (for example CQL: 38.291 on FQE, 82.622 on WISt), so we read that cohort as weak evidence in either direction. Our conclusions rest on the comorbid cohort.

WD3QNE and HRRT. "evaluations of WD3QNE in the AKI cohort (which it was trained on!) and HRRT in the sepsis cohort are missing"

Answer: Table 2 already contains the WD3QNE sepsis evaluation, and we have now added both methods on the joint cohort, where BIFOCAL is higher on both estimators.

WD3QNE is the sepsis method and HRRT is the renal replacement method. Table 2 evaluates each on its own disease cohort: WD3QNE on sepsis-only (WISt 2.683 vs 56.277 for BIFOCAL) and HRRT on AKI-only (WISt 71.787 vs 74.434).

New: both methods on the joint cohort. Each single-disease method controls the two treatment dimensions of its own disease, so BIFOCAL is compared with it on the same two dimensions.

Compared onMethodFQEWISt, cap 100
AKI actions (fluid, ultrafiltration)BIFOCAL72.46480.342
AKI actions (fluid, ultrafiltration)HRRT (new)64.82678.480
Sepsis actions (fluid, vasopressor)BIFOCAL72.46479.636
Sepsis actions (fluid, vasopressor)WD3QNE (new)50.20850.720

On the joint cohort BIFOCAL is higher than HRRT by 7.6 on FQE and 1.9 on WISt, and higher than WD3QNE by 22.3 on FQE and 28.9 on WISt.

Scope: BIFOCAL's branches start from disease-expert pretraining (Sec. 2.1), which the single models do not use, so training budgets are not matched. All values are offline estimates.

AuthorsOpenReview ↗

[3/4] WIS to WISt drop and Q2: weight diagnostics, cap sensitivity and FQE (new results)

The drop from WIS to WISt / DR. "the drop in policy value from WIS to WISt/DR seems to require some discussion"

Answer: the drop comes from heavy-tailed trajectory weights, which affect every learned method. Untruncated WIS alone is not reliable here, so we now report it as a reference only and base the comparison on truncated WIS.

  • Why it happens. A trajectory weight is the product of per-step probability ratios over about 11 four-hour steps and 125 joint actions. A few trajectories receive weights far larger than all others and dominate the untruncated average.
  • It is not specific to BIFOCAL. The untruncated effective sample size is 1.00 to 1.26 for every learned method. Truncation moves the estimates in both directions: Independent Q-Learning rises by 9.7, while VDN, B3C, QMIX and CF-CQL drop by 16.6, 14.1, 10.8 and 7.4; BIFOCAL drops by 11.3.
  • After truncation BIFOCAL remains the highest learned method.

A. WIS and WISt with weight diagnostics (Table 1(a) of the submission, 6,392 stays; one new column)

MethodWISESS before truncation (new)WISt, cap 100ESSt
BIFOCAL87.7001.0676.4434,174
CF-CQL83.4881.0276.0994,125
Independent Q-Learning66.2431.2675.9474,306
B3C86.7261.0272.6053,177
QMIX83.3241.0772.5391,492
VDN82.6171.0065.98882

On best practice for reporting WIS. "along with whatever best practices are for reporting WIS estimates"

Answer: our reporting is broader than that of each published clinical RL study we checked, including those in Nature Medicine, npj Digital Medicine and JAMA. Each of those reports one value estimator in its main text. We report six, with weight diagnostics before and after truncation, sensitivity to the cap, and a blinded physician assessment.

StudyValue estimatorsESS reportedCap sensitivityFQEBlinded physician rating
Komorowski et al., Nat Med 2018WISnononono
Peine et al., npj Digit Med 2021WISnononono
Zhang et al., npj Digit Med 2022WISyesnonono
Wu et al., npj Digit Med 2023DRnononono
Kalimouttou et al., JAMA 2025WISnononono
This workWIS, WISt, WISb, WISbt, DR, FQE (FQE new)yes, before (new) and after truncationyes (new)yes (new)yes

This table is based on the main text of each paper. The check on the behavior model itself is under Q2 below.

Q2. "Can you strengthen the WIS measures by using alternative trajectory weight estimates and studying the robustness to these weights a bit more?"

Answer: done. With trajectory weights re-estimated from a jointly fitted behavior cloning model, as you suggest, BIFOCAL remains first in point estimate and changes least, although support under these weights is limited. Three further checks point the same way.

B. WISt under two behavior models (new) (setting of Table 1(a), cap 100)

MethodOriginal behavior modelBC behavior model (new)Change
BIFOCAL76.44375.017-1.4
CF-CQL76.09970.966-5.1
Independent Q-Learning75.94770.321-5.6
QMIX72.53962.012-10.5
B3C72.60558.699-13.9
VDN65.98846.914-19.1
  • In point estimates BIFOCAL stays first and moves least (by 1.4; the baselines by 5.1 to 19.1), and the top three keep their order.
  • Support under the BC weights is limited, so we read this as a directional check. BIFOCAL's truncated effective sample size falls from 4,174 to 204 (about 3% of stays).
  • A limit we state plainly. The estimates do depend on the behavior model, as you suspected. For claim (a), we therefore rely on the comparison with the BC policy in part 2/4.
  • Further check: an estimator with no trajectory weights. FQE does not depend on the behavior model. BIFOCAL has the highest FQE in the joint cohort: 72.464, against 69.222 (CQL), 65.373 (BC) and 65.186 (dBCQ).
  • Further check: the truncation cap. For the greedy policy on the strict test subset, WISt is 78.883, 79.357, 79.747 and 80.763 at caps 1, 10, 100 and 1,000, a range of 1.88 (95% interval at cap 100: 76.540 to 82.902).
  • Further check: weight diagnostics. Effective sample size before and after truncation is in Table A.
  • The original behavior model. 750 state clusters with empirical clinician action frequencies; the two disease-specific probabilities are combined into the 125 joint actions (App. C.4, C.5).
AuthorsOpenReview ↗

[4/4] Leakage and presentation

Backward-in-time leakage. "Additional experimental details to confirm no backward-in-time data leakage into the predictors would be useful. Binning can be deceptive if not employed carefully."

Answer: the predictors see only the past at decision time, and every fitted quantity comes from the training split only. The details are below.

  • The risk predictor reads only the history window. Its input is the four most recent 4-hour states. The next six steps (24 hours) are used only during training, to build the deterioration labels and as the target of the future-representation stage. The future encoder and the decoder are discarded after training, so nothing from the future is available at decision time (Sec. 2.2, App. B.4, B.5).
  • The conflict router reads only current quantities. Its inputs are the current state representations and the current proposals of the two pretrained branches (Sec. 2.2).
  • Splits are by ICU stay. Train and test are split 60/40 by ICU stay with a fixed random state (Table 6). A patient with several stays can therefore appear on both sides, which is why the new results use the strict subset below.
  • Normalization and bins come from the training split only. The scaler is fitted on the training split and reused on the test split, and action bins are the quartiles of non-zero training doses (App. C.3).
  • Evaluation on patients no component has seen. The new results use a strict test subset that removes every patient used to train any component: 6,392 test stays reduce to 4,069.
  • Binning. States and actions use the same 4-hour grid. Actions are aggregated within each window: total fluid in the window and the maximum vasopressor dose in the window (App. B.1, C.3). The submission does not include a test that rebuilds each state from measurements taken before that step's action window, which is the finer check behind your remark on binning.

Presentation. "I would recommend colour-coding models [...] based on which cohort they were trained in" and "The figures should be more professional in appearance"

Answer: both suggestions are adopted.

  • Tables 1 and 2 will mark the training cohort of every model (joint, AKI-only or sepsis-only). This also removes the ambiguity about WD3QNE and HRRT discussed in part 2/4.
  • Independent Q-Learning will be renamed so that it is not read as Implicit Q-Learning.
  • The figures will be redrawn with consistent fonts, colors and axis labels.

On your claim (c). We agree that BIFOCAL's advantage over existing MARL methods is the best supported claim, and the new baselines in part 2/4 are meant to bring (a) and (b) to the same standard.

Reviewer_btWuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

treatment decisions in patients with sepsis and acute kidney injury (AKI). Two disease-conditioned value branches share a fluid action. Predicted 24-hour deterioration sets the direction of disease priority, while disagreement between pretrained fluid preferences controls the strength of asymmetric TD-gradient weighting. At decision time, the same signals guide gated Q-value fusion along the shared fluid dimension. The evaluation uses MIMIC-III, MIMIC-IV, eICU, and a private Chinese cohort. It includes off-policy evaluation (OPE), component ablations, MPE-style routing tests, and a two-stage retrospective assessment by six physicians on 30 cases. The reported results support further investigation of the coordination mechanism, but the reliability of the estimated gains and their relationship to the specified decision policy need clarification.

Critical strengths

1 A meaningful coordination problem and an intuitive mechanism Sepsis and AKI can require conflicting treatments and changing priorities. Predicted deterioration sets priority; present disagreement controls its influence. The authors distinguish learning updates from causal credit assignment and acknowledge differing training and decision rules. 2 Substantial evaluation and methodological detail Four ICU datasets, baselines, ablations, and detailed appendices provide useful evidence. Six physicians judged priority on 30 cases before seeing model outputs. Their retrospective ratings inform plausibility and interpretability.

Critical weaknesses

1 The estimated value improvement needs stronger validation The estimated gain depends heavily on the estimator. In Table 1(a), BIFOCAL improves on clinicians by 11.653 return units under WIS and 11.261 under WISb, but only 0.396, 0.420, and 0.183 under WISt, WISbt, and DR. These are estimated returns, not observed survival gains. Several WIS estimates match terminal-return values implied by the reward and discount: 87.752 across four variants, 82.617 across several single-disease results, and 84.294 for BIFOCAL on AKI-only patients. Uniform routing gives WIS 87.752 but WISb 34.844. These patterns raise weight-concentration concerns and affect the WISb-based ablations and external-cohort comparisons. They do not alone establish dominance by a few trajectories; report the underlying weights. Appendix C.5 caps trajectory weights at 100 and uses 100 bootstrap samples. Truncation changes both variance and bias. Please report paired patient-level confidence intervals for gains over clinicians and the strongest learned baseline, effective sample size (ESS) before truncation, the largest normalized weight, top-1% weight share, and the proportion capped. Test sensitivity to truncation and DR clipping. The reported truncated ESS of 4,174 does not establish stability before truncation. Fitted Q evaluation (FQE) could provide a complementary check under its own support and modeling assumptions. Figure 4(a) also differs from Table 1(a). At 100% bootstrap sampling, WISbt is approximately 78.5, 75–76, and 60 for BIFOCAL, VDN, and QMIX; the table reports 76.437, 65.961, and 72.597. The baseline lists also differ. Please reconcile the cohorts, checkpoints, methods, and estimator settings before using this figure as evidence of stability. Algorithm 2 chooses the highest-scoring action, whereas Appendix C.5 converts 125-action scores into probabilities through softmax. Rescaling scores can preserve the chosen action while changing these probabilities and OPE estimates. Explain how Eq. (3)'s five-fluid fusion produces 125-action joint scores. Specify each method's temperature and evaluated policy, and test score-scale, entropy, and support sensitivity. The 99.661% agreement between greedy rules does not establish agreement between probabilistic policies or explain Table 5. Appendices B.1 and C.3 combine treatment-related state features with four-hour action windows, using an AI-Clinician-style 750-cluster behavior model. Specify observation cutoffs and verify that features precede decisions through a timeline and pre-decision or lagged reconstruction. Tang et al. (2026), Off by a beat: the effects of temporal misalignment in reinforcement learning for sepsis treatment, npj Digital Medicine, motivates this check. The pipeline description does not establish leakage. Table 6 specifies training/test splits but no separate validation subset. Figure 4(c) plots indexed candidate models, and Appendix A.4 selects the 24-hour horizon using prediction and policy value. Define the candidate set, its size, and the patients and metrics used for selection. If test patients influenced selection, repeat it within training data and evaluate on an untouched test set.

2 Attribution and generality require more precise controls Tables 3 and 5 show large changes across routing and decision rules. Under Uniform learning, Action Mean gives WISb 81.214 and DR 7.224, versus 34.844 and 5.162 with Conflict-gated Q-fusion. Table 3 also gives DR 3.924, 1.986, and −0.806 for Uniform, Risk-only, and Future-label, compared with 76.047 for clinicians. Hold initialization, conflict-weighted anchoring, training budget, and decision/OPE rules fixed, and vary only routing weights. Report uncertainty and policy support to distinguish policy changes from estimation instability. Appendix A.6 changes 251 of 7,874 selected high-conflict decisions, or 3.19% (95% CI: 2.78–3.62%); this restricted comparison does not isolate training gains. Supply all five estimators promised in Section 3.4 for Tables 3–5; Tables 4–5 currently show only WISb and DR. Qualify the claim that removing warm-up has little effect: WISb changes from 87.287 to 86.162, but DR falls from 76.230 to 12.706. Proposition 1 needs a clearer scope. Equation (2) preserves exact zero-error solutions of the TD component, but state-dependent weights can change how gradients cancel across transitions with shared neural parameters. The full objective also includes anchoring. Specify whether the result concerns individual transitions, tabular models, or realizable zero-error solutions. Account for gradient scale and anchoring when claiming recovery of VDN under equal weights. Appendix B.4 derives disease risks from different markers. A shared sigmoid range does not make clinical severity comparable, although the raw score difference sets priority. Specify normalization, aggregation, and temperature, and test rescaling. Because targets include treatment-dependent markers such as RRT initiation, fluid balance, and vasopressor intensity, distinguish predicted changes under observed care from intrinsic deterioration or treatment benefit. Appendix A.5 measures agreement with constructed routing targets, which already incorporate priority and conflict. Explain whether targets are defined independently of the proposed rule. An independently defined team-performance objective would provide stronger validation. Figure 3(e) compares external-cohort results only with clinicians. Add learned-policy baselines and uncertainty intervals, clarify transfer versus retraining, and test zero-filling of missing variables for both policy and behavior models. This matters especially for the private cohort's 59 test patients. The physician ratings concern plausibility, not comparative efficacy or safety.

Questions for authors

See questions.

7 comments
AuthorsOpenReview ↗

[1/7] Overview: your 17 points, numbered W1.1 to W2.9, and how each is addressed

Code (anonymous): https://anonymous.4open.science/r/BIFOCAL-7DFF/README.md

Thank you for the detailed and checkable review. Your two weaknesses contain 17 separate points. To make this response easy to check, we number them W1.1 to W1.8 and W2.1 to W2.9 in the order you raise them, and answer each under its number. Each line below says how the concern is addressed and where.

Weakness 1, "The estimated value improvement needs stronger validation"

  • W1.1 Weights behind the WIS estimates ("report the underlying weights"). Addressed by reporting ESS before truncation (1.00 to 1.26 for every learned method), the top-1% weight share and the proportion capped, and by resting all comparisons on truncated WIS. Part 2/7.
  • W1.2 Paired intervals ("paired patient-level confidence intervals for gains over clinicians and the strongest learned baseline"). Addressed with paired intervals from 2,000 resamples: +0.396 over clinicians and +0.344 over CF-CQL. The difference is positive in every setting, with intervals that include zero. Part 2/7.
  • W1.3 Truncation sensitivity ("Test sensitivity to truncation and DR clipping"). Addressed by varying the cap from 1 to 1,000: WISt moves by 1.88. Part 2/7.
  • W1.4 FQE ("could provide a complementary check"). Addressed by adding FQE as a supplementary check; the comparison rests on truncated WIS. Part 2/7.
  • W1.5 Figure 4(a) and Table 1(a) ("reconcile [...] before using this figure as evidence of stability"). Addressed by resting stability on W1.2 and W1.3 instead, and regenerating the figure. Part 3/7.
  • W1.6 Greedy vs softmax ("Algorithm 2 chooses the highest-scoring action, whereas Appendix C.5 converts 125-action scores into probabilities through softmax"). Addressed by evaluating the greedy policy directly: WISt 79.747 (95% interval 76.540 to 82.902), the highest of the six methods evaluated. Construction and temperature are specified. Part 3/7.
  • W1.7 Temporal alignment ("Specify observation cutoffs"). Addressed by specifying the cutoff the pipeline uses and what each learned component reads. Part 4/7.
  • W1.8 Validation and selection ("Define the candidate set [...] and the patients and metrics used for selection"). Addressed by documenting the split unit, the patient overlap and each selection, and by evaluating new results on a strict subset with no training overlap. Part 4/7.

Weakness 2, "Attribution and generality require more precise controls"

  • W2.1 Routing ablations ("Report uncertainty and policy support"). Addressed by reporting the support of each variant. Against the well-supported variant BIFOCAL is higher by 3.5; the low-support rows are read as directional; evidence that does not rely on OPE is added. Part 5/7.
  • W2.2 App. A.6 ("does not isolate training gains"). Addressed by stating its scope: it is the controlled test of the decision step, where 95.62% of changes go toward the higher-weight disease. Part 5/7.
  • W2.3 Five estimators for Tables 3 to 5. Table 3 already reports all five. Addressed for Tables 4 and 5 by correcting Section 3.4 and adding truncated WIS and ESS. Part 5/7.
  • W2.4 Warm-up ("Qualify the claim"). Qualified as you ask: DR falls from 76.230 to 12.706 without warm-up, so it is a needed component. Part 5/7.
  • W2.5 Proposition 1 ("needs a clearer scope"). Addressed by stating the scope: individual transitions, TD term only. Part 5/7.
  • W2.6 Risk scores ("A shared sigmoid range does not make clinical severity comparable"). Addressed by specifying the construction and what the priority means: an ordering under observed care, consistent with the priority that physicians judged before seeing the model. Part 6/7.
  • W2.7 App. A.5 ("whether targets are defined independently"). Addressed by stating what A.5 tests. Part 6/7.
  • W2.8 External cohorts ("only with clinicians"). Addressed by stating the documented protocol and what Fig. 3(e) is used for. Part 6/7.
  • W2.9 Physician ratings ("plausibility, not comparative efficacy or safety"). Addressed by stating exactly what the physicians judged: frontline ICU physicians, the intended users, rated BIFOCAL's recommendations positively on all six dimensions, including safety awareness and trust. None of the published clinical RL studies we checked reports such an evaluation. Part 7/7.

What the evidence supports after these checks. On the comorbid cohort, BIFOCAL is first on truncated WIS in each of three settings (the softmax policy of the submission, a jointly fitted behavior model, and the greedy policy on the strict subset), the estimate is stable across truncation caps, and routing acts in the intended direction in 95.62% of the decisions it changes.

AuthorsOpenReview ↗

[2/7] W1.1 to W1.4: weight diagnostics, paired intervals, truncation sensitivity, FQE

W1.1 Weights behind the WIS estimates. "The estimated gain depends heavily on the estimator [...] These patterns raise weight-concentration concerns [...] report the underlying weights [...] effective sample size (ESS) before truncation, the largest normalized weight, top-1% weight share, and the proportion capped."

We address this concern by reporting the diagnostics you list and by resting every comparison on truncated WIS. Untruncated WIS and WISb are kept as a reference only.

  • Before truncation (new). The effective sample size is 1.00 to 1.26 for every learned method on the 6,392 test stays, so this is a property of the task and not of BIFOCAL. The top 1% of BIFOCAL's trajectories carry 97.032% of the weight, and an ESS of 1.06 bounds the largest normalized weight from below: it cannot be smaller than 1/ESS, that is 0.94. Your reading is therefore right: an untruncated estimate here is close to the return of one trajectory (87.752 is 100 x 0.99^13, one surviving stay of 14 steps; 82.617 and 84.294 are 100 x 0.99^19 and 100 x 0.99^17). This is why the comparison rests on truncated WIS.
  • After truncation at 100 (new). BIFOCAL's effective sample size is 4,174 (65% of stays), and 54.991% of trajectories (3,515) are capped. All capped trajectories carry the same weight, so the largest normalized weight is below 0.03%, and the top 1% of trajectories carry less than 2% of the weight.
MethodWISESS before truncation (new)WISt, cap 100ESSt
BIFOCAL87.7001.0676.4434,174
CF-CQL83.4881.0276.0994,125
Independent Q-Learning66.2431.2675.9474,306
B3C86.7261.0272.6053,177
QMIX83.3241.0772.5391,492
VDN82.6171.0065.98882

W1.2 Paired intervals. "Please report paired patient-level confidence intervals for gains over clinicians and the strongest learned baseline"

We address this concern with patient-level paired bootstrap intervals (2,000 resamples, in place of the 100 of App. C.5) for both comparisons you name, and for two further settings.

  • Table 1(a), BIFOCAL minus clinicians, WISt: +0.396 (95% interval -0.520 to 1.292).
  • Table 1(a), BIFOCAL minus CF-CQL, the strongest learned baseline: +0.344 (-0.568 to 1.282).
  • Greedy policy on the strict test subset (W1.6), BIFOCAL minus dBCQ, the strongest baseline there: +0.619 (-2.035 to 3.386).
  • Jointly fitted behavior model (W1.6), BIFOCAL minus CF-CQL: +4.051 (-1.229 to 9.902).

The difference is positive in every setting, and the intervals include zero. What we claim is therefore a consistent ordering across target policies and behavior models, not a statistically separated gap, and, as you say, estimated returns rather than observed survival gains.

W1.3 Sensitivity to truncation and DR clipping. "Truncation changes both variance and bias [...] Test sensitivity to truncation and DR clipping. The reported truncated ESS of 4,174 does not establish stability before truncation."

We address this concern by varying the cap over three orders of magnitude: the estimate moves by 1.88.

CapWISt95% intervalESSt
178.88376.083 to 81.7581,071
1079.35776.507 to 82.1321,012
10079.74776.540 to 82.902746
1,00080.76376.925 to 84.434498

This is BIFOCAL's greedy policy on the strict test subset (3,948 patients, 4,069 stays). The lower bound of the interval stays between 76.1 and 76.9 across caps. We agree that an ESSt of 4,174 says nothing about stability before truncation; the numbers before truncation are in W1.1. The DR clipping threshold was not varied.

W1.4 FQE. "Fitted Q evaluation (FQE) could provide a complementary check under its own support and modeling assumptions."

We address this concern by adding FQE for the greedy policies on the strict subset as a supplementary check. As you note, FQE rests on its own support and modeling assumptions, so the comparison in this response rests on truncated WIS with the diagnostics above.

AuthorsOpenReview ↗

[3/7] W1.5 and W1.6: Fig. 4(a), and the greedy policy evaluated directly

This part addresses your concerns W1.5 (Fig. 4(a)) and W1.6 (greedy vs softmax policy). Key result: the deployed greedy policy is now evaluated directly, and BIFOCAL has the highest truncated WIS of the six methods (79.747, 95% interval 76.540 to 82.902). It is first at both ends of the temperature range and under a second behavior model.

W1.5 Figure 4(a) and Table 1(a). "At 100% bootstrap sampling, WISbt is approximately 78.5, 75-76, and 60 for BIFOCAL, VDN, and QMIX; the table reports 76.437, 65.961, and 72.597. The baseline lists also differ. Please reconcile the cohorts, checkpoints, methods, and estimator settings before using this figure as evidence of stability."

We address your concern about Fig. 4(a) by replacing it with two direct stability results: the estimate moves by only 1.88 across truncation caps from 1 to 1,000 (W1.3), and BIFOCAL's difference to the strongest baseline is positive in every setting (W1.2). You are right that Fig. 4(a) and Table 1(a) do not match, so the figure is not used as evidence here. It is regenerated from the cohort, checkpoints, methods and estimator settings of Table 1(a).

W1.6 Greedy policy vs softmax policy. "Algorithm 2 chooses the highest-scoring action, whereas Appendix C.5 converts 125-action scores into probabilities through softmax. Rescaling scores can preserve the chosen action while changing these probabilities and OPE estimates. Explain how Eq. (3)'s five-fluid fusion produces 125-action joint scores. Specify each method's temperature and evaluated policy, and test score-scale, entropy, and support sensitivity. The 99.661% agreement between greedy rules does not establish agreement between probabilistic policies or explain Table 5."

We address your concern about the greedy/softmax mismatch by evaluating the deployed greedy policy directly. This estimate does not depend on the softmax temperature or on a common rescaling of the scores, and BIFOCAL is first of the six methods.

  • The greedy policy, evaluated directly (new). Strict test subset (3,948 patients, 4,069 stays, none used to train any component; W1.8), a single training seed per method, WISt with cap 100:
MethodWISt, greedy policy
BIFOCAL79.747
dBCQ79.128
Implicit Q-Learning77.691
Dynamic-weights CN77.054
BC76.840
CQL74.164

BIFOCAL's 95% interval is 76.540 to 82.902.

  • Entropy: BIFOCAL is first at both ends of the temperature range. At temperature 1 (Table 1(a)) it is first among the six learned methods with WISt 76.443; at temperature zero, the greedy policy, it is first with 79.747.
  • Support: BIFOCAL is first under a second behavior model. With a behavior cloning model fitted jointly over the 125 actions, BIFOCAL stays first and changes least (76.443 to 75.017, against drops of 5.1 to 19.1 for the five baselines), with a smaller effective sample size of 204. The truncated effective sample size is 4,174 of 6,392 stays for the softmax policy and 746 of 4,069 for the greedy policy.
  • Score scale. You are right that softmax estimates depend on it. In the test reported to Reviewer Xkdt (part 4/4), standardizing the Q-values of each head, which rescales the two heads differently, changes the joint greedy action in 7.051% of states and moves the softmax WISt from 76.443 to 65.700. A common positive rescaling of the fused scores leaves the greedy action unchanged, which is why the greedy policy is the estimand we now report.
  • How the 125 joint scores are built. A joint action is a triple of fluid level f, ultrafiltration level u and vasopressor level v (5 x 5 x 5). Its score is w-AKI x Q-AKI(f, u) + w-SEP x Q-SEP(f, v), where the two branches share f and the weights are those of Eq. (3).
  • Evaluated policy and temperature in the submission. A softmax at temperature 1 for all six learned methods (App. C.5). For BIFOCAL it is one softmax over the 125 fused scores. For the five baselines it is a softmax over each of the two 25-action heads, combined into joint-action probabilities by the action-mean rule.
  • On 99.661%. It is a statement about greedy actions, and we use it only as such. The swings in Table 5 are discussed in W2.1.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response [1/3]: thanks, what the reviewers recognized, and the scope of this study

Code (anonymous): https://anonymous.4open.science/r/BIFOCAL-7DFF/README.md

We sincerely thank all five reviewers. Your comments have clearly improved this paper. We apologize for the late reply: we used the time to run almost every experiment that was requested, and the results are reported here and in the individual responses. Three parts follow: (1/3) what the reviewers recognized and what we wish to underline, (2/3) the shared concerns and how each is addressed, (3/3) all new results in one place.

1. What the reviewers recognized

  • A meaningful clinical problem and a new mechanism. Reviewer btWu: "A meaningful coordination problem and an intuitive mechanism". Reviewer WjTg: "a novel creative solution". Reviewer Xkdt: "Separating direction from activation is a reasonable inductive bias for competing objectives."
  • Evaluation by the intended users, beyond the standard of the venue. Reviewer S6Ln: the blinded clinician evaluation "goes above and beyond for the venue and should be considered a serious point in the favour of the paper". Reviewer btWu: six physicians judged priority "before seeing model outputs".
  • Broad and fair comparison. Reviewer S6Ln: "significant effort into constructing fair comparisons, with a number of models and evaluation measures". Reviewer btWu: "Four ICU datasets, baselines, ablations, and detailed appendices provide useful evidence." Reviewer WjTg: comparison with "a large number of multi-agent RL methods".
  • Precise claims. Reviewer Xkdt: the manuscript "carefully delineates what the routing rule does", which makes the contribution "substantially easier to assess and trust".
  • Clinical thinking and clear writing. Reviewer WjTg: the authors "put some thought into the clinical implications"; Reviewers tw5d and WjTg find the writing clear.

2. What we wish to underline

  • Judged by the intended users. Six frontline ICU physicians from four hospitals judged disease priority on 30 real comorbid cases before seeing the model, and BIFOCAL's priorities were consistent with their judgments. After seeing its recommendations, they rated all six dimensions positively, including safety awareness, trust and willingness to adopt (priority appropriateness and interpretability: 5.5 to 6.3 out of 7; App. C.6). This evidence does not depend on off-policy evaluation.
  • Validated across patient populations. Sepsis and AKI are highly heterogeneous, so one cohort is not enough. The study covers four ICU datasets from two countries (MIMIC-IV, MIMIC-III, eICU and a private cohort from China; 87,566 trajectories, 1,285,977 decision points) and three populations with one model (comorbid, AKI-only, sepsis-only).
  • Twelve compared methods: five multi-agent baselines, two disease-specific methods and five single-model baselines (new).
  • Interpretable and inexpensive. Every recommendation exposes each disease's fluid preference, the conflict score and the priority weights; training takes under 9 minutes and a decision under 0.5 ms.
  • To our knowledge, the broadest evaluation among the published clinical RL studies we checked, and the only one among them with a blinded evaluation by the intended users:
StudyValue estimatorsESS reportedCap sensitivityFQEBlinded physician rating
Komorowski et al., Nat Med 2018WISnononono
Peine et al., npj Digit Med 2021WISnononono
Zhang et al., npj Digit Med 2022WISyesnonono
Wu et al., npj Digit Med 2023DRnononono
Kalimouttou et al., JAMA 2025WISnononono
This workWIS, WISt, WISb, WISbt, DR, FQEyes, before (new) and after truncationyes (new)yes (new)yes

Based on the main text of each paper: each reports one value estimator, and none reports a blinded physician rating. This response further adds a direct evaluation of the greedy policy, a second behavior model and paired patient-level intervals (all new).

3. The result in one sentence

On the comorbid cohort, BIFOCAL has the highest truncated WIS in every setting we evaluated: the softmax policy of the submission, the greedy policy on a strict test subset, and a jointly fitted behavior model (point estimates; paired intervals are in the individual responses).

AuthorsOpenReview ↗

General Response [2/3]: six shared concerns and how each is addressed (new results)

The main concerns of the five reviews fall into six groups. For each we state who raised it, how it is addressed, and why the result answers it. Tables A to G are in part 3/3.

A. Is a design with disease branches needed, compared with one model? Raised by Reviewer tw5d (Q1, CW2), Reviewer S6Ln (claims (a) and (b), Q1) and Reviewer WjTg (W4). Addressed by five new single-model baselines trained on the joint cohort: BIFOCAL has the highest truncated WIS of the six methods (79.747). (new, Table A)

  • The baselines are the ones asked for: CQL and Implicit Q-Learning (one Q-function over the 125 combined actions), a dynamic-weights network, behavior cloning and dBCQ.
  • The closest test: the dynamic-weights network has the same size as BIFOCAL (424K vs 427K parameters) and also receives priority weights, but has no disease branches. It is lower (77.054 vs 79.747).
  • The extra cost is small: 8.6 minutes of training and 0.43 ms per decision (new, Table G).

B. Are the importance-sampling estimates reliable? Raised by Reviewer tw5d (CW1), Reviewer S6Ln (drop from WIS to WISt, Q2), Reviewer btWu (W1) and Reviewer Xkdt (W1). Addressed by four checks; BIFOCAL remains first wherever methods are compared.

  • Weights before and after truncation (new, Table B). The untruncated effective sample size is 1.00 to 1.26 for every learned method, so the drop from WIS to WISt is a property of the task and not of BIFOCAL. Untruncated WIS is now a reference only. After truncation BIFOCAL is the highest learned method.
  • Truncation cap (new, Table C). Across caps from 1 to 1,000 the estimate moves by 1.88.
  • A second behavior model (new, Table D). With a behavior cloning model fitted jointly over the 125 actions, as Reviewers S6Ln and Xkdt suggest, BIFOCAL stays first and changes least (by 1.4, against 5.1 to 19.1 for the baselines).
  • Paired patient-level intervals (new), 2,000 resamples. The difference to the strongest baseline is positive in every setting (+0.344, +0.619 and +4.051); the intervals are in the responses to Reviewers Xkdt and btWu.

C. The submission evaluated a softmax policy, but the deployed policy is greedy. Raised by Reviewer Xkdt (W1, Q2) and Reviewer btWu (W1). Addressed by evaluating the greedy policy directly: WISt 79.747 (95% interval 76.540 to 82.902), the highest of the six methods. (new, Table A) BIFOCAL is therefore first at both ends of the temperature range.

D. Data splits, selection and leakage. Raised by Reviewer Xkdt (Q1), Reviewer btWu (W1) and Reviewer S6Ln (smaller concern). Addressed by evaluating all new greedy-policy results on a strict test subset on which no component was trained (3,948 patients, 4,069 ICU stays, 46,470 decision points), where BIFOCAL is first. The split unit and each selection are documented in the individual responses. The predictors read only the past at decision time.

E. Does the benefit come from the routing mechanism, and do the weights act as intended? Raised by Reviewer WjTg (W1, W2), Reviewer Xkdt (W2), Reviewer btWu (W2) and Reviewer tw5d (Q1). Addressed by controlled comparisons that isolate the mechanism.

  • With both Q-heads held fixed, routing changes 3.19% of 7,874 high-conflict decisions, and 95.62% of the changes move toward the disease at higher predicted risk (App. A.6).
  • The scale-normalized fusion suggested by Reviewer Xkdt does not improve the policy (WISt 65.700 vs 76.443) (new).
  • Independent of OPE: in high-conflict states the one-step reduction in TD error is 1.34 with routing and 1.25 with equal weights (Fig. 2c).

F. Do the results hold in other cohorts and against disease-specific methods? Raised by Reviewer S6Ln (Q1, and the missing WD3QNE and HRRT evaluations). Addressed by evaluating the new baselines on all three cohorts and both disease-specific methods on the joint cohort. (new, Tables E and F) On WISt BIFOCAL is first on the comorbid and AKI-only cohorts and second on the sepsis-only cohort, and on the joint cohort it is above HRRT (80.342 vs 78.480) and WD3QNE (79.636 vs 50.720).

Why these results answer the concerns together. The concerns were of two kinds: whether the comparison was complete, and whether the estimates can be trusted. The first is answered by the new baselines. The second is answered by testing the same claim under different target policies, behavior models, truncation caps and test subsets: BIFOCAL's first place on the comorbid cohort does not change. The physician evaluation adds evidence that needs no off-policy estimate: frontline ICU physicians, the intended users, judged the disease priorities appropriate and interpretable, and rated safety awareness and trust positively.

AuthorsOpenReview ↗

General Response [3/3]: all new results in one place (Tables A to G)

All new results in one place. (new) marks results added in this response. The best value in each column is in bold. WISt is truncated WIS with cap 100.

A. BIFOCAL vs single-model methods (new) (comorbid cohort, greedy policies, strict test subset of 3,948 patients and 4,069 stays, a single training seed per method)

MethodWISt
BIFOCAL79.747
dBCQ (new)79.128
Implicit Q-Learning (new)77.691
Dynamic-weights network (new)77.054
Behavior cloning (new)76.840
CQL (new)74.164

B. Weights before and after truncation (Table 1(a) of the submission, 6,392 stays; one new column)

MethodWISESS before truncation (new)WIStESSt
BIFOCAL87.7001.0676.4434,174
CF-CQL83.4881.0276.0994,125
Independent Q-Learning66.2431.2675.9474,306
B3C86.7261.0272.6053,177
QMIX83.3241.0772.5391,492
VDN82.6171.0065.98882

C. Sensitivity to the truncation cap (new) (BIFOCAL, greedy policy, strict subset, 2,000 patient-level bootstrap resamples)

CapWISt95% intervalESSt
178.88376.083 to 81.7581,071
1079.35776.507 to 82.1321,012
10079.74776.540 to 82.902746
1,00080.76376.925 to 84.434498

D. WISt under two behavior models (new) (setting of Table 1(a))

MethodOriginal behavior modelJoint behavior cloning model (new)Change
BIFOCAL76.44375.017-1.4
CF-CQL76.09970.966-5.1
Independent Q-Learning75.94770.321-5.6
QMIX72.53962.012-10.5
B3C72.60558.699-13.9
VDN65.98846.914-19.1

E. New baselines on three cohorts (new) (WISt; baselines trained on the joint cohort; greedy policies, so not directly comparable with Table 2 of the submission, which evaluates softmax policies)

MethodComorbidAKI-onlySepsis-only
BIFOCAL79.74777.72181.470
Behavior cloning (new)76.84067.04673.460
CQL (new)74.16461.32082.622
dBCQ (new)79.12861.51776.406

F. Disease-specific methods on the joint cohort (new) (WISt; each compared on the two treatment dimensions it controls)

Compared onBIFOCALDisease-specific method
AKI actions (fluid, ultrafiltration)80.342HRRT (new): 78.480
Sepsis actions (fluid, vasopressor)79.636WD3QNE (new): 50.720

G. Cost (new)

MethodParametersTraining time, with all pretrainingInference per decision
BIFOCAL427K518.4 s0.43 ms
Dynamic-weights network424K271.3 s0.19 ms
Implicit Q-Learning251K84.1 s0.03 ms
CQL251K42.1 s0.03 ms

What the tables show together. On the comorbid cohort, the primary setting of the paper, BIFOCAL is first on truncated WIS under the softmax policy (B), under the greedy policy on the strict subset (A), and under a second behavior model (D), and its estimate is stable across caps (C). It is also first on the AKI-only cohort and second on the sepsis-only cohort (E), and above both disease-specific methods on the joint cohort (F), at a cost of under 9 minutes of training and under half a millisecond per decision (G). All values are offline estimates.

Together with the blinded evaluation by frontline ICU physicians, who rated BIFOCAL's recommendations positively on all six dimensions, this is to our knowledge the broadest evaluation among the published clinical RL studies we checked (comparison table in part 1/3).

Independent Q-Learning (Table 1 of the submission) and Implicit Q-Learning (new baseline) are different methods.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.