- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates offline reinforcement learning for managing patients with concurrent sepsis and acute kidney injury (AKI). The authors formulate the problem using two disease-conditioned value branches and propose BIFOCAL, which dynamically allocates learning credit based on two signals: predicted future deterioration risk determines which disease receives greater weight, while current treatment conflict determines when and how strongly asymmetric weighting is applied. At decision time, the same risk–conflict principle is used to fuse disease-specific Q-values along the shared fluid-action dimension.
Critical strengths
Separating direction from activation is a reasonable inductive bias for competing objectives. More importantly, the manuscript carefully delineates what the routing rule does: it modifies branch-specific optimization rates while preserving the forward additive value, recovers equal routing at zero conflict, and does not claim causal credit assignment. This precision makes the contribution substantially easier to assess and trust.
Critical weaknesses
- BIFOCAL’s strongest claim concerns improved estimated policy value. However, the policy executed in the paper relies on greedy Q fusion, whereas the OPE target-policy probabilities are derived from softmax-transformed Q-values. For the 125-action joint space, the behavior policy is likewise constructed from disease-specific behavior probabilities rather than explicitly estimated as a joint clinician policy. Given the sensitivity of importance-sampling OPE to the specification of both the target and behavior policies, these choices may alter method rankings. This concern is compounded by the lack of a clearly documented validation split, especially because the risk horizon is selected partly on the basis of downstream OPE performance.
This is especially important in clinical offline RL, where prior work has shown that conclusions can vary substantially with evaluation and MDP design choices. Proceedings of Machine Learning Research I would need an untouched final test evaluation, target-policy/temperature sensitivity, behavior-policy sensitivity, and paired uncertainty on the differences against the strongest baselines before treating the policy-value advantage as robust.
- BIFOCAL assigns semantic meaning to the risk-derived disease weights and then applies those weights directly to the AKI and Sepsis Q profiles. Yet the individual branch values of an additive decomposition are not uniquely identified in a way that automatically guarantees comparable scales. The paper's own sensitivity analysis states that correspondence between routing weights and actual Q influence deteriorates as branch scales diverge and explicitly notes that it does not establish globally aligned or identifiable Q-head scales. This matters because a nominal weight of, for example, 0.6 versus 0.4 does not necessarily imply 60/40 disease influence if one Q-head has substantially larger variation. A scale-invariant fusion rule, explicit calibration/normalization, or a convincing invariance analysis would materially strengthen the method.
Questions for authors
-
What exact patient subsets are used to select the 24-hour risk horizon, conflict threshold , routing temperature , anchor coefficient, checkpoints, and other hyperparameters? Table 6 documents 60/40 train/test splits for the public datasets, while the 24-hour horizon is selected partly using downstream OPE. Was the final test cohort completely untouched during all such selections? Please also confirm that disease-specific expert pretraining, the risk module, and the conflict router never use patients assigned to the corresponding comorbidity test set.
-
The executable BIFOCAL policy is greedy, whereas Appendix C.5 defines target probabilities through softmax-transformed Q-values. Are the reported WIS/WISt/WISb/WISbt/DR values therefore estimates for a stochastic surrogate policy rather than the deployed greedy policy? Please report sensitivity to the softmax temperature and explain why the chosen stochastic policy is the appropriate estimand.
4 comments
[2/4] W1 and Q2: greedy policy evaluated directly; BIFOCAL first under both target policies and both behavior models
W1 and Q2. "the policy executed in the paper relies on greedy Q fusion, whereas the OPE target-policy probabilities are derived from softmax-transformed Q-values" and "Are the reported WIS/WISt/WISb/WISbt/DR values therefore estimates for a stochastic surrogate policy rather than the deployed greedy policy? Please report sensitivity to the softmax temperature and explain why the chosen stochastic policy is the appropriate estimand."
Answer: yes, the values in the submission are for softmax policies. We have now evaluated the deployed greedy policy directly: BIFOCAL has the highest truncated WIS of the six methods evaluated (79.747). It also stays first, and changes least, when the behavior policy is replaced by a jointly fitted clinician model.
- Why a softmax policy was used. Importance sampling gives zero weight to every stay that departs once from a deterministic policy, so few stays carry the estimate. A softmax over the Q-values keeps more of them (4,174 of 6,392 effective stays after truncation). The temperature is 1 for all six learned methods (App. C.5). The softmax policy is a means of comparing methods with adequate support. The policy that would be deployed is the greedy one, so we now evaluate it directly.
- The greedy policy, evaluated directly (new). Strict test subset (3,948 patients, 4,069 ICU stays, none used to train any component; part 3/4), a single training seed per method, WISt with cap 100:
| Method | WISt, greedy policy |
|---|---|
| BIFOCAL | 79.747 |
| dBCQ | 79.128 |
| Implicit Q-Learning | 77.691 |
| Dynamic-weights CN | 77.054 |
| BC | 76.840 |
| CQL | 74.164 |
BIFOCAL's 95% interval is 76.540 to 82.902, with 746 effective stays after truncation (18%).
The four checks you ask for
- 1. Untouched final test evaluation. The table above uses a strict subset on which no component was trained. A fully untouched re-split is not available in this response (part 3/4).
- 2. Target-policy and temperature sensitivity. We report the two ends of the temperature range. At temperature 1 (Table 1(a)), BIFOCAL is first among the six learned methods with WISt 76.443. At temperature zero, which is the greedy policy, it is first with 79.747. The greedy estimate is also stable across truncation caps: 78.883, 79.357, 79.747 and 80.763 at caps 1, 10, 100 and 1,000.
- 3. Behavior-policy sensitivity (new). We replaced the behavior model of the submission (two disease-specific models combined; App. C.5) with a behavior cloning model fitted jointly over the 125 actions, keeping the policies and the test set of Table 1(a) fixed:
| Method | WISt, original behavior model | WISt, joint behavior model | Change |
|---|---|---|---|
| BIFOCAL | 76.443 | 75.017 | -1.4 |
| CF-CQL | 76.099 | 70.966 | -5.1 |
| Independent Q-Learning | 75.947 | 70.321 | -5.6 |
| QMIX | 72.539 | 62.012 | -10.5 |
| B3C | 72.605 | 58.699 | -13.9 |
| VDN | 65.988 | 46.914 | -19.1 |
BIFOCAL stays first and changes least, and the order of the first three methods is unchanged. The weights are more concentrated under this model (BIFOCAL's truncated effective sample size is 204), which is why the corresponding interval below is wider.
-
4. Paired uncertainty against the strongest baseline (new). Patient-level paired bootstrap, 2,000 resamples; BIFOCAL minus the baseline, with 95% interval:
- Greedy policy, strict subset, vs dBCQ: +0.619 (-2.035 to 3.386).
- Table 1(a), vs CF-CQL: +0.344 (-0.568 to 1.282).
- Joint behavior model, vs CF-CQL: +4.051 (-1.229 to 9.902).
The difference is positive in all three settings, and the intervals include zero. What we claim is therefore a consistent ordering across target policies and behavior models, not a statistically separated gap.
On the comorbid cohort, in each of the three settings (the softmax policy of the submission, the joint behavior model, and the greedy policy on the strict subset), BIFOCAL is first on truncated WIS among the methods compared.
[3/4] Q1: split unit, overlap counts and selection subsets; strict subset used for new results
Q1. "What exact patient subsets are used to select the 24-hour risk horizon, conflict threshold, routing temperature, anchor coefficient, checkpoints, and other hyperparameters? [...] Was the final test cohort completely untouched during all such selections? Please also confirm that disease-specific expert pretraining, the risk module, and the conflict router never use patients assigned to the corresponding comorbidity test set."
Answer: not completely, and we give the exact numbers. The split was made by ICU stay, so some comorbid test patients have another stay in a training source, and two selections used test-set metrics. All new greedy-policy results use a strict subset from which these patients are removed, and BIFOCAL is first there.
- Split unit and overlap. The 60/40 split of Table 6 was made by ICU-stay ID. A patient with several stays can therefore fall on both sides of the split, or in a single-disease cohort and in the comorbid cohort. Of the 5,884 comorbid test patients (6,392 ICU stays), 1,936 also appear in a training source.
- The components you name. Comorbid test patients who also appear in the training data of each: AKI expert pretraining 1,126; sepsis expert pretraining 528; risk module 653; conflict router and joint policy 907. The groups overlap. So we cannot give the confirmation you ask for on the original test set.
- Strict subset. Removing all 1,936 patients leaves 3,948 patients, 4,069 ICU stays and 46,470 decision points, none used to train any component. On this subset BIFOCAL has the highest WISt of the six methods evaluated (79.747; part 2/4).
- What was selected, and on which patients.
- 24-hour risk horizon: eight horizons from 4 to 32 hours, compared on the original test set by prediction correlation, NMSE and downstream OPE (App. A.4).
- Conflict threshold, routing temperature and anchor coefficient: 0.5, 1 and 1, as in the saved configuration. We have no selection record for these three values.
- Checkpoint: 5 training epochs. Its saved selection statistic is WIS on the original test set.
- Nothing was re-tuned for this response. The settings and the checkpoint are those of the submission.
- Untouched final test. The strict subset removes the patient overlap; it does not change how the settings above were chosen. An evaluation on a new patient-level split, with every component retrained and selected inside the training data, is not available in this response.
The revision states the split unit and these counts in Table 6.
[4/4] W2: weights set a direction of priority; scale-normalized fusion tested
W2. "a nominal weight of, for example, 0.6 versus 0.4 does not necessarily imply 60/40 disease influence if one Q-head has substantially larger variation. A scale-invariant fusion rule, explicit calibration/normalization, or a convincing invariance analysis would materially strengthen the method."
Answer: we agree on the reading. The weights set a direction of priority, not a 60/40 share of influence, and the paper claims only the former. We ran the scale-normalized fusion you suggest: it does not improve the policy (WISt 65.700, against 76.443 for the fusion rule of the paper), so we keep fusion on the raw Q scale, where both heads are expressed in the same units of return.
- Why the two heads are on a comparable scale. As you note, the TD loss alone fixes only the sum of the two branch values. Their scales are tied by construction instead: both heads start from disease experts trained on the same reward (terminal reward of plus or minus 100, discount 0.99), and the anchor keeps each head close to its expert, most strongly where conflict is low.
- What the weights do: set a direction. With both Q-heads held fixed, replacing equal weights by the adaptive weights changes the fluid decision in 3.19% of 7,874 high-conflict decisions, and 95.62% of these changes (95% interval 92.89 to 97.93) move toward the fluid level preferred by the higher-weight disease (App. A.6).
- What we do not claim: a proportion. The paper's own sensitivity analysis (App. Fig. 11), which you cite, already states that it does not establish aligned or identifiable Q-head scales. Remark 1 makes the same point for learning: equal credit means an equal gradient coefficient, not equal gradient norms.
- Scale-normalized fusion (new). The model and the routing weights are frozen. In each state, the 25 Q-values of each disease head are standardized (z-score) before fusion. Nothing else changes.
- Decisions: the joint greedy action changes in 7.051% of states. The greedy action of each head taken alone is unchanged.
- Estimated value, setting of Table 1(a): WISt falls from 76.443 to 65.700, paired difference -10.743 (95% interval -14.646 to -7.099).
- Why raw-scale fusion is the better choice here. Both heads estimate the same quantity, the expected discounted 90-day survival return. A wide spread across a head's actions means that the choice matters more for that disease in this state. Standardizing each head in each state removes exactly this information: a head that is almost indifferent between its actions gets the same say as one for which the choice is decisive. Standardization also flattens the softmax policy used in OPE, so we read the result as "normalization does not help", not as a precise effect size.
In short: the weights decide which disease has priority, the shared reward scale and the anchor keep the two heads comparable, and normalizing the heads did not improve on this.
[1/4] Overview: greedy policy evaluated directly, joint behavior model tested, splits documented
Code (anonymous): https://anonymous.4open.science/r/BIFOCAL-7DFF/README.md
Thank you for the precise review, and for noting that the paper states what the routing rule does and does not do. We ran the checks you ask for. This overview follows your review point by point.
Where to find what. Part 2/4: target policy, behavior policy and the four checks you ask for (W1, Q2). Part 3/4: data splits and selection (Q1). Part 4/4: Q-head scales (W2).
Critical Weaknesses
Questions For Authors
What holds after these checks. On the comorbid cohort, BIFOCAL is first on truncated WIS in each of the three settings: the softmax policy of the submission, the joint behavior model, and the greedy policy on the strict subset.