acceptodds
Under review as a conference paper at ICLR 2027

Not All Old Knowledge Is Yours Alone: Direction-Wise Budgeted Release for Continual Multi-Agent Coordination in Comorbid Treatment

Abstract

Clinical policies are usually learned one disease at a time, but real patients present with combinations of conditions whose treatments interact. We study offline reinforcement learning over a growing team of disease-specific agents, introduced in stages of increasing comorbidity. This couples two problems: agents must learn to condition on one another, yet the updates that install these dependencies overwrite representations earlier policies rely on. We let a single spectral memory serve both roles — a private singular-value subspace that identifies which historical directions an update would disturb, and a shared basis that defines the coordinates through which co-present agents exchange patient information. Rather than protecting all historical directions uniformly, we compare spectral responses across agents on time-aligned observations and allocate a bounded adaptation budget as a continuous knapsack problem. We evaluate on sepsis, kidney injury, and respiratory failure cohorts, measuring every checkpoint on all single-, two-, and three-disease test sets.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, multi-agent reinforcement learning, healthcare application, clinical decision support

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8UVPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper considers continual offline multi-agent reinforcement learning for comorbid ICU treatment. Three disease-specific agents, representing sepsis, acute kidney injury, and respiratory failure, are trained sequentially on single-disease, pairwise-comorbidity, and three-disease cohorts. COSTAR-MARL stores historical representation directions in private SVD memories, transmits teammate information through a shared spectral interface, and estimates cross-agent functional support by correlating component responses on aligned patient observations. It then allocates a per-layer budget for releasing historically protected gradient directions according to current gradient alignment, historical spectral importance, and estimated teammate support. The paper proves an exact retained-response drift identity and the optimality of greedy allocation for its linear surrogate, and reports improvements in final and across-stage truncated weighted importance-sampling estimates on MIMIC-IV and eICU.

Critical strengths
  1. Relevant problem and a reasonably coherent mechanism. The stability–plasticity problem becomes more challenging as a fixed set of disease agents is reused in new team configurations. The combination of private spectral memory, cross-agent response comparison, and direction-wise release provides a meaningful extension of the historical-subspace-protection viewpoint.

  2. Substantial empirical reporting and careful theoretical qualifications. The manuscript evaluates seven clinical cohorts on two ICU datasets, provides a broad comparison set, component and communication ablations, seed-level variation, effective sample sizes, and a permutation audit.

Critical weaknesses
  1. Training uses teammate messages and team masks, but functional_context.py computes actor/critic correlations with zero-filled team contexts. The code also replaces the stated directional match-fraction statistic with an averaged correlation score. Both discrepancies can alter which directions are released.

  2. Correlated responses do not establish that a teammate can replace historical knowledge, particularly when that teammate is absent from an earlier single-disease team. The ablations do not isolate genuine support from altered release budgets.

  3. The drift guarantee applies to virtual SGD, not the clipped Adam updates used in training, while the knapsack result optimizes a chosen surrogate. Key subspace baselines use different optimizers or eligible data, small across-stage margins over SGP across only three runs.

Questions for authors
  1. Why functional support for actor.0 and critic.0 is calculated using zero team contexts rather than the actual teammate messages and masks?

  2. Is the intended (\phi_{ij}) the fraction of matched directions in a specific layer, or the symmetric mean-best-correlation graph score used in functional_relation_graph?

  3. When a supported historical direction is released because of a new teammate, what protects the previous policy on a singleton team where that teammate is absent? Can a matched-budget perturbation/permutation experiment show that genuine support predicts a lower historical-task cost than random or merely high-correlation support?

  4. To my knowledge, the two datasets used in the paper (MIMIC-IV and eICU) require an application for access and a usage certificate; could the authors provide relevant documentation to verify this?

  5. What joint treatment action is executed when sepsis and AKI agents recommend incompatible fluid levels? Are the reported per-agent WIS scores intended as marginal disease-specific policy evaluations or as a valid estimate of the combined team's value?

  6. How are all target-action probabilities in Eq. (29) computed in the actual evaluation script? Specifically, are the full aligned team contexts supplied, and why is the fitted behavior reference conditioned on (o_i) rather than the information on which the logged clinical action plausibly depends? Can the authors provide cohort-level ESS/overlap diagnostics and a second OPE estimator?

  7. Can the authors provide optimizer- and sample-matched GPM/SGP and selective-release baselines, paired uncertainty intervals, and a comparison of actual Adam-induced historical drift against the virtual-SGD quantity in Theorem 2? Is the budget intended to be per layer or global across the team?

10 comments
Reviewer_8UVPOpenReview ↗

Thank you for your response.

While some of my concerns have been addressed, I still believe that the paper falls substantially short of the minimum standards for acceptance at ICLR. Therefore, I will maintain my original rating.

We sincerely apologize for not responding sooner. We are still working on additional experiments to address your concerns and have not yet posted any response to your review. However, your comment mentions that some of your concerns have already been addressed, so we were wondering if this comment might have been intended for another paper.

Although our response is coming rather late, we would be very grateful if you could still take the time to carefully consider it once it is posted. We hope it may help address your concerns and possibly lead you to reconsider your rating.

We apologize again for the delay and any inconvenience caused.

AuthorsOpenReview ↗

[1/8] Overview: three weaknesses and seven questions answered with new controls on MIMIC-IV

Code: https://anonymous.4open.science/r/COSTAR-MARL-rebuttal-206D/README.md

Thank you for reading the code and for asking questions that can be checked directly. We ran the controls you requested on MIMIC-IV and report them below.

Protocol for all new numbers. All new experiments are on MIMIC-IV. Every method uses the same five training seeds (42 to 46), weighted equally, unless a larger seed count is stated. WISt truncates trajectory weights at 100. Paired 95% intervals describe variation across training seeds on the fixed test set.

Main results.

  • Matched training conditions. With optimizer, samples and batch order matched, COSTAR-MARL is above GPM by 5.57 [3.85, 7.29], above SGP by 2.16 [0.27, 4.04] and above hard projection by 5.55 [3.40, 7.69] on final WISt.
  • Second estimator. FQE places COSTAR-MARL above SGP by 14.29 [7.20, 21.37], in 5 of 5 seeds.
  • Teammate support. Over 20 seeds, COSTAR-MARL is above its no-support variant by 2.40 [1.23, 3.57].
  • Code. The two discrepancies you found are real and do not change the results.

Critical Weaknesses

  • W1, "[the code] computes actor/critic correlations with zero-filled team contexts [...] replaces the stated directional match-fraction statistic with an averaged correlation score". Both confirmed in the code; neither changes the results. True team context changes final WISt by -0.20 [-0.65, 0.25]; the match-fraction statistic changes support scores by at most 1e-5. Part 2/8.
  • W2, "Correlated responses do not establish that a teammate can replace historical knowledge [...] The ablations do not isolate genuine support from altered release budgets". We agree on the first point, and the paper says so. Support has a measured contribution: +2.40 [1.23, 3.57] over 20 seeds. The matched-budget controls you proposed give the same perturbation for true and permuted support. Part 3/8.
  • W3, "The drift guarantee applies to virtual SGD, not the clipped Adam updates used in training [...] Key subspace baselines use different optimizers or eligible data". Matched baselines added; drift under Adam measured and explained. COSTAR-MARL is above GPM by 5.57 [3.85, 7.29] and above SGP by 2.16 [0.27, 4.04] on final WISt. Parts 7/8 and 8/8.

Questions For Authors

  • Q1, "zero team contexts rather than the actual teammate messages and masks". Tested: no effect on the results. -0.20 [-0.65, 0.25]. Part 2/8.
  • Q2, "the fraction of matched directions in a specific layer, or the symmetric mean-best-correlation graph score". The latter; the text is corrected to match the code. The swap changes support scores by at most 1e-5. Part 2/8.
  • Q3, "what protects the previous policy on a singleton team [...] matched-budget perturbation/permutation experiment". The agent's own memory, caps and budget: with the filter, old AKI behaviour stays closer (KL 1.72 vs 9.88 without it). Both experiments you proposed were run. Part 3/8.
  • Q4, "could the authors provide relevant documentation to verify this?". Yes. The authors who accessed and analysed the two databases hold the required credentialed access. We can provide the access records and training certificates through a confidential channel designated by the Area Chair, consistent with anonymous review. Answered here.
  • Q5, "What joint treatment action is executed [...] marginal disease-specific policy evaluations or [...] the combined team's value?". Marginal; a joint-action evaluation is added. Joint WISt 72.42, the highest among entries with ESS above 30. Part 4/8.
  • Q6, "How are all target-action probabilities in Eq. (29) computed [...] cohort-level ESS/overlap diagnostics and a second OPE estimator?". Described; both added. FQE on MIMIC-IV: +14.29 [7.20, 21.37] over SGP. Parts 5/8 and 6/8.
  • Q7, "optimizer- and sample-matched GPM/SGP and selective-release baselines, paired uncertainty intervals [...] per layer or global". All provided; the budget is per layer. Above GPM by 5.57 and SGP by 2.16 under matched conditions. Parts 7/8 and 8/8.
Reviewer_h8AQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This is a solid methodological paper to introduce a novel approach for RL in a multi-agent and continual learning set-up. Its innovation lies in the integration of several existing algorithm elements in a logical and practical way. The chosen use case was based on MIMIC database and framed as an offline RL and off-policy evaluation problem hence inhering their limitations that lower the level of clinical utility that one can claim with this novel approach.

Critical strengths
  1. The problem was well articulated and motivated the choice of various algorithm elements
  2. Logical and practical integration of individual algorithm elements SVD, Knapsack etc.
  3. Proposed selective filtering approach is novel and sound
Critical weaknesses
  1. Offine RL and OPE carry inherent challenges with using observational data for causal ML. Having chosen offline RL and OPE to develop and test novel ideas in this study will need to carefully address those challenges. The proposed approach can potentially be adapted and expanded as a solution to more general problem of continual learning that involve applications of multiple models or model versions - but with more defensible outcome.

  2. Authors chosen those direct observations as state variable. However, the true/hidden state of patients may not completely captured by these direct observations.

  3. Sepsis, AKI, and respiratory failure in ICU settings are all dynamic processes. It seems the identification of the three cohors is done once to create training data, which may not reflect the clinical set-up. what is the the starting point (t0) of each trajectory? Is it the first time the condition is noted by the clicians or the first treatment initiated?

  4. A critical baseline is missing: frozen disease experts with lightweight coordination adapters. I think this is the most important issue for the authors to address. The paper presents COSTAR-MARL as a solution to the stability–plasticity problem that arises as the disease team expands, but it never tests the simplest solution to that problem. Once each agent is trained in D1, one could freeze it and add a small adapter, in the spirit of progressive networks (Rusu et al., 2016) or adapters (Houlsby et al., 2019). The adapter reads the teammates' messages and is switched on only when more than one agent is active. This rules out forgetting on single-disease cohorts by construction. With three diseases there are only four combinations, so per-combination adapters frozen after their stage would rule out forgetting everywhere. The design also suits the data: the three-disease cohort has only 1,106 training stays on MIMIC-IV (Table 3). The authors may have tried to justify excluding methods that rely on task identity by asserting that "our setting has no such identity" (Appendix A), because the active team changes within a stage and the same agent joins several teams. However, this justification seems weak, because the composition is in fact known for every sample, in training and at evaluation. The team mask is concatenated into every agent's policy input (Eq. 3), and the full set of agents is fixed in advance, with "a fixed slot [...] whether or not it is in the current team." The lack of task index per stage does not prevent conditioning on the composition, which is all this baseline needs.

  5. The evaluation never scores the team, which leaves the core motivation of the work somewhat unclear. The paper motivates the multi-agent formulation through interacting treatments: "Fluid administration, for example, may support perfusion in sepsis while aggravating fluid overload in AKI or pulmonary edema in RF," so "Comorbid treatment therefore requires not only disease-specific decision making, but also coordination across interacting treatment objectives" (Section 1). However, WISt is computed per agent (Eqs. 29–30, Appendix E.3). The importance weight for agent j multiplies only agent j's action probabilities, so every other intervention is implicitly left as recorded care. Each reported score therefore estimates the value of one agent acting while clinicians handle everything else, and the joint policy that would actually be deployed is never evaluated. The authors acknowledge that agents sharing a treatment variable "are evaluated independently, so reconciling conflicting recommendations at the bedside remains open" (Section 5), but this is precisely the fluid example used to motivate the work. As a result, the reported gains cannot be attributed to coordination.

Citations: Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., & Gelly, S. (2019). Parameter-efficient transfer learning for NLP (arXiv:1902.00751). arXiv. https://arxiv.org/abs/1902.00751

Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., & Hadsell, R. (2022). Progressive neural networks (arXiv:1606.04671). arXiv. https://arxiv.org/abs/1606.04671

Questions for authors

1.Large gains on a survival-based return, such as 87.2 against the clinicians’ 47.7 for sepsis-only patients in one run, can come from confounding by severity. Please comment on this.

  1. AI Clinician and Ventilation FQI score below zero. If these policies are near-deterministic, WIS keeps very few stays for them. The paper does not say how they were made stochastic for evaluation

  2. Each agent is graded on its own actions. Sepsis and AKI agents both set fluids, but the joint plan is never evaluated. How would a frontline clinician be given such information?

  3. Releasing every cap scores below plain training. The authors offer an explanation they have not tested.

  4. Only training and test splits are described, with no separate validation set for choosing β, the caps or the 0.35 threshold.

7 comments
AuthorsOpenReview ↗

[1/7] Overview: frozen-expert baseline added (+7.04 for COSTAR-MARL), team now scored with a joint-action evaluation, second estimator added

Code: https://anonymous.4open.science/r/COSTAR-MARL-rebuttal-206D/README.md

Thank you for the constructive review and for proposing the frozen-expert baseline. We implemented it at two adapter sizes, added a joint-action evaluation of the team and a second off-policy estimator, and answer each point below.

Protocol for all new numbers. All new experiments are on MIMIC-IV. Every method uses the same five training seeds (42 to 46), weighted equally, unless another seed set is stated. WISt truncates trajectory weights at 100. Paired 95% intervals describe variation across training seeds on the fixed test set.

Main results.

  • Frozen experts with adapters. COSTAR-MARL's final WISt is higher by 7.04 [5.03, 9.05] with 16-unit adapters and by 7.36 [4.78, 9.93] with 64-unit adapters.
  • The team is scored. Under an explicit rule for the shared fluid variable, COSTAR-MARL's joint WISt is 72.42, the highest among entries with ESS above 30.
  • Second estimator. FQE places COSTAR-MARL above SGP by 14.29 [7.20, 21.37], in 5 of 5 seeds.

Critical Weaknesses

  • "[...] RL and OPE carry inherent challenges with using observational data for causal ML". Addressed with a second estimator, coverage for every cohort-agent pair, and severity stratification. FQE: +14.29 [7.20, 21.37] over SGP. Part 4/7.
  • "the true/hidden state of patients may not completely captured by these direct observations". We agree; every method is compared on the same observations. Part 5/7.
  • "[...] starting point (t0) of each trajectory?". The start of the earliest retained 4-hour window of each disease view, a median of 2.9 to 12 hours before the disease marker. Part 5/7.
  • "A critical baseline is missing: frozen disease experts with lightweight coordination adapters. I think this is the most important issue for the authors to address." Added at two adapter sizes. COSTAR-MARL is higher by 7.04 [5.03, 9.05]. Part 2/7.
  • "The evaluation never scores the team, which leaves the core motivation of the work somewhat unclear." Joint-action evaluation added. Joint WISt 72.42. Part 3/7.

Questions For Authors

  • Q1, "can come from confounding by severity". The gap is present inside severity strata, and we do not read it as a survival gain. +21.3, +34.9 and +29.9 WISt points in the three SOFA strata with adequate support. Part 4/7.
  • Q2, "The paper does not say how they were made stochastic for evaluation". They were not; we now report their coverage and a smoothing check. 1.24% and 0.61% of stays are fully matched. Part 6/7.
  • Q3, "How would a frontline clinician be given such information?". As one joint recommendation per 4-hour window, with a single fluid level. Part 3/7.
  • Q4, "The authors offer an explanation they have not tested." Tested. Two-sided release at the same budget cuts historical drift from 0.0729 to 0.0331. Part 7/7.
  • Q5, "no separate validation set for choosing β, the caps or the 0.35 threshold". Correct; sensitivity sweeps added. Every β from 0.05 to 0.25 gives a final WISt between 59.17 and 59.79. Part 7/7.
AuthorsOpenReview ↗

[2/7] W4 Frozen experts with coordination adapters: baseline added at two sizes; COSTAR-MARL is higher by 7.04 and 7.36 final WISt (new results)

W4. "A critical baseline is missing: frozen disease experts with lightweight coordination adapters. I think this is the most important issue for the authors to address. [...] The lack of task index per stage does not prevent conditioning on the composition, which is all this baseline needs."

Answer: baseline added, at two adapter sizes. COSTAR-MARL's final WISt is higher by 7.04 [5.03, 9.05] against 16-unit adapters and by 7.36 [4.78, 9.93] against 64-unit adapters. You are right that the team mask identifies the composition, and we use it exactly as you describe to select the adapter. This addresses your concern that the simplest solution to the stability-plasticity problem was not tested: it removes forgetting, and it also gives up the gains of continued adaptation.

Setup: MIMIC-IV, five seeds. Both methods start from the same D1 single-disease expert weights. (new)

  • The expert networks and their message modules are frozen after D1. Single-disease cohorts use the frozen experts directly.
  • For each of the four comorbid compositions (three pairs and the triple), every participating expert gets a residual adapter that reads its own state features and the teammates' messages. The team mask selects the adapter. Each adapter is frozen once its composition has been trained.
  • The 16-unit version adds 31,071 parameters over the four compositions. D2 and D3 use the same number of updates as COSTAR-MARL.
MetricFrozen + adapters (16)COSTAR-MARLPaired difference
Final WISt, all 12 pairs52.3059.33+7.04 [5.03, 9.05]
Final WISt, three-disease cohort53.0158.14+5.12 [3.91, 6.34]
Backward transfer, nine earlier pairs0.00+6.42+6.42 [1.59, 11.26]

The higher value in each row is in bold.

  • The structural advantage you predicted is confirmed. The frozen design keeps every earlier policy exactly: backward transfer is 0 in all five seeds.
  • Continued adaptation is worth more than freezing. COSTAR-MARL is higher by 7.04 over all pairs, and higher in all five seeds on the three-disease cohort, the small cohort you point to (1,106 training stays).
  • The gap is not a capacity effect. Enlarging the adapters from 16 to 64 hidden units gives 51.97 (16 units: 52.30); COSTAR-MARL is higher by 7.36 [4.78, 9.93].
  • Appendix A. We will rewrite Appendix A. The argument that the setting has no task identity is removed: the team mask gives the composition, composition-specific adapters are feasible, and they are now a baseline.

This suggests that freezing solves forgetting at the price of adaptation: on this task, letting the experts keep learning under a release budget gives a higher final value, including on the smallest cohort.

Scope: COSTAR-MARL's backward transfer is positive on average and uneven across cohorts, as Section 4.3 reports: it gains on sepsis cohorts and loses on the AKI-only and RF-only cohorts, where the frozen design by construction loses nothing. All values are offline estimates.

AuthorsOpenReview ↗

[3/7] W5, Q3 Scoring the team: explicit fluid rule, joint WISt with ESS for all methods (COSTAR-MARL 72.42, highest among supported entries); one fluid recommendation per window for the clinician

W5. "The evaluation never scores the team, which leaves the core motivation of the work somewhat unclear. [...] As a result, the reported gains cannot be attributed to coordination."

Q3. "Each agent is graded on its own actions. Sepsis and AKI agents both set fluids, but the joint plan is never evaluated. How would a frontline clinician be given such information?"

Answer: the team is now scored. For the shared fluid variable we define one executed action, the mean of the two recommended doses, and evaluate the joint action over the whole stay. Under this rule COSTAR-MARL has the highest joint WISt among all entries with ESS above 30 (72.42). A clinician would receive one fluid recommendation per 4-hour window, together with the sepsis agent's vasopressor level and the AKI agent's ultrafiltration level.

The rule:

  • Each agent's fluid level is converted to the median dose of that level in its training data. The two doses are averaged and mapped back to one fluid level.
  • This level, the sepsis agent's vasopressor level and the AKI agent's ultrafiltration level form the joint action.
  • The joint probability sums over all pairs of proposals that produce the recorded action (normalization checked). WISt is computed over the whole stay.

Setup: Sepsis+AKI, 5,654 test stays, five seeds, cap 100. ESS is the five-seed mean. (new)

MethodJoint WIStESS
CFCQL84.234.75
CQL81.856.56
COSTAR-MARL72.42140.45
COSTAR-MARL, no messages71.31145.37
Balanced Joint67.3090.16
COMAD63.4948.83
Adam-NSCL62.8068.89
LwF59.22186.08
EWC59.19226.00
Factored Single Policy58.44265.40
Sequential FT54.91472.39
SGP53.98364.99
Scratch-D252.53749.30
GPM48.82332.90
Cumulative Replay48.20342.23
GEM48.20894.71
B3C47.6430.25
Scratch-D344.84597.40

Higher is better in both columns; the best value in each column is in bold.

  • Among entries with usable support, COSTAR-MARL is highest. It is above every continual-learning baseline by at least 8.9 (next: COMAD 63.49), above Balanced Joint by 5.1 and above SGP by 18.4.
  • The two entries above it rest on fewer than 7 effective stays. CFCQL and CQL put positive weight on 4.8 and 6.6 stays on average, about 0.1% of the cohort. Without truncation they fall to 51.59 and 64.17.
  • What the clinician receives. One joint recommendation per window: a single fluid level, a vasopressor level and an ultrafiltration level. The two agents' original fluid proposals remain available and can be displayed when they differ.

This suggests that the agents form one executable team policy, and that its estimated value is the highest among the methods with usable support. The fluid conflict that motivates the paper is resolved by an explicit rule and evaluated.

Scope: this evaluation scores the team's joint action; it does not isolate the effect of message passing, since the no-message variant is close (71.31). It covers Sepsis+AKI, where the two views are synchronized; the three-disease cohort has too few strictly synchronized windows (2 stays) for a joint estimate. COSTAR-MARL's own ESS is 140 of 5,654 stays (2.5%). The joint evaluation uses inputs from the previous complete window. All values are offline estimates.

Reviewer_ibW3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper frames ICU care for patients with multiple conditions as a continual, offline, multi-agent RL problem, with one agent per disease: sepsis, acute kidney injury and respiratory failure. Training proceeds in three stages: single-disease cohorts, then all two-disease cohorts, and finally the three-disease cohort. The method, COSTAR-MARL, builds on gradient projection memory (GPM). For each agent and layer, it stores the top SVD directions of past layer inputs. In each minibatch, it then "releases" part of the gradient along selected protected directions instead of blocking them completely. Experiments are performed on multiple datasets.

Critical strengths

S1. The method is sound. The idea is well-motivated.

S2. The experiments are extensive. The experiments cover two ICU databases and 17 baselines spanning continual learning, offline MARL, clinical RL and joint/scratch training, plus ablations of individual components and of communication.

Critical weaknesses
  1. All agents receive the same reward; there is no disease-specific outcome such as renal recovery or ventilator-free days. The "disease-specific value functions" therefore all predict the same mortality outcome.

  2. Cohorts, preprocessing and data splits are not described; patient-level leakage is possible, and there is no validation set. The splits appear to be made by ICU stay rather than by patient: counts refer to "ICU stays rather than unique patients", "dependence between repeated stays of the same person is not modelled", and the audit uses only stay identifiers. Many MIMIC-IV patients have several ICU stays, so the same person can appear in both the training and test sets. Only training and test sets are described. How the baseline hyperparameters were chosen is not explained either.

  3. Some baseline results suggest unfair or under-tuned comparisons. On MIMIC-IV, Cumulative Replay performs worse than Sequential FT, and Balanced Joint, which is usually an upper bound in continual learning, performs worse than COSTAR-MARL. Replay rarely loses to naive fine-tuning. Training budgets also differ ("Balanced Joint and Scratch variants retain their respective training budgets"). The offline MARL baselines (COMAD, CFCQL, B3C, CQL) are "independently implemented" with stricter alignment rules, so they are trained on fewer transitions.

  4. The main text is hard to follow, and key information appears only in the appendix. Sections 3.1–3.4 introduce many symbols and terms ("filtered layer", "native", "virtual step", "zero completion") with little intuitive explanation. Figure 1 is abstract and does not use real data. The basic clinical and data information needed to judge validity (cohort definitions, features, action bins, outcome windows and the behaviour model) is missing from both the main text and the appendix. The abstract does not mention that all results are off-policy estimates with overlapping intervals, or that BWT varies greatly across runs. There is no qualitative clinical analysis, e.g., of how the recommended fluid, vasopressor or PEEP doses differ from clinicians' choices for comorbid patients. Such an analysis would show whether the "coordination" is clinically meaningful.

Questions for authors

Please refer to above

8 comments
AuthorsOpenReview ↗

[1/8] Overview: second estimator, joint-action evaluation, patient-overlap check and cohort definitions added; your five weaknesses answered point by point

Code: https://anonymous.4open.science/r/COSTAR-MARL-rebuttal-206D/README.md

Thank you for the detailed review. It led to a second estimator, a joint-action evaluation, a patient-level overlap check and a full description of cohorts and baselines. Your Questions For Authors repeat the five Critical Weaknesses word for word, so each answer below covers both.

Protocol for all new numbers. All new experiments are on MIMIC-IV. Every method uses the same five training seeds (42 to 46), weighted equally, unless another seed set is stated. WISt truncates trajectory weights at 100. Paired 95% intervals describe variation across training seeds on the fixed test set.

W1. Single estimator

  • "No second estimator." FQE added for all 17 methods. COSTAR-MARL is above SGP by 14.29 [7.20, 21.37] and above all nine continual-learning baselines. Part 2/8.
  • "these figures would imply very large reductions in mortality [...] which is not credible". We agree that they should not be read as mortality reductions; FQE gives much smaller values. Part 2/8.
  • "Without truncation, the rankings change". Untruncated WIS rests on about one effective trajectory (ESS 1.15). We agree that stability across caps is weak evidence and rely on FQE instead. Part 3/8.
  • "Undescribed behaviour model [...] Per-cohort ESS is missing." Described; ESS reported for all 12 pairs. A small ESS is common to all 17 methods (63 to 1,333 in Table 11) and to published work in this field. Part 3/8.
  • "recommended doses are not checked against clinician actions". Checked. Part 8/8.

W2. The team is never evaluated

  • "A joint-action evaluation, with a rule for resolving shared treatment variables, is needed." Added. Joint WISt 72.42, the highest among entries with ESS above 30. Part 4/8.
  • "All agents receive the same reward". Correct, by design; the term is renamed. Part 4/8.

W3. Cohorts, splits, validation

  • "The paper gives no inclusion criteria". Given, with features, missing-data rules and action bins. Part 5/8.
  • "'Ultrafiltration output' [...] applies only to patients on renal replacement therapy". Quantified: 3.28% of AKI-cohort test stays have a positive record. Part 5/8.
  • "the same person can appear in both the training and test sets". Confirmed and tested. Without overlapping patients COSTAR-MARL moves from 59.33 to 59.73 and stays above SGP, GPM, Sequential FT and Replay. Part 6/8.
  • "How the baseline hyperparameters were chosen is not explained". Documented for all 21 methods; sweeps added. Part 6/8.

W4. Baselines

  • "Replay rarely loses to naive fine-tuning. Training budgets also differ". With the same five seeds, Replay is level with Sequential FT and Balanced Joint is above COSTAR-MARL; all three run 24,184 updates. Part 7/8.
  • "trained on fewer transitions", "departs substantially from the original method", "Across-stage Avg." Quantified (4.8% to 6.7% in three cohorts), relabelled HRRT-lite, and explained. Part 7/8.
  • Matched comparison (new). With optimizer, samples and batch order matched, COSTAR-MARL is above GPM by 5.57 [3.85, 7.29] and above SGP by 2.16 [0.27, 4.04]. Part 6/8.

W5. Presentation

  • "The main text is hard to follow, and key information appears only in the appendix." Revised. Part 8/8.
  • "There is no qualitative clinical analysis". Fluid recommendations compared with clinicians' for single-disease and comorbid patients. Part 8/8.
AuthorsOpenReview ↗

[2/8] W1 (points 1 and 2) Second estimator: FQE added for 17 methods (+14.29 over SGP); Table 8 magnitudes are not mortality reductions, and FQE gives much smaller values

W1, points 1 and 2. "No second estimator. There is no FQE, doubly robust or model-based estimate [...]" and "[...] these figures would imply very large reductions in mortality from a policy learned on the same logged data, which is not credible."

Answer: FQE added for all 17 methods on MIMIC-IV. It confirms the paper's main comparison: COSTAR-MARL is above SGP by 14.29 [7.20, 21.37] and above all nine continual-learning baselines. On Table 8 we agree: the figures are WISt points from a single run and should not be read as mortality reductions. FQE gives much smaller values for the two pairs you cite, and what we claim is a comparison between learned methods, not a gain over clinicians.

Second estimator. Policies are frozen. One Q-network is fitted for each cohort-agent pair and policy seed, with the auxiliary seed, held-out split and minibatch order fixed across methods: 17 methods x 5 seeds x 12 pairs = 1,020 Q-models. (new)

  • Against the strongest subspace baseline on WISt: COSTAR-MARL 42.96 vs SGP 28.67, +14.29 [7.20, 21.37], higher in 5 of 5 seeds.
  • Both estimators agree against the subspace methods. WISt: 59.33 vs 55.34 (SGP) and 52.64 (GPM). FQE: 42.96 vs 28.67 (SGP) and 28.77 (GPM).
  • Against all continual-learning baselines: COSTAR-MARL (42.96) is above all nine: Sequential FT 42.24, Cumulative Replay 41.47, COMAD 39.40, EWC 39.08, Adam-NSCL 33.11, LwF 33.06, GEM 30.83, GPM 28.77, SGP 28.67.

The two pairs of Table 8. Seed 42, sepsis agent. (new)

CohortWIStFQEClinician return
Sepsis only87.1945.7947.68
Sepsis+RF53.3322.426.22
  • The distance to the clinician reference depends on the estimator. Sepsis only: +39.51 on WISt and -1.89 on FQE. Sepsis+RF: +47.11 on WISt and +16.20 on FQE.
  • The Sepsis+RF pair rests on little data. The sepsis agent's ESS in this run is 20.6 (4.6% of 452 test stays), and the clinician reference itself is uncertain: 6.22 with a 95% interval of [-1.92, 14.01] (Table 14).
  • Table 8 is one run and reports WISt points, as its caption states. We make no claim of clinical benefit from these figures.

This suggests that the comparison among methods, which is what the paper claims, is supported by two estimators, while the distance to the clinician reference is estimator-dependent and should not be read clinically.

Scope: FQE carries its own bias. The FQE margin over Sequential FT is small (+0.72 [-4.98, 6.41]), and three methods outside the continual-learning group are higher on FQE (CQL 51.00, CFCQL 46.03, Balanced Joint 43.61); CQL and CFCQL have the lowest WISt ESS in Table 11 (110 and 63). A doubly robust estimator was not run. The check of recommended doses against clinician actions is in part 8/8.

AuthorsOpenReview ↗

[3/8] W1 (points 3 and 4) Truncation, behaviour model and coverage: behaviour reference described; ESS reported for all 12 pairs; small ESS is common to all 17 methods and to published work

W1, point 3. "Without truncation, the rankings change [...] Stability across caps of 10–1000 is to be expected once the weights are capped, so it is weak evidence of robustness."

Answer: we agree that stability across caps is weak evidence, and we no longer use it as an argument for robustness. Untruncated WIS is not informative in either direction: for COSTAR-MARL it has an effective sample size of 1.15 on MIMIC-IV and 1.41 on eICU, that is, about one trajectory. The check that does not depend on truncation is FQE (part 2/8).

  • Untruncated estimates rest on one trajectory. With an ESS of 1.15 and 1.41, COSTAR-MARL's untruncated value on either database reflects a single heavily weighted stay, so its lead in that column on MIMIC-IV and its deficit on eICU are equally uninformative. (new)
  • Under truncation, the estimate and its support are reported together. COSTAR-MARL on MIMIC-IV at caps 10, 100 and 1000: WISt 60.08, 59.33 and 58.11, with ESS 917.5, 738.1 and 581.5. (new)

This suggests that the untruncated column should not be used to rank any method, ours included, and that robustness is better judged by a second estimator than by the range of caps.

Scope: truncation trades variance for bias. The finite-cap results describe sensitivity within the stated range only.

W1, point 4. "Undescribed behaviour model. [...] Per-cohort ESS is missing."

Answer: described below, and ESS is now reported for all 12 cohort-agent pairs. A small ESS is a problem of importance-sampling evaluation in this field, not of our method: it affects every method in our comparison and published work alike. All methods are scored with the same behaviour reference on the same test stays, so the comparison between them is like for like.

  • Behaviour model. For each cohort-agent pair: mini-batch K-means on the agent's local clinical state, then smoothed action frequencies from the clinician actions within each cluster. It is fitted on the training split only and reused for all methods and seeds.
  • Calibration. Expected calibration error on the test split (10 equal-width bins of the predicted probability of the top action), for the two small cohorts you name: Sepsis+RF 0.277 (sepsis agent) and 0.237 (RF agent); AKI+RF 0.172 (AKI agent) and 0.240 (RF agent). (new)
  • An indexing detail found while checking. For non-terminal transitions the reference paired the current state with the next recorded action. Refitting on the current action moves COSTAR-MARL's three-seed mean from 59.79 to 60.31; all comparisons keep the original shared reference.

Coverage. MIMIC-IV, the three runs behind Table 11 (seeds 42, 43, 45), cap 100. Each cell gives the mean ESS and its share of test stays. (new)

CohortStaysSepsis agentAKI agentRF agent
Sepsis4,128322.8 (7.8%)n/an/a
AKI9,294n/a3,040.8 (32.7%)n/a
RF1,240n/an/a126.3 (10.2%)
Sepsis+AKI5,654291.4 (5.2%)4,049.2 (71.6%)n/a
Sepsis+RF45278.9 (17.5%)n/a55.1 (12.2%)
AKI+RF514n/a348.9 (67.9%)25.4 (4.9%)
Sepsis+AKI+RF73898.3 (13.3%)642.2 (87.0%)69.6 (9.4%)

The 12 values average to 762.416, as in Table 11 (a mean over 36 run-pair values).

A small ESS is a problem of the field and of all methods.

  • Why it arises. Importance weights concentrate on the stays where the evaluated policy keeps agreeing with the recorded actions, and over many decisions per stay such agreement becomes rare. Gottesman et al. (Nature Medicine, 2019) describe this for sequential treatment policies in general and report a few dozen effective patients on a sepsis cohort of 3,855.
  • Every method in our comparison is affected. In Table 11 the average ESS of the 17 methods ranges from 63 (CFCQL) to 1,333 (COMAD), on pairs with about 2,510 test stays on average. COSTAR-MARL (762) is in the middle: 7 methods are lower, among them GPM (542), SGP (670) and Balanced Joint (664), and 9 are higher.
  • Published studies are at the same level. Tang et al. (NeurIPS 2022): a test ESS of 178 for both the baseline and the proposed method, on a 15% test split of 19,287 sepsis patients, about 6%. Liu et al. (UAI 2022): 207 to 437 on 2,995 test trajectories for all five learned methods, 7% to 15%. Our sepsis and RF agents are at 4.9% to 17.5%, and the AKI agent at 32.7% to 87.0%.

This suggests that limited coverage is a limit of the estimator in this field and applies to all compared methods; it is not a weakness specific to COSTAR-MARL. WISt remains the common reference, and FQE is added as a second one.

Scope: a low ESS limits what any single pair can show, for every method. Five of the 12 pairs have an ESS below 100; for these we rely on the comparison across methods and on FQE rather than on the size of the score.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response [3/3]: the new results in one place (Tables A to E)

The new results in one place. (new) marks results added in this response. The best value in each row (Tables A and C) or column (Tables B and D) is in bold. WISt is truncated weighted importance sampling with cap 100; FQE is fitted Q evaluation. MIMIC-IV, the same five seeds for every method unless stated.

A. COSTAR-MARL against the subspace-protection baselines, in every evaluation

EvaluationCOSTAR-MARLSGPGPM
Final WISt, all test stays59.3355.3452.64
Final WISt, unseen patients only (new)59.7356.9154.53
Final WISt, optimizer-matched baselines (new)59.3357.1853.76
FQE (new)42.9628.6728.77
Joint-action WISt, Sepsis+AKI (new)72.4253.9848.82

Row 3 uses GPM and SGP retrained with Adam. Paired differences: +5.57 [3.85, 7.29] over GPM and +2.16 [0.27, 4.04] over SGP in row 3; +14.29 [7.20, 21.37] over SGP in row 4.

B. Second estimator: FQE (new) (1,020 Q-models; the methods reported in the individual responses)

MethodFQE
CQL51.00
CFCQL46.03
Balanced Joint43.61
COSTAR-MARL42.96
Sequential FT42.24
Cumulative Replay41.47
COMAD39.40
EWC39.08
Adam-NSCL33.11
LwF33.06
GEM30.83
GPM28.77
SGP28.67

COSTAR-MARL is above all nine continual-learning baselines. CQL and CFCQL have the lowest WISt ESS (110 and 63 in Table 11).

C. Frozen experts with coordination adapters (new)

MetricFrozen, 16 unitsFrozen, 64 unitsCOSTAR-MARL
Final WISt, all 12 pairs52.3051.9759.33
Backward transfer0.000.00+6.42

Paired differences in final WISt: +7.04 [5.03, 9.05] and +7.36 [4.78, 9.93].

D. Joint-action evaluation of the team (new) (Sepsis+AKI, 5,654 test stays)

MethodJoint WIStESS
CFCQL84.234.75
CQL81.856.56
COSTAR-MARL72.42140.45
COSTAR-MARL, no messages71.31145.37
Balanced Joint67.3090.16
COMAD63.4948.83
Adam-NSCL62.8068.89
LwF59.22186.08
EWC59.19226.00
Factored Single Policy58.44265.40
Sequential FT54.91472.39
SGP53.98364.99
Scratch-D252.53749.30
GPM48.82332.90
Cumulative Replay48.20342.23
GEM48.20894.71
B3C47.6430.25
Scratch-D344.84597.40

COSTAR-MARL is the highest among entries with ESS above 30; the two above it rest on fewer than 7 effective stays.

E. Coverage: ESS for every cohort-agent pair (new) (the three runs behind Table 11; mean ESS and its share of test stays)

CohortStaysSepsis agentAKI agentRF agent
Sepsis4,128322.8 (7.8%)n/an/a
AKI9,294n/a3,040.8 (32.7%)n/a
RF1,240n/an/a126.3 (10.2%)
Sepsis+AKI5,654291.4 (5.2%)4,049.2 (71.6%)n/a
Sepsis+RF45278.9 (17.5%)n/a55.1 (12.2%)
AKI+RF514n/a348.9 (67.9%)25.4 (4.9%)
Sepsis+AKI+RF73898.3 (13.3%)642.2 (87.0%)69.6 (9.4%)

What the tables show together. On MIMIC-IV, COSTAR-MARL is above GPM and SGP in all five evaluations (A) and above frozen experts with adapters at both sizes (C). Under an explicit joint-action rule it has the highest value among the entries with usable support (D). It is not first everywhere: Balanced Joint, which trains on all cohorts at once, is higher on per-agent WISt (62.83) and on FQE, and CQL and CFCQL are higher on FQE with very low WISt support (B). Coverage differs by agent (E), which is why the conclusions rest on comparisons between methods and on two estimators. All values are offline estimates.

AuthorsOpenReview ↗

General Response [2/3]: five shared concerns and how each is addressed (new results)

The main concerns of the three reviews fall into five groups. For each we state who raised it, how it is addressed, and why the result answers it. Tables A to E are in part 3/3.

A. Can a single importance-sampling estimator be trusted?

Raised by Reviewer ibW3 (W1, points 1 to 4), Reviewer h8AQ (W1, Q1) and Reviewer 8UVP (Q6).

Addressed by a second estimator and full coverage diagnostics: on FQE, COSTAR-MARL is above SGP by 14.29 [7.20, 21.37], in 5 of 5 seeds, and above all nine continual-learning baselines. (new, Tables A, B and E)

  • FQE uses no importance weights, so it does not share the weakness of WISt, and it gives the same direction against GPM and SGP.
  • ESS is now reported for all 12 cohort-agent pairs (4.9% to 87.0% of test stays). A small ESS is common to all 17 methods and to published studies, and all methods are scored with the same behaviour reference on the same stays.
  • We do not read the distance to the clinician reference as a clinical gain; the claims are comparisons between methods.

B. The team was never scored.

Raised by Reviewer ibW3 (W2, point 1), Reviewer h8AQ (W5, Q3) and Reviewer 8UVP (Q5).

Addressed by an explicit rule for the shared fluid variable and a joint-action evaluation: COSTAR-MARL's joint WISt is 72.42, the highest among entries with ESS above 30. (new, Table D)

  • The two fluid recommendations are merged into one executed dose, so what is evaluated is the policy that would be deployed.
  • COSTAR-MARL is above every continual-learning baseline by at least 8.9 and above Balanced Joint by 5.1. The two entries above it (CFCQL, CQL) rest on fewer than 7 effective stays.

C. Are the baselines fair and complete?

Raised by Reviewer ibW3 (W3 point 4, W4), Reviewer h8AQ (W4) and Reviewer 8UVP (W3, Q7).

Addressed by matched and new baselines: with optimizer, samples and batch order matched, COSTAR-MARL is above GPM by 5.57 [3.85, 7.29] and above SGP by 2.16 [0.27, 4.04]; it is above frozen experts with adapters by 7.04 [5.03, 9.05]. (new, Tables A and C)

  • The matched comparison removes the differences in optimizer and data that were questioned.
  • Frozen experts with adapters are the simplest answer to forgetting. They keep earlier policies exactly and give up about 7 points of final value, at both adapter sizes.
  • With the same five seeds for every method, Replay is level with Sequential FT, and Balanced Joint, which sees all cohorts at once, is above COSTAR-MARL (62.83 vs 59.33), as expected of a joint-training reference.

D. Cohorts, patient overlap and tuning.

Raised by Reviewer ibW3 (W3, points 1 to 4) and Reviewer h8AQ (W3, Q5).

Addressed by full cohort definitions and two checks: on test stays of unseen patients COSTAR-MARL moves from 59.33 to 59.73 and stays above SGP, GPM, Sequential FT and Replay; every β from 0.05 to 0.25 gives a final WISt between 59.17 and 59.79. (new, Table A)

  • Inclusion criteria, trajectory start, missing-data rules and action bins are in the response to Reviewer ibW3 (part 5/8).
  • No separate validation set was used; the sweeps measure sensitivity on the test set.

E. Does the mechanism do what the paper says, and does the code match?

Raised by Reviewer 8UVP (W1, W2, W3, Q1 to Q3, Q7) and Reviewer h8AQ (Q4).

Addressed by four controls. (new)

  • Code. Both differences are confirmed. True team context changes final WISt by -0.20 [-0.65, 0.25]; the match-fraction statistic changes support scores by at most 1e-5.
  • Teammate support. Over 20 seeds, COSTAR-MARL is above its no-support variant by 2.40 [1.23, 3.57] (this control also lowers the cap; see the response to Reviewer 8UVP, part 3/8).
  • Theory. Theorem 2 holds for the virtual SGD step it is stated for (error 2.28e-8). Under Adam the realized drift is larger, and the budget still orders it on average in every seed.
  • Full release. Two-sided release at the same budget cuts historical drift from 0.0729 to 0.0331, which supports the first link of the explanation given in the paper.

Why these results answer the concerns together. The concerns were of two kinds: whether the estimates can be trusted, and whether the comparison is fair and complete. The first is answered by a second estimator of a different kind and by coverage for every pair. The second is answered by matched baselines, a new frozen baseline and common seeds. Across all of them one claim does not change: in final value, budgeted release is above the methods that fully protect a stored subspace.

AuthorsOpenReview ↗

General Response [1/3]: thanks, what the reviewers recognized, and what we wish to underline

Code (anonymous): https://anonymous.4open.science/r/COSTAR-MARL-rebuttal-206D/README.md

We sincerely thank all three reviewers. Your comments have clearly improved this paper. We apologize for the late reply: we used the time to run almost every experiment that was requested, and the results are reported here and in the individual responses. Three parts follow: (1/3) what the reviewers recognized and what we wish to underline, (2/3) the shared concerns and how each is addressed, (3/3) the new results in one place.

1. What the reviewers recognized

  • A well-motivated problem and a sound method. Reviewer ibW3: "The method is sound. The idea is well-motivated." Reviewer h8AQ: "a solid methodological paper" whose problem "was well articulated". Reviewer 8UVP: "Relevant problem and a reasonably coherent mechanism."
  • A novel release rule. Reviewer h8AQ: "Proposed selective filtering approach is novel and sound". Reviewer 8UVP: "a meaningful extension of the historical-subspace-protection viewpoint".
  • Extensive experiments. Reviewer ibW3: "The experiments are extensive", with "17 baselines spanning continual learning, offline MARL, clinical RL and joint/scratch training". Reviewer 8UVP: "Substantial empirical reporting" and "a broad comparison set".
  • Careful theory and design. Reviewer 8UVP: "careful theoretical qualifications". Reviewer h8AQ: "Logical and practical integration of individual algorithm elements".

2. What we wish to underline

  • Almost every requested experiment was run. A second estimator for all 17 methods (1,020 fitted Q-models), optimizer-matched GPM and SGP, frozen experts with adapters at two sizes, a joint-action evaluation of the team for all methods, a 20-seed control of teammate support, a check on unseen patients, and a measurement of drift under Adam (1,680 observations). (new)
  • One conclusion holds in every evaluation of final value. COSTAR-MARL is above GPM and SGP, the subspace-protection methods it extends, on WISt, on FQE, under optimizer-matched training, on unseen patients and in the joint-action evaluation (Table A, part 3/3). On the across-stage average it is level with SGP (response to Reviewer 8UVP, part 7/8).
  • Checked against clinicians. The agents' fluid recommendations are compared with clinicians' recorded choices for single-disease and comorbid patients (response to Reviewer ibW3, part 8/8). (new)
  • Open to verification. Code and trained weights are public at the link above. The two differences between code and paper found by Reviewer 8UVP were confirmed and shown not to change the results.
  • To our knowledge, on the criteria below, a broader off-policy evaluation than in the published ICU reinforcement-learning studies we checked:
StudyValue estimatorsLearned methods comparedESS reportedCap sensitivity
Komorowski et al., Nat Med 2018WIS1nonot reported
Zhang et al., npj Digit Med 2022WIS3yesnot reported
Tang et al., NeurIPS 2022WIS2yesnot reported
Liu et al., UAI 2022truncated IS5yesnot reported
This workWISt, FQE (new)17yes, for all 12 cohort-agent pairs (new)yes (Tables 12 and 13)

Based on the main text of each paper; reference policies such as clinician, random or zero-drug are not counted as learned methods. All four studies report one importance-sampling estimator; this work adds an estimator without importance weights and reports coverage for every cohort and agent.

3. The result in one sentence

On MIMIC-IV, the final value of COSTAR-MARL is above that of the subspace-protection baselines GPM and SGP in every evaluation we ran: per-agent WISt, FQE, optimizer-matched training, unseen patients, and the joint-action evaluation (point estimates; intervals are in the individual responses).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.