- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper considers continual offline multi-agent reinforcement learning for comorbid ICU treatment. Three disease-specific agents, representing sepsis, acute kidney injury, and respiratory failure, are trained sequentially on single-disease, pairwise-comorbidity, and three-disease cohorts. COSTAR-MARL stores historical representation directions in private SVD memories, transmits teammate information through a shared spectral interface, and estimates cross-agent functional support by correlating component responses on aligned patient observations. It then allocates a per-layer budget for releasing historically protected gradient directions according to current gradient alignment, historical spectral importance, and estimated teammate support. The paper proves an exact retained-response drift identity and the optimality of greedy allocation for its linear surrogate, and reports improvements in final and across-stage truncated weighted importance-sampling estimates on MIMIC-IV and eICU.
Critical strengths
-
Relevant problem and a reasonably coherent mechanism. The stability–plasticity problem becomes more challenging as a fixed set of disease agents is reused in new team configurations. The combination of private spectral memory, cross-agent response comparison, and direction-wise release provides a meaningful extension of the historical-subspace-protection viewpoint.
-
Substantial empirical reporting and careful theoretical qualifications. The manuscript evaluates seven clinical cohorts on two ICU datasets, provides a broad comparison set, component and communication ablations, seed-level variation, effective sample sizes, and a permutation audit.
Critical weaknesses
-
Training uses teammate messages and team masks, but functional_context.py computes actor/critic correlations with zero-filled team contexts. The code also replaces the stated directional match-fraction statistic with an averaged correlation score. Both discrepancies can alter which directions are released.
-
Correlated responses do not establish that a teammate can replace historical knowledge, particularly when that teammate is absent from an earlier single-disease team. The ablations do not isolate genuine support from altered release budgets.
-
The drift guarantee applies to virtual SGD, not the clipped Adam updates used in training, while the knapsack result optimizes a chosen surrogate. Key subspace baselines use different optimizers or eligible data, small across-stage margins over SGP across only three runs.
Questions for authors
-
Why functional support for actor.0 and critic.0 is calculated using zero team contexts rather than the actual teammate messages and masks?
-
Is the intended (\phi_{ij}) the fraction of matched directions in a specific layer, or the symmetric mean-best-correlation graph score used in functional_relation_graph?
-
When a supported historical direction is released because of a new teammate, what protects the previous policy on a singleton team where that teammate is absent? Can a matched-budget perturbation/permutation experiment show that genuine support predicts a lower historical-task cost than random or merely high-correlation support?
-
To my knowledge, the two datasets used in the paper (MIMIC-IV and eICU) require an application for access and a usage certificate; could the authors provide relevant documentation to verify this?
-
What joint treatment action is executed when sepsis and AKI agents recommend incompatible fluid levels? Are the reported per-agent WIS scores intended as marginal disease-specific policy evaluations or as a valid estimate of the combined team's value?
-
How are all target-action probabilities in Eq. (29) computed in the actual evaluation script? Specifically, are the full aligned team contexts supplied, and why is the fitted behavior reference conditioned on (o_i) rather than the information on which the logged clinical action plausibly depends? Can the authors provide cohort-level ESS/overlap diagnostics and a second OPE estimator?
-
Can the authors provide optimizer- and sample-matched GPM/SGP and selective-release baselines, paired uncertainty intervals, and a comparison of actual Adam-induced historical drift against the virtual-SGD quantity in Theorem 2? Is the budget intended to be per layer or global across the team?
10 comments
We sincerely apologize for not responding sooner. We are still working on additional experiments to address your concerns and have not yet posted any response to your review. However, your comment mentions that some of your concerns have already been addressed, so we were wondering if this comment might have been intended for another paper.
Although our response is coming rather late, we would be very grateful if you could still take the time to carefully consider it once it is posted. We hope it may help address your concerns and possibly lead you to reconsider your rating.
We apologize again for the delay and any inconvenience caused.
[1/8] Overview: three weaknesses and seven questions answered with new controls on MIMIC-IV
Code: https://anonymous.4open.science/r/COSTAR-MARL-rebuttal-206D/README.md
Thank you for reading the code and for asking questions that can be checked directly. We ran the controls you requested on MIMIC-IV and report them below.
Protocol for all new numbers. All new experiments are on MIMIC-IV. Every method uses the same five training seeds (42 to 46), weighted equally, unless a larger seed count is stated. WISt truncates trajectory weights at 100. Paired 95% intervals describe variation across training seeds on the fixed test set.
Main results.
- Matched training conditions. With optimizer, samples and batch order matched, COSTAR-MARL is above GPM by 5.57 [3.85, 7.29], above SGP by 2.16 [0.27, 4.04] and above hard projection by 5.55 [3.40, 7.69] on final WISt.
- Second estimator. FQE places COSTAR-MARL above SGP by 14.29 [7.20, 21.37], in 5 of 5 seeds.
- Teammate support. Over 20 seeds, COSTAR-MARL is above its no-support variant by 2.40 [1.23, 3.57].
- Code. The two discrepancies you found are real and do not change the results.
Critical Weaknesses
- W1, "[the code] computes actor/critic correlations with zero-filled team contexts [...] replaces the stated directional match-fraction statistic with an averaged correlation score". Both confirmed in the code; neither changes the results. True team context changes final WISt by -0.20 [-0.65, 0.25]; the match-fraction statistic changes support scores by at most 1e-5. Part 2/8.
- W2, "Correlated responses do not establish that a teammate can replace historical knowledge [...] The ablations do not isolate genuine support from altered release budgets". We agree on the first point, and the paper says so. Support has a measured contribution: +2.40 [1.23, 3.57] over 20 seeds. The matched-budget controls you proposed give the same perturbation for true and permuted support. Part 3/8.
- W3, "The drift guarantee applies to virtual SGD, not the clipped Adam updates used in training [...] Key subspace baselines use different optimizers or eligible data". Matched baselines added; drift under Adam measured and explained. COSTAR-MARL is above GPM by 5.57 [3.85, 7.29] and above SGP by 2.16 [0.27, 4.04] on final WISt. Parts 7/8 and 8/8.
Questions For Authors
- Q1, "zero team contexts rather than the actual teammate messages and masks". Tested: no effect on the results. -0.20 [-0.65, 0.25]. Part 2/8.
- Q2, "the fraction of matched directions in a specific layer, or the symmetric mean-best-correlation graph score". The latter; the text is corrected to match the code. The swap changes support scores by at most 1e-5. Part 2/8.
- Q3, "what protects the previous policy on a singleton team [...] matched-budget perturbation/permutation experiment". The agent's own memory, caps and budget: with the filter, old AKI behaviour stays closer (KL 1.72 vs 9.88 without it). Both experiments you proposed were run. Part 3/8.
- Q4, "could the authors provide relevant documentation to verify this?". Yes. The authors who accessed and analysed the two databases hold the required credentialed access. We can provide the access records and training certificates through a confidential channel designated by the Area Chair, consistent with anonymous review. Answered here.
- Q5, "What joint treatment action is executed [...] marginal disease-specific policy evaluations or [...] the combined team's value?". Marginal; a joint-action evaluation is added. Joint WISt 72.42, the highest among entries with ESS above 30. Part 4/8.
- Q6, "How are all target-action probabilities in Eq. (29) computed [...] cohort-level ESS/overlap diagnostics and a second OPE estimator?". Described; both added. FQE on MIMIC-IV: +14.29 [7.20, 21.37] over SGP. Parts 5/8 and 6/8.
- Q7, "optimizer- and sample-matched GPM/SGP and selective-release baselines, paired uncertainty intervals [...] per layer or global". All provided; the budget is per layer. Above GPM by 5.57 and SGP by 2.16 under matched conditions. Parts 7/8 and 8/8.
Thank you for your response.
While some of my concerns have been addressed, I still believe that the paper falls substantially short of the minimum standards for acceptance at ICLR. Therefore, I will maintain my original rating.