acceptodds
Under review as a conference paper at ICLR 2027

Feedback-Robust AI for Patient Knowledge Graphs

Abstract

Patient knowledge graphs from bedside monitoring should type their relations and state whether the data support their signs. In anesthesia and intensive care, clinicians titrate drugs and ventilation in response to the physiology, so temporal relations mix the patient's response with the clinician's policy. We introduce ClosedLoopBench: 29 relations with signs fixed by physics, pharmacology or clinical practice, on 3,442 VitalDB surgical cases (12,653 h) with negative-control action streams. When each patient's actions are replaced by another patient's, six of 12 estimators declare on average 11-18 of their 19-29 distinct relation estimates significant without calibration, and after calibration cross-correlation and Granger tests still assign ventilator rateend-tidal CO the sign of the clinician's policy. We propose feedback-robust patient graphs that combine concept nodes with evidence pointers, typed relations admitted against negative controls, and beat-level couplings. On VitalDB under null streams, our graphs contain 0.06-0.10 false concept-level relation instances per graph, versus 10-12 for correlational construction. Patient-specific estimates of 11 slow drug and ventilator responses predict later data no better than population estimates, whereas the pulse-arrival-time-systolic-pressure slope is negative in 94.3% of 2,884 cases and patient-specific (early-late correlation 0.67 [0.63, 0.70]).

Primary area:
learning on time series and dynamical systems
Keywords:
patient knowledge graphs, temporal causal discovery, clinical feedback, negative controls, benchmark, physiological time series, concept-based models, anesthesia, healthcare

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JH1cOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the construction of patient knowledge graphs from bedside physiological monitoring under clinical feedback, where clinicians adapt drugs and ventilator settings in response to the same physiology that those interventions subsequently affect. The authors argue that standard temporal-relation estimators can therefore conflate clinician policy with patient response and even recover relations with the wrong sign.

Critical strengths
  1. The paper demonstrates that replacing a patient's intervention stream by an unrelated patient's stream still causes several commonly used estimators to report large numbers of significant relations. On real data, some temporal estimators additionally recover the direction of clinician titration rather than the expected direction of the physiological intervention response. This is consequential because physiological monitoring is inherently closed-loop: treatment is chosen because of the state being modeled. The paper therefore provides a useful warning against interpreting temporal edges in patient graphs as direct physiological effects.

  2. The paper pre-specifies cohorts, estimands, estimators, nulls, decision rules, and several hypotheses; hashes the corresponding protocols; records deviations; and reports hypotheses that failed. It also distinguishes results that are confirmatory from post-hoc analyses. This substantially increases my confidence in the central empirical observation, particularly given the large number of possible analysis choices in high-frequency physiological time series.

Critical weaknesses
  1. The paper itself states that its real-data evaluation measures concordance with pre-specified sign expectations, not agreement with causal ground truth. Moreover, those expectations differ in epistemic strength: respiratory physics is relatively clear, pharmacological and physiological effects are context- and horizon-dependent, and clinician-policy labels are inferred from standard practice rather than the actual decision policy used by the clinicians in these records. Therefore, an estimator disagreeing with some drug or policy signs is not necessarily incorrect in the same sense that a method is incorrect on a synthetic benchmark with an observed structural graph.

  2. The manuscript reports approximately 0.06–0.10 false relation instances per graph for the proposed construction versus roughly 10–12 for CKG. This is visually striking, but the comparison combines substantially different statistical procedures. The proposed method admits relations at the population level across 176 candidate relations using 38 null replicates and then instantiates them in individual graphs. CKG instead tests temporal relations independently within each patient. The appendix explicitly acknowledges that the resulting per-graph counts “compare constructions, not a matched test.” Moreover, stronger per-patient controls substantially narrow the gap: pairwise Granger with BH yields 2.78/1.86 false edges and PCMCI+ 0.79/0.85 on VitalDB, compared with 12.40/10.07 for raw CKG.

Questions for authors

Could the authors report all major ClosedLoopBench estimator rankings separately for:

  1. respiratory-physics relations;
  2. pharmacological/physiological relations;
  3. policy relations? The manuscript explicitly acknowledges that these sign labels have different evidential strength. I would especially like to know whether the central conclusion—that design-based estimators are substantially more reliable than lag-based approaches—remains unchanged when evaluation is restricted to the highest-confidence sign subset.
1 comment
AuthorsOpenReview ↗

Replying Reviewer JH1c

Thank you for the careful review. Both of your points led to new analyses, which we summarize here and will add to the revised paper.

On W1 and your question (does the conclusion hold on the highest-confidence relations?)

You are right that a sign fixed by respiratory physics is much stronger evidence than one taken from documented clinical practice. We split the 29 relations into three tiers using the sign source the paper already lists for each relation: physics (5 relations), pharmacology and physiology (14), and documented practice (10). Nothing was re-run; the table re-counts the existing calibrated calls within each tier.

EstimatorPhysics (5): right/wrong sign, RTPharmacology (14): right/wrong sign, RTRank, physicsRank, pharmacology
Positive-peak cross-corr.1/3, 01/6, 81112
Signed-peak cross-corr.2/2, 17/0, 7910
Direction-aware cross-corr.2/2, 18/0, 396
Pairwise Granger2/3, 310/0, 81211
PCMCI+0/0, 01/0, 044
VAR-LiNGAM5/0, 310/0, 487
VARX4/1, 06/1, 055
Naive event contrast4/0, 24/4, 269
Naive level contrast3/0, 07/2, 238
Matched event design4/0, 05/0, 012
Interrupted trend4/0, 06/0, 011
Sequential-trial contrast4/0, 25/0, 062

(RT: false admissions under the random-time null. Ranks use the reliability-first rule we declared in advance.)

The conclusion holds on the physics tier, which is where we would most like it to be judged. The matched and interrupted-trend designs admit 4 of the 5 physics relations, all with the physical sign and with no random-time false admissions, and they rank first under both of our ranking rules. Cross-correlation and Granger, by contrast, admit set rate → EtCO₂ with the sign of the clinician's policy and end up at ranks 9–12. Two estimators need a caveat. The sequential-trial contrast gets the signs right but has two random-time false admissions. VAR-LiNGAM gets all five signs right, yet calls every one of its physics estimates significant on every transplant null, so its error control is poor even though its signs are correct.

Pharmacology looks similar: the three design-based estimators take the top ranks. In the practice tier no estimator stands out, and two policies (P08, P10) come out with the opposite sign for almost every estimator that admits them, which makes us think the practice label is the problem there rather than the estimators. We agree with your reading, and the revision will describe the policy tier as agreement with documented practice, outside the reliability claim.

On W2 (the 0.06 vs. 12 comparison)

Thank you for pressing on this. Table 2 compared our full pipeline with a per-patient correlational graph built under different procedures, and the baseline used a single null replicate. We have replaced it with a ladder that starts from the per-patient baseline and adds our ingredients one at a time. Every row uses the same candidate relations, the same 38 null replicates and the same metric.

ConstructionFalse instances per graph, transplant nullFalse instances per graph, shift null
Per-patient correlational graph (as in Table 2)12.4810.07
Per-patient Granger + BH2.781.86
Per-patient PCMCI+ + BH (400 cases)0.750.83
Per-patient correlational graph, density-matched to ours9.587.99
Same statistics pooled across patients + BH24.451.83
+ our negative-control rule (the pre-specified matched comparison)2.62 [2.25, 3.03]0.23 [0.11, 0.35]
+ the hazard model's case-phase covariates (added after the fact)0.070 [0.001, 0.164]0.31 [0.16, 0.49]
Our construction0.057 [0.014, 0.107]0.103 [0.035, 0.184]

Read from top to bottom, the gap closes in identifiable steps. Per-patient rules have no way to abstain when the whole graph is null, so they always return edges. Pooling alone makes the transplant count worse, because with three million rows the pooled tests pick up the case-phase structure that transplanted series share. The negative-control rule removes most of what remains, and adjusting for case phase removes the transplant excess altogether. Between that last ladder row and our construction there is a further 2–3× on the shift null, and those intervals overlap. This makes the claim sharper than Table 2 could: the improvement comes from how relations are constructed and admitted, and those ingredients would help any pooled statistic; it does not come from a new estimator. The revised paper will say this and will no longer set 12.4 against 0.06.

Reviewer_j2wPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the problem of discovering reliable physiological relationships from clinical monitoring data under feedback from clinician interventions. The authors introduce ClosedLoopBench, a benchmark comprising 29 physiologically motivated relations evaluated using 12 estimators on 3,442 surgical cases from VitalDB. They demonstrate that conventional correlation-based and temporal relation discovery methods can identify spurious or incorrectly signed relationships due to clinical feedback and temporal dependencies.

The authors further propose a framework for constructing patient knowledge graphs (KGs), combining physiological concept extraction, pooled relation estimation, negative-control calibration, typed relations, and patient-specific beat-level physiological couplings. The framework is evaluated in terms of false relation discoveries, temporal consistency, concept grounding, and the reproducibility of patient-specific physiological relationships.

Critical strengths
  1. The paper addresses a fundamental challenge in analyzing observational clinical time series: physiological measurements and clinical interventions are coupled through feedback mechanisms. The distinction between clinician policies, physiological responses, and observational associations is well motivated and important for developing reliable clinical AI systems.

  2. The use of cross-patient transplantation, random-time interventions, and temporal shifts provides an interesting approach to studying spurious relationship discovery in real-world physiological data. These experiments highlight important limitations of conventional statistical significance testing under temporal dependence.

  3. The paper includes extensive supplementary experiments, uncertainty estimates, multiple-testing corrections, and an unusually transparent discussion of limitations and negative findings. The investigation of patient-specific beat-level couplings versus slower intervention responses is particularly interesting.

Critical weaknesses
  1. My primary concern is that the proposed framework combines several established statistical and machine-learning techniques without introducing a clearly identifiable new methodological contribution. The main components include transformer-based concept extraction, pooled logistic hazard models, VARX, negative-control calibration, multiple-testing corrections, and rule-based graph construction. Although their integration is meaningful from an application perspective, the manuscript does not convincingly establish what constitutes the fundamental methodological innovation. In summary, it remains unclear what new learning or inference principle is introduced beyond applying existing statistical safeguards to clinical KG construction.

  2. Several central components rely on linear or additive models, including VARX-based estimation of intervention responses and linear regression for beat-level physiological couplings. These assumptions appear restrictive given that physiological responses to clinical interventions are often nonlinear, state-dependent, and characterized by complex temporal dynamics. Drug concentration–effect relationships, in particular, may exhibit nonlinear dose responses, saturation effects, delayed responses, and substantial interactions with the patient's physiological state. While linear models may offer interpretability and computational advantages, the paper does not sufficiently justify why they are appropriate for the investigated relationships. More importantly, it remains unclear whether the reported robustness is a property of the proposed framework or partly a consequence of its restrictive modeling assumptions.

  3. The proposed framework contains numerous fixed thresholds and heuristic decisions, including physiological concept definitions, persistence tiers, minimum sample-size requirements, sign-consistency criteria, significance thresholds, lag windows, and graph-edge admission rules. Some thresholds are clinically motivated and reasonable. However, several algorithmic choices appear insufficiently justified. For example, why is a sign-consistency criterion of 75% considered appropriate? Why are five contributing cases sufficient for admitting a relationship? How sensitive are the resulting graphs to these and other design choices?

  4. The reported reduction in false relation instances is striking. For example, Table 2 reports approximately 0.06 false concept-level relations per graph for the proposed method, compared with approximately 12.4 for the correlational KG baseline under the transplant null. However, the comparison involves fundamentally different estimation procedures. The proposed approach pools information across patients, restricts candidate relations, applies negative-control calibration, and uses conservative population-level admission criteria before instantiating relations in individual graphs. In contrast, the correlational baseline estimates temporal relationships primarily at the individual-patient level. Consequently, the reported improvement may reflect differences in pooling, candidate selection, statistical power, multiple-testing procedures, and graph sparsity rather than an intrinsic advantage of the proposed graph-construction methodology. A more convincing evaluation would compare the methods under matched conditions, including identical candidate relation sets, comparable pooling strategies, equivalent calibration procedures, and controlled graph density. Without such comparisons, I find the magnitude of the reported improvement difficult to interpret.

  5. The experiments demonstrate that certain estimators identify relationships under artificially constructed null conditions. However, successfully avoiding false discoveries under negative controls is not equivalent to correctly identifying genuine physiological relationships. In particular, replacing intervention trajectories with those of other patients or redistributing intervention times changes aspects of the underlying temporal and treatment distributions. These procedures do not necessarily reproduce all relevant characteristics of the null hypothesis in a closed-loop physiological system. The random-time experiments also reveal important limitations: several estimators continue to admit false relations despite negative-control calibration. Furthermore, the concept-level experiments report false population-level admissions in a non-negligible proportion of null replicates, even though the average number of false relation instances per patient graph is small. A central missing component is a systematic evaluation using simulated closed-loop physiological systems with known ground-truth relationships. Such experiments should investigate varying feedback strengths, degrees of unobserved confounding, nonlinear response functions, sampling frequencies, and intervention policies. Crucially, the evaluation should consider both false-positive control and the ability to recover true relationships. Otherwise, highly conservative procedures may appear robust simply because they reject most candidate edges.

  6. Although the paper emphasizes patient knowledge graphs, most slow relations are estimated at the population level and subsequently instantiated within individual patients based on the occurrence of corresponding events. However, observing two temporally related events in an individual patient does not establish that the underlying population-level relationship is valid for that specific patient. The evidence pointers improve traceability but do not, by themselves, validate the physiological or causal interpretation of individual graph edges. This concern is reinforced by the authors' finding that none of the 11 slow intervention-response relationships demonstrates reliable patient-specific predictive improvement in the split-half experiments. Therefore, the resulting graphs appear to represent patient-specific observations annotated with population-level relationships rather than independently learned patient-specific physiological dependencies.

  7. More importantly, the practical benefits of this representation remain unclear. The paper provides limited evidence that the proposed knowledge graphs improve downstream predictive performance or generalization compared with models directly using the underlying physiological time series and intervention histories. The authors also acknowledge that typed relations improve held-out concept-onset prediction over a history-only baseline in only one of three codings. A convincing demonstration of downstream utility, such as prediction of physiological deterioration, intervention-response forecasting, or improved robustness to distribution shifts, would substantially strengthen the contribution.

  8. The distinction between clinician policy and physiological response is central to the paper. However, the proposed statistical models do not resolve the causal identification challenges arising from time-varying confounding and partially observed clinical decision processes. Relevant drivers of intervention decisions, including surgical stimulation, unrecorded medication administration, and other clinical events, may not be captured in the available data. Negative-control calibration can help detect spurious associations but does not generally eliminate bias from unobserved confounding or identify causal intervention effects. This is particularly relevant because the concept-level response relations fail the authors' prespecified pharmacological sign checks.

  9. The principal feedback-related experiments are conducted using VitalDB, a single-center surgical dataset. Although MIMIC-IV waveform data provide additional evaluation, the corresponding experiments do not externally validate the central intervention-response findings because the necessary action records are unavailable. Consequently, it remains uncertain whether the reported results generalize to other hospitals, clinical intervention policies, or patient populations.

Questions for authors
  1. What precisely constitutes the novel methodological contribution of the proposed framework beyond combining established statistical estimators, negative-control calibration, and rule-based graph construction?

  2. Why are linear or additive models considered sufficiently expressive for the physiological relationships studied? Do the main findings remain valid with nonlinear estimators?

  3. How sensitive are the reported false-discovery rates and graph structures to the numerous fixed thresholds and heuristic admission criteria?

  4. Can the authors provide a comparison against correlational KG approaches using identical candidate relations, population-level pooling, calibration procedures, and comparable graph densities?

  5. Can the framework be evaluated on simulated closed-loop systems with known ground-truth relationships, reporting both true-positive and false-positive recovery under varying feedback and confounding conditions?

  6. What evidence demonstrates that the proposed patient knowledge graphs provide useful information beyond the original physiological observations and treatment histories?

  7. Will an anonymized implementation of ClosedLoopBench, including standardized evaluation scripts and preprocessing procedures, be made available during the review process?

3 comments
AuthorsOpenReview ↗

Reply to Reviewer j2wP

We thank Reviewer j2wP for the detailed review. Each point is answered below with new analyses.

W1/Q1 (what is new). The individual tools are established; what is new is a construction for relation discovery when an agent acts on the data, and the evidence that it is needed. Three parts: (1) a real-data benchmark in which the clinician acts on the recorded variables and the signs of both the policy and the response are fixed in advance (synthetic ones have no acting agent); (2) negative-control calibration as the admission rule for any temporal estimator, which separates schedule artifacts from serial-dependence artifacts (Table 1); (3) typed relations with a sign status, so that clinician behaviour is never read as a drug effect. The ladder under W4 shows what removing them costs.

W4/Q4 (Table 2). We ran exactly this comparison: a ladder that gives correlational construction our ingredients one at a time, with the same candidates, the same 38 null replicates and the same metric (full table in our reply to JH1c). False relation instances per graph, transplant / shift null: per-patient correlational graph 12.48 / 10.07; density-matched to ours 9.58 / 7.99; its statistics pooled, with BH and our negative-control rule, 2.62 / 0.23; ours 0.057 / 0.103. With identical candidates, pooling, multiplicity control and calibration, correlational statistics still leave 46× and 2× more false instances. Only when we also add our case-phase adjustment (post hoc) do they approach our level (0.070 / 0.31). The false edges are removed by the construction, which is our contribution and transfers to other pooled statistics. Table 2 will be replaced by this ladder.

W2/Q2 (linear models). The best estimators in Table 1 are design-based before/after contrasts with no dose-response form, so their signs are invariant to any monotone transformation. We also re-ran the lag-based ones in nonlinear form. Rank cross-correlation still gives set rate → EtCO₂ the policy sign (+0.136). A boosted-tree Granger test gets the physical sign, but so does a ridge version of the same design, so the reversal comes from pairwise estimation under feedback rather than from linearity; both also fail calibration (3.25 and 8.1 false admissions per held-out draw, against 0 for every Table 1 estimator). Dose-binned VARX changes no sign, and rank and isotonic PAT–SBP slopes are negative in 94.9% and 93.8% of cases (linear: 94.3%).

W3/Q3 (thresholds). All were fixed in hashed protocols before any run (App. I). Across 400 benchmark settings (q 0.01–0.10, BY or BH, stability 0.5–1.0, 3–50 cases) the set of estimators with wrong-sign admissions never changes, and across 84 graph settings no relation changes sign; 76 of 81 relations survive all of them. Only q matters (q = 0.01: 0.013 / 0.054 false instances per graph, with 5 fewer relations). The 7/19 and 9/19 replicates are what the max-over-nulls rule implies at that null rate, not an excess.

W5/Q5 (power with known truth). We ran all 12 estimators and the admission rule on the paper's simulator with known effects, including an inert drug (600 patients per setting, two replicates; feedback, unrecorded drivers, saturation, sampling and policy varied). Nine of the 12 estimators find 92–100% of the true relations, so low false-positive rates are not bought by finding nothing. Without feedback no estimator makes a false positive. Under feedback the design-based estimators admit 8–25% of the inert relations, against 67–100% for Granger, VAR-LiNGAM, VARX and the naive contrasts. When the clinician reacts to an unrecorded driver or acts in anticipation, no estimator, design-based included, can separate the effect from the clinician’s response (50–83% inert admissions; wrong signs for a third of relations): the boundary our theory predicts, which calibration cannot remove.

W6–W9. W6: this is by design and is one of our findings. Concepts and beat-level couplings are patient-specific; slow relations are estimated on the population and instantiated with patient evidence, because a patient's own estimate predicts its later data worse (0 of 11 pass; per-patient weights cost 0.257 nats per onset). Sec. 4 will make this explicit. W7: the paper claims reliability of what a graph contains; where it does test prediction, typed relations beat correlational ones in all three codings (+0.038 to +0.045 nats per onset). W8: the paper makes no causal-identification claim; the simulation now measures what negative controls remove and what they cannot. W9: the central result is single-center because VitalDB is, to our knowledge, the only public dataset with time-stamped drug and ventilator actions at this scale (MIMIC-IV yields about 50 usable records). The code reads the PhysioNet format, so other centers can run the benchmark directly.

Q7. The code is at https://anonymous.4open.science/r/ClosedLoopBench/ (with 334 tests; the analyses above are in scripts/extended/).

Reviewer_j2wPOpenReview ↗

Reply to authors

Response to Q1: I really acknowledge the author's contribution from an application perspective. However, I see too limited methodological novelty for an ICLR contribution. Response to Q4: Thank you, this is much appreciated. Response to Q2: Thank you for running these tests. Notably, rank correlation relies on the assumption of monotonicity. So, replacing Pearson by rank correlation still imposes strong limitations. Response to Q3: I must admit that I have difficulties to follow your answer, because it is rather shortened. My concern was about the fixation of numerous thresholds and heuristic decisions, which may impact the observed results. Response to Q5: Thank you for clarification. Response to Q6-Q9: Thank you for clarification.

AuthorsOpenReview ↗

Reply to Reviewer j2wP

Thank you for taking the time to actively engage in the discussion. Discussions genuinely improve the work, and we are grateful for your careful reading. Brief response to each remaining point below.

Novelty (Q1). Thank you for recognising the value of the contribution; we fully respect your view on methodological novelty. If it is helpful, we would gently note that, beyond the benchmark, the paper also contributes two results that we see as methodological, each with a proof: the sign-reversal proposition (a lagged or event contrast converges to the effect plus a policy term that opposes it below a crossover horizon) and the future-inaction bias of control pools restricted by later treatment. Both are checked on the simulator with known effects (App. D), and the new simulation grid extends that check to all 12 estimators. Since ICLR's call includes applications in healthcare, we hope the paper may be seen as a contribution in that area: it shows, on real data and with known ground truth, which estimators can be trusted under clinical feedback and why.

Monotonicity (Q2). Thank you, it is a good point: a rank correlation does assume monotonicity, which is why we included it as one of four checks. Two of the others make no such assumption: the boosted-tree Granger test (out-of-fold predictive gain of the action history over the vitals' own history, with any functional form) and PCMCI+ with the nonparametric GPDC test. Dose-binned VARX allows a separate response for small, medium and large steps. The design-based estimators make no dose-response assumption either: each compares the change after an observed titration step with matched control times, so it estimates the average effect of the steps clinicians actually made, whatever the shape of the dose-response. None of these checks changes the paper's conclusions.

Thresholds (Q3). Thank you for letting us know that our earlier answer was too concise, and we hope this fuller version is easier to follow. Every threshold below was registered in a hashed protocol before the main runs (App. I), and we re-applied the admission rules over the stated ranges without refitting anything.

ChoiceValue used, and whyRange testedEffect
Sign consistency over null draws75%, i.e. 6 of 8 draws50% to 100%No sign changes; requiring 8 of 8 removes 2 design-based and 9 lag-based admissions
Minimum contributing cases5, the smallest number for which a case bootstrap is not degenerate3 to 50No change from 3 to 20; at 50 only PCMCI+ (60-case subsample) changes
FDR level and correctionq = 0.05, Benjamini–Yekutieliq = 0.01 to 0.10; BY or BHReal-data admissions almost unchanged; only the null false admissions of lag-based estimators move
Graphs: testability gateat least 20 exposed events from 10 patients10/5 to 40/20False instances per graph 0.056–0.057 (transplant) and 0.099–0.103 (shift)
Graphs: null rulemust exceed all 38 null replicatesmaximum, 95th percentile, noneWithout the rule: 0.070 / 0.175
Graphs: FDR levelq = 0.050.01 to 0.10Transplant 0.013 to 0.057; shift 0.054 to 0.285; q = 0.01 removes 5 of the 81 relations

Across all 400 benchmark settings and all 84 graph settings, no relation is admitted with the opposite sign, the design-based estimators admit set rate → EtCO₂ with the physical sign while cross-correlation and Granger admit it with the policy sign, and the matched design has no false admission under either null. 76 of the 81 graph relations are admitted in every setting.

The remaining choices are fixed by physiology rather than tuned. Concept definitions are standard clinical thresholds (for example, hypotension as MAP below 65 mmHg). Persistence tiers label graph nodes and do not enter relation admission. Lag windows (up to 10 min for concept relations, 60 s for ventilation and 120 s for drugs in the benchmark) follow pharmacological and respiratory latency and were set in the protocol. The full threshold grid will be added as an appendix table, and the code is in scripts/extended/E_sensitivity.py. We would be glad to provide any further detail that would be useful.

Reviewer_XQE8OpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

As I understand, the core idea concerns when clinicians adjust treatment in response to a patient's condition, then correlations between treatments and measurements can be misleading. The key view is that even established methods such as Granger causality can mistake the clinician's corrective behavior for the treatment's physiological effect. The authors introduce ClosedLoopBench, a benchmark using surgical cases with defined relationships to evaluate methods under feedback.

Critical strengths

I guess this was somewhat surprising to me, not that they dont work, but how poor current approaches are. But this is a hard problem. The authors demonstrate this nicely. I like the use of negative control action as this helps mitigate spurious associations that many other evaluations seem to overlook. The authors investigate several sources of bias and distinguish between clinician policies and physiological responses - to me this is key to being a robust analysis.

The graphs the authors introduce, to my understanding, distinguish relation types and link back to supporting observations. The supplementary information is really extensive and thorough.

Critical weaknesses

I guess there is no "new" method, and while this might be a weakness and i do add it here, i do not think it should discount from the novelty of the paper. I think the proposed framework is a really assimilation of existing approaches into something useful, and this is arguably better than just some new method with no practical application.

I guess given my point above, the one weakness is the need for more evaluations and matched calibration and statistical testing procedures. If i understand the evaluation if focused on a single center dataset, and this is a limitation.

Questions for authors

No additional questions

2 comments
AuthorsOpenReview ↗

Reply to Reviewer XQE8

We thank Reviewer XQE8 for the encouraging review, and for reading the appendix so closely. We are glad the negative-control idea and the policy/response distinction came across; they are the parts of the work we care most about. Thank you also for judging the paper as a benchmark and a construction rather than as a new learner. That is how we see it too, and the revised introduction will say so directly.

Matched calibration and testing. This was the right thing to ask for, and the other reviewers asked for it as well. We ran a construction ladder that gives correlational graph construction our ingredients one at a time, on the same candidate relations, the same 38 null replicates and the same metric (full table in our reply to JH1c). False relation instances per graph, transplant / shift null: per-patient correlational construction 12.48 / 10.07; the same statistics pooled across patients with BH and our negative-control rule 2.62 / 0.23; ours 0.057 / 0.103. Adding our case-phase adjustment as well (post hoc) brings the correlational statistics to 0.070 / 0.31. The improvement therefore comes from the construction rather than from the test statistic, and this ladder will replace Table 2.

More evaluations. We added four:

  • A simulation with known effects, including an inert drug. Without feedback no estimator makes a false positive. Under feedback the design-based estimators admit 8–25% of the inert relations, against 67–100% for Granger, VAR-LiNGAM, VARX and the naive contrasts, while still finding 92–100% of the true ones.
  • Nonlinear versions of the lag-based estimators (rank cross-correlation, boosted-tree Granger, dose-binned VARX). None of them changes the paper's conclusions: rank correlation keeps the policy sign, and the flexible models that recover the physical sign lose error control.
  • A sweep over the admission thresholds (400 benchmark and 84 graph settings), in which no sign changes.
  • The benchmark split by how strong the evidence for each sign is. The design-based estimators lead on the physics tier, where the signs are least disputable.

Single-center data. This is the main limitation of the paper. VitalDB is, as far as we know, the only public dataset with time-stamped drug and ventilator actions at this scale; linking MIMIC-IV yields about 50 usable records, too few for the benchmark. The code reads the PhysioNet format, so another center can run the benchmark on its own data: https://anonymous.4open.science/r/ClosedLoopBench/.

Thank you again. If anything else would make you more confident in the work, we would be glad to address it.

Thank you for the response, it would be great if you add these evaluations to the paper (especially the sweep), i am happy to maintain my positive score and really like this work

Comments on the paper

1 comment
AuthorsOpenReview ↗

General comment

Summary of the discussion-period additions (revised PDF uploaded)

We thank the three reviewers for constructive and detailed reviews. Their requests converged on a few points, and for each we ran a new analysis. The revised PDF with all of these results is now uploaded, and every result below is reproducible from the anonymized code: https://anonymous.4open.science/r/ClosedLoopBench/.

  1. A matched comparison with correlational construction (all three reviewers). We built a construction ladder that gives correlational graph construction our ingredients one at a time. Every row uses the same candidate relations, the same 38 null replicates and the same metric.
ConstructionFalse relation instances per graph (transplant / shift null)
Per-patient correlational graph, as in the submitted Table 212.48 / 10.07
Per-patient Granger + BH2.78 / 1.86
Per-patient correlational graph, density-matched to ours9.58 / 7.99
Correlational statistics pooled across patients + BH + our negative-control calibration2.62 / 0.23
... + our case-phase adjustment (post hoc)0.070 / 0.31
Our construction0.057 / 0.103

The ladder shows which ingredient removes which artifact: per-patient rules have no way to abstain on an all-null graph, calibration removes the within-record artifact, and case-phase adjustment removes the one that transplanted series share. It now replaces Table 2 (Sec. 6.2), with the full comparison in App. K.

  1. Results by strength of evidence (JH1c). Splitting the benchmark by the source of each expected sign, the matched and interrupted-trend designs rank first on the respiratory-physics relations, where the signs are least disputable: 4 of the 5 relations admitted with the physical sign, none with the opposite sign and no false admissions under the random-time null, while cross-correlation and Granger assign set rate → EtCO₂ the clinician's policy sign. The policy relations are reported as agreement with documented practice (App. C.9).

  2. Ground truth (j2wP). On the paper's clinician-in-the-loop simulator with known effects, including an inert drug, nine of the 12 estimators recover 92–100% of the true relations. Without feedback no estimator admits a false relation; under feedback the design-based estimators admit 8–25% of the inert relations, against 67–100% for Granger, VAR-LiNGAM, VARX and the naive contrasts. The simulation also maps the boundary that the paper's theory predicts, where the clinician responds to drivers absent from the record (App. M).

  3. Nonlinear models and thresholds (j2wP). Rank cross-correlation, boosted-tree Granger, dose-binned VARX and rank and isotonic PAT–SBP slopes leave the paper's conclusions unchanged (App. N). Across 400 benchmark and 84 graph-level threshold settings, no sign changes and 76 of 81 graph relations are admitted in every setting (App. L).

What changed in the revised PDF. Table 2 is replaced by the ladder, and the abstract now reports the matched comparison (0.06-0.10 versus 0.23-2.6 false relation instances per graph). Evidence tiers are added to the benchmark (App. C.9), and new appendices K to N contain the full matched comparison, the threshold sweep, the simulation with known effects and the nonlinear estimators. The contributions are stated as a benchmark and a construction for relation discovery under feedback, a table referenced from Sec. 4 (App. H) shows which graph elements are estimated per patient and which on the population, and the limitations state the single-center data scope. Section, table and appendix numbers of the submission are unchanged, so the references in our replies still hold.

Taken together, the new analyses support the paper's central finding under matched procedures, with known ground truth, under nonlinear models and across thresholds: under clinical feedback, lag-based estimators assign physical relations the sign of the clinician's policy, while design-based estimators and our negative-control-calibrated construction recover the physical signs and keep false relations near zero.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.