acceptodds
Under review as a conference paper at ICLR 2027

From Safety Notices to Monitoring Policies: Evidence-Grounded LLM Agents for Statistical Process Control

Abstract

Safety notices require hospitals to map natural language onto local exposure cohorts before statistical monitoring can begin. We formulate this step as a probability distribution over executable cohort predicates, linked to monitoring actions by an equal- expected-loss solver. An LLM uses six calls to five read-only tools to acquire cell-suppressed aggregate evidence and update its cohort distribution. Deterministic code handles calibration and action selection, while patient rows, outcomes, and monitoring losses remain outside the model's interface. We evaluate 56 synthetic notices for 14 cohorts held out from semantic-method development, using cohort memberships derived from real MIMIC-IV exposures and two prespecified process-shift templates. The Agent reduces normalized regret by 38.6% relative to the same LLM without tools and by 64.9% relative to a fixed-query non-LLM pipeline with the same six-call budget. Mean detection delay falls to 53.92 patient arrivals from 58.24 and 67.14, respectively. Fifteen of sixteen preregistered conditions were met; the largest target's 27.86% share of net gain against fixed-query evidence exceeded the 25% concentration ceiling, leaving the conjunctive confirmation criterion unmet. These results quantify how model-directed aggregate evidence use improves cohort-conditioned monitoring decisions within a fixed executable catalog.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
large language model agents, statistical process control, clinical monitoring, evidence-grounded reasoning, aggregate queries

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PkmvOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to map natural-language safety notices onto executable exposure cohorts for statistical monitoring. The core method allows an LLM to query aggregate evidence through tools and output a probability distribution over candidate cohorts, while a statistical solver selects the optimal monitoring action at an equal-ARL0ARL_0 level. Evaluated using real MIMIC-IV records, synthetic notices, and simulated process shifts, the results show that this method reduces normalized regret and detection delay compared to a no-tool LLM and a fixed-query baseline, although it did not pass all preregistered success conditions.

Critical strengths

One strength of this paper is its clear practical motivation, as it attempts to bridge language understanding with downstream monitoring decisions. The authors integrate the LLM into a complete pipeline consisting of notice parsing, cohort identification, and monitoring action selection.

Additionally, the experimental design is relatively rigorous and standardized. All competing methods are connected to the same statistical solver, calibrated to an equal-ARL0ARL_0 false-alarm level, and use strictly isolated data streams for action selection and verification. This controlled setup enhances fairness in comparison. Furthermore, the authors transparently report that not all preregistered conditions were met, which supports the transparency of the evaluation.

Critical weaknesses

The core method of this paper relies primarily on prompt engineering and tool calls using a proprietary model (gpt-5.5), with limited algorithmic novelty in machine learning. The method follows a predict-then-optimize formulation but does not introduce a new decision-focused learning algorithm. As the authors acknowledge, the LLM is used without task-specific training, and downstream losses are used for action selection and evaluation. The contribution therefore lies primarily in system integration, while its methodological advances over existing LLM-based tool-use approaches remain insufficiently established.

In addition, the evaluation is limited by a very small candidate catalog of only 20 predicates and 14 held-out test targets. The task is restricted to identifying cohorts within this fixed catalog, with an initial BM25 shortlist to guide the model. This limited evaluation scope leaves its generalizability to larger and more diverse cohort catalogs unclear.

Questions for authors
  1. Compared with existing LLM and tool-use methods, what is the main methodological innovation of this paper, and which specific designs actually improve monitoring performance?

  2. What happens if we feed the fixed-query information directly into the same LLM? It would be helpful to compare this under the same query budget to test the extra value of letting the LLM choose its own queries.

  3. Why does the Log Score get worse from 0.834 to 1.357 after using tools? Could the authors analyze the cases where the model was very confident but wrong, and how they affected the final monitoring choices?

  4. What is the reason for splitting the unknown probability mass equally among all candidates? If we change this way of allocation, how much will the final monitoring results and regret change?

Reviewer_uJodOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how LLM agents can translate natural-language safety notices into executable patient cohorts and use aggregate evidence to guide statistical process monitoring. The proposed agent makes six adaptive calls to a restricted set of read-only tools and outputs a probability distribution over candidate cohorts. A deterministic solver then selects a monitoring action, with all actions calibrated to the same average run length (ARL0).

The authors construct a MIMIC-IV-grounded benchmark with real patient exposure records, synthetic safety notices, and simulated process shifts. On 56 held-out notices covering 14 cohorts, the proposed agent reduces normalized decision regret by 38.6% compared with the same LLM without tools and by 64.9% compared with a fixed-query non-LLM baseline.

Critical strengths
  1. The paper evaluates cohort identification through downstream monitoring regret rather than classification accuracy alone, providing a meaningful connection between LLM predictions and operational decisions.
  2. The use of real EHR-derived cohorts, equal-ARL calibration, multiple baselines, and a preregistered evaluation protocol strengthens the credibility of the reported results.
Critical weaknesses
  1. The evaluation uses synthetic notices and a fixed catalog of only 20 cohorts, with the true target always included. It is unclear whether the findings extend to real-world safety notices or more complex cohort identification problems.
  2. The paper does not compare against a non-adaptive tool-using LLM with the same query budget. Consequently, the contribution of adaptive evidence acquisition cannot be clearly separated from the general benefit of providing an LLM with additional evidence.
Questions for authors
  1. How representative are the synthetic notices of real-world safety notices?
  2. Could the authors include a baseline using the same LLM with a fixed, non-adaptive six-query strategy? This would help clarify the contribution of adaptive query selection.
  3. How would the proposed approach perform with a larger candidate catalog or when the true cohort is not included among the candidates?
Reviewer_QBxDOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper develops a framework for identifying cohorts relevant for safety notices in hospitals by formulating safety notice interpretations as probability distributions over patient cohorts and then selecting monitoring actions using an equal-ARL0 expected loss solver. The paper also introduces a benchmark with EHR events from the first 24 hours using MIMIC IV and evaluate their method using the benchmark compared to baselines.

Critical strengths
  • The motivation to find better ways to determine who is affected by safety notices in a less burdensome way is commendable and an important problem to address
Critical weaknesses

W1 - I have some concerns about the definition of safety notices. What exactly is encompassed in "safety notices"? This seems to be a pretty broad category and it is unclear how this is categorized. The paper would benefit from a clean definition and description of what and how such safety notices are included and tested on. Also, why is the goal to identify cohort-level safety notices and not individualized ones? Do we not ultimately want individualized safety alerts to know which patients are affected? (See Q1-2).

W2 - I have some soundness concerns in the methodology and experimental setup. 2a. In the problem formulation (section 3.1) why are logistic models used to estimate outcome specific risks instead of not using the real ground truth? This seems to introduce possibility of bias. (See Q3).

2b. I have concerns about the generation of the safety notices. Section 3.3. does not provide details about how these notices are generated and where they come from. If I am correct in my understanding, they are derived from EHR events in the MIMIC data, but these events are not necessarily safety notices so this seems to be a mismatch from the goal of the work. (See Q4).

W3 - The paper presentation is hard to follow at times, and leaves out important details. There are lots of terms that are undefined and it is unclear what is meant by them. For example: section 3.1 what is meant by "matured patient encounters?"; section 3.2 what is BM25 configuration, this hasn't been introduced yet; section 3.3 what is "DGP". Section 4, the experimental section does not clearly present what is being tested or the complete experimental setup, and is missing important experimental details including what data/features were used (e.g., what parts of the mimic data), how many patients, the total number and type of safety events being evaluated against etc.

Questions for authors
  1. What exactly is included in safety notices? Does the approach cover all types or are there only specific types?
  2. Should the safety notices be personalized as opposed to cohort level? Put another way, why is the goal cohort selection as opposed to individual personalized selection? Do we not ultimately want individualized safety alerts to know which patients are affected?
  3. Why are logistic models used to estimate outcome specific risks in the problem set up as opposed to using the ground truth outcomes?
  4. How are the safety notices generated? Why are real safety notices not used as ground truth?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.