- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper develops a hospital operations simulator to study how healthcare providers respond to changes in payment, performance incentives, and resource allocation. The paper integrates coding, patient acceptance, treatment deferral, treatment effort, and bed requests into a shared environment to examine how distortions shift across channels. It also represents administrator and provider policies as executable code that LLMs can modify, while retaining historical policies to test resilience to continued adaptation. Its main contribution is a unified experimental environment and evaluation approach. The simulator follows an “Identify–Produce–Settle” cycle. Administrators set financial and quality incentives, capacity, and bonuses. Providers choose actions based on returns, patient complexity, and workload. Between simulation episodes, an LLM modifies policy code. The framework first jointly searches for a high-performing policy pair, then allows each party to pursue its own interests, and finally evaluates interactions with historical policies to identify remaining profitable deviations and changes in health and access.
Critical strengths
- This paper introduces a hospital operations simulator, which is a quite innovative and interesting study. The paper evaluates policies under continued provider adaptation and reveals cross-channel effects, such as improved billing accuracy accompanied by reduced patient access.
- Provides interpretable policies and a comprehensive analysis. Executable policies make behavioral changes inspectable, while experiments jointly examine returns, patient health, and strategic stability rather than relying solely on aggregate rewards.
Critical weaknesses
- The simulator’s real-world validity is insufficiently established. Patient arrivals, provider incentives, treatment costs, and health outcomes are prescribed rather than calibrated against hospital data, and the paper reports no structured independent expert validation. Literature consistency checks cannot establish that the combined environment or discovered strategies are realistic. Consequently, the findings demonstrate behavior within the specified simulator, but their relevance to actual hospital policy remains unsubstantiated. Experimental sugessions for the authors: Calibrate key components using hospital data and validate predictions against independent observations. If such data are unavailable, at minimum obtain structured assessments from (expert) clinicians, or hospital administrators, including blinded reviews of generated strategies. Revise implausible assumptions (particularly payment despite zero treatment effort).
- The methodological contribution and added value of LLM-based program search are not sufficiently isolated. The framework primarily integrates existing evolutionary search and game-theoretic tools. Current comparisons do not clearly establish whether gains arise from meaningful LLM-generated changes to policy logic, additional search opportunities, or the underlying optimization machinery. Authors should compare against non-LLM evolutionary search, random program mutations, and parameter-only optimization under matched simulation budgets. Ablate the LLM editing component while retaining the remaining framework, conduct multiple independent searches, and report policy outcomes alongside runtime and token costs. These controls are necessary to demonstrate what the LLM contributes beyond the integration of existing methods.
Questions for authors
- Although the paper reports rejection and waiting alongside treated-patient health, could the authors evaluate health benefits across a fixed cohort of all arriving patients, explicitly accounting for rejected patients and those still waiting at the evaluation horizon? Do the main policy rankings remain unchanged under this assessment?
- The paper shows that soft penalties and reduced profit incentives do not eliminate all harmful responses. Could the authors test clinically and operationally grounded safeguards (such as minimum-service requirements, payments linked to delivered care, or access constraints) and assess whether they continue to protect patients under further provider adaptation?
5 comments
Response to Reviewer T9jH (Part 2/4)
W2. The contribution of LLM-based program search is not isolated from the evolutionary and game-theoretic machinery. Compare with non-LLM evolutionary search, random program mutations and parameter-only optimization under matched budgets; ablate the LLM editor; run multiple independent searches; report runtime and token costs.
We add a direct-generation baseline (B0) against the paper's E0 joint-design result. B0 uses DeepSeek to generate 200 complete administrator–provider programs independently from the task description and DSL schema, without a supplied program or rollout feedback. Following the paper's selection rule, the first valid sample and the five highest-scoring training candidates are compared on validation seeds. We then assess the selected program and the archived E0 program on the same 20 held-out seeds.
| Method | Initialization and search | Held-out (submitted parameters) |
|---|---|---|
| Paper E0 (DeepSeek, original prompt, high thinking) | 17 supplied programs + 200 feedback-guided proposals | 15.45 |
| B0, first sample | One independently generated program | 5.68 |
| B0, best of 200 | 200 independent generations, followed by validation selection | 14.74 |
The paper's E0 exceeds B0 by 0.71 in held-out . Under the Chinese parameters, the same frozen E0 and B0 programs score 15.36 and 14.13, respectively; in the SPARCS scenario, on 20 fresh common seeds, they score 14.34 and 12.70. This is a complete-pipeline comparison: E0 uses the supplied starts, while B0 uses the current DeepSeek endpoint following a provider update. The paper also reports three independent response searches per task and ABR versus CSRO–MGCCE comparisons to examine search variability and history retention (App. L.8–L.9).
Cost. B0 used 200 calls, all successful on the first attempt, totaling 1.07M prompt and 2.04M completion tokens (API-reported). Re-assessing the archived E0 program required no model calls. All 200 generations with token usage, the selection and the held-out assessment are in rebuttal/direct_generation.
Response to Reviewer T9jH (Part 3/4)
Q1. Evaluate health over a fixed cohort of all arriving patients, counting rejected patients and those still waiting at the horizon. Do the main rankings survive?
We agree that the treated-patient mean conditions on access, so a rule can raise it by treating fewer or easier patients. Using the submitted parameters, we re-assessed every endpoint on the cohort of all arrivals. Each arriving patient is followed to exactly one state at the horizon : treated, rejected, left the queue untreated, or still waiting; the tracker verifies that these four counts sum to arrivals in every rollout. Cohort health is where is the health credited to an arrival who received no treatment. Our headline uses (health actually delivered per arrival); we then vary .
| Editor configuration | Endpoint | Treated | Treated | Rejected | Left untreated | Waiting at | Urgent | |
|---|---|---|---|---|---|---|---|---|
| DeepSeek, original, off | E0 | 0.946 | 62.7% | 4.1% | 32.5% | 0.73% | 0.593 | 0.595 |
| ABR-A endpoint | 0.677 | 66.4% | 0.0% | 32.9% | 0.74% | 0.449 | 0.173 | |
| DeepSeek, original, high | E0 | 0.984 | 97.3% | 0.0% | 2.5% | 0.26% | 0.957 | 0.946 |
| ABR-A endpoint | 0.791 | 99.6% | 0.0% | 0.3% | 0.10% | 0.787 | 0.664 | |
| DeepSeek, revised, off | E0 | 0.995 | 61.2% | 35.4% | 3.1% | 0.25% | 0.609 | 0.963 |
| ABR-A endpoint | 0.816 | 63.4% | 17.8% | 18.2% | 0.55% | 0.517 | 0.014 | |
| DeepSeek, revised, high | E0 | 1.009 | 99.1% | 0.5% | 0.4% | 0.00% | 1.000 | 0.920 |
| ABR-A endpoint | 0.148 | 99.7% | 0.0% | 0.3% | 0.00% | 0.147 | 0.122 | |
| GLM-5.3 | E0 | 0.965 | 88.7% | 0.0% | 10.8% | 0.46% | 0.857 | 0.935 |
| ABR-A endpoint | 0.631 | 99.0% | 0.1% | 0.7% | 0.27% | 0.625 | 0.534 |
Counts and cohort health are pooled over the paper's 20 held-out environment seeds (≈48,000 arrivals per endpoint); treated H uses the paper's original aggregation. Every replay reproduces its released record, with no model calls. The full table with ABR-D and both MGCCE endpoints, the replay records and the scripts are in rebuttal/fixed_cohort.
- The main erosion result holds. Erosion of jointly designed pair under provider adaptation (E0 → ABR-A endpoint) holds on full cohort in 5/5 configurations, including GLM. Across five main endpoints of each configuration (E0, ABR-A, ABR-D, MGCCE 200×3, MGCCE 50×12), 47 of 50 pairwise orderings agree between treated and , and none of the reversals involves the E0 → ABR-A comparison. Two are E0 versus ABR-D endpoint, where treated differs by less than 0.01. The third is E0 versus MGCCE 200×3 in original/off, where the mixture treats more arrivals (66.2% vs 62.7%).
- The cohort view strengthens the harm finding for urgent patients. Health delivered per urgent arrival falls from 0.595 to 0.173 and from 0.963 to 0.014 in the two thinking-off configurations, a loss that the treated mean (0.946 → 0.677; 0.995 → 0.816) understates.
- Robust to the credit given to untreated patients. For , E0 remains above the ABR-A endpoint in 5/5 configurations, and at most 3 of 50 within-configuration orderings change relative to .
- Few patients remain waiting. Patients still waiting at the horizon account for at most 0.78% of arrivals at any endpoint.
- The cohort measure reveals differences in access. For two E0 pairs, treated is 0.946 and 0.995, while is 0.593 and 0.609. Across configurations, 48 of 300 pairwise orderings reverse. We will make the primary health measure in the revision, with treated as a secondary column.
On the paper's five selection-validation seeds of the four DeepSeek configurations, the conclusions are the same: erosion holds in 4/4, and 32 of 36 orderings agree.
Response to Reviewer T9jH (Part 1/4)
We thank the reviewer for the positive assessment and the concrete suggestions, which guided the additional evaluations below.
W1. Real-world validity: public-data alignment, external comparisons and LLM mechanism assessment. Reviewer CvHR shares this concern; the completed comparisons are reported in CvHR (W1, parts a–d). Code, records and verification scripts for all additional evaluations are in the anonymous repository under
rebuttal/. In brief:W1 (continued). Implausible assumption: payment despite zero treatment effort. We agree that this settlement rule should be revised; our S2 experiment below tests payment linked to delivered care.
effort = 0.0and H < 0.001 (App. L.6). Sec. 4.3 and App. M identify this as a boundary case of the settlement rule.