acceptodds
Under review as a conference paper at ICLR 2027

CareLoop: Measuring the Gap Between Knowing Medicine and Practicing It in the Context of Patients' Lives

Abstract

Medical benchmarks usually ask whether an AI system knows the right answer. Patient-facing care also requires discovering incomplete or incorrect information, understanding a person's beliefs and constraints, adapting feasible plans, verifying execution, incorporating delayed evidence, and retaining responsibility for unresolved risk. We introduce CareLoop, a clinical-world sandbox that measures this gap between knowing medicine and practicing it in the context of patients' lives. Its 120 deidentified EHR-derived case contracts distribute state across clinical truth, patient and family beliefs, records, and executed actions; prospectively authored frictions create controlled behavioral forks with observable downstream consequences. Ten models generated 1,200 trajectories. Four blinded LLM Judges produced 4,800 complete-context assessments, and 139 physicians produced 6,000 structured assessments. GPT-5.6 Sol ranked first on the Friction Capability Composite (FCC, 0.824) and its safety-gated counterpart C-RWR (0.753). Across 15,537 realized high-order opportunity assessments, models acted in 0.935, changed the trajectory meaningfully in 0.745, and completed the opportunity effectively in 0.697; hidden-state discovery was the largest shared bottleneck. Individual assessments varied, yet disjoint 60-case halves recovered stable model rankings (median Spearman for FCC and for C-RWR), and aggregate physician and four-Judge rankings converged ( and ). Matched model–Judge analyses showed no uniform self-favoring pattern. Together, these results show how execution-grounded evaluation can extend from directly testable state change to consequential action under partial observability and limited control: the benchmark makes the patient's lived context causally operative and aggregates structured observations over broad case coverage. Anonymous code and data: https://anonymous.4open.science/r/careloop-iclr2027-6e55. The simulation does not establish clinical effectiveness or deployment safety.

Primary area:
datasets and benchmarks
Keywords:
clinical-world simulation, medical AI agents, executable evaluation, partially observable environments, limited-control agency, trajectory-based evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wWP8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors proposed a sandbox called CareLoop, aiming to measure the gap between possessing medical knowledge and practicing medicine in the context of patients’ lives. Within this sandbox, the authors defined four interrelated states: underlying clinical state, patient and family beliefs, recorded evidence, and executed actions. They examined whether AI could identify mismatches between these states and advance care through actions, verification, and plan revision. Based on de-identified EHR-derived data, the authors constructed a dataset containing 120 case contracts and evaluated ten LLMs, generating 1,200 interaction trajectories. Subsequently, four LLM judges and 139 physicians conducted structured blinded assessments of these trajectories. The authors compared agreement between the two groups of evaluators and examined the stability of model rankings.

Critical strengths
  1. I think the authors’ research topic is very meaningful: the gap between LLMs possessing medical knowledge and practicing medicine in the context of patients’ lives.

  2. The authors invited physicians to participate in the evaluation, which greatly strengthens the validity of the study.

Critical weaknesses
  1. The authors claim that their sandbox can simulate the dynamic evolution of patient care in the real world. However, based on the main text and appendix, I still lack sufficiently concrete details to understand how the sandbox produces this dynamic evolution. I also remain unclear about details such as the prompts used to generate trajectories and how physicians apply the stated evaluation criteria to assess individual trajectories.

    1.1. Many concepts are still not explained concretely enough, including distributed state, case contract, friction, and HO opportunity.

    1.2. Although the paper organizes its results around overall performance, process-level performance, and evaluator reliability, the transitions between different levels of analysis—cases, opportunities, trajectories, and model rankings—are not sufficiently clear. The relationships among the different classification and scoring systems are also explained mainly in the appendix.

  2. In the abstract and introduction, the authors emphasize that CareLoop can measure the gap between possessing medical knowledge and practicing medicine in the context of patients’ lives. However, substantial space in the main text and appendix is devoted to the stability of model rankings, while the discussion of simulation realism, scoring validity, and how to distinguish insufficient medical knowledge from failures in practice remains relatively weak.

Questions for authors
  1. Could you explain the specific design and implementation details of your sandbox more clearly and in greater detail in the methodology and experimental sections?
  2. Please clearly distinguish the different levels of your framework in the methodology section. For example, as I understand it, the items listed in Lines 161–163—“opening presentation, hidden clinical state, actor-specific knowledge and behavior, workspace evidence, disclosure and reliability rules, delayed events, friction lifecycle, external actions, minimum duties, high-order (HO) opportunities, false-closure traps, and bounded closure criteria”—serve different functions within the framework, but their hierarchy and relationships are not clearly explained. There are many similarly confusing descriptions throughout the paper.
  3. Please explain the reasons for choosing parameters such as the weights (0.20, 0.20, 0.15, 0.20, 0.10, 0.15) in Line 191.
  4. What evidence from the existing experiments distinguishes insufficient medical knowledge from failures in information acquisition or execution, and supports the central claim that CareLoop measures the gap between medical knowledge and medical practice?
Reviewer_eHs5OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces CareLoop, a simulated clinical environment for evaluating medical agents under partial observability and real-world constraints. It evaluates ten LLMs on 120 EHR-derived cases using trajectory-based metrics, four LLM judges, and physician assessments. The authors report model rankings and analyze failures in information discovery, action execution, and clinical follow-up.

Critical strengths
  1. Relevant problem setting. Evaluating medical agents beyond static knowledge tests is meaningful, particularly under patient-specific constraints.
  2. Extensive evaluation effort. The inclusion of multiple models, structured trajectories, and physician assessments provides useful empirical observations.
Critical weaknesses
  1. Insufficient validation of the central claim. The paper claims to measure the knowledge-to-practice gap but provides neither a matched knowledge-only baseline nor controlled counterfactuals isolating the effects of real-world friction. Consequently, it remains unclear whether the measured differences reflect clinical agency, general model capability, or simulator-specific artifacts.
  2. Questionable evaluation reliability and generalizability. The proposed FCC and C-RWR metrics depend heavily on subjective trajectory assessments and hand-designed scoring rules. Individual judge agreement is modest (ICC = 0.257 for FCC), while aggregate ranking consistency does not establish clinical validity. Furthermore, evaluating only one trajectory per model-case pair on 120 simulated cases leaves substantial uncertainty about robustness to stochastic variation and broader clinical settings.
Questions for authors
  1. How can the authors demonstrate that CareLoop measures the knowledge-to-practice gap rather than general model capability? Can matched knowledge-only and friction-free baselines be provided?
  2. How are the FCC/C-RWR scoring rules independently validated against clinically meaningful outcomes, beyond agreement on aggregate model rankings?
  3. How stable are the conclusions across repeated trajectories, alternative simulation configurations, and independently constructed clinical cases?
Reviewer_AvYCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CareLoop, a clinical simulation benchmark designed to evaluate the gap between medical knowledge and effective care in patients' real-life contexts. Rather than focusing solely on the correctness of medical recommendations, CareLoop evaluates whether agents can navigate practical constraints, adapt their decisions, verify execution, and manage unresolved risks. The authors evaluate ten models using 120 clinical cases, with assessments from both LLM judges and 139 physicians. The experiment scale is large.

Critical strengths

The paper addresses an important and meaningful problem: whether medical knowledge can be translated into effective care under real-world constraints. I particularly appreciate the shift from evaluating the correctness of recommendations to assessing their downstream impact on patient trajectories. The friction-centered benchmark design is interesting, and the extensive evaluations involving multiple models and clinical experts provide valuable insights into current medical agents' limitations. Overall, the paper is well motivated and carefully presented. I appreciate authors efforts on the experiment, especially using 139 physicians for judgement.

Critical weaknesses

My main concern is the distinction from existing medical agent benchmarks, particularly PatientAgentBench and HealthAgentBench. Although the paper discusses these works, more detailed comparisons would help establish what fundamentally new evaluation capabilities CareLoop provides as these benchmarks also addressing the similar problems. In particular, it remains unclear whether the proposed friction mechanisms introduce genuinely new evaluation signals beyond those captured by existing interactive healthcare benchmarks.

Another concern is simulation fidelity and practical scalability. Patient behaviors, treatment adherence, and real-world constraints are difficult to simulate faithfully, raising questions about how well the observed trajectories reflect actual clinical interactions. Furthermore, the reliance on extensive physician evaluations introduces substantial annotation costs and expert availability challenges, which may limit the scalability of the benchmark.

Questions for authors

Could the authors provide a more detailed comparison with PatientAgentBench and HealthAgentBench, particularly regarding the unique evaluation capabilities of CareLoop?

How can the authors demonstrate that the proposed friction mechanisms capture capabilities not already measured by existing benchmarks? Would matched counterfactual cases with and without friction help isolate their contributions?

How is the realism of simulated patient behaviors and clinical trajectories validated against real-world clinical interactions?

How could the evaluation framework scale given the costs and limited availability of clinical experts?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.