- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The authors proposed a sandbox called CareLoop, aiming to measure the gap between possessing medical knowledge and practicing medicine in the context of patients’ lives. Within this sandbox, the authors defined four interrelated states: underlying clinical state, patient and family beliefs, recorded evidence, and executed actions. They examined whether AI could identify mismatches between these states and advance care through actions, verification, and plan revision. Based on de-identified EHR-derived data, the authors constructed a dataset containing 120 case contracts and evaluated ten LLMs, generating 1,200 interaction trajectories. Subsequently, four LLM judges and 139 physicians conducted structured blinded assessments of these trajectories. The authors compared agreement between the two groups of evaluators and examined the stability of model rankings.
Critical strengths
-
I think the authors’ research topic is very meaningful: the gap between LLMs possessing medical knowledge and practicing medicine in the context of patients’ lives.
-
The authors invited physicians to participate in the evaluation, which greatly strengthens the validity of the study.
Critical weaknesses
-
The authors claim that their sandbox can simulate the dynamic evolution of patient care in the real world. However, based on the main text and appendix, I still lack sufficiently concrete details to understand how the sandbox produces this dynamic evolution. I also remain unclear about details such as the prompts used to generate trajectories and how physicians apply the stated evaluation criteria to assess individual trajectories.
1.1. Many concepts are still not explained concretely enough, including distributed state, case contract, friction, and HO opportunity.
1.2. Although the paper organizes its results around overall performance, process-level performance, and evaluator reliability, the transitions between different levels of analysis—cases, opportunities, trajectories, and model rankings—are not sufficiently clear. The relationships among the different classification and scoring systems are also explained mainly in the appendix.
-
In the abstract and introduction, the authors emphasize that CareLoop can measure the gap between possessing medical knowledge and practicing medicine in the context of patients’ lives. However, substantial space in the main text and appendix is devoted to the stability of model rankings, while the discussion of simulation realism, scoring validity, and how to distinguish insufficient medical knowledge from failures in practice remains relatively weak.
Questions for authors
- Could you explain the specific design and implementation details of your sandbox more clearly and in greater detail in the methodology and experimental sections?
- Please clearly distinguish the different levels of your framework in the methodology section. For example, as I understand it, the items listed in Lines 161–163—“opening presentation, hidden clinical state, actor-specific knowledge and behavior, workspace evidence, disclosure and reliability rules, delayed events, friction lifecycle, external actions, minimum duties, high-order (HO) opportunities, false-closure traps, and bounded closure criteria”—serve different functions within the framework, but their hierarchy and relationships are not clearly explained. There are many similarly confusing descriptions throughout the paper.
- Please explain the reasons for choosing parameters such as the weights (0.20, 0.20, 0.15, 0.20, 0.10, 0.15) in Line 191.
- What evidence from the existing experiments distinguishes insufficient medical knowledge from failures in information acquisition or execution, and supports the central claim that CareLoop measures the gap between medical knowledge and medical practice?