- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces MedEVM to study whether LLMs commit to diagnoses when sufficient evidence is available. It analyzes evidence–value misalignment and evaluates predictive features, an oracle-waiting intervention, and a diagnosis-verification harness.
Critical strengths
-
This paper explores an important distinction: arriving at a correct diagnosis is not the same as determining whether the available evidence is sufficient to justify accepting that diagnosis. This perspective broadens evaluation beyond a narrow focus on the accuracy of the final answer.
-
Controlled sequences of evidence facilitate a systematic analysis of the timing of information submission, the order of evidence presentation, and the impact of misleading information. The predictive experiments, intervention analyses, and ablation studies presented in this work offer valuable observations and hypotheses for future research.
Critical weaknesses
-
The paper reports many “main findings,” but I am still unsure what they contribute to the AI and healthcare communities. RQ1–RQ4 each seem reasonable, but the paper needs to explain what researchers in these communities should learn from the results and why these lessons matter.
-
The paper already discusses more interactive settings such as AgentClinic, so the value of studying this intermediate setting needs clearer justification (maybe in the very beginning). There is also no expert behavioral baseline under the same evidence conditions. Would experts make similar decisions, and would expert-informed interventions work better? These comparisons would help separate model limitations from effects of the task design.
-
I have concerns about the scope of the claims in Table 3 and Section 5. The authors use these results to argue that EVM features can predict submission timing and diagnostic errors, and that reducing EVM can improve accuracy. The clinical meaning of these findings remains unclear without expert assessment of the behavioral changes and errors. The authors should narrow these claims and state their boundaries from the beginning.
-
The findings depend on the selected models, and their relevance to other models remains unclear. Comparisons with medical models would help show whether the observed behavior also occurs in systems designed for this domain.
-
The presentation also contributes to my confusion. After the initial experiments and discussion, Sections 5 and 6 introduce further experiments and findings. This structure, together with the extensive formal notation, makes the main argument difficult to follow.
Questions for authors
Please refer to the weaknesses above.
6 comments
Responses to Reviewer vQ6h (2)
Thank you for your valuable comments!
The paper already discusses more interactive settings such as AgentClinic, so the value of studying this intermediate setting needs clearer justification (maybe in the very beginning). There is also no expert behavioral baseline under the same evidence conditions. Would experts make similar decisions, and would expert-informed interventions work better? These comparisons would help separate model limitations from effects of the task design.
We appreciate this central question. We agree that the paper should explain at the outset what important question this intermediate setting addresses and why the design is needed. The original manuscript did not make this motivation sufficiently clear. We chose progressive-evidence diagnosis to study a fundamental capability that full diagnostic agents also require: using the distinguishing value of currently available patient facts to decide whether to continue observing or submit a diagnosis.
The setting allows us to examine information acquisition separately from decisions made using information already available. In fully interactive environments such as AgentClinic, agents choose questions, examinations, and tools, so different systems follow different information-acquisition paths. A diagnostic failure may arise because an agent never acquired a decisive clue, or because it had sufficient clues but did not act appropriately. Final accuracy alone makes these causes difficult to distinguish. MedEVM fixes the candidate set and evidence sequence, reveals one patient observation per turn, and lets the model choose WAIT or ANSWER, with submission ending the trajectory. The facts available at each decision point are therefore explicit, and behavior can be compared under identical information.
The design retains three key features: progressive evidence arrival, autonomous submission timing, and irreversible submission. Single-turn, full-information question answering does not expose these processes, while differing information paths in fully interactive environments complicate attribution. The case-level sufficiency boundary t* lets us identify submissions made before decisive evidence and waiting that continues after sufficient evidence is available. Controlled changes to order, repetition, and misleading clues then allow us to examine what changes these behaviors. This intermediate setting thus connects static diagnostic capability to full-agent performance through process-level evaluation. It isolates failures that persist after information has been supplied, providing a clear basis for improving evidence verification and submission policies.
We further integrated Codex and OpenCode into the same environment to test whether these problems persist with general-purpose agent capabilities. On the shared 200 cases, Codex CLI 0.167, OpenCode 2.1.4, and EVD-Harness use the same corresponding models, evidence order, and frozen CDW retrieval cards. Both general-purpose harnesses can manage their own histories, invoke tools, and organize review, while accessing only patient facts revealed so far. Output budgets are fixed in advance; autonomous histories and tool interfaces lead to longer input prompts for the general-purpose harnesses. Each method is scored on its first valid submission, with accuracy computed over all cases and unanswered cases counted as errors.
Table R2. General-purpose harness comparison (%, n = 200). Note: n = 200; DeepSeek-V4-Flash LLM-only: n = 199.
| Model | Method | Accuracy | Premature | At t* | No answer |
|---|---|---|---|---|---|
| Claude-Opus-4-8 | LLM-only | 37.5 | 11.5 | 27.0 | 56.0 |
| EVD-Harness | 84.0 | 5.0 | 67.5 | 9.5 | |
| Codex | 74.0 | 13.5 | 56.0 | 16.5 | |
| OpenCode | 74.5 | 12.0 | 57.0 | 16.5 | |
| DeepSeek-V4-Flash | LLM-only | 45.7 | 22.1 | 26.6 | 39.2 |
| EVD-Harness | 67.5 | 18.5 | 70.0 | 1.0 | |
| Codex | 50.5 | 17.5 | 55.5 | 17.5 | |
| OpenCode | 49.5 | 7.0 | 51.5 | 22.5 | |
| Kimi-K2.6 | LLM-only | 61.5 | 19.5 | 36.0 | 5.0 |
| EVD-Harness | 77.0 | 17.5 | 67.0 | 2.5 | |
| Codex | 68.5 | 2.0 | 53.0 | 29.0 | |
| OpenCode | 57.0 | 2.5 | 39.0 | 29.0 |
As shown in Table R2, premature submissions and failures to answer in time persist even with autonomous state management and tool orchestration. Claude still submits prematurely in approximately 12%–14% of cases under Codex and OpenCode. Kimi has a lower premature-submission rate, but both harnesses leave 29.0% of cases unanswered. EVD-Harness reduces Kimi's unanswered rate to 2.5% and achieves higher accuracy. These results indicate that general-purpose agent capabilities need to be coupled with submission decisions grounded in patient evidence. MedEVM reveals differences between harnesses at precisely this stage and tests whether dedicated verification improves diagnosis, informing the design of more complete agent systems.
Your proposed expert behavioral comparison will further help distinguish limitations in models' evidence use from reasonable decisions under the task design. Clinical experts will review the same 200 cases in the same evidence order, choose WAIT or ANSWER at each turn, and record their reasoning. They will not see the reference diagnosis, t*, future observations, or model outputs. We will compare diagnostic accuracy, submission timing, and unanswered cases. We will also derive expert-informed submission rules from separate development cases, freeze them, and compare them with the Witness on the same held-out cases to test whether expert experience further improves submission decisions. Completing and checking the turn-by-turn comparison across all 200 cases takes time; we expect to provide these results and analyses within the next two days.
We will make the motivation for this design explicit at the beginning of the revision and use the general-purpose harness and expert comparisons to illustrate its value: identifying decision failures under the same patient facts and translating the findings into more reliable agent submission mechanisms. Thank you again for helping us explain the necessity of this intermediate setting and its relevance to practical systems.
Responses to Reviewer vQ6h (1)
Thank you for your valuable comments!
We appreciate this important suggestion. We agree that the paper should explain clearly which problems these findings help AI and healthcare researchers address. The original manuscript presents the experiments by research question but does not sufficiently articulate their collective significance. Our central contribution is to establish how models decide when to act on patient evidence as a measurable, interpretable, and improvable aspect of translating medical knowledge into reliable diagnosis. The experiments offer three main insights.
Together, our work connects capability evaluation, failure analysis, and method development: it identifies when existing diagnostic beliefs fail to guide action, tests which factors disrupt evidence use, and evaluates whether interventions on submission improve outcomes. This provides an experimental foundation for reliable decision-making research in AI and concrete guidance for diagnosing failures and designing evidence-verification mechanisms in medical AI. We will emphasize these implications in the introduction and the conclusions of each research question. Thank you again for helping us make the substantive value to both communities clearer.