acceptodds
Under review as a conference paper at ICLR 2027

Can LLMs Value the Right Evidence? Evidence–Value Misalignment in Dynamic Medical Diagnosis

Abstract

A correct diagnosis reached from insufficient or misleading evidence can pose a clinical hazard, yet outcome-based accuracy may reward such lucky guesses. We call this mismatch between diagnostic decisions and the value of available evidence **Evidence-Value Misalignment (EVM)**. To disentangle evidential grounding independently from diagnostic accuracy, we introduce MedEVM, a dynamic benchmarking environment comprising 1,050 cases across 24 disease systems. Observations arrive turn by turn, requiring models to continuously calibrate its decision by deciding whether to wait for more evidence or submit a diagnosis. Across 9 LLMs, four interesting patterns are observed. (1) **Miscalibrated evidence tracking**. Making a diagnosis often fails to calibrate evidence sufficiency, even in more capable models, and even worsens in reasoning mode. (2) **Misaligned diagnosis submission**. Confidence in the correct diagnosis often fails to ensure timely submission despite sufficient evidence. (3) **Evidence order matters**. Reordering the same evidence changes diagnoses even when model confidence remains similar. (4) Misleading evidence remains influential. Added misleading evidence redirects diagnoses even after prior evidence becomes sufficient. We further verify that EVM predicts errors and that preventing premature submission improves accuracy. These findings motivate **Evidence-Verified Diagnosis Harness (EVD-Harness)**. It decouples diagnosis generation from submission through an offline Contrastive Diagnostic Wiki and three online control stages, namely observation management, proposal and witness verification, and diagnosis submission control. Across five LLMs, EVD-Harness improves accuracy by 12.0–51.1 percentage points while mitigating EVM-related failures. Our results demonstrate that verifying evidential support before submission can make diagnostic decisions more reliable.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Large Language Model, Dynamic Medical Diagnosis, Clinical Diagnosis Harness

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vQ6hOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces MedEVM to study whether LLMs commit to diagnoses when sufficient evidence is available. It analyzes evidence–value misalignment and evaluates predictive features, an oracle-waiting intervention, and a diagnosis-verification harness.

Critical strengths
  1. This paper explores an important distinction: arriving at a correct diagnosis is not the same as determining whether the available evidence is sufficient to justify accepting that diagnosis. This perspective broadens evaluation beyond a narrow focus on the accuracy of the final answer.

  2. Controlled sequences of evidence facilitate a systematic analysis of the timing of information submission, the order of evidence presentation, and the impact of misleading information. The predictive experiments, intervention analyses, and ablation studies presented in this work offer valuable observations and hypotheses for future research.

Critical weaknesses
  1. The paper reports many “main findings,” but I am still unsure what they contribute to the AI and healthcare communities. RQ1–RQ4 each seem reasonable, but the paper needs to explain what researchers in these communities should learn from the results and why these lessons matter.

  2. The paper already discusses more interactive settings such as AgentClinic, so the value of studying this intermediate setting needs clearer justification (maybe in the very beginning). There is also no expert behavioral baseline under the same evidence conditions. Would experts make similar decisions, and would expert-informed interventions work better? These comparisons would help separate model limitations from effects of the task design.

  3. I have concerns about the scope of the claims in Table 3 and Section 5. The authors use these results to argue that EVM features can predict submission timing and diagnostic errors, and that reducing EVM can improve accuracy. The clinical meaning of these findings remains unclear without expert assessment of the behavioral changes and errors. The authors should narrow these claims and state their boundaries from the beginning.

  4. The findings depend on the selected models, and their relevance to other models remains unclear. Comparisons with medical models would help show whether the observed behavior also occurs in systems designed for this domain.

  5. The presentation also contributes to my confusion. After the initial experiments and discussion, Sections 5 and 6 introduce further experiments and findings. This structure, together with the extensive formal notation, makes the main argument difficult to follow.

Questions for authors

Please refer to the weaknesses above.

6 comments
AuthorsOpenReview ↗

Responses to Reviewer vQ6h (1)

Thank you for your valuable comments!

The paper reports many “main findings,” but I am still unsure what they contribute to the AI and healthcare communities. RQ1–RQ4 each seem reasonable, but the paper needs to explain what researchers in these communities should learn from the results and why these lessons matter.

We appreciate this important suggestion. We agree that the paper should explain clearly which problems these findings help AI and healthcare researchers address. The original manuscript presents the experiments by research question but does not sufficiently articulate their collective significance. Our central contribution is to establish how models decide when to act on patient evidence as a measurable, interpretable, and improvable aspect of translating medical knowledge into reliable diagnosis. The experiments offer three main insights.

  1. The gap between diagnostic belief and submission behavior provides a more informative basis for evaluating AI capabilities and improving medical models. RQ1–RQ2 jointly measure sufficiency, probabilities over candidate diagnoses, and submission behavior for the same case and evidence prefix. Among trajectories that reach the sufficiency boundary, the mean normalized probability of the correct diagnosis is already 0.70–0.86, yet submission timing remains misaligned. Recognizing the correct diagnosis therefore does not necessarily translate into appropriate action. For AI research, this provides a concrete task and measurements for studying how beliefs become decisions. For medical AI research, it helps distinguish failures that require additional diagnostic knowledge from those that require better submission policies, allowing training and system improvements to target specific failure points.
  2. Evidence presentation can alter diagnostic decisions, and our controlled experiments provide a reproducible way to test the reliability of evidence use. RQ3 reorders earlier observations while preserving the evidence set at the sufficiency boundary; RQ4 separately controls repeated information and misleading-evidence strength. We find that even when both orderings reach the boundary and the model forms similar diagnostic beliefs, final outcomes can differ. Stronger misleading evidence further degrades diagnostic performance. These findings show that actions are sensitive to the path by which information arrives and to its surface salience. For AI research, this enables controlled tests of whether decisions track evidence value. For medical AI research, it motivates evaluating decision stability under progressively arriving information, repeated descriptions, and distracting clues, and checking whether decisive findings continue to guide diagnosis.
  3. Submission timing is an actionable target for improving diagnostic outcomes and motivates a concrete system-design principle. Natural Gate blocks premature submissions before the annotated sufficiency boundary t* and restores the model's own decisions from t* onward. As shown in Table A8, this intervention yields net accuracy gains of 5.3–44.9 percentage points over the full case set, showing that premature termination can prevent models from correcting diagnoses using subsequent decisive evidence. For AI research, this provides direct experimental support for improving existing models through decision control. For medical AI research, it motivates checking whether current patient facts distinguish competing diagnoses before submission. EVD-Harness implements this principle through distinguishing knowledge, evidence verification, and submission control, turning the findings into an executable and evaluable improvement.

Together, our work connects capability evaluation, failure analysis, and method development: it identifies when existing diagnostic beliefs fail to guide action, tests which factors disrupt evidence use, and evaluates whether interventions on submission improve outcomes. This provides an experimental foundation for reliable decision-making research in AI and concrete guidance for diagnosing failures and designing evidence-verification mechanisms in medical AI. We will emphasize these implications in the introduction and the conclusions of each research question. Thank you again for helping us make the substantive value to both communities clearer.

AuthorsOpenReview ↗

Responses to Reviewer vQ6h (2)

Thank you for your valuable comments!

The paper already discusses more interactive settings such as AgentClinic, so the value of studying this intermediate setting needs clearer justification (maybe in the very beginning). There is also no expert behavioral baseline under the same evidence conditions. Would experts make similar decisions, and would expert-informed interventions work better? These comparisons would help separate model limitations from effects of the task design.

We appreciate this central question. We agree that the paper should explain at the outset what important question this intermediate setting addresses and why the design is needed. The original manuscript did not make this motivation sufficiently clear. We chose progressive-evidence diagnosis to study a fundamental capability that full diagnostic agents also require: using the distinguishing value of currently available patient facts to decide whether to continue observing or submit a diagnosis.

The setting allows us to examine information acquisition separately from decisions made using information already available. In fully interactive environments such as AgentClinic, agents choose questions, examinations, and tools, so different systems follow different information-acquisition paths. A diagnostic failure may arise because an agent never acquired a decisive clue, or because it had sufficient clues but did not act appropriately. Final accuracy alone makes these causes difficult to distinguish. MedEVM fixes the candidate set and evidence sequence, reveals one patient observation per turn, and lets the model choose WAIT or ANSWER, with submission ending the trajectory. The facts available at each decision point are therefore explicit, and behavior can be compared under identical information.

The design retains three key features: progressive evidence arrival, autonomous submission timing, and irreversible submission. Single-turn, full-information question answering does not expose these processes, while differing information paths in fully interactive environments complicate attribution. The case-level sufficiency boundary t* lets us identify submissions made before decisive evidence and waiting that continues after sufficient evidence is available. Controlled changes to order, repetition, and misleading clues then allow us to examine what changes these behaviors. This intermediate setting thus connects static diagnostic capability to full-agent performance through process-level evaluation. It isolates failures that persist after information has been supplied, providing a clear basis for improving evidence verification and submission policies.

We further integrated Codex and OpenCode into the same environment to test whether these problems persist with general-purpose agent capabilities. On the shared 200 cases, Codex CLI 0.167, OpenCode 2.1.4, and EVD-Harness use the same corresponding models, evidence order, and frozen CDW retrieval cards. Both general-purpose harnesses can manage their own histories, invoke tools, and organize review, while accessing only patient facts revealed so far. Output budgets are fixed in advance; autonomous histories and tool interfaces lead to longer input prompts for the general-purpose harnesses. Each method is scored on its first valid submission, with accuracy computed over all cases and unanswered cases counted as errors.

Table R2. General-purpose harness comparison (%, n = 200). Note: n = 200; DeepSeek-V4-Flash LLM-only: n = 199.

ModelMethodAccuracyPrematureAt t*No answer
Claude-Opus-4-8LLM-only37.511.527.056.0
EVD-Harness84.05.067.59.5
Codex74.013.556.016.5
OpenCode74.512.057.016.5
DeepSeek-V4-FlashLLM-only45.722.126.639.2
EVD-Harness67.518.570.01.0
Codex50.517.555.517.5
OpenCode49.57.051.522.5
Kimi-K2.6LLM-only61.519.536.05.0
EVD-Harness77.017.567.02.5
Codex68.52.053.029.0
OpenCode57.02.539.029.0

As shown in Table R2, premature submissions and failures to answer in time persist even with autonomous state management and tool orchestration. Claude still submits prematurely in approximately 12%–14% of cases under Codex and OpenCode. Kimi has a lower premature-submission rate, but both harnesses leave 29.0% of cases unanswered. EVD-Harness reduces Kimi's unanswered rate to 2.5% and achieves higher accuracy. These results indicate that general-purpose agent capabilities need to be coupled with submission decisions grounded in patient evidence. MedEVM reveals differences between harnesses at precisely this stage and tests whether dedicated verification improves diagnosis, informing the design of more complete agent systems.

Your proposed expert behavioral comparison will further help distinguish limitations in models' evidence use from reasonable decisions under the task design. Clinical experts will review the same 200 cases in the same evidence order, choose WAIT or ANSWER at each turn, and record their reasoning. They will not see the reference diagnosis, t*, future observations, or model outputs. We will compare diagnostic accuracy, submission timing, and unanswered cases. We will also derive expert-informed submission rules from separate development cases, freeze them, and compare them with the Witness on the same held-out cases to test whether expert experience further improves submission decisions. Completing and checking the turn-by-turn comparison across all 200 cases takes time; we expect to provide these results and analyses within the next two days.

We will make the motivation for this design explicit at the beginning of the revision and use the general-purpose harness and expert comparisons to illustrate its value: identifying decision failures under the same patient facts and translating the findings into more reliable agent submission mechanisms. Thank you again for helping us explain the necessity of this intermediate setting and its relevance to practical systems.

Reviewer_8siROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether LLMs know when the available evidence justifies committing to a diagnosis, describing premature answers and unnecessary waiting as Evidence–Value Misalignment. It introduces MedEVM, a benchmark of 1,050 cases from published medical reports. Clinical observations arrive sequentially, and the model must either wait or submit a final diagnosis from a fixed candidate set. Each case has an annotated earliest point at which the evidence distinguishes the correct diagnosis from its alternatives.

Across nine LLMs, the authors find that greater diagnostic capability does not consistently produce better commitment decisions, and that evidence order and misleading observations can change the outcome. An oracle intervention that prevents submission before the annotated boundary allows many premature errors to be corrected. The proposed EVD-Harness combines a contrastive diagnostic Wiki with a separate witness that checks whether a proposed diagnosis is supported by the current patient evidence. Evaluations on five models report improved accuracy, fewer premature submissions and non-answers, and greater stability under evidence perturbations.

Critical strengths
  1. MedEVM gives evidence sufficiency a useful temporal reference. Linking the candidate diagnoses to the first distinguishing observation allows premature commitment and unnecessary waiting to be evaluated along the same case trajectory. This extends the progressive-evidence setting of MINT[1] with an explicit reference for when commitment becomes justified. It provides a concrete way to examine how diagnostic behavior changes as evidence arrives.
  2. The contrastive Wiki addresses the reasoning required by this task. Evidence supporting one disease may also support its closest alternatives; the important information is what separates them. Organizing the same retrieved documents around these distinctions improves performance, giving the design a specific empirical rationale. Keeping reference knowledge separate from patient observations also preserves an important distinction between what a disease can cause and what has actually been observed in this case.

[1]: https://arxiv.org/abs/2604.04325

Critical weaknesses
  1. The conceptual advance is narrower than the broad evidence-valuation framing suggests. ClinDet-Bench[1] already finds premature judgment and unnecessary abstention despite demonstrated knowledge in rule-based clinical decisions. Sufficient Context[2] separates information sufficiency from answer correctness and uses it to guide abstention. MedEVM's contribution is the temporal analysis of differential diagnosis, anchored to the first distinguishing observation. This is a useful benchmark extension, but the findings provide a more limited advance in understanding evidence valuation than the introduction of EVM as a general failure mode implies.
  2. The witness's benefit is supported by the ablations, including controls for history and execution. What remains uncertain is whether checking the cited, distinguishing evidence improves stopping beyond an additional confidence or sufficiency assessment. The witness reduces premature answers but also increases unanswered cases relative to submitting proposals without review, so its benefit includes a caution–coverage trade-off. A simpler stopping gate given the same proposals and retrieved knowledge, with comparable inference budgets, would help establish the value of the proposed verification. Confidence- and sufficiency-based controllers such as those in MediQ[3] motivate this comparison; the relevant test is within MedEVM's own fixed-evidence setting.

[1]: https://arxiv.org/abs/2602.22771

[2]: https://arxiv.org/abs/2411.06037

[3]: https://arxiv.org/abs/2406.00922

Questions for authors
  1. When the witness rejects a correct proposal at or after the annotated sufficiency boundary, what gaps does its recorded rationale identify? How often do these rejections lead to unnecessary waiting or a final non-answer? This would clarify where the added caution conflicts with the benchmark's commitment criterion.

  2. Given the same diagnosis proposals, CDW knowledge, and execution conditions, does the witness improve stopping over a simpler confidence- or sufficiency-based gate with a comparable inference budget? Both premature answers and unnecessary waiting are relevant to establishing the benefit of checking distinguishing evidence.

8 comments
AuthorsOpenReview ↗

Responses to Reviewer 8siR (1)

Thank you for your valuable comments!

The conceptual advance is narrower than the broad evidence-valuation framing suggests. ClinDet-Bench[1] already finds premature judgment and unnecessary abstention despite demonstrated knowledge in rule-based clinical decisions. Sufficient Context[2] separates information sufficiency from answer correctness and uses it to guide abstention. MedEVM's contribution is the temporal analysis of differential diagnosis, anchored to the first distinguishing observation. This is a useful benchmark extension, but the findings provide a more limited advance in understanding evidence valuation than the introduction of EVM as a general failure mode implies.

We appreciate your thoughtful discussion of the connections to prior work. As you note, ClinDet-Bench identifies premature judgment and unnecessary abstention despite demonstrated knowledge of clinical rules. MINT further studies premature answers, evidence order, and subsequent self-correction as medical evidence is progressively revealed. Sufficient Context explicitly distinguishes context sufficiency from answer correctness and uses sufficiency signals to improve selective answering. These studies provide an important foundation for understanding how models use evidence. Your comment helped us recognize the need to explain more clearly what our work adds to this foundation about the relationship between evidence value and diagnostic decisions. We see three main contributions.

1. Jointly examining evidence sufficiency, diagnostic belief, and submission behavior within the same trajectory reveals a gap between recognizing the correct diagnosis and taking the appropriate action.

MedEVM annotates t*, the earliest turn at which accumulated evidence distinguishes the reference diagnosis from its confounders. At each evidence prefix, we probe the model's probabilities over candidate diagnoses and record its decision to wait or submit. This allows us to ask whether the model recognizes the correct diagnosis when patient evidence becomes sufficient, and whether that recognition actually guides submission. Among trajectories that reach t*, the mean normalized probability assigned to the correct diagnosis is already 0.70–0.86, yet submission timing can still deviate from the sufficiency boundary. This locates the knowledge–behavior gap in the belief–action relationship for the same case under the same known facts, providing new empirical evidence about why diagnostic capability may fail to translate into reliable decisions.

2. Controlled perturbations and submission interventions identify factors that drive actions away from evidence value and establish their diagnostic consequences.

Using t* as a common reference, we separately control evidence order, repeated information, and misleading-evidence strength, and examine changes in both diagnostic belief and action. For example, reordering pre-boundary observations produces substantial output changes even though the paired trajectories contain the same evidence set at t*. This tests whether actions respond consistently to the distinguishing value of patient facts. We further use Natural Gate to block premature submissions before t* and restore the model's own decisions from the boundary onward. This intervention yields net accuracy gains of 5.3–44.9 percentage points over the full case set (Table A8), showing that premature termination can prevent models from using subsequent decisive evidence to correct their diagnoses. Our work thereby connects failures in evidence use to diagnostic consequences that can be improved through intervention, and identifies a concrete target for better agent decision-making.

3. Translating these findings into submission control based on distinguishing evidence, and testing its value beyond simple gates.

EVD-Harness turns the question of whether current patient facts support submission into an explicit verification procedure. CDW provides knowledge that distinguishes candidate diseases; the Witness checks whether patient evidence supports the proposal over competing diagnoses; Citation Check verifies references against the revealed patient record; and unsuccessful verification leads to further evidence acquisition. Following your suggestion, we compare this procedure with confidence- and sufficiency-based gates while fixing the cases, proposals, and knowledge inputs. As shown in Table R4 in our response below, structured evidence verification achieves higher overall accuracy and more submissions at the boundary for all three models shown. Explicitly connecting distinguishing evidence to the current patient record therefore helps translate sufficiency judgments into effective submission decisions, providing a concrete and testable method for improving diagnostic agents.

Together, our work establishes an approach that jointly analyzes evidence, belief, and action against a case-level sufficiency boundary, and uses interventions to improve diagnostic outcomes. It shows that translating medical knowledge into reliable diagnosis also depends on recognizing the distinguishing role of currently available evidence and acting at the appropriate time. These insights inform model evaluation, failure analysis, and the design of evidence-verification mechanisms. We will make these contributions more explicit in the introduction and related work, while fully acknowledging their connections to prior studies. Thank you again for helping us articulate the distinctive value and significance of our work more clearly.

AuthorsOpenReview ↗

Responses to Reviewer 8siR (2)

Thank you for your valuable comments!

The witness's benefit is supported by the ablations, including controls for history and execution. What remains uncertain is whether checking the cited, distinguishing evidence improves stopping beyond an additional confidence or sufficiency assessment. The witness reduces premature answers but also increases unanswered cases relative to submitting proposals without review, so its benefit includes a caution–coverage trade-off. A simpler stopping gate given the same proposals and retrieved knowledge, with comparable inference budgets, would help establish the value of the proposed verification. Confidence- and sufficiency-based controllers such as those in MediQ[3] motivate this comparison; the relevant test is within MedEVM's own fixed-evidence setting.

Given the same diagnosis proposals, CDW knowledge, and execution conditions, does the witness improve stopping over a simpler confidence- or sufficiency-based gate with a comparable inference budget? Both premature answers and unnecessary waiting are relevant to establishing the benefit of checking distinguishing evidence.

Thank you for suggesting this important comparison. We agree that the Witness should be tested directly against simpler stopping rules under the same diagnosis proposals, knowledge inputs, and comparable computational budgets. Following your suggestion, we added a stopping-gate experiment that measures premature submissions, submissions at t*, delayed submissions, unanswered cases, and overall accuracy, capturing both the benefits and costs of each policy.

Experimental setup and procedure. On the same 200 cases, we fix the proposal model, evidence order, CDW retrieval cards, and history management. At each decision point, we generate a diagnosis proposal and pass the identical proposal and revealed patient evidence to each gate. Direct submits immediately. Confidence asks the model to assess the probability that the proposal is correct, returning a brief rationale and a score from 0 to 100. Sufficiency asks whether the current evidence is sufficient to justify immediate submission and returns a sufficiency score. The Witness checks whether current patient findings support the proposal over competing diagnoses and provides citations that can be located in the revealed record. We calibrate the Confidence and Sufficiency thresholds on 100 separate cases, choosing thresholds whose answering coverage is closest to that of the Witness, then freeze them for the main evaluation.

To compare the gates at the same decision opportunities, we first generate a shared proposal sequence: each proposal is reviewed by all gates before the next observation is revealed. We then identify the first point in this sequence at which each gate authorizes submission. Thus, all methods select from the same diagnosis proposals. Each review-based setting allows at most 10 reviews per case, with an output budget of 1,024 tokens per review. The Witness may make one citation-repair attempt when needed, with its calls and tokens included in the cost. Cases without a submission when evidence or review budgets are exhausted are recorded as unanswered and count as errors.

Table R4. Stopping gates with shared proposals and frozen thresholds (%, n = 200)

ModelGatePrematureAt t*DelayedNo answerAccuracyCalls/caseTokens/case (k)
Qwen3.8-27BDirect18.065.015.51.570.019.534
Confidence15.560.519.54.566.521.337
Sufficiency10.562.521.06.071.521.337
Witness13.572.510.53.576.521.238
Kimi-K2.6Direct20.059.519.01.571.019.033
Confidence16.560.518.54.570.020.737
Sufficiency18.058.014.010.067.520.436
Witness17.567.013.02.577.020.637
Claude-Opus-4-8Direct10.064.024.02.077.520.157
Confidence5.563.015.016.573.523.569
Sufficiency4.555.526.513.576.025.074
Witness5.067.518.09.584.023.369

As shown in Table R4, the Witness achieves higher accuracy and more submissions at t* under the same proposals and knowledge inputs. For Kimi-K2.6, accuracy is 70.0% and 67.5% with the two simpler gates, compared with 77.0% with the Witness, which also reduces delayed submissions and unanswered cases. Average call counts and token costs are similar across the three review methods, indicating that explicit verification of distinguishing evidence selects submission times more effectively at comparable computational cost. Across the complete five-model experiment, the Witness achieves higher accuracy and more submissions at t* than either score-based gate.

Reviewer_vXHrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether LLMs submit diagnoses when the evidence is sufficient. MedEVM contains 1,050 cases with ten diagnosis options, observations revealed one at a time, and an annotation of the first sufficient turn. The behavior study examines premature answers, delayed answers, and sensitivity to reordered or misleading observations. An intervention blocking early answers recovers many errors. EVD-Harness then combines medical retrieval, diagnosis proposals, witness review, and citation checks to control submission without the boundary annotation. It reports improved accuracy and timing across five models.

Critical strengths
  • The benchmark measures whether an answer is supported, not just correct. Its annotated stopping point distinguishes justified diagnoses from lucky early guesses and unnecessary waiting.
  • Better stopping decisions recover diagnostic errors. Blocking early answers and using the proposed harness both improve accuracy, even when unanswered cases count as failures.
Critical weaknesses
  • The baselines need reconciling. Kimi's premature-answer rate changes from 17.0% in Table A1 to 49.6% in Table 4 under the same timing definition. The configuration used to measure harness gains needs clarification.
  • Some gains come from changes outside evidence checking. On 200 cases, history and response-handling changes alone raise Opus accuracy from 37.5% to 75.0%. Separating these effects would clarify the method's contribution.
  • The added value for practical agents is unclear. The baseline workflow restricts models to WAIT/ANSWER. A general-purpose harness (e.g., Claude Code or Codex), adapted to the same task with matched knowledge and computation, would provide a more informative comparison.
Questions for authors
  1. What configuration differences explain the baseline changes between Figure 3/Table A1 and Table 4?
  2. What does the specialized controller add over a general-purpose harness such as Claude Code or Codex, adapted to the same task with matched knowledge and computation?
5 comments
AuthorsOpenReview ↗

Responses to Reviewer vXHr (1)

Thank you for your valuable comments!

The baselines need reconciling. Kimi's premature-answer rate changes from 17.0% in Table A1 to 49.6% in Table 4 under the same timing definition. The configuration used to measure harness gains needs clarification.

What configuration differences explain the baseline changes between Figure 3/Table A1 and Table 4?

We sincerely appreciate your careful observation of the discrepancy between the two baselines. Your comment prompted us to re-examine the run logs, model-serving configurations, and metric definitions for both experiments. We confirmed that Table 4 reports a separate baseline run conducted to evaluate EVD-Harness, rather than reusing the results in Figure 3/Table A1, and the two experiments accessed Kimi-K2.6 through different model-serving sources. The earlier experiment used our self-hosted SGLang deployment, while the later experiment used an external API provider. We apologize for not making this distinction clear in the original manuscript, which made the results difficult to interpret and compare.

Both experiments define premature submission as the first submission occurring before the annotated evidence-sufficiency boundary t*; the metric definition is unchanged. Our further checks of behavior under the different serving configurations confirmed that the change in model-serving source was the main source of the baseline discrepancy. To address this issue, we will standardize the model-serving configuration for Kimi-K2.6 and align the relevant experimental results using the same model source in the revision. Thank you for helping us identify and correct this issue of experimental comparability.


Thank you for your comment!

Some gains come from changes outside evidence checking. On 200 cases, history and response-handling changes alone raise Opus accuracy from 37.5% to 75.0%. Separating these effects would clarify the method's contribution.

We appreciate this important question about attribution. We agree that separating the contributions of execution handling and evidence verification is essential to understanding the method. We have therefore examined Claude's improvement more closely, as its source was insufficiently explained in the original manuscript.

As shown in Appendix Table A15, Claude leaves 56.0% of cases unanswered under the original protocol. Shortening the action history and improving response handling reduces this rate to 12.5%, while accuracy rises from 37.5% to 75.0%. These results indicate that Claude already has substantial diagnostic capability, but its overall performance is depressed by the large number of cases without a valid submission. Improved execution handling allows more of its diagnostic judgments to become valid answers, accounting for the substantial gain. Your suggestion motivated us to examine the additional contributions of knowledge retrieval and evidence verification on top of this improvement.

Experimental setup and procedure. We added a series of component controls on the same 200 cases, revealing patient evidence one observation at a time in a fixed order. Models can access only the evidence revealed so far, with no access to future observations, the reference diagnosis, or t*. Every setting retains the complete revealed patient record. The first valid submission ends the case, and unanswered cases count as errors. We compare five settings:

  1. Natural: Retains the full action history and the original WAIT/ANSWER output and parsing protocol; the model decides when to submit.
  2. Short: Retains only the two most recent actions. Patient evidence, the output protocol, and parsing remain the same as in Natural.
  3. Direct: Adds the harness's structured output, format repair, and state updates to Short, and submits diagnosis proposals directly. Knowledge retrieval and evidence verification are not yet included.
  4. Direct + CDW: Adds frozen Contrastive Diagnostic Wiki (CDW) retrieval results to Direct and continues to submit proposals directly.
  5. Full: Keeps the same execution procedure and CDW inputs, and adds the Evidence Witness, Citation Check, and submission control. The system checks whether current patient evidence supports the proposal over competing diagnoses and whether its citations match the revealed record. It submits only after these checks pass; otherwise, it waits for the next observation.

Table R1. Component controls with a fixed model endpoint (%, n = 200)

ModelSettingAccuracyPrematureAt t*No answer
Qwen3.8-27BNatural59.531.541.03.5
Short63.535.040.02.5
Direct57.552.525.52.5
Direct + CDW70.018.065.01.5
Full76.513.572.53.5
Kimi-K2.6Natural61.519.536.05.0
Short65.533.037.53.5
Direct60.050.521.06.0
Direct + CDW71.020.059.51.5
Full77.017.567.02.5

As shown in Table R1, two findings emerge. First, the distinguishing knowledge supplied by CDW helps models use patient evidence more accurately. Adding CDW improves both Qwen3.8-27B's and Kimi-K2.6's accuracy by more than 10 percentage points, substantially reduces premature submissions, and concentrates more submissions at the first sufficient turn. This suggests that organizing knowledge around competing diagnoses helps models identify which patient findings support the proposed diagnosis.

Second, evidence verification further improves diagnostic outcomes under the same execution procedure and knowledge inputs. Full improves accuracy over Direct + CDW by 6.5 and 6.0 percentage points for the two models, respectively, while also increasing submissions at t*. Verification blocks some proposals with insufficient evidential support and allows the model to reassess them as further observations arrive. The resulting accuracy gains show that the additional correct diagnoses outweigh the costs of extra waiting and unanswered cases. Accuracy improves for all four models in the supplementary component experiment, supporting the consistency of this effect across models.

These results clarify the respective roles of execution handling, distinguishing knowledge, and evidence verification. We will present the ablations in this order in the revised manuscript and discuss Claude's execution-protocol effect alongside the new component results. Thank you again for helping us explain the contributions of the individual components more clearly.

AuthorsOpenReview ↗

Responses to Reviewer vXHr (2)

Thank you for your valuable comments!

The added value for practical agents is unclear. The baseline workflow restricts models to WAIT/ANSWER. A general-purpose harness (e.g., Claude Code or Codex), adapted to the same task with matched knowledge and computation, would provide a more informative comparison.

We appreciate your suggestion to include practical agent baselines. We agree that the original manuscript would benefit from a clearer account of the controlled task's practical relevance and a more complete comparison with general-purpose harnesses. Following your suggestion, we conducted additional experiments and clarified the connection between our benchmark design and diagnostic agents.

MedEVM examines a fundamental issue in diagnostic agents that final accuracy can obscure: whether the available patient evidence supports submitting a diagnosis at that moment. Benchmarks such as AgentClinic simulate a broader diagnostic process through patient interactions, examinations, and tool use, providing valuable environments for evaluating practical agents. In these tasks, outcomes jointly reflect information acquisition, tool use, and diagnosis submission. Final accuracy alone makes it difficult to distinguish a correct answer supported by sufficient evidence from a correct guess made before decisive evidence arrives.

To measure this issue directly, we introduce MedEVM as a progressive-evidence diagnosis benchmark. Given a fixed set of candidate diagnoses, the environment reveals one atomic observation from a case report per turn. The model can wait for the next observation or submit a diagnosis, which ends the trajectory. Each case is annotated with t*, the earliest turn at which the accumulated evidence distinguishes the reference diagnosis from all competing candidates. A fixed evidence stream ensures that different systems receive the same patient facts and obtain the same next observation when they wait, enabling a direct comparison of how they use evidence to decide when to submit.

This design reveals that even when additional challenges such as active questioning and tool selection are reduced, models can submit before the available medical facts are sufficient or continue waiting after they become sufficient. These behaviors affect diagnostic outcomes as well as timing. For example, the intervention in Section 5 blocks submission on originally premature cases until t*, then restores the model's own decision policy. It improves accuracy over the full benchmark by 5.3–44.9 percentage points, with unanswered cases still counted as errors. This analysis uses the annotated boundary to test the effect of waiting; EVD-Harness checks evidential support at runtime without access to t*. Our work thus provides both a process-level evaluation dimension for diagnostic agents and an implementation of evidence-based submission control that can complement richer interactive evaluations.

Following your suggestion, we also evaluated operational general-purpose harnesses. We ran Codex CLI 0.167 and OpenCode 2.1.4 on the same 200 cases, using the same corresponding models and evidence order, and supplying the same frozen CDW retrieval cards as EVD-Harness for each case. The general-purpose harnesses can manage their own state, invoke retrieval, and organize review, while accessing only patient evidence revealed so far. We set output-token budgets with reference to EVD-Harness's generation costs and allow additional capacity for tool calls, providing comparable generation budgets. Their autonomous history management and tool interfaces produce longer input prompts. We evaluate both diagnostic accuracy and submission timing; accuracy is computed over all cases, with unanswered cases counted as errors.

Table R2. General-purpose harness comparison (%, n = 200). Note: n = 200; DeepSeek-V4-Flash LLM-only: n = 199.

ModelMethodAccuracyPrematureAt t*No answer
Claude-Opus-4-8LLM-only37.511.527.056.0
EVD-Harness84.05.067.59.5
Codex74.013.556.016.5
OpenCode74.512.057.016.5
DeepSeek-V4-FlashLLM-only45.722.126.639.2
EVD-Harness67.518.570.01.0
Codex50.517.555.517.5
OpenCode49.57.051.522.5
Kimi-K2.6LLM-only61.519.536.05.0
EVD-Harness77.017.567.02.5
Codex68.52.053.029.0
OpenCode57.02.539.029.0

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.