acceptodds
Withdrawn from ICLR 2027

From Hindsight to Foresight: Dissecting Clinical Evidence Acquisition in Large Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) often diagnose full sight, static clinical cases well yet struggle in foresight, dynamic diagnosis. Prior work has improved this limitation by quantifying the information models acquire and by developing stronger follow-up strategies to improve diagnostic accuracy. In contrast, we introduce a distinctive hindsight to systematically dissect why LLMs fail to acquire diagnostically critical evidence, tracing the failure from evidence identification, utilization and downstream diagnostic updating through not only diagnostic accuracy, but also evidence coverage, trajectory dynamics, and elicited probability and information entropy. Starting experiments from the breakdown at the level of capabilities, to the diagnosis of the pre-event state before the evidence arrives, and to the post-event state transition after the evidence arrives, our research has revealed the profound limitations and internal causes existing in the dynamic diagnostic process. At the same time, we have also conducted sufficient verification and generalization experiments to prove that these limitations are widespread. Together, these findings shift the central challenge from whether LLMs possess relevant clinical knowledge to whether they can identify, acquire, and effectively use the right evidence under clinical diagnosis and conduct a more detailed mechanism analysis to identify the exact location of the capability gap.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Clinical Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GFAdOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper talks about the medial diagnosis by LLM and categorize them into three categories: Foresight, hintsight and fullsight, especially compared the contribution of hintsight and fullsight (conventional medical QA). The authors tested several models based on these datasets: 320 cases: 140 AgentClinic MedQA, 80 dermatology, and 100 DiagnosisArena, and provide evaulation.

Critical strengths
  • this paper categorize the medical diagnosis as fullsight, hintsight and foresight
  • it tested the hypotheses on a set of dataset: 320 cases: 140 AgentClinic MedQA, 80 dermatology, and 100 DiagnosisArena.
  • it proposed several, might be interesting evaluation metrics
Critical weaknesses
  • this paper is extremely hard to follow. The authors could not ellaborate what they want to tell and how to prove that what they want to tell.
  • the basic categorizations are not well explained, it is unclear what is the hintsight, what is foresight and what is fullsight. There are no examples to explain. The authors said that fullsight is the one most medicalQA goes showing all information, but it does not say what hintsight does (only provide information one by one, or choose some information to represent, or only present info that agent will request)? same for foresight. But the medical diagnosis is differential diagnosis that doctors collect some initial information, making some hypotheses and then collect more information to verify or deny some hypotheses. Also based on different diseases, different medical evidences will be requested, and different medical evidences weights differently based on different diseases. These are not discussed in the paper.
  • the datasets are not clear whether they are medical QA or medical simulations, or covering which kinds of diseases, or they are rare diseases, it is all unclear.
  • the conclusion is vague and hard to understand. No clinician evaluation.
Questions for authors
  • please clearly definite the three categories, provide examples, showing why we have to care about these three categories
  • clearly talk about your datasets, are they medical QA or simulated agentic environment,, covering which diseases, derived from where.
  • explain your hypothesis, and being concrete on what you want to explain and provide more examples to support your arguments.
Reviewer_uiaPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies why LLMs struggle to acquire useful clinical evidence even when they can diagnose the same cases with complete information. It introduces Hindsight, which ranks available evidence after seeing the full case, with the reference diagnosis withheld in the strict protocol. The authors compare this reference with prospective evidence acquisition using fact-count matching, evidence reordering, and controlled evidence insertion. On 181 cases selected for correct diagnosis with complete information, a performance gap remains after matching the number of acquired facts. Further experiments examine whether the findings hold across models, data sources, and EviMed.

Critical strengths
  1. By separating requests, returned facts, and diagnostic outcomes on cases solvable with complete information, the paper provides a useful framework for studying evidence acquisition failures.
  2. Fact-count matching, same-set reranking, and evidence insertion test several explanations for the gap, with paired comparisons and robustness checks supporting the observations.
Critical weaknesses
  1. The lexical matcher can return blood pressure for a laboratory request (Appendix C.1), making it difficult to separate model acquisition failures from environment errors.
  2. The main text and appendices disagree on the backbone (GPT-5.5 versus DeepSeek V3.2) and evaluation horizon (six versus twelve requests), leaving the main results difficult to verify.
  3. EviMed, CDEG, and MT-InfoSeek already study acquisition bottlenecks, critical evidence, and query ordering, so the paper needs to explain which findings go beyond these works.
Questions for authors
  1. Which model performed acquisition, retrospective ranking, and diagnosis in the strict primary study?
  2. Were the 181 primary cases evaluated through twelve requests, and how do the results in Tables 1 and 2 reconcile with Appendices B.2 and D.2?
  3. What fraction of returned facts actually answer the issued requests, based on a clinical audit of the request-to-fact mappings?
  4. Does the acquisition gap persist with clinically validated or semantic matching instead of the current lexical matcher?
  5. Which new findings are established by the Hindsight analysis beyond EviMed, CDEG, and MT-InfoSeek?
Reviewer_qwZ3OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates why LLMs perform better when diagnosing complete clinical cases than when acquiring evidence through sequential requests. It introduces a retrospective Hindsight reference that selects diagnostically useful facts from a completed case and compares it with prospective Foresight acquisition. The analysis distinguishes request realization, acquired-evidence composition and ordering, and diagnostic responses after evidence arrival. For experiments, the main controlled study uses 181 cases selected for complete-case diagnostic correctness. Validation includes fact-count matching, same-set reranking, controlled insertion of missed evidence, selector perturbations, cross-model comparisons, and an external EviMed evaluation. The central finding is that prospective acquisition frequently fails to obtain retrospectively useful evidence, while supplying such evidence can improve diagnosis. Elicited probability distributions provide additional, explicitly qualified analyses of diagnostic updating.

Critical strengths
  1. The paper studies an important problem: whether LLMs can proactively acquire diagnostically useful information.

  2. It decomposes interactive diagnosis into meaningful stages, separating evidence acquisition, evidence quality, and diagnostic response.

  3. Controlled evidence-insertion experiments show that supplying certain missed facts can improve diagnostic correctness.

Critical weaknesses
  1. Limited evaluation scale. The primary study includes only 181 Fullsight-correct cases, with 140 retained for twelve-step continuation. By comparison, KnowGuard [1] evaluates on a benchmark comprising 3,061 cases across three datasets. Although the protocols differ, the relatively small, selected cohort warrants stronger justification and broader validation of the stage-wise conclusions.

  2. Unclear incremental contribution. The paper should better distinguish its findings from the established difficulty of interactive information acquisition. Hindsight has privileged access to realized evidence, so its advantage alone offers limited insight into avoidable acquisition failures.

  3. Simulator confounding. The lexical matcher can return clinically irrelevant facts, confounding model-level acquisition failures with interface errors. The menu experiment retains the same matcher and therefore does not eliminate this concern.

  4. Substantive protocol inconsistencies. Section 3.3 names GPT-5.5 as the strict-study model, while Appendix A.3 names DeepSeek V3.2. Tables 1–2 also report twelve-step results for 181 cases, despite Appendices B.2 and D.2 stating that their diagnostic trajectories end at six steps.

  5. Unreliable probabilistic readouts. Identical-evidence controls exhibit substantial probability and Top-1 variation, weakening the attribution of distributional changes to new evidence. The mechanistic claims require clearer separation of evidence effects from measurement variability.

  6. Presentation needs improvement. Key definitions and assumptions are buried in the appendix. The organization and table labeling make it difficult to track cohorts, protocols, and the conclusions of individual analyses.

[1] Dang et al. KnowGuard: Knowledge-Driven Abstention for Multi-Round Clinical Reasoning. ICLR 2026.

Questions for authors
  1. Can the authors reconcile the model identities and analysis horizons, particularly the twelve-step results reported for the 181-case cohort?

  2. Would the main findings persist with a clinically validated request-to-fact mapping instead of the lexical matcher?

  3. Do the probability and entropy effects remain distinguishable from unchanged-evidence variability under repeated, consistent readouts?

  4. What motivated the cohort sizes, and which stage-wise findings have been replicated across models under the same strict protocol?

  5. What new conclusions does each stage establish beyond the known acquisition gap, and what implications follow for prospective evidence-seeking strategies?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.