- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper talks about the medial diagnosis by LLM and categorize them into three categories: Foresight, hintsight and fullsight, especially compared the contribution of hintsight and fullsight (conventional medical QA). The authors tested several models based on these datasets: 320 cases: 140 AgentClinic MedQA, 80 dermatology, and 100 DiagnosisArena, and provide evaulation.
Critical strengths
- this paper categorize the medical diagnosis as fullsight, hintsight and foresight
- it tested the hypotheses on a set of dataset: 320 cases: 140 AgentClinic MedQA, 80 dermatology, and 100 DiagnosisArena.
- it proposed several, might be interesting evaluation metrics
Critical weaknesses
- this paper is extremely hard to follow. The authors could not ellaborate what they want to tell and how to prove that what they want to tell.
- the basic categorizations are not well explained, it is unclear what is the hintsight, what is foresight and what is fullsight. There are no examples to explain. The authors said that fullsight is the one most medicalQA goes showing all information, but it does not say what hintsight does (only provide information one by one, or choose some information to represent, or only present info that agent will request)? same for foresight. But the medical diagnosis is differential diagnosis that doctors collect some initial information, making some hypotheses and then collect more information to verify or deny some hypotheses. Also based on different diseases, different medical evidences will be requested, and different medical evidences weights differently based on different diseases. These are not discussed in the paper.
- the datasets are not clear whether they are medical QA or medical simulations, or covering which kinds of diseases, or they are rare diseases, it is all unclear.
- the conclusion is vague and hard to understand. No clinician evaluation.
Questions for authors
- please clearly definite the three categories, provide examples, showing why we have to care about these three categories
- clearly talk about your datasets, are they medical QA or simulated agentic environment,, covering which diseases, derived from where.
- explain your hypothesis, and being concrete on what you want to explain and provide more examples to support your arguments.