- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents the Chinese evidence-based medicine benchmark MedEviBench, which includes 240 doctor questions covering 12 medical specialties and ten clinical intents. Given clinical questions and reference paragraphs, the model needs to select relevant evidence and provide cited answers. The benchmark is scored using nine weighted criteria, which are divided into three parts: evidence quality, clinical content, and answer presentation. The authors evaluated 18 models, used blind doctors to compare and verify the consistency between automatic scoring and doctor ranking, and analyzed the main failure modes of the models.
Critical strengths
-
The direction of the paper is a real but rarely explored topic – whether a model can answer with evidence rather than merely provide the correct answer, which is of great significance to evidence-based medicine.
-
There is a blind review by doctors for verification. The automatic ranking and the doctor ranking are generally consistent, indicating that this benchmark has a certain connection with clinical judgment.
Critical weaknesses
-
Although the overall ranking was consistent, when it came to each specific question, the differences among doctors were substantial, with a Krippendorff's alpha of 0.0402. Moreover, 38.16% of the comparison items had opposite votes. The paper also says that agreement of the overall ranking and disagreement on single comparisons exist at the same time. Model scores are added up from single question scores, so if single comparisons are this noisy, the small gaps between models may not be reliable.
-
The robustness check only covers two systems, but there are 18 main experiments. Moreover, there is overlap between the GPT series models used for scoring and the GPT series being evaluated, which may introduce self-preference bias.
Questions for authors
-
How do you plan to deal with the large disagreement on single comparisons?
-
Does the judge prefer GPT models, and how do control for it?
-
Can the robustness checks covers more systems, especially non-GPT and medical models?