- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper propose a benchmark construction pipeline through progressive difficulty filtering followed by physician validation. Applying the pipeline on 47 medical benchmarks results in HealthCoreBench, which contains 13987 samples. Several LLMs are then evaluated on HealthCoreBench.
Critical strengths
- The paper focus on an important problem: the quality of benchmark is crucial for reliable LLM evaluation.
Critical weaknesses
- The novelty of the paper is limited. Difficulty filtering and expert validation are common strategies for constructing benchmarks, e.g. MMLU-Pro [1]. Though the authors try to formulate the pipeline as failure attribution, it is actually achieved through manual review. The authors should clarify the contribution beyond existing filtering and validation practices.
- The filtering process may introduce bias to the resulting benchmark. As only the most difficulty questions are preserved, the pipeline may change the distribution of the benchmark. The performance on HealthCoreBench could not reflect LLM's mastery of other clinical scenario. Moreover, questions that are filtered out by frontier proprietary models may still be difficult for less capable models. Therefore, more detailed analysis on the coverage would be valuable.
[1] MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark, NeurIPS 2024
Questions for authors
- The purpose of progressive filtering could be better explained. Why not just filtering with most strong models in tier 5? How many samples are solved by smaller models but are difficult for frontier closed-source models?
- The system or model used for automated pre-screening is not specified.
- Combining temperature=0 with reasoning mode may lead to performance decrease. Would using each model's suggested sampling parameters change the result?