- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces PharmEval, a pharmaceutical benchmark containing 6,174 questions across 9 task families and 47 subcategories. Beyond answer accuracy and explanation quality, it incorporates four diagnostic studies: web-based retrieval, compositional knowledge use, counterfactual sensitivity, and response modality variation. Experiments on 13 LLMs reveal that high accuracy does not necessarily translate into reliable knowledge use across different conditions.
Critical strengths
-
Evaluation beyond accuracy. The paper provides a systematic framework for examining LLM behavior from multiple perspectives. The controlled comparisons and item-level analyses reveal limitations that conventional accuracy metrics may overlook.
-
Broad coverage and extensive evaluation. The benchmark covers diverse pharmaceutical tasks and links questions to structured source records. Experiments across 13 models provide useful insights into their heterogeneous performance.
Critical weaknesses
- Limited attribution and generalizability of diagnostic findings.
The diagnostic experiments do not fully isolate the capabilities being investigated. In the retrieval study, performance differences may reflect retrieval quality rather than knowledge utilization. In the compositional study, some target questions already provide facts tested by separate atomic probes, and improvements under guided prompting may partly reflect general prompting effects rather than compositional reasoning. Moreover, the compositional and counterfactual studies use only 451 instances from one subcategory, limiting the generalizability of their findings.
While these experiments provide useful behavioral diagnostics, stronger controls or more cautious interpretations are needed to support capability-specific conclusions.
- Insufficient validation of open-ended and explanation-quality evaluation.
The substantial MCQ-to-open-ended accuracy drop (55.55–67.50 percentage points) requires further validation. Removing options may introduce multiple valid answers beyond the original reference. Although semantic evaluation is employed, it remains unclear whether alternative medically correct answers are consistently accepted.
Additionally, explanation quality is evaluated solely using GPT-4o-mini, without reported systematic human-expert validation. This limits confidence in the explanation-quality rankings and their interpretation.
Questions for authors
-
Retrieval: Have the authors evaluated retrieval quality or evidence sufficiency? Could an additional analysis using identical verified evidence help distinguish retrieval failures from knowledge-utilization failures?
-
Composition: What proportion of target questions already contain the tested atomic facts? Could the authors clarify how general prompting effects are distinguished from compositional reasoning improvements?
-
Open-ended evaluation: How are alternative medically valid answers handled? For instance, would Cladribine be accepted for the hairy cell leukemia question in Appendix C.4, where Pentostatin is the reference? Could the authors audit a representative sample of incorrectly judged responses?
-
Explanation quality: Could the authors provide expert agreement or independent-judge validation on a representative subset to establish the reliability of GPT-4o-mini's explanation scores?
2 comments
Response to Reviewer BiK1 (2/2)
Q3: Handling of alternative medically valid answers and audit of representative error cases. We agree that Cladribine should be credited as a correct answer, as the initial reference omitted this clinically valid alternative. We conducted two follow-up analyses:
- Expansion and re-scoring of open-ended reference answers: In open-ended evaluation, while the judge prompt supports recognizing semantically equivalent clinical expressions, relying on a single initial reference can still underestimate valid clinical alternatives. Therefore, we asked the expert team to review 288 stratified-sampled questions across task families to expand the open-ended reference sets by incorporating clinically equivalent drugs and valid alternatives, followed by re-scoring the outputs of five models using GPT-4o-mini. The re-scoring results are summarized below:
| Model | Original open-ended | Expanded open-ended |
|---|---|---|
| GPT-5.6 Terra | 13.54% | 24.31% |
| DeepSeek-V4.1-Flash | 15.97% | 24.65% |
| Qwen-3.8-Max | 14.24% | 23.61% |
| Gemma-4-E4B-it | 10.07% | 16.32% |
| Hulu-Med-7B | 9.03% | 14.58% |
- Qualitative audit of representative error cases: The expert team reviewed the incorrect answers in the sample and found that most model errors stemmed from incorrect understanding of drug interaction conditions, metabolic pathways, and logical reasoning under multiple constraints.
Q4: Multi-judge and expert validation of explanation quality on a representative subset. In response to this comment, we used DeepSeek-V4.1-Flash and Qwen3.8-Flash as additional LLM judges to score the full set of explanation samples, while the expert team scored 160 explanations (10% out of 1600 per model), stratified-sampled across task families. All raters followed the original four-dimensional evaluation criteria and were blinded to model identities during scoring. After pairing each evaluator's scores with GPT-4o-mini at the individual explanation level, all evaluation results showed strong correlations with GPT-4o-mini and preserved the overall model rankings, indicating that GPT-4o-mini provides robust judging results.
| Judge | Pearson r | Spearman ρ |
|---|---|---|
| DeepSeek-V4.1-Flash | 0.832 | 0.801 |
| Qwen3.8-Flash | 0.859 | 0.848 |
| Pharmaceutical Expert Team | 0.708 | 0.685 |
Response to Reviewer BiK1 (1/2)
We sincerely thank the reviewer for the helpful suggestions and address each point below using new retrieval controls, prompt ablations, and blind expert reviews.
Q1: Evaluation of retrieval quality and distinguishing retrieval failures from knowledge utilization failures. Regarding your suggestion to assess evidence sufficiency and use standardized evidence to separate retrieval defects from evidence-utilization defects, we conducted two additional experiments:
Q2: Proportion of target questions containing atomic facts and isolating the effect of generic prompting. Among the target questions, 12.64% of the stems contained at least one tested atomic fact. Keeping the original Direct and Guided settings, we introduced a generic step-by-step prompt as a control. Generic step-by-step prompting improved accuracy by an average of 7.05% compared with Direct, and Guided brought a further 12.20% gain over the generic step-by-step prompt. The former reflects the gains from general step-by-step reasoning, while the latter highlights the additional benefits of decomposing and answering atomic facts beforehand.