acceptodds
Under review as a conference paper at ICLR 2027

PharmEval: A Diagnostic Benchmark for Pharmacological Knowledge and Reasoning of Large Language Models

Abstract

Existing pharmaceutical LLM benchmarks primarily focus on accuracy, making it difficult to determine whether model failures arise from insufficient knowledge, ineffective information use, or unstable reasoning. We introduce PharmEval, a pharmaceutical benchmark comprising 6,174 questions across 9 task families and 47 subcategories. Beyond conventional accuracy-based evaluation, PharmEval assesses explanation quality and incorporates four diagnostics of pharmaceutical knowledge use, including web-based knowledge retrieval, compositional knowledge use, counterfactual sensitivity, and response modality variation. These studies examine how external information and explicit premises support knowledge use, and whether models maintain or correctly update their judgments as conditions and response formats change. We evaluate a diverse set of LLMs and conduct an in-depth analysis of representative models. Our results show that benchmark accuracy alone does not fully reflect how reliably models use pharmaceutical knowledge. High accuracy can coexist with weak open-ended performance and failures to update judgments correctly when key conditions change. These findings suggest that a key challenge for current LLMs lies not only in acquiring relevant knowledge, but also in applying it consistently and reliably in context. By evaluating models from both performance and behavioral perspectives, PharmEval provides a more nuanced characterization of their capabilities and limitations in pharmaceutical reasoning.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Pharmaceutical Benchmark, Diagnostic Evaluation, Knowledge Reasoning, Model Behavior Analysis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BiK1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PharmEval, a pharmaceutical benchmark containing 6,174 questions across 9 task families and 47 subcategories. Beyond answer accuracy and explanation quality, it incorporates four diagnostic studies: web-based retrieval, compositional knowledge use, counterfactual sensitivity, and response modality variation. Experiments on 13 LLMs reveal that high accuracy does not necessarily translate into reliable knowledge use across different conditions.

Critical strengths
  1. Evaluation beyond accuracy. The paper provides a systematic framework for examining LLM behavior from multiple perspectives. The controlled comparisons and item-level analyses reveal limitations that conventional accuracy metrics may overlook.

  2. Broad coverage and extensive evaluation. The benchmark covers diverse pharmaceutical tasks and links questions to structured source records. Experiments across 13 models provide useful insights into their heterogeneous performance.

Critical weaknesses
  1. Limited attribution and generalizability of diagnostic findings.

The diagnostic experiments do not fully isolate the capabilities being investigated. In the retrieval study, performance differences may reflect retrieval quality rather than knowledge utilization. In the compositional study, some target questions already provide facts tested by separate atomic probes, and improvements under guided prompting may partly reflect general prompting effects rather than compositional reasoning. Moreover, the compositional and counterfactual studies use only 451 instances from one subcategory, limiting the generalizability of their findings.

While these experiments provide useful behavioral diagnostics, stronger controls or more cautious interpretations are needed to support capability-specific conclusions.

  1. Insufficient validation of open-ended and explanation-quality evaluation.

The substantial MCQ-to-open-ended accuracy drop (55.55–67.50 percentage points) requires further validation. Removing options may introduce multiple valid answers beyond the original reference. Although semantic evaluation is employed, it remains unclear whether alternative medically correct answers are consistently accepted.

Additionally, explanation quality is evaluated solely using GPT-4o-mini, without reported systematic human-expert validation. This limits confidence in the explanation-quality rankings and their interpretation.

Questions for authors
  1. Retrieval: Have the authors evaluated retrieval quality or evidence sufficiency? Could an additional analysis using identical verified evidence help distinguish retrieval failures from knowledge-utilization failures?

  2. Composition: What proportion of target questions already contain the tested atomic facts? Could the authors clarify how general prompting effects are distinguished from compositional reasoning improvements?

  3. Open-ended evaluation: How are alternative medically valid answers handled? For instance, would Cladribine be accepted for the hairy cell leukemia question in Appendix C.4, where Pentostatin is the reference? Could the authors audit a representative sample of incorrectly judged responses?

  4. Explanation quality: Could the authors provide expert agreement or independent-judge validation on a representative subset to establish the reliability of GPT-4o-mini's explanation scores?

2 comments
AuthorsOpenReview ↗

Response to Reviewer BiK1 (1/2)

We sincerely thank the reviewer for the helpful suggestions and address each point below using new retrieval controls, prompt ablations, and blind expert reviews.

Q1: Evaluation of retrieval quality and distinguishing retrieval failures from knowledge utilization failures. Regarding your suggestion to assess evidence sufficiency and use standardized evidence to separate retrieval defects from evidence-utilization defects, we conducted two additional experiments:

  • Blind evaluation of retrieved snippet quality: We stratified-sampled about 5% of the 6,174 questions across task families. Blinded to model identities and outputs, three pharmaceutical experts assigned binary relevance and sufficiency scores to each retrieved snippet, based on whether it contained the target property, relation, or rule, and whether it, together with the question stem, provided complete and accurate supporting evidence. Each metric is normalized by the sample size of its respective task category and averaged across all models:
Task FamilyRelevance RateSufficiency Rate
PhysChem Props.92.82%55.13%
Drug Relations90.77%50.51%
Toxicity & Safety88.46%46.35%
Drug Comb.91.28%44.36%
Ontology & Norm.89.49%52.82%
PhysChem Rules83.33%35.38%
Indication Reasoning76.15%30.96%
Drug Reposition.65.64%33.85%
Metab. & Clinical77.38%46.15%
  • Controlled comparison and error analysis using identical expert-verified evidence: We independently stratified-sampled another 100 questions and tested all 13 models. After substituting self-retrieved snippets with unified expert-verified evidence, the number of correct pairs rose from 985/1,300 (75.8%) to 1,093/1,300 (84.1%). Among 315 self-retrieval errors, 203 lacked sufficient evidence (124 fixed with unified evidence), while 112 already had sufficient evidence (95 remained incorrect). This indicates that errors stem from both insufficient retrieved evidence and improper utilization of available evidence. An audit of 1,315 pairs that were correct under closed-book settings but failed with web retrieval further revealed only 74 cases (5.63%) involving conflicts between web snapshots and benchmark rules.

Q2: Proportion of target questions containing atomic facts and isolating the effect of generic prompting. Among the target questions, 12.64% of the stems contained at least one tested atomic fact. Keeping the original Direct and Guided settings, we introduced a generic step-by-step prompt as a control. Generic step-by-step prompting improved accuracy by an average of 7.05% compared with Direct, and Guided brought a further 12.20% gain over the generic step-by-step prompt. The former reflects the gains from general step-by-step reasoning, while the latter highlights the additional benefits of decomposing and answering atomic facts beforehand.

ModelDirectGeneric Step-by-StepGuided
GPT-5.6 Terra78.27%86.47%98.89%
DeepSeek-V4.1-Flash72.28%82.04%99.11%
Qwen-3.8-Max74.06%84.26%98.67%
Gemma-4-E4B-it68.96%74.72%89.36%
Hulu-Med-7B47.89%49.22%51.66%
Average68.29%75.34%87.54%
AuthorsOpenReview ↗

Response to Reviewer BiK1 (2/2)

Q3: Handling of alternative medically valid answers and audit of representative error cases. We agree that Cladribine should be credited as a correct answer, as the initial reference omitted this clinically valid alternative. We conducted two follow-up analyses:

  • Expansion and re-scoring of open-ended reference answers: In open-ended evaluation, while the judge prompt supports recognizing semantically equivalent clinical expressions, relying on a single initial reference can still underestimate valid clinical alternatives. Therefore, we asked the expert team to review 288 stratified-sampled questions across task families to expand the open-ended reference sets by incorporating clinically equivalent drugs and valid alternatives, followed by re-scoring the outputs of five models using GPT-4o-mini. The re-scoring results are summarized below:
ModelOriginal open-endedExpanded open-ended
GPT-5.6 Terra13.54%24.31%
DeepSeek-V4.1-Flash15.97%24.65%
Qwen-3.8-Max14.24%23.61%
Gemma-4-E4B-it10.07%16.32%
Hulu-Med-7B9.03%14.58%
  • Qualitative audit of representative error cases: The expert team reviewed the incorrect answers in the sample and found that most model errors stemmed from incorrect understanding of drug interaction conditions, metabolic pathways, and logical reasoning under multiple constraints.

Q4: Multi-judge and expert validation of explanation quality on a representative subset. In response to this comment, we used DeepSeek-V4.1-Flash and Qwen3.8-Flash as additional LLM judges to score the full set of explanation samples, while the expert team scored 160 explanations (10% out of 1600 per model), stratified-sampled across task families. All raters followed the original four-dimensional evaluation criteria and were blinded to model identities during scoring. After pairing each evaluator's scores with GPT-4o-mini at the individual explanation level, all evaluation results showed strong correlations with GPT-4o-mini and preserved the overall model rankings, indicating that GPT-4o-mini provides robust judging results.

JudgePearson rSpearman ρ
DeepSeek-V4.1-Flash0.8320.801
Qwen3.8-Flash0.8590.848
Pharmaceutical Expert Team0.7080.685
Reviewer_DoTmOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PharmEval, a pharmaceutical benchmark for evaluating the pharmaceutical knowledge and reasoning capabilities of LLMs. Beyond conventional answer accuracy, the authors also evaluate explanation quality and introduce four diagnostic studies: (1) web-based knowledge retrieval, (2) compositional knowledge use, (3) counterfactual sensitivity, and (4) response modality variation.

Critical strengths

The paper provides a unified evaluation that examines pharmaceutical knowledge and reasoning from multiple perspectives (IR with external knowledge, compositional reasoning, counterfactual sensitivity, and response modality (mcq vs open-ended questions) robustness. This goes beyond conventional accuracy-based benchmarks and provides a more detailed understanding of model behaviour under different conditions.

Critical weaknesses

While I do find the results interesting, I also have some concerns regarding the validity of the experimental settings.

  1. The paper relies on GPT-4o-mini to evaluate open-ended correctness and explanation quality. Using multiple LLM judges and reporting their agreement would strengthen the reliability of the results. If possible, agreement between human evaluators and LLM judges should also be reported.
  2. Validation of intervention and open-ended questions: Manual or expert validation of the constructed intervention (are they intervened correctly?) and open-ended questions appears necessary to establish their correctness and validity. This is particularly important given the extremely low open-ended accuracy reported in the experiments, which raises concerns about potential ambiguity.
  3. The finding that web-based knowledge retrieval has substantially different effects across task families is interesting. However, the paper does not sufficiently investigate the factors underlying these differences. For example, retrieval difficulty and the quality or relevance of retrieved information may differ across task families. Some tasks may be easier to retrieve relevant evidence for, while others may yield less relevant or lower-quality search results.
Questions for authors

As I discussed in the weaknesses: a) Could the authors evaluate the agreement among multiple LLM judges and, if possible, between LLM judges and human evaluators? b) Were the intervention and MCQ-to-open-ended questions manually reviewed or validated by domain experts? c) Could the authors further analyse retrieval difficulty and the quality or relevance of retrieved information across task families?

Also, on paraphrase generation, how were the paraphrased questions generated? Were LLMs used, and how was semantic equivalence to the original questions verified?

2 comments
AuthorsOpenReview ↗

Response to Reviewer DoTm

We thank the reviewer for the detailed suggestions. We added blind evaluations, expert checks, and retrieval analyses, and addressed each point below.

Q1: Consistency between different LLM judges and human expert scoring. We introduced DeepSeek-V4.1-Flash, Qwen3.8-Flash, and a three-member pharmaceutical expert team to conduct three independent blind evaluations:

  • Native open-ended question correctness: Across 210 native open-ended questions stratified and sampled by task family, each judge independently determined correctness. Consistency with GPT-4o-mini is as follows:
JudgeConsistencyCohen’s κ
DeepSeek-V4.1-Flash88.32%0.766
Qwen3.8-Flash87.77%0.755
Pharmaceutical Experts84.51%0.692
  • Explanation quality: DeepSeek-V4.1-Flash and Qwen3.8-Flash re-scored the full explanation evaluation set, while the expert team scored 160 explanations (10% out of 1600 per model), stratified-sampled across task families. By pairing each explanation's overall score with its GPT-4o-mini score, the correlations at the individual explanation level show high consistency in overall model rankings.
JudgePearson rSpearman ρ
DeepSeek-V4.1-Flash0.8320.801
Qwen3.8-Flash0.8590.848
Pharmaceutical Experts0.7080.685

b) Were the intervention and MCQ-to-open-ended questions manually reviewed or validated by domain experts?

Q2: Expert verification of counterfactual intervention and converted open-ended questions. We initially involved experts in designing templates and defining rules for counterfactual intervention and MCQ‑derived open‑ended items. In response to this comment, we asked the expert team to further conduct a blind review of both question types. For counterfactual intervention items, we sampled 2 questions from each of the 48 original sentence templates, totaling 96 items and covering all 6 pharmacokinetic endpoints. Without reference answers, experts’ judgments on outcomes, change directions, and single-enzyme activity perturbations were consistent with the reference answers. For converted open-ended questions, experts reviewed 288 stratified-sampled items with options masked to verify stem clarity and supplement valid reference answers, confirming the quality of the converted items.

Q3: Retrieval difficulty, evidence quality, and relevance analysis across task families. We stratified-sampled about 5% of the 6,174 questions across task families. Blinded to model identities and answer conditions, the expert team assigned 0 or 1 annotations to all retrieved passages for each question. Passages addressing the target property, relation, or rule received a relevance score of 1, while those combining with the question stem to provide sufficient, correct, and complete evidence for answering received a sufficiency score of 1, and 0 otherwise. All rates were computed per model using the sample size of each corresponding task family as the denominator and then averaged across models.

Task FamilyRelevance RateSufficiency Rate
PhysChem Props.92.82%55.13%
Drug Relations90.77%50.51%
Toxicity & Safety88.46%46.35%
Drug Comb.91.28%44.36%
Ontology & Norm.89.49%52.82%
PhysChem Rules83.33%35.38%
Indication Reasoning76.15%30.96%
Drug Reposition.65.64%33.85%
Metab. & Clinical77.38%46.15%

Q4: Generation and semantic equivalence verification of synonymous paraphrase questions. We used GPT-6 Sol to generate one paraphrased version for each of the 451 counterfactual source questions under strict constraints: preserving the drug and enzyme names, the direction of enzyme activity change, pharmacokinetic assumptions, prediction endpoints, and answering requirements, while retaining the original reference answers. We sampled 2 pairs from each of the 48 original sentence templates, totaling 96 pairs and covering all 6 prediction endpoints. Through blind review by pharmaceutical experts comparing the queried endpoints, change directions, and constraints, 93 pairs directly satisfied semantic equivalence.

Thank you for the additional analyses and detailed explanations. I do believe they strengthen the paper. I will maintain the positive score.

Best,

Reviewer DoTm

Reviewer_b3rROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper aims to distinguish pharmaceutical answer accuracy from reliable use of pharmaceutical knowledge. It introduces PharmEval, a source-linked benchmark of 6,174 questions across nine task families and 47 subcategories. Beyond answer correctness and explanation scoring, it evaluates search access, premise organization and provision, sensitivity to changed conditions, and response-format variation. Thirteen models undergo the main and retrieval evaluations, with five receiving additional diagnostics. The reported results show task-dependent retrieval effects, model-dependent benefits from explicit premises, failures under enzyme-activity reversals, and large multiple-choice-to-open-ended performance gaps.

Critical strengths
  1. The motivation appropriately moves beyond an aggregate accuracy leaderboard.

  2. The paper retains paired outcomes and provides extensive source-level examples.

Critical weaknesses
  1. Confounded response-format comparison. Removing options can make questions under-specified or admit multiple valid answers, while unrelated distractors simplify MCQs. The reported format gap therefore does not cleanly measure answer-cue dependence.

  2. Unclear construction and validation. The roles of templates, LLM agents, and experts are insufficiently documented, particularly how counterfactual reference answers are generated and independently validated.

  3. Limited diagnostic validity. Composition and counterfactual tests cover only 451 enzyme-activity questions, often supplying the relevant premises. This narrow design does not cleanly separate factual knowledge from compositional reasoning.

  4. Unvalidated evaluation. GPT-4o-mini alone scores open-ended answers and explanations, without reported expert agreement or an independent-judge audit. Missing uncertainty estimates further weaken comparisons.

  5. Unclear novelty and trade-offs. The contribution is primarily integrative. Comparisons emphasize coverage but omit sample size, validation quality, and task realism, leaving pharmaceutical-specific advances and trade-offs unclear.

  6. Presentation needs clarification. The paper should better distinguish native and converted open-ended questions and clarify premise conditions, intervention diagrams, and position range.

Questions for authors
  1. How are candidate-dependent MCQs converted into self-contained open-ended questions, and how are references expanded to accept alternative valid answers? Could the format comparison be repeated on an expert-verified subset with plausible, category-matched distractors?

  2. What roles do templates, deterministic programs, LLM agents, and experts play in construction? In particular, how are counterfactual reference answers generated and independently validated? Please also report distinct entity and template counts.

  3. What operational criteria distinguish Rule from Composition? Could the composition study compare targets with and without the drug–enzyme premise and include balanced increase, decrease, and no-change conditions?

  4. Could blinded experts and an independent judge audit open-ended correctness and explanation quality, especially high-scoring incorrect explanations and cases driving the format gap? How would rescoring affect the conclusions?

  5. Could the benchmark comparison include sample size, expert validation, task realism, and evaluation cost, and clarify the specific methodological contribution and trade-offs of PharmEval?

  6. How often do retrieval-induced errors reflect conflicts with the database snapshot or benchmark-specific rules rather than reasoning failures? Please provide an audit, paired uncertainty estimates accounting for shared entities/templates, and unchanged-input controls for permutation variability.

4 comments
AuthorsOpenReview ↗

Response to Reviewer b3rR (1/2)

We thank the reviewer for recognizing our research motivation, experimental design, and rich illustrative examples. We addressed the questions below.

Q1: Conversion of MCQs to open-ended questions and reference expansion. We initially converted MCQs into open-ended questions by removing options and using the original correct options as references. To resolve this concern, we invited three pharmaceutical experts to revise a 10% stratified sample of 288 out of 2,877 items. For open-ended items, they ensured the questions were self-contained and added valid alternatives. For MCQs, they retained original questions and correct options, and replaced weak options with plausible, category-matched distractors. We then used GPT-4o-mini to regrade the original open-ended answers and reevaluated five models on the revised MCQs. A substantial performance gap persisted across all models:

ModelOriginal MCQRevised MCQOriginal open-endedExpanded open-ended
GPT-5.6 Terra78.82%70.14%13.54%24.31%
DeepSeek-V4.1-Flash79.17%73.61%15.97%24.65%
Qwen-3.8-Max77.43%71.18%14.24%23.61%
Gemma-4-E4B-it67.36%59.03%10.07%16.32%
Hulu-Med-7B65.28%55.90%9.03%14.58%

Q2: Roles in dataset construction and building counterfactual items. a) Roles in dataset construction. Templates defined question structures, while deterministic programs filled in DrugMAP records and derived answers. LLMs only paraphrased text without changing meaning, and experts verified the results. b) Counterfactual answer generation and validation. Deterministic programs generated reference answers by inverting only main metabolic enzyme activity. Experts then reviewed 96 randomly sampled items from the 451 counterfactuals (20%), covering all 48 source templates. 96.9% passed semantic equivalence, while 2 items omitting conditions and 1 item leaking the conclusion were revised and verified again. c) Distinct entity and template counts. The benchmark contains 4,952 distinct entity groups and 86 template groups.

Q3: Criteria for Rule and Composition and premise controls**.** Rule tasks provide needed facts and apply one clear rule, while Composition tasks link multiple facts across steps to reach an answer. Following your suggestion, we conducted a premise control study. We sampled 45 source questions from the 451 counterfactual items (10%). For each question, we built six paired versions across decreased, increased, and unchanged enzyme activity, both with and without stating that the enzyme is the main metabolic enzyme. Experts verified all reference answers. Providing this drug enzyme premise improved accuracy for all five models in every condition.

Q3: Criteria for Rule and Composition and premise controls**.** Rule tasks provide needed facts and apply one clear rule, while Composition tasks link multiple facts across steps to reach an answer. Following your suggestion, we conducted a premise control study. We sampled 45 source questions from the 451 counterfactual items (10%). For each question, we built six paired versions across decreased, increased, and unchanged enzyme activity, both with and without stating that the enzyme is the main metabolic enzyme. Experts verified all reference answers. Providing this drug enzyme premise improved accuracy for all five models in every condition.

ModelDecreased w/ premiseDecreased w/o premiseIncreased w/ premiseIncreased w/o premiseUnchanged w/ premiseUnchanged w/o premise
GPT-5.6 Terra10077.7897.7866.6793.3360
DeepSeek-V4.1-Flash10073.331006095.5651.11
Qwen-3.8-Max10073.3397.7862.2291.1153.33
Gemma-4-E4B-it93.3368.8984.4453.3375.5640
Hulu-Med-7B48.8946.6742.2237.7833.3328.89
AuthorsOpenReview ↗

Response to Reviewer b3rR (2/2)

Q4: Blind audit of open-ended correctness and explanation quality. Following your suggestion, we conducted a blind audit using DeepSeek-V4.1-Flash and Qwen3.8-Flash on the full set, alongside human experts on a stratified sample of 210 native open-ended questions and 160 explanations. Both the model judges and experts showed consistency with GPT-4o-mini on answer correctness and explanation scores. Reviewing high-scoring incorrect answers showed that some gave detailed text without key evidence for the final conclusion, which inflated their scores. Cases driving the format gap confirmed that errors came from recall and reasoning failures without option cues. Overall, rescoring by the model judges or experts did not change our model rankings or conclusions.

Response level consistency on native open-ended questions:

JudgeConsistencyCohen's κ
DeepSeek-V4.1-Flash88.32%0.766
Qwen3.8-Flash87.77%0.755
Pharmaceutical expert team84.51%0.692

Paired correlation of per-explanation overall scores:

JudgePearson rSpearman ρ
DeepSeek-V4.1-Flash0.8320.801
Qwen3.8-Flash0.8590.848
Pharmaceutical expert team0.7080.685

Q5: Benchmark comparison dimensions and methodological trade-offs. Following your suggestion, we added sample sizes and expert validation tiers (full, sampled, or none) to the main comparison table, and placed the task realism analysis in Appendix K. About 2/3 of the evaluation uses automated grading, which keeps costs low and allows flexible module selection. PharmEval's main contribution is using paired question variants to pinpoint where models fail. By linking base knowledge with controlled question changes, it separates errors in evidence gathering, premise use, condition updates, and output format. While isolating these steps trades some degree of clinical realism, this deliberate design choice is essential for achieving fine-grained diagnostic clarity.

Q6: Audit of retrieval-induced errors and uncertainty controls. We audited all 1,315 retrieval-induced errors using GPT-6 Astra. Only 5.63% came from database snapshot or benchmark rule conflicts, confirming that most errors reflect genuine reasoning or evidence integration failures. Then we computed paired uncertainty estimates across 6,174 items using a two-way cluster bootstrap grouped by shared entities and templates. All five models still showed positive retrieval gains with 95% confidence intervals above zero. Finally, we tested unchanged inputs on 288 items (10%) to measure natural variability under original and permuted option orders.

Web retrieval accuracy and paired 95% confidence intervals:

ModelClosed-book (%)Search-enabled (%)Two-way cluster 95% CI
GPT-5.6 Terra77.4283.25[4.10, 7.56]
DeepSeek-V4.1-Flash77.3679.77[0.95, 3.87]
Qwen-3.8-Max74.6278.77[2.50, 5.80]
Gemma-4-E4B-it62.7863.51[0.08, 1.38]
Hulu-Med-7B52.3853.94[0.30, 2.80]

Answer flip rates under original and permuted option orders:

ModelOriginalPermuted
GPT-5.6 Terra1.39%4.86%
DeepSeek-V4.1-Flash2.43%6.25%
Qwen-3.8-Max2.08%5.90%
Gemma-4-E4B-it3.13%8.33%
Hulu-Med-7B4.17%8.68%
Reviewer_b3rROpenReview ↗

response to rebuttal

Thank you very much for your carefully written responses. The premise-control experiment, additional model judges and expert audit, and paired uncertainty estimates address several of my concerns. In particular, the comparison with and without the drug–enzyme premise provides useful evidence about how models use that information on the enzyme-activity tasks.

My main remaining concern is the interpretation of the response-format gap. As I understand the rebuttal, the models answered the revised MCQs, whereas their original open-ended responses were regraded against expanded references. Expanding the references can correct grading errors, but it cannot show how a model would have answered a clarified question. This matters for candidate-dependent items such as Examples B.37 and B.39, where removing the options also removes the information needed to identify the intended entity. On the 288-item subset, the reported GPT gap narrows from 65.28 to 45.83 percentage points. The remaining difference is substantial, but the comparison does not establish how much of it reflects reliance on answer cues rather than ambiguity introduced by conversion. The reported human–judge agreement is encouraging, though it is quantified for native rather than converted open-ended questions.

Overall, the rebuttal strengthens my confidence in several findings, particularly the premise-control result. However, because the format comparison is central to the paper’s diagnostic contribution and its interpretation remains unresolved, I am maintaining my weak-reject rating.

AuthorsOpenReview ↗

Follow-up Response to Reviewer b3rR

Thank you for the follow-up. For candidate-dependent items (such as Examples B.37 and B.39), we place the target entity into the stem and query its entity category directly, as each item is built from a structured source record:

  • Example B.37: Converted to “What entity category does ‘Mitochondrial complex 1 deficiency, nuclear type 36’ belong to?” → Reference: “Disease or clinical condition.”
  • Example B.39: Converted to “What entity category does ‘PSMA-11 Ga-68’ belong to?” → Reference: “Drug or therapeutic product.”

While the phrasing shifts from multi-candidate selection to entity classification, both formats still test the underlying entity-category knowledge of the same source record. For other MCQs whose stems already specify the target entity and relation, removing the options directly yields a self-contained open-ended question.

To verify that the format gap is not driven by reformulated items, we split the 2,877 items into two subsets:

  • Direct option-removal subset (91.6%): The MCQ–open-ended accuracy gap across the five models remains 58.65–68.21% (MCQ: 67.07–81.18%, open-ended: 8.42–15.21%).
  • Reformulated recognition subset (8.4%): The gap is 29.88–35.27% (MCQ: 45.64–60.17%, open-ended: 15.77–26.97%).

These results confirm that the substantial format gap persists even when restricting the comparison strictly to items whose stems are unchanged. The gap is larger in the direct option-removal subset for every model, indicating that the overall pattern is not driven by the reformulated recognition items.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.