acceptodds
Under review as a conference paper at ICLR 2027

PedMed-Bench: A Multi-turn Benchmark for Large Language Models on Pediatric Medication Effectiveness and Safety

Abstract

Evaluating Medical Large Language Models (LLMs) in complex clinical scenarios has gained increasing attention. Yet most existing benchmarks emphasize general or adult-centered medicine, making them poorly aligned with pediatric care, where age-dependent physiology, longitudinal disease evolution, and medication safety require specialized reasoning. Specifically, current evaluation systems regarding pediatric care are still facing severe data scarcity challenges, thereby hindering the evaluation and development of pediatric Medical LLMs. Motivated by this, we introduce a novel benchmark (PedMed-Bench) to address the above challenges. Further, we conduct a systematic evaluation involving numerous representative LLMs on PedMed-Bench. The results suggest that contemporary LLMs still struggle to accurately predict adverse drug events (ADEs) and suspected drugs. Even though models equipped with reasoning capabilities demonstrate notable enhancements, their absolute performance remains constrained, with the highest ADE extraction F1 score reaching only 12.89%. To the best of our knowledge, PedMed-Bench is the first fine-grained, event-driven benchmark for real-world pediatric medication evaluation. The benchmark is released in both Chinese and English, with human-verified translation validated for cross-lingual equivalence. We hope it can serve as a helpful resource, contributing to the ongoing efforts in developing more reliable clinical AI tools for pediatric healthcare.

Primary area:
datasets and benchmarks
Keywords:
Pediatric Medication, Benchmark, Medication Safety

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_oKszOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduce a benchmark, PedMed-Bench, to evaluate large language models on pediatric medication effectiveness and safety. The benchmark consists of longitudinal inpatient electronic health records of 316 patients from two tertiary hospitals in China. The data records cover diagnosis, symptoms, therapeutic response, adverse drug events, suspected medications, and causality assessment. The paper evaluated 14 LLMs across multiple metrics.

Critical strengths
  1. The benchmark presents relevant clinical settings and provides a potentially valuable dataset.
  2. The authors made substantial annotation efforts, which provide a rather solid foundation for a benchmark.
Critical weaknesses
  1. The uniqueness of a pediatric benchmark is not clearly presented in the paper. It is not clear what is specially designed or prepared for the pediatric medication in the problem formulation, data processing, and evaluation.
  2. The problem is not well formulated at the moment. It defines the problem as sequential decision evaluation tasks. It predict outcomes within an observation window. However, the target mixes symptoms at window onset, severity of illness under treatment, improvement across windows, future adverse drug effects, drug-event causality, etc. The paper currently fails to identify explicitly which target concern current state and which concern future outcomes. The ambiguity affects the interpretation of low scores.
Questions for authors
  1. Can you provide an evaluation example of a complete treatment episode?
  2. How are those metrics calculated in details?
  3. Can you illustrate in details on how different LLM model respond to a single patient's clinical records? It is also better to highlight the key difference as well, especially for the pediatric-related ones.
  4. I still see substantial contents of the recording are Chinese when I access the dataset on October 6th, including family_history, history_of_present_illness, clinical_trajectory, etc.
  5. The readme file shows it is previously submitted to NeurIPS 2026. Is it possible to disclose how you have improved the benchmark after the submission?
  6. What expert-validated evidence demonstrates that the medication interventions have the expected outcomes?
  7. Any evidence can demonstrate a meaningful portion of the benchmark specifically requires pediatric reasoning?
  8. Several inconsistencies are noticed in the manuscript. For instance, table 2 reports an ADE-severity score of -10.00, while the metric is defined as accuracy. Please check all metric definition, evaluation protocols and results to ensure fair comparison. These affect the accuracy of interpretation.
10 comments

We thank the reviewer for recognizing the clinical relevance of our dataset and annotation work. We have revised the manuscript to address the concerns about pediatric specificity, task formulation, evaluation pipelines, reference labels, and reporting inconsistencies. All changes are colored in blue.

Q1. What is specifically designed for pediatrics in the formulation, data, and evaluation?

A: Three characteristics of pediatric medication motivate the benchmark: age-related developmental variation in drug disposition and response, weight-dependent dosing, and frequent off-label use amid limited pediatric evidence. Each is carried into one of the three components the reviewer asks about.

Formulation. Each hospitalization is a sequence of treatment nodes, with inference conditioned on age, body weight, medication regimen, and clinical history; at each node the model assesses the current state and predicts subsequent treatment response and ADEs. The three characteristics map directly onto this design: developmental stage governs how presentations and drug responses are interpreted, body weight governs how administered doses are interpreted, and prior treatment observations supply patient-specific evidence where pediatric prescribing guidance is thin, including off-label use.

Data. 316 patients aged 28 days to 18 years, 2,515 treatment nodes, and 186 distinct diagnoses, including pediatric-specific conditions such as Kawasaki disease, congenital heart disease, and infantile idiopathic thrombocytopenia. The ADE spectrum likewise contains age-specific categories such as neonatal hyperbilirubinemia and infantile diarrhea — concrete pediatric content, not general medical question answering.

Evaluation. Appendix C.1 stratifies every metric across five developmental groups (infancy, toddlerhood, preschool, school age, adolescence) to test whether aggregate scores conceal age-related differences, so performance is reported against the developmental variation that motivates the benchmark, not pooled over it.

Given the main-text length limit, we have added Appendix A.8 ("Pediatric-Specific Design") mapping these challenges to our formulation, data, and analyses.

Q2. The task targets mix current state (symptoms, illness severity) with future outcomes (cross-window improvement, future ADEs), and the relationship between "sequential decision evaluation" and within-window outcome prediction is unclear, making low scores ambiguous to interpret.

A: The reviewer correctly observes that our tasks span different temporal targets. This is a deliberate choice, made with pediatric physicians, to reproduce the cognitive load a clinician carries at each treatment decision, and it follows established distinctions: clinical prediction modelling separates diagnostic prediction (present state) from prognostic prediction (future outcomes), whose target definition, validation, and interpretation must be handled separately [1]; situation-awareness theory describes three ascending levels — perception of current elements, comprehension of the present state, projection of future states [2]; and clinical time-series benchmarks separate tasks on the same grounds [3]. We therefore group our metrics by temporal target:

Class A — current state (at tt). Diagnosis and symptom extraction/severity, scored against the progress note and physical examination at that timestamp, measure immediate integration of clinical evidence.

Class B — within-window outcome (tt to t+Δt+\Delta). SI, GI, and EI measure prospective prediction of the course under the recorded regimen; ADE occurrence and extraction measure prospective identification of adverse events. Both are scored against how the course actually evolved.

Class C — retrospective causal inference. Suspected drug and causality measure attribution after an ADE has occurred. Experts first determine under WHO-UMC and RUCAM whether a suspected ADE occurred within 8 h of the node's regimen; if so, the preceding 8 h are reviewed retrospectively and annotated independently.

"Sequential decision evaluation" denotes this complete loop executed across successive treatment nodes, not a prediction at a single timestamp.

To remove the ambiguity in reading low scores, the appendix now analyses capability by task class rather than as one undifferentiated verdict, with per-class input-truncation and time-anchor diagrams making each class's temporal semantics explicit.

[1] Steyerberg E W. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. Springer, 2009.

[2] Endsley M R. Toward a theory of situation awareness in dynamic systems. In: Situational Awareness, Routledge, 2017: 9–42.

[3] Harutyunyan H, Khachatrian H, Kale D C, et al. Multitask learning and benchmarking with clinical time series data. Scientific Data, 2019, 6(1): 96.

Q3: Can you provide a complete evaluation example of a treatment process?

A: Below is a de-identified real case (830becd9) giving the task-by-task evaluation at Node 2: gold standard, raw outputs from three models, and how each metric is scored. We have added this case study as Appendix C.7 and Table 21.

Case. A 3.5-year-old male (13.8 kg), hospitalized 10 days across 8 treatment nodes, discharge diagnosis Kawasaki disease with drug-induced liver injury. Chief complaint was "fever for 5 days", with conjunctival congestion during febrile spikes. Kawasaki disease was established on Day 4 (2024-06-15 09:10), initiating IVIG pulse therapy.

Node 2 (2024-06-12 19:53, "Initial Course Record") returned CRP 84.35 mg/L↑\text{CRP } 84.35\text{ mg/L}\uparrow, PCT 12.1 ng/mL\text{PCT } 12.1\text{ ng/mL}, negative Mycoplasma/Chlamydia and viral antibodies, and an unremarkable chest CT. The gold standard requires the model to prospectively predict the ADE emerging in the following window — "Liver injury (Severe)" — whose evidence (ALT 94 U/L↑\text{ALT } 94\text{ U/L}\uparrow, γ-GT 180.6 U/L↑\gamma\text{-GT } 180.6\text{ U/L}\uparrow) returned two hours later at Node 3 (21:09).

TaskGold Standard (rationale)Gemini 3.8 FlashMedGemma-4B-itDeepSeek V3.2Scoring
Diag. (Admission)Fever — the only diagnosis confirmed at admission; Kawasaki disease and liver injury depend on later labs.Fever of unknown origin, suspected Kawasaki disease, urticaria, acute URIPneumonia, skin erythemaAcute URI, lymphadenitis, urethritisF1 0.00 / 0.00 / 0.00 — no output contains the exact term "fever".
Symp. Ext.Pharyngeal redness and pain — from this window’s note and examination.Fever, conjunctival congestion, rash, pharyngeal congestion, cervical lymphadenopathy, urethral swellingFever, erythematous rashes (multiple sites), pharyngeal congestion, coarse breath sounds, urethral rednessFever, rash, pharyngeal congestion, cervical lymphadenopathy, urethral rednessF1 0.00 / 0.00 / 0.00 — all three output "pharyngeal congestion"; exact matching fails.
Symp. Sev.Mild — congestion without dysphagia or suppuration (three-tier grading).MildMildModerateNo scorable pairs — the symptom entity never matched.
SICGI-S 5, severely ill — HR 130 bpm, coarse breath sounds, CRP and PCT elevated, Level I nursing; conscious, no cardiorespiratory instability.545Abs. error 0 / 1 / 0
GICGI-I 4, unchanged — admitted the same day, no pre-treatment comparison; 36.3∘C36.3^\circ\text{C}, symptoms as at admission.234Abs. error 2 / 1 / 0
EI0.25 — efficacy not yet manifest given the very short window.130.0——✗ / ✗ / ✗
ADE Occ.Yes (liver injury) — implicating cefoperazone-sulbactam and Xiyanping, given at this node, within 8 h.NoYesYes✗ / ✓ / ✓ (Gemini predicted No)
ADE Ext.Liver injury (MedDRA LLT normalized).NoneDrug rash, drug allergyDiarrhea, rashF1 0.00 / 0.00 / 0.00 — both detecting models named the wrong event.
Susp. DrugCefoperazone-sulbactam, Xiyanping — both started 19:53, found by retrospective review of the prior 8 h (causality 3, Possible).NoneCefoperazone-sulbactam (×2\times 2)Cefoperazone-sulbactam (×2\times 2)F1 0.00 / 0.00 / 0.00 — right drug, wrong paired ADE, so no (ADE,drug)(\text{ADE}, \text{drug}) tuple intersects.
ADE Sev.Severe, CTCAE Grade 3 — transaminase rise requiring hepatoprotective treatment.None1, 11, 1No scorable pairs — no ADE matched.
Causality3 for both pairs — WHO-UMC: plausible timing and labelled reaction, but no dechallenge evidence and disease confounding.None4, 45, 4No scorable pairs — no (ADE,drug)(\text{ADE}, \text{drug}) tuple intersects.

What this case shows. Three properties are visible at this node.

- The targets are genuinely prospective. At 19:53 the liver-injury evidence is not yet in the record; it returns at 21:09. The gold standard is reachable only by reasoning from the regimen just administered and the patient's state, not by retrieving a documented conclusion.

- Tasks are dependent, not independent. Severity, suspected drug, and causality are all conditioned on first identifying the right entity, so an extraction failure propagates: the symptom mismatch leaves Symp. Sev. with no scorable pairs, and naming the wrong ADE empties ADE Sev. and Causality. Conditional metrics therefore carry small, model-dependent denominators, so we report nn alongside them.

- Low scores reflect task difficulty, not a lenient rubric. Some zeros come from strict exact matching against MedDRA-normalized labels, others from substantive clinical error: the strongest model in our evaluation predicted no ADE at all, while the two that detected one attributed it to rash or diarrhea. Predicting that an ADE will occur is far easier than naming it and attributing it correctly — the gap this benchmark is built to measure.

Q4. What are the detailed procedures for metric calculation?

A: We have expanded Appendix B.4 with complete specifications, including derivations, rating-scale mappings, and pseudocode. The 11 metrics in Table 2 fall into two families.

Patient-level macro-averaging (Diag., Symp. Ext., Symp. Sev., ADE Occ., ADE Ext., Susp. Drug), in four steps:

  1. At each node, exact set or label matching against the gold set gives node-level TP\text{TP}, FP\text{FP}, FN\text{FN} (and TN\text{TN} for binary accuracy).
  2. Pool these counts across all nodes within each patient.
  3. From the pooled counts compute patient-level F1 =2TP2TP+FP+FN= \frac{2\text{TP}}{2\text{TP}+\text{FP}+\text{FN}} or accuracy =TP+TNTP+TN+FP+FN= \frac{\text{TP}+\text{TN}}{\text{TP}+\text{TN}+\text{FP}+\text{FN}}.
  4. Take the arithmetic mean over all patients in the evaluated partition (×100\times 100), so the statistical unit is the patient — N=316N = 316 for the full cohort and N=121N = 121 for the subset on which medical-domain models are evaluated.

Entity matching is performed after normalization to MedDRA LLT level with parenthetical annotations stripped. A missing output is treated as an empty prediction, counted as FN\text{FN}, contributing zero precision and recall.

Node-level pooled QWK (SI, GI, EI, ADE Sev., Causality). All scorable (gold,pred)(\text{gold}, \text{pred}) pairs — both non-empty — are pooled across nodes into a single Quadratic Weighted Kappa: κw=1−∑i,jWijOij∑i,jWijEij,Wij=(i−j)2(C−1)2\kappa_w = 1 - \frac{\sum_{i,j} W_{ij} O_{ij}}{\sum_{i,j} W_{ij} E_{ij}}, \quad W_{ij} = \frac{(i-j)^2}{(C-1)^2} where OijO_{ij} and EijE_{ij} are the observed and chance-expected agreement matrices and CC is the number of ordinal categories (SI/GI 8; EI 4; ADE Sev. 5, CTCAE v6.0; Causality 6, WHO-UMC). κw∈[−1,1]\kappa_w \in [-1, 1]; negative values are mathematically valid and indicate systematic inverse divergence. ADE Sev. and Causality are conditional, with denominators equal to the number of patients with true-positive ADEs and the number of true-positive ADE–drug pairs respectively; effective nn is annotated in Table 10.

Reviewer_yTVXOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

PedMed-Bench is a benchmark for evaluating language models on pediatric inpatient medication effectiveness and safety, built from EHRs at two hospitals. Records are organized around treatment-regimen changes, and the tasks cover diagnosis and symptom extraction, severity and efficacy assessment, ADE occurrence and extraction, suspected drug identification, and causality grading. Fourteen LMs are evaluated; fine-grained medication-safety attribution proves especially challenging.

Critical strengths
  1. Pediatric inpatient medication safety is clinically important and poorly covered by existing benchmarks. Organizing records around treatment changes so that clinical state, efficacy, and safety can be evaluated together is a useful design that goes beyond isolated medical QA.
Critical weaknesses
  1. The reported scores are hard to interpret because the task definitions, cohorts, and scoring procedure are not described consistently. Patient, node, and plan counts don't clearly map to each table or to the released data, and general versus medical-domain models turn out to use different cohorts. The evaluator computes both accuracy and QWK, but which one appears in which table is unclear. The ADE parser can also misread negative responses, and the effect on scores is unknown. On a separate note, the two independent inference calls don't match the claimed multi-turn interaction.
  2. The ADE occurrence task needs a majority-class baseline for context: an always-negative predictor gets about 72.4% patient-macro accuracy on the released cohort. The safety labels also deserve closer scrutiny. Outcomes observed after the next treatment change may reflect the new regimen, so the attribution rule and clinical adjudication process need to explain how such cases are handled. Temporal association plus drug-label compatibility alone can't isolate a drug's causal contribution. The age discrepancy between structured fields and clinical narratives is another sign that the data feeding clinical-reasoning evaluation needs a targeted consistency check.
Questions for authors
  1. Can you provide a cohort flow diagram linking the reported patient, node, and plan counts to the released records and each result table, with inclusion criteria?
  2. Can general and medical-domain models be compared on the same records? If they stay on different cohorts, which comparisons and rankings shouldn't be read as model differences?
  3. What state, predictions, or conversation history is passed between treatment steps or inference calls? If the calls are independent, what does "multi-turn" mean here?
  4. Which data version, output files, metric columns, and aggregation rules produced the main tables? Please clarify accuracy vs. QWK, including patient-macro accuracy vs. QWK over pooled record pairs.
  5. How many model responses are actually affected by the ADE negation-parsing issue, and how do scores change after fixing it?
  6. Can you report always-negative and always-positive baselines using the exact same cohort, labels, aggregation, and missing-output rules as the model results?
6 comments

We thank the reviewer for recognizing the clinical value of PedMed-Bench. We have revised the cohort reporting, inference description, and metric definitions, and provide quantitative evidence on the parser correction and baseline performance below. All changes are colored in blue in the revised manuscript.

Q1: Can you provide a cohort flow diagram linking the patient, node, and plan counts to the released records and each result table?

A: We have added a cohort-to-experiment flow diagram (Figure 6) and a dedicated subsection (Appendix B.1) that comprehensively trace the data from initial patient screening through treatment-node construction to the evaluation settings and corresponding result tables/figures.

Specifically, the data construction and experimental setup follow this detailed pipeline:

  1. Patient Screening & Final Cohort: Starting from 1,000 de-identified candidate inpatient EHRs across two hospitals (500 from Center 1 and 500 from Center 2), we apply the inclusion criteria (inpatients aged 28 days to 18 years) and exclusion criteria (hospital stay ≤3\le 3 days, missing diagnosis, or incomplete records). This yields a Final Cohort of 316 patients (216 from Center 1 and 100 from Center 2).
  2. Treatment Node Construction: Each patient's hospitalization course is decomposed into sequential treatment events. Through a rigorous two-annotator labeling and three-reviewer verification protocol, treatment plan adjustments are identified based on changes in medication prescriptions, with each adjustment corresponding to a treatment node. This results in 2,515 treatment nodes in total (1,530 from Center 1 and 985 from Center 2).
  3. Evaluation & Experiment Mapping: As detailed in Appendix B.1, the constructed tasks are partitioned into two evaluation settings:
    • Full Set (316 patients; 2,515 treatment nodes), which supports the main evaluation of general models ( Table 2 ) and method exploration ( Table 9 ).
    • Subset (121 patients; 858 treatment nodes), which supports the main evaluation of medical-specific models ( Table 2 ), ablation studies ( Table 3 ), diagnostic error analysis ( Figure 3 ), model scaling ( Table 8 ), and additional analyses ( Table 13, Figure 7, Figure 8, etc. ).

Together, Appendix B.1 and the revised figure provide an explicit, end-to-end mapping from the raw candidate records to the exact patient and node counts underlying every reported result.

Q2: What information passes between inference calls and treatment steps, and what does “multi-turn” mean?

A: Between successive treatment nodes, no model-generated outputs or hidden conversational states are passed forward; each inference call at node kk (timestamp TkT_k) is strictly stateless and conditioned solely on the cumulative, verified ground-truth EHR from admission up to TkT_k (including chronological progress notes, exams, lab/imaging reports, and prior regimens). In our benchmark, “multi-turn” therefore refers to a longitudinal sequence of interdependent clinical decision points (~8 nodes per hospitalization) evaluated over progressively updated patient records rather than a stateful chat history. Because the EHR context expands cumulatively at every turn, accurate prediction at later nodes still requires reasoning across the patient's longitudinal trajectory established in earlier steps, mirroring real-world inpatient rounds where clinicians reassess patients based on accumulated medical records rather than past tentative guesses.

Methodologically, decoupling model state across turns is an intentional choice to preserve benchmark integrity. Chaining model predictions forward would allow an early hallucination at node 1 to contaminate all downstream evaluations, conflating late-stage clinical reasoning with early extraction errors. By conditioning each node strictly on the verified EHR snapshot at TkT_k, every treatment decision remains independently reproducible and auditable without fragile interaction traces. We have clarified this distinction explicitly in Section 2 and Appendix B.3 of the revised manuscript.

Q3: Can general and medical-domain models be compared on the same records?

A: Yes. Both model families were first evaluated on the same 121 patients; we then extended the general-purpose evaluation to all 316.

Supplementary Table 1 gives the matched comparison on the shared 121-patient cohort under identical reference labels and scoring. There the leading general-purpose models beat the strongest medical-domain model by a clear margin (Gemini-3.8-Flash 34.84, Gemini-3.1-Pro-Preview 32.65, versus 22.65 for Baichuan-M3); per-dimension scores are in Table 2.

We also extended Baichuan-M3 to the remaining 195 patients, completing its evaluation on all 316 (Supplementary Table 2). Its overall score and rank are stable — 22.65 (8/12) to 22.72 (6/12) — so the 121-patient subset is representative and the larger sample does not change our comparative conclusions.

Supplementary Table 1: Overall score (unweighted mean of the nine task metrics) and rank on the two cohorts.

ModelTotal (121)Rank (121)Total (316)Rank (316)
Gemini-3.8-Flash34.84136.711
Gemini-3.1-Pro-Preview32.65232.162
DeepSeek-V3.227.95327.064
Gemini-3-Pro-Preview27.09429.013
Claude-Sonnet-4.625.27521.567
GPT-5.425.11623.045
Qwen3-235B-Instruct23.46721.148
Baichuan-M322.65822.726
GPT-5.221.95919.559
Claude-Opus-520.271018.3410
GPT-519.491117.7111
Qwen3-30B-Instruct19.341216.5112

Supplementary Table 2: Baichuan-M3 task-level scores on the two cohorts.

CohortDiag.Symp. Ext.Symp. Sev.SIGIEIADE Occ.ADE Ext.Susp. DrugTotal
121 patients24.3246.0756.0434.128.83−0.9431.792.720.8922.65
316 patients23.9944.8957.0936.6210.813.5925.690.950.8122.72

Q4: Which data, metric columns, and aggregation rules produced the main tables?

A: We apologize for the incomplete description. Section 5, Table 2, and Appendix B.4 now state all three explicitly.

  • Data scope. Table 2 reports general-purpose models on the full 316-patient cohort and medical-domain models on the 121-patient subset; Appendix B.1 maps every partition to its result table.
  • Metric definitions. Each column is labelled F1, exact accuracy, or QWK×100\text{QWK}\times100, correcting ADE Severity from accuracy to QWK×100\text{QWK}\times100. The five ordinal columns (SI, GI, EI, ADE Sev., Causality) carry †^\dagger, with scales in the caption: CGI 0–7 (SI, GI), 1–4 composite (EI), CTCAE Grade 1–5 (ADE Sev.), WHO-UMC (Causality).
  • Aggregation. Entity-level and binary metrics use patient-level macro-averaging: TP/FP/FN pooled within each patient across nodes, then averaged uniformly over patients. Ordinal metrics are QWK over pooled reference–prediction pairs within a partition, weighted across partitions by scorable pair count.
  • Sample support. Table 10 reports QWK×100\text{QWK}\times100, patient-macro exact accuracy, and effective nn together for ADE Severity and Causality across all 14 models; valid nn accompanies every conditional safety metric.

Appendix B.4 also gives the QWK formula, matching criteria, and the handling of missing predictions and undefined QWK.

Q5: How many responses were affected by ADE negation parsing, and how did the scores change?

A: We audited negation handling in parse_ade_severity() — which could read {"ADE": "no"} as a positive prediction — across all 14 models. The largest change in ADE Extraction F1 was +0.11 points (Qwen3-235B-A22B-Instruct, 1.01 → 1.12); two models moved by +0.02 and +0.01 and the other eleven were unchanged. The ADE Extraction ranking is unchanged.

Even at the highest affected ratio (17.60%) the shift stays within +0.11: patient-level macro-averaging bounds cohort-level impact when errors concentrate in a few patients, and correcting "no" removes false positives without adding true positives. The fix is in the released codebase and all affected results are updated.

Supplementary Table 3: Negation-parsing audit; the eleven models not listed had a 0.00–0.70% affected ratio and no change in F1.

ModelAffected recordsPre-fix ADE Ext.Post-fix ADE Ext.ΔF1
Qwen3-235B-A22B-Instruct17.60%1.011.12+0.11
Gemini 3.1 Pro Preview0.93%6.396.41+0.02
Gemini 3 Pro Preview0.47%4.314.32+0.01

Q6: Can you report always-negative and always-positive baselines under the same evaluation rules?

A: Both baselines use the same cohort, reference labels, patient-macro aggregation, and missing-output rules as our model evaluations: always-negative reaches 79.47 ADE-occurrence accuracy and always-positive 20.53. Both are now in the table note of Table 2, and Table 11 adds precision and recall for the baselines and every model.

Supplementary Table 4: ADE-occurrence accuracy, precision, and recall of the constant-prediction baselines (per-model values: Table 11).

PredictorAccuracyPrecisionRecall
Always-negative (constant)79.470.000.00
Always-positive (constant)20.5320.53100.00

Q7: How are events observed after the next treatment change attributed?

A: Our annotation separates the window used to identify an outcome from the assessment used to attribute it to a drug.

(1) 8-hour extension with active-regimen attribution (Nelson et al., 2023). The observation window runs from TstartT_{\text{start}} to Tnext+8hT_{\text{next}}+8\text{h} to capture delayed-onset or residual effects. An adjudicated ADE is still assigned only to the node whose active medication list (administered between TstartT_{\text{start}} and TnextT_{\text{next}}) contains the causal drug: the extension widens detection for drugs already active in that node but never transfers attribution to a node that did not prescribe the causal drug. If a drug started at TnextT_{\text{next}} causes an ADE two hours later, the ADE belongs solely to the new node.

(2) Cross-window lookback (Appendix A.5). All medication records within 8 hours before ADE onset are reviewed retrospectively and attribution is traced to the node(s) prescribing the causal drug. Candidate pairs are then verified by independent annotation, third-reviewer adjudication, and multidisciplinary pharmacist review.

Section 3 and Appendix A.5 now state this rule explicitly.

Q8: Does the benchmark rely on temporal association rather than causal inference?

A: Drug–ADE attribution uses structured clinical causality assessment, not temporal association or label compatibility alone. Section 3 states that causality is assessed with WHO-UMC (WHO, 2005) and RUCAM (Danan & Teschke, 2015), severity with CTCAE v6.0 and Hy's Law (FDA, 2009, 2010); Appendix A.5 now describes how each is applied.

WHO-UMC requires the evidence to meet the criteria of a given category, weighing exposure timing, alternative explanations (underlying disease, concomitant drugs), and the course after withdrawal or re-exposure. An event merely following exposure therefore does not by itself support “Probable/Likely”.

RUCAM, for suspected drug-induced liver injury, scores onset timing, course after withdrawal, concomitant medications, alternative causes, known hepatotoxicity, and re-exposure response.

Neither instrument allows “Probable” or “Certain” from temporal association alone. The gold standard was independently annotated by two blinded annotators, adjudicated by a third party, and reviewed by a multidisciplinary pharmacist team (κ = 0.7758).

Q9: How are discrepancies between structured and narrative age handled?

A: These reflect natural variation in recording granularity: clinicians write age in completed years, rounded years, or years plus months, so a child aged 5 years 6 months may appear as 5 or 6 across notes. Two points:

  1. Unified ground truth. Structured admission age is computed from date of birth and admission date, never extracted from narrative text. It is the sole authoritative reference for cohort stratification, pediatric dosing checks, and all metric computation; narrative age is never used in labeling or evaluation.
  2. Model input. At inference the model receives this date-derived age in the structured header alongside the unmodified narratives. We preserve their rounding variation rather than rewriting raw clinical text, since reconciling structured metadata with clinician shorthand is inherent to real-world EHR decision support.

Section 3.1 and Appendix A.1 now document this procedure.

Reviewer_zbyAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PedMed-Bench, a multi-turn benchmark for evaluating large language models on pediatric medication effectiveness and safety. The benchmark is constructed from longitudinal electronic health records of 316 pediatric inpatients across two tertiary hospitals. The authors organize clinical records into treatment episodes based on medication adjustments and introduce effective observation windows to capture subsequent therapeutic outcomes and adverse drug events. The benchmark evaluates 11 metrics across three dimensions: diagnostic capability, therapeutic efficacy, and medication safety.

The authors evaluate 14 representative LLMs and find that current models struggle particularly with fine-grained adverse drug event (ADE) identification and suspected-drug attribution. Additional analyses, including temporal perturbations, counterfactual medication interventions, and oracle-based evaluations, investigate potential limitations in temporal grounding and medication-level causal reasoning. The paper also explores the effects of model scaling, reasoning, prompting, retrieval, and memory.

Critical strengths
  1. A clinically grounded benchmark for longitudinal pediatric medication evaluation. The paper addresses an important and relatively underexplored setting by constructing a benchmark from real-world pediatric inpatient EHRs. Organizing clinical trajectories around treatment adjustments, rather than evaluating isolated medical questions, provides a more realistic setting for assessing medication-related reasoning. The inclusion of expert annotations, standardized clinical terminology, and multiple evaluation dimensions further increases the potential value of the benchmark as a research resource.

  2. Fine-grained evaluation and diagnostic analyses of model failures. The benchmark evaluates medication safety at multiple levels, including ADE occurrence, ADE identification, suspected-drug attribution, and causality assessment. This decomposition is useful for distinguishing coarse risk detection from more demanding medication-level reasoning. Furthermore, the temporal perturbation, medication intervention, and oracle analyses go beyond conventional leaderboard comparisons and provide potentially useful insights into model behavior.

Critical weaknesses
  1. Inconsistency in the evaluation metric definition. Section 5 and Appendix B.3 describe ADE Severity as an accuracy-based metric. However, Table 2 reports a negative ADE Severity score (-10.00) for Claude Sonnet 4.6, which is inconsistent with the definition of accuracy. The authors should clarify the actual scoring formula and explain how this negative value arises.

  2. Insufficient clarity regarding the causality evaluation. Table 2 reports a Causality score of 100.00 for Claude Opus 5, despite its very low ADE Extraction F1 (2.69) and Suspected Drug F1 (2.34). Appendix B.3 indicates that patients without valid predictions are excluded from the corresponding metric. Therefore, a perfect causality score may be based on a very small number of valid predictions and may not reflect the model's overall ability to assess drug-event causality. Reporting the number of valid evaluation instances and prediction coverage for each model would make these results more interpretable.

Questions for authors

Q1. Could the authors clarify the exact evaluation metric used for ADE Severity? In particular, how is the negative score (-10.00) in Table 2 obtained if the metric is defined as accuracy?

Q2. Could the authors report the number of valid samples used to calculate Causality for each model, especially Claude Opus 5, which achieves 100.00? How should this score be interpreted given its low ADE Extraction and Suspected Drug F1 scores? Would it be possible to report prediction coverage alongside the conditional causality accuracy?

2 comments

We thank the reviewer for recognizing the clinical relevance of PedMed-Bench and the value of its fine-grained evaluation and diagnostic analyses. Below, we address the reviewer’s questions and describe the revisions made to improve the clarity and completeness of our reporting. All changes are colored in blue in the revised manuscript.

Q1: Table 2 reports an ADE Severity of −10.00 for Claude Sonnet 4.6, which contradicts the definition of accuracy.

A: We are sorry for the confusion caused by incomplete description of the metric. The ADE Severity column reports quadratic weighted kappa multiplied by 100 (QWK × 100). ADE severity is graded on the ordinal CTCAE Grade 1–5 scale. QWK accounts for chance-expected agreement and assigns greater penalties to disagreements spanning more grades. Consequently, QWK can be negative when weighted agreement falls below that expected from the observed marginal distributions.

Claude Sonnet 4.6’s reported −10.00 corresponds to QWK = −0.10. For transparency, this result is based on 11 scorable instances, with nine exact matches and two one-grade disagreements. Ten of the eleven reference labels are Grade 1. The pooled exact accuracy is therefore 81.82% (9/11), while the patient-macro exact accuracy is 83.33%, reflecting equal weighting of contributing patients rather than individual instances. These accuracy values and the negative QWK describe different statistical properties of the same predictions and are not numerically contradictory. Their interpretation requires attention to the small sample size and concentrated grade distribution.

Therefore, we agree that reporting exact accuracy, QWK, and the effective sample size together provides a more comprehensive and interpretable assessment of ADE severity. Accordingly, we have: (i) revised the Methods section to clarify metric definitions and aggregation procedures; (ii) updated the results to include these complementary statistics, as detailed in Table 10 of the manuscript, and (iii) updated the corresponding descriptions and table labels. Appendix B.4 now also specifies the scoring formulas, matching criteria, and handling of missing predictions and undefined QWK values.

Q2: Claude Opus 5 attains a Causality score of 100.00 while its ADE Extraction F1 is only 2.69%, suggesting the score is based on too few valid cases.

A: The ADE extraction F1 scores and the causality score assess different stages of the evaluation pipeline and use different evaluation subsets. Specifically, causality is evaluated only on patients for whom the model has already correctly identified both the ADE name and the suspected drug. In the case of Claude Opus 5, only three instances satisfy the evaluation requirements for causality scoring, and all three receive the correct causality grade, resulting in a Causality score of 100.00.

We agree that the causality scores should be reported alongside their sample sizes and evaluation coverage. Accordingly, we have updated Supplementary Table 1 below, which presents ADE Extraction F1, Suspected Drug F1, the valid causality sample count, and the corresponding causality score for each listed model. This makes the sample support for each conditional score explicit, including the three instances underlying Claude Opus 5’s score of 100.00.

We have also clarified the conditional evaluation criteria in the Methods section and revised the results discussion to acknowledge that small valid sample counts limit conclusions about downstream causality-grading capability.


Supplementary Table 1: Coverage of Conditional Metrics(N=316)

ModelADE Ext.Susp. DrugValid n (Causality)Causality
Claude Opus 52.69%2.34%3100.00
GPT-5.42.16%1.05%1553.92
Gemini 3.1 Pro6.39%4.71%833.33
Gemini 3 Pro4.31%2.75%522.19
DeepSeek V3.24.10%2.19%714.21
Gemini 3.8 Flash12.89%12.90%0—
Baichuan-M32.72%0.89%312.90
Sonnet 4.62.44%5.47%0—

Note: Valid n requires both correct ADE extraction and correct suspected drug attribution simultaneously to qualify for joint causality evaluation.

We hope these clarifications and the revised reporting address the reviewer’s concerns and strengthen confidence in PedMed-Bench as a clinically grounded resource for evaluating longitudinal pediatric medication effectiveness and safety.

Dear Reviewer,

We apologize for posting our response a bit late, as we wanted to ensure every point you raised was carefully examined and thoroughly addressed in our revised manuscript and supplementary materials.

As the discussion period draws to a close (in about a day), we would love to know whether our rebuttal has resolved your concerns. If so, we would be truly grateful if you could consider updating your rating.

Should any questions remain, we would welcome your feedback and gladly provide further details before the deadline.

Thank you once more for your time and constructive review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.