- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper introduce a benchmark, PedMed-Bench, to evaluate large language models on pediatric medication effectiveness and safety. The benchmark consists of longitudinal inpatient electronic health records of 316 patients from two tertiary hospitals in China. The data records cover diagnosis, symptoms, therapeutic response, adverse drug events, suspected medications, and causality assessment. The paper evaluated 14 LLMs across multiple metrics.
Critical strengths
- The benchmark presents relevant clinical settings and provides a potentially valuable dataset.
- The authors made substantial annotation efforts, which provide a rather solid foundation for a benchmark.
Critical weaknesses
- The uniqueness of a pediatric benchmark is not clearly presented in the paper. It is not clear what is specially designed or prepared for the pediatric medication in the problem formulation, data processing, and evaluation.
- The problem is not well formulated at the moment. It defines the problem as sequential decision evaluation tasks. It predict outcomes within an observation window. However, the target mixes symptoms at window onset, severity of illness under treatment, improvement across windows, future adverse drug effects, drug-event causality, etc. The paper currently fails to identify explicitly which target concern current state and which concern future outcomes. The ambiguity affects the interpretation of low scores.
Questions for authors
- Can you provide an evaluation example of a complete treatment episode?
- How are those metrics calculated in details?
- Can you illustrate in details on how different LLM model respond to a single patient's clinical records? It is also better to highlight the key difference as well, especially for the pediatric-related ones.
- I still see substantial contents of the recording are Chinese when I access the dataset on October 6th, including family_history, history_of_present_illness, clinical_trajectory, etc.
- The readme file shows it is previously submitted to NeurIPS 2026. Is it possible to disclose how you have improved the benchmark after the submission?
- What expert-validated evidence demonstrates that the medication interventions have the expected outcomes?
- Any evidence can demonstrate a meaningful portion of the benchmark specifically requires pediatric reasoning?
- Several inconsistencies are noticed in the manuscript. For instance, table 2 reports an ADE-severity score of -10.00, while the metric is defined as accuracy. Please check all metric definition, evaluation protocols and results to ensure fair comparison. These affect the accuracy of interpretation.
10 comments
Q3: Can you provide a complete evaluation example of a treatment process?
A: Below is a de-identified real case (830becd9) giving the task-by-task evaluation at Node 2: gold standard, raw outputs from three models, and how each metric is scored. We have added this case study as Appendix C.7 and Table 21.
Case. A 3.5-year-old male (13.8 kg), hospitalized 10 days across 8 treatment nodes, discharge diagnosis Kawasaki disease with drug-induced liver injury. Chief complaint was "fever for 5 days", with conjunctival congestion during febrile spikes. Kawasaki disease was established on Day 4 (2024-06-15 09:10), initiating IVIG pulse therapy.
Node 2 (2024-06-12 19:53, "Initial Course Record") returned , , negative Mycoplasma/Chlamydia and viral antibodies, and an unremarkable chest CT. The gold standard requires the model to prospectively predict the ADE emerging in the following window — "Liver injury (Severe)" — whose evidence (, ) returned two hours later at Node 3 (21:09).
| Task | Gold Standard (rationale) | Gemini 3.8 Flash | MedGemma-4B-it | DeepSeek V3.2 | Scoring |
|---|---|---|---|---|---|
| Diag. (Admission) | Fever — the only diagnosis confirmed at admission; Kawasaki disease and liver injury depend on later labs. | Fever of unknown origin, suspected Kawasaki disease, urticaria, acute URI | Pneumonia, skin erythema | Acute URI, lymphadenitis, urethritis | F1 0.00 / 0.00 / 0.00 — no output contains the exact term "fever". |
| Symp. Ext. | Pharyngeal redness and pain — from this window’s note and examination. | Fever, conjunctival congestion, rash, pharyngeal congestion, cervical lymphadenopathy, urethral swelling | Fever, erythematous rashes (multiple sites), pharyngeal congestion, coarse breath sounds, urethral redness | Fever, rash, pharyngeal congestion, cervical lymphadenopathy, urethral redness | F1 0.00 / 0.00 / 0.00 — all three output "pharyngeal congestion"; exact matching fails. |
| Symp. Sev. | Mild — congestion without dysphagia or suppuration (three-tier grading). | Mild | Mild | Moderate | No scorable pairs — the symptom entity never matched. |
| SI | CGI-S 5, severely ill — HR 130 bpm, coarse breath sounds, CRP and PCT elevated, Level I nursing; conscious, no cardiorespiratory instability. | 5 | 4 | 5 | Abs. error 0 / 1 / 0 |
| GI | CGI-I 4, unchanged — admitted the same day, no pre-treatment comparison; , symptoms as at admission. | 2 | 3 | 4 | Abs. error 2 / 1 / 0 |
| EI | 0.25 — efficacy not yet manifest given the very short window. | 130.0 | — | — | ✗ / ✗ / ✗ |
| ADE Occ. | Yes (liver injury) — implicating cefoperazone-sulbactam and Xiyanping, given at this node, within 8 h. | No | Yes | Yes | ✗ / ✓ / ✓ (Gemini predicted No) |
| ADE Ext. | Liver injury (MedDRA LLT normalized). | None | Drug rash, drug allergy | Diarrhea, rash | F1 0.00 / 0.00 / 0.00 — both detecting models named the wrong event. |
| Susp. Drug | Cefoperazone-sulbactam, Xiyanping — both started 19:53, found by retrospective review of the prior 8 h (causality 3, Possible). | None | Cefoperazone-sulbactam () | Cefoperazone-sulbactam () | F1 0.00 / 0.00 / 0.00 — right drug, wrong paired ADE, so no tuple intersects. |
| ADE Sev. | Severe, CTCAE Grade 3 — transaminase rise requiring hepatoprotective treatment. | None | 1, 1 | 1, 1 | No scorable pairs — no ADE matched. |
| Causality | 3 for both pairs — WHO-UMC: plausible timing and labelled reaction, but no dechallenge evidence and disease confounding. | None | 4, 4 | 5, 4 | No scorable pairs — no tuple intersects. |
What this case shows. Three properties are visible at this node.
- The targets are genuinely prospective. At 19:53 the liver-injury evidence is not yet in the record; it returns at 21:09. The gold standard is reachable only by reasoning from the regimen just administered and the patient's state, not by retrieving a documented conclusion.
- Tasks are dependent, not independent. Severity, suspected drug, and causality are all conditioned on first identifying the right entity, so an extraction failure propagates: the symptom mismatch leaves Symp. Sev. with no scorable pairs, and naming the wrong ADE empties ADE Sev. and Causality. Conditional metrics therefore carry small, model-dependent denominators, so we report alongside them.
- Low scores reflect task difficulty, not a lenient rubric. Some zeros come from strict exact matching against MedDRA-normalized labels, others from substantive clinical error: the strongest model in our evaluation predicted no ADE at all, while the two that detected one attributed it to rash or diarrhea. Predicting that an ADE will occur is far easier than naming it and attributing it correctly — the gap this benchmark is built to measure.
Q4. What are the detailed procedures for metric calculation?
A: We have expanded Appendix B.4 with complete specifications, including derivations, rating-scale mappings, and pseudocode. The 11 metrics in Table 2 fall into two families.
Patient-level macro-averaging (Diag., Symp. Ext., Symp. Sev., ADE Occ., ADE Ext., Susp. Drug), in four steps:
- At each node, exact set or label matching against the gold set gives node-level , , (and for binary accuracy).
- Pool these counts across all nodes within each patient.
- From the pooled counts compute patient-level F1 or accuracy .
- Take the arithmetic mean over all patients in the evaluated partition (), so the statistical unit is the patient — for the full cohort and for the subset on which medical-domain models are evaluated.
Entity matching is performed after normalization to MedDRA LLT level with parenthetical annotations stripped. A missing output is treated as an empty prediction, counted as , contributing zero precision and recall.
Node-level pooled QWK (SI, GI, EI, ADE Sev., Causality). All scorable pairs — both non-empty — are pooled across nodes into a single Quadratic Weighted Kappa: where and are the observed and chance-expected agreement matrices and is the number of ordinal categories (SI/GI 8; EI 4; ADE Sev. 5, CTCAE v6.0; Causality 6, WHO-UMC). ; negative values are mathematically valid and indicate systematic inverse divergence. ADE Sev. and Causality are conditional, with denominators equal to the number of patients with true-positive ADEs and the number of true-positive ADE–drug pairs respectively; effective is annotated in Table 10.
We thank the reviewer for recognizing the clinical relevance of our dataset and annotation work. We have revised the manuscript to address the concerns about pediatric specificity, task formulation, evaluation pipelines, reference labels, and reporting inconsistencies. All changes are colored in blue.
A: Three characteristics of pediatric medication motivate the benchmark: age-related developmental variation in drug disposition and response, weight-dependent dosing, and frequent off-label use amid limited pediatric evidence. Each is carried into one of the three components the reviewer asks about.
Formulation. Each hospitalization is a sequence of treatment nodes, with inference conditioned on age, body weight, medication regimen, and clinical history; at each node the model assesses the current state and predicts subsequent treatment response and ADEs. The three characteristics map directly onto this design: developmental stage governs how presentations and drug responses are interpreted, body weight governs how administered doses are interpreted, and prior treatment observations supply patient-specific evidence where pediatric prescribing guidance is thin, including off-label use.
Data. 316 patients aged 28 days to 18 years, 2,515 treatment nodes, and 186 distinct diagnoses, including pediatric-specific conditions such as Kawasaki disease, congenital heart disease, and infantile idiopathic thrombocytopenia. The ADE spectrum likewise contains age-specific categories such as neonatal hyperbilirubinemia and infantile diarrhea — concrete pediatric content, not general medical question answering.
Evaluation. Appendix C.1 stratifies every metric across five developmental groups (infancy, toddlerhood, preschool, school age, adolescence) to test whether aggregate scores conceal age-related differences, so performance is reported against the developmental variation that motivates the benchmark, not pooled over it.
Given the main-text length limit, we have added Appendix A.8 ("Pediatric-Specific Design") mapping these challenges to our formulation, data, and analyses.
A: The reviewer correctly observes that our tasks span different temporal targets. This is a deliberate choice, made with pediatric physicians, to reproduce the cognitive load a clinician carries at each treatment decision, and it follows established distinctions: clinical prediction modelling separates diagnostic prediction (present state) from prognostic prediction (future outcomes), whose target definition, validation, and interpretation must be handled separately [1]; situation-awareness theory describes three ascending levels — perception of current elements, comprehension of the present state, projection of future states [2]; and clinical time-series benchmarks separate tasks on the same grounds [3]. We therefore group our metrics by temporal target:
Class A — current state (at t). Diagnosis and symptom extraction/severity, scored against the progress note and physical examination at that timestamp, measure immediate integration of clinical evidence.
Class B — within-window outcome (t to t+Δ). SI, GI, and EI measure prospective prediction of the course under the recorded regimen; ADE occurrence and extraction measure prospective identification of adverse events. Both are scored against how the course actually evolved.
Class C — retrospective causal inference. Suspected drug and causality measure attribution after an ADE has occurred. Experts first determine under WHO-UMC and RUCAM whether a suspected ADE occurred within 8 h of the node's regimen; if so, the preceding 8 h are reviewed retrospectively and annotated independently.
"Sequential decision evaluation" denotes this complete loop executed across successive treatment nodes, not a prediction at a single timestamp.
To remove the ambiguity in reading low scores, the appendix now analyses capability by task class rather than as one undifferentiated verdict, with per-class input-truncation and time-anchor diagrams making each class's temporal semantics explicit.
[1] Steyerberg E W. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. Springer, 2009.
[2] Endsley M R. Toward a theory of situation awareness in dynamic systems. In: Situational Awareness, Routledge, 2017: 9–42.
[3] Harutyunyan H, Khachatrian H, Kale D C, et al. Multitask learning and benchmarking with clinical time series data. Scientific Data, 2019, 6(1): 96.