- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes OntoSkill, an ontology-aware agentic skill-evolution framework for clinical prediction from longitudinal electronic health records. The authors first construct an EHR-oriented ontology from medical references, organizing measurements into clinical entities, typed relations, and ordered reasoning rules. Each root entity is associated with a persistent textual skill containing an activation policy and an ordered set of relation-specific instructions.
OntoSkill evolves these skills through three mechanisms. Ontology-aware Semantic Grounding maps raw longitudinal measurements to clinically interpretable status and trend states and activates the corresponding skills. Ontology-aware Empirical Prioritization assigns each activated skill a normalized error score based on its local outcome probability and selects one skill for refinement. Ontology-aware Contrastive Evolution compares successful and failed traces under the same ontology rules and uses LLM-based evaluation and editing to propose an ontology-constrained edit package, which is accepted or rolled back according to validation performance.
The framework is evaluated on MIMIC-IV, TJH, and CDSL for mortality and readmission prediction using Qwen3-8B, Claude-Sonnet-4.6, and DeepSeek-V4-Flash. The paper compares against direct prompting, retrieval, tool-augmented, multi-agent, and skill-evolution baselines and reports improvements across several settings, together with transfer, ablation, cost, and qualitative analyses.
The paper is exceptionally clear and visually polished, and the ontology-constrained skill lifecycle is a coherent system design. However, the current evaluation does not establish that the reported gains arise from leakage-free prospective prediction or clinically validated reasoning. The reuse of labeled evolution cases and the heuristic interpretation of local skill losses as responsibility also weaken the claimed skill-localization mechanism.
Critical strengths
-
A coherent and inspectable ontology-constrained skill lifecycle. OntoSkill does not allow an LLM to rewrite clinical prompts without restrictions. It grounds observations into named clinical entities, limits edits to ontology-supported relations, preserves rule order, exposes relation-level traces, and accepts or rolls back candidate edits. This structure is more inspectable than unconstrained experience summarization and provides a useful framework for organizing persistent clinical reasoning instructions.
-
Broad system-level evaluation and analysis. The paper evaluates multiple datasets, prediction tasks, and LLM backbones and compares against several agentic paradigms. It also includes component ablations, cross-model and cross-dataset transfer, cost analysis, and qualitative skill-lifecycle examples. This breadth is valuable for assessing a complex agentic framework.
Critical weaknesses
-
Leakage-free clinical prediction and clinical validity are not established. The paper does not clearly specify the index time, observation window, and prediction horizon for each task. The prompts describe longitudinal ICU or hospitalization records, raising the possibility that measurements close to death or discharge are included. Without a fixed prediction time, the task may become retrospective outcome classification rather than prospective risk prediction. In addition, the ontology and evolved skills are produced through retrieval, LLM generation, structural checks, and labeled experience, but no independent clinician evaluation of relation correctness, reasoning order, or evolved-skill safety is reported. Structural validity and support from medical references do not by themselves establish clinically valid reasoning.
-
The adaptive evolution protocol does not establish reliable skill attribution, control over overfitting, or generality beyond EHR. OEP treats a skill's local cross-entropy as normalized responsibility even though the final probability is produced by a nonlinear aggregation of evidence from multiple correlated skills. A poor local probability therefore does not show that the skill caused the final error. Moreover, successful and failed cases are used to generate an edit, and candidate acceptance appears to sample up to 50 activated cases from the same evolution set. Repeatedly selecting edits with labeled cases can overfit a small evolution cohort, especially for TJH, which contains only 140 evolution examples. Calling the method training-free is also potentially misleading: model parameters are frozen, but labels are still used for non-parametric skill optimization and model selection. Finally, the paper presents the where/which/how decomposition as a unified skill-evolution framework, but every major component exploits EHR-specific structure: a fixed measurement schema, clinical reference ranges, longitudinal trends, a stable expert ontology, and binary patient outcomes. No non-EHR task is evaluated, so it remains unclear whether OntoSkill is a general ontology-guided self-evolution method or a domain-specific clinical prompting system.
Questions for authors
-
For each dataset and prediction task, what are the exact index time, observation window, and prediction horizon? Does mortality prediction use only information available within a fixed period after admission, or the complete hospitalization trajectory? Please explain how measurements recorded near or after death, discharge, or the prediction endpoint are excluded, and report results under fixed observation windows such as the first 24 and 48 hours.
-
How were the constructed ontology and evolved skills clinically validated? Please report the number of entities, relations, and rules for each dataset and obtain independent clinician assessments of relation correctness, reasoning-order validity, and the clinical safety of evolved skill instructions. Structural checks and retrieval from medical references do not by themselves establish clinical validity.
-
Are the cases used to identify skill failures, construct success–failure contrasts, generate skill edits, accept or reject candidate skills, and evaluate final performance mutually disjoint? Please provide the exact evolution, validation, and test splits. If candidate acceptance repeatedly uses cases from the same evolution set that generated the edit, how is adaptive overfitting controlled?
-
OEP assigns responsibility using each skill's local cross-entropy even though the final prediction is produced by a nonlinear aggregation of multiple skill-generated evidence statements. Does a high OEP responsibility score identify the skill that materially affects the final error? Please validate this using leave-one-skill-out or counterfactual replacement experiments and compare OEP against random selection, activation frequency, and direct marginal-influence baselines.
-
Which components of OntoSkill are intended to be general, and which fundamentally depend on EHR-specific structure? OSG relies on clinical reference ranges and longitudinal trends, OEP assumes skill-local outcome probabilities, and OCE edits skills within a fixed expert ontology. Can the authors validate the where/which/how framework on at least one non-EHR structured reasoning or prediction task, or otherwise demonstrate how it operates when such clinical thresholds, measurement schemas, and ontology rules are unavailable? If not, the paper should narrow its general agentic skill-evolution claims and position the contribution explicitly as an EHR-specific system.
2 comments
Final Decision
I appreciate the authors' efforts. However, in my view, this paper does not meet the standards I expect for an ICLR conference paper. I apologize for this.
Response to Reviewer #isGw#
Thanks for your helpful comments. We have carefully addressed your main concerns with additional experiments and clarifications. The revised manuscript has been uploaded to the submission system, with all revisions highlighted in orange.
W1-1&Q1: Clarification of Task Settings. For reproducibility and fairness, we follow the EHR benchmark task definitions adopted in [1–3], using the complete hospitalization trajectory as input. Specifically, in-hospital mortality is evaluated under the retrospective setting, using “structured EHR data from the entire hospital stay ... with the prediction made at the time of discharge” [1], whereas 30-day readmission is defined as “a prospective binary classification task to predict whether a patient will be readmitted within 30 days of discharge” [1]. We have further specified these task definitions in the revised Problem Formulation (p. 3). MIMIC-IV in-hospital mortality results using a fixed first-48-hour observation window are shown below, further validating OntoSkill.
W1-2&Q2: Clinical Evaluation. Three physicians from a well-known local hospital, specializing in Critical Care Medicine, Pulmonary Medicine, and General Internal Medicine, assessed ontology-relation validity, clinical reasoning order, and evolved-skill safety across MIMIC-IV, TJH, and CDSL, covering 62 clinical relations and 23 reasoning rules in total. For example, they evaluated the clinical rationale of CONDITIONAL (Respiratory Rate|Body Temperature) → MUTUAL (Oxygenation, Respiratory Rate), examining whether the contextual interpretation appropriately precedes joint physiological assessment. Their assessments indicated that the ontology and evolved skills were generally aligned with RAG-retrieved medical references and reasonably consistent with clinical practice. The ontology statistics are summarized below.
W2-1&Q4: Validation of OEP. Empirically, we compare OEP with Counterfactual Attribution (CA) under the same experimental settings. OEP selects the same skill as CA in 8 of 10 evolution rounds. Theoretically, OEP uses skill-local loss as an efficient proxy for error responsibility. While CA requires additional aggregation calls scaling linearly with the number of activated skills, OEP requires only a single aggregation and reuses existing predictions, achieving a favorable trade-off between prioritization effectiveness and computational efficiency. Further comparisons with alternative strategies on MIMIC-IV Mortality are provided below.
W2-2&Q3: Data Splits and Overfitting. As we had already detailed in Data Statistics (p. 18-19, Table 6), we follow established data-splitting and evaluation protocols [3]. The 140 TJH evolution cases constitute the entire available training set. Candidate skills are validated on up to 50 randomly sampled training cases, ensuring test-set isolation. For reproducibility and fair comparison, we apply no additional overfitting-mitigation techniques (e.g., data augmentation), as their investigation is beyond this study's scope.
W2-3&Q5: Clarification of Scope. OntoSkill aims to address clinical semantic fragmentation and insufficient structural guidance in EHR clinical prediction by proposing a unified ontology-aware agentic skill evolution framework, integrating the Where/Which/How mechanisms for skill formation, prioritization, and refinement. In fact, we explicitly stated in the Limitations (p. 24, Appendix E.4) that “skill evolution is constrained by a fixed EHR-oriented clinical ontology.” To clarify these misunderstandings, we have revised the contribution statement to explicitly emphasize its EHR focus as follows: “We propose OntoSkill, a unified ontology-aware agentic skill evolution framework that operationalizes an EHR-oriented clinical ontology throughout the skill lifecycle to address clinical semantic fragmentation and insufficient structural guidance in EHR clinical prediction.” To avoid ambiguity, we have replaced “training-free” with “without parameter updates”.
References
[1] ClinicRealm: Re-evaluating Large Language Models with Conventional Machine Learning for Non-Generative Clinical Prediction Tasks. npj Digital Medicine, 2026.
[2] ColaCare: Enhancing Electronic Health Record Modeling through Large Language Model-Driven Multi-Agent Collaboration. WWW, 2025.
[3] MedAgentBoard: Benchmarking Multi-Agent Collaboration with Conventional Methods for Diverse Medical Tasks. NeurIPS, 2025.