acceptodds
Desk rejected at ICLR 2027

OntoSkill: Ontology-aware Agentic Skill Evolution for EHR Clinical Prediction

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Electronic Health Records (EHR) clinical prediction requires integrating heterogeneous observations and their complex temporal and clinical dependencies, while agents driven by Large Language Models (LLMs) provide a promising technical direction. Recent studies increasingly explore the agent skill paradigm for experience-driven capability evolution across medical reasoning tasks. However, extending agent skills to EHR clinical prediction faces two fundamental challenges: (1) clinical semantic fragmentation, where isolated measurements lack explicit mappings and dependencies for coherent patient states, and (2) insufficient structural guidance, where performance feedback alone cannot induce clinically meaningful skill structures. To address these issues, we construct an EHR-oriented ontology that structures fragmented observations into meaningful clinical entities, atomic relations, and reasoning rules. We further propose OntoSkill, an ontology-aware agentic skill evolution framework that autonomously forms and evolves reusable clinical skills from experience. OntoSkill resolves three coupled decisions in skill evolution. (i) Where to form: grounding fragmented longitudinal observations into ontology-grounded patient states. (ii) Which to evolve: prioritizing skills based on accumulated prediction experience. (iii) How to evolve: contrasting successful and failed experiences under ontology rules to localize reasoning deficiencies and refine the selected skill. Comprehensive experiments on multiple EHR datasets demonstrate that OntoSkill consistently outperforms state-of-the-art baselines, validating its effectiveness for EHR clinical prediction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Agentic Reasoning, Skill Evolution, Electronic Health Records, Clinical Ontology

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_isGwOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OntoSkill, an ontology-aware agentic skill-evolution framework for clinical prediction from longitudinal electronic health records. The authors first construct an EHR-oriented ontology from medical references, organizing measurements into clinical entities, typed relations, and ordered reasoning rules. Each root entity is associated with a persistent textual skill containing an activation policy and an ordered set of relation-specific instructions.

OntoSkill evolves these skills through three mechanisms. Ontology-aware Semantic Grounding maps raw longitudinal measurements to clinically interpretable status and trend states and activates the corresponding skills. Ontology-aware Empirical Prioritization assigns each activated skill a normalized error score based on its local outcome probability and selects one skill for refinement. Ontology-aware Contrastive Evolution compares successful and failed traces under the same ontology rules and uses LLM-based evaluation and editing to propose an ontology-constrained edit package, which is accepted or rolled back according to validation performance.

The framework is evaluated on MIMIC-IV, TJH, and CDSL for mortality and readmission prediction using Qwen3-8B, Claude-Sonnet-4.6, and DeepSeek-V4-Flash. The paper compares against direct prompting, retrieval, tool-augmented, multi-agent, and skill-evolution baselines and reports improvements across several settings, together with transfer, ablation, cost, and qualitative analyses.

The paper is exceptionally clear and visually polished, and the ontology-constrained skill lifecycle is a coherent system design. However, the current evaluation does not establish that the reported gains arise from leakage-free prospective prediction or clinically validated reasoning. The reuse of labeled evolution cases and the heuristic interpretation of local skill losses as responsibility also weaken the claimed skill-localization mechanism.

Critical strengths
  1. A coherent and inspectable ontology-constrained skill lifecycle. OntoSkill does not allow an LLM to rewrite clinical prompts without restrictions. It grounds observations into named clinical entities, limits edits to ontology-supported relations, preserves rule order, exposes relation-level traces, and accepts or rolls back candidate edits. This structure is more inspectable than unconstrained experience summarization and provides a useful framework for organizing persistent clinical reasoning instructions.

  2. Broad system-level evaluation and analysis. The paper evaluates multiple datasets, prediction tasks, and LLM backbones and compares against several agentic paradigms. It also includes component ablations, cross-model and cross-dataset transfer, cost analysis, and qualitative skill-lifecycle examples. This breadth is valuable for assessing a complex agentic framework.

Critical weaknesses
  1. Leakage-free clinical prediction and clinical validity are not established. The paper does not clearly specify the index time, observation window, and prediction horizon for each task. The prompts describe longitudinal ICU or hospitalization records, raising the possibility that measurements close to death or discharge are included. Without a fixed prediction time, the task may become retrospective outcome classification rather than prospective risk prediction. In addition, the ontology and evolved skills are produced through retrieval, LLM generation, structural checks, and labeled experience, but no independent clinician evaluation of relation correctness, reasoning order, or evolved-skill safety is reported. Structural validity and support from medical references do not by themselves establish clinically valid reasoning.

  2. The adaptive evolution protocol does not establish reliable skill attribution, control over overfitting, or generality beyond EHR. OEP treats a skill's local cross-entropy as normalized responsibility even though the final probability is produced by a nonlinear aggregation of evidence from multiple correlated skills. A poor local probability therefore does not show that the skill caused the final error. Moreover, successful and failed cases are used to generate an edit, and candidate acceptance appears to sample up to 50 activated cases from the same evolution set. Repeatedly selecting edits with labeled cases can overfit a small evolution cohort, especially for TJH, which contains only 140 evolution examples. Calling the method training-free is also potentially misleading: model parameters are frozen, but labels are still used for non-parametric skill optimization and model selection. Finally, the paper presents the where/which/how decomposition as a unified skill-evolution framework, but every major component exploits EHR-specific structure: a fixed measurement schema, clinical reference ranges, longitudinal trends, a stable expert ontology, and binary patient outcomes. No non-EHR task is evaluated, so it remains unclear whether OntoSkill is a general ontology-guided self-evolution method or a domain-specific clinical prompting system.

Questions for authors
  1. For each dataset and prediction task, what are the exact index time, observation window, and prediction horizon? Does mortality prediction use only information available within a fixed period after admission, or the complete hospitalization trajectory? Please explain how measurements recorded near or after death, discharge, or the prediction endpoint are excluded, and report results under fixed observation windows such as the first 24 and 48 hours.

  2. How were the constructed ontology and evolved skills clinically validated? Please report the number of entities, relations, and rules for each dataset and obtain independent clinician assessments of relation correctness, reasoning-order validity, and the clinical safety of evolved skill instructions. Structural checks and retrieval from medical references do not by themselves establish clinical validity.

  3. Are the cases used to identify skill failures, construct success–failure contrasts, generate skill edits, accept or reject candidate skills, and evaluate final performance mutually disjoint? Please provide the exact evolution, validation, and test splits. If candidate acceptance repeatedly uses cases from the same evolution set that generated the edit, how is adaptive overfitting controlled?

  4. OEP assigns responsibility using each skill's local cross-entropy even though the final prediction is produced by a nonlinear aggregation of multiple skill-generated evidence statements. Does a high OEP responsibility score identify the skill that materially affects the final error? Please validate this using leave-one-skill-out or counterfactual replacement experiments and compare OEP against random selection, activation frequency, and direct marginal-influence baselines.

  5. Which components of OntoSkill are intended to be general, and which fundamentally depend on EHR-specific structure? OSG relies on clinical reference ranges and longitudinal trends, OEP assumes skill-local outcome probabilities, and OCE edits skills within a fixed expert ontology. Can the authors validate the where/which/how framework on at least one non-EHR structured reasoning or prediction task, or otherwise demonstrate how it operates when such clinical thresholds, measurement schemas, and ontology rules are unavailable? If not, the paper should narrow its general agentic skill-evolution claims and position the contribution explicitly as an EHR-specific system.

2 comments
AuthorsOpenReview ↗

Response to Reviewer #isGw#

Thanks for your helpful comments. We have carefully addressed your main concerns with additional experiments and clarifications. The revised manuscript has been uploaded to the submission system, with all revisions highlighted in orange.

W1-1&Q1: Clarification of Task Settings. For reproducibility and fairness, we follow the EHR benchmark task definitions adopted in [1–3], using the complete hospitalization trajectory as input. Specifically, in-hospital mortality is evaluated under the retrospective setting, using “structured EHR data from the entire hospital stay ... with the prediction made at the time of discharge” [1], whereas 30-day readmission is defined as “a prospective binary classification task to predict whether a patient will be readmitted within 30 days of discharge” [1]. We have further specified these task definitions in the revised Problem Formulation (p. 3). MIMIC-IV in-hospital mortality results using a fixed first-48-hour observation window are shown below, further validating OntoSkill.

MethodsAUROCAUPRC
Direct82.59±4.9637.63±10.95
OntoSkill85.39±4.6752.38±11.78

W1-2&Q2: Clinical Evaluation. Three physicians from a well-known local hospital, specializing in Critical Care Medicine, Pulmonary Medicine, and General Internal Medicine, assessed ontology-relation validity, clinical reasoning order, and evolved-skill safety across MIMIC-IV, TJH, and CDSL, covering 62 clinical relations and 23 reasoning rules in total. For example, they evaluated the clinical rationale of CONDITIONAL (Respiratory Rate|Body Temperature) → MUTUAL (Oxygenation, Respiratory Rate), examining whether the contextual interpretation appropriately precedes joint physiological assessment. Their assessments indicated that the ontology and evolved skills were generally aligned with RAG-retrieved medical references and reasonably consistent with clinical practice. The ontology statistics are summarized below.

DatasetsEntitiesRelationsRules
MIMIC-IV111411
TJH23246
CDSL23246

W2-1&Q4: Validation of OEP. Empirically, we compare OEP with Counterfactual Attribution (CA) under the same experimental settings. OEP selects the same skill as CA in 8 of 10 evolution rounds. Theoretically, OEP uses skill-local loss as an efficient proxy for error responsibility. While CA requires additional aggregation calls scaling linearly with the number of activated skills, OEP requires only a single aggregation and reuses existing predictions, achieving a favorable trade-off between prioritization effectiveness and computational efficiency. Further comparisons with alternative strategies on MIMIC-IV Mortality are provided below.

Methods (DeepSeek-V4-Flash)Selection Consistency (CA as Reference)Number of Aggregations↓AUROC↑AUPRC↑
CA10/10K+195.86±2.0980.99±7.75
Random0/10193.75±3.4078.76±9.30
Activation Frequency0/10193.47±3.2780.99±8.51
OEP8/10195.52±2.5383.01±7.61

W2-2&Q3: Data Splits and Overfitting. As we had already detailed in Data Statistics (p. 18-19, Table 6), we follow established data-splitting and evaluation protocols [3]. The 140 TJH evolution cases constitute the entire available training set. Candidate skills are validated on up to 50 randomly sampled training cases, ensuring test-set isolation. For reproducibility and fair comparison, we apply no additional overfitting-mitigation techniques (e.g., data augmentation), as their investigation is beyond this study's scope.

W2-3&Q5: Clarification of Scope. OntoSkill aims to address clinical semantic fragmentation and insufficient structural guidance in EHR clinical prediction by proposing a unified ontology-aware agentic skill evolution framework, integrating the Where/Which/How mechanisms for skill formation, prioritization, and refinement. In fact, we explicitly stated in the Limitations (p. 24, Appendix E.4) that “skill evolution is constrained by a fixed EHR-oriented clinical ontology.” To clarify these misunderstandings, we have revised the contribution statement to explicitly emphasize its EHR focus as follows: “We propose OntoSkill, a unified ontology-aware agentic skill evolution framework that operationalizes an EHR-oriented clinical ontology throughout the skill lifecycle to address clinical semantic fragmentation and insufficient structural guidance in EHR clinical prediction.” To avoid ambiguity, we have replaced “training-free” with “without parameter updates”.

References

[1] ClinicRealm: Re-evaluating Large Language Models with Conventional Machine Learning for Non-Generative Clinical Prediction Tasks. npj Digital Medicine, 2026.

[2] ColaCare: Enhancing Electronic Health Record Modeling through Large Language Model-Driven Multi-Agent Collaboration. WWW, 2025.

[3] MedAgentBoard: Benchmarking Multi-Agent Collaboration with Conventional Methods for Diverse Medical Tasks. NeurIPS, 2025.

Reviewer_isGwOpenReview ↗

Final Decision

I appreciate the authors' efforts. However, in my view, this paper does not meet the standards I expect for an ICLR conference paper. I apologize for this.

Reviewer_6QAbOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OntoSkill, an ontology-aware framework for evolving reusable clinical reasoning skills for EHR outcome prediction. A medical-reference-grounded ontology defines clinical entities, relations, and reasoning precedence. OntoSkill grounds patient observations into entity-level states, prioritizes skills using outcome-based prediction discrepancies, and refines selected skills by contrasting successful and failed reasoning traces under ontology constraints. Experiments evaluate mortality and readmission prediction across three datasets and multiple LLM backbones, with additional transfer and ablation studies.

Critical strengths
  1. The framework provides an explicit structure for experience-driven skill refinement. Anchoring skills to clinical entities and restricting edits to ontology-supported relations makes the adaptation process more inspectable than unconstrained instruction rewriting. The separation between a fixed ontology and evolving skills is a useful design choice.

  2. The experiments cover multiple datasets and model backbones, and some improvements are substantial. For example, on CDSL with Claude-Sonnet-4.6, AUPRC increases from 70.52 for Direct to 84.85 for OntoSkill. The cross-model transfer experiments and skill lifecycle examples also provide useful evidence that the learned instructions can be reused and inspected.

Critical weaknesses
  1. The evaluation does not sufficiently establish the reliability of the reported gains. Each task uses a single sample of 200 test patients, and the reported deviations come from bootstrap resampling rather than independently repeated skill evolution runs. These estimates do not capture variation from training sample selection or skill generation and refinement. Several comparisons with strong baselines are close, such as readmission AUPRC of 27.31 versus 26.91 for MedLA with DeepSeek. Paired uncertainty estimates for performance differences and repeated evolution runs would help determine which improvements are robust.

  2. The additional contribution of skill evolution over static ontology guidance remains insufficiently established. In Table 4, mortality AUPRC increases from 71.99 to 81.77 with the retrieval-grounded clinical ontology, reaches 82.08 with OSG, and reaches 83.01 with the complete framework. Thus, much of the observed improvement is already obtained before the full evolution mechanism is introduced. Although the readmission AUROC results suggest a larger contribution from subsequent components, the current evidence does not clearly establish the benefit of the proposed prioritization and contrastive refinement over simpler alternatives under comparable supervision and computational budgets.

  3. The interpretation of skill-local prediction discrepancy as responsibility for the final error is not adequately validated. A skill observes a restricted clinical context, so a high local outcome loss can reflect insufficient information rather than defective reasoning. Moreover, the final aggregator consumes textual evidence, whereas OEP assigns responsibility using independently estimated skill probabilities. Similarly, partitioning traces by final prediction correctness does not establish whether an individual skill reasoned correctly. The proposed mechanism is a plausible heuristic, but stronger evidence is needed to support its interpretation as targeted reasoning-error localization.

  4. Several evaluation details materially affect the interpretation and reproducibility of the results. The relationship between the experience set and candidate-validation cases, the acceptance metric, and the prediction time and observation window are not sufficiently explicit. In addition, Appendix C.1 states that cross-dataset transfer uses two shared indicators, but their identities and the corresponding input restrictions for each baseline are unclear. Several baseline entries in Table 2 are identical to those in Table 1, making the exact transfer comparison difficult to assess.

Questions for authors
  1. Are the cases used to accept or reject candidate edits disjoint from those used to construct the success–failure contrasts? Are validation cases reused across rounds, and what exact metric determines acceptance?

  2. What prediction time and observation cutoff are used for each task? How are measurements near death or discharge handled, and are multiple admissions from the same patient kept within a single split?

  3. Can you report paired confidence intervals for differences against the strongest baselines and variability across independently repeated skill evolution runs? Do the main conclusions hold on a larger test cohort?

  4. Under matched computational and supervision budgets, how does the full method compare with a frozen initialized skill library and with refinement based on failure examples without rule-aligned success–failure contrasts?

  5. Which two indicators are used in the TJH–CDSL transfer experiments? Do all methods receive the same restricted inputs, and why are several baseline results identical to the corresponding full-feature results in Table 1?

3 comments
AuthorsOpenReview ↗

Response to Reviewer #6QAb#

Thanks for your helpful feedback. We have carefully addressed your main concerns with additional experiments and clarifications. The revised manuscript has been uploaded to the submission system, with all revisions highlighted in green.

W1&Q3: Uncertainty Estimation. Following [4], we report mean±SD over 100 bootstraps on 200 test patients. Paired bootstrap comparisons against the strongest baseline, Skill-Pro, yield positive mean differences (Skill-Pro → OntoSkill) on both metrics at both test-set sizes. Although two 95% CIs marginally cross zero, OntoSkill still achieves substantial positive mean gains on both metrics, with the upper CI bounds reaching +4.45 and +15.80, respectively. These positive gains persist when expanding the test set to 400 cases.

BaselineMetricPaired Difference95% CI
Skill-Pro → OntoSkill (200 cases)MIMIC Mortality AUROC↑+1.86[+0.21, +4.45]
Skill-Pro → OntoSkill (200 cases)MIMIC Mortality AUPRC↑+6.72[-0.53, +15.80]
Skill-Pro → OntoSkill (400 cases)MIMIC Mortality AUROC↑+1.18[-0.24, 2.45]
Skill-Pro → OntoSkill (400 cases)MIMIC Mortality AUPRC↑+6.23[1.17, 12.50]

W2&Q4: Gain of Evolution. The substantial gains achieved before full evolution arise from two factors: (1) Limited evolution data. To control computational costs, we use only 500 cases for skill evolution (Table 6, p. 18). Expanding to 1000 cases yields further performance improvements. (2) Strong ontology-based initialization. The clinical ontology provides a strong foundation for initial skills, enabling effective clinical reasoning. Further comparisons are presented below.

Methods (DeepSeek-V4-Flash)MIMIC-IV Mortality AUROC (95% CI)↑MIMIC-IV Mortality AUPRC (95% CI)↑
Initialized Skills94.90±2.85 [88.54, 99.47]82.08±8.31 [65.80, 95.49]
Failure-Only Refinement94.38±2.96 [87.31, 98.94]78.74±8.55 [60.97, 92.83]
OntoSkill (500 cases)95.52±2.53 [90.22, 99.03]83.01±7.61 [64.51,94.01]
OntoSkill (1000 cases)96.79±1.64 [92.84, 99.14]83.69±7.43 [66.17, 94.60]

W3: Justification of Local Observations. Guided by the clinical ontology, each skill jointly interprets clinically related physiological indicators and their trends to derive meaningful clinical evidence. This approach aligns with established clinical practice [1, 2], reducing the likelihood that high skill-local prediction losses arise merely from insufficient clinical context. Additionally, Counterfactual Attribution (CA) directly measures each skill's marginal impact on the aggregated prediction. OEP selects the same skill as CA in 8 of 10 evolution rounds, while requiring only 1/(K+1)1/(K+1) of CA's aggregation cost. OCE aims to leverage final prediction correctness to construct success–failure contrasts and identify reusable reasoning patterns.

W4-1&Q1&Q2: Clarification of Experimental Protocol. The experience set corresponds to the training set (shown in Table 6). Candidate validation uses up to 50 randomly sampled training cases activating the target skill. Given the limited training cohorts, particularly TJH (140 cases), candidate validation uses the same training pool as contrastive editing, resulting in some overlap between the two sets and reuse of validation cases across evolution rounds. All admissions from the same patient are assigned to the same split. As shown in Fig. 6, an edit is accepted only if it reduces validation CE loss. Following the dataset task definitions and evaluation protocols [3, 4], we use the complete hospitalization trajectory as input. Specifically, in-hospital mortality follows the retrospective setting, using “structured EHR data from the entire hospital stay ... at the time of discharge” [3], whereas 30-day readmission is defined as “a prospective binary classification task to predict whether a patient will be readmitted within 30 days of discharge” [3]. We have further clarified these settings in the revised Implementation Details (p. 8) and Problem Formulation (p. 3).

W4-2&Q5: Clarification of Transfer Settings. We apologize for the typo in the previous version and have corrected Appendix C.1 to clarify that TJH and CDSL share 50 clinical indicators, which are used by all methods. The dashed line in Table 2 separates transferable methods (transferring evolved experience across datasets) from non-transferable methods (which lack cross-dataset transfer and thus use their original Table 1 results as in-dataset reference baselines).

References

[1] Acute Respiratory Distress Syndrome: The Berlin Definition. JAMA, 2012.

[2] Circulatory Shock. NEJM, 2013.

[3] ClinicRealm: Re-evaluating Large Language Models with Conventional Machine Learning for Non-Generative Clinical Prediction Tasks. npj Digital Medicine, 2026.

[4] MedAgentBoard: Benchmarking Multi-Agent Collaboration with Conventional Methods for Diverse Medical Tasks. NeurIPS, 2025.

Reviewer_6QAbOpenReview ↗

Comments

Thanks for your reply. The scaling evolution experiment addresses one of my main concerns. I agree that improving agent skills through clinical use is an important and valuable research direction and have raised my score.

I believe it would be valuable to discuss a practical question: how can clinical skills be iteratively refined through human–AI collaboration, rather than through autonomous agent improvement alone? In clinical practice, software updates will always be reviewed by clinicians and are not deployed by the agent itself. That said, a stronger capacity for self-improvement could still provide a valuable foundation for this collaborative process. Moreover, as the number of cases grows, the context may become increasingly long. Could the authors clarify whether this raises concerns about the method’s scalability?

I encourage the authors to discuss these points in the paper.

AuthorsOpenReview ↗

Response to Reviewer #6QAb#

We appreciate your positive assessment and insightful suggestions, and have incorporated both discussions into the revised manuscript (Appendix E.5–E.6, p. 25).

Q1: Human–AI Collaboration. OntoSkill provides a natural foundation for clinician-supervised skill evolution. OEP identifies skills requiring refinement, while OCE analyzes successful and failed reasoning traces to generate candidate skill revisions. In clinical practice, clinicians can review, revise before validation and deployment (between Eqs. (21) and (22), p. 7), with their feedback incorporated into subsequent evolution rounds. Our current clinical evaluation involved physicians in assessing ontology validity, reasoning order, and evolved-skill safety, which could be further extended by integrating interactive clinician feedback into the skill evolution and validation process for human–AI collaboration.

Q2: Context Scalability. To further clarify this issue, we discuss two scenarios: (1) Increasing the initial evolution set. A larger evolution set provides broader clinical experience, offering more comprehensive evidence for OEP-based skill prioritization and helping OCE identify more representative clinical reasoning patterns. However, processing more reasoning traces also increases the context length and computational cost of OCE, creating a trade-off between clinical experience coverage and context efficiency. By controlling the amount of contrastive evidence used by OCE and integrating Agent Memory for managing and retrieving historical experience, we can achieve a practical balance between the two [1]. (2) Continuously arriving clinical cases. When new clinical cases continuously arrive, there is similarly a trade-off between the timeliness of skill updates and the sufficiency of evidence for each update. More frequent updates can incorporate emerging clinical patterns sooner and reduce the context required for each update, but each update is based on less evidence and may be more sensitive to the current case distribution. Less frequent updates can accumulate broader clinical experience and support more stable skill refinement, but require longer contexts and delay adaptation to new clinical patterns. Periodic evolution, together with clinician-supervised skill validation and deployment, can support more timely and reliable clinical skill refinement [2].

Methods (DeepSeek-V4-Flash)MIMIC-IV Mortality AUROC (95% CI) ↑MIMIC-IV Mortality AUPRC (95% CI) ↑OCE Input Tokens ↓
OntoSkill (500 cases)95.52±2.53 [90.22, 99.03]83.01±7.61 [64.51, 94.01]25.68K
OntoSkill (1000 cases)96.79±1.64 [92.84, 99.14]83.69±7.43 [66.17, 94.60]40.96K

References

[1] A-Mem: Agentic Memory for LLM Agents. NeurIPS, 2025.

[2] Rethinking Clinical Trials for Medical AI with Dynamic Deployments of Adaptive Systems. npj Digital Medicine 2025.

Reviewer_f8EbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OntoSkill, an ontology-aware agentic skill evolution framework for EHR clinical prediction. It constructs a clinical ontology consisting of entities, relations, and reasoning rules to organize heterogeneous physiological measurements. Building on this ontology, OntoSkill introduces three components: Ontology-aware Semantic Grounding (OSG), Ontology-aware Empirical Prioritization (OEP), and Ontology-aware Contrastive Evolution (OCE), enabling clinical skills to be activated and refined through accumulated prediction experience. Experiments on MIMIC-IV, TJH, and CDSL demonstrate improvements in mortality and readmission prediction across multiple LLM backbones.

Critical strengths
  1. The paper provides a structured approach to organizing heterogeneous physiological measurements through clinical entities, relations, and reasoning rules, improving the clinical interpretability of LLM-based EHR prediction.
  2. The experiments cover multiple datasets and LLM backbones, with ablation studies and cross-dataset transfer evaluations demonstrating the effectiveness of ontology-guided skill evolution.
Critical weaknesses
  1. Insufficient justification and validation of clinical reasoning rules. Although the clinical relations are grounded in external medical knowledge, the construction and ordering of reasoning rules remain insufficiently justified. In particular, the paper does not clearly explain how candidate relation chains are generated or why specific reasoning orders, such as Conditional followed by Mutual, are clinically appropriate. Moreover, these rules remain globally fixed despite potential differences in patient conditions. While the paper includes a rule-removal ablation, it lacks direct evaluations of rule correctness, ordering validity, and patient-specific adaptability.
  2. Insufficient baseline comparisons and limited clinical generalizability. The experiments mainly compare OntoSkill with LLM-based reasoning methods and Skill-Pro, a general-purpose skill evolution framework, without evaluating more relevant medical skill evolution approaches such as SkeMex [1]. Furthermore, the evaluation is limited to binary mortality and readmission prediction, without comparisons against established deep learning models for EHR prediction, such as GraphCare [2], or evaluations on more complex clinical tasks, such as diagnosis prediction, treatment recommendation, and multi-step clinical reasoning. These limitations make it difficult to assess the broader applicability and practical advantages of the proposed ontology-guided LLM agent framework.

References
[1] Experience Makes Skillful: Enabling Generalizable Medical Agent Reasoning via Self-Evolving Skill Memory, NeurIPS, 2026.
[2] GraphCare: Enhancing Healthcare Predictions with Personalized Knowledge Graphs, ICLR, 2024.

Questions for authors
  1. How are candidate connected relation chains generated and validated before being passed to the Rule Builder? What clinical evidence justifies the ordering of relations within each rule? Can the authors provide an end-to-end construction example and independent clinical validation of the generated rules?
  2. Given that the clinical ontology and reasoning rules remain fixed, how does OntoSkill accommodate differences in clinical reasoning across patients?
  3. Can the authors include representative medical skill evolution methods (such as SkeMex [1]) and established deep learning baselines (such as GraphCare [2]) for EHR prediction under comparable evaluation settings?
  4. Can the authors extend the evaluation beyond binary mortality and readmission prediction to more complex clinical tasks, such as diagnosis prediction or treatment recommendation, to demonstrate the generalizability of the learned clinical skills?

References
[1] Experience Makes Skillful: Enabling Generalizable Medical Agent Reasoning via Self-Evolving Skill Memory, NeurIPS, 2026.
[2] GraphCare: Enhancing Healthcare Predictions with Personalized Knowledge Graphs, ICLR, 2024.

2 comments
AuthorsOpenReview ↗

Response to Reviewer #f8Eb#

We appreciate your insightful comments. We have carefully addressed your main concerns with additional experiments and clarifications. The revised manuscript has been uploaded to the submission system, with all revisions highlighted in red.

W1-1&Q1-1&Q1-3: Clarification of Relation and Rule Construction. We construct clinical relations and reasoning rules through three steps: (1) Relation Adjudication. We initialize entities C\mathcal{C} from the EHR feature catalog and retrieve medical evidence from MSD Manuals and StatPearls. Candidate CONDITIONAL, MUTUAL, and JOINT relations are proposed and adjudicated against the retrieved evidence, with only accepted relations retained in R\mathcal{R}. (2) Chain Generation. We enumerate distinct pairs of accepted relations that share a common entity node and apply structural checks to ensure valid compositions. (3) Rule Construction. The Rule Builder determines evidence-supported reasoning orders and validates relation dependencies and acyclicity. The resulting ontology O=(C,R,Q)\mathcal{O=(C,R,Q)} is frozen before skill evolution. We have added an end-to-end rule construction example in Fig. 7 (p. 16).

W1-2&Q1-2: Clinical Evaluation. Three physicians from a well-known local hospital, specializing in Critical Care Medicine, Pulmonary Medicine, and General Internal Medicine, assessed ontology-relation validity, clinical reasoning order, and evolved-skill safety across MIMIC-IV, TJH, and CDSL, covering 62 clinical relations and 23 reasoning rules in total. For example, they evaluated the clinical rationale of CONDITIONAL (Respiratory Rate | Body Temperature) → MUTUAL (Oxygenation, Respiratory Rate), examining whether the contextual interpretation appropriately precedes joint physiological assessment. Their assessments indicated that the ontology and evolved skills were generally aligned with RAG-retrieved medical references and reasonably consistent with clinical practice.

W1-3&Q2: Clarification of Patient-Specific Reasoning. Patient-specific reasoning is achieved through skills activated by each patient's physiological indicators and longitudinal trends. The fixed clinical ontology and rules provide a sound foundation for initial skills, which are refined through experience from patients with similar physiological states. This mirrors how physicians develop expertise: first acquiring foundational clinical knowledge through formal medical training, then adapting and refining their clinical reasoning through experience with different patients.

W2-1&Q3: Additional Comparisons. We have newly added comparisons with SkeMex under matched experimental settings, as reported in Table 1 (p. 7). Our comparisons focus on LLM-based methods with frozen backbones. In contrast, conventional EHR models typically rely on supervised training of task-specific neural architectures, including RNNs and GNNs (e.g., GraphCare's BAT-GNN), differing substantially in model architecture, parameter scale, and pretraining data. Thus, performance differences cannot objectively reflect the merits of the methods themselves.

MethodsMIMIC-IV Mortality AUROC↑MIMIC-IV Mortality AUPRC↑MIMIC-IV Readmission AUROC↑MIMIC-IV Readmission AUPRC↑
Direct94.08±2.8971.99±9.8663.86±7.6024.25±6.65
SkeMex [1]95.34±1.8080.51±6.7066.23±6.4724.33±6.67
ConCare [2]94.08±3.7080.65±6.9879.17±4.4264.27±6.97
GRASP [3]93.14±3.0372.55±8.3677.76±4.1762.42±7.08
OntoSkill95.52±2.5383.01±7.6170.18±6.1727.31±7.68
Note: Direct, SkeMex, and OntoSkill share the same frozen DeepSeek-V4-Flash backbone. ConCare (GRU-based) and GRASP (GCN-based) are trained using supervised learning on over 8k cases.

W2-2&Q4: Extension to Broader Clinical Tasks. Following [4], we further evaluate OntoSkill on multi-label discharge diagnosis prediction using structured EHR data from MIMIC-IV. OntoSkill achieves consistent gains over the Direct baseline, extending its effectiveness beyond binary outcome prediction.

Methods (DeepSeek-V4-Flash)Micro-F1 (%)↑
Direct34.47
+ OntoSkill37.46

References

[1] Experience Makes Skillful: Enabling Generalizable Medical Agent Reasoning via Self-Evolving Skill Memory, NeurIPS, 2026.

[2] ConCare: Personalized Clinical Feature Embedding via Capturing the Healthcare Context, AAAI 2020.

[3] GRASP: Generic Framework for Health Status Representation Learning Based on Incorporating Knowledge from Similar Patients, AAAI 2021.

[4] Less Can Be Better: Decomposing Clinical Data Modalities in Large Language Model-based Healthcare Applications, JAMIA 2026.

AuthorsOpenReview ↗

Response to Reviewer #f8Eb#

Thank you very much for your time and consideration. We would like to kindly follow up, as we have not yet received your response. As the system deadline (November 18, AOE) is approaching, we would greatly appreciate it if you could kindly provide your feedback at your earliest convenience.

Thank you very much for your understanding, time, and support!

Reviewer_KJTEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes OntoSkill, an ontology-aware agentic skill evolution framework for EHR clinical prediction. The framework constructs a clinical ontology consisting of entities, relations, and reasoning rules, and introduces three components: Ontology-aware Semantic Grounding (OSG), Ontology-aware Empirical Prioritization (OEP), and Ontology-aware Contrastive Evolution (OCE), to guide skill formation, selection, and refinement. Experiments on MIMIC-IV, TJH, and CDSL with three LLM backbones demonstrate competitive performance in mortality and readmission prediction.

Critical strengths
  1. The paper addresses a meaningful problem in EHR clinical prediction by exploring how structured clinical knowledge can guide the evolution of reusable reasoning skills.
  2. The proposed framework integrates clinical ontology into different stages of skill evolution. The overall methodology is well organized, and the roles of OSG, OEP, and OCE are clearly presented.
  3. The experiments cover multiple EHR datasets and LLM backbones, with comparisons against representative baselines. The additional ablation and transferability studies further support the potential effectiveness of the proposed framework.
Critical weaknesses
  1. The main contribution of OntoSkill is integrating clinical ontology into skill formation, prioritization, and evolution. However, the individual mechanisms resemble existing knowledge-guided reasoning, experience-driven skill refinement, and automatic prompt optimization. Although Table 4 provides ablation results, it remains unclear whether the improvements primarily come from ontology constraints or iterative LLM-based skill editing and validation. Stronger comparisons with ontology-free skill optimization methods would better establish the unique contribution of the proposed framework.
  2. The paper introduces an LLM Evaluator and an LLM Editor in Equations (19) and (21), but does not clearly specify their underlying models. It is unclear whether they share the same backbone as the prediction agent or use separate, potentially stronger models. This ambiguity affects reproducibility and makes it difficult to determine whether the improvements, particularly for smaller backbones, are attributable to the proposed method or external model capabilities.
  3. In Equation (14), OEP uses normalized skill-local cross-entropy losses to estimate each skill's responsibility for prediction errors. However, a higher local loss does not necessarily indicate a greater contribution to the final error, especially when skills provide correlated or complementary evidence. Since the final prediction aggregates evidence from multiple skills, the proposed score may not accurately capture each skill's actual influence on the prediction. Further justification and validation of this prioritization strategy are needed.
  4. The results show task-dependent and metric-dependent differences across LLM backbones, but these variations are not sufficiently analyzed. For example, on MIMIC-IV mortality prediction, Qwen3-8B achieves an AUROC of 92.73%, relatively close to Claude-Sonnet-4.6 (94.11%) and DeepSeek-V4-Flash (95.52%), while showing substantially lower AUPRC. It remains unclear how backbone capabilities and the skill evolution process contribute to these different performance patterns.
Questions for authors
  1. Could the authors provide comparisons with ontology-free skill evolution or automatic prompt optimization under matched experimental settings to better demonstrate the independent contribution of ontology constraints?
  2. Please clarify the specific LLMs used for the Evaluator and Editor, including whether they share the prediction backbone and how their configurations affect the reported results.
  3. Could the authors further justify the use of normalized skill-local cross-entropy losses for estimating error responsibility and compare OEP with alternative skill prioritization strategies?
  4. Please provide further analysis of the performance variations across LLM backbones and evaluation metrics, particularly the relatively small AUROC gaps but substantially larger AUPRC differences observed for Qwen3-8B on certain tasks.
3 comments
AuthorsOpenReview ↗

Response to Reviewer #KJTE#

Thanks for the constructive comments. We have carefully addressed your main concerns with additional experiments and clarifications. The revised manuscript has been uploaded to the submission system, with all revisions highlighted in blue.

Q1:Gain of Ontology. Empirically, we further compare OntoSkill with its w/o ontology variant, the general agentic skill evolution method Skill-Pro [1], and the automatic prompt optimization method GEPA [2] under the same experimental settings. OntoSkill consistently outperforms these methods. Conceptually, the ontology resolves semantic fragmentation in EHR data, enabling clinically meaningful skill formation and targeted evolution beyond generic experience-driven optimization.

Methods (DeepSeek-V4-Flash)MIMIC-IV Mortality AUROC↑MIMIC-IV Mortality AUPRC↑MIMIC-IV Readmission AUROC↑MIMIC-IV Readmission AUPRC↑
Skill-Pro [1]93.66±3.2076.29±9.4668.45±6.9625.73±7.53
GEPA [2]95.10±2.3880.27±6.7867.25±6.7223.44±6.34
w/o Ontology94.66±2.4078.54±8.4365.74±6.1023.24±5.66
OntoSkill95.52±2.5383.01±7.6170.18±6.1727.31±7.68

Q2: Clarification of Experimental Settings. To clarify our experimental setup, we have added the following to Implementation Details (p. 8): “For fairness, the prediction agent, Evaluator, and Editor share the same LLM backbone, differing only in role-specific prompts.”

Q3: Justification of OEP. Empirically, we compare OEP with Counterfactual Attribution (CA) under the same experimental settings. OEP selects the same skill as CA in 8 of 10 evolution rounds. In terms of computational cost, OEP uses skill-local loss as an efficient proxy for error responsibility. Whereas CA incurs K+1K+1 aggregation calls, with the computational cost scaling linearly with the number of activated skills, OEP reuses existing predictions and requires only a single aggregation, thereby achieving a favorable trade-off between prioritization effectiveness and computational efficiency.

Methods (DeepSeek-V4-Flash)Selection Consistency (CA as Reference)Number of Aggregations↓MIMIC-IV Mortality AUROC↑MIMIC-IV Mortality AUPRC↑
CA10/10K+195.86±2.0980.99±7.75
Random0/10193.75 ± 3.4078.76 ± 9.30
Activation Frequency0/10193.47±3.2780.99±8.51
OEP8/10195.52±2.5383.01±7.61

Q4: Clarification of Performance Disparities. As shown below, substantial AUPRC differences already exist among the original backbones, despite relatively small AUROC gaps. Importantly, OntoSkill consistently improves AUPRC by 10.64/13.55/11.02 pp. across Qwen3-8B, Claude-Sonnet-4.6, and DeepSeek-V4-Flash, respectively. These results indicate that the performance disparities primarily stem from backbone capabilities, while OntoSkill delivers consistent gains across different models.

BackboneMethodsMIMIC-IV Mortality AUROC↑MIMIC-IV Mortality AUPRC↑
Qwen3-8BDirect90.52±2.7444.05±9.50
+ OntoSkill92.73±2.3454.69±9.55
Claude-Sonnet-4.6Direct92.70±2.9160.91±10.39
+ OntoSkill94.11±3.1274.46±9.29
DeepSeek-V4-FlashDirect94.08±2.8971.99±9.86
+ OntoSkill95.52±2.5383.01±7.61

References

[1] Skill-Pro: Learning Reusable Skills from Experience via Non-Parametric PPO for LLM Agents, ICML 2026.

[2] GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning, ICLR 2026.

Reviewer_KJTEOpenReview ↗

Response to Authors

Thank you for the additional experiments and clarifications. Most of my concerns have been addressed.

However, Q4 remains partially unresolved. The authors show that the AUPRC differences already exist in the original backbones, but this does not explain why the AUROC values are relatively close while the AUPRC values differ substantially.

Overall, I appreciate the authors' efforts and will raise my rating from 2 to 3.

AuthorsOpenReview ↗

Thank you very much for your positive feedback and recognition (score increased from 2 to 3).

Thank you very much for your positive feedback and recognition. However, we noticed that the score currently displayed in the system is still 2 points. As the system deadline (November 18, AOE) is approaching, we would greatly appreciate it if you could kindly look into this matter at your earliest convenience and confirm whether the score can be updated before the deadline.

Thank you again for your understanding, support, and assistance!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank the AC and all reviewers for their time, effort, and constructive feedback.


Core Storyline with Contributions.

We addressed two challenges in EHR clinical prediction: clinical semantic fragmentation and insufficient structural guidance for skill evolution. To this end, we constructed an EHR-oriented clinical ontology and proposed OntoSkill, a unified ontology-aware agentic skill evolution framework that operationalized the ontology throughout the skill lifecycle, guiding where to form, which to evolve, and how to evolve skills.

Without LLM parameter updates, OntoSkill achieved top performance in 16/24 comparisons across three EHR datasets and three backbones. Cross-dataset and cross-model transfer improved AUPRC by up to 9.79% and 18.20%, respectively.


Summary of Reviews and Responses.

Reviews recognized the clinical relevance, methodological insight, and empirical breadth, and raised questions about the clarity of experimental details, the sufficiency of ablation studies, the breadth of baseline comparisons, and the need for clinical expert validation.

We further clarified the experimental protocols and expanded the ablation studies to evaluate the contributions of the ontology and skill evolution. We added comparisons with GEPA and SkeMex under matched experimental settings, and validated the feasibility and effectiveness of OEP through comparison with counterfactual attribution. We also invited three physicians to assess the clinical validity of ontology relations, clinical reasoning order, and evolved-skill safety, covering 62 relations and 23 reasoning rules.

Our revisions addressed the major concerns raised during review and were acknowledged in the reviewers’ follow-up feedback (with both Reviewers KJTE and 6QAb increasing their scores from 2 to 3), providing further empirical and clinical support for OntoSkill.

We sincerely hope the AC and reviewers will carefully consider our work and render a fair decision. Best wishes!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.