acceptodds
Desk rejected at ICLR 2027

MedSkill-E²: Evaluating and Evolving Skills for Reliable Execution in Medical Agents

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language model (LLM) agents increasingly use skills to access domain knowledge and complete real-world tasks. We observe that even when a skill specifies the rules and procedures needed to complete a task, agents may still fail to apply them effectively during autonomous execution, a phenomenon we call skill Enable But Underuse (EBU). This issue is particularly consequential in medical tasks, where overlooking even a single critical constraint can lead to a plausible but incorrect result. To systematically evaluate and address this phenomenon, we introduce MedSkill-E², a framework for evaluating and evolving skills for medical agents. We construct EBU-Bench, a challenging benchmark covering 12 medically relevant task groups and their associated skills, to evaluate agents' ability to use available skills autonomously. We quantify the EBU gap as the difference in skill-use accuracy between autonomous use and explicitly guided reading and application of the same skill. On EBU-Bench, all evaluated model–harness combinations exhibit EBU, with a mean gap of 43.94 percentage points (pp). For the same model, skill-use accuracy under autonomous use differs by up to 38.96 pp across harnesses. Motivated by these observations, we propose EBU-Evolve, which uses execution feedback from autonomous and explicitly guided skill use to distill effective guidance into reusable skill instructions. Relative to their pre-evolution versions, the revised skills improve skill-use accuracy during autonomous execution by 33.33–55.17 pp and continue to yield improvements when transferred across multiple models and harnesses. Our results show that skill-use accuracy under autonomous execution varies substantially across harnesses even when the model and skill are held fixed, and model rankings also change; EBU can persist even after a skill is opened. EBU-Evolve distills effective guidance into reusable skill instructions, improving skill-use accuracy under autonomous execution on unseen tasks within the same skill, and still yields improvements when the evolved skills are transferred to other models and harnesses.

Primary area:
datasets and benchmarks
Keywords:
Medical Agent;AI Agent;Benchmark, Skill

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4ekLOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

Nowadys, LLM agents increasingly receive “skills”: folders of procedural instructions (a SKILL.md entry file with a name and description, plus reference files) that the agent can open on demand. The authors observe that even when an applicable skill is installed, agents often fail to apply it during autonomous work. Moreover, they claim that the same model with the same skill can behave very differently under different execution harnesses. This is called by the papers, Skill Enable But Underuse (EBU). The paper develops EBU-Bench (331 medical and biomedical tasks over 12 task groups, each paired with a skill). For this, the authors measure the EBU gap, which is defined as the difference between the accuracy when the gent is explicitly told to read and apply the skill (“Operationally Supported Skill Use”) minus the accuracy when it decides for itself (“Natural Skill Use”). The other contribution of the paper is the definition of EBU-Evolve. This metric compares paired executions of he same task under the two conditions, asks a reflection model to write down what the supported run did that the natural run did not, and rewrites the skill’s description and body accordingly.

Critical strengths

The main contribition and strong point of the paper is to show to which extent the same AI model uses a harness changes natural-use accuracy by a lot while leaving the other two conditions ( No Skill and Operationally Supported ) almost unchanged. The authors tested 9 AI models in two different environments (called harnesses), Claude Code and Codex. Each model got the same procedures and the same tasks. When the authors told the model to read and follow the procedure, both environments gave almost the same score. When the model was not confronted with a procedure at all, both also scored similarly

However, if the procedure was available for the model to decide for itself whether to use it, the scores are significantly different. In Claude Code the models averaged about 26%, and in Codex about 50%. Very probable, one can assume that the models in one environment rarely used the procedure. In Claude Code they used it in about 12% of runs, and in Codex in about 65%. The models weren’t smarter in one place. They just had different frequencies of using it.

What is also interesting is that fifferent harnesses expose skills in different ways: how the list of skills is shown, how a skill is loaded, and which tool the model calls to open it. Some models might be also tuned for particular harnesses, so the same model may react differently to the cues. The authors take this into account and show that interface problems can dominate a result. So part of what looks like model behavior may be plumbing.

Critical weaknesses

The reviewer tried to understand how this benchmark relate to known existing prior work. The authors cover this in the Related Work section (page 9) but only in words, failing to provide a quantitative comparison with any other benchmark. No experiment on SkillsBench, SWE-Skills-Bench or any other skill benchmark, so its imposible to assess whether the Natural-versus-Supported gap shows up there. There is also no comparison of scale, domain coverage or design in a table. No non-medical control. The title and framing are about medical agents, but nothing tests whether EBU is specific to medical tasks or general to skills. The paper does not provide a "how-to for measuring your own gap " to someone interested to apply in practice. The Natural-versus-Supported contrast is the paper’s method, but it isn’t offered as a recipe, and there’s no guidance on how many tasks or runs will be needed. There’s no limitations section, no section on how a practitioner should apply the findings, and no definition of tasks, scorers or evolved skills that can be reused. That last one limits how far an interested reader can reuse the work as it stands. If harnesses automatically inject the relevant skill into context based on the task (retrieval, routing or a forced read step), the Natural condition becomes the same as the Supported condition and thus the whole contribution of the paper is limited.

Questions for authors

How does this benchmark relate to other known benchmarks in the literature ? The title and framing are about medical agents, but nothing tests whether EBU is specific to medical tasks or general to skills. Is the EBU specific to this area only ? Stage 3 keeps only tasks that fail under Natural use and succeed under Supported use, mainly using GLM-5.1. How many screening runs were used per task? What is the gap on tasks selected by an independent screen, or on a fresh rerun? Without that, how much of the 43.94 pp is regression to the mean? Why do the models behave so differently for the harnesses ? Can the authors Ffx the discovery interface across harnesses, t vary one feature at a time and then classify each trace as not opened, opened but the rule not retrieved, retrieved but not applied, or applied wrongly? Which results from the current paper will still be relevant if LLM evolves towards harness vendors making skill loading automatic ? In this case, the whole contribution of the paper will stay relevant ?

Reviewer_hxwBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates a failure mode of LLM agents in medical application setting, where an agent given a skill description does not actively use that skill for the target task. It introduces a medical domain benchmark to measure the gap between with and without explict guidance in the context for using the provided skill. The experiments show that LLM agents have a large performance gap, termed as the phenomenon of Enable But Underuse. This paper further proposes a method to revise these skills based on a paired agent traces with and without the guidance for one medical task. The experiments show that this largely help improve the agents' skill usage.

Critical strengths
  • Interesting problem: this paper studies an interesting and practial agent application problem.

  • Domain-specific benchmark: this paper curates a highly specializied domain-specific benchmark in medical domain, which should be beneficial for the community..

Critical weaknesses
  • Unclear practial significance: The main concer I have for this paper is that, this operational-support condition requires knowing which skill and procedure match the task. The main failure for LLM agents shown in this paper is that they simply did not explictly use the provided skill. However, for a real-world medical agent usage, people would deliberately ask the agents to carefully select a skill for usage since medical application is a critical domain. It's unclear in which practical setting that this EBU gap may really occur.

  • Not specific to the medical domain: The paper frames EBU as a medical problem, but the phenomenon itself does not seem specific to medicine. It's unclear what makes the benchmark and the method medical beyond the choice of tasks.

  • The EBU gap is built by task selection: The curated benchmark mainly keeps tasks that fail under natural use and succeed under guidance with one LLM. Therefore, the large performance gap is largely a result of the construction process. It's unclear how often this problem happens without this filtering step. Authors should provide an analysis of the ratio of such phenomenon in natural medical tasks.

  • Missing simple baseline: The evaluation setup mainly compares simple usage where the skill description are put in the system prompt for agents. However, simply revising the skill on with/without guidance instruction recover most of the performance gap between natural skill usage and operationlly supported skill use, which raises the question whether simple system prompt optimziation would already fix the EBU gap, e.g., add several instruction in the system prompt for the agent to deliberately reason which skill for usage to solve the medical task.

Questions for authors

Please see weakness.

Reviewer_rhkwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MedSkill-E², comprising a challenge benchmark for skill underuse and a skill-rewriting method. EBU-Bench contrasts autonomous execution with explicit instructions to read and apply the associated skill. EBU-Evolve uses paired execution traces to revise skill descriptions and bodies. Experiments show large performance gaps across execution conditions and substantial gains from revised skills, including transfer across models and harnesses.

Critical strengths
  • Skill availability, retrieval, and successful application are meaningfully different, especially for tasks governed by explicit procedural specifications.
  • The nine-model, two-harness evaluation provides useful evidence that skill performance depends strongly on the execution environment.
  • The paper includes generic prompting, skill-opening analyses, optimizer comparisons, description/body ablations, and frozen-artifact transfer. It explicitly identifies the benchmark as EBU-enriched rather than representative.
Critical weaknesses
  1. It seems that the EBU gap does not isolate skill underuse. Operational support jointly changes resource identification, reading requirements, procedural emphasis, and potentially computational behavior. The outcome difference therefore measures the benefit of a bundled intervention. Opening-status analyses are informative but do not disentangle these factors. Controls separating forced loading, explicit selection, procedural reminders, and skill-independent execution guidance are needed.
  2. Description-only editing retains 85.42% of the joint improvement (Appendix B.8.1). The description may directly supply the decisive rule. This is a useful result, but it weakens attribution to improved acquisition and execution of the skill body. A description-only condition with the body unavailable, and a simple rule-promotion baseline, would clarify the mechanism.
  3. Comparisons with GEPA, SkillOpt, and generic rewriting are welcome. However, the generic control differs simultaneously in paired evidence, example selection, and utilization diagnosis. It cannot identify which component explains the improvement. In the separate generic-rewriting comparison, the advantage is seven answers out of 87, five from one task group; repeated optimization runs and component-level ablations are needed.
  4. Final-answer scoring does not directly verify procedural adherence. The Lite benchmark also heavily weights a small family of acid-base calculations, while related task variants complicate task-level uncertainty estimates. Moreover, the Atria clinical-screening regression demonstrates that aggregate gains can conceal deterioration. Source-clustered uncertainty, repeated runs, and regression reporting would materially strengthen the conclusions.
Questions for authors

Answer the questions raised in the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.