- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
Nowadys, LLM agents increasingly receive “skills”: folders of procedural instructions (a SKILL.md entry file with a name and description, plus reference files) that the agent can open on demand. The authors observe that even when an applicable skill is installed, agents often fail to apply it during autonomous work. Moreover, they claim that the same model with the same skill can behave very differently under different execution harnesses. This is called by the papers, Skill Enable But Underuse (EBU). The paper develops EBU-Bench (331 medical and biomedical tasks over 12 task groups, each paired with a skill). For this, the authors measure the EBU gap, which is defined as the difference between the accuracy when the gent is explicitly told to read and apply the skill (“Operationally Supported Skill Use”) minus the accuracy when it decides for itself (“Natural Skill Use”). The other contribution of the paper is the definition of EBU-Evolve. This metric compares paired executions of he same task under the two conditions, asks a reflection model to write down what the supported run did that the natural run did not, and rewrites the skill’s description and body accordingly.
Critical strengths
The main contribition and strong point of the paper is to show to which extent the same AI model uses a harness changes natural-use accuracy by a lot while leaving the other two conditions ( No Skill and Operationally Supported ) almost unchanged. The authors tested 9 AI models in two different environments (called harnesses), Claude Code and Codex. Each model got the same procedures and the same tasks. When the authors told the model to read and follow the procedure, both environments gave almost the same score. When the model was not confronted with a procedure at all, both also scored similarly
However, if the procedure was available for the model to decide for itself whether to use it, the scores are significantly different. In Claude Code the models averaged about 26%, and in Codex about 50%. Very probable, one can assume that the models in one environment rarely used the procedure. In Claude Code they used it in about 12% of runs, and in Codex in about 65%. The models weren’t smarter in one place. They just had different frequencies of using it.
What is also interesting is that fifferent harnesses expose skills in different ways: how the list of skills is shown, how a skill is loaded, and which tool the model calls to open it. Some models might be also tuned for particular harnesses, so the same model may react differently to the cues. The authors take this into account and show that interface problems can dominate a result. So part of what looks like model behavior may be plumbing.
Critical weaknesses
The reviewer tried to understand how this benchmark relate to known existing prior work. The authors cover this in the Related Work section (page 9) but only in words, failing to provide a quantitative comparison with any other benchmark. No experiment on SkillsBench, SWE-Skills-Bench or any other skill benchmark, so its imposible to assess whether the Natural-versus-Supported gap shows up there. There is also no comparison of scale, domain coverage or design in a table. No non-medical control. The title and framing are about medical agents, but nothing tests whether EBU is specific to medical tasks or general to skills. The paper does not provide a "how-to for measuring your own gap " to someone interested to apply in practice. The Natural-versus-Supported contrast is the paper’s method, but it isn’t offered as a recipe, and there’s no guidance on how many tasks or runs will be needed. There’s no limitations section, no section on how a practitioner should apply the findings, and no definition of tasks, scorers or evolved skills that can be reused. That last one limits how far an interested reader can reuse the work as it stands. If harnesses automatically inject the relevant skill into context based on the task (retrieval, routing or a forced read step), the Natural condition becomes the same as the Supported condition and thus the whole contribution of the paper is limited.
Questions for authors
How does this benchmark relate to other known benchmarks in the literature ? The title and framing are about medical agents, but nothing tests whether EBU is specific to medical tasks or general to skills. Is the EBU specific to this area only ? Stage 3 keeps only tasks that fail under Natural use and succeed under Supported use, mainly using GLM-5.1. How many screening runs were used per task? What is the gap on tasks selected by an independent screen, or on a fresh rerun? Without that, how much of the 43.94 pp is regression to the mean? Why do the models behave so differently for the harnesses ? Can the authors Ffx the discovery interface across harnesses, t vary one feature at a time and then classify each trace as not opened, opened but the rule not retrieved, retrieved but not applied, or applied wrongly? Which results from the current paper will still be relevant if LLM evolves towards harness vendors making skill loading automatic ? In this case, the whole contribution of the paper will stay relevant ?