- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces EmbodiedSkillBench, a benchmark for studying how reusable embodied skills affect LLM agents on manipulation tasks. The authors define a common skill format and build a repository of 250 skills from 1,052 existing resources, including action, perception, and experience skills. The benchmark evaluates native control, skill-based execution, and Agent-Directed Reuse (ADR) on 100 perturbed LIBERO-Pro tasks. The main finding is that skills are most useful when they provide capabilities that the base agent lacks. For weaker agents, they substantially improve task success and reduce execution cost. For a stronger agent, however, prescribed skill workflows can hurt performance, and skills are more useful as optional tools for delegation. I see the main contribution as an empirical study of the marginal utility of reusable embodied capabilities across agents with different levels of native competence.
Critical strengths
-
The paper brings a broad set of heterogeneous embodied resources into a common format and provides a benchmark for studying how agents use them. Beyond the size of the repository itself, this gives a useful common setting for comparing skill selection, composition, and reuse across agents.
-
The most interesting result is that skills are not uniformly helpful. They fill capability gaps for weaker agents, but can constrain a stronger agent when used through a prescribed workflow. The GPT-6-Astra experiments suggest a useful shift from capability augmentation toward selective, efficiency-oriented delegation as base agents become stronger.
Critical weaknesses
-
The main results lack uncertainty estimates. The experiments appear to use a single rollout/seed per task and configuration (Clean100, seed 26), without repeated runs or confidence intervals. Given the stochasticity of LLM agents and learned robot policies, it is difficult to judge the robustness of some reported gaps, especially smaller differences such as 85% versus 92%. Multiple seeds or other uncertainty estimates would make the conclusions more convincing.
-
The experiments do not isolate the value of the embodied-skill abstraction itself. This is my main concern. In the primary comparisons, the skill-enabled agent often receives additional VLA controllers, perception modules, and procedural guidance, while the no-skill baseline mainly uses native low-level robot control. As a result, the experiments clearly show that additional embodied capabilities can help, but they do not show how much of the gain comes from the proposed skill representation itself. A more controlled comparison would keep the underlying modules fixed and expose them either as raw tools/APIs or through the proposed skill format. This would help separate the value of the underlying VLA/perception modules from the value of skill packaging, instructions, and reuse mechanisms.
Questions for authors
Please see weaknesses.
4 comments
Repeated runs and uncertainty
W1: The main results lack uncertainty estimates and appear to use a single rollout/seed per task and configuration. Given LLM and policy stochasticity, repeated runs or confidence intervals are needed to assess robustness, especially for gaps such as 85% versus 92%.
We extend the GPT-6 Astra evaluation on LIBERO-Pro Clean100 across three random seeds. Table 2 reports the mean and sample standard deviation for native control, same-tool no skill, with skill, and agent-directed reuse (ADR). Native control achieves the highest mean success rate, while the with-skill setting uses fewer actions, tokens, and agent minutes. ADR recovers much of the success lost under the with-skill setting.
Table 2. GPT-6 Astra on LIBERO-Pro Clean100 across three random seeds. Entries report mean ± sample standard deviation across seeds. SR is in percent, Tok. in millions, and Time in minutes. Act. counts state-changing calls.
Overall
| Setting | SR (%) | Act. | Tok. (M) | Time (min) |
|---|---|---|---|---|
| With skill | 58.0±2.6 | 2.33±0.13 | 0.510±0.063 | 2.26±1.00 |
| No skill | 85.7±1.2 | 9.36±0.86 | 0.861±0.171 | 4.05±1.14 |
| Native control | 89.3±2.5 | 16.71±0.33 | 1.975±0.306 | 9.58±9.47 |
| Agent-directed reuse | 87.7±2.5 | 13.74±0.76 | 0.880±0.947 | 7.88±8.84 |
L10
| Setting | SR (%) | Act. | Tok. (M) | Time (min) |
|---|---|---|---|---|
| With skill | 31.9±4.8 | 3.11±0.38 | 0.810±0.170 | 3.29±1.52 |
| No skill | 72.2±2.4 | 16.31±1.78 | 1.708±0.490 | 7.35±2.84 |
| Native control | 72.2±2.4 | 26.50±0.89 | 3.690±0.834 | 15.44±15.92 |
| Agent-directed reuse | 61.1±2.4 | 25.17±3.14 | 1.617±1.751 | 13.58±14.90 |
Goal
| Setting | SR (%) | Act. | Tok. (M) | Time (min) |
|---|---|---|---|---|
| With skill | 59.5±7.4 | 1.94±0.22 | 0.318±0.032 | 1.70±0.79 |
| No skill | 86.9±2.1 | 8.33±0.72 | 0.743±0.155 | 3.91±1.07 |
| Native control | 96.4±3.6 | 13.12±1.46 | 1.380±0.118 | 7.05±6.26 |
| Agent-directed reuse | 94.0±4.1 | 10.80±1.03 | 0.768±0.817 | 6.85±7.71 |
Object
| Setting | SR (%) | Act. | Tok. (M) | Time (min) |
|---|---|---|---|---|
| With skill | 80.6±2.4 | 1.32±0.02 | 0.334±0.014 | 1.81±0.80 |
| No skill | 95.8±4.2 | 4.69±2.06 | 0.363±0.206 | 1.83±0.69 |
| Native control | 94.4±4.8 | 14.06±1.17 | 1.387±0.197 | 7.40±6.83 |
| Agent-directed reuse | 98.6±2.4 | 6.97±2.33 | 0.281±0.166 | 2.73±1.77 |
Spatial
| Setting | SR (%) | Act. | Tok. (M) | Time (min) |
|---|---|---|---|---|
| With skill | 59.7±2.4 | 3.00±0.57 | 0.609±0.129 | 2.32±0.93 |
| No skill | 87.5±4.2 | 8.26±0.50 | 0.652±0.070 | 3.12±0.34 |
| Native control | 93.1±2.4 | 13.75±0.52 | 1.542±0.350 | 8.86±9.43 |
| Agent-directed reuse | 95.8±4.2 | 12.53±2.41 | 0.874±1.077 | 8.53±11.19 |
We extend the no-skill/with-skill comparison to RoboCasa365 and to RoboDojo for gpt-5.6-sol. Skill guidance improves overall success on both benchmarks, with gains varying by task category. RoboCasa365 shows gains on Atomic Seen and Composite Seen, while Composite Unseen remains unchanged. RoboDojo shows gains on Open and Memory and decreases on Generalization and Long Horizon. These experiments provide evidence for transfer across simulation environments.
Table 3. With skill vs. no skill for gpt-5.6-sol/high, evaluated across three random seeds on RoboCasa365 and three random layouts on RoboDojo. Entries report mean ± sample standard deviation across seeds or layouts. SR is in percent, Tok. in millions, and Time in minutes. Act. counts agent-level actions.
| Benchmark | Task category | No skill SR | Act. | Tok. | Time | With skill SR | Act. | Tok. | Time |
|---|---|---|---|---|---|---|---|---|---|
| RoboCasa365 | Atomic Seen | 59.3±11.6 | 2.41±0.32 | 0.291±0.092 | 2.24±0.09 | 72.2±19.2 | 3.26±1.51 | 0.405±0.139 | 2.17±0.53 |
| Composite Seen | 12.5±12.5 | 10.92±0.95 | 2.926±1.031 | 11.47±3.37 | 22.9±3.6 | 9.60±1.16 | 1.483±0.161 | 9.65±2.27 | |
| Composite Unseen | 10.4±9.5 | 11.21±0.58 | 3.088±1.171 | 12.33±4.38 | 10.4±3.6 | 10.54±1.30 | 1.842±0.426 | 11.28±2.62 | |
| Overall | 28.7±9.9 | 7.95±0.50 | 2.029±0.734 | 8.42±2.48 | 36.7±4.6 | 7.62±0.33 | 1.210±0.122 | 7.48±1.45 | |
| RoboDojo | Generalization | 19.4±4.8 | 3.81±2.21 | 1.864±0.671 | 12.10±1.01 | 13.9±12.7 | 7.44±0.82 | 0.242±0.018 | 12.81±1.10 |
| Open | 8.3±7.2 | 9.13±2.49 | 1.926±0.593 | 11.66±1.15 | 25.0±0.0 | 6.83±0.80 | 0.215±0.032 | 9.55±1.56 | |
| Precision | 0.0±0.0 | 2.83±2.13 | 1.472±0.483 | 10.08±1.42 | 0.0±0.0 | 8.25±0.78 | 0.284±0.046 | 11.23±0.58 | |
| Long Horizon | 12.5±12.5 | 7.13±3.03 | 2.118±0.866 | 15.51±1.49 | 8.3±7.2 | 7.79±1.09 | 0.235±0.017 | 14.23±1.83 | |
| Memory | 0.0±0.0 | 13.39±4.99 | 3.634±1.064 | 15.83±1.52 | 11.1±9.6 | 6.67±1.20 | 0.180±0.032 | 9.98±1.60 | |
| Overall | 9.5±2.4 | 6.63±1.77 | 2.102±0.507 | 12.81±1.10 | 11.9±4.8 | 7.44±0.57 | 0.235±0.014 | 11.75±0.44 |
Dear Reviewer,
I hope this message finds you well. As the discussion period is nearing its end with less than two days remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are of great importance to us, and we’re eager to address any remaining issues to improve our work.
Thank you for your time and effort in reviewing our paper.
Thank you for the detailed rebuttal and additional experiments. The new same-tool baseline and multi-seed and cross-environment evaluations address most of my concerns. I appreciate the authors' efforts and have decided to update my assessment to weak accept.
Same-tool baseline
W2: The experiments do not isolate the value of the embodied-skill abstraction itself. A controlled comparison should keep the underlying modules fixed and expose them as raw tools/APIs or through the proposed skill format, separating module gains from skill packaging, instructions, and reuse mechanisms.
We add a controlled same-tool, no-skill comparison on LIBERO-Pro Clean100 for gpt-5.6-sol, Kimi-K3, and claude-opus-5, finding that skill guidance improves overall success and reduces action, token, and time costs for all three models (Table 1). In the added baseline, agents access the same underlying models and tools through their APIs as in the with-skill setting, allowing us to evaluate the additional value of the embodied-skill abstraction and its procedural guidance while holding tool availability fixed. We rename the previous no-skill setting native control to distinguish it from this new baseline. Table 1 summarizes the evaluation across the four task families and overall, and results vary by model and task family.
Table 1. Added “no skill” experiments on LIBERO-Pro Clean100 for gpt-5.6-sol, kimi-k3, and opus-5. SR is in percent, Tok. in millions, and Time in minutes.