- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
Summary
The paper introduces Streaming Embodied Skill Discovery (SESD). A vision-language model processes an ordered stream of manipulation videos chunk by chunk. For each manipulation event it predicts the temporal boundaries and a symbolic "skill call" (a schema plus argument bindings). It also maintains a persistent, append-only skill library: each event either reuses an existing schema or creates a new one. The authors build the Video2Skill benchmark from HD-EPIC (egocentric kitchen) and ROBOINTER (DROID/RH20T robot tabletop) videos. The reference segments are human-verified and organised into 48 canonical transformation classes. Evaluation has three parts:
- temporal coverage ();
- naming-invariant grouping agreement with the reference classes (pair precision/recall, ARI);
- CREATE/REUSE recall, derived from the first occurrence of each class or name.
The authors evaluate 19 open-source VLMs zero-shot under a unified paradigm (one model perceives and updates the library) and a factorized paradigm (event description first, then text-only library update). Unified models tend to over-merge and factorized models tend to fragment. They then fine-tune two backbones with oracle-history SFT, a counterfactual library-state augmentation (CLARE), and on-policy correction. Grouping improves, but the models rarely create new schemas: the final library holds about 21–22 schemas against 46 reference classes, and classes unseen in training are almost never created.
Critical strengths
Strengths
- Clarity. The paper is easy to follow, and the high-level problem statement and Figure 1 communicate the intended setting well.
- Breadth of zero-shot evaluation. Nineteen open-source VLMs of several sizes and families are evaluated under two inference paradigms. This gives a broad picture of current open-model behaviour.
- Two domains. The benchmark covers both human egocentric and robot manipulation video, and the reference annotations were manually verified (12,429 calls).
- Some diagnostic value. The contrast between over-merging (unified) and fragmentation (factorized), and the presentation-order analysis, are reasonable observations about how library-conditioned prediction fails.
Critical weaknesses
Weaknesses
A. Motivation
A1. The claimed benefit for embodied agents is never tested. The paper motivates SESD by arguing that "planning with these skills helps embodied agents generalize" and that schemas "can be composed by planners." However, it also states that schemas are "semantic abstractions … rather than executable controllers" (L265), and it defers downstream utility to future work (Limitations). No experiment links the discovered library to a planner, a policy or robot execution. A schema is only a name, a free-form definition and argument slots. It has no preconditions or effects, so it is unclear how a symbolic planner would parse or compose it. As evaluated, the task is streaming temporal action segmentation plus category discovery over video. The "reusable embodied skills" framing in the title and abstract is therefore not supported.
A2. It is unclear why a discovered skill library is needed at all. Existing pipelines already combine pretrained foundation models with structured symbolic planning, without learning a skill vocabulary from video. For example, TiPToP (Shen et al., 2026, arXiv:2603.09971) uses a VLM to ground language goals and GPU-parallel task-and-motion planning for execution. On a DROID-style platform, the same source as part of ROBOINTER, it matches or exceeds a strong VLA without task-specific robot training data. A frontier VLM/LLM could likewise write a symbolic domain for a TAMP planner, or output step-level plans directly. The paper does not explain what a progressively discovered library provides that these alternatives lack.
A3. The value of "discovery" is undermined by the paper's own evaluation. A library is judged good only to the extent that it reproduces a hand-curated set of 48 classes. If a curated taxonomy is available and treated as correct, it is unclear why it should be rediscovered online instead of simply adopted. The case for discovery has to come from a criterion that is independent of this reference, such as downstream task success. No such criterion is provided.
Requested evidence for A1–A3. At minimum, a simulated downstream study (e.g., LIBERO or CALVIN) in which the same planner or hierarchical policy uses each of the following:
- the discovered library;
- the 48-class reference;
- a fixed primitive set;
- a VLM + TAMP pipeline such as TiPToP;
- direct VLM planning without a library.
Report task success. The authors should also show that higher Video2Skill scores predict better downstream performance. Otherwise it is unclear why ranking models on this benchmark matters for embodied agents.
B. Benchmark and method
B1. The benchmark has no grounded notion of a useful schema; this is the fundamental flaw of the evaluation. Every grouping and create/reuse metric rewards agreement with one hand-curated partition of 48 classes. This leaves the paper in a dilemma.
- If this partition is the correct target, there is little reason to discover skills: the curated vocabulary could simply be used, for example by classifying into it. After fine-tuning, the task already largely reduces to this. Train and test are disjoint by video but not by class: all 15 ROBOINTER test classes and 35 of the 46 HD-EPIC test classes appear in training.
- If it is not the correct target, then the metrics do not measure what matters. A library that is consistently coarser or finer than the reference, for instance 20 well-chosen schemas instead of 46, may be equally or more useful downstream (e.g., a smaller action space for a planner), yet it is scored as a failure.
The headline conclusion that libraries "stall below half the reference size" depends entirely on this unvalidated reference, whose granularity is not justified: no annotation protocol, agreement study or class list is provided. The paper also offers no evidence that agreement with the reference correlates with downstream utility. Without such a link (cf. A1–A3), the benchmark measures conformity to a particular annotation scheme rather than the quality of a skill library.
B2. The "progressive" discovery is too restricted to support the paper's claims. The library is append-only (). Once a schema is created, it can never be merged, split or revised, so any early duplication or over-merging becomes permanent. Progressively building a taxonomy should involve refining earlier abstractions as more evidence arrives, but the formulation rules this out. In addition, only this text library persists across videos; no past event instances are retained. The paper also never shows that the persistent library helps: there is no comparison with resetting it at every video or removing it entirely. The order study (Fig. 2) shows a final library size of about 21–22 regardless of presentation order. This suggests that library growth reflects the model's prior vocabulary rather than the experience stream.
B3. Key baselines are missing.
- No proprietary frontier VLMs. All 19 evaluated models are open-source. If frontier models largely solve the task zero-shot, the claim that novelty recognition is "the central challenge" weakens considerably.
- No closed-set or trivial baselines. Natural ones include a classifier over the training classes plus a "new" option, and the strategies "always one name" and "always a new name."
- No human agreement. Without an upper bound, absolute scores such as "ARI " cannot be interpreted.
B4. The metrics are easy to game and miss key parts of the output.
- Free predictions. Unmatched predictions are not penalised, and one prediction can match several references, which inflates pair recall.
- Degenerate strategies maximise individual metrics. A single shared name gives at ; unified LLaVA-OV-2-8B on ROBOINTER does exactly this.
- Reuse R does not check correctness. It only checks whether a name occurred before, not whether the right schema was reused.
- Grouping metrics depend on coverage. They are computed only on matched events, so covering fewer events can raise them (MiniCPM-V-4.5 has the best HD-EPIC ARI at 7.5% coverage).
- Argument bindings are never evaluated. These are what would make the schemas symbolic and composable.
A single primary metric that combines temporal precision, coverage and grouping is needed.
B5. Clarity and reproducibility.
- Formulation. is called a "policy," although there is no environment or reward. The event tuple is not fully defined. is undefined and appears to be the reference class . CONTINUE and "library-management transactions" are used but never defined. Chunking and "balanced order" are not specified.
- No supplementary material or code. There is no appendix (no prompts, output formats, training details, annotation protocol or class definitions) and no released code, so the benchmark and the streaming implementation cannot be reproduced or verified.
- Statistics. No variance or seeds are reported, and several conclusions rest on very small subsets (e.g., 11 unseen classes, 59 segments).
- Text and table disagree. Numbers quoted in Sec. 4.2 (30.0 / 12.7 / 69.9) do not match Table 2 (36.7 / 12.5 / 61.4).
Questions for authors
N/A
5 comments
Response to Reviewer bESG (2/n)
B1. Could a library of 20 coarser schemas be useful? Why treat disagreement with the reference as failure?
A library of 20 schemas is not a failure by itself; 46 is not a success by itself. The reference sets the scoring granularity, and models choose their own names. Pair Precision penalizes merging distinct transformations (e.g., placing and throwing); Pair Recall penalizes splitting recurring ones.
Training can improve several abilities together. In Table 2, fine-tuning unified LLaVA-OneVision-2-8B raises event coverage from 37.8 to 54.1, grouping ARI from 0.0 to 60.9, and Create Recall from 7.1 to 38.5 on RoboInter, while Reuse Recall remains high (98.6).
The new API runs show strength on more than one metric. Among the four API models, Opus 5.5 ranks first on both grouping and creation on HD-EPIC (ARI 27.1, Create Recall 48.8). Its library is larger than the reference (67 vs. 46), illustrating why we assess these abilities directly rather than ranking libraries by size.
Why not supply the reference vocabulary? Pre-seeding Sonnet 5.5 with all 48 reference classes improves grouping, but the model still uses manipulate for 217 events and misses 12 classes.
| ARI ↑ | Discover from an empty library | Start with all 48 classes |
|---|---|---|
| RoboInter | 37.1 | 44.2 |
| HD-EPIC | 14.7 | 15.9 |
Why do grouping errors matter downstream? A missing wash schema leaves the planner without a cleaning action.
B2. What does an append-only, persistent library add?
Keeping the library across videos helps the model recognize recurring skills. To isolate this effect, we ran Sonnet 5.5 with and without resetting its library at each video boundary during rebuttal.
| Dataset | Metric ↑ | Persistent library | Reset after each video |
|---|---|---|---|
| RoboInter | Pair Recall | 52.1 | 29.4 |
| RoboInter | ARI | 37.1 | 27.2 |
| HD-EPIC | Pair Recall | 22.6 | 11.1 |
| HD-EPIC | Reuse Recall | 95.6 | 81.4 |
| HD-EPIC | Event Coverage | 57.7 | 47.7 |
This control tests persistence; it does not compare different rules for revising the library.
B3–B4. Are the baselines and metrics enough to rule out trivial strategies?
The metrics are designed to be read together. Coverage asks how many events the model finds. Pair Precision, Pair Recall, and ARI ask how well it groups the events it found. Create and Reuse Recall track when it introduces or repeats a name. Separating these scores lets us tell a model that misses events apart from one that finds them but groups them poorly. One RoboInter run in §3 gets 100.0 on both Pair Recall and Reuse Recall but 0.0 ARI: the full metric set exposes the collapse to a single name. We added four closed-source API baselines during rebuttal:
| Model | RoboInter ARI ↑ | HD-EPIC ARI ↑ | HD-EPIC library size (reference: 46) | HD-EPIC Create R ↑ |
|---|---|---|---|---|
| Opus 5.5 | 55.7 | 27.1 | 67 | 48.8 |
| Sonnet 5.5 | 37.1 | 14.7 | 48 | 32.6 |
| Sonnet 4.6 | 8.2 | 10.0 | 60 | 32.5 |
| Haiku 4.5 | 6.1 | 12.3 | 13 | 16.2 |
| Best open-source ARI in the paper, per dataset | 27.0 | 25.6 | — | — |
Opus 5.5 improves grouping most clearly on RoboInter, but its HD-EPIC Create Recall is still below 50.
| Additional check | Result |
|---|---|
| One name for every event | Pair Recall 100, ARI 0 |
| A new name for every event | Create Recall 100, despite no reuse |
| Penalizing misses (ARI_all) | MiniCPM-V-4.5 on HD-EPIC: 25.6 → 0.5 when unmatched reference events are included as singletons |
| Checking whether the right name was reused | Qwen3.5-4B: Reuse Recall 99.3 / 98.5 vs. 67.8 / 49.8 when reuse must match the reference class's majority name (RoboInter / HD-EPIC) |
The new scores complement this diagnosis. One-to-one Event Precision and F1 measure extra predictions; Pair R_all and ARI_all show how missed events affect grouping across the full reference set. This adds an end-to-end view while preserving the ability to tell whether errors come from finding events or grouping them.
Response to Reviewer bESG (3/n)
B5. Can the evaluation be reproduced, and are the reported numbers consistent?
We will update the appendix with the prompts, evaluation and training settings, and annotation rules needed to reproduce the results.
Thank you for pointing this out. Table 2 reports the final results. We will update the three values quoted in §4.2 to match it: RoboInter Pair Precision (36.7), HD-EPIC Pair Precision (12.5), and HD-EPIC Event Coverage (61.4). The trends described in §4.2 remain unchanged.
Response to Reviewer bESG (1/n)
Thank you for asking what the discovered skills can do beyond matching the reference annotations. To test their usefulness directly, we ran new downstream experiments during rebuttal. We also tested what happens when the library is reset and added stricter scoring and four API baselines.
A1–A3. Why discover a library, and what is it useful for if it has not been tested in a planner?
We ran two downstream evaluations during rebuttal: action selection in ALFWorld and closed-loop robot execution in LIBERO.
1. ALFWorld: Can the library help a planner choose actions?
ALFWorld is a text-based household simulator. We gave the same Qwen3-32B planner 48 unseen tasks and varied its action vocabulary. Skill calls were mapped to environment commands via reference classes (e.g.,
wash→clean). The environment judged task success. Each condition was run once.washA library changes which actions the planner actually considers. The planner could issue
cleanwithout a library but never did on these tasks. A schema discovered from kitchen video made that action available as an explicit choice. An incomplete library can also hurt, as the Qwen3.5-4B result shows. What matters is whether the library covers the actions the task needs.2. GPT-6-astra as a robot policy: Can it execute the actions in LIBERO?
We paired the video-discovered schemas with scripted motion examples from separate simulator episodes, using reference-class alignment. GPT-6-astra received a task instruction, camera image, and ground-truth object coordinates, then generated arm and gripper waypoints. A controller executed them, and the policy used the updated observation to choose its next action. LIBERO judged task success. Only the prompt context changed.
Knowing what to do does not tell the policy how to do it. The name
graspand the reported bottle center do not specify the higher grasp point. The object-specific examples supply that detail. Examples alone already solve 10/12 of these trials, so the main gain here comes from the motion knowledge the library carries.Does it matter which skill an example is attached to? We tested this on a simpler LIBERO suite with Qwen3-32B and the same four generic examples in each condition:
With the same examples and schemas, correct assignment outperformed shuffled assignment.
Together, these tests show how a skill library can carry knowledge across tasks: it makes useful actions explicit and can store details about how to execute them.