acceptodds
Under review as a conference paper at ICLR 2027

Video2Skill: From Streaming Experience to Reusable Embodied Skills

Abstract

Manipulation behaviors vary widely across objects and scenes, but they share a small set of reusable skills, and planning with these skills helps embodied agents generalize to new tasks. Yet an agent can only plan with skills it knows. Recovering skills from observed experience, the inverse of planning, builds this knowledge over time and yields skill data for training future agents. Vision-Language Models (VLMs) describe individual manipulation events well, but can they organize a stream of events into reusable skills? We formulate this problem as Streaming Embodied Skill Discovery (SESD): a model watches videos in sequence and maintains a persistent skill library that shapes its later decisions. To systematically measure this ability, we introduce Video2Skill, a benchmark that covers robot tabletop manipulation and human kitchen activity and tests three core capabilities: (i) locating manipulation events in time, (ii) grouping events of the same transformation, and (iii) deciding when to reuse an existing skill or create a new one. Across 19 open-source VLMs, many models group events at near-chance level, and scale does not consistently help. Their errors depend on how perception and library updates are coupled: joint models merge distinct transformations into one skill, while models that update the library from text descriptions duplicate recurring ones. Supervised fine-tuning, including our Counterfactual Library-state Rebalancing (CLaRe), improves grouping but exposes a deeper bottleneck: trained models consolidate familiar skills yet rarely expand the library. Their libraries stall below half the reference size, and transformations unseen in training are located in time but almost never given a new skill. Recognizing when existing skills are insufficient thus emerges as the central challenge.

Primary area:
datasets and benchmarks
Keywords:
Embodied AI, Streaming Video Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bESGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

Summary

The paper introduces Streaming Embodied Skill Discovery (SESD). A vision-language model processes an ordered stream of manipulation videos chunk by chunk. For each manipulation event it predicts the temporal boundaries and a symbolic "skill call" (a schema plus argument bindings). It also maintains a persistent, append-only skill library: each event either reuses an existing schema or creates a new one. The authors build the Video2Skill benchmark from HD-EPIC (egocentric kitchen) and ROBOINTER (DROID/RH20T robot tabletop) videos. The reference segments are human-verified and organised into 48 canonical transformation classes. Evaluation has three parts:

  • temporal coverage (tIoU≥0.3\mathrm{tIoU} \geq 0.3);
  • naming-invariant grouping agreement with the reference classes (pair precision/recall, ARI);
  • CREATE/REUSE recall, derived from the first occurrence of each class or name.

The authors evaluate 19 open-source VLMs zero-shot under a unified paradigm (one model perceives and updates the library) and a factorized paradigm (event description first, then text-only library update). Unified models tend to over-merge and factorized models tend to fragment. They then fine-tune two backbones with oracle-history SFT, a counterfactual library-state augmentation (CLARE), and on-policy correction. Grouping improves, but the models rarely create new schemas: the final library holds about 21–22 schemas against 46 reference classes, and classes unseen in training are almost never created.

Critical strengths

Strengths

  1. Clarity. The paper is easy to follow, and the high-level problem statement and Figure 1 communicate the intended setting well.
  2. Breadth of zero-shot evaluation. Nineteen open-source VLMs of several sizes and families are evaluated under two inference paradigms. This gives a broad picture of current open-model behaviour.
  3. Two domains. The benchmark covers both human egocentric and robot manipulation video, and the reference annotations were manually verified (12,429 calls).
  4. Some diagnostic value. The contrast between over-merging (unified) and fragmentation (factorized), and the presentation-order analysis, are reasonable observations about how library-conditioned prediction fails.
Critical weaknesses

Weaknesses

A. Motivation

A1. The claimed benefit for embodied agents is never tested. The paper motivates SESD by arguing that "planning with these skills helps embodied agents generalize" and that schemas "can be composed by planners." However, it also states that schemas are "semantic abstractions … rather than executable controllers" (L265), and it defers downstream utility to future work (Limitations). No experiment links the discovered library to a planner, a policy or robot execution. A schema s=(ns,ds,As)s = (n_s, d_s, A_s) is only a name, a free-form definition and argument slots. It has no preconditions or effects, so it is unclear how a symbolic planner would parse or compose it. As evaluated, the task is streaming temporal action segmentation plus category discovery over video. The "reusable embodied skills" framing in the title and abstract is therefore not supported.

A2. It is unclear why a discovered skill library is needed at all. Existing pipelines already combine pretrained foundation models with structured symbolic planning, without learning a skill vocabulary from video. For example, TiPToP (Shen et al., 2026, arXiv:2603.09971) uses a VLM to ground language goals and GPU-parallel task-and-motion planning for execution. On a DROID-style platform, the same source as part of ROBOINTER, it matches or exceeds a strong VLA without task-specific robot training data. A frontier VLM/LLM could likewise write a symbolic domain for a TAMP planner, or output step-level plans directly. The paper does not explain what a progressively discovered library provides that these alternatives lack.

A3. The value of "discovery" is undermined by the paper's own evaluation. A library is judged good only to the extent that it reproduces a hand-curated set of 48 classes. If a curated taxonomy is available and treated as correct, it is unclear why it should be rediscovered online instead of simply adopted. The case for discovery has to come from a criterion that is independent of this reference, such as downstream task success. No such criterion is provided.

Requested evidence for A1–A3. At minimum, a simulated downstream study (e.g., LIBERO or CALVIN) in which the same planner or hierarchical policy uses each of the following:

  • the discovered library;
  • the 48-class reference;
  • a fixed primitive set;
  • a VLM + TAMP pipeline such as TiPToP;
  • direct VLM planning without a library.

Report task success. The authors should also show that higher Video2Skill scores predict better downstream performance. Otherwise it is unclear why ranking models on this benchmark matters for embodied agents.

B. Benchmark and method

B1. The benchmark has no grounded notion of a useful schema; this is the fundamental flaw of the evaluation. Every grouping and create/reuse metric rewards agreement with one hand-curated partition of 48 classes. This leaves the paper in a dilemma.

  • If this partition is the correct target, there is little reason to discover skills: the curated vocabulary could simply be used, for example by classifying into it. After fine-tuning, the task already largely reduces to this. Train and test are disjoint by video but not by class: all 15 ROBOINTER test classes and 35 of the 46 HD-EPIC test classes appear in training.
  • If it is not the correct target, then the metrics do not measure what matters. A library that is consistently coarser or finer than the reference, for instance 20 well-chosen schemas instead of 46, may be equally or more useful downstream (e.g., a smaller action space for a planner), yet it is scored as a failure.

The headline conclusion that libraries "stall below half the reference size" depends entirely on this unvalidated reference, whose granularity is not justified: no annotation protocol, agreement study or class list is provided. The paper also offers no evidence that agreement with the reference correlates with downstream utility. Without such a link (cf. A1–A3), the benchmark measures conformity to a particular annotation scheme rather than the quality of a skill library.

B2. The "progressive" discovery is too restricted to support the paper's claims. The library is append-only (Lt=Lt−1∪DtL_t = L_{t-1} \cup D_t). Once a schema is created, it can never be merged, split or revised, so any early duplication or over-merging becomes permanent. Progressively building a taxonomy should involve refining earlier abstractions as more evidence arrives, but the formulation rules this out. In addition, only this text library persists across videos; no past event instances are retained. The paper also never shows that the persistent library helps: there is no comparison with resetting it at every video or removing it entirely. The order study (Fig. 2) shows a final library size of about 21–22 regardless of presentation order. This suggests that library growth reflects the model's prior vocabulary rather than the experience stream.

B3. Key baselines are missing.

  • No proprietary frontier VLMs. All 19 evaluated models are open-source. If frontier models largely solve the task zero-shot, the claim that novelty recognition is "the central challenge" weakens considerably.
  • No closed-set or trivial baselines. Natural ones include a classifier over the training classes plus a "new" option, and the strategies "always one name" and "always a new name."
  • No human agreement. Without an upper bound, absolute scores such as "ARI ≤27\leq 27" cannot be interpreted.

B4. The metrics are easy to game and miss key parts of the output.

  • Free predictions. Unmatched predictions are not penalised, and one prediction can match several references, which inflates pair recall.
  • Degenerate strategies maximise individual metrics. A single shared name gives Pair R=Reuse R=100\text{Pair R} = \text{Reuse R} = 100 at ARI≈0\text{ARI} \approx 0; unified LLaVA-OV-2-8B on ROBOINTER does exactly this.
  • Reuse R does not check correctness. It only checks whether a name occurred before, not whether the right schema was reused.
  • Grouping metrics depend on coverage. They are computed only on matched events, so covering fewer events can raise them (MiniCPM-V-4.5 has the best HD-EPIC ARI at 7.5% coverage).
  • Argument bindings are never evaluated. These are what would make the schemas symbolic and composable.

A single primary metric that combines temporal precision, coverage and grouping is needed.

B5. Clarity and reproducibility.

  • Formulation. πθ\pi_\theta is called a "policy," although there is no environment or reward. The event tuple e=(be,de,se,ae)e = (b_e, d_e, s_e, a_e) is not fully defined. τ(⋅)\tau(\cdot) is undefined and appears to be the reference class cic_i. CONTINUE and "library-management transactions" are used but never defined. Chunking and "balanced order" are not specified.
  • No supplementary material or code. There is no appendix (no prompts, output formats, training details, annotation protocol or class definitions) and no released code, so the benchmark and the streaming implementation cannot be reproduced or verified.
  • Statistics. No variance or seeds are reported, and several conclusions rest on very small subsets (e.g., 11 unseen classes, 59 segments).
  • Text and table disagree. Numbers quoted in Sec. 4.2 (30.0 / 12.7 / 69.9) do not match Table 2 (36.7 / 12.5 / 61.4).
Questions for authors

N/A

5 comments
AuthorsOpenReview ↗

Response to Reviewer bESG (1/n)

Thank you for asking what the discovered skills can do beyond matching the reference annotations. To test their usefulness directly, we ran new downstream experiments during rebuttal. We also tested what happens when the library is reset and added stricter scoring and four API baselines.

A1–A3. Why discover a library, and what is it useful for if it has not been tested in a planner?

We ran two downstream evaluations during rebuttal: action selection in ALFWorld and closed-loop robot execution in LIBERO.

1. ALFWorld: Can the library help a planner choose actions?

ALFWorld is a text-based household simulator. We gave the same Qwen3-32B planner 48 unseen tasks and varied its action vocabulary. Skill calls were mapped to environment commands via reference classes (e.g., wash → clean). The environment judged task success. Each condition was run once.

Action vocabularyTask success ↑
No library29.2%
Fixed Video2Skill reference vocabulary41.7%
Library discovered by Sonnet 5.552.1%
Library discovered by Opus 5.545.8%
Library discovered by Qwen3.5-4B18.8%
Cleaning-task comparisonTasks solved ↑
No library0/17
Video-discovered libraries with a schema mapped to wash6–13/17

A library changes which actions the planner actually considers. The planner could issue clean without a library but never did on these tasks. A schema discovered from kitchen video made that action available as an explicit choice. An incomplete library can also hurt, as the Qwen3.5-4B result shows. What matters is whether the library covers the actions the task needs.

2. GPT-6-astra as a robot policy: Can it execute the actions in LIBERO?

We paired the video-discovered schemas with scripted motion examples from separate simulator episodes, using reference-class alignment. GPT-6-astra received a task instruction, camera image, and ground-truth object coordinates, then generated arm and gripper waypoints. A controller executed them, and the policy used the updated observation to choose its next action. LIBERO judged task success. Only the prompt context changed.

Prompt contextSuccess on trials requiring hidden grasp offsets
No prior1/12
Opus schemas + generic motion examples0/12
Opus schemas + generic and object-specific motion examples12/12

Knowing what to do does not tell the policy how to do it. The name grasp and the reported bottle center do not specify the higher grasp point. The object-specific examples supply that detail. Examples alone already solve 10/12 of these trials, so the main gain here comes from the motion knowledge the library carries.

Does it matter which skill an example is attached to? We tested this on a simpler LIBERO suite with Qwen3-32B and the same four generic examples in each condition:

Qwen3-32B policy contextSuccesses (15 trials) ↑
Opus schemas + examples under the wrong schemas3/15
Opus schemas + correctly assigned examples12/15

With the same examples and schemas, correct assignment outperformed shuffled assignment.

Together, these tests show how a skill library can carry knowledge across tasks: it makes useful actions explicit and can store details about how to execute them.

AuthorsOpenReview ↗

Response to Reviewer bESG (2/n)

B1. Could a library of 20 coarser schemas be useful? Why treat disagreement with the reference as failure?

A library of 20 schemas is not a failure by itself; 46 is not a success by itself. The reference sets the scoring granularity, and models choose their own names. Pair Precision penalizes merging distinct transformations (e.g., placing and throwing); Pair Recall penalizes splitting recurring ones.

Training can improve several abilities together. In Table 2, fine-tuning unified LLaVA-OneVision-2-8B raises event coverage from 37.8 to 54.1, grouping ARI from 0.0 to 60.9, and Create Recall from 7.1 to 38.5 on RoboInter, while Reuse Recall remains high (98.6).

The new API runs show strength on more than one metric. Among the four API models, Opus 5.5 ranks first on both grouping and creation on HD-EPIC (ARI 27.1, Create Recall 48.8). Its library is larger than the reference (67 vs. 46), illustrating why we assess these abilities directly rather than ranking libraries by size.

Why not supply the reference vocabulary? Pre-seeding Sonnet 5.5 with all 48 reference classes improves grouping, but the model still uses manipulate for 217 events and misses 12 classes.

ARI ↑Discover from an empty libraryStart with all 48 classes
RoboInter37.144.2
HD-EPIC14.715.9

Why do grouping errors matter downstream? A missing wash schema leaves the planner without a cleaning action.

B2. What does an append-only, persistent library add?

Keeping the library across videos helps the model recognize recurring skills. To isolate this effect, we ran Sonnet 5.5 with and without resetting its library at each video boundary during rebuttal.

DatasetMetric ↑Persistent libraryReset after each video
RoboInterPair Recall52.129.4
RoboInterARI37.127.2
HD-EPICPair Recall22.611.1
HD-EPICReuse Recall95.681.4
HD-EPICEvent Coverage57.747.7

This control tests persistence; it does not compare different rules for revising the library.

B3–B4. Are the baselines and metrics enough to rule out trivial strategies?

The metrics are designed to be read together. Coverage asks how many events the model finds. Pair Precision, Pair Recall, and ARI ask how well it groups the events it found. Create and Reuse Recall track when it introduces or repeats a name. Separating these scores lets us tell a model that misses events apart from one that finds them but groups them poorly. One RoboInter run in §3 gets 100.0 on both Pair Recall and Reuse Recall but 0.0 ARI: the full metric set exposes the collapse to a single name. We added four closed-source API baselines during rebuttal:

ModelRoboInter ARI ↑HD-EPIC ARI ↑HD-EPIC library size (reference: 46)HD-EPIC Create R ↑
Opus 5.555.727.16748.8
Sonnet 5.537.114.74832.6
Sonnet 4.68.210.06032.5
Haiku 4.56.112.31316.2
Best open-source ARI in the paper, per dataset27.025.6——

Opus 5.5 improves grouping most clearly on RoboInter, but its HD-EPIC Create Recall is still below 50.

Additional checkResult
One name for every eventPair Recall 100, ARI 0
A new name for every eventCreate Recall 100, despite no reuse
Penalizing misses (ARI_all)MiniCPM-V-4.5 on HD-EPIC: 25.6 → 0.5 when unmatched reference events are included as singletons
Checking whether the right name was reusedQwen3.5-4B: Reuse Recall 99.3 / 98.5 vs. 67.8 / 49.8 when reuse must match the reference class's majority name (RoboInter / HD-EPIC)

The new scores complement this diagnosis. One-to-one Event Precision and F1 measure extra predictions; Pair R_all and ARI_all show how missed events affect grouping across the full reference set. This adds an end-to-end view while preserving the ability to tell whether errors come from finding events or grouping them.

AuthorsOpenReview ↗

Response to Reviewer bESG (3/n)

B5. Can the evaluation be reproduced, and are the reported numbers consistent?

We will update the appendix with the prompts, evaluation and training settings, and annotation rules needed to reproduce the results.

Thank you for pointing this out. Table 2 reports the final results. We will update the three values quoted in §4.2 to match it: RoboInter Pair Precision (36.7), HD-EPIC Pair Precision (12.5), and HD-EPIC Event Coverage (61.4). The trends described in §4.2 remain unchanged.

Reviewer_kmHkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The work proposes a new task, Streaming Embodied Skill Discovery, which measures the ability of VLMs to summarize skills from videos. To evaluate this ability, the Video2Skill benchmark is proposed to measure event coverage, naming-invariant skill grouping, and creation–reuse decisions across streaming videos. Post-training improvements are performed but do not yield consistent benefits, which shows that the problem remains challenging.

Critical strengths
  1. Summarizing skills from videos may be an important ability for VLMs under agentic frameworks. The work defines the problem and constructs the Video2Skill benchmark to measure it. The evaluation includes both unified and factorized processing paradigms, which helps analyze the bottleneck.
  2. Post-training improvements are also performed to analyze the ability boundary. SFT on ground-truth history, data augmentation (CLARE), and on-policy correction are conducted, but they do not yield consistent benefits, which shows that the problem is challenging.
  3. Further discussions provide interesting analysis on discovery order, knowledge sharing across skills, and behavior on unseen transformations.
Critical weaknesses
  1. The diversity and accuracy of the dataset raise concerns. HD-EPIC contains only a few kitchen activity videos with low scenario diversity, while ROBOINTER comes from DROID and RH20T with low visual quality and event diversity. The distribution of the training and test datasets greatly influences performance. Given such limited diversity, the consistency of the evaluation is also questionable.
  2. The definition of events is crucial, and how annotation quality is ensured needs further clarification. For example, "pick object," "approach object," and "grasp object" represent different granularities of event definition for the same video clip.
  3. Beside the description of event, robot skill has more broader requirements, such as failure detection, correction, grasping pose for object with specific pose.
  4. Leading video understanding models are not included, such as API models.
Questions for authors

This work finds an interesting and valuable research topic, but seems not ready to be a solid benchmark. Could you show the definition of the event and the annotated skill distribution among train and test dataset?

1 comment
AuthorsOpenReview ↗

Response to Reviewer kmHk

Thank you for your helpful comments. We appreciate your recognition of the value of the task, the two inference approaches, and the analysis of unseen transformations. We respond to each concern below.

1. How diverse are the data, and what is the train/test skill distribution?

The benchmark spans first-person kitchen activity (HD-EPIC) and robot tabletop manipulation (RoboInter, drawn from DROID and RH20T). No source video is shared between training and test.

DatasetTraining dataTest dataTest transformation classes
HD-EPIC42 annotated videos; 5,726 calls40 clips from 29 held-out recordings; 1,137 segments46
RoboInter2,992 videos; 5,032 verified calls132 videos; 534 verified segments15

2. What counts as an event, and how do you keep annotation granularity consistent?

An event is one manipulation with a start and end time, expressed as a skill call.

What we checkExample or evidence
Event granularityA reach alone does not count as a grasp.
Naming consistencyCalling every grasp take_hold instead of grasp leaves grouping scores unchanged.
Annotation qualityVideo review corrected 241 of 12,429 verified calls and removed unsupported RoboInter events.

3. Is this the same kind of skill a robot needs to execute an action?

The original paper focuses on language-level skills. As explained in Section 2.1, a skill schema describes a physical transformation, such as grasping or placing an object, in language with named arguments. It does not specify how the robot should move.

During the rebuttal period, we added a closed-loop experiment at the action level. In LIBERO, GPT-6-astra used the video-discovered schemas and scripted motion examples to generate arm and gripper waypoints. A controller executed them, and the policy used the updated observations to choose its next action. The simulator judged task success.

Policy contextAll trials ↑Trials needing a hidden grasp offset ↑
No library or examples31/481/12
Video-discovered schemas + generic motion examples33/480/12
Video-discovered schemas + all motion examples48/4812/12

The goal is to generalize from streaming experience. A model should recognize the same skill across different objects and scenes, then reuse it when facing a new task. Our experiment tests this transfer by pairing schemas discovered from video with motion examples from separate simulator episodes. The examples supply most of the execution gains; the schemas organize them into skills the policy can call again.

4. Why were strong API models missing?

Thank you for suggesting this comparison. We added four API models:

ModelRoboInter ARI ↑HD-EPIC ARI ↑HD-EPIC library size (reference: 46)HD-EPIC Create R ↑
Opus 5.555.727.16748.8
Sonnet 5.537.114.74832.6
Sonnet 4.68.210.06032.5
Haiku 4.56.112.31316.2
Best open-source ARI in the paper, per dataset27.025.6——

Opus gains 28.7 ARI points on RoboInter and 1.5 on HD-EPIC. Its Create Recall of 48.8 still leaves over half of first matched occurrences without a new schema name.

Reviewer_k7UrOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the problem of streaming skill discovery and reuse in embodied intelligence by introducing the Streaming Embodied Skill Discovery (SESD) task and constructing the Video2Skill benchmark. The benchmark aims to evaluate whether models can identify manipulation events from continuous video streams, group behaviors involving the same physical transformations into unified symbolic skills, and maintain a skill library that can be continuously expanded and reused. Overall, the paper is clearly presented and well structured. However, my main concerns lie in the proposed benchmark. Specifically, since the benchmark is constructed by reorganizing and annotating existing datasets, its distinctions from existing tasks, such as action discovery and incremental clustering, are not sufficiently clarified. Furthermore, the paper provides limited discussion and empirical validation of the practical utility of the constructed skill library.

Critical strengths
  1. The paper presents a clear research motivation and addresses a practically meaningful problem in embodied intelligence, focusing on whether models can extract, organize, and reuse skills from continuous visual experience.

  2. The proposed benchmark integrates multiple capabilities into a unified evaluation framework. Rather than focusing solely on temporal event localization, it jointly evaluates Event Localization, Skill Grouping, and Skill Creation and Reuse, demonstrating practical relevance and potential value for embodied intelligence research.

Critical weaknesses
  1. The proposed benchmark is primarily constructed by reorganizing and annotating existing datasets. However, its fundamental distinctions from existing tasks, such as action discovery, incremental clustering, and continual category discovery, are not sufficiently clarified, which limits the overall novelty of the work.

  2. Regarding the evaluation metrics, the current protocol does not penalize unmatched predicted events, and skill grouping is evaluated only on successfully matched events. Can these metrics comprehensively reflect the models' actual skill discovery capabilities?

  3. The paper lacks validation of the practical utility of the constructed skill library. The experiments mainly focus on skill grouping and creation/reuse decisions, without evaluating downstream tasks. Therefore, it remains unclear whether the constructed skill library can effectively support robotic planning and skill reuse.

Questions for authors
  1. What are the fundamental differences between the proposed Video2Skill benchmark and existing tasks, such as action discovery, incremental clustering, and continual category discovery? Compared with these existing settings, what unique contributions does Video2Skill offer for real-world robotic deployment?

  2. Regarding the evaluation protocol, unmatched predicted events are not penalized, and skill grouping is evaluated only on successfully matched events. Can such an evaluation protocol accurately and comprehensively reflect the models' actual skill discovery capabilities?

3 comments
AuthorsOpenReview ↗

Response to Reviewer k7Ur

Thank you for your helpful comments. We appreciate your recognition that the task is useful and that evaluating event localization, grouping, and reuse together adds value. We respond to each concern below.

1. How does this differ from action discovery, incremental clustering, and continual category discovery?

An embodied agent learns from a stream: earlier observations shape what it can reuse later. To our knowledge, Video2Skill is the first benchmark for building, reusing, and expanding a persistent library of symbolic manipulation skills from passive video streams.

Section 2.3 measures three distinct abilities:

AbilityMetric in Video2Skill
Find events in timeCoverage: fraction of reference events matched at temporal IoU ≥ 0.3.
Group instances of the same skillPair Precision, Pair Recall, and Adjusted Rand Index (ARI) on matched events. Low precision signals over-merging; low recall signals fragmentation. ARI adjusts grouping agreement for chance.
Know when to create or reuseCreate Recall and Reuse Recall on first and repeated occurrences in the matched sequence.
Representative settingLocalizes events in videoGroups instancesProcesses inputs incrementallyUses a persistent library of named skills with argument slots and tests when to create or reuse them
Offline action discovery (Kukleva et al.)✓✓——
Incremental clustering of data points (Guo et al.)—✓✓—
Continual category discovery on images (Zhang et al.)—✓✓—
Video2Skill✓✓✓✓

2. Can a model miss events or overpredict and still get a good grouping score?

High grouping scores can hide missed events. Ranks below are from 19 open-source models on the same test split in the unified setting.

ModelWhere it ranks firstWhat that rank hides
MiniCPM-V-4.51st in grouping17th in event coverage; it finds only 7.5% of reference events.

During rebuttal, we evaluated four additional closed-source API models: Opus 5.5, Sonnet 5.5, Sonnet 4.6, and Haiku 4.5. Opus 5.5 ranks 1st of 4 on both grouping and creation in one test split. A strong model for streaming skill discovery needs balanced performance: it should find events reliably, group them correctly, and know when to create or reuse a skill.

LLaVA-OneVision-2-8B (unified), before and after fine-tuning on reference histories.

MetricBeforeAfterChange (points)
Event Coverage37.854.1+16.3
Skill Grouping (ARI)0.060.9+60.9
Create Recall7.138.5+31.4
Reuse Recall100.098.6−1.4

Fine-tuning improves three abilities; creation remains the bottleneck.

3. What is the discovered library useful for?

Video → skill schemas → simulator execution. Opus 5.5 built the library from kitchen videos. We attached scripted motion examples from separate simulator episodes. GPT-6-astra used the library, a camera image, and ground-truth coordinates to generate waypoints; LIBERO executed them and checked task success.

What GPT-6-astra receivedOverall successTasks with hidden grasp offsets
No library or examples31/48 (65%)1/12
Video-discovered schemas + generic examples33/48 (69%)0/12
Video-discovered schemas + all motion examples48/48 (100%)12/12

Object-specific examples supply missing execution details, such as grasping above a bottle’s reported center. See bESG’s response for the examples-only control.

Separate downstream tests: ALFWorld executes text commands; the 130-task LIBERO test scores plan steps without executing them.

Downstream testWithout a libraryWith a library discovered from video
ALFWorld: task success, Qwen3-32B planner (48 unseen tasks)29.2%52.1% with Sonnet 5.5's library; 45.8% with Opus 5.5's library
LIBERO: plan step F1 against a goal-derived reference (130 tasks)3061–89

In ALFWorld, no library solved 0/17 cleaning tasks; libraries with a schema mapped to wash solved 6–13. A library missing wash, open, and close reduced overall success to 18.8%.

The authors have provided a detailed rebuttal, including comparisons with three categories of related tasks and additional downstream experiments on LIBERO and ALFWorld, which have substantially addressed my previous concerns. I also recognize the value of the proposed benchmark and believe it has the potential to contribute to future research in embodied intelligence, particularly in improving the generalization capabilities of embodied agents. Nevertheless, I encourage the authors to further refine the evaluation protocol to ensure a more comprehensive and reliable assessment. Overall, given the clarifications and additional experimental evidence provided in the rebuttal, I have decided to raise my rating.

AuthorsOpenReview ↗

Many thanks for your thoughtful review and support

Thank you for your careful review and positive recommendation! We will revise the paper accordingly and further refine the evaluation protocol based on your suggestions.

Comments on the paper

2 comments

Dear Reviewers,

Please review the authors' rebuttal and the other reviewers' comments, and participate in the discussion before it closes on November 18 (AoE). Kindly indicate whether your concerns have been adequately addressed and raise any remaining questions or concerns.

Thank you for your time and contributions.

Best regards,

Area Chair

Dear Reviewers,

A quick reminder that the discussion period ends on November 18 (AoE). If you haven't already, please review the authors' rebuttal and provide a brief follow-up indicating whether your concerns have been adequately addressed or if further clarification is needed.

Your timely feedback would be greatly appreciated and will help facilitate the final decision.

Thank you for your time and contributions!

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.