acceptodds
Under review as a conference paper at ICLR 2027

What to Attend, What to Keep: Skill-Conditioned Visuotactile Representation with Progress-Guided Event Memory

Abstract

Robotic manipulation integrates vision, touch, and language, whose importance shifts across stages: vision guides reaching, while touch, through its evolution over time, decides grasping, alignment, and contact. Yet existing multi-modal manipulation policies typically use fixed temporal contexts and fusion strategies, despite shifts in what each modality contributes across different skills. We study how vision and touch should be combined at the level of primitive skills, asking what each skill needs from each sensor, and propose a language-conditioned representation in which the queried skill selects modality-specific encoders and horizons and attends to a sparse event memory holding the terminal states of the last K completed skills. Evaluated by skill progress estimation on three contact-rich tasks, it reduces slip-detection delay by 87% against fine-tuned SOTA progress models, twist-completion delay by 67.5% against a vision-only ablation, and progress error on a blind search task by 92% through sparse event memory. Gains concentrate exactly where completion is defined by contact or task history. More broadly, our results suggest that observation formation—not only policy architecture—is a central challenge in multi-modal representation. All data, models, and code will be open-sourced upon release.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-Modal Representation, Contact-Rich Manipulation, Tactile Sensing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sj5pOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces Skillformer, a representation model that combines visual and tactile observations conditioned on a language description of the queried skill. It uses different observation windows for vision and touch and stores sensory windows at predicted skill completion events. Evaluations on lamp replacement, bottle capping, and blind cube search examine how these components support progress estimation and recognition of contact events.

Critical strengths
  • The paper identifies a meaningful need to interpret sensory observations according to the skill being performed. Reaching, tightening, and object inspection require different evidence, and the chosen tasks make these differences concrete. Conditioning perception on the queried skill is therefore well motivated.

  • Storing sensory windows at skill completion is a simple and interpretable way to preserve brief but important contact events. This connects event detection with later state estimation and addresses situations where the current observation cannot reveal what happened earlier.

Critical weaknesses
  • Another concern is that progress prediction does not by itself establish the broader representation quality claimed by the paper. For continuous skills, the target measures elapsed time within an annotated interval, which can differ from physical progress during pauses, reversals, or recovery.
  • The event memory depends on progress predictions that are themselves conditioned on that memory. A premature completion prediction could therefore store misleading context and influence subsequent event decisions
  • The formulation does not clearly explain how repeated executions of the same primitive are distinguished. When a completed skill is attempted again, the same language query must distinguish progress in the new attempt from the completion retained in memory. This becomes important for retries and recovery, where the assumption of advancing through a predefined skill sequence may no longer hold.
Questions for authors
  • How is the queried skill instance selected when constructing event memory at test time, and does this selection use annotated skill boundaries?
  • Since language identifies skills from a predefined vocabulary, what benefit does the frozen CLIP text embedding provide over a learned skill identifier, particularly when related skills share similar descriptions?
3 comments
AuthorsOpenReview ↗

Response to Reviewer sj5p (part 1/2)

We sincerely thank the reviewer for their positive assessment of our work and for recognizing that “the chosen tasks make these differences concrete” and that storing sensory windows at skill completion is “a simple and interpretable way to preserve brief but important contact events.” We also appreciate the constructive feedback and insightful questions, which have helped us improve the clarity and presentation of our work. Below, we address each concern and question in detail.

W1. Time-Based Progress Supervision Our training dataset consists of expert demonstrations without pauses, reversals, or recovery, making elapsed time a reasonable approximation of physical progress for supervision. However, elapsed time is only used to generate training targets; SkillFormer learns to associate sensory observations with progress rather than relying on time itself. For example, in Figure 6, higher progress for "Reach bottle cap" corresponds to the gripper approaching the cap. If the gripper moves backward, the model can predict a reduction in skill progress despite time advancing, as its predictions are not constrained to increase monotonically. The "Rewind" video in the Robustness Tests section of our supplementary website demonstrates SkillFormer's ability to predict non-monotonic progress during repeated reaching and grasping, even when previous skill completion events remain stored in memory.

W2. Robustness of Event Memory to Premature Skill Completion SkillFormer predicts progress using both current observations and previously stored event memory. Therefore, a stored completion event does not necessarily imply that the skill's outcome remains valid throughout the task. For example, if the robot grasps the bottle cap and proceeds to "Reach bottle," but the cap subsequently slips from the gripper, the current observations may resemble an earlier stage of the task. SkillFormer can recognize this change and predict a reduction in "Reach bottle" progress despite the previously stored grasp event. This behavior is encouraged by querying inactive skills during training (Section 3.5), which teaches the model to estimate progress for active and inactive skills throughout the whole task. The "cap collision" video in the Robustness Tests section of our supplementary website further illustrates this behavior videos/bottlecapcol.mp4videos/bottle_cap_col.mp4.

To further investigate the effect of premature completion predictions, we evaluate SkillFormer by inserting events into memory 2, 5, 10, and 20 frames earlier than their originally detected timestamps. The results are shown below.

MethodBottle MSELamp MSEBlind MSE
SkillFormer4.6 ± 10.33.4 ± 7.711.6 ± 34.1
SkillFormer, 2 early5.3 ± 10.93.7 ± 8.515.2 ± 41.6
SkillFormer, 5 early5.0 ± 10.03.8 ± 8.714.1 ± 39.3
SkillFormer, 10 early5.2 ± 9.63.9 ± 9.318.0 ± 49.8
SkillFormer, 20 early5.1 ± 9.04.8 ± 9.321.4 ± 85.1
FiLM11.6 ± 20.614.7 ± 22.1116.0 ± 253.4
w/o Events11.2 ± 18.19.0 ± 26.4143.6 ± 220.0

Even when events are stored up to 20 frames prematurely, SkillFormer maintains substantially lower MSE than FiLM and w/o Events across all three tasks. Although performance degrades with increasingly premature events, particularly in blind search, these results suggest that the model is reasonably robust to errors in event timing and does not rely merely on previously stored completion events.

W3. Repeated Skill Execution When a long-horizon task involves multiple instances of the same primitive skill, we differentiate them using instance-specific skill descriptions. For example, in blind cube search, we annotate instances as "Reach box 1" and "Reach box 2," allowing us to evaluate whether sparse event memory retains sufficient interaction history to distinguish progress across different boxes. For retries and recovery, progress predictions are not constrained to increase monotonically, and previously stored completion events do not imply that a skill remains completed. As demonstrated in the "Rewind" and "Cap Collision" videos in the Robustness Tests section of our supplementary website, SkillFormer can predict reductions in progress and track repeated skill executions when the robot returns to earlier stages of the task.

AuthorsOpenReview ↗

Response to Reviewer sj5p (part 2/2)

Q1. Test-Time Skill Selection and Event Memory Construction At test time, we assume having access to a predefined skill graph that specifies how primitive skills are organized to accomplish a long-horizon task. This graph determines which skill is queried, while SkillFormer uses current visuotactile observations and previously stored events to predict its progress and timing. When the predicted progress reaches the completion threshold defined in Section 3.4, the corresponding observation window is added to event memory. Importantly, this process does not rely on ground-truth skill boundaries at test time, since skill completion is determined by the model's predictions. Annotated boundaries are used only to generate progress targets during training. Further details on training-time event memory construction are provided in Section 3.5.

Q2: Frozen CLIP Text Embeddings vs. Learned Skill Identifiers We use frozen CLIP text embeddings to provide a shared semantic prior across related skills, such as "Reach box," "Reach bottle," and "Reach bulb." This gives the model a structured starting point for learning how similar primitive skills should attend to visual, tactile, and event-memory information across different tasks. In contrast, learned skill IDs are independently initialized and must learn these relationships from task-specific demonstrations. Additionally, language-based conditioning improves scalability by allowing users to introduce entirely new skills, even those without shared action verbs, without having to learn completely independent embeddings for each new skill.

Thank the authors for the rebuttal, most of my concerns addressed. So, i will keep my current positve rating.

Reviewer_JhvmOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SKILLFORMER, a language-conditioned visuotactile representation combining modality-specialized encoders. On skill-progress estimation for lamp replacement, bottle capping, and blind cube search, the model reduces slip-detection delay by 87% relative to fine-tuned progress models.

Critical strengths
  • The paper addresses the limitation of multi-modal manipulation encoders with fixed temporal context and task-level instruction. The proposed SKILLFORMER is well-motivated and enhances modality fusion at skill-level.

  • The reported results are detailed and show the high quality of representation of SKILLFORMER. Across the three tasks, skill-conditioned fusion lowers progress MSE by 37–80% compared with vision-only baseline. I also appreciate the Figure 7, which demonstrates the robustness of SKILLFORMER to visual and temporal perturbations.

Critical weaknesses
  • The paper is motivated by multi-modal manipulation, but the evaluation covers only offline skill-progress estimation. Although the authors leave the use of predicted progress as a reward for future work, SKILLFORMER does not validate its benefits to downstream policies, whereas ReWiND and Robometer did.

  • Progress estimation depends on a queried skill and on primitive skill annotations. The skill identity selects encoders, horizons, and memory attention, so the model may be told which skill is active. Event memory may stores terminal states of predicted completions, which allows an early detection error to enter the context of later skills. Robustness to an incorrect skill query or to noisy completion predictions is not established.

Questions for authors
  • Can the estimated progress signal be used as a reward, or can the representation condition a policy? Such experiments on even one of the three tasks would significantly enhance the quality of the paper.

  • At test time, is the language skill query taken from ground-truth primitive annotations? How much does progress error increase under an incorrect skill query, and when predicted completions written into the event memory are wrong?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Jhvm (part 1/2)

We thank the reviewer for the encouraging feedback and for highlighting the strengths of our approach. We are glad that the reviewer found SkillFormer's skill-level modality fusion well-motivated and that the experimental results demonstrate the high quality of the learned representations. We also appreciate the positive comments on our robustness analysis in Figure 7. The reviewer raises valuable points about robustness to prediction errors, which we address below.

Q1. Using Skill Progress as a Reward Our primary contribution is to investigate how multimodal representations should combine brief contact-dependent events captured by touch, visual context from RGB observations, and relevant interactions beyond the recent observation window. We evaluate these design choices through skill-progress prediction, which requires distinguishing meaningful changes in manipulation state, including tightening, slipping, and inspection outcomes that depend on earlier interactions.

The predicted progress can be used as a reward signal or for trajectory ranking, while the learned representation can also condition a downstream policy, as motivated by progress-based approaches such as ReWiND and Robometer. Our focus, however, is on multimodal representation learning rather than policy learning. Beyond progress accuracy, we provide ablations, diagnostic evaluations, and robustness tests to understand and debug the learned representations, examining how different modalities, temporal horizons, and event memory contribute to performance. While downstream policy evaluation would provide additional validation, we believe these findings offer valuable insights into representation learning for contact-rich manipulation.

Q2. Skill Query Selection and Robustness to Incorrect Event Detection

Test-Time Skill Selection Without Ground-Truth Annotations

At test time, we only have access to a predefined skill graph that specifies how primitive skills are composed to perform a long-horizon task. The active skill is selected by advancing through this graph based on predicted skill completion. Specifically, when the predicted progress crosses the corresponding completion threshold (Section 3.4), the observation window is stored in event memory, and the next skill is selected according to the graph. For conditional skills, such as inspection, the predicted outcome (success or failure) determines which branch of the skill graph to follow. Therefore, this process does not require ground-truth skill boundaries at test time. We will explicitly clarify this test-time skill selection and transition procedure, including conditional branching, in the revised paper.

Inactive Skill Queries

Importantly, the skill query identifies which skill's progress to estimate, not whether that skill is currently being executed. We explicitly train with both active and inactive skill queries (Section 3.5) and evaluate every skill throughout each held-out episode (Section 5). Thus, the model must use current observations and event memory to determine whether a queried skill has started, is ongoing, or has completed, including whether a conditional skill has succeeded or failed. This allows us to evaluate progress predictions for inactive skills without assuming that the queried skill is currently being executed.

AuthorsOpenReview ↗

Response to Reviewer Jhvm (part 2/2)

Robustness to Premature Completion Predictions

To further evaluate robustness to premature completion predictions, we conducted additional experiments in which events are stored 2, 5, 10, 20, and 30 frames earlier than their detected completion times.

MethodBottle MSEBottle DelayBottle PearsonLamp MSELamp DelayLamp PearsonBlind MSEBlind DelayBlind Pearson
SkillFormer4.6 ± 10.310.70 ± 15.780.985 ± 0.0353.4 ± 7.77.60 ± 9.200.971 ± 0.04511.6 ± 34.116.10 ± 37.200.953 ± 0.082
SkillFormer, 2 frames early5.3 ± 10.910.61 ± 15.390.983 ± 0.0353.7 ± 8.57.0 ± 8.90.978 ± 0.03315.2 ± 41.619.7 ± 50.70.947 ± 0.091
SkillFormer, 5 frames early5.0 ± 10.010.77 ± 15.580.984 ± 0.0333.8 ± 8.77.1 ± 9.00.978 ± 0.03314.1 ± 39.320.5 ± 53.00.952 ± 0.078
SkillFormer, 10 frames early5.2 ± 9.610.88 ± 15.370.984 ± 0.0303.9 ± 9.38.8 ± 12.90.976 ± 0.04218.0 ± 49.819.3 ± 46.30.949 ± 0.083
SkillFormer, 20 frames early5.1 ± 9.011.53 ± 15.840.984 ± 0.0294.8 ± 9.39.5 ± 14.20.969 ± 0.03921.4 ± 85.117.7 ± 38.70.940 ± 0.123
SkillFormer, 30 frames early5.5 ± 8.711.91 ± 15.710.983 ± 0.0255.8 ± 10.57.8 ± 9.10.956 ± 0.05019.1 ± 72.118.8 ± 26.80.951 ± 0.096
FiLM11.6 ± 20.614.27 ± 15.700.914 ± 0.17614.7 ± 22.18.97 ± 14.550.828 ± 0.276116.0 ± 253.477.81 ± 118.550.727 ± 0.263
w/o Events11.2 ± 18.117.36 ± 15.500.961 ± 0.0619.0 ± 26.47.92 ± 10.410.906 ± 0.173143.6 ± 220.063.46 ± 119.430.683 ± 0.301

MSE (×10⁻³), Pearson correlation, and absolute detection delay (frames). Values are mean ± standard deviation.

We observe an increase in overall MSE as events are stored earlier, suggesting that premature event storage introduces additional error into progress predictions. However, Pearson correlation remains consistently high, and event-detection delay remains relatively stable, even when events are stored up to 30 frames prematurely. For example, Pearson correlation changes only from 0.985 to 0.983 for Bottle Capping, 0.971 to 0.956 for Lamp Replacement, and 0.953 to 0.951 for Blind Search. This indicates that although absolute progress estimates are affected, the model continues to capture the overall progress trajectory and reliably detect skill completion. Importantly, shifting events by 30 frames represents an extreme robustness test rather than a typical event-localization error. SkillFormer also continues to outperform FiLM and w/o Events across all three tasks, demonstrating the robustness of its progress estimation and sparse event memory.

Furthermore, during training, event memory is constructed using the model's own predicted completions (warmup epochs) and periodically updated (Section 3.5), rather than relying exclusively on ground-truth completion times. While these experiments specifically evaluate premature event storage rather than arbitrary incorrect skill-graph transitions or spurious event identities, they provide additional evidence that the benefits of event memory persist even when completion timing is inaccurate.

Robustness to Failures and Repeated Skill Execution We also provide complementary evidence in the Robustness Tests section of our supplementary website, where execution failures introduce retries and deviations from the original skill sequence. In the "Cap Collision," "Loose Grasp," and "Rewind" videos, SkillFormer tracks changes in skill progress as the robot returns to earlier stages of the task, even when previous completion events remain stored in memory. These examples demonstrate that progress predictions are not determined solely by stored events or the expected skill order, but also depend on current visuotactile observations. Together with our event-timing perturbation experiments, these results provide further evidence of robustness to execution failures and non-monotonic skill progress.

I appreciate the authors' detailed response, which has addressed my concerns regarding Skill Queries. However, while the authors reiterate their contribution as representation learning rather than policy learning, I remain concerned about the practical application of the learned representations. I therefore maintain my current rating.

We thank the reviewer for their follow-up. We agree that demonstrating the practical utility of the learned representation for downstream policy learning would strengthen our work. To address this concern, we conducted an additional policy-learning experiment using SkillFormer's learned representation directly as input to a manipulation policy.

Specifically, we built a simulated replica of our bottle-capping setup in Genesis, including calibrated camera geometry and simulated GelSight tactile sensing. We trained an MLP policy using DAgger to execute three consecutive skills: Reach Bottle Cap → Grasp Bottle Cap → Reach Bottle. The policy receives SkillFormer's output representation token, which was trained solely through progress supervision and remains frozen during policy training. Skill transitions and event-memory updates are determined by predicted progress, without ground-truth skill boundaries or privileged simulator states during execution.

We compare policies trained using SkillFormer, FiLM, and Vision Only representations under the same evaluation setting with randomized initial configurations. Success rates are computed over 100 evaluation trials for each model.

Due to challenges in accurately simulating complex contact interactions, particularly thread alignment and cap sliding, we limited this experiment to the first three skills, ending with reaching the bottle while holding the cap. Although this is a relatively simple pick-and-place task, the results provide evidence that SkillFormer's progress-supervised representation captures spatial and manipulation-relevant information that can be used directly for action prediction. Most observed failures arise from premature grasping, and SkillFormer achieves higher success rates than both alternative representations. We expect its advantages to be more pronounced in longer contact-rich tasks, where tactile feedback and interaction history become increasingly important, although this remains to be evaluated.

ModelSuccess Rate (100 Trials) ↑
SkillFormer (Ours)80%
FiLM75%
Vision Only56%

We have included both successful and failed policy rollouts on our supplementary website. While our primary contribution remains multimodal representation learning, this experiment provides initial downstream validation that the learned features are useful for robotic control beyond offline progress estimation.

Reviewer_f6e1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2(+1)
Presentation
2(+1)
Soundness
1
Read review
Paper summary

This paper introduces SkillFormer, a skill-conditioned visuotactile representation for manipulation progress estimation. The model combines recent visual and tactile observations with a sparse memory of observation windows retained at predicted skill completions. A language query identifies the primitive skill whose progress is estimated. The approach is evaluated on three collected datasets: lamp replacement, bottle capping, and blind cube search.

Critical strengths
  • Relevant motivation. Different manipulation stages require different sensory evidence, and studying tactile temporal context together with event memory is well motivated.

  • Useful component analysis. The tactile, temporal-horizon, and event-memory ablations provide evidence for their contributions within the evaluated tasks. The perturbation experiments offer additional insight into the model’s sensory dependencies.

Critical weaknesses
  • The presentation obscures the central problem and method. The introduction moves between modality fusion, temporal context, event memory, and progress estimation without clearly prioritizing the central research question. The method’s operational flow only becomes clear after reading the later event-construction and training sections. There is also a nearly verbatim repetition of the opening paragraph of Section 3.1 across pages 3–4.

  • The evaluation is too narrow to establish broader representation utility. Experiments cover three specific task domains, with approximately 10% of episodes from each domain held out for testing. This demonstrates performance on unseen episodes within familiar task structures, but does not establish transfer to unseen tasks, objects, or skill compositions. Although the paper acknowledges these limitations, the current evidence remains insufficient to substantiate the broader motivation concerning reusable manipulation representations.

  • Scalability under the required skill supervision is unaddressed. The method assumes a predefined skill vocabulary and annotated start and end times for each skill instance, together with progress types and conditional outcomes. The limitations section additionally acknowledges dependence on a predefined skill graph.

Questions for authors
  • Can the authors provide evidence of generalization to unseen skill compositions or objects, and assess sensitivity to alternative skill decompositions or noisy boundary annotations?

  • How were skill boundaries, progress types, and success/failure outcomes annotated? What was the annotation effort, and how were ambiguous boundaries or disagreements handled?

5 comments
AuthorsOpenReview ↗

Response to Reviewer f6e1 (part 1/2)

We thank the reviewer for their careful reading and constructive feedback. We appreciate their recognition of the motivation behind studying skill-dependent sensory requirements and the value of our component analysis and robustness experiments. We have revised the paper to clarify the central research question and flow, and conducted additional experiments to address concerns about generalization to unseen skill sequences and sensitivity to noisy skill annotations. We also provide further details on our annotation procedure and its associated effort. We address each concern below.

W2 & Q1. Generalization to Unseen Skill Sequences and Sensitivity to Skill Annotations We thank the reviewer for raising these concerns. Our work focuses on evaluating SkillFormer across contact-rich manipulation tasks with distinct sensory, temporal, and memory requirements. To this end, we designed and collected three new, challenging manipulation datasets specifically targeting these requirements. Through extensive ablations, we investigate how different primitive skills benefit from tactile sensing, temporal context, multimodal fusion, and event memory.

Generalization to Unseen Skill Sequences. To further evaluate robustness beyond the skill sequences observed during training, we conducted additional experiments involving unseen execution orders caused by manipulation failures and retries, presented in the Robustness Tests section of our supplementary website. In the Cap Collision, Loose Grasp, and Rewind experiments, failures cause the robot to revisit earlier skills or repeat previously completed interactions. Despite these deviations from training skill orders, SkillFormer interestingly captures non-monotonic progress even when previous completion events remain in memory. These results provide evidence that the learned representation can interpret changes in execution order rather than relying solely on the expected skill sequence.

Sensitivity to Noisy Skill-Boundary Annotations. We additionally retrained SkillFormer with every training skill boundary randomly shifted by a uniformly sampled offset of up to ±2, ±5, ±10, or ±15 frames, while evaluating against clean test annotations. Due to the limited rebuttal period, we conducted these additional training experiments on Bottle Capping only.

Overall performance under noisy training annotations

Training boundary noiseMSE (×10⁻³) ↓Completion delay (frames) ↓Pearson r ↑
Original4.610.70.985
±2 frames5.810.40.982
±5 frames8.215.70.977
±10 frames12.213.20.955
±15 frames7.714.00.976

To further investigate the effect of annotation noise on different skill types, we report the per-skill Completion Delay @ 0.9 ↓

SkillOriginal±2±5±10±15
Reach bottle cap5.45.45.05.06.4
Reach bottle8.78.06.58.57.0
Align bottle cap6.36.013.312.018.5
Slide bottle cap11.714.145.530.231.6

The results reveal distinct sensitivities across skill types. Continuous reaching skills remain relatively robust to boundary perturbations because their progress is reflected in observations throughout the motion. In contrast, discrete, contact-dependent skills rely on brief completion signals that must be accurately localized. With small perturbations (±2 frames), the true completion event is still likely to fall within the observation window associated with the annotated boundary, allowing the model to learn from the relevant sensory evidence. However, with larger perturbations (±5 or ±10 frames), the annotated boundary is more likely to shift away from the actual contact event, potentially placing the critical completion signal outside the corresponding observation window. This introduces a mismatch between the supervision target and the available sensory evidence, explaining the greater sensitivity of discrete skills. For example, Slide Bottle Cap completion delay increases from 11.7 to 14.1 frames under ±2-frame perturbations but reaches 45.5 frames under ±5-frame perturbations.

Importantly, overall Pearson correlation remains above 0.95 across all perturbation levels, indicating that SkillFormer largely preserves the temporal progression of skills despite increased absolute prediction and event-localization errors. These findings also motivate our annotation procedure, where brief contact-dependent completion events receive additional human verification.

AuthorsOpenReview ↗

Response to Reviewer f6e1 (part 2/2)

Scope of Generalization. We agree that transfer to entirely unseen objects, tasks, and alternative skill decompositions remains an important future direction. Such evaluation relies on substantially larger and more diverse synchronized visuotactile datasets, which remain scarce to collect. Therefore, as part of this work, we have designed and collected three new contact-rich manipulation datasets, and we view broader generalization evaluation as a natural extension to our contribution: learning skill-conditioned representations that selectively integrate relevant sensory information and retain important past interactions.

Q2: Skill Annotation Procedure and Effort We thank the reviewer for raising this question. To reduce annotation effort, we developed a VLM-assisted skill segmentation pipeline. We uniformly sample trajectory frames at 3 FPS and provide them to a VLM along with predefined skill descriptions and a small set of labeled example trajectories. The VLM then identifies and segments the primitive skills within each demonstration. Since contact-rich manipulation often involves brief completion signals that are difficult to localize precisely, we developed a lightweight annotation tool for human verification of skill boundaries. This ensures that critical completion events, such as cap slippage or the final bulb twist, are accurately localized in the RGB or tactile observations and included in the corresponding observation windows. Ambiguous boundaries are resolved through this manual inspection, particularly for discrete skills whose progress changes sharply at completion. Manual verification takes approximately 15 seconds per episode, totaling roughly 107 minutes across all 429 demonstrations. The importance of accurate boundary localization is further examined in our additional noisy-label experiments (part 1), which quantify the sensitivity of progress estimation to perturbed training annotations.

Progress types are defined according to each primitive skill's completion characteristics. For example, reaching is continuous because its advancement can be observed throughout the motion, whereas alignment and sliding are discrete because completion is determined by a specific event. For conditional skills, such as object inspection, success or failure labels are determined by the subsequent skill sequence specified in the skill graph. For instance, proceeding to another candidate box indicates an unsuccessful inspection, whereas transitioning toward the puzzle box indicates that the target object was found. These definitions are consistent across demonstrations and summarized in Appendix Table 5. We will add these annotation details to the revised paper to clarify the supervision requirements, annotation effort, and treatment of ambiguous skill boundaries.

Thank you for the authors' detailed responses and additional experiments. I appreciate the efforts made to address my concerns. However, my main concerns remain only partially resolved.

  1. Presentation and logical coherence.

The authors mention that the manuscript has been revised to improve clarity and organization. However, after downloading the latest PDF, I could not identify the corresponding changes. Could the authors highlight the revisions in a different color to facilitate verification?

More importantly, my concern is not merely about wording but about the logical structure of the paper. The Introduction moves between modality fusion, temporal context, event memory, and progress estimation without establishing a clear and coherent research narrative. Similarly, the Method section lacks sufficient logical connections between key components, making the overall framework difficult to follow without considerable effort. I believe these issues require substantial restructuring rather than minor textual revisions.

  1. Generalization and sensitivity to skill annotations.

I appreciate the additional experiments on unseen execution orders, which provide useful evidence of robustness to failures and retries. However, these experiments do not fully address my concern about generalization to unseen skill compositions. I am particularly interested in whether the learned representation can generalize to novel compositions of existing primitive skills, rather than different execution orders within familiar task structures.

Furthermore, the annotation noise experiments reveal considerable sensitivity in discrete contact-dependent skills. For example, the completion delay for Slide Bottle Cap increases from 11.7 to 45.5 frames under ±5-frame boundary perturbations. This suggests that accurate skill-boundary annotations remain important for reliable event detection, raising concerns about the scalability of the required supervision.

  1. Scalability of skill supervision.

I appreciate the clarification regarding the VLM-assisted annotation pipeline and the reported manual verification time. However, the framework still relies on a predefined skill vocabulary, skill graphs, and labeled examples. The reported annotation effort mainly reflects boundary verification within these predefined task structures.

How would the supervision requirements scale to a substantially larger and more diverse collection of manipulation tasks? In particular, what would be the additional effort required to define new skills, construct skill graphs, prepare labeled examples, and verify skill boundaries for previously unseen tasks?

Overall, while the additional experiments and clarifications are valuable, they do not fully resolve my concerns regarding the presentation quality, broader representation generalization, and scalability of the proposed framework.

Comments on the paper

1 comment

Dear Reviewers,

Thank you again for your reviews of this submission. The authors have now posted their responses, and the author–reviewer discussion period ends on November 18, 2026 (AoE).

Please read the authors' rebuttal and let them know whether it addresses your concerns. A short reply is enough, for example:

  • which of your concerns are now resolved;
  • which remain open, and what evidence or clarification would resolve them;
  • whether you plan to update your rating.

If the rebuttal changes your assessment, please update your review and rating before the discussion period ends. If you have already engaged, thank you, and no further action is needed unless the authors reply again.

Your participation is important for a fair and well-informed decision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.