- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces Skillformer, a representation model that combines visual and tactile observations conditioned on a language description of the queried skill. It uses different observation windows for vision and touch and stores sensory windows at predicted skill completion events. Evaluations on lamp replacement, bottle capping, and blind cube search examine how these components support progress estimation and recognition of contact events.
Critical strengths
-
The paper identifies a meaningful need to interpret sensory observations according to the skill being performed. Reaching, tightening, and object inspection require different evidence, and the chosen tasks make these differences concrete. Conditioning perception on the queried skill is therefore well motivated.
-
Storing sensory windows at skill completion is a simple and interpretable way to preserve brief but important contact events. This connects event detection with later state estimation and addresses situations where the current observation cannot reveal what happened earlier.
Critical weaknesses
- Another concern is that progress prediction does not by itself establish the broader representation quality claimed by the paper. For continuous skills, the target measures elapsed time within an annotated interval, which can differ from physical progress during pauses, reversals, or recovery.
- The event memory depends on progress predictions that are themselves conditioned on that memory. A premature completion prediction could therefore store misleading context and influence subsequent event decisions
- The formulation does not clearly explain how repeated executions of the same primitive are distinguished. When a completed skill is attempted again, the same language query must distinguish progress in the new attempt from the completion retained in memory. This becomes important for retries and recovery, where the assumption of advancing through a predefined skill sequence may no longer hold.
Questions for authors
- How is the queried skill instance selected when constructing event memory at test time, and does this selection use annotated skill boundaries?
- Since language identifies skills from a predefined vocabulary, what benefit does the frozen CLIP text embedding provide over a learned skill identifier, particularly when related skills share similar descriptions?
3 comments
Response to Reviewer sj5p (part 2/2)
Q1. Test-Time Skill Selection and Event Memory Construction At test time, we assume having access to a predefined skill graph that specifies how primitive skills are organized to accomplish a long-horizon task. This graph determines which skill is queried, while SkillFormer uses current visuotactile observations and previously stored events to predict its progress and timing. When the predicted progress reaches the completion threshold defined in Section 3.4, the corresponding observation window is added to event memory. Importantly, this process does not rely on ground-truth skill boundaries at test time, since skill completion is determined by the model's predictions. Annotated boundaries are used only to generate progress targets during training. Further details on training-time event memory construction are provided in Section 3.5.
Q2: Frozen CLIP Text Embeddings vs. Learned Skill Identifiers We use frozen CLIP text embeddings to provide a shared semantic prior across related skills, such as "Reach box," "Reach bottle," and "Reach bulb." This gives the model a structured starting point for learning how similar primitive skills should attend to visual, tactile, and event-memory information across different tasks. In contrast, learned skill IDs are independently initialized and must learn these relationships from task-specific demonstrations. Additionally, language-based conditioning improves scalability by allowing users to introduce entirely new skills, even those without shared action verbs, without having to learn completely independent embeddings for each new skill.
Thank the authors for the rebuttal, most of my concerns addressed. So, i will keep my current positve rating.
Response to Reviewer sj5p (part 1/2)
We sincerely thank the reviewer for their positive assessment of our work and for recognizing that “the chosen tasks make these differences concrete” and that storing sensory windows at skill completion is “a simple and interpretable way to preserve brief but important contact events.” We also appreciate the constructive feedback and insightful questions, which have helped us improve the clarity and presentation of our work. Below, we address each concern and question in detail.
W1. Time-Based Progress Supervision Our training dataset consists of expert demonstrations without pauses, reversals, or recovery, making elapsed time a reasonable approximation of physical progress for supervision. However, elapsed time is only used to generate training targets; SkillFormer learns to associate sensory observations with progress rather than relying on time itself. For example, in Figure 6, higher progress for "Reach bottle cap" corresponds to the gripper approaching the cap. If the gripper moves backward, the model can predict a reduction in skill progress despite time advancing, as its predictions are not constrained to increase monotonically. The "Rewind" video in the Robustness Tests section of our supplementary website demonstrates SkillFormer's ability to predict non-monotonic progress during repeated reaching and grasping, even when previous skill completion events remain stored in memory.
W2. Robustness of Event Memory to Premature Skill Completion SkillFormer predicts progress using both current observations and previously stored event memory. Therefore, a stored completion event does not necessarily imply that the skill's outcome remains valid throughout the task. For example, if the robot grasps the bottle cap and proceeds to "Reach bottle," but the cap subsequently slips from the gripper, the current observations may resemble an earlier stage of the task. SkillFormer can recognize this change and predict a reduction in "Reach bottle" progress despite the previously stored grasp event. This behavior is encouraged by querying inactive skills during training (Section 3.5), which teaches the model to estimate progress for active and inactive skills throughout the whole task. The "cap collision" video in the Robustness Tests section of our supplementary website further illustrates this behavior videos/bottlecapcol.mp4.
To further investigate the effect of premature completion predictions, we evaluate SkillFormer by inserting events into memory 2, 5, 10, and 20 frames earlier than their originally detected timestamps. The results are shown below.
Even when events are stored up to 20 frames prematurely, SkillFormer maintains substantially lower MSE than FiLM and w/o Events across all three tasks. Although performance degrades with increasingly premature events, particularly in blind search, these results suggest that the model is reasonably robust to errors in event timing and does not rely merely on previously stored completion events.
W3. Repeated Skill Execution When a long-horizon task involves multiple instances of the same primitive skill, we differentiate them using instance-specific skill descriptions. For example, in blind cube search, we annotate instances as "Reach box 1" and "Reach box 2," allowing us to evaluate whether sparse event memory retains sufficient interaction history to distinguish progress across different boxes. For retries and recovery, progress predictions are not constrained to increase monotonically, and previously stored completion events do not imply that a skill remains completed. As demonstrated in the "Rewind" and "Cap Collision" videos in the Robustness Tests section of our supplementary website, SkillFormer can predict reductions in progress and track repeated skill executions when the robot returns to earlier stages of the task.