acceptodds
Withdrawn from ICLR 2027

FEEL: Fine-grained Embodied Representation Learning for Robotic Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multisensory feedback enables humans to perform daily tasks. When opening a soda bottle, vision locates the cap, tactile feedback reveals slip, force feedback indicates changes in resistance, and audio signals the release of gas. Learning comparable representations for robots requires synchronized sensory observations of the same manipulation event. We therefore aim to build fine-grained embodied representations that allow robots to understand the ongoing action and judge its interaction state from multiple senses. This leads to three questions: how can robots acquire such multisensory experience, how can it be organized into manipulation events, and how can representations be learned from it? Accordingly, we develop our framework along three corresponding aspects: data collection, dataset construction, and multimodal representation learning. For data collection, we develop FeelCap, an egocentric multisensory capture system that synchronously records RGB, depth, tactile, audio, force/torque, and IMU signals. Using FeelCap, we construct Feeling, an egocentric multisensory dataset of real-world manipulation events. We divide each trajectory into fine-grained interaction segments. Each segment is annotated with the action and manipulated objects, together with their visual and physical attributes. We propose Feel, trained with three objectives: semantic-space alignment associates sensory representations with the corresponding visual–geometry, physical, and event semantics; intra-modal masked reconstruction retains modality-specific sensory information; and cross-modal prediction learns the relationships among different sensory signals within the same manipulation event. We evaluate Feel on the Feeling Benchmark using sensor–text retrieval and modality generation. We further assess the learned representations on real-world robot manipulation tasks. Experimental results show that Feel effectively captures relationships across sensory modalities and improves policy performance on real-world tasks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
representation learning,multimodal alignment,multimodal perception

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2aLYOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents FeelCap, a multisensory data-collection device; FEELING, a dataset of manipulation recordings; and FEEL, a representation learning method. The method combines semantic alignment and sensory reconstruction, with evaluation on retrieval, reconstruction, and real-robot manipulation.

Critical strengths

I like the hardware design. Integrating vision, depth, touch, force, and audio into a device that supports both handheld collection and robot use is a useful contribution. The modular design looks well thought out.

The dataset is another strong point. It contains 9,636 episodes and 37,391 action segments, covering a range of materials and interactions. The synchronized recordings and segment-level annotations could be valuable beyond the proposed model.

The paper is clear and easy to follow. I also appreciate the real-robot experiments. Both policies improve on all three tasks in the reported trials, which provides some evidence of practical value.

Critical weaknesses

My main concern is environmental diversity. Many examples show similar green tabletop setups. This does not establish that collection was limited to one environment, but I couldn’t find a breakdown of locations, backgrounds, lighting, or acoustic conditions. The paper reports material and action diversity, while the benchmark holds out objects. Neither tells us how well the representations transfer to different recording environments.

The robot evaluation needs more detail. Section 5.4 gives the tasks, demonstration counts, and success criteria, but leaves the policy integration and test conditions unclear. Which sensors does each baseline receive? How are FEEL features added? What changes between training and testing? There are also only 10 trials per setting. For example, π0’s pouring improvement from 40% to 50% is one additional success. The results are encouraging, but still preliminary.

The learning method would benefit from cleaner comparisons. I’d like to see a simple fusion baseline with the same sensors and encoders, plus ablations of the three-space design and reconstruction losses. The Stage-1/Stage-2 results use different evaluation protocols, so they do not isolate the effect of the second training stage.

Questions for authors
  • How many collection locations and setups are included? How much variation is there in backgrounds, lighting, and acoustic conditions? Are any environments held out for evaluation?
  • What microphone is used, and how do you handle background noise and robot motor noise during deployment?
  • Which sensors do the baseline and FEEL-augmented policies receive? How are FEEL features integrated and sensor streams synchronized?
  • What varies across robot trials—for example, object poses, initial contents for pouring, or board placement? Are more trials or same-input fusion comparisons available?
Reviewer_VuSSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies fine-grained multisensory representation learning for robotic manipulation. It has three main contributions: (1) FEELCAP, an egocentric multisensory capture system that synchronously records RGB, depth, tactile, audio, force/torque, IMU, and gripper-state signals; (2) FEELING, a dataset of 1,928 task groups, 9,636 episodes, and 37,391 action segments with fine-grained action and object annotations, plus event, visual–geometry, and physical text descriptions; and (3) FEEL, a representation learning framework trained with semantic-space alignment, intra-modal masked reconstruction, and cross-modal generation. The method is evaluated on cross-modal retrieval, modality generation, and three real-world robot manipulation tasks with π0 and EgoVLA policies. The paper claims that FEEL captures cross-modal relationships while preserving modality-specific information and improves downstream policy success rates.

Critical strengths
  1. A practical synchronized multisensory data-collection platform and dataset. FEELCAP and FEELING address a real bottleneck in multisensory robot learning: synchronized observations of the same manipulation event. The dataset construction, annotation protocol, and three text descriptions are potentially useful beyond this paper. This strength is consequential because data scarcity is a central obstacle in this area.

  2. A coherent three-space, three-objective representation learning design. Dividing supervision into event, visual–geometry, and physical spaces is well motivated, and combining semantic alignment with intra-modal reconstruction and cross-modal generation is a reasonable way to preserve shared semantics and modality-specific details. This strength supports the paper’s technical contribution, although the current ablations are insufficient to verify each component.

Critical weaknesses
  1. The five-modality claim is not matched by the evaluation, especially for Force/Torque. The paper repeatedly emphasizes RGB, depth, tactile, audio, and force/torque, and the physical space is defined over tactile, force/torque, and audio. However, Table 1 reports retrieval only for vision, depth, audio, and tactile, and Table 2 does not clearly report force/torque generation results. Without force/torque evaluation, the central claim of fine-grained multisensory representation learning is not fully supported. This is consequential because force/torque is one of the most distinctive contact modalities for manipulation.
Questions for authors

1.Where are the Force/Torque retrieval and generation results? Can you report sensor–text retrieval and intra-/cross-modal reconstruction for force/torque, and quantify its contribution to the physical space and downstream tasks? 2.Can you provide a complete, clearly aligned Table 2 with all target modality × method × setting combinations? Were all baselines trained or adapted on the same FEELING training set with comparable decoder capacity and training budget? 3.How are FEEL features integrated into π0 and EgoVLA? Does the +FEEL setting increase model parameters or training budget? Can you report confidence intervals, multiple seeds, or additional trials for the real-robot experiments?

Reviewer_57XUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents FeelCap, a multisensory capture system; Feeling, a manipulation dataset; and Feel, a representation-learning model. It combines synchronized sensory recordings with fine-grained action and object-attribute annotations, and integrates semantic-space alignment, intra-modal reconstruction, and cross-modal prediction to learn representations that capture both shared semantics and sensory details. Evaluation includes retrieval, reconstruction, and real-world robotic manipulation.

Critical strengths
  • The integrated pipeline for data collection, annotation, and learning provides practical value as a dataset and system contribution.

  • Combining semantic alignment with sensory reconstruction is well motivated, addressing the preservation of both shared semantics and modality-specific information.

  • Retrieval and real-robot experiments demonstrate the potential usefulness of the learned representations.

Critical weaknesses
  1. The most critical issue is the poor readability of both the paper’s narrative and its experimental presentation. The paper appears hastily assembled, with substantial AI-generated content and frequent logical jumps. Unusual uses of bold and italics, combined with a confusing experimental design, further undermine clarity. Although the topic is interesting, the paper does not meet the standard for acceptance.

  2. The core design lacks controlled ablations evaluating the role of each modality under matched encoders, data, and training budgets. Additional egocentric pretraining data and heterogeneous modality-specific encoders also make it difficult to distinguish gains from the proposed method from those attributable to stronger initialization and additional training.

  3. The retrieval evaluation protocol is unclear, as are the downstream policy experiments. Do these tasks require audio input? What are the respective contributions of tactile and force information?

  4. The evidence for fine-grained cross-modal interaction understanding remains indirect. Text retrieval and MAE-based metrics only establish semantic matching, making it particularly difficult to assess the method’s effectiveness on contact-rich tasks.

  5. The paper repeatedly emphasizes force sensing in the earlier sections, but the subsequent experiments omit this modality altogether.

Questions for authors

see above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.