FEEL: Fine-grained Embodied Representation Learning for Robotic Manipulation

Multisensory feedback enables humans to perform daily tasks. When opening a soda bottle, vision locates the cap, tactile feedback reveals slip, force feedback indicates changes in resistance, and audio signals the release of gas. Learning comparable representations for robots requires synchronized sensory observations of the same manipulation event. We therefore aim to build fine-grained embodied representations that allow robots to understand the ongoing action and judge its interaction state from multiple senses. This leads to three questions: how can robots acquire such multisensory experience, how can it be organized into manipulation events, and how can representations be learned from it? Accordingly, we develop our framework along three corresponding aspects: data collection, dataset construction, and multimodal representation learning. For data collection, we develop FeelCap, an egocentric multisensory capture system that synchronously records RGB, depth, tactile, audio, force/torque, and IMU signals. Using FeelCap, we construct Feeling, an egocentric multisensory dataset of real-world manipulation events. We divide each trajectory into fine-grained interaction segments. Each segment is annotated with the action and manipulated objects, together with their visual and physical attributes. We propose Feel, trained with three objectives: semantic-space alignment associates sensory representations with the corresponding visual–geometry, physical, and event semantics; intra-modal masked reconstruction retains modality-specific sensory information; and cross-modal prediction learns the relationships among different sensory signals within the same manipulation event. We evaluate Feel on the Feeling Benchmark using sensor–text retrieval and modality generation. We further assess the learned representations on real-world robot manipulation tasks. Experimental results show that Feel effectively captures relationships across sensory modalities and improves policy performance on real-world tasks.