What to Attend, What to Keep: Skill-Conditioned Visuotactile Representation with Progress-Guided Event Memory

Robotic manipulation integrates vision, touch, and language, whose importance shifts across stages: vision guides reaching, while touch, through its evolution over time, decides grasping, alignment, and contact. Yet existing multi-modal manipulation policies typically use fixed temporal contexts and fusion strategies, despite shifts in what each modality contributes across different skills. We study how vision and touch should be combined at the level of primitive skills, asking what each skill needs from each sensor, and propose a language-conditioned representation in which the queried skill selects modality-specific encoders and horizons and attends to a sparse event memory holding the terminal states of the last K completed skills. Evaluated by skill progress estimation on three contact-rich tasks, it reduces slip-detection delay by 87% against fine-tuned SOTA progress models, twist-completion delay by 67.5% against a vision-only ablation, and progress error on a blind search task by 92% through sparse event memory. Gains concentrate exactly where completion is defined by contact or task history. More broadly, our results suggest that observation formation—not only policy architecture—is a central challenge in multi-modal representation. All data, models, and code will be open-sourced upon release.