WHAT WILL THE FINGERS FEEL NEXT? A PROGRESSIVE MULTI-STEP TACTILE WORLD MODEL FOR PERSISTENT IN-HAND MANIPULATION

Dexterous in-hand manipulation requires a hand to repeatedly change contacts while sustaining object motion and avoiding drops. Vision and touch provide complementary observations, but simply combining their features leaves the consequences of actions on future contact implicit. We present TWIRL (Tactile World Model for In-hand Reinforcement Learning), a predictive visual–tactile policy built on a progressive multi-step tactile world model. TWIRL first learns immediate object and tactile dynamics, then extends this representation to anticipate contact evolution, rotation progress, and drop risk over a longer horizon. It predicts compact, action-conditioned object–tactile variables to inform control without future-image generation or online planning. A complementary fingertip-aligned fusion module uses calibrated kinematics to associate local visual features with tactile observations, grounding the policy in where contact occurs as well as how it may evolve. Persistent-rotation experiments across diverse objects in simulation show improved success over direct visual–tactile fusion, while real-world trials on multiple objects demonstrate effective transfer of the learned policy to a physical dexterous hand. Our results suggest that integrating progressive tactile dynamics prediction with fingertip-aligned visual feedback improves policy learning for sustained in-hand manipulation.