Real-Time Action Chunking for Relative-Action Robot Policies

Relative-action representations offer an appealing action space for robot learning, as they reduce dependence on a specific robot configuration and facilitate scalable robot-free data collection. In such policies, each action chunk is expressed relative to its initial end-effector pose, referred to as the anchor pose. Deploying these policies on robots requires real-time execution for fast and continuous operation. Existing real-time execution methods achieve this by aligning the overlapping actions of consecutive chunks. However, for relative-action policies, such alignment does not ensure consistent absolute target poses because consecutive chunks use different anchor poses. Controller tracking error can further shift these anchors during execution, introducing additional cross-chunk inconsistency. Robot-free demonstrations make this problem more challenging, as they provide no direct supervision for deployment-time tracking errors. In this work, we introduce RRTC, a tracking-error-aware Relative-action Real-Time Chunking framework. RRTC re-expresses the overlapping trajectory of the preceding chunk relative to the robot's realized pose, thereby preserving cross-chunk absolute-target consistency. It further conditions action generation on the tracking error, allowing the policy to adapt accordingly. To enable such training with robot-free demonstration data, RRTC synthesizes pseudo tracking errors from temporal offsets along demonstration trajectories. Experiments across diverse simulated and real-world manipulation tasks show that RRTC enables smoother and more efficient real-time execution while maintaining robust task performance. Source code is available at https://anonymous.4open.science/r/RRTC-ICLR-2141