SKIP: Efficient Trajectory Prediction via Sparse Keyframe Interaction Propagation

Trajectory predictors often recompute agent and map interactions at every historical frame, even though nearby frames can share similar spatial context. We propose Sparse Keyframe Interaction Propagation (SKIP), which reduces these updates while retaining the full motion history. SKIP executes the native spatial block at selected keyframes and forms an interaction residual by subtracting the block input from its output. At each skipped frame, it adds the residual of the nearest keyframe to the same agent’s own feature. The receiving frame keeps its motion state; only the interaction increment is reused. SKIP introduces no extra parameters and leaves temporal processing, the decoder, and the training objective unchanged. We analyze its approximation error and computational cost, and evaluate it on QCNet and HiVT. On Argoverse 2, reducing QCNet’s spatial updates from 50 to 17 per layer increases forward throughput by 24.5% and reduces peak GPU memory by 43.0% at batch size 4, while validation minFDE changes from 1.5103 to 1.5045 m. On Argoverse 1, reducing HiVT’s local interaction updates from 20 to 7 increases forward throughput by 21.8% at batch size 32, with minFDE changing from 1.0304 to 1.0264 m. On these two backbones, spatial updates can be executed less frequently without discarding motion history or increasing validation minFDE.