TGTP: General Trajectory-Guided Token Pruning for Visual Tracking via Reinforcement Learning

Token pruning in visual tracking aims to reduce computational cost by removing redundant tokens, with the key challenge lying in reliable token importance estimation. Existing methods often rely on current-frame cues such as attention responses, token similarity, or static pruning criteria, while making limited use of the temporal continuity inherent to tracking. Since target position, scale, and motion evolve continuously over time, historical trajectories can provide useful priors for token selection. To this end, we propose Trajectory-Guided Token Pruning (TGTP), a general reinforcement learning-based token pruning framework for visual tracking. The core of TGTP is a Trajectory-Guided Reinforcement Learning (TGRL) module, which formulates token pruning as a sequential decision-making problem. TGRL constructs states from recent target trajectories and learns pruning actions through a Trajectory Policy Network, predicting motion priors, motion uncertainty, and token keep ratios for the current frame. The policy is optimized with rewards that balance tracking accuracy, FLOPs, uncertainty regularization, and target-token preservation. As a lightweight plug-in module, TGTP can be integrated into different tracking architectures without redesigning their original structures. Experiments on multiple single-modal and multi-modal baselines show that TGTP substantially reduces search tokens and model FLOPs while maintaining or improving tracking performance, demonstrating its effectiveness and generality for computation-efficient visual tracking.