ReTrack: Learning Recovery Decisions from Trajectory Replay

Visual trackers commonly assume limited target displacement across consecutive frames and localize the target within a local search region cropped around the previous prediction. However, the tracker may lose a small target when rapid motion moves it outside this local window or brief occlusion disrupts localization. It may then drift onto background clutter and fail to recover. To recover a lost target, existing trackers combine local tracking with full-frame re-detection, often relying on a separately trained external detector and hand-crafted rules based on current-frame confidence to trigger recovery. Nevertheless, these rules decide whether to switch from current-frame cues, without training on how staying and switching affect the subsequent trajectory. To close this gap, we introduce ReTrack, which learns recovery decisions through trajectory-replay supervision. By rolling out the stay and switch trajectories over subsequent frames and evaluating both against the ground-truth trajectory, we directly measure the relative advantage of switching. At inference, a lightweight policy predicts this advantage from relative geometric, confidence, and temporal cues. It then decides whether to retain the tracker prediction or switch to a full-frame candidate. These candidates are provided by a full-frame branch that shares the same backbone as the tracker, eliminating the need for a separately trained detector. With full-frame candidates and learned recovery decisions, ReTrack achieves 49.07% AUC on UAV-Anti-UAV, surpassing the previous best result by 5.04% AUC, and improves performance on three additional drone tracking benchmarks.