TrackRerank: Semantic Enhancement for Two-Stage Open-Vocabulary Tracking

A two-stage open-vocabulary tracker can follow an object correctly while repeatedly assigning it the wrong category. We introduce \method, a causal post-association semantic head that revisits category decisions without changing boxes, detection confidence, or identities. It gathers candidates from detector predictions, current-frame vision–language retrieval, and past track observations. A category-shared scorer interprets each candidate through its source confidence, relative competition, uncertainty, cross-source support, and track context. On a custom video-disjoint OVT-B split, the complete pipeline raises YOLOE–ByteTrack from 13.307 to 25.300 ClsA (+11.993 points), exceeding causal voting and EMA baselines. Under shared candidates, supervision, and optimization, the full representation gains 1.050 points over a missingness-aware score-fusion MLP with nearly identical parameter count, with positive gains in all three paired seeds. The same trained head, connected through an observation-aligned adapter, improves native OVTR on TAO by 1.390 causal ClsA points without target-domain fitting. Its within-group accuracy gains are larger when current sources conflict or are uncertain. These results distinguish the benefit of gathering additional semantic evidence from the benefit of interpreting its candidate-level context.