RMOT-Compass: A Query-Conditioned Error Attribution Framework for Referring Multi-Object Tracking

Referring multi-object tracking (RMOT) tracks the objects described by a language query and preserves their identities over time. The query decides which scene objects are targets and which are distractors. Thus, a trajectory can be accurate and consistent, yet still wrong for the query. However, an aggregate score such as HOTA does not reveal how different output errors contribute to the recoverable gap. We argue that error diagnosis in RMOT should answer three distinct questions: how often an error occurs, what is gained by repairing it alone, and how much of the jointly recoverable gap it accounts for. In this work, we propose RMOT-Compass, a query-conditioned error attribution framework. It combines the language query with full-scene annotations to define target and distractor roles, and freezes diagnostic correspondences to keep error labels fixed across repairs. Joint repairs address five error types, covering missing targets, distractors, background or duplicate outputs, association, and localization, while coordinating target recovery with identity and box correction across frames. Standard Shapley allocation accounts for repair interactions by averaging marginal gains over all repair orders. The credited gains sum exactly to the jointly recoverable HOTA gap. Across five RMOT methods evaluated on the Refer-KITTI dataset, missing targets are consistently the largest recoverable error. Methods with higher overall scores can still have larger recoverable contributions from specific error types. Both findings hold on the Refer-KITTI-V2 dataset. Therefore, RMOT-Compass turns a single aggregate score into a query-aware error budget. It identifies which output errors account for the recoverable gap and directs focused model investigation.