acceptodds
Under review as a conference paper at ICLR 2027

RMOT-Compass: A Query-Conditioned Error Attribution Framework for Referring Multi-Object Tracking

Abstract

Referring multi-object tracking (RMOT) tracks the objects described by a language query and preserves their identities over time. The query decides which scene objects are targets and which are distractors. Thus, a trajectory can be accurate and consistent, yet still wrong for the query. However, an aggregate score such as HOTA does not reveal how different output errors contribute to the recoverable gap. We argue that error diagnosis in RMOT should answer three distinct questions: how often an error occurs, what is gained by repairing it alone, and how much of the jointly recoverable gap it accounts for. In this work, we propose RMOT-Compass, a query-conditioned error attribution framework. It combines the language query with full-scene annotations to define target and distractor roles, and freezes diagnostic correspondences to keep error labels fixed across repairs. Joint repairs address five error types, covering missing targets, distractors, background or duplicate outputs, association, and localization, while coordinating target recovery with identity and box correction across frames. Standard Shapley allocation accounts for repair interactions by averaging marginal gains over all repair orders. The credited gains sum exactly to the jointly recoverable HOTA gap. Across five RMOT methods evaluated on the Refer-KITTI dataset, missing targets are consistently the largest recoverable error. Methods with higher overall scores can still have larger recoverable contributions from specific error types. Both findings hold on the Refer-KITTI-V2 dataset. Therefore, RMOT-Compass turns a single aggregate score into a query-aware error budget. It identifies which output errors account for the recoverable gap and directs focused model investigation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Referring Multi-Object Tracking, Error Attribution, Query-Conditioned Analysis, Error Diagnosis, Shapley Value

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Ny3EOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes an error attribution framework for referring multi-object tracking (RMOT) named RMOT-Compass, which can identify five observable error types and provide isolated gains for fixing each type of error. The paper presents a comparative diagnosis of five RMOT methods on the Refer-KITTI dataset and identifies that error due to missing targets is the dominant recoverable component.

Critical strengths
  • The paper is well-motivated, as existing RMOT frameworks require a deeper analysis of the failure modes so that future works can divert their focus to those directions.
  • The types of errors and gains after repair are well categorized to distinguish the performance of different RMOT methods.
  • The evaluation makes comparisons for isolated gains and joint attribution for repairing different error types.
Critical weaknesses
  • The novelty is incremental compared to the prior work TIDE, despite the clarifications in the introduction section. The main difference comes from the RMOT task itself, which requires not just detection and association, but also conditioning on text queries. The proposed method is still largely based on the TIDE pipeline.
  • It is unclear how these categorized errors can inform the design. For example, the "MISS" error could be due to failures in different parts of the RMOT framework, and therefore, the number alone is not very helpful. In addition, the Shapley attribution is based on the assumption of perfectly repairing the errors, which is usually not true, so an error type that contributes the most to the overall error rate may not be the best direction to optimize.
Questions for authors
  • Can the authors provide a concrete example where the joint attribution based on Sharpley provides a more helpful diagnosis for RMOT model design?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Ny3E

We sincerely thank you for your thorough review and valuable insights. We address your comments below.

W1: The novelty is incremental compared to the prior work TIDE, despite the clarifications in the introduction section. The main difference comes from the RMOT task itself, which requires not just detection and association, but also conditioning on text queries. The proposed method is still largely based on the TIDE pipeline.

RMOT-Compass builds on TIDE’s output-diagnosis paradigm. Our methodological contribution lies in defining errors and composable repairs for query-dependent target selection and temporal identity in RMOT and in allocating the jointly recoverable score gap beyond reporting isolated repair gains.

Our error taxonomy addresses query constraints and temporal identity in RMOT. Two key distinctions from TIDE’s taxonomy are query-distractor errors (DIST) and identity-association errors (ASS). DIST describes a real scene object that falls outside the current query’s target set. Even if its category and localization are correct, outputting it remains an error, and DIST repair removes that output. ASS captures temporal identity errors, which TIDE’s static detection taxonomy does not define. When restoring missing targets, we coordinate the identities of inserted detections with ASS repair while keeping evaluation ground truth fixed and diagnostic correspondences frozen across repair combinations (Sections 3.2, 3.3, and 3.4 of the manuscript).

TIDE recognizes non-additivity and reports isolated repair gains. We retain these gains and additionally evaluate all 32 repair combinations, using the established Shapley rule to average marginal contributions across repair contexts (Section 3.5 of the manuscript). The resulting allocations sum to the jointly recoverable gap. For TransRMOT, the manuscript reports 40.645 HOTA points for the sum of isolated gains and 54.524 points for the joint gap under the same repair protocol (Figure 1 and Table 9 of the manuscript). The 13.879-point difference illustrates why isolated gains alone do not provide a complete allocation of joint recovery.

W2: It is unclear how these categorized errors can inform the design. For example, the "MISS" error could be due to failures in different parts of the RMOT framework, and therefore, the number alone is not very helpful. In addition, the Shapley attribution is based on the assumption of perfectly repairing the errors, which is usually not true, so an error type that contributes the most to the overall error rate may not be the best direction to optimize.

Our error taxonomy quantifies output failure modes across RMOT methods, while repair analysis measures their impact on evaluation metrics to inform model inspection and design improvement. MISS can originate from multiple internal stages. One can inspect whether candidates cover the target and where the corresponding candidates are rejected or lost during filtering or tracking. These checks narrow the investigation.

Shapley attribution allocates the score gap recoverable under the specified oracle repairs. A large allocation highlights a failure worth investigating, but practical optimization priorities also depend on feasibility, implementation cost, and effects on other errors. We use attribution and repair interactions to formulate design hypotheses, then evaluate actual interventions, as illustrated in Q1.

Q1: Can the authors provide a concrete example where the joint attribution based on Sharpley provides a more helpful diagnosis for RMOT model design?

We use TempRMOT as an example. DIST repair yields 6.050 HOTA points in isolation and 8.577 after MISS repair. Together with the positive Shapley interaction between MISS and DIST, this motivates jointly testing foreground supervision and query-distractor suppression.

We conduct an exploratory four-condition experiment. C retains the original losses. A adds positive supervision for matched low-confidence foreground candidates. B adds negative referring supervision for matched non-target objects with high referring scores. AB combines both terms. For all conditions, we start from the same pretrained checkpoint, train for ten epochs, and evaluate the final checkpoint on the Refer-KITTI test set using identical inference thresholds. The experimental results are shown below.

ConditionHOTAChange from C
C52.2870
A51.769−0.518
B51.855−0.432
AB52.427+0.140

As shown in the table, adding B alone decreases HOTA by 0.432 points, whereas adding B when A is present increases HOTA by 0.658 points. AB yields a net HOTA gain of 0.140 points over C. This case illustrates how joint diagnosis identifies design directions worth investigating together. In this experiment, judging modifications solely by their individual effects misses a favorable combination, while joint diagnosis provides a reason to test it.

I would like to thank the authors for the clarification and extra experiments, which helped make the contribution clearer. I would maintain a positive score for this paper.

AuthorsOpenReview ↗

Official Comment by Authors

Thank you for your thoughtful feedback and for recognizing the value of our clarifications and additional experiments. We are also grateful that you maintain a positive score.

Reviewer_JyhpOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RMOT-Compass, an error attribution framework for Referring Multi-Object Tracking (RMOT). The framework categorizes tracking failures into five error types and measures their contributions to HOTA through oracle repairs and Shapley decomposition. By accounting for interactions among different error types, it provides a quantitative breakdown of the recoverable performance gap. Experiments on five RMOT trackers demonstrate that missing targets consistently contribute the most to performance degradation.

Critical strengths
  1. Useful framework for tracking error analysis.

RMOT performance is challenging to diagnose. The paper provides a systematic and controlled framework for analyzing different types of tracking failures and their impact on performance.

  1. Principled error attribution.

The use of Shapley decomposition accounts for interactions among different error types and provides a quantitative attribution of their contributions to HOTA.

  1. Extensive evaluation across trackers.

The authors analyze five RMOT methods with different architectures and demonstrate consistent findings across methods and datasets.

Critical weaknesses
  1. Limited additional insights beyond existing evaluation metrics.

Many of the paper's main findings can be inferred from the breakdown metrics of HOTA (e.g., DetA, AssA, and LocA) or through manual error analysis. While the proposed framework automates and quantifies this analysis, it is unclear how much additional insight it provides beyond existing evaluation practices.

  1. Limited explanation of the underlying causes of tracking failures.

The framework attributes HOTA degradation to different observable error types but does not explain why the tracker makes these mistakes. For example, missing targets may result from detection failures, incorrect language grounding, or tracking failures.

  1. Limited methodological novelty.

Although the paper introduces an automated analyze framework, its core methodology relies on the existing Shapley decomposition. Moreover, as discussed above, some of the resulting insights can already be obtained from existing metrics, while the framework provides limited understanding of the underlying model failures. These limitations raise questions about the significance of the methodological contribution and the additional value of the proposed analysis.

Questions for authors

Please see Weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer Jyhp (1/2)

We appreciate your careful review and valuable feedback. We address your comments point by point below.

W1: Limited additional insights beyond existing evaluation metrics. Many of the paper's main findings can be inferred from the breakdown metrics of HOTA (e.g., DetA, AssA, and LocA) or through manual error analysis. While the proposed framework automates and quantifies this analysis, it is unclear how much additional insight it provides beyond existing evaluation practices.

RMOT-Compass complements HOTA components by distinguishing error occurrence, isolated repair gains, and joint contributions. Four results illustrate this additional diagnostic information.

1. Metric components do not map one-to-one to error types.

DetA does not separate missing targets, tracked non-target objects, and background/duplicates. Likewise, the recoverable AssA gap does not arise only from ASS. The table reports raw AssA and Shapley contributions in AssA points (Tables 6 and 7 in the manuscript). Contributions are weighted averages of marginal gains across repair contexts.

MethodRaw AssAMISS contribution to AssAASS contribution to AssA
TransRMOT57.96925.75513.177
TempRMOT66.86119.3017.395
DKGTrack66.08020.5647.657
iKUN62.03616.7606.022
FlexHook66.91619.0966.451

MISS contributes more than ASS to the recoverable AssA gap for every method. Therefore, an association score alone cannot determine the contribution of association errors. This diagnosis extends beyond subdividing DetA.

2. Higher scores can coexist with larger error contributions.

On Refer-KITTI, FlexHook scores higher on HOTA and all components, yet its DIST and LOC Shapley contributions are larger (HOTA points, Tables 6 and 7 in the manuscript).

Metric or attributionTransRMOTFlexHook
HOTA45.47653.824
DetA35.81343.354
AssA57.96966.916
LocA90.24090.872
DIST contribution to HOTA6.7309.275
LOC contribution to HOTA5.8846.787

These contributions identify error categories worth investigating beyond the overall ranking. Higher component scores do not imply smaller recoverable contributions from every error type.

3. Lower scores across datasets accompany different error shifts.

We use each dataset's corresponding checkpoint. Contributions are in HOTA points (Tables 7 and 16 in the manuscript).

MethodDatasetHOTAMISS contribution to HOTADIST contribution to HOTA
TempRMOTRefer-KITTI52.01226.3358.336
TempRMOTRefer-KITTI-V234.34547.6069.235
FlexHookRefer-KITTI53.82424.8379.275
FlexHookRefer-KITTI-V242.52623.47217.489

Both methods score lower on Refer-KITTI-V2, but TempRMOT's MISS contribution rises, whereas FlexHook's MISS contribution decreases slightly and its DIST contribution rises. These profiles suggest different investigation priorities that the overall score drop alone does not identify.

4. Error counts and isolated gains do not capture joint contributions.

Under the near-exact box-equality criterion, all five methods have a LOC rate of 1.0000, yet LOC's HOTA contributions range from 5.884 to 14.493 points (Tables 7 and 8 in the manuscript). Therefore, occurrence rates do not measure recoverable impact.

For all five methods, the sum of isolated repair gains is smaller than the joint recoverable gap (HOTA points, Table 9 in the manuscript).

MethodJoint recoverable gapSum of isolated repair gainsDifference
TransRMOT54.52440.64513.879
TempRMOT47.98838.4439.545
DKGTrack50.69841.0759.623
iKUN55.66348.2387.425
FlexHook46.17638.4237.753

The rankings also differ. For TransRMOT, repairing LOC alone yields a gain of 3.54 points, compared with 3.47 for ASS. Joint attribution reverses this order, assigning 6.120 points to ASS and 5.884 points to LOC (Fig. 4 and Table 7 in the manuscript). Removing DIST yields a gain of 4.139 points alone and 6.716 points after MISS repair (Fig. 1 in the manuscript).

Manual inspection can identify errors. Obtaining these contribution profiles, conditional gains, and interaction-aware rankings additionally requires explicit repairs, coalition evaluation, and joint analysis. RMOT-Compass makes this process systematic and reproducible.

AuthorsOpenReview ↗

Response to Reviewer Jyhp (2/2)

W2: Limited explanation of the underlying causes of tracking failures. The framework attributes HOTA degradation to different observable error types but does not explain why the tracker makes these mistakes. For example, missing targets may result from detection failures, incorrect language grounding, or tracking failures.

The same output error can arise from different internal processes, and RMOT methods differ in their architectures and processing pipelines. Therefore, RMOT-Compass uses a common output-level diagnostic interface to quantify each error type's contribution to the recoverable performance gap under the specified repair protocol. This provides a basis for further investigation. Identifying specific internal failure stages requires intermediate records from the corresponding model.

To examine a concrete failure pathway, we remove target candidates before FlexHook's grounding stage while keeping its checkpoint and queries fixed. The table shows the final MISS rate at baseline and at two removal proportions. Changes are absolute rate differences (Section 4.4 and Table 12 in Appendix E of the manuscript).

Target candidate removal proportionFinal MISS rateAbsolute change from baseline
0, baseline0.34650
0.100.4168+0.0703
0.300.5517+0.2052

In this controlled setting, reduced candidate coverage increases final omissions. This shows how upstream candidate omissions can propagate through a two-stage pipeline to the final RMOT output.

W3: Limited methodological novelty. Although the paper introduces an automated analysis framework, its core methodology relies on the existing Shapley decomposition. Moreover, as discussed above, some of the resulting insights can already be obtained from existing metrics, while the framework provides limited understanding of the underlying model failures. These limitations raise questions about the significance of the methodological contribution and the additional value of the proposed analysis.

Our methodological contribution lies in constructing query-conditioned diagnosis and temporally coordinated joint repairs for RMOT. We use the query and full-scene annotations to determine target and distractor roles and define observable output errors. Freezing the original diagnostic correspondences prevents subsequent repairs from changing the original error definitions. The joint repair operator then coordinates target insertion, identity correction, and localization correction. Each combination of errors selected for repair yields a well-defined output that can be evaluated independently (Sections 3.2 to 3.4 in the manuscript).

For example, restoring a missing target requires both inserting boxes and assigning its identity across frames. When only MISS is repaired, existing prediction IDs remain unchanged, and inserted predictions receive IDs through a safe-fill rule. When MISS and ASS are repaired together, existing and inserted predictions for the same target receive a common identity (Section 3.4 and Appendix A.3 of the manuscript). These rules determine the output and evaluation score for each repair combination. Shapley then uses these scores to allocate contributions. Therefore, constructing the joint repairs is necessary for analyzing error interactions and contributions.

AuthorsOpenReview ↗

Looking Forward to Your Feedback on the Rebuttal

Dear Reviewer Jyhp,

Thank you again for your thoughtful review and for raising important questions about RMOT-Compass. We have posted point-by-point responses addressing your concerns, including additional quantitative analyses and clarifications of our diagnostic framework and methodological contribution.

As the discussion phase nears its deadline, we would be grateful if you could take another look at our responses and let us know whether they adequately address your concerns or whether any further clarification would be helpful.

We understand you may have limited time and sincerely appreciate your consideration.

Best regards,

The Authors

Reviewer_YLoFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the evaluation of Referring Multi-Object Tracking (RMOT) by proposing a query-conditioned error attribution framework called RMOT-Compass. The authors argue that aggregate metrics like HOTA fail to reveal how specific output errors contribute to performance gaps. In RMOT, whether an object is a "target" or a "distractor" depends on the language query; therefore, the role of an object must be determined based on the query before errors can be defined. The authors further contend that error diagnosis should distinguish between three aspects: the frequency of error occurrences, the gain achieved by fixing a specific type of error in isolation, and the share of the total performance gap attributable to a particular error type when multiple fixes are applied collectively. The paper terms the latter two concepts "individual gain" and "joint attribution," respectively.

Methodologically, the framework leverages language queries and full-scene annotations to categorize objects as either targets or distractors. It establishes and freezes a diagnostic correspondence between predictions and ground-truth objects, based on which five observable error types are defined: missed targets, outputs corresponding to distractors, outputs corresponding to background or redundant detections, association errors, and localization errors. Each error type is associated with a specific correction operator: MISS involves inserting the ground-truth bounding box (with the target's identity coordinated based on whether an associated ASS error is also corrected); DIST and BKG-DUP involve removing the corresponding outputs; ASS involves assigning the correct target ID; and LOC involves replacing a predicted bounding box with the ground-truth box. The framework performs independent re-evaluations using TrackEval for all 32 possible subsets of these corrections to derive coalition values ​​(in the context of cooperative game theory). It then employs the standard Shapley value to precisely allocate the total recoverable HOTA gap among the five error types, while using the Shapley interaction index to characterize the complementarity between different types of corrections.

In the experiments, the authors utilize the official model weights for five RMOT methods—TransRMOT, TempRMOT, DKGTrack, iKUN, and FlexHook—without retraining. These models are diagnosed on the Refer-KITTI test set, and the same protocol is applied to two of these methods on the Refer-KITTI-V2 dataset. There are three main conclusions: Among all methods, MISS yields the largest attribution, ranging from 24.8 to 31.3 HOTA points; methods with higher HOTA scores may, paradoxically, exhibit greater attribution for certain types of errors; and the rankings produced vary depending on whether one adopts the perspective of incidence, individual gain, or joint attribution. The appendix further presents sensitivity analyses regarding thresholds, confidence intervals derived via the query-macro bootstrap method, controlled perturbations of FlexHook input candidates, experiments involving query rewriting, and a regression analysis of task difficulty.

Critical strengths
  1. The diagnostic metrics are clearly defined, and the attribution calculations are precise and reproducible. The paper explicitly distinguishes between three types of diagnostic metrics—incidence, individual gain, and joint attribution—and uses TransRMOT to demonstrate that TIDE-style individual-fix analysis systematically underestimates the total performance gap in multi-object tracking: the sum of individual gains is 40.645 points, whereas the joint recoverable gap is 54.524 points. The framework performs exact evaluations for all 32 possible coalitions (avoiding sampling approximations), keeps diagnostic correspondences fixed to prevent label drift during the "fixing" process, and provides comprehensive implementation specifications and numerical audits. This constitutes the primary value of the paper as an evaluation tool and is why I believe it offers meaningful insights for the research community.

  2. The experimental reporting is transparent, with no selective presentation of results. The authors utilize the official model weights for five representative methods—covering both end-to-end and two-stage paradigms—and faithfully report findings that run counter to their own narrative: three controlled perturbations failed to yield the expected direction or statistical significance; none of the "query-rewriting" experiments remained significant after Benjamini-Hochberg (BH) correction; and discrepancies were observed between the global and "query-macro" estimators. While this level of transparency enhances the credibility of the results, it simultaneously exposes certain validity issues, which are discussed further below.

Critical weaknesses
  1. Query conditioning fails to separate the two core failure modes. It merges “never detected” and “detected but rejected by the referring module” into MISS, so the decomposition cannot distinguish whether fixes should target detection-tracking or the referring module. Its added value over standard MOT error decomposition is unclear.
  2. Shapley attributions largely follow inherent HOTA mathematical properties. The observed ranking and magnitude of gains are predictable from HOTA’s formula and miss rates. It is not demonstrated what extra diagnostic insight Shapley brings compared with closed-form baselines or simple event counting.
  3. Attributions depend on arbitrary, consequential modeling choices. Results are sensitive to game design decisions including how MISS is defined, ID assignment rules, and uneven event granularity. Stability under reasonable alternative setups is not verified, and the choice of Shapley over other indices lacks justification.
Questions for authors

Q1. Query conditioning fails to separate RMOT's two essential failure modes.Query conditioning only partitions false positives into DIST and BKG-DUP—both are FPs under HOTA with identical deletion effects—while the distinction that matters (never detected vs. tracked but rejected by the referring module) is collapsed into MISS, the largest attribution (24.8–31.3). The framework therefore cannot say whether to fix detection-tracking or referring, contrary to its stated goal. For iKUN/FlexHook, MISS could be split into "absent from candidates" and "present but unselected." Explain why this was not done and what query conditioning adds over generic MOT error decomposition.

Q2. The findings follow from HOTA's algebra.DetA = T/(T+F+P). Deleting FPs alone gains T·P/((T+F)(T+F+P)); repairing misses first, then deleting FPs, gains P/(T+F+P); the ratio is (T+F)/T = 1/(1−miss rate) > 1. TransRMOT's 0.4285 miss rate predicts 1.75 vs. the reported ≈1.62, the rest attributable to AssA and threshold averaging. Likewise, ASS–LOC is near-tautological (LOC repair raises TPs at high thresholds), the DIST/BKG-DUP attribution ratio roughly tracks event counts (1.33 vs. 1.51; 4.73 vs. 4.06), and MISS dominance follows from 0.35–0.43 miss rates and low DetRe. Show what Shapley attribution adds beyond HOTA's sub-metrics and counts—e.g., against a closed-form or regression baseline.

Q3. Full-recovery results are tautological; validation is internal only.Repairs use GT boxes and IDs, so full repair yields HOTA 100 and the jointly recoverable gap ≡ 100−HOTA; Table 3/App. C.1 (zero residual, 1e-9 efficiency error, non-negative marginals) only confirm implementation and the efficiency identity. No external validity is tested: do real detector/association/referring improvements yield HOTA gains matching the corresponding attribution, and does attribution ranking guide improvements better than standalone gains or incidence? Otherwise "error budget," "recoverable," and the Section 5 claims are unsupported.

Q4. Attributions hinge on arbitrary game design choices.As Sundararajan & Najmi note, Shapley values depend on how the game is operationalized. Three choices here are consequential: (i) MISS inserts a score-1 exact GT box, charging unmatched targets' localization loss to MISS while matched-but-poorly-localized targets charge LOC—an asymmetry that may inflate MISS; (ii) the ID rule (safe original vs. canonical IDs) determines MISS's gain and the MISS–ASS interaction; uniformly new IDs would change all results; (iii) granularity is highly uneven (37,982 LOC vs. 11,495 BKG-DUP events) and Shapley is not split-invariant. Report stability under equally reasonable alternatives, and justify symmetry/additivity over Banzhaf or Owen values for this diagnostic use.

Q5. Diagnostic labels do not match HOTA's matching.Diagnostics match at IoU 0.05 with a max-cardinality lexicographic objective; HOTA matches separately at every 0.05–0.95 threshold using association scores. A 0.06-IoU prediction can be labeled LOC (and ASS) yet is FP+FN at most HOTA thresholds, so labels need not correspond to metric penalties. The 0.05 choice is unjustified; in Table 4 only MISS's rank survives, while iKUN's LOC drops 14.493→11.922 (≈18%) and at 0.30 TempRMOT's BKG-DUP exceeds ASS. State the threshold rationale and report the diagnostic/metric mismatch rate per threshold.

Q6. The LOC definition is degenerate, making the incidence argument circular.With LOC defined as IoU < 1−1e-6, incidence is 1.000 for all methods (Table 8) and stays saturated under perturbations (Table 12). The claim that "identical incidence, attributions 5.88–14.49" shows incidence is uninformative thus rests on this degenerate definition. Re-test under the non-degenerate IoU < 0.5 severe-error label already defined in the appendix.

Q7. Uncertainty is absent and the bootstrap basis is weak.Refer-KITTI has only 3 test videos (0005, 0011, 0013), so video-level block bootstrap gives 10 resamples with no coverage guarantee; V2 has 4 videos. All global attributions (Fig. 3; Tables 4, 7, 9, 10) lack uncertainty despite tiny key gaps (ASS 6.12 vs. LOC 5.88; standalone 3.47 vs. 3.54), and query-macro intervals are very wide (BKG-DUP 1.054–13.968). Provide uncertainty estimates and justify the bootstrap with three clusters.

Q8. Global and query-macro estimates disagree; the target estimand is undefined.iKUN's MISS is 25.154 globally (second smallest) but 52.764 under query-macro (largest); LOC goes 14.493→9.015. TransRMOT's ASS exceeds LOC globally (6.120 vs. 5.884) but not under query-macro (3.879 vs. 5.717), erasing the Section 4.3/Fig. 4 reversal. Since the diagnostic answer depends on the estimator, state which estimand is intended and why, and whether the 4.2/4.3 conclusions hold under the other.

Q9. Perturbations neither validate attribution nor inject controlled error types.Appendix E.1 confirms post-perturbation 32-coalition attributions are never computed, so the core output is untested against known interventions; perturbations precede FlexHook, conflating framework validity with model response. Only target removal behaves as expected and significantly; ID fragmentation crosses zero, distractor insertion shifts DIST oppositely, and LOC stays saturated. High-dose distractor insertion adds 20,429 predictions but only 4,696 DIST events, leaving ~15.7k unexplained (possible duplicates or other queries' targets). Inject known-count/type errors into real outputs and test whether attribution recovers them specifically.

13 comments
AuthorsOpenReview ↗

Response to Reviewer YLoF (1/8)

Thank you for your careful review and constructive comments, which we address point by point below.

W1: Query conditioning fails to separate the two core failure modes. It merges “never detected” and “detected but rejected by the referring module” into MISS, so the decomposition cannot distinguish whether fixes should target detection-tracking or the referring module. Its added value over standard MOT error decomposition is unclear.

RMOT-Compass uses final predictions as a common interface for end-to-end and two-stage methods, whose intermediate outputs differ. MISS identifies unmatched query targets, while determining its internal source requires additional records. Query conditioning separates outputs on annotated non-target objects (DIST) from background or duplicate outputs (BKG-DUP). Q1 below provides the iKUN/FlexHook candidate-source audit and numerical comparisons showing what these distinctions reveal.

Q1: Query conditioning fails to separate RMOT's two essential failure modes. Query conditioning only partitions false positives into DIST and BKG-DUP—both are FPs under HOTA with identical deletion effects—while the distinction that matters (never detected vs. tracked but rejected by the referring module) is collapsed into MISS, the largest attribution (24.8–31.3). The framework therefore cannot say whether to fix detection-tracking or referring, contrary to its stated goal. For iKUN/FlexHook, MISS could be split into "absent from candidates" and "present but unselected." Explain why this was not done and what query conditioning adds over generic MOT error decomposition.

Our main analysis uses final predictions as a common interface to compare methods using the same type of available information. This interface alone cannot determine the internal source of an omission. Therefore, we adopt a unified MISS category. For iKUN and FlexHook, we use candidate and selection records to further distinguish its sources.

We keep the final predictions used in the manuscript and their original MISS events fixed, covering 150 Refer-KITTI queries per method. The counting unit is query × frame × GT object. Replaying intermediate records under the original selection rules reproduces the final outputs. The candidate coverage threshold is IoU=0.05, matching the original MISS diagnostic threshold. Events with no candidate reaching this threshold are classified as uncovered. Events with one-to-one candidate coverage but no retained threshold-qualified candidate are classified as covered but not retained. Candidate competition and cases with retained overlapping outputs but an unmatched target are classified as ambiguous/other. The results are shown in the table below:

MethodOriginal MISS eventsNo candidate coverageCovered but not retainedAmbiguous/other
iKUN25,75227.38%53.93%18.69%
FlexHook23,02843.69%31.58%24.73%

As shown in the table, iKUN's MISS events predominantly fall into the covered-but-not-retained category, whereas for FlexHook, the proportion of uncovered events is higher than that of covered-but-not-retained events. This difference provides provenance information beyond the unified MISS value.

All 13,888 covered-but-not-retained events for iKUN have recorded scores showing that every qualifying candidate is rejected by the original selection rule. Of the 7,273 corresponding events for FlexHook, 6,134 have this evidence, while 1,139 involve candidates without scores for the corresponding query. These events cannot all be classified as referring rejection. Candidate coverage is assessed at the specified IoU threshold, so an uncovered target may have poorly localized candidates and is not equivalent to an object that is never detected.

Query conditioning uses full-scene annotations to distinguish outputs on real non-target objects (DIST) from background or duplicate outputs (BKG-DUP). For FlexHook versus TransRMOT, HOTA is 53.824 versus 45.476, DIST attribution is 9.275 versus 6.730, and BKG-DUP attribution is 2.284 versus 4.465 HOTA points. Higher overall HOTA coexists with larger DIST and smaller BKG-DUP contributions. Both categories use deletion repairs, but apply to different output sets. Pooling them removes this distinction, while the intermediate-record audit supplies additional omission-source information.

AuthorsOpenReview ↗

Response to Reviewer YLoF (2/8)

W2: Shapley attributions largely follow inherent HOTA mathematical properties. The observed ranking and magnitude of gains are predictable from HOTA’s formula and miss rates. It is not demonstrated what extra diagnostic insight Shapley brings compared with closed-form baselines or simple event counting.

HOTA's algebra and error frequencies explain some patterns.

First, a proportional isolated-gain baseline allocates the same joint gap as Shapley. It preserves 48/50 within-method category-pair orderings across five methods, but the mean absolute category difference ranges from 0.709 to 2.498 HOTA points. TransRMOT illustrates the magnitude differences:

CategoryProportional baseline (HOTA points)Shapley (HOTA points)
MISS37.57131.325
BKG-DUP1.9994.465

The total allocation is identical, so these differences reflect how the joint gap is distributed. The simple baseline recovers most rankings but does not reproduce contributions averaged over repair orders.

Second, a count-based approximation with fixed AssA closely matches TransRMOT's isolated DIST gain, at 4.126 versus 4.139 HOTA points. After MISS repair, the approximate and evaluated gains are 5.694 and 6.716 points. Across all five methods, this approximation underestimates the evaluated DIST gain after MISS repair by 1.022 to 1.321 points.

RMOT-Compass evaluates all 32 repair subsets, and Shapley averages each category's marginal gains across repair orders. The added information is the magnitude of each contribution and its dependence on other repairs. Q2 gives the baseline definitions and complete comparisons, including where event statistics preserve attribution rankings.

AuthorsOpenReview ↗

Response to Reviewer YLoF (3/8)

Q2: The findings follow from HOTA's algebra. DetA = T/(T+F+P). Deleting FPs alone gains T·P/((T+F)(T+F+P)); repairing misses first, then deleting FPs, gains P/(T+F+P); the ratio is (T+F)/T = 1/(1−miss rate) > 1. TransRMOT's 0.4285 miss rate predicts 1.75 vs. the reported ≈1.62, the rest attributable to AssA and threshold averaging. Likewise, ASS–LOC is near-tautological (LOC repair raises TPs at high thresholds), the DIST/BKG-DUP attribution ratio roughly tracks event counts (1.33 vs. 1.51; 4.73 vs. 4.06), and MISS dominance follows from 0.35–0.43 miss rates and low DetRe. Show what Shapley attribution adds beyond HOTA's sub-metrics and counts—e.g., against a closed-form or regression baseline.

Your DetA derivation holds under fixed matching and the specified count changes. LOC repair can also change matching at higher thresholds and affect association evaluation. Therefore, the direction of an interaction alone does not establish additional diagnostic value. Diagnostic MISS uses frozen correspondences and is not equivalent to threshold-specific HOTA FNs. We test which attribution information simpler baselines recover.

First, we compare normalized isolated gains with the full Shapley allocation. Let V(S)V(S) denote global HOTA after repairing subset SS, and let CC contain the five error categories. Define the isolated gain as gk=V({k})−V(∅)g_k=V(\{k\})-V(\varnothing) and the joint gap as J=V(C)−V(∅)J=V(C)-V(\varnothing). The proportional baseline is

bk=Jgk∑j∈Cgj.b_k=J\frac{g_k}{\sum_{j\in C}g_j}.

This baseline allocates the same joint gap as Shapley. Mean absolute differences across the five categories are relative to the full Shapley allocation. The results are shown in the table below:

MethodMean absolute difference (HOTA points)Concordant category pairs
TransRMOT2.4989/10
TempRMOT1.1779/10
DKGTrack1.35310/10
iKUN1.23010/10
FlexHook0.70910/10

As shown in the table, the baseline preserves 48/50 category-pair orderings, with mean absolute differences of 0.709 to 2.498 HOTA points. For TransRMOT, its MISS allocation is 37.571 versus Shapley's 31.325, while BKG-DUP is 1.999 versus 4.465 points. These differences arise despite identical total allocation and show what proportional isolated gains do not reproduce about contributions averaged over repair orders.

Second, we extend your count analysis to approximate conditional HOTA gains. At each evaluation threshold, let T,F,PT,F,P denote the original TP, FN, and FP counts, with N=T+F+PN=T+F+P. Assuming that mm FNs become TPs and dd FPs are subsequently deleted, and additionally holding the original AssA fixed at AA, we obtain

=\sqrt{A\frac{T+m}{N-d}}-\sqrt{A\frac{T+m}{N}}.$$ Fixed AssA is an additional approximation we introduce for this comparison. We set $d$ to the diagnostic DIST count, with $m=0$ for isolated deletion and $m=F$ for the approximation after MISS repair. We average the approximate gains over all 19 HOTA thresholds. Evaluated gains come from rerunning TrackEval after applying the manuscript's repair protocol. The results are shown in the table below: | Method | Isolated DIST | | DIST after MISS | | |---|:---:|:---:|:---:|:---:| | | Approximate (HOTA points) | Evaluated (HOTA points) | Approximate (HOTA points) | Evaluated (HOTA points) | | TransRMOT | 4.126 | 4.139 | 5.694 | 6.716 | | TempRMOT | 5.703 | 6.050 | 7.493 | 8.577 | | DKGTrack | 6.613 | 6.838 | 9.059 | 10.265 | | iKUN | 6.727 | 7.318 | 9.263 | 10.584 | | FlexHook | 6.604 | 7.052 | 8.440 | 9.697 | As shown in the table, for TransRMOT, the approximation closely matches the evaluated isolated DIST gain (4.126 versus 4.139), but differs more after MISS repair (5.694 versus 6.716). Across the five methods, evaluated gains after MISS repair exceed the approximation by 1.022 to 1.321 HOTA points. These are approximation errors under fixed AssA and the specified count changes, not separately identified causal interaction components. We also compare error statistics with Shapley contributions across the five methods. DIST event counts, DIST incidence rates, and ASS incidence rates each preserve the cross-method attribution ordering for the corresponding category, with agreement on 10/10 method pairs in each comparison. HOTA sub-metrics and counts describe the current outputs. Evaluating repair combinations measures how each category's gain depends on other repairs. Shapley averages these marginal gains across repair orders and allocates the joint gap. The proportional baseline does not fully reproduce this allocation, and the tested closed-form approximation does not fully reproduce the reevaluated conditional gains.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their thoughtful comments and suggestions. We are encouraged by the reviewers' positive feedback on our motivation (Ny3E), systematic error analysis (Jyhp), and clear diagnostic definitions and reproducible attribution (YLoF).

Before addressing the individual comments, we would like to clarify the motivation and contribution of this work. RMOT requires a tracker to identify the objects described by a language query and preserve their identities over time. An output can follow a real object accurately yet still be incorrect for the query. Existing tracking metrics summarize performance through detection, association, and localization scores, but do not directly quantify how these query-dependent output failures contribute to the performance gap. Moreover, repairing one error can change the benefit of repairing another. This motivates distinguishing how often errors occur, what is gained by repairing each type alone, and how the jointly recoverable gap is allocated among error types.

RMOT-Compass provides a unified framework for answering these questions across RMOT architectures. It uses final predictions, language queries, and full-scene annotations as a common interface for end-to-end and two-stage methods. Query-defined target and distractor roles distinguish outputs on query distractors from background or duplicate outputs. Frozen diagnostic correspondences keep error labels consistent across repairs, while joint repairs coordinate target recovery with identity and geometry across frames. Standard Shapley allocation then accounts for repair interactions by averaging marginal gains over all repair orders. Our contribution lies in the RMOT-specific diagnostic formulation and repair protocol.

RMOT-Compass enables comparisons across architectures and quantifies how the gain from repairing one error changes when other errors are repaired. An exploratory training case also illustrates how a measured interaction motivates a concrete hypothesis about combining model changes. Our aim is to give the RMOT community a reproducible basis for comparing failure patterns and formulating more focused model investigations.

Updates to the Manuscript. The revised manuscript incorporates the following updates, with all changes highlighted in blue:

  1. Clarify the meaning and scope of oracle attribution (Secs. 3.5 and 5, Appendix C.1).
  2. Add comparisons with isolated-gain and count-based baselines (Appendix D.2.1).
  3. Add sensitivity analyses for repair and allocation choices (Sec. 4.2, Appendix C.3).
  4. Report diagnostic versus HOTA matching differences (Appendix C.4).
  5. Add localization severity and jitter-response analyses (Secs. 4.3 and 4.4, Appendices D.1 and E.2.1).
  6. Clarify aggregation choices and add video-composition sensitivity analyses (Secs. 4.1 and 4.3, Fig. 4, Appendix D.3.1).
  7. Provide an exploratory training design case (Secs. 4.1 and 4.3, Appendix D.4).
  8. Add controlled output injections and single-error calibration (Sec. 4.4, Appendix E.3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.