Q1. Query conditioning fails to separate RMOT's two essential failure modes.Query conditioning only partitions false positives into DIST and BKG-DUP—both are FPs under HOTA with identical deletion effects—while the distinction that matters (never detected vs. tracked but rejected by the referring module) is collapsed into MISS, the largest attribution (24.8–31.3). The framework therefore cannot say whether to fix detection-tracking or referring, contrary to its stated goal. For iKUN/FlexHook, MISS could be split into "absent from candidates" and "present but unselected." Explain why this was not done and what query conditioning adds over generic MOT error decomposition.
Q2. The findings follow from HOTA's algebra.DetA = T/(T+F+P). Deleting FPs alone gains T·P/((T+F)(T+F+P)); repairing misses first, then deleting FPs, gains P/(T+F+P); the ratio is (T+F)/T = 1/(1−miss rate) > 1. TransRMOT's 0.4285 miss rate predicts 1.75 vs. the reported ≈1.62, the rest attributable to AssA and threshold averaging. Likewise, ASS–LOC is near-tautological (LOC repair raises TPs at high thresholds), the DIST/BKG-DUP attribution ratio roughly tracks event counts (1.33 vs. 1.51; 4.73 vs. 4.06), and MISS dominance follows from 0.35–0.43 miss rates and low DetRe. Show what Shapley attribution adds beyond HOTA's sub-metrics and counts—e.g., against a closed-form or regression baseline.
Q3. Full-recovery results are tautological; validation is internal only.Repairs use GT boxes and IDs, so full repair yields HOTA 100 and the jointly recoverable gap ≡ 100−HOTA; Table 3/App. C.1 (zero residual, 1e-9 efficiency error, non-negative marginals) only confirm implementation and the efficiency identity. No external validity is tested: do real detector/association/referring improvements yield HOTA gains matching the corresponding attribution, and does attribution ranking guide improvements better than standalone gains or incidence? Otherwise "error budget," "recoverable," and the Section 5 claims are unsupported.
Q4. Attributions hinge on arbitrary game design choices.As Sundararajan & Najmi note, Shapley values depend on how the game is operationalized. Three choices here are consequential: (i) MISS inserts a score-1 exact GT box, charging unmatched targets' localization loss to MISS while matched-but-poorly-localized targets charge LOC—an asymmetry that may inflate MISS; (ii) the ID rule (safe original vs. canonical IDs) determines MISS's gain and the MISS–ASS interaction; uniformly new IDs would change all results; (iii) granularity is highly uneven (37,982 LOC vs. 11,495 BKG-DUP events) and Shapley is not split-invariant. Report stability under equally reasonable alternatives, and justify symmetry/additivity over Banzhaf or Owen values for this diagnostic use.
Q5. Diagnostic labels do not match HOTA's matching.Diagnostics match at IoU 0.05 with a max-cardinality lexicographic objective; HOTA matches separately at every 0.05–0.95 threshold using association scores. A 0.06-IoU prediction can be labeled LOC (and ASS) yet is FP+FN at most HOTA thresholds, so labels need not correspond to metric penalties. The 0.05 choice is unjustified; in Table 4 only MISS's rank survives, while iKUN's LOC drops 14.493→11.922 (≈18%) and at 0.30 TempRMOT's BKG-DUP exceeds ASS. State the threshold rationale and report the diagnostic/metric mismatch rate per threshold.
Q6. The LOC definition is degenerate, making the incidence argument circular.With LOC defined as IoU < 1−1e-6, incidence is 1.000 for all methods (Table 8) and stays saturated under perturbations (Table 12). The claim that "identical incidence, attributions 5.88–14.49" shows incidence is uninformative thus rests on this degenerate definition. Re-test under the non-degenerate IoU < 0.5 severe-error label already defined in the appendix.
Q7. Uncertainty is absent and the bootstrap basis is weak.Refer-KITTI has only 3 test videos (0005, 0011, 0013), so video-level block bootstrap gives 10 resamples with no coverage guarantee; V2 has 4 videos. All global attributions (Fig. 3; Tables 4, 7, 9, 10) lack uncertainty despite tiny key gaps (ASS 6.12 vs. LOC 5.88; standalone 3.47 vs. 3.54), and query-macro intervals are very wide (BKG-DUP 1.054–13.968). Provide uncertainty estimates and justify the bootstrap with three clusters.
Q8. Global and query-macro estimates disagree; the target estimand is undefined.iKUN's MISS is 25.154 globally (second smallest) but 52.764 under query-macro (largest); LOC goes 14.493→9.015. TransRMOT's ASS exceeds LOC globally (6.120 vs. 5.884) but not under query-macro (3.879 vs. 5.717), erasing the Section 4.3/Fig. 4 reversal. Since the diagnostic answer depends on the estimator, state which estimand is intended and why, and whether the 4.2/4.3 conclusions hold under the other.
Q9. Perturbations neither validate attribution nor inject controlled error types.Appendix E.1 confirms post-perturbation 32-coalition attributions are never computed, so the core output is untested against known interventions; perturbations precede FlexHook, conflating framework validity with model response. Only target removal behaves as expected and significantly; ID fragmentation crosses zero, distractor insertion shifts DIST oppositely, and LOC stays saturated. High-dose distractor insertion adds 20,429 predictions but only 4,696 DIST events, leaving ~15.7k unexplained (possible duplicates or other queries' targets). Inject known-count/type errors into real outputs and test whether attribution recovers them specifically.
Response to Reviewer Ny3E
We sincerely thank you for your thorough review and valuable insights. We address your comments below.
RMOT-Compass builds on TIDE’s output-diagnosis paradigm. Our methodological contribution lies in defining errors and composable repairs for query-dependent target selection and temporal identity in RMOT and in allocating the jointly recoverable score gap beyond reporting isolated repair gains.
Our error taxonomy addresses query constraints and temporal identity in RMOT. Two key distinctions from TIDE’s taxonomy are query-distractor errors (DIST) and identity-association errors (ASS). DIST describes a real scene object that falls outside the current query’s target set. Even if its category and localization are correct, outputting it remains an error, and DIST repair removes that output. ASS captures temporal identity errors, which TIDE’s static detection taxonomy does not define. When restoring missing targets, we coordinate the identities of inserted detections with ASS repair while keeping evaluation ground truth fixed and diagnostic correspondences frozen across repair combinations (Sections 3.2, 3.3, and 3.4 of the manuscript).
TIDE recognizes non-additivity and reports isolated repair gains. We retain these gains and additionally evaluate all 32 repair combinations, using the established Shapley rule to average marginal contributions across repair contexts (Section 3.5 of the manuscript). The resulting allocations sum to the jointly recoverable gap. For TransRMOT, the manuscript reports 40.645 HOTA points for the sum of isolated gains and 54.524 points for the joint gap under the same repair protocol (Figure 1 and Table 9 of the manuscript). The 13.879-point difference illustrates why isolated gains alone do not provide a complete allocation of joint recovery.
Our error taxonomy quantifies output failure modes across RMOT methods, while repair analysis measures their impact on evaluation metrics to inform model inspection and design improvement. MISS can originate from multiple internal stages. One can inspect whether candidates cover the target and where the corresponding candidates are rejected or lost during filtering or tracking. These checks narrow the investigation.
Shapley attribution allocates the score gap recoverable under the specified oracle repairs. A large allocation highlights a failure worth investigating, but practical optimization priorities also depend on feasibility, implementation cost, and effects on other errors. We use attribution and repair interactions to formulate design hypotheses, then evaluate actual interventions, as illustrated in Q1.
We use TempRMOT as an example. DIST repair yields 6.050 HOTA points in isolation and 8.577 after MISS repair. Together with the positive Shapley interaction between MISS and DIST, this motivates jointly testing foreground supervision and query-distractor suppression.
We conduct an exploratory four-condition experiment. C retains the original losses. A adds positive supervision for matched low-confidence foreground candidates. B adds negative referring supervision for matched non-target objects with high referring scores. AB combines both terms. For all conditions, we start from the same pretrained checkpoint, train for ten epochs, and evaluate the final checkpoint on the Refer-KITTI test set using identical inference thresholds. The experimental results are shown below.
As shown in the table, adding B alone decreases HOTA by 0.432 points, whereas adding B when A is present increases HOTA by 0.658 points. AB yields a net HOTA gain of 0.140 points over C. This case illustrates how joint diagnosis identifies design directions worth investigating together. In this experiment, judging modifications solely by their individual effects misses a favorable combination, while joint diagnosis provides a reason to test it.