- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TempoTrack, a temporal distribution matching framework for open-vocabulary multi-object tracking (OV-MOT). To address identity ranking ambiguities, it represents historical matching responses as query-conditioned distributions characterized by statistical descriptors. It further introduces adaptive fusion of recent and long-term historical evidence and a structured association scorer to improve identity discrimination. The association module is trained using only Base-category identity supervision while keeping the visual frontend frozen. Experiments on TAO and BDD MOT demonstrate improvements in association performance, particularly for Novel categories. Additional experiments across different tracking frontends and ablation studies evaluate the effectiveness of the proposed components.
Critical strengths
-
Well-motivated investigation of identity ranking errors in open-vocabulary tracking. The paper provides a useful diagnostic analysis that distinguishes candidate availability from identity ranking failures. Figure 2 shows that, although the correct identity is frequently not ranked first, it remains within the top 64 candidates in most evaluated cases. This observation suggests that improving identity ranking can be an effective direction for addressing association errors without necessarily modifying the visual representation or candidate retrieval mechanism. The analysis provides meaningful motivation for investigating more discriminative historical association evidence.
-
Practical framework with relatively comprehensive experimental evaluation. TempoTrack introduces a lightweight association module that can be integrated with frozen visual frontends without retraining the underlying feature representations. The authors evaluate the method on multiple benchmarks and tracking frontends, demonstrating consistent improvements in association-related metrics. In addition, the ablation studies investigate the contributions of historical evidence representations, temporal fusion strategies, and scoring architectures. These results support the practical applicability of the proposed approach.
Critical weaknesses
-
Limited methodological novelty and an insufficiently established connection between the MGF formulation and the observed performance gains. The paper represents historical matching responses as empirical probability distributions and derives compact descriptors using their means, variances, and moment-generating functions (MGFs). However, the underlying mathematical formulations are based on established statistical properties, and the resulting MGF descriptor, (m_i^\rho=\mu_i^\rho+\frac{1}{2}v_i^\rho), is merely a deterministic combination of the mean and variance, without introducing additional independent statistical information. Furthermore, recent and long-term history fusion, adaptive gating, and residual scoring are relatively common design strategies, making the overall methodological novelty somewhat limited.
-
Table 7 shows that removing the two MGF-derived descriptors reduces Novel AssocA only from 43.3 to 43.0, whereas the full model outperforms a parameter-matched MLP by 1.3 percentage points. These results suggest that the improvements are more strongly associated with the structured association architecture than with the MGF descriptors themselves. The unique necessity and practical contribution of the MGF formulation therefore remain insufficiently demonstrated.
-
The Method section lacks a sufficiently coherent presentation of the relationships among its components. Although the paper provides detailed mathematical definitions and derivations, the overall presentation of Section 3 is not sufficiently streamlined, making the proposed framework more difficult to follow than necessary. In particular, Section 3.3 devotes considerable attention to empirical matching distributions, MGFs, and Gaussian surrogates, while Section 3.4 subsequently introduces a structured association score based on prototype similarity, historical means, and variance penalization. The connection between these formulations is not explained clearly enough, making it difficult to understand how the MGF-derived descriptors contribute to the final association decision.
Questions for authors
Q1. Do Tables 1 and 2 use the same detector and visual feature configurations? Table 2 reports substantially higher overall tracking performance than Table 1. Could the authors clarify whether the same detector and visual feature extractor are used in both experiments? If different configurations are adopted, please specify the differences. If the same detector and visual features are used, please explain the reasons for the large performance gap between the two tables, particularly the influence of different evaluation protocols and category settings.
Q2. Why are the improvements on Novel categories generally larger than those on Base categories, despite using only Base-category identity supervision? Several results in Table 1 and Table 3 indicate that TempoTrack produces larger association improvements for Novel categories than for Base categories. For example, Table 3 reports Novel AssocA improvements of 3.6, 3.1, and 3.1 percentage points when integrated with OVTrack, VOVTrack, and MASA, respectively, compared with Base AssocA improvements of 2.1, 2.3, and 2.0 percentage points.
Since the association scorer is trained exclusively with Base-category identity supervision, could the authors explain why the improvements are more pronounced for unseen categories in these settings? Is this phenomenon related to differences in baseline association difficulty, historical feature distributions, or candidate ranking ambiguity between Base and Novel categories? A more detailed analysis would help clarify the open-vocabulary generalization behavior of the proposed association mechanism.
10 comments
Response to Reviewer R53Q (Part 2)
(Continued from Part 1)
This formulation provides three practical advantages. First, all retained observations contribute to identity matching, reducing the influence of an unusual viewpoint. Second, the variance captures the consistency of historical support, distinguishing identities with similar means but different dispersion. Third, the representation is compact and category-independent: each temporal range requires only the projected mean and variance, without explicitly constructing a covariance matrix. We agree that the MGF-derived coordinate introduces no independent statistical information. Its individual contribution is 0.3 points in Novel AssocA (Table 7), while the main improvement comes from how the scorer jointly exploits the mean and variance.
-
3. Structured use of distributional evidence for identity association. Building on the distributional representation, TempoTrack develops a structured scoring mechanism that converts historical identity evidence into more discriminative association decisions. The structured score combines prototype agreement and mean historical support while accounting for response dispersion. The distinction lies in how these components are organized around the query-conditioned historical response distribution. Each temporal branch evaluates the average historical support and its consistency. The candidate-specific gate balances recent and long-term evidence according to the candidate's temporal state, while the residual scorer refines the resulting scores in the context of competing identities. Thus, these components jointly exploit the statistical structure of historical identity evidence rather than operating as independent add-on modules. Table 6 shows that adaptive fusion improves Novel AssocA from 41.9 to 43.3 compared with fixed fusion. Table 7 further shows that the structured scorer achieves 43.3, outperforming a capacity-matched MLP (42.0) by 1.3 points, despite having slightly fewer parameters.
We have also extended the frontend integration experiments by adding COVTrack + Ours to Table 3. The association improvements across all four frontends are summarized below.
Frontend Novel AssocA Gain Base AssocA Gain COVTrack +3.0 +2.1 OVTrack +3.6 +2.1 VOVTrack +3.1 +2.3 MASA +3.1 +2.0 TempoTrack consistently improves association performance across all four frontends, with larger gains on Novel categories. In particular, integrating TempoTrack into the complete COVTrack pipeline improves Novel AssocA from 41.3 to 44.3 and Base AssocA from 42.0 to 44.1. This further demonstrates that our distribution-based association mechanism is complementary to existing multi-cue tracking designs. Taken together, TempoTrack connects query-conditioned historical distributions, adaptive temporal integration, and candidate-relative scoring into a unified association framework. We have revised Sections 4.3 and 4.4 to make these relationships clearer.
W2. Connection between the MGF formulation and the performance gains.
"Table 7 shows that removing the two MGF-derived descriptors reduces Novel AssocA only from 43.3 to 43.0, whereas the full model outperforms a parameter-matched MLP by 1.3 percentage points. ... The unique necessity and practical contribution of the MGF formulation therefore remain insufficiently demonstrated."
Thank you for this insightful reading of Table 7. We agree that the improvement comes mainly from the structured architecture rather than from the two MGF coordinates. Below we clarify what Table 7 measures, show what drives the gain, and explain how we have revised the paper.
- 1. What Table 7 measures. The control without m removes only the two coordinates of the recent and long term branches. It keeps their constituent means and variances, so the second order evidence is still available to the scorer (Appendix A). The drop from 43.3 to 43.0 therefore measures the added value of m given μ and v, and we agree that this value is small. It does not measure the value of the distributional evidence itself, which Table 5 examines.
Response to Reviewer R53Q (Part 3)
(Continued from Part 2)
-
2. Distributional evidence matters beyond the mean. Table 5 compares representations of historical evidence under the same tracking configuration.
Historical evidence Novel AssocA Base AssocA Prototype matching 38.2 37.3 Mean history 39.7 38.5 Max history 40.5 39.1 Mean + variance 40.1 38.8 Single distribution 41.2 40.2 TempoTrack (full) 43.3 42.2 Adding variance to the mean alone does not guarantee an improvement, as Mean + Variance (40.1) does not outperform Max History (40.5). The more informative comparison is between Mean History (39.7) and Single Distribution (41.2). The latter evaluates the historical response distribution within the structured scorer and improves Novel AssocA by 1.5 points. This suggests that the benefit does not arise simply from appending a variance statistic to a mean-based representation. It emerges from how the historical response distribution is incorporated into the association scoring process. The full model further combines recent and long-term distributional evidence with adaptive fusion and candidate-relative refinement, reaching 43.3.
-
3. The gain over the MLP is stable. Both MLPs in Table 7 receive exactly the same 33 dimensional descriptor as TempoTrack, and the capacity matched MLP even has slightly more parameters. To check that the gap is not caused by training randomness or by a few videos, we train the capacity matched MLP with the same three seeds as TempoTrack in Appendix E. We also compare both models on the same videos with 2,000 paired video bootstrap resamples.
Seed TempoTrack Novel AssocA MLP Novel AssocA Difference 0 43.3 42.0 +1.3 1 43.3 41.8 +1.5 2 43.3 41.9 +1.4 Mean ± std 43.30 ± 0.00 41.90 ± 0.10 +1.40 ± 0.10 The gap is positive for every seed. On resampled videos, the difference is +1.3 [0.5, 2.8] for Novel AssocA and +1.0 [0.4, 2.1] for Base AssocA (95% CI), and both intervals exclude zero. The paired differences remain positive across all three seeds, with a mean gap of 1.40 ± 0.10 points. This indicates that the advantage of the structured scorer is consistent across the evaluated seeds.
-
4. What the structure adds. The MLP must learn from data how the moments relate to identity, while TempoTrack builds this relation into the score. Each temporal branch combines prototype agreement with mean support and subtracts a learned positive penalty on the response dispersion. The penalty is constrained to be positive, so a history whose responses vary widely receives less support unless other evidence compensates. The residual then interprets each candidate relative to its competitors. An MLP with the same inputs can represent these relations, but it has to discover them from Base identities alone. We believe this constrained form acts as a useful inductive bias, because the scorer is trained only on Base identities and must transfer to Novel categories.
-
5. Revised role of the MGF formulation. We have revised the paper. Section 4.3 now describes m as a compact parameterization of second-order statistics, and the contribution list attributes the gains to the structured use of distributional evidence. In short, the MGF formulation explains and efficiently computes the second order evidence, while the gain comes from how the structured scorer uses this evidence. We therefore position the MGF as the mathematical formulation that connects the empirical matching distribution to its projected moments and provides a compact second-order coordinate. Its role is to support the representation and computation of distributional evidence, while the empirical gains are primarily associated with the structured use of the projected mean and variance in the association scorer. We have revised the contribution statements accordingly and do not attribute the main performance improvement to the MGF coordinate alone.
Response to Reviewer R53Q (Part 1)
Dear Reviewer R53Q, We sincerely appreciate your thoughtful evaluation and constructive suggestions. We are encouraged that you recognized the value of our identity-ranking diagnosis and the practical potential of TempoTrack. Your comments have helped us strengthen both the methodological explanation and experimental evidence. In this revision, we have clarified the role of distributional evidence and the MGF formulation, streamlined Section 3, and added paired-seed, video-level, candidate-budget, and temporal-fusion analyses to further examine the effectiveness and robustness of the proposed scoring architecture. We have also clarified the experimental configurations and expanded the analysis of generalization to Novel categories. We hope these revisions address your concerns and make the contributions of TempoTrack clearer.
W1. Methodological novelty and the role of the MGF formulation.
We sincerely appreciate your thoughtful assessment. Your comments have encouraged us to reconsider how we present the methodological contribution of TempoTrack. In the revision, we clarify that our central contribution is the distributional representation and structured exploitation of historical identity evidence, rather than the introduction of new statistical properties or individual scoring components. We have also strengthened the experimental evidence to clarify how this formulation contributes to the overall association improvement.
1. Distributional representation of historical identity evidence. We represent each identity through a distribution of historical matching responses because a single visual feature is often insufficient to characterize an identity in Novel categories. Without task-specific identity supervision, frozen visual features may vary considerably across observations of the same object. An individual observation captures only one appearance state, and changes in pose, viewpoint, scale, and occlusion can make it an unreliable reference for subsequent association. Similarly, a single prototype compresses multiple historical observations into one representation, overlooking how differently these observations support the current query. We therefore collect the matching responses between the current query and all retained historical features of each candidate identity, forming a query-conditioned empirical distribution. Its mean measures the overall matching support, while its variance characterizes the consistency of this support across observations. This distinction is important because two identities may have similar mean similarities but substantially different distributions of historical support. Moreover, the distribution is conditioned on the current query and constructed from each identity's own history, making it applicable to Novel categories without category-specific identity supervision. Table 5 supports this design: Prototype Matching achieves 38.2 Novel AssocA, Mean History 39.7, Single Distribution 41.2, and the full model 43.3. These results demonstrate the benefit of exploiting the distributional characteristics of historical matching evidence for open-vocabulary identity association.
2. Statistical interpretation and role of the MGF. The MGF serves as a mathematical tool to express the matching response distribution in a unified functional form and naturally connect it with the mean and covariance of historical identity features. The mean vector describes the typical appearance of an identity, while the covariance matrix captures the extent and directions of its appearance variations. Since each matching response dik=xt⊤hik is a linear projection of a historical feature onto the current query, averaging these responses gives the projected mean, μiρ=xt⊤hˉiρ, while their variance gives the projected covariance, viρ=xt⊤Σiρxt. These statistics are naturally derived from the matching process rather than manually selected. The first and second derivatives of the log-MGF at zero yield the mean and variance, respectively. Although higher-order statistics could provide a more detailed characterization, the recent branch contains at most eight observations, making their estimation potentially noisy. We therefore focus on the first two moments. Using a moment-matched Gaussian surrogate, we evaluate its log-MGF at β=1 and obtain the compact second-order descriptor:miρ=logMiρ(1∣xt)=μiρ+21viρ.