acceptodds
Under review as a conference paper at ICLR 2027

TempoTrack: Temporal Distribution Matching for Open-Vocabulary Multi-Object Tracking

Abstract

Open-vocabulary multi-object tracking requires identity association for categories without task-specific identity supervision. Existing tracking methods retain appearance histories, yet prototypes or pooled similarities compress this history and can obscure how past observations support the current query. We propose TempoTrack, which represents each query–identity pair through the empirical distribution of its historical matching responses. Recent and long-term distributional evidence is combined by a candidate-specific temporal gate and candidate-relative scorer, while a moment-generating formulation provides a compact second-order parameterization without explicit covariance construction. TempoTrack introduces a lightweight, frame-causal association layer on top of frozen visual frontends and is trained using only Base identity supervision. On TAO, TempoTrack reaches 43.3 Novel AssocA on validation and 33.6 on test. These results support query-conditioned temporal distributions as a lightweight and effective representation of historical identity evidence for open-vocabulary association.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-Object Tracking

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_R53QOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TempoTrack, a temporal distribution matching framework for open-vocabulary multi-object tracking (OV-MOT). To address identity ranking ambiguities, it represents historical matching responses as query-conditioned distributions characterized by statistical descriptors. It further introduces adaptive fusion of recent and long-term historical evidence and a structured association scorer to improve identity discrimination. The association module is trained using only Base-category identity supervision while keeping the visual frontend frozen. Experiments on TAO and BDD MOT demonstrate improvements in association performance, particularly for Novel categories. Additional experiments across different tracking frontends and ablation studies evaluate the effectiveness of the proposed components.

Critical strengths
  1. Well-motivated investigation of identity ranking errors in open-vocabulary tracking. The paper provides a useful diagnostic analysis that distinguishes candidate availability from identity ranking failures. Figure 2 shows that, although the correct identity is frequently not ranked first, it remains within the top 64 candidates in most evaluated cases. This observation suggests that improving identity ranking can be an effective direction for addressing association errors without necessarily modifying the visual representation or candidate retrieval mechanism. The analysis provides meaningful motivation for investigating more discriminative historical association evidence.

  2. Practical framework with relatively comprehensive experimental evaluation. TempoTrack introduces a lightweight association module that can be integrated with frozen visual frontends without retraining the underlying feature representations. The authors evaluate the method on multiple benchmarks and tracking frontends, demonstrating consistent improvements in association-related metrics. In addition, the ablation studies investigate the contributions of historical evidence representations, temporal fusion strategies, and scoring architectures. These results support the practical applicability of the proposed approach.

Critical weaknesses
  1. Limited methodological novelty and an insufficiently established connection between the MGF formulation and the observed performance gains. The paper represents historical matching responses as empirical probability distributions and derives compact descriptors using their means, variances, and moment-generating functions (MGFs). However, the underlying mathematical formulations are based on established statistical properties, and the resulting MGF descriptor, (m_i^\rho=\mu_i^\rho+\frac{1}{2}v_i^\rho), is merely a deterministic combination of the mean and variance, without introducing additional independent statistical information. Furthermore, recent and long-term history fusion, adaptive gating, and residual scoring are relatively common design strategies, making the overall methodological novelty somewhat limited.

  2. Table 7 shows that removing the two MGF-derived descriptors reduces Novel AssocA only from 43.3 to 43.0, whereas the full model outperforms a parameter-matched MLP by 1.3 percentage points. These results suggest that the improvements are more strongly associated with the structured association architecture than with the MGF descriptors themselves. The unique necessity and practical contribution of the MGF formulation therefore remain insufficiently demonstrated.

  3. The Method section lacks a sufficiently coherent presentation of the relationships among its components. Although the paper provides detailed mathematical definitions and derivations, the overall presentation of Section 3 is not sufficiently streamlined, making the proposed framework more difficult to follow than necessary. In particular, Section 3.3 devotes considerable attention to empirical matching distributions, MGFs, and Gaussian surrogates, while Section 3.4 subsequently introduces a structured association score based on prototype similarity, historical means, and variance penalization. The connection between these formulations is not explained clearly enough, making it difficult to understand how the MGF-derived descriptors contribute to the final association decision.

Questions for authors

Q1. Do Tables 1 and 2 use the same detector and visual feature configurations? Table 2 reports substantially higher overall tracking performance than Table 1. Could the authors clarify whether the same detector and visual feature extractor are used in both experiments? If different configurations are adopted, please specify the differences. If the same detector and visual features are used, please explain the reasons for the large performance gap between the two tables, particularly the influence of different evaluation protocols and category settings.

Q2. Why are the improvements on Novel categories generally larger than those on Base categories, despite using only Base-category identity supervision? Several results in Table 1 and Table 3 indicate that TempoTrack produces larger association improvements for Novel categories than for Base categories. For example, Table 3 reports Novel AssocA improvements of 3.6, 3.1, and 3.1 percentage points when integrated with OVTrack, VOVTrack, and MASA, respectively, compared with Base AssocA improvements of 2.1, 2.3, and 2.0 percentage points.

Since the association scorer is trained exclusively with Base-category identity supervision, could the authors explain why the improvements are more pronounced for unseen categories in these settings? Is this phenomenon related to differences in baseline association difficulty, historical feature distributions, or candidate ranking ambiguity between Base and Novel categories? A more detailed analysis would help clarify the open-vocabulary generalization behavior of the proposed association mechanism.

10 comments
AuthorsOpenReview ↗

Response to Reviewer R53Q (Part 1)

Dear Reviewer R53Q,  We sincerely appreciate your thoughtful evaluation and constructive suggestions. We are encouraged that you recognized the value of our identity-ranking diagnosis and the practical potential of TempoTrack. Your comments have helped us strengthen both the methodological explanation and experimental evidence. In this revision, we have clarified the role of distributional evidence and the MGF formulation, streamlined Section 3, and added paired-seed, video-level, candidate-budget, and temporal-fusion analyses to further examine the effectiveness and robustness of the proposed scoring architecture. We have also clarified the experimental configurations and expanded the analysis of generalization to Novel categories. We hope these revisions address your concerns and make the contributions of TempoTrack clearer.


W1. Methodological novelty and the role of the MGF formulation.

"The underlying mathematical formulations are based on established statistical properties, and the resulting MGF descriptor ... . Furthermore, recent and long-term history fusion, adaptive gating, and residual scoring are relatively common design strategies, making the overall methodological novelty somewhat limited."

We sincerely appreciate your thoughtful assessment. Your comments have encouraged us to reconsider how we present the methodological contribution of TempoTrack. In the revision, we clarify that our central contribution is the distributional representation and structured exploitation of historical identity evidence, rather than the introduction of new statistical properties or individual scoring components. We have also strengthened the experimental evidence to clarify how this formulation contributes to the overall association improvement.

  • 1. Distributional representation of historical identity evidence. We represent each identity through a distribution of historical matching responses because a single visual feature is often insufficient to characterize an identity in Novel categories. Without task-specific identity supervision, frozen visual features may vary considerably across observations of the same object. An individual observation captures only one appearance state, and changes in pose, viewpoint, scale, and occlusion can make it an unreliable reference for subsequent association. Similarly, a single prototype compresses multiple historical observations into one representation, overlooking how differently these observations support the current query. We therefore collect the matching responses between the current query and all retained historical features of each candidate identity, forming a query-conditioned empirical distribution. Its mean measures the overall matching support, while its variance characterizes the consistency of this support across observations. This distinction is important because two identities may have similar mean similarities but substantially different distributions of historical support. Moreover, the distribution is conditioned on the current query and constructed from each identity's own history, making it applicable to Novel categories without category-specific identity supervision. Table 5 supports this design: Prototype Matching achieves 38.2 Novel AssocA, Mean History 39.7, Single Distribution 41.2, and the full model 43.3. These results demonstrate the benefit of exploiting the distributional characteristics of historical matching evidence for open-vocabulary identity association.

  • 2. Statistical interpretation and role of the MGF. The MGF serves as a mathematical tool to express the matching response distribution in a unified functional form and naturally connect it with the mean and covariance of historical identity features. The mean vector describes the typical appearance of an identity, while the covariance matrix captures the extent and directions of its appearance variations. Since each matching response dik=xt⊤hikd_i^k=x_t^\top h_i^k is a linear projection of a historical feature onto the current query, averaging these responses gives the projected mean, μiρ=xt⊤hˉiρ\mu_i^\rho=x_t^\top\bar h_i^\rho, while their variance gives the projected covariance, viρ=xt⊤Σiρxtv_i^\rho=x_t^\top\Sigma_i^\rho x_t. These statistics are naturally derived from the matching process rather than manually selected. The first and second derivatives of the log-MGF at zero yield the mean and variance, respectively. Although higher-order statistics could provide a more detailed characterization, the recent branch contains at most eight observations, making their estimation potentially noisy. We therefore focus on the first two moments. Using a moment-matched Gaussian surrogate, we evaluate its log-MGF at β=1\beta=1 and obtain the compact second-order descriptor:miρ=log⁡M~iρ(1∣xt)=μiρ+12viρ.m_i^\rho=\log\widetilde{M}_i^\rho(1\mid x_t) =\mu_i^\rho+\frac{1}{2}v_i^\rho.

AuthorsOpenReview ↗

Response to Reviewer R53Q (Part 2)

(Continued from Part 1)

This formulation provides three practical advantages. First, all retained observations contribute to identity matching, reducing the influence of an unusual viewpoint. Second, the variance captures the consistency of historical support, distinguishing identities with similar means but different dispersion. Third, the representation is compact and category-independent: each temporal range requires only the projected mean and variance, without explicitly constructing a covariance matrix. We agree that the MGF-derived coordinate miρm_i^\rho introduces no independent statistical information. Its individual contribution is 0.3 points in Novel AssocA (Table 7), while the main improvement comes from how the scorer jointly exploits the mean and variance.

  • 3. Structured use of distributional evidence for identity association. Building on the distributional representation, TempoTrack develops a structured scoring mechanism that converts historical identity evidence into more discriminative association decisions. The structured score combines prototype agreement and mean historical support while accounting for response dispersion. The distinction lies in how these components are organized around the query-conditioned historical response distribution. Each temporal branch evaluates the average historical support and its consistency. The candidate-specific gate balances recent and long-term evidence according to the candidate's temporal state, while the residual scorer refines the resulting scores in the context of competing identities. Thus, these components jointly exploit the statistical structure of historical identity evidence rather than operating as independent add-on modules. Table 6 shows that adaptive fusion improves Novel AssocA from 41.9 to 43.3 compared with fixed fusion. Table 7 further shows that the structured scorer achieves 43.3, outperforming a capacity-matched MLP (42.0) by 1.3 points, despite having slightly fewer parameters.

    We have also extended the frontend integration experiments by adding COVTrack + Ours to Table 3. The association improvements across all four frontends are summarized below.

    FrontendNovel AssocA GainBase AssocA Gain
    COVTrack+3.0+2.1
    OVTrack+3.6+2.1
    VOVTrack+3.1+2.3
    MASA+3.1+2.0

    TempoTrack consistently improves association performance across all four frontends, with larger gains on Novel categories. In particular, integrating TempoTrack into the complete COVTrack pipeline improves Novel AssocA from 41.3 to 44.3 and Base AssocA from 42.0 to 44.1. This further demonstrates that our distribution-based association mechanism is complementary to existing multi-cue tracking designs. Taken together, TempoTrack connects query-conditioned historical distributions, adaptive temporal integration, and candidate-relative scoring into a unified association framework. We have revised Sections 4.3 and 4.4 to make these relationships clearer.


W2. Connection between the MGF formulation and the performance gains.

"Table 7 shows that removing the two MGF-derived descriptors reduces Novel AssocA only from 43.3 to 43.0, whereas the full model outperforms a parameter-matched MLP by 1.3 percentage points. ... The unique necessity and practical contribution of the MGF formulation therefore remain insufficiently demonstrated."

Thank you for this insightful reading of Table 7. We agree that the improvement comes mainly from the structured architecture rather than from the two MGF coordinates. Below we clarify what Table 7 measures, show what drives the gain, and explain how we have revised the paper.

  • 1. What Table 7 measures. The control without m removes only the two coordinates of the recent and long term branches. It keeps their constituent means and variances, so the second order evidence is still available to the scorer (Appendix A). The drop from 43.3 to 43.0 therefore measures the added value of m given μ and v, and we agree that this value is small. It does not measure the value of the distributional evidence itself, which Table 5 examines.
AuthorsOpenReview ↗

Response to Reviewer R53Q (Part 3)

(Continued from Part 2)

  • 2. Distributional evidence matters beyond the mean. Table 5 compares representations of historical evidence under the same tracking configuration.

    Historical evidenceNovel AssocABase AssocA
    Prototype matching38.237.3
    Mean history39.738.5
    Max history40.539.1
    Mean + variance40.138.8
    Single distribution41.240.2
    TempoTrack (full)43.342.2

    Adding variance to the mean alone does not guarantee an improvement, as Mean + Variance (40.1) does not outperform Max History (40.5). The more informative comparison is between Mean History (39.7) and Single Distribution (41.2). The latter evaluates the historical response distribution within the structured scorer and improves Novel AssocA by 1.5 points. This suggests that the benefit does not arise simply from appending a variance statistic to a mean-based representation. It emerges from how the historical response distribution is incorporated into the association scoring process. The full model further combines recent and long-term distributional evidence with adaptive fusion and candidate-relative refinement, reaching 43.3.

  • 3. The gain over the MLP is stable. Both MLPs in Table 7 receive exactly the same 33 dimensional descriptor as TempoTrack, and the capacity matched MLP even has slightly more parameters. To check that the gap is not caused by training randomness or by a few videos, we train the capacity matched MLP with the same three seeds as TempoTrack in Appendix E. We also compare both models on the same videos with 2,000 paired video bootstrap resamples.

    SeedTempoTrack Novel AssocAMLP Novel AssocADifference
    043.342.0+1.3
    143.341.8+1.5
    243.341.9+1.4
    Mean ± std43.30 ± 0.0041.90 ± 0.10+1.40 ± 0.10

    The gap is positive for every seed. On resampled videos, the difference is +1.3 [0.5, 2.8] for Novel AssocA and +1.0 [0.4, 2.1] for Base AssocA (95% CI), and both intervals exclude zero. The paired differences remain positive across all three seeds, with a mean gap of 1.40 ± 0.10 points. This indicates that the advantage of the structured scorer is consistent across the evaluated seeds.

  • 4. What the structure adds. The MLP must learn from data how the moments relate to identity, while TempoTrack builds this relation into the score. Each temporal branch combines prototype agreement with mean support and subtracts a learned positive penalty on the response dispersion. The penalty is constrained to be positive, so a history whose responses vary widely receives less support unless other evidence compensates. The residual then interprets each candidate relative to its competitors. An MLP with the same inputs can represent these relations, but it has to discover them from Base identities alone. We believe this constrained form acts as a useful inductive bias, because the scorer is trained only on Base identities and must transfer to Novel categories.

  • 5. Revised role of the MGF formulation. We have revised the paper. Section 4.3 now describes m as a compact parameterization of second-order statistics, and the contribution list attributes the gains to the structured use of distributional evidence. In short, the MGF formulation explains and efficiently computes the second order evidence, while the gain comes from how the structured scorer uses this evidence. We therefore position the MGF as the mathematical formulation that connects the empirical matching distribution to its projected moments and provides a compact second-order coordinate. Its role is to support the representation and computation of distributional evidence, while the empirical gains are primarily associated with the structured use of the projected mean and variance in the association scorer. We have revised the contribution statements accordingly and do not attribute the main performance improvement to the MGF coordinate alone.


Reviewer_45yCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TempoTrack learns a small association scorer over frozen visual frontends. For each observation and candidate identity, it matches the current features against stored historical features and summarizes recent and longer-term support. It combines these summaries with prototype agreement and candidate-relative information, then uses a temporal gate and residual network to produce the score. Training uses Base identity supervision. At inference, histories are updated only after the frame’s assignments have been committed. On TAO, the reported Novel AssocA is 43.3 on validation and 33.6 on test, compared with 41.3 and 32.6 for COVTrack. The evaluation also includes matched frontend integrations, architectural controls, and runtime measurements.

Critical strengths
  1. The comparisons with learned scorers using the same inputs are convincing. In Table 7, the full model reaches 43.3 Novel AssocA against 42.0 for a capacity-matched MLP, with slightly fewer parameters. Table 6 compares adaptive and fixed temporal fusion at 43.3 and 41.9. These controls distinguish the architecture’s contribution from the benefit of training an MLP on a better historical descriptor. The paper also reports the limited contribution of the two deterministic MGF coordinates: removing them reduces AssocA only to 43.0.
  2. Appendix A documents safeguards in training and selection. Optimization and internal holdout partitions are video-disjoint. Normalization, checkpoint selection, and validation selection of inference thresholds use Base data only. The matched integrations improve Novel AssocA for OVTrack, VOVTrack, and MASA, and the main configuration retains a gain on TAO test. Appendix E reports three training seeds and paired end-to-end throughput of 6.5 versus 6.2 FPS for OVTrack. The evidence therefore extends beyond an improvement on a single validation configuration.
Critical weaknesses
  1. Candidate coverage under the deployed prefilter is not quantified directly. Section 2 reports roughly 96–98% Novel Recall@64 under the original frontend scores. Inference ranks identities by similarity to their last actual observation, retaining 64 for context and eight for assignment. Section 2.2 and Appendix I distinguish these sets and explain that identities outside the eligible set cannot be recovered. Appendix G provides a matched-candidate analysis in which Base holdout Rank-1 accuracy improves from 83.73% to 87.70%. As stated in the appendix, that holdout participates in checkpoint selection, so this result does not independently measure Novel-category generalization. Reporting how often the correct identity survives the actual top-eight prefilter would connect the candidate analysis to inference. The reported evaluations support the end-to-end gains, and this analysis would clarify how candidate coverage affects them.
  2. The contribution rests mainly on the empirical scoring architecture. Section 3.3 defines m = μ + v/2 as a deterministic coordinate, and Section 3.5 gives an information bound for normalized identity scores. These results offer limited additional theoretical insight. Table 7 provides stronger support for the architecture: it gains 1.3 points over the capacity-matched MLP, while removing the MGF coordinates leaves 43.0 Novel AssocA. The three-seed study measures variation in the final model. It does not show whether the paired gap between architectures is stable. Appendix I discusses the potential cost of dispersion penalties and discarded temporal order. Analyses of gate behavior, candidate-budget sensitivity, or paired variation would help explain when the architecture helps and would support a stronger recommendation.
Questions for authors
  1. Section 2.2 and Appendix I distinguish diagnostic Recall@64 from assignment eligibility. If saved traces permit, what are the Base and Novel candidate recalls under the actual last-observation prefilter with a budget of eight? This would quantify the coverage of the eligible set.
  2. Please give the margin thresholds selected for each frontend, the validation search grid, and the rule for resolving conflicting proposals. Appendix A gives the selection criterion and rules out a second-choice fallback, but these final settings remain unspecified.
  3. Does the 43.3 versus 42.0 difference in Table 7 persist across paired seeds or a paired video-level analysis? Appendix E tests variation in the final model, rather than variation in this difference. Existing per-video outputs could support that comparison.
  4. If results are available, how do the gate weights and the gain over fixed fusion vary with temporal gap and historical response dispersion? This would help explain what the scorer learns. A new subgroup study can be left to the final version.
8 comments
AuthorsOpenReview ↗

Response to Reviewer 45yC (Part 1)

Dear Reviewer 45yC,

Thank you very much for your careful and constructive review. We sincerely appreciate your recognition of the matched scorer controls, the safeguards in training and selection, and the evaluations across frontends. Your comments point precisely to the analyses that connect our diagnosis to inference. We address each point below and have revised the paper accordingly.


W1. Candidate coverage under the deployed prefilter.

"Candidate coverage under the deployed prefilter is not quantified directly. ... Reporting how often the correct identity survives the actual top-eight prefilter would connect the candidate analysis to inference."

Thank you for highlighting this important distinction between broad candidate availability and actual assignment eligibility. We agree that the high Recall@64 reported under the original frontend scores does not directly establish candidate coverage in the deployed inference pipeline, where identities are ranked using their last actual observations and only the top eight are eligible for assignment. To address this gap, we have directly quantified the survival rate of correct identities under the actual prefilter and further examined how the remaining errors are distributed between candidate retrieval and identity ranking.

  • 1. Coverage of the deployed prefilter. We use the same diagnostic events as Figure 2 (COVTrack, 4,648 Base and 457 Novel events). Candidates are ranked exactly as at inference, by similarity to their last actual observation.

    SplitR@1R@2R@4R@8R@16R@32R@64
    Base66.5078.6887.3592.2895.3397.2998.99
    Novel61.4976.1582.2889.7294.3196.7298.91

    The correct identity survives the top eight prefilter in 92.28% of Base events and 89.72% of Novel events. The gap between Novel and Base is 5.01 points at R@1 and only 2.56 points at R@8. The Base--Novel coverage gap narrows from 5.01 percentage points at R@1 to 2.56 points at R@8, indicating that the deployed budget retains most correct Novel identities and substantially reduces the initial coverage disparity.

  • 2. The diagnosis holds under the deployed budget. We group each event by the rank of the correct identity under the same prefilter.

    SplitRank 1Ranks 2 to 8 (ranking error)Outside top 8 (retrieval failure)
    Base66.50%25.77%7.72%
    Novel61.49%28.23%10.28%

    Across all diagnostic events, ranking errors are approximately 3.3 times as frequent as retrieval failures for Base and 2.7 times for Novel. Events outside the top eight cannot be recovered by any scorer. These events cannot be recovered by the association scorer under the deployed candidate budget, thereby limiting the proportion of errors that can be corrected through candidate re-ranking, as discussed in Appendix D and Appendix I.

  • 3. Matched-candidate evaluation on the frozen diagnostic set. We agree that the Base holdout analysis in Appendix G does not measure Novel generalization by itself. We therefore repeat the matched candidate comparison on validation events whose correct identity lies in the top eight. Both methods rank the same eligible candidate set, containing up to eight identities. Novel events are never used for training or selection. Confidence intervals come from 2,000 video bootstrap resamples.

    SplitEventsLast observation Top-1 (%)TempoTrack Top-1 (%)Difference (pp) [95% CI]
    Base4,28972.0774.54+2.47 [1.50, 3.51]
    Novel41068.5470.98+2.44 [−0.25, 5.33]

    With identical candidates, TempoTrack improves Top-1 by 2.47 points for Base and by 2.44 points for Novel. The two gains are almost identical. The Novel interval is wider because it contains only 410 events, and it narrowly includes zero. The matching point estimates are consistent with the ranking improvement extending to categories that are never used for training or selection.


AuthorsOpenReview ↗

Response to Reviewer 45yC (Part 2)

(Continued from Part 1)

W2. Theoretical contribution and evidence for when the architecture helps.

"The contribution rests mainly on the empirical scoring architecture. ... Analyses of gate behavior, candidate-budget sensitivity, or paired variation would help explain when the architecture helps and would support a stronger recommendation."

We sincerely appreciate your thoughtful comments, which have helped us clarify the contribution of TempoTrack and strengthen its empirical validation. We agree that its main contribution lies in the structured scoring architecture for exploiting historical distributional evidence in identity association. The MGF descriptor and information-theoretic analysis provide supporting formulations, while the primary evidence comes from the architecture's empirical performance. As shown in Table 7, our scorer outperforms a capacity-matched MLP by 1.3 points, while removing the MGF coordinates reduces Novel AssocA by only 0.3 points. We further provide complementary analyses of candidate-budget sensitivity, paired performance stability, and adaptive temporal fusion to examine the robustness of these gains and the conditions under which the architecture is most beneficial.

  • 1. Role of the theoretical components. The MGF formulation is mainly a computational device. It shows that the mean and variance of scalar responses correspond to the projected mean and covariance of the retained features. The scorer can therefore use second order feature statistics without building any covariance matrix. At the deployed history length of 64, computing these statistics and updating the history take about 10 ms per frame, and the history features of all identities occupy only 4.0 MiB. As you note, the deterministic coordinate m adds only 0.3 Novel AssocA in Table 7. The information bound explains how the listwise objective relates to identity information. It is an interpretation of the training objective rather than a performance guarantee, and the architecture itself is evaluated by the matched controls. Regarding the potential cost of the dispersion penalty discussed in Appendix I, the subgroup analysis in Q4 shows that decisions with high response dispersion keep a low error of about 7%, so the penalty does not harm identities whose appearance varies validly.

  • 2. Candidate budget sensitivity. We vary the decision budget while keeping the context of 64 candidates fixed.

    Decision budgetNovel AssocABase AssocAFPS
    443.0341.626.29
    8 (ours)43.3042.206.20
    1643.3241.696.19
    3243.3341.926.18
    6443.3341.916.00

    TempoTrack is stable once the budget reaches eight. Larger budgets change Novel AssocA by at most 0.03 points, and Base AssocA is highest at eight. Together with W1, this shows that the scorer works on a compact eligible set and does not depend on a large candidate pool.

  • 3. Candidate-budget sensitivity, paired variation, and gate behavior. We address these three aspects in detail in our following responses. In Q2, we report the candidate-budget sensitivity analysis to justify the choice of eight assignment candidates. In Q3, we provide a paired comparison between TempoTrack and the capacity-matched MLP across three random seeds, together with a paired video-level bootstrap analysis to examine the consistency of their performance gap. In Q4, we present a subgroup analysis of adaptive temporal fusion across different temporal-gap and historical-response-dispersion groups, showing when the learned gate provides improvements over fixed fusion.


Q1. Candidate recall under the deployed prefilter with a budget of eight.

"If saved traces permit, what are the Base and Novel candidate recalls under the actual last-observation prefilter with a budget of eight?"

Thank you for raising this important question. We directly evaluate candidate recall using the actual last-observation prefilter. For each query, candidate identities are ranked by cosine similarity to their last actual observations. The top 64 are retained for contextual evidence, while only the top 8 are eligible for assignment. The complete Recall@K results are reported below.

SplitR@1R@2R@4R@8R@16R@32R@64
Base66.5078.6887.3592.2895.3397.2998.99
Novel61.4976.1582.2889.7294.3196.7298.91

The results show that the correct identity survives the deployed top-eight prefilter in 92.28% of Base events and 89.72% of Novel events. The Base--Novel coverage gap narrows from 5.01 percentage points at R@1 to 2.56 points at R@8.

AuthorsOpenReview ↗

Response to Reviewer 45yC (Part 3)

(Continued from Part 2)

We select eight candidates as a practical trade-off between association accuracy and computational efficiency. Our candidate-budget ablation, with the contextual budget fixed at 64, shows that increasing the assignment budget from 4 to 8 improves Novel AssocA from 43.03 to 43.30 and Base AssocA from 41.62 to 42.20. Further increasing the budget to 16, 32, or 64 brings almost no additional Novel improvement, with a maximum gain of only 0.03 points over the eight-candidate setting. Base AssocA also remains highest at a budget of 8. Meanwhile, the throughput decreases from 6.293 FPS at K=4 to 6.201 FPS at K=8 and 6.002 FPS at K=64. Thus, eight candidates retain most correct identities while providing the best observed Base association performance and avoiding the additional computational cost of larger assignment sets. Larger candidate sets may also introduce more ambiguous alternatives without yielding meaningful accuracy gains. The remaining 7.72% of Base events and 10.28% of Novel events fall outside the eligible set and cannot be recovered by the subsequent association scorer. These results connect actual candidate availability with the remaining identity-ranking errors and support the use of eight candidates in our inference pipeline. The complete prefilter analysis is provided in Appendix D, and the candidate-budget sensitivity study is reported in Appendix E.


Q2. Margin thresholds, search grid, and conflict resolution.

"Please give the margin thresholds selected for each frontend, the validation search grid, and the rule for resolving conflicting proposals."

Thank you for pointing this out. As you note, Appendix A gives the selection criterion and the rule against a second choice fallback, but not the final settings. We agree that these settings are necessary for reproducibility. We list them below and have added them to Appendix A.

  • 1. Search grid. The score threshold is fixed at τs = 0. The scorer is trained to rank candidates relative to each other, so the margin over the second candidate is the main acceptance test, and fixing τs leaves a single threshold to select. For the margin threshold, we first compute the margin between the top two scores of every validation query. The grid contains five values, namely zero and the 5th, 15th, 30th, and 50th percentiles of these margins. These values correspond to rejecting roughly none, 5%, 15%, 30%, and half of the least confident decisions. Rejecting more decisions leaves more observations unmatched and fragments the tracks, so the grid is denser at low rejection rates and stops at one half. The percentile positions are fixed in advance, and computing the margins requires no labels. Among the five values, we select the one with the highest Base AssocA, since τm controls identity association. The tie breakers in Appendix A were never needed. Novel, Overall, and test metrics are never used. The small grid also limits the freedom of selection.

  • 2. Selected thresholds. The same procedure is applied to each frontend.

    FrontendSelected percentileSelected τm
    COVTrack30th0.58
    OVTrack30th0.56
    VOVTrack30th0.51
    MASA15th0.45
    TETer15th0.32
  • 3. Conflict resolution. Each query first proposes its best identity only if it passes both thresholds. Two or more queries may still propose the same identity in one frame. In this case, the query with the higher final score for that identity keeps it. If the scores are equal, the query with the larger margin keeps it. The other queries do not fall back to their second ranked candidates. They are treated as unmatched observations and follow the track initialization rule of the frontend. A query with only one eligible candidate is accepted by the score threshold alone, without a margin test.


Reviewer_aVp8OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Targeting the identity association problem in open-vocabulary multi-object tracking (OVMOT), this paper represents the historical support of each query–identity pair as a query-conditioned temporal matching distribution: for every candidate identity, a feature history bank is maintained, the cosine responses between the current query and the retained historical features are collected as a set, and its mean, variance, and the closed-form second-order evaluation of the log-MGF at β = 1 (m = μ + v/2) are extracted. A candidate-specific gate then fuses evidence from the recent and long-term temporal ranges, combined with a candidate-relative residual scorer. The visual frontend remains frozen and the association layer is trained with Base-only identity supervision (~4.7k trainable parameters). On TAO, TempoTrack reaches 43.3 Novel AssocA on validation (vs. 41.3 for COVTrack) and 33.6 on test (vs. 32.6), and further reports consistent gains across four frontends, results on BDD MOT, and ablation studies.

Critical strengths

Clear and practically meaningful motivation The paper addresses a relevant and practical problem in open-vocabulary MOT. Since open-vocabulary detectors and feature extractors are often frozen, appearance features for novel, blurred, or deformed objects may vary significantly over time. The quantified finding that "the correct identity is retained in the candidate pool over 96% of the time and is merely ranked below rank 1" is independently useful to the OVMOT community, and it is also the starting point of the paper.

Training-free and modular design TempoTrack does not require retraining the detector, feature extractor, or association head. Instead, it works as an inference-time temporal association enhancement module. This makes the method easy to integrate into existing open-vocabulary trackers and gives it practical deployment value.

Strong empirical support across benchmarks and detector settings The paper reports results on the open-vocabulary MOT benchmark with both validation and test-set evaluations, and includes different detector/backbone settings. The additional results on TAO-TETA and BDD100K further demonstrate cross-dataset generalization. These experiments make the empirical evidence relatively complete.

Useful ablation, efficiency, and consolidation analysis The paper includes ablations for prototype association and set consolidation, as well as runtime and parameter analysis. In addition, the trajectory fragmentation analysis shows that EMD-based consolidation reduces the number of track IDs and increases the average track length. The set-distance ablation comparing mean pooling, Chamfer distance, Hausdorff distance, and EMD also provides empirical justification for choosing EMD.

Critical weaknesses

In Table 1, "TempoTrack (Ours)" does not state which frontend it is built on presumably COVTrack. Yet in Table 3, VOVTrack+Ours reaches 44.0 Novel AssocA on the validation set, higher than the 43.3 headline number in the abstract. Why is the strongest combination not the one reported in Table 1? And why is COVTrack — the reference frontend in the diagnostic section and in Table 1 — absent from the frontend-generalization table (Table 3)?

Contradiction between the diagnosis and the method. Section 2.2 uses Recall@64 ≈ 96%–98% to argue that "the bottleneck lies in ranking, not retrieval," yet at inference time TempoTrack retains only 8 candidates via the last-observation prefilter. The recall of this 8-candidate prefilter itself is never reported in the main text — if it falls well below 96% on Novel categories, the connection between the motivation of "ranking ambiguity" diagnosis and the implementation of the experiments is broken.

Efficiency claims lack evidence. "Lightweight" and "without explicit covariance construction" are presented in the paper, but the results report no runtime, FPS, or memory figures anywhere. Does the cost of redundancy suppression and history-bank updates grow with the history length L? At minimum, the per-frame association overhead should be reported.

The paper has no dedicated Related Work section. Relevant literature is only scattered through the introduction. Recent methods published in the past year — e.g., GOVTrack, COVTrack++, DOVTrack, and SOVTrack — should be discussed and positioned.

Questions for authors

I have reviewed this paper several times and have seen its progress, so I am able to accept it. However, given the ceiling on its originality, I can only offer a weak accept.

6 comments
AuthorsOpenReview ↗

Response to Reviewer aVp8 (Part1)

Dear Reviewer aVp8,

Thank you very much for your careful and generous review! We were genuinely touched to learn that you have read this work several times and followed its development across rounds. It is a rare and fortunate coincidence for a paper to meet the same attentive reader again, and reviewing one manuscript repeatedly is a substantial investment of time and care. Your earlier comments have already shaped this version. We are sincerely grateful for this sustained effort and address each point below.


W1. Frontend of the main result, the VOVTrack + Ours result, and COVTrack in Table 3.

"In Table 1, "TempoTrack (Ours)" does not state which frontend it is built on presumably COVTrack. Yet in Table 3, VOVTrack+Ours reaches 44.0 Novel AssocA on the validation set, higher than the 43.3 headline number in the abstract. Why is the strongest combination not the one reported in Table 1? And why is COVTrack — the reference frontend in the diagnostic section and in Table 1 — absent from the frontend-generalization table (Table 3)?"

We appreciate you drawing our attention to these related issues. Your questions helped us recognize that the relationship between the controlled comparison in Table 1, the frontend-specific integration results in Table 3, and the choice of COVTrack as our reference frontend was not sufficiently clear. We have revised the corresponding descriptions and added the missing COVTrack integration results. We clarify each point below.

  • 1. Frontend of Table 1. The "TempoTrack (Ours)" row is built on the COVTrack architecture, sharing its ViLD-based detector and frozen visual features (hence the † mark). For identity association, we do not use COVTrack's multi-cue fusion strategy; identity decisions are made by the TempoTrack scorer described in Section 4.4. The experimental configuration is clarified in Appendix A.

  • 2. Why VOVTrack + Ours is not the headline result. Table 1 compares methods under a shared detector (†), so that differences reflect association rather than detection. VOVTrack follows a different training protocol that changes both parts of the pipeline: it adds a prompt-guided attention mechanism to improve the localization and classification of time-varying objects, and it trains association with self-supervision on raw, unannotated TAO videos (about 534.1k frames and 399.9k boxes, versus 18.1k frames and 54.7k boxes in the 1-FPS annotations). Our scorer, in contrast, is trained only on Base identity events from the annotated frames. Placing VOVTrack + Ours in Table 1 would mix these frontend-level gains with the effect of our association mechanism, so we report it in Table 3, where each frontend is compared only with its own baseline. There, TempoTrack still improves VOVTrack from 40.9 to 44.0 Novel AssocA, suggesting that it is complementary to video-centric frontend training. Importantly, the two results serve different evaluation purposes. The 43.3 result is obtained under the controlled COVTrack-based configuration reported in Table 1, where the detector and visual features are shared to isolate the association contribution. The 44.0 result comes from integrating TempoTrack with the separately trained VOVTrack frontend, which uses additional raw-video self-supervision and a different training protocol. We therefore use 43.3 as the headline result for the controlled comparison, while reporting 44.0 as an additional frontend-specific integration result. This distinction ensures that the main comparison does not conflate improvements from frontend training with those from the proposed association scorer.

  • 3. COVTrack in Table 3. COVTrack is an excellent recent work, and its adaptive multi cue fusion is a strong association design in its own right. For completeness, we have added COVTrack + Ours to Table 3, integrating TempoTrack into the complete COVTrack pipeline in the same way as for the other frontends. Both COVTrack and COVTrack + Ours use the same detector, frozen visual features, and validation protocol, with TempoTrack added to the complete COVTrack association pipeline.

    SettingNovel TETANovel LocANovel AssocANovel ClsABase TETABase LocABase AssocABase ClsA
    COVTrack34.358.241.33.539.657.342.019.6
    COVTrack + Ours35.759.944.32.940.858.544.119.9
    Δ+1.4+1.7+3.0−0.6+1.2+1.2+2.1+0.3

    TempoTrack improves Novel AssocA by 3.0 points and Base AssocA by 2.1 points on top of the full COVTrack pipeline, which shows that it remains effective even when combined with a strong multi cue association. Since this setting keeps the multi cue fusion of COVTrack in addition to our scorer, we report it in Table 3 rather than as the standalone result of Table 1. The table below summarizes the gains over the baseline of each frontend.

AuthorsOpenReview ↗

Response to Reviewer aVp8 (Part 2)

(Continued from Part 1)

FrontendNovel AssocA gainBase AssocA gain
COVTrack+3.0+2.1
OVTrack+3.6+2.1
VOVTrack+3.1+2.3
MASA+3.1+2.0

TempoTrack consistently improves association across all four frontends, with larger gains on Novel categories. We emphasize that the 43.3 result in Table 1 evaluates TempoTrack as a standalone association scorer under a controlled COVTrack-based configuration. In contrast, the 44.3 COVTrack + Ours and 44.0 VOVTrack + Ours results in Table 3 evaluate the integration of TempoTrack with the respective frontend's existing association pipeline. These combined configurations retain additional frontend-specific association mechanisms or training strategies and therefore represent different experimental settings. We use 43.3 as the headline result to report the controlled contribution of our scorer, while Table 3 demonstrates its compatibility with and additional gains over existing frontend pipelines.


W2. Candidate coverage of the prefilter with eight candidates.

"Contradiction between the diagnosis and the method. Section 2.2 uses Recall@64 ≈ 96%–98% to argue that "the bottleneck lies in ranking, not retrieval," yet at inference time TempoTrack retains only 8 candidates via the last-observation prefilter. The recall of this 8-candidate prefilter itself is never reported in the main text — if it falls well below 96% on Novel categories, the connection between the motivation of "ranking ambiguity" diagnosis and the implementation of the experiments is broken."

We appreciate this careful observation. You are right that the original Recall@64 analysis alone cannot establish candidate availability under the actual inference procedure, since the original frontend ranking and the last-observation prefilter use different scoring criteria. This distinction is important for connecting our diagnostic motivation to the implemented association pipeline. We therefore re-evaluate candidate coverage using the exact inference prefilter and further examine the trade-off between candidate coverage, association accuracy, and efficiency. Our original broad-candidate analysis is intended to characterize the potential availability of correct identities and reveal the extent of identity-ranking errors, rather than to imply that all retrieved candidates are used for final assignment.

In our implementation, we adopt a top-eight assignment budget as a practical trade-off between candidate coverage, association accuracy, and efficiency. The budget sensitivity study shows that increasing the candidate budget from 4 to 8 improves both Novel and Base AssocA. Further increasing it to 16, 32, or 64 yields only marginal changes in Novel AssocA (at most 0.03 points), while Base AssocA does not improve. A larger candidate set may also introduce additional ambiguous identities and increase candidate competition without providing meaningful gains. We therefore use eight candidates for assignment while retaining up to 64 identities for contextual evidence.

We further evaluate the exact last-observation prefilter used during inference. It retains the correct identity for 89.72% of Novel events and 92.28% of Base events. Within the deployed top-eight set, ranking errors are approximately 2.7 times as frequent as retrieval failures for Novel and 3.3 times for Base. Thus, most events in which the correct identity is not ranked first still have the correct identity available for reassociation. These results confirm that the ranking problem identified by our original diagnosis remains relevant under the actual inference budget. We have added the complete coverage analysis to Appendix D and the budget sensitivity results to Appendix E.

  • 1. Recall of the eight candidate prefilter. We measure the recall on the same diagnostic events as Figure 2 (COVTrack, 4,648 Base and 457 Novel events). Candidates are ranked exactly as at inference, by similarity to their last actual observation.

    SplitR@1R@2R@4R@8R@16R@32R@64
    Base66.5078.6887.3592.2895.3397.2998.99
    Novel61.4976.1582.2889.7294.3196.7298.91

    Novel R@8 is 89.72%, which is lower than R@64 as you anticipated. However, the smaller budget does not hurt Novel identities more than Base ones. The gap between Novel and Base is 5.01 points at R@1 and only 2.56 points at R@8.

AuthorsOpenReview ↗

Response to Reviewer aVp8 (Part 3)

(Continued from Part 2)

  • 2. The diagnosis still holds with eight candidates. The diagnosis concerns whether errors come mainly from ranking or from retrieval. We therefore divide all events by the position of the correct identity under the deployed prefilter.

    SplitRank 1Ranks 2 to 8 (ranking error)Outside top 8 (retrieval failure)
    Base66.50%25.77%7.72%
    Novel61.49%28.23%10.28%

    Even with only eight candidates, ranking errors occur 2.7 times as often as retrieval failures for Novel and 3.3 times as often for Base. In these events, the correct identity is already among the candidates but is not ranked first. This is exactly the setting in which TempoTrack operates. The excluded events (10.28% for Novel) set an upper bound on what any scorer can recover, as discussed in Appendix D and Appendix I.

  • 3. A larger budget does not bring further gains. We also vary the decision budget while keeping the context of 64 candidates fixed.

    Decision budgetNovel AssocABase AssocAFPS
    443.0341.626.293
    8 (ours)43.3042.206.201
    1643.3241.696.194
    3243.3341.926.181
    6443.3341.916.002

    Importantly, increasing the budget improves candidate recall, but does not translate proportionally into association accuracy. This distinction suggests that retrieving more identities is not necessarily beneficial once the candidate set already provides sufficient coverage. A compact assignment set also avoids unnecessary competition among additional candidates. Four candidates run slightly faster but lower accuracy. Budgets larger than eight change Novel AssocA by at most 0.03 points, and Base AssocA is highest at eight. Although larger budgets increase raw candidate recall, these additional candidates translate into negligible improvements in AssocA. They also increase the number of competing identities and computational cost. We therefore adopt eight candidates as a practical balance between candidate coverage, association accuracy, and efficiency. We therefore use eight candidates as a good trade off between accuracy and speed.


W3. Evidence for the efficiency claims.

"Efficiency claims lack evidence. ..."

We appreciate your emphasis on the need for quantitative evidence behind the efficiency claims. This prompted us to profile the history-dependent operations in greater detail, rather than relying only on the theoretical complexity of the proposed formulation. We now report per-frame latency, throughput, feature memory, and peak memory increase across different history lengths, using the same frontend and hardware. We have now profiled every part of TempoTrack that depends on the history length L. All numbers are averaged per frame on TAO validation, with the same frontend and hardware for every row.

History length LResponse (ms)Gate and residual (ms)History update (ms)Total per frame (ms)FPSHistory features (MiB)Peak memory increase (MiB)
81.370.480.95153.06.540.58.6
161.510.481.84153.76.511.09.2
322.290.483.43155.96.412.010.8
64 (ours)2.880.497.16161.36.204.015.1
1284.570.4913.72171.35.848.027.8

This breakdown is important because the frontend accounts for most of the total runtime, whereas our contribution introduces only a small history-dependent overhead. We therefore report both the total throughput and the incremental costs of the operations introduced by TempoTrack.

  • 1. The overhead is small. At L = 64, the three history dependent parts take about 10.5 ms per frame in total. The remaining cost of approximately 150 ms per frame is mainly due to frontend computation and remains largely unchanged as L varies. The resulting speed of 6.20 FPS agrees with the 6.2 FPS reported in Appendix E. The history features of all identities occupy only 4.0 MiB.

  • 2. The cost grows at most linearly with L. The response computation and the history update scale with L, because each candidate is summarized by L scalar responses. The learned gate and residual stay at about 0.48 ms for every L. They operate on a fixed 33 dimensional descriptor, so their cost does not depend on the history length. The row with L = 128 exceeds our deployed capacity of 64 and is shown only to illustrate the trend.

  • 3. Why scalar statistics are efficient. TempoTrack computes the projected mean and variance directly from scalar matching responses, requiring O(L) storage and O(Ld) computation per candidate. It therefore avoids explicitly constructing and storing the feature-dimensional covariance matrix, which would require O(d²) memory per identity. No matrix inversion is required. This allows historical evidence to be maintained for each identity with a small additional memory cost.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.