acceptodds
Under review as a conference paper at ICLR 2027

TrackRerank: Semantic Enhancement for Two-Stage Open-Vocabulary Tracking

Abstract

A two-stage open-vocabulary tracker can follow an object correctly while repeatedly assigning it the wrong category. We introduce \method, a causal post-association semantic head that revisits category decisions without changing boxes, detection confidence, or identities. It gathers candidates from detector predictions, current-frame vision–language retrieval, and past track observations. A category-shared scorer interprets each candidate through its source confidence, relative competition, uncertainty, cross-source support, and track context. On a custom video-disjoint OVT-B split, the complete pipeline raises YOLOE–ByteTrack from 13.307 to 25.300 ClsA (+11.993 points), exceeding causal voting and EMA baselines. Under shared candidates, supervision, and optimization, the full representation gains 1.050 points over a missingness-aware score-fusion MLP with nearly identical parameter count, with positive gains in all three paired seeds. The same trained head, connected through an observation-aligned adapter, improves native OVTR on TAO by 1.390 causal ClsA points without target-domain fitting. Its within-group accuracy gains are larger when current sources conflict or are uncertain. These results distinguish the benefit of gathering additional semantic evidence from the benefit of interpreting its candidate-level context.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
open-vocabulary multi-object tracking, semantic refinement, multi-source evidence fusion, candidate reranking, vision-language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_mG58OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents TrackRerank, a post-association semantic reranking module for two-stage open-vocabulary tracking. The method constructs candidate categories from detector predictions, current-frame CLIP retrieval, and causal track memory, and represents each candidate using a 34-dimensional feature vector containing source scores, ranks, margins, uncertainty, source agreement, and track statistics. A lightweight MLP then scores these candidates to produce the final semantic prediction. On a custom video-disjoint OVT-B split, TrackRerank improves ClsA from 13.307 to 25.300, while the controlled comparison with Score+Mask-PM reports a 1.050-point improvement. Additional experiments on TAO and OVTR investigate transferability and conditional behavior. Although the paper provides a relatively comprehensive empirical study, I find that the core methodological contribution is limited, as the proposed approach mainly combines manually designed statistics from multiple semantic sources with a small MLP, and the incremental gain over a strong parameter-matched fusion baseline is relatively modest. Thus, I recommend reject.

Critical strengths
  1. Comprehensive experimental analysis. The paper evaluates multiple semantic fusion baselines and includes controlled comparisons, multi-seed experiments, bootstrap analysis, and conditional error analysis. This provides a relatively detailed study of when semantic reranking is effective.
  2. Strong experimental control. The comparison with Score+Mask-PM uses the same candidate pool, supervision, optimization budget, and missingness information, while keeping the parameter counts close. The consistent improvement across three seeds provides useful evidence that the additional contextual features contribute beyond a simple score-fusion model.
  3. Useful transfer and diagnostic experiments. The TAO and OVTR experiments provide additional evidence beyond the main OVT-B setting, while the analysis of conflicting sources and high-uncertainty observations helps clarify the cases in which TrackRerank provides the most benefit.
Critical weaknesses
  1. The core methodological novelty is insufficient. The proposed TrackRerank largely consists of combining manually designed statistics from multiple semantic sources and feeding them into a small MLP. The input features include scores, ranks, margins, entropy, source agreement, historical support, and track statistics. These are intuitive feature engineering choices rather than a substantially new learning paradigm. The strongest controlled comparison against Score+Mask-PM provides only a 1.050-point ClsA improvement, despite the overall improvement over the original detector being much larger. Therefore, the main performance gain appears to come largely from introducing additional semantic sources and temporal information rather than from a fundamentally new reranking mechanism.

  2. The main experimental improvement is strongly tied to the specific evaluation and supervision protocol. The principal results are obtained on a custom OVT-B split rather than an official benchmark evaluation, and the training process relies on OWLv2 teacher pseudo-labels. The paper itself reports that removing Novel positive supervision decreases Novel ClsA from 25.820 to 23.400, indicating a substantial dependence on the positive supervision scope. This makes it difficult to determine whether the reported gains reflect a generally effective open-vocabulary tracking method or primarily an effective resolver under this particular pseudo-label and candidate construction protocol.

  3. The evidence for generalization is not sufficiently convincing. The TAO experiment provides some transfer evidence, but the method is still applied to frozen upstream outputs through a specially designed adapter. Moreover, the gain over Score+Mask-PM becomes only 0.530 ClsA on native OVTR outputs. This is a relatively small improvement considering the additional semantic processing and carefully constructed evidence representation. The experiments therefore do not convincingly establish that the proposed method generalizes broadly across open-vocabulary tracking settings or different tracking architectures.

  4. The candidate construction fundamentally limits the scope of the method. TrackRerank only reranks categories that are already present in its candidate pool. It cannot recover a target category that is absent from the detector, CLIP retrieval, or memory candidates, nor can it correct upstream localization or association errors. Consequently, the method is primarily a semantic post-processing module rather than a complete open-vocabulary tracking solution. This limitation substantially weakens the broader significance of the contribution.

  5. The training observation filtering may introduce a favorable data bias. The training procedure retains 499,845 of 603,678 frozen observations after teacher matching, confidence filtering, and candidate reachability filtering. The supplementary material acknowledges that confidence-based selection may favor easier observations. Thus, the reported training population may not represent the full difficulty distribution of the tracking problem, which raises concerns about how robustly the learned reranker handles difficult or unrecoverable cases.

  6. The practical benefit is not sufficiently demonstrated. Although the reranking head itself is small, the complete method requires additional crop extraction and CLIP inference. The paper does not provide a convincing end-to-end latency or throughput comparison with the original tracker. Therefore, the practical significance of the reported semantic accuracy improvement remains unclear.

Questions for authors
  1. What is the key methodological novelty of TrackRerank beyond a feature-engineered MLP for score fusion? Which component is fundamentally new rather than an intuitive combination of existing semantic and temporal statistics?

  2. How much of the 12-point improvement from 13.307 to 25.300 ClsA comes from adding CLIP retrieval and track memory, and how much comes specifically from the proposed 34-dimensional representation and MLP architecture?

  3. Can the authors demonstrate comparable gains on additional datasets and substantially different upstream open-vocabulary trackers without designing a dataset-specific adapter?

  4. How does TrackRerank perform when the OWLv2 pseudo-label teacher is replaced by another teacher or when pseudo-label quality is substantially reduced?

  5. What is the end-to-end inference overhead of CLIP feature extraction and candidate construction compared with the original tracker?

Reviewer_oBFxOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies category recognition errors in open-vocabulary object tracking: existing trackers can continuously track the same object, but their category predictions may be incorrect or unstable. The authors propose TrackRerank, which combines candidate categories from the detector, current-frame CLIP, and historical trajectories without changing the detection boxes or track identities, and uses a small shared network to select the category again based on prediction scores, uncertainty, agreement between sources, and trajectory context. Experiments show that this method can improve classification performance on OVT-B and can be transferred to TAO and native OVTR outputs. Controlled comparisons further show that candidate context features provide additional benefits over simple score fusion. I read this abstract several times, and I found that the logical connections between the sentences are not explained. For example, there seems to be no connection between the first and second sentences. This paper is a bit difficult to read. I suggest that the authors revise and improve this abstract.

Critical strengths

The paper focuses on the problem that trackers can continuously track objects, but their category predictions may be incorrect or unstable. The proposed method can correct categories without changing detection boxes or track identities, making it easy to integrate into existing trackers, and has shown its effectiveness in two different tracking frameworks.

The main controlled experiments are reasonably designed. The authors compare the full model with score-fusion models using the same candidate sets, the same training supervision, and similar parameter counts, and check the results through feature ablations and three random seeds. These experiments provide some support for the benefits of the additional context features.

Critical weaknesses

I think the evaluation of generalization to novel categories is insufficient. Positive pseudo-labels from Novel categories are used during training in the main experiments, so these results cannot directly show how the method generalizes to categories not included in positive supervision. Although the authors conducted an experiment using only positive labels from Base categories, they did not compare with the strongest fusion model under the same setting, so it is still unclear whether the proposed context features retain an advantage in this case.

The additional value of the full feature design is not clear enough. The existing control model only uses source scores and missingness indicators, while the full model additionally uses ranks, score margins, uncertainty, and historical information. The ablation experiments show that the group of rank, score margin, and uncertainty features has the largest effect, but the paper does not report a comparison where only this feature group is added to the control model. Therefore, it is still unclear whether the full 34-dimensional design is needed, or whether simply adding a few confidence statistics can achieve similar results.

Questions for authors

When trained using only positive labels from Base categories, does Full34 still outperform Score+Mask-PM on Novel categories? Could you provide a comparison under the same setting?

If only rank, score margin, and uncertainty features are added to Score+Mask-PM, how much additional improvement does the full 34-dimensional model still provide?

Reviewer_KTfFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TrackRerank, a post-association semantic refinement module for OVMOT that keeps boxes, confidences, and track IDs fixed while reranking category candidates from the detector, current-frame CLIP, and history using a 34-dimensional handcrafted evidence vector scored by a lightweight class-shared MLP.

Critical strengths

Strengths

  1. Simple and effective approach. The paper presents a straightforward approach to semantic refinement in open-vocabulary multi-object tracking (OVMOT). The method is easy to understand, and the reported results demonstrate its potential effectiveness.

  2. Extensive analytical experiments. The experimental evaluation goes beyond reporting overall performance gains. Multiple controlled comparisons and ablation studies help explain the sources of improvement and provide insight into the proposed design.

  3. Well-motivated research problem. Improving classification is a meaningful direction for OVMOT, where semantic recognition remains a substantial challenge alongside detection and association. Addressing classification errors on existing trajectories is therefore a reasonable and practically relevant research objective.

Critical weaknesses

Weaknesses

  1. Insufficient writing precision and polish. The presentation would benefit from more careful editing. Several descriptions lack precision and rigor, and the overall exposition is not sufficiently polished, making the technical details and experimental conclusions harder to assess.

  2. Insufficient comparison with recent strong methods. The paper lacks a sufficiently comprehensive comparison with recent representative OVMOT methods on established benchmarks under consistent evaluation protocols. This makes it difficult to assess its competitiveness. Although integration with OVTR is reported, broader validation with additional recent strong trackers (VOVTrack, COVTrack, COVTrack++, etc) is needed. Comparing each original tracker with its TrackRerank-enhanced counterpart would provide stronger evidence of the module’s general applicability and consistent benefits.

  3. Limited methodological depth and insufficient justification of handcrafted features. While the approach is simple and empirically effective, its technical novelty and underlying principled justification appear relatively limited. In particular, the 34-dimensional evidence representation relies substantially on manually designed features and prior assumptions. The rationale for selecting this particular feature set, and the sensitivity of performance to these choices, require further clarification. Beyond the reported analyses, more systematic feature-group ablations, sensitivity studies, and evaluations using a fixed feature configuration across datasets and tracking backbones would help establish the robustness and generality of the design.

Questions for authors
  1. Beyond OVTR, can it consistently improve additional recent strong trackers when plugged in, with side-by-side results for each original tracker and its enhanced version?

  2. What principles motivate the 34 evidence features, which feature groups are essential, and how sensitive are results to removing or replacing feature groups?

  3. With the same feature definitions and trained scoring network, can the method transfer across datasets, backbones, and category distributions without adaptation, and which results come from direct transfer versus adaptation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.