acceptodds
Under review as a conference paper at ICLR 2027

TORUS: Scaling Open-Vocabulary Tracking with Heterogeneous Data and State-Space Models

Abstract

Open-vocabulary multi-object tracking (OVMOT) is short of training data: images with large vocabularies are plentiful, but annotated tracking videos are scarce and temporally sparse. Existing methods either simulate motion from images—which poorly represents real-world deformation and occlusion—or label video by hand, which fails to scale. We introduce TORUS, an online tracker that bypasses this trade-off by unifying both data sources, jointly learning from detection images, disjoint-vocabulary videos, and synthetic pseudo-videos—without using any TAO annotations. TORUS freezes an open-vocabulary detector, adding a selective state-space (Mamba) memory to propagate tracks across frames. To learn jointly from this data mixture, we introduce two key designs: i) a class-agnostic confidence head that decouples track lifecycle management from classification, keeping tracks alive when predicted classes flip; and ii) a spatial-semantic federated loss that masks objectness supervision to avoid background penalization where labels are incomplete. With a ResNet-50 backbone, TORUS achieves 42.5 Base and 42.1 Novel TETA on TAO OVMOT, outperforming the state of the art by +2.9 and +6.8 points. A single set of untuned weights generalizes zero-shot across DanceTrack (57.7 HOTA), YouTube-VIS 2021 (42.9 mAP), and VisDrone-MOT (42.3 TETA)—operating entirely online at 14.7 FPS under 1,203 prompt

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Open Vocabulary, Multiple Object Tracking, Segmentation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ijyuOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes TORUS, an online open-vocabulary multi-object tracker trained with heterogeneous data sources. It builds on a frozen MOBIUS open-vocabulary detector and introduces a Mamba-based temporal memory, a class-agnostic confidence head, and a spatial-semantic federated loss. The model is jointly trained on detection images, real videos with restricted vocabularies, and synthetic pseudo-videos, without using TAO annotations. Experiments are reported on TAO OVMOT, DanceTrack, YouTube-VIS 2021, and VisDrone-MOT.

Critical strengths
  1. The idea of separating large-vocabulary semantic learning from temporal association learning is interesting and potentially useful. Using large-vocabulary images for semantics and restricted-vocabulary videos for temporal modeling is a reasonable way to reduce the dependence on densely annotated open-vocabulary tracking videos.

  2. The paper evaluates the framework across several tracking and video instance segmentation benchmarks, and the reported numerical results are strong.

Critical weaknesses
  1. The most serious issue is the fairness of the main experimental comparison. TORUS is built on the frozen MOBIUS detector, whose open-vocabulary detection capability is substantially stronger than the detectors used by most competing OVMOT methods in Table 1. This makes the reported TETA comparison fundamentally difficult to interpret. A stronger detector directly improves localization and classification quality, and therefore naturally benefits LocA and ClsA. More importantly, better detection quality, recall, and localization also make temporal association considerably easier, so AssocA is affected as well. Therefore, the gains in LocA, ClsA, AssocA, and consequently TETA cannot be cleanly attributed to the proposed tracking framework. Comparing complete systems with substantially different detector capabilities does not convincingly demonstrate that TORUS is a better tracker. A meaningful comparison should control the detector/backbone and preferably the training data, or provide detector-matched baselines using the same MOBIUS detector. Without such controlled comparisons, the central state-of-the-art claim is not sufficiently supported. This is my main reason for rejection.
  2. Several generalization claims are overstated or not supported by the evaluation protocol. The paper repeatedly describes the DanceTrack result as zero-shot generalization, although DanceTrack is explicitly included in the training mixture (about 3% of the samples). The TAO Novel categories were also already observed by the frozen MOBIUS detector during LVIS pretraining, so the Novel results primarily measure retention of pretrained semantic knowledge rather than genuine unseen-category generalization. Similarly, on YouTube-VIS, TORUS has seen the novel classes through LVIS and has been trained with video masks from other datasets, which makes the comparison with existing zero-shot/open-vocabulary methods difficult to interpret. In addition, the main TAO OVMOT evaluation is reported only on the validation set, while test-set results are missing. Together, these issues substantially weaken the claimed cross-dataset and open-vocabulary generalization.
Questions for authors

1.Can the authors provide a detector-controlled comparison in which TORUS and the main competing tracking methods use the same MOBIUS detector, or otherwise control the detector/backbone and training data? Without such an experiment, how can the improvements in LocA, ClsA, and especially AssocA be attributed to the tracking method rather than to the much stronger detector?

2.Why is DanceTrack described as a zero-shot evaluation when it is included in the training mixture? Please also clarify the intended meaning of “zero-shot” for TAO Novel classes, since these categories have already been seen by MOBIUS during LVIS pretraining.

3.Why are TAO OVMOT results reported only on the validation set? Please provide the corresponding test-set results.

4.The paper claims that the class-agnostic confidence head decouples track lifecycle management from classification, but track birth, update, and deletion are still based on a fused score containing the maximum classification score. What exactly is being decoupled in this formulation?

5 comments

We are happy that the reviewer finds separating semantic from temporal learning a useful way to reduce the dependence on densely annotated tracking videos, and we thank them for the detailed review.

Q1: Detector-controlled comparison.

We agree that Table 1 compares complete systems whose detectors differ in strength and pretraining data; we will state this explicitly and add the detector and its pretraining data to Table 1. The paper does contain a detector-matched baseline (Table 5, standard TAO), which we will make more prominent: associating the detections of the same frozen MOBIUS detector with MinVIS, instead of our tracker.

Standard TAO (Table 5)TETALocAAssocAClsA
SLAck (Swin-T)35.552.238.915.6
COVTrack36.955.141.114.6
MOBIUS-R50 + MinVIS (offline)36.248.438.022.2
TORUS-R50 (same detector)42.860.643.024.8
Δ\Delta TORUS vs. MOBIUS + MinVIS+6.6+12.2+5.0+2.6

Two observations address the question about AssocA. First, with the detector held fixed, the tracker accounts for +6.6 TETA and +5.0 AssocA. Second, the stronger detector alone does not yield better association: MOBIUS with MinVIS reaches 38.0 AssocA, below COVTrack (41.1) and SLAck (38.9), and its TETA (36.2) is on par with COVTrack (36.9).

For scale and data: TORUS-R50 has a parameter count comparable to OVTR and COVTrack (71.3M vs. 58.8M and 74.4M, Table 10) and uses 120k tracking samples and no TAO annotation, whereas COVTrack trains on ~4.9M samples of densified TAO annotations and VOVTrack on unlabeled TAO videos.

We recognize that this does not fully control the detector and training data for every baseline. We are therefore making an effort to re-train TORUS on top of the OVTR detector, using OVTR's released weights, CLIP text embeddings and LVIS-base data. In this setting, only the tracker differs from OVTR, and the Novel classes are genuinely unseen.

Q2: Meaning of "zero-shot" for DanceTrack and TAO Novel.

We agree that our use of "zero-shot" was inconsistent. DanceTrack is part of the training mixture (about 3% of samples, Table 12), so calling it zero-shot was incorrect, and we will correct the abstract and introduction (see also our response to JzTQ). The intended claim is a single set of weights evaluated on all benchmarks without benchmark-specific fine-tuning; true zero-shot transfer applies only to VisDrone-MOT.

For TAO Novel, we agree the classes are not unseen. As stated in the Table 1 footnote, Sec. 4.2 and the Limitations, Novel scores measure how well the detector's knowledge of these classes is retained through tracking training, not zero-shot discovery. Our first contribution used "zero-shot to unseen categories" inconsistently with this, and we will correct it. The same holds for YouTube-VIS, where, as noted in the Table 2 caption, only the base-class mAP compares directly with OV2Seg and OVFormer. For a setting in which Novel classes are genuinely unseen, see the OVTR-detector experiment above.

Q3: Why are TAO OVMOT results reported only on the validation set?

Validation is the standard evaluation split in this field and the one on which every compared method reports results. The TETA protocol of TETer, which defines the standard TAO benchmark, is evaluated on the validation set; OVMOT works since OVTrack report their main results and ablations on it; and some baselines, such as MASA, report validation results only. Validation is therefore the only split that allows a complete comparison in Tables 1 and 5. With limited compute, we have prioritized the detector-controlled experiments above; test-set evaluation requires no retraining, and we aim to add it if time permits.

Q4: What exactly is decoupled?

Reviewer JzTQ raised the same point; please see our response to their Q1. In short, the decoupling concerns what each head learns: the classifier is trained on matched queries only and learns semantics, the confidence head learns occlusion, duplicates and background, and the stop-gradient fused score with a maximum over classes keeps tracks alive through class flips while gating out objects outside the prompted vocabulary. We are running the requested comparison with the confidence score alone; as explained there, we expect only a small difference on TAO, where all classes are prompted, and the class factor to matter when only a subset of classes is prompted.

Thank you for the response. However, my main concerns remain insufficiently addressed.

Q1. Detector-controlled comparison. The MOBIUS+MinVIS comparison is conducted under the standard closed-set TAO protocol rather than the main OVMOT setting. More importantly, although both methods share the frozen MOBIUS encoder, their LocA differs substantially (48.4 vs. 60.6). Therefore, attributing the +5.0 AssocA improvement solely to the tracking framework is not sufficiently justified, as differences in detection and localization quality can also affect association performance. The proposed OVTR-detector experiment would be relevant, but no results have been provided.

Q2. Zero-shot generalization. I appreciate the authors' acknowledgment of the incorrect zero-shot claims. However, the current manuscript remains unchanged, and these claims are still present in the abstract and introduction. The promised corrections have therefore not yet been incorporated.

Q3. TAO test-set evaluation. While validation-set evaluation is common, several representative OVMOT methods, including OVTrack, COVTrack, and VOVTrack, also report test-set results. Thus, the explanation that validation is the standard evaluation split does not sufficiently justify the absence of test-set results, particularly given the paper's generalization claims.

Q4. Confidence-head decoupling. The response clarifies the separation of the training objectives, but track birth, update, and deletion still depend on the fused confidence-classification score. The proposed confidence-only ablation has not yet been provided, so the concern remains unresolved.

Overall, the response provides useful clarifications but no new experimental evidence directly addressing my central concern about the fairness and attribution of the main OVMOT improvements. The proposed additional experiments and manuscript corrections remain pending. Therefore, my assessment and recommendation remain unchanged at this stage.

We thank the reviewer for the prompt follow-up.

Q1: Detector-controlled comparison.

We agree that the MOBIUS+MinVIS comparison is closed-set only. On the LocA gap, the two methods do not differ in their per-frame detector. Both use the same frozen backbone and encoder. TORUS fine-tunes the decoder, which makes its frame-level detection slightly worse than MOBIUS on COCO and LVIS (−1.7 and −2.3 AP, Table 4). The higher LocA of TORUS therefore does not come from a stronger detector. It comes from the tracking framework: track queries carry each object's state into the next frame and act as priors for detection.

The TAO detection experiment in our response to Reviewer zMiV (W2) isolates this effect. With the same TORUS checkpoint, frame-level box AP on the annotated TAO val frames is 20.5 when each frame is processed independently and 33.6 when the track queries of previous frames are used (+13.1 AP, +14.0 AP$_r$).

We agree that better localization also eases association, so the AssocA gain cannot be separated completely from the LocA gain. However, both gains come from the same component: propagating track queries through the detector is how TORUS associates, and it is also what improves detection. We consider this joint improvement part of the tracking contribution, not a detector advantage. We will make this argument and the TAO detection results part of the detector-controlled comparison in the final revision.

The OVTR-detector experiment is in progress. We will share its results if it completes within the discussion period.

Q2: Zero-shot claims in the manuscript.

We have uploaded a revision that corrects the zero-shot claims in the abstract and introduction, and states the YouTube-VIS majority-voting exception (Sec. 4.1) and the fused decision rule after Eq. (3). Please note that we are likely going to make further changes to the manuscript, such as adding any new ablations, until the end of the discussion period.

Q3: TAO test-set evaluation.

We acknowledge that OVTrack, VOVTrack and COVTrack also report test-set results. We are working on a test-set evaluation, but cannot guarantee that it will complete within the discussion period.

Q4: Confidence-only lifecycle ablation.

The ablation is now complete; please see our follow-up to Reviewer JzTQ (Q1) for the full tables. With the same checkpoint, lifecycle management on the confidence score alone is within 0.4 TETA of the fused score on TAO (Base 42.9 vs. 42.5, Rare 41.9 vs. 42.1), where all classes are prompted. On DanceTrack, where only person is prompted, it starts tracks for every confident object: false positives increase about tenfold and HOTA drops from 57.7 to 47.6, with association unaffected (AssA 44.6 vs. 43.3). The confidence head thus carries the track lifecycle on its own, and the class factor acts as a vocabulary gate.

Reviewer_zMiVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work tackles the problem of open-vocabulary multi object tracking by combining training data from multiple sources — images, videos, and pseudo-videos. It extends an open-vocabulary detector by adding memory to propagate tracks across frames of a video. Learning the semantics of classes from their temporal tracking are decoupled into two separate problems to tackle in their architecture / training (for eg. you can learn semantics from images only and temporal continuity from videos/pseudo-videos). Zero-shot evaluation on standard tracking benchmarks shows improvements.

Critical strengths
  • the method is well motivated and the architecture for using images and videos for learning different concepts seems well-designed. Ablations establish that both images and videos are important training sources and the two main designs in training (loss and confidence head) contribute to performance
  • zero-shot results on the considered benchmarks beat the prior SOTA by a considerable margin across standard tracking metrics, and on the TAO benchmark suite for open vocabulary tracking
  • since the method starts from a frozen open-vocabulary detector, it is efficient in terms of number of training samples needed for open-vocabulary tracking
Critical weaknesses
  • it is not so clear what the benefit of using pseudo-videos is? Do they just add more video training data, are they useful for covering more open-vocabulary ground. There are no ablations for this and existing set of ablations don’t differentiate between the two kind of video sources.
  • this may not be standard but I’m wondering what the detection metrics on TAO benchmark look like — does the tracking machinery on top of MOBIUS improve detection AP or worsen it, since static images are also included in the training.
  • There is qualitative analysis missing from the paper —3 tracking results are shown but there is no comparison with other baseline trackers which could have shed light on what kind of issues the proposed tracker solves over the base model. Similarly, for all architectural ablations it is not clear intuitively what issues are solved by each of the components.
Questions for authors

see above

2 comments

We appreciate that the reviewer finds the method well motivated and its design well suited to learning different concepts from images and videos.

W1: What is the benefit of pseudo-videos: more video data, or more vocabulary?

Table 3 isolates the pseudo-videos: "Images + videos" and the full recipe differ only in whether pseudo-videos are used, at the same number of iterations. Pseudo-videos replace 20% of the independent image samples, so they add no training data. We agree the table is hard to read and will add explicit data-source columns. The relevant rows:

LVIS AP$_r$Base TETABase AssocABase ClsARare TETARare AssocARare ClsA
Images + videos37.942.042.323.240.943.216.5
+ pseudo-videos (full)39.342.543.124.142.143.819.3
Δ\Delta+1.4+0.5+0.8+0.9+1.2+0.6+2.8

If pseudo-videos acted mainly as additional motion data, we would expect gains in association on both splits. Instead, the gain concentrates on classification of the classes the real videos cover sparsely (Rare ClsA +2.8, LVIS-rare AP +1.4), while AssocA changes by less than one point. Their benefit is therefore vocabulary coverage: they expose the memory to every category of the image datasets over time. They cannot replace real video, as they show camera motion but not deformation; images with pseudo-videos alone reach only 6.5 Rare ClsA. We will add this analysis to Sec. 4.5.

W2: Does the tracking machinery improve or worsen detection on TAO?

The question has two parts. What tracking training does to the detector is measured on COCO and LVIS in Table 4: compared with the frozen MOBIUS detector, tracking training costs 1.7 COCO AP and 2.3 LVIS AP, a small degradation as expected, and gains 4.6 LVIS-rare AP.

What temporal context does at test time was not measured in the paper. We therefore evaluated box AP (LVIS protocol, top 300 detections per frame, no thresholds) on the annotated TAO val frames with the same TORUS checkpoint, once frame by frame without track queries, and once while tracking, where each frame also receives the track queries of the previous frames:

TAO val frames, box APAPAP50AP$_r$AP$_c$AP$_f$
TORUS, frame by frame20.526.817.815.124.6
TORUS, tracking33.644.631.830.236.2
Δ\Delta+13.1+17.8+14.0+15.1+11.6

Temporal propagation improves detection substantially, across all frequency groups including rare classes. Absolute values should be read with care, as TAO was designed for tracking, not for frame-level AP. It annotates only a subset of the objects in each video, so correct detections of unannotated objects are penalized, and TORUS is trained to detect such objects rather than suppress them (federated mask). Conversely, TAO keeps labeling objects along their tracks, including frames in which they are heavily occluded or blurred, instances that image datasets would typically not annotate and that a single-frame detector can hardly recover. These are the cases in which we expect temporal propagation to help most. The comparison is between two inference modes of the same model under the same evaluation. We will add this analysis to the appendix.

W3: Qualitative comparison with baselines and per-component intuition.

We agree that this would make the contribution of each component clearer. We will add qualitative comparisons of TORUS with baseline trackers on the same sequences, as well as qualitative examples for the ablated components.

We thank the reviewer again for the constructive review. As the discussion period ends, we summarize how each point is addressed in the revised manuscript.

W1: Benefit of pseudo-videos. Table 3 now shows the training data as explicit Images / Videos / Pseudo-videos columns. The caption states that the last two rows train for the same number of iterations and that pseudo-videos replace image samples instead of adding data. Sec. 4.5 adds the corresponding analysis: pseudo-videos raise Novel ClsA by 2.8 and LVIS-rare AP by 1.4, while AssocA changes by less than one point. Their benefit is vocabulary coverage, not additional motion data.

W2: Effect of tracking on detection. The frame-level evaluation on TAO is now in the appendix (Extended ablation study). With the same TORUS checkpoint, box AP on the annotated TAO val frames rises from 20.5 frame by frame to 33.6 while tracking, consistently across rare, common and frequent classes. The section also discusses why absolute AP on TAO, a benchmark designed for tracking, should be read with care. We also connect this result to the detector-controlled comparison with MinVIS: the localization gain of TORUS over the same detector comes from track queries acting as detection priors.

W3: Qualitative comparison. Within the discussion period and our compute budget, we were not able to produce the qualitative comparisons with baseline trackers. We prioritized the quantitative analyses above and the detector-controlled and lifecycle experiments requested by the other reviewers, as these test the paper's claims directly. We agree that side-by-side qualitative examples would make the contribution of each component more intuitive.

We hope these revisions address the reviewer's concerns and would be glad to clarify any remaining questions.

Reviewer_JzTQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TORUS, an architecture that leverages heterogeneous data to train an open-vocabulary multi-object tracking model. The authors argue that static images provide rich category supervision, whereas videos provide temporal and motion information, but category annotations in videos are typically incomplete. To address this issue, TORUS freezes a pretrained MOBIUS detector and introduces a Mamba-based temporal memory module, a separate tracking confidence branch, and a federated mask to handle incomplete category annotations. The training data consist of static images, real videos, and synthetic pseudo-videos. The paper evaluates the proposed method on benchmarks including TAO OVMOT, DanceTrack, YouTube-VIS 2021, and VisDrone-MOT, and analyzes the contributions of different data sources and model components through ablation studies. The authors report a Base TETA of 42.5 and a Novel TETA of 42.1 on TAO OVMOT, as well as a HOTA of 57.7 on DanceTrack.

Critical strengths
  1. The research problem is clearly defined, and the heterogeneous-data training scheme has practical value. This paper addresses the issue of rich image annotations but insufficient video category annotations in open-vocabulary tracking and attempts to leverage the complementary information from different data sources. The design of the federated mask also targets the problem of erroneous negative supervision under incomplete annotations, providing a reasonable technical motivation.

  2. The experiments cover a broad range of benchmarks and include meaningful ablation analyses. The paper not only reports open-vocabulary tracking results on TAO OVMOT but also evaluates the method on DanceTrack, YouTube-VIS 2021, and VisDrone-MOT, and compares different backbones and memory modules. These experiments indicate that the proposed method has potential for application across different tasks.

Critical weaknesses
  1. The Core Methodological Claim Does Not Fully Align with the Actual Decision Mechanism. The paper claims that the independent confidence head can decouple track lifecycle management from category classification. However, track creation, memory updates, and track deletion all use the fused score (Equation (3)). This means that category scores still influence track management decisions.

  2. The Experimental Claims Regarding Cross-Benchmark Generalization and Online Inference Have Not Been Fully Validated Consistently. The paper reports competitive results on multiple benchmarks, but the training data and evaluation protocols used in different experiments are not entirely consistent. DanceTrack is included in the training mixture, while the caption of the table for YouTube-VIS 2021 mentions trajectory-level majority voting, which is inconsistent with the paper's general characterization of online inference in the main text. Furthermore, the authors state that all results are obtained from a single run, without reporting the mean and variance across multiple random seeds.

Questions for authors
  1. Equation (3) multiplies the independent confidence score by the highest category score to obtain the fused score, which is used for track creation, memory updates, and track deletion. Under this design, in what sense is track lifecycle management decoupled from category classification? Could the authors further explain the distinction between the two and, where feasible, compare the track management results obtained using only the independent confidence score with those obtained using the fused score?

  2. Table 2 shows that DanceTrack accounts for approximately 3% of the training mixture, whereas the abstract describes the model's generalization to DanceTrack as zero-shot generalization.

  3. Section 4.1 claims that all TORUS results are obtained using online inference without majority voting over completed trajectories. However, the caption of Table 2 states that all methods on YouTube-VIS 2021 use majority voting. Does the TORUS result on this benchmark also use trajectory-level majority voting? Please explicitly clarify whether this exception applies and distinguish between online category prediction and post-processing that depends on complete trajectories.

  4. Stability of the Main Experimental Results: All reported results are obtained from a single run. Could the authors repeat the main experiments on TAO OVMOT, as well as the comparison between Mamba and Transformer memory modules on DanceTrack, using different random seeds and report the mean and standard deviation?

3 comments

We thank the reviewer for the thorough review, and we are glad they see practical value in training on heterogeneous data and potential for applying TORUS across tasks.

Q1: In what sense is track lifecycle management decoupled from classification, given the fused score?

We agree that our wording overstated this. The decoupling acts on what the two heads learn, not on the decision rule. In a single-head query tracker, every unmatched query receives an all-zero class target, so the classifier must also learn whether a query is responsible for a visible object in the current frame. Under occlusion these goals conflict: a track query whose person is hidden behind a car must score car low, even though the region shows a car. The same holds for duplicate queries on one object.

TORUS applies the class loss to matched queries only, so the classifier learns semantics alone, while the class-agnostic confidence head learns occlusion, duplicates and background. The fused score s^=σ(s)⋅max⁡cσ(zc)\hat s=\sigma(s)\cdot\max_c \sigma(z_c) combines these two independently trained factors, with the class term stop-gradient, so lifecycle decisions never shape the classifier. Because it takes the maximum over classes, s^\hat s is nearly invariant to flips between similar classes; it drops only when no prompted class fits the object, which acts as an intended vocabulary gate. Letting the class score govern the track instead costs 7.4 Novel AssocA and 2.5 Novel LocA (Table 4).

As requested, we are running the comparison of lifecycle management (birth, memory update, deletion) on σ(s)\sigma(s) alone vs. on s^\hat s, with the same checkpoint. On TAO OVMOT we expect only a small difference: all 1,203 LVIS classes are prompted, so nearly every object matches some class, and detecting and tracking all objects, which the confidence head already does on its own, is exactly what this benchmark asks for. The class factor in the lifecycle matters when only a subset of classes is prompted, e.g. person on DanceTrack or the eight VisDrone classes, where confident objects of unprompted classes should not start tracks. We will revise the abstract and introduction accordingly, and state the fused decision rule explicitly after Eq. (3).

Q2: DanceTrack is in the training mixture but described as zero-shot.

The reviewer is right. DanceTrack is about 3% of the training samples (Table 12), as stated in Table 2 and Sec. 4.3; describing it as zero-shot in the abstract and introduction was incorrect, and we will correct it. The intended claim is that a single set of weights, without benchmark-specific fine-tuning or threshold tuning, is evaluated on all benchmarks. True zero-shot transfer applies only to VisDrone-MOT, whose domain no training set covers.

Q3: Does TORUS use trajectory-level majority voting on YouTube-VIS 2021?

Yes, this exception applies, and only to YouTube-VIS. Track-mAP assigns a single category to each video instance, so without aggregation every class change within a track would count as a new instance and fragment it, unlike for the compared methods, which all aggregate per instance. Tracking, masks and per-frame class prediction remain online on every benchmark: the class at frame tt uses only frames up to tt, through an EMA over class logits. Only to produce the YouTube-VIS output, after tracking, does each instance take the class with the highest summed per-frame score. All TAO, DanceTrack and VisDrone results use no trajectory-level post-processing. We will state this exception explicitly in Sec. 4.1 and the Table 2 caption.

Q4: Results from a single run.

We agree that seed variance should be reported. Evaluation is deterministic for a fixed checkpoint, so the variance comes from training alone. We are re-training the main TORUS-R50 model with additional seeds. Because of limited compute, repeating every ablation, including the Mamba vs. Transformer comparison (two full trainings per seed), is unlikely within the rebuttal period. The seed variance of the main model indicates the run-to-run variation to expect for the ablations, which share its training recipe, e.g. relative to the 4.8 AssA gap between Mamba and Transformer memory (Table 6).

AuthorsOpenReview ↗

Follow-up to Q1: lifecycle management with the confidence score alone

As promised in our first response, we compared lifecycle management (birth, memory update, deletion) on the confidence score σ(s)\sigma(s) alone against the fused score s^=σ(s)⋅max⁡cσ(zc)\hat s=\sigma(s)\cdot\max_c \sigma(z_c), with the same checkpoint and thresholds.

TAO valBase TETABase LocABase AssocABase ClsARare TETARare LocARare AssocARare ClsA
s^\hat s (fused)42.560.443.124.142.163.343.819.3
σ(s)\sigma(s) only42.961.843.223.841.963.742.519.4
DanceTrack valHOTAAssADetAMOTAIDF1FP
s^\hat s (fused)57.743.377.487.959.7~12k
σ(s)\sigma(s) only47.644.651.033.851.6~132k

The results match the expectation stated in our first response:

  • TAO. The two rules are within 0.4 TETA. All 1,203 LVIS classes are prompted, so nearly every object matches some class, and the class factor rarely changes a lifecycle decision. The confidence head alone already does what this benchmark asks: detect and track all objects.
  • DanceTrack. Only person is prompted. With the confidence score alone, every confidently detected object starts a track, whether it is a person or not. False positives increase about tenfold, DetA drops by 26.4 and MOTA from 87.9 to 33.8, while association is unaffected (AssA +1.3).

The class factor in the lifecycle is therefore not what keeps tracks alive; it acts as the vocabulary gate, and it is needed whenever only a subset of classes is prompted. We will add both comparisons to the ablations and state the fused decision rule explicitly.

We thank the reviewer again for the constructive review. As the discussion period ends, we summarize how each point is addressed in the revised manuscript.

Q1: Decoupling of lifecycle management and classification. We revised the abstract and introduction to state that the separation is in what the two heads learn, not in the decision rule, and we now state the fused decision rule explicitly after Eq. (3). The requested comparison of lifecycle management on the confidence score alone is in the appendix (Extended ablation study). With the same checkpoint and thresholds:

  • TAO: the confidence-only rule is within 0.4 TETA of the fused score (Base 42.9 vs. 42.5, Novel 41.9 vs. 42.1). All 1,203 classes are prompted there.
  • DanceTrack: only person is prompted. The confidence-only rule starts tracks for every confident object: false positives increase about tenfold, and HOTA drops from 57.7 to 47.6, while association is unaffected.

The confidence head therefore carries the track lifecycle on its own, and the class factor acts as a vocabulary gate.

Q2: DanceTrack described as zero-shot. Corrected in the abstract and introduction. The claim is now a single set of weights evaluated on all benchmarks, and only VisDrone-MOT is described as an unseen domain.

Q3: Majority voting on YouTube-VIS. The exception is now stated in Sec. 4.1, in addition to the Table 2 caption. All other results are fully online.

Q4: Results from a single run. Within the discussion period, we were not able to complete additional training runs with different seeds. We prioritized the analyses that test our central claims: the confidence-only lifecycle ablation, frame-level detection on TAO, and the detector-controlled comparison requested by Reviewer ijyu. Evaluation is deterministic for a fixed checkpoint, so the variance comes from training alone. The conclusions we draw from the ablations rest on margins well above the 0.5 TETA that the paper marks as inconclusive: 4.8 AssA for Mamba vs. Transformer memory, 7.4 Novel AssocA without the separate head, and 12.8 Novel ClsA without real video. The paper states the single-run limitation explicitly.

We hope these revisions address the reviewer's concerns and would be glad to clarify any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.