- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes TORUS, an online open-vocabulary multi-object tracker trained with heterogeneous data sources. It builds on a frozen MOBIUS open-vocabulary detector and introduces a Mamba-based temporal memory, a class-agnostic confidence head, and a spatial-semantic federated loss. The model is jointly trained on detection images, real videos with restricted vocabularies, and synthetic pseudo-videos, without using TAO annotations. Experiments are reported on TAO OVMOT, DanceTrack, YouTube-VIS 2021, and VisDrone-MOT.
Critical strengths
-
The idea of separating large-vocabulary semantic learning from temporal association learning is interesting and potentially useful. Using large-vocabulary images for semantics and restricted-vocabulary videos for temporal modeling is a reasonable way to reduce the dependence on densely annotated open-vocabulary tracking videos.
-
The paper evaluates the framework across several tracking and video instance segmentation benchmarks, and the reported numerical results are strong.
Critical weaknesses
- The most serious issue is the fairness of the main experimental comparison. TORUS is built on the frozen MOBIUS detector, whose open-vocabulary detection capability is substantially stronger than the detectors used by most competing OVMOT methods in Table 1. This makes the reported TETA comparison fundamentally difficult to interpret. A stronger detector directly improves localization and classification quality, and therefore naturally benefits LocA and ClsA. More importantly, better detection quality, recall, and localization also make temporal association considerably easier, so AssocA is affected as well. Therefore, the gains in LocA, ClsA, AssocA, and consequently TETA cannot be cleanly attributed to the proposed tracking framework. Comparing complete systems with substantially different detector capabilities does not convincingly demonstrate that TORUS is a better tracker. A meaningful comparison should control the detector/backbone and preferably the training data, or provide detector-matched baselines using the same MOBIUS detector. Without such controlled comparisons, the central state-of-the-art claim is not sufficiently supported. This is my main reason for rejection.
- Several generalization claims are overstated or not supported by the evaluation protocol. The paper repeatedly describes the DanceTrack result as zero-shot generalization, although DanceTrack is explicitly included in the training mixture (about 3% of the samples). The TAO Novel categories were also already observed by the frozen MOBIUS detector during LVIS pretraining, so the Novel results primarily measure retention of pretrained semantic knowledge rather than genuine unseen-category generalization. Similarly, on YouTube-VIS, TORUS has seen the novel classes through LVIS and has been trained with video masks from other datasets, which makes the comparison with existing zero-shot/open-vocabulary methods difficult to interpret. In addition, the main TAO OVMOT evaluation is reported only on the validation set, while test-set results are missing. Together, these issues substantially weaken the claimed cross-dataset and open-vocabulary generalization.
Questions for authors
1.Can the authors provide a detector-controlled comparison in which TORUS and the main competing tracking methods use the same MOBIUS detector, or otherwise control the detector/backbone and training data? Without such an experiment, how can the improvements in LocA, ClsA, and especially AssocA be attributed to the tracking method rather than to the much stronger detector?
2.Why is DanceTrack described as a zero-shot evaluation when it is included in the training mixture? Please also clarify the intended meaning of “zero-shot” for TAO Novel classes, since these categories have already been seen by MOBIUS during LVIS pretraining.
3.Why are TAO OVMOT results reported only on the validation set? Please provide the corresponding test-set results.
4.The paper claims that the class-agnostic confidence head decouples track lifecycle management from classification, but track birth, update, and deletion are still based on a fused score containing the maximum classification score. What exactly is being decoupled in this formulation?
5 comments
Thank you for the response. However, my main concerns remain insufficiently addressed.
Q1. Detector-controlled comparison. The MOBIUS+MinVIS comparison is conducted under the standard closed-set TAO protocol rather than the main OVMOT setting. More importantly, although both methods share the frozen MOBIUS encoder, their LocA differs substantially (48.4 vs. 60.6). Therefore, attributing the +5.0 AssocA improvement solely to the tracking framework is not sufficiently justified, as differences in detection and localization quality can also affect association performance. The proposed OVTR-detector experiment would be relevant, but no results have been provided.
Q2. Zero-shot generalization. I appreciate the authors' acknowledgment of the incorrect zero-shot claims. However, the current manuscript remains unchanged, and these claims are still present in the abstract and introduction. The promised corrections have therefore not yet been incorporated.
Q3. TAO test-set evaluation. While validation-set evaluation is common, several representative OVMOT methods, including OVTrack, COVTrack, and VOVTrack, also report test-set results. Thus, the explanation that validation is the standard evaluation split does not sufficiently justify the absence of test-set results, particularly given the paper's generalization claims.
Q4. Confidence-head decoupling. The response clarifies the separation of the training objectives, but track birth, update, and deletion still depend on the fused confidence-classification score. The proposed confidence-only ablation has not yet been provided, so the concern remains unresolved.
Overall, the response provides useful clarifications but no new experimental evidence directly addressing my central concern about the fairness and attribution of the main OVMOT improvements. The proposed additional experiments and manuscript corrections remain pending. Therefore, my assessment and recommendation remain unchanged at this stage.
We thank the reviewer for the prompt follow-up.
Q1: Detector-controlled comparison.
We agree that the MOBIUS+MinVIS comparison is closed-set only. On the LocA gap, the two methods do not differ in their per-frame detector. Both use the same frozen backbone and encoder. TORUS fine-tunes the decoder, which makes its frame-level detection slightly worse than MOBIUS on COCO and LVIS (−1.7 and −2.3 AP, Table 4). The higher LocA of TORUS therefore does not come from a stronger detector. It comes from the tracking framework: track queries carry each object's state into the next frame and act as priors for detection.
The TAO detection experiment in our response to Reviewer zMiV (W2) isolates this effect. With the same TORUS checkpoint, frame-level box AP on the annotated TAO val frames is 20.5 when each frame is processed independently and 33.6 when the track queries of previous frames are used (+13.1 AP, +14.0 AP$_r$).
We agree that better localization also eases association, so the AssocA gain cannot be separated completely from the LocA gain. However, both gains come from the same component: propagating track queries through the detector is how TORUS associates, and it is also what improves detection. We consider this joint improvement part of the tracking contribution, not a detector advantage. We will make this argument and the TAO detection results part of the detector-controlled comparison in the final revision.
The OVTR-detector experiment is in progress. We will share its results if it completes within the discussion period.
Q2: Zero-shot claims in the manuscript.
We have uploaded a revision that corrects the zero-shot claims in the abstract and introduction, and states the YouTube-VIS majority-voting exception (Sec. 4.1) and the fused decision rule after Eq. (3). Please note that we are likely going to make further changes to the manuscript, such as adding any new ablations, until the end of the discussion period.
Q3: TAO test-set evaluation.
We acknowledge that OVTrack, VOVTrack and COVTrack also report test-set results. We are working on a test-set evaluation, but cannot guarantee that it will complete within the discussion period.
Q4: Confidence-only lifecycle ablation.
The ablation is now complete; please see our follow-up to Reviewer JzTQ (Q1) for the full tables. With the same checkpoint, lifecycle management on the confidence score alone is within 0.4 TETA of the fused score on TAO (Base 42.9 vs. 42.5, Rare 41.9 vs. 42.1), where all classes are prompted. On DanceTrack, where only person is prompted, it starts tracks for every confident object: false positives increase about tenfold and HOTA drops from 57.7 to 47.6, with association unaffected (AssA 44.6 vs. 43.3). The confidence head thus carries the track lifecycle on its own, and the class factor acts as a vocabulary gate.
We are happy that the reviewer finds separating semantic from temporal learning a useful way to reduce the dependence on densely annotated tracking videos, and we thank them for the detailed review.
Q1: Detector-controlled comparison.
We agree that Table 1 compares complete systems whose detectors differ in strength and pretraining data; we will state this explicitly and add the detector and its pretraining data to Table 1. The paper does contain a detector-matched baseline (Table 5, standard TAO), which we will make more prominent: associating the detections of the same frozen MOBIUS detector with MinVIS, instead of our tracker.
Two observations address the question about AssocA. First, with the detector held fixed, the tracker accounts for +6.6 TETA and +5.0 AssocA. Second, the stronger detector alone does not yield better association: MOBIUS with MinVIS reaches 38.0 AssocA, below COVTrack (41.1) and SLAck (38.9), and its TETA (36.2) is on par with COVTrack (36.9).
For scale and data: TORUS-R50 has a parameter count comparable to OVTR and COVTrack (71.3M vs. 58.8M and 74.4M, Table 10) and uses 120k tracking samples and no TAO annotation, whereas COVTrack trains on ~4.9M samples of densified TAO annotations and VOVTrack on unlabeled TAO videos.
We recognize that this does not fully control the detector and training data for every baseline. We are therefore making an effort to re-train TORUS on top of the OVTR detector, using OVTR's released weights, CLIP text embeddings and LVIS-base data. In this setting, only the tracker differs from OVTR, and the Novel classes are genuinely unseen.
Q2: Meaning of "zero-shot" for DanceTrack and TAO Novel.
We agree that our use of "zero-shot" was inconsistent. DanceTrack is part of the training mixture (about 3% of samples, Table 12), so calling it zero-shot was incorrect, and we will correct the abstract and introduction (see also our response to JzTQ). The intended claim is a single set of weights evaluated on all benchmarks without benchmark-specific fine-tuning; true zero-shot transfer applies only to VisDrone-MOT.
For TAO Novel, we agree the classes are not unseen. As stated in the Table 1 footnote, Sec. 4.2 and the Limitations, Novel scores measure how well the detector's knowledge of these classes is retained through tracking training, not zero-shot discovery. Our first contribution used "zero-shot to unseen categories" inconsistently with this, and we will correct it. The same holds for YouTube-VIS, where, as noted in the Table 2 caption, only the base-class mAP compares directly with OV2Seg and OVFormer. For a setting in which Novel classes are genuinely unseen, see the OVTR-detector experiment above.
Q3: Why are TAO OVMOT results reported only on the validation set?
Validation is the standard evaluation split in this field and the one on which every compared method reports results. The TETA protocol of TETer, which defines the standard TAO benchmark, is evaluated on the validation set; OVMOT works since OVTrack report their main results and ablations on it; and some baselines, such as MASA, report validation results only. Validation is therefore the only split that allows a complete comparison in Tables 1 and 5. With limited compute, we have prioritized the detector-controlled experiments above; test-set evaluation requires no retraining, and we aim to add it if time permits.
Q4: What exactly is decoupled?
Reviewer JzTQ raised the same point; please see our response to their Q1. In short, the decoupling concerns what each head learns: the classifier is trained on matched queries only and learns semantics, the confidence head learns occlusion, duplicates and background, and the stop-gradient fused score with a maximum over classes keeps tracks alive through class flips while gating out objects outside the prompted vocabulary. We are running the requested comparison with the confidence score alone; as explained there, we expect only a small difference on TAO, where all classes are prompted, and the class factor to matter when only a subset of classes is prompted.