acceptodds
Under review as a conference paper at ICLR 2027

VastMAT: A Large-Scale Multi-Category Benchmark for Multi-Animal Tracking

Abstract

Multi-animal tracking (MAT) supports the study of animal movement, behavior, and group interactions. However, general multi-object tracking (MOT) benchmarks primarily focus on pedestrians and vehicles, whereas dedicated MAT benchmarks remain limited in jointly supporting broad animal coverage, large-scale video data, and extensive within-video multi-instance association. To address this gap, we introduce VastMAT, which has four key characteristics: (1) Large scale. It comprises 2,947 videos with 1,002,562 annotated frames, totaling 27.85 hours. (2) Broad category coverage. These videos cover 337 animal categories with diverse morphologies and motion patterns. (3) Extensive instance annotations. It provides 3,663,248 bounding boxes and 22,883 identity trajectories—to our knowledge, the largest numbers of both among dedicated MAT benchmarks. (4) High-quality annotations. To ensure reliability, annotations undergo iterative expert review and correction, and quality is assessed through an independent reannotation audit. To systematically assess tracking performance and cross-category generalization, we establish Seen-category and category-disjoint Unseen-category protocols, and evaluate eight representative MOT methods under both protocols. Under these protocols, the highest baseline HOTA scores are 66.37% and 52.90%, respectively, highlighting the challenge of tracking unseen animals. To address the low-overlap association challenge revealed by our analysis, we propose Center-Distance-Augmented Association (CDA), a lightweight module that adaptively combines IoU with center similarity normalized by the boxes' own scales. Without additional training, CDA improves TrackTrack's HOTA by 1.58 and 1.31 percentage points under the two protocols, respectively. To facilitate further MAT research, we will publicly release our benchmark and code.

Primary area:
datasets and benchmarks
Keywords:
Multi-animal tracking, Multi-object tracking, Large-scale tracking benchmark, Animal tracking dataset, Cross-species generalization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ceZhOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes a new benchmark for MAT. This benchmark is featured by the large scale of video and frame numbers, broad animal category coverage, extensive animal instance annotations, and high quality annotations. They build seen-category and unseen-category protocols for evaluation. They propose CDA to address the low-overlap association challenge, which combines IoU with center similarity normalized by the box scales. Multiple MOT methods are evaluated on the benchmark.

Critical strengths
  1. This is a large and diverse dataset for MAT, especially on the number of bounding boxes and identity trajectories. Although SAM3 is used for initial labeling, expert review and correction are conducted, and quality is assessed through the independent audit. This improves the annotation quality.
  2. They establish seen and unseeen category protocols, and evaluate multiple MOT methods under both protocols. These comprehensive experiments are good baselines for following research.
Critical weaknesses

The tracking performance degeneration on unseen-category protocol needs furthey analysis. It's unclear whether the main bottleneck is the detection or the association stage. One more experiment where Ground Truth detections are provided should be conducted. Then we can check the association performance under such situation and get a more clear understanding of the tracking performance degeneration.

Questions for authors

Please check the weakness part.

2 comments
AuthorsOpenReview ↗

**Response to Reviewer ceZh**

Thank you for suggesting the GT-detection control.

Detection versus association. We evaluated TrackTrack and DiffMOT with normal detections and GT boxes under both protocols. Within each pair, the tracker, checkpoint, and association settings are fixed. GT boxes use one foreground class and confidence 1, without GT identities; appearance features are recomputed from the corresponding crops.

Normal → GT detections; scores in %.

TrackerProtocolHOTADetAAssA
TrackTrackP165.10 → 85.7165.08 → 95.1665.26 → 77.20
TrackTrackP252.61 → 86.1448.75 → 95.4056.97 → 77.78
DiffMOTP166.37 → 85.9865.97 → 92.4366.89 → 79.98
DiffMOTP252.85 → 84.6048.66 → 89.7657.54 → 79.76

The P1–P2 HOTA gap shrinks from 12.49 to −0.43 points for TrackTrack and 13.52 to 1.38 points for DiffMOT. Their AssA gaps also shrink from 8.29/9.35 to −0.57/0.22 points. This identifies detection quality as the dominant contributor to the observed protocol gap for these trackers.

The table reports paired diagnostic runs. DiffMOT's normal P2 score is 52.852 for the checkpoint used in this control; its original Table 5 result remains 52.90. We will include the control in the revised manuscript.

Reviewer_ceZhOpenReview ↗

Post Rebuttal Comments

The author's rebuttal has addressed my concerns, and I will maintain the positive rating.

Reviewer_qi2KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This article proposes the multi animal tracking benchmark VastMAT and provides frame by frame bounding boxes and intra video identity annotation. The author also conducted detection, correlation, and video difficulty analysis on eight tracking methods. In addition, the author proposes a CDA module that does not require additional training, improves correlation by fusing IoU with normalized center distance at the target scale, and adjusts fusion weights based on the number of high confidence detections.

Critical strengths
  1. Data resources have clear value. VastMAT combines a wide range of animal categories with frame by frame multi instance identity annotation, which can supplement the shortcomings of existing animal tracking benchmarks in terms of category diversity and video size.
  2. The verification is comprehensive and some limitations have been disclosed. The author provided independent re annotation audits, animal population analysis, and video by video results. The CDA experiment includes geometric controls with shared gating, different trackers, and sensitivity analysis of post-processing.
Critical weaknesses
  1. The submitted annotations have specific identity or category inconsistencies, and the current audit is insufficient to verify the reliability of the complete trajectory. There are multiple videos with incorrect animal categories in the GT annotation in the supplementary file, for example: ID 11 of 'Pig-24' is first labeled as Pig and then as Elephant; `The ID 29 of Buffalo-2 varies between Buffalo and Heron, and so on. At the very least, it needs to be clarified whether these situations are due to video editing errors or missing annotations. At the same time, each segment only has 20 frames, which is about 2 seconds at the 10 FPS mentioned in the article. Its identity consistency indicator checks whether an original ID is always mapped to the same re annotated ID, and cannot fully exclude trajectory fragmentation of multiple original IDs corresponding to the same animal. Therefore, 99.30% of intra segment consistency should not be interpreted as long-term occlusion, reproduction, and identity continuity being verified in the entire video.
  2. VastMAT uses different annotation frame rates compared to other benchmarks, directly comparing the displacement and IoU of adjacent annotation frames, which mixes time intervals with real motion differences, such as in the TAO dataset. The complete training configuration and pre training sources for the baseline have not yet been provided. For papers that focus on evaluating protocols as their core contribution, this information is crucial.
Questions for authors
  1. Is the above cross category trajectory caused by category errors, ID reuse, or clipping boundaries? Please explain the specific processing results and evaluate whether a different cleaning version than the submitted GT was used. `How do class_id=-1 'and zero area boxes participate in training, category statistics, and TrackEval evaluation?
  2. Is there a complete trajectory or statistical result of occlusion reproduction available? How does the existing audit handle the situation where multiple original IDs correspond to the same re annotated ID? Please clarify the range of conclusions that can be supported by the 99.30% consistency indicator.
  3. Have all source videos undergone uniform frame rate conversion? Please provide the actual time sampling rules, frame rate recording for each video, and the corresponding table of original video ID and clip interval. Is the training/testing division isolated based on the original video or overlapping segments?
  4. What pre training data and training configurations are used for each of the eight baselines? Does' no category seen 'only refer to those who did not participate in VastMAT task training, without excluding exposure to these animals during the pre training phase?
  5. In Table 6, the pure IoU configuration under shared gating has reached 53.56 HOTA, with fixed fusion at 53.83 and adaptive fusion at 53.92; The adaptive relative fixed fusion in Protocol 1 only increases by 0.02. Please distinguish the contributions of gate control/geometric item replacement, center distance, and adaptive detection quantity, and explain whether these minor differences are stable in the existing video by video results.
3 comments
AuthorsOpenReview ↗

**Response to Reviewer qi2K**

Thank you for the detailed questions.

Q1. Annotation corrections and label handling.

Pig-24/ID11 involved ID reuse after a 36-frame gap and an erroneous Elephant label. Buffalo-2/ID29 contained a merged buffalo/heron box with inconsistent category labels. Both have been corrected in 2947GT/.

Reevaluating 71 stored prediction configurations before and after the test-GT patch changes HOTA by +0.002–+0.085 points (mean +0.0125), preserving the main conclusions. The reported models retain their original training history.

Category statistics use corrected semantic GT, before single-class conversion. Our converter maps valid animals to foreground class 1 and unresolved −1 labels to distractor class 8 for unmodified TrackEval. Corrected test GT contains no −1 labels. The inspected P1 training export contained no −1 labels and 12 nonpositive-area boxes among 3,389,474 records. TransTrack/ByteTrack filter these boxes; MOTIP/TCEI readers and DiffMOT label generation do not apply that filter. Corrected GT removes them.

Q2. Identity audit. On 96 twenty-frame clips from 48 videos, forward consistency is 424/427=99.30% and reverse consistency 420/425=98.82%. The reverse check identifies five potential fragmentation cases. These are within-clip identity-mapping statistics.

Five training-sequence examples illustrate ID continuity across 25–50 intermediate frames without boxes. Before/during/after figures are in long_term_occlusion_cases/.

Q3. Sampling and source separation. VastMAT annotations use a uniform 10 FPS timeline: adjacent GT frame indices represent 0.1 s. Table 2 uses gap=1 on the 266-video P2-only subset (278,138 pairs); tracking evaluation uses all 315 test videos per protocol. We will clarify Figure 3's sampling intervals and interpretation.

Clips from the same source use temporally disjoint intervals. The provenance table documents 33 shared sources, 36 P1 test clips, and 66 P1 training clips; all 78 listed cross-split pairs have zero interval overlap. There is no data leakage across protocols. The source/interval records are in source_provenance_and_intervals.csv.

Q4. Baseline configuration. The eight methods use the following initialization/components and training schedules:

MethodInitialization/componentsVastMAT training
MOTIPMOTIP-release DETR*~42k iterations
TCEIMOTIP-release DETR*~42k iterations
TransTrackMOTIP-release DETR*20 epochs
ByteTrackCOCO-initialized YOLOX-X20 epochs (~379k iterations)
TrackTrackShared YOLOX-X; official ReID/AFLinkShared detector; no additional association training
DiffMOTShared YOLOX-X; DanceTrack-trained motion; official ReIDShared detector
QDTrackOfficial Faster R-CNN COCO weights20 epochs; LR decay at 15
FairMOTOfficial pre-trained DLA-34 weights20 epochs; LR decay at 15; batch 14

*Recorded DETR init: Official r50_deformable_detr COCO weights (with DanceTrack fine-tuning). "Unseen" refers to VastMAT task training, not external pretraining. GT-detection isolates association performance. Per-protocol configs are in the appendix.

Q5. Attribution and stability. In the main P2 table, fixed CDA adds +0.27 HOTA over pure IoU under the shared gate; adaptation adds +0.09. The additional matched control separates score and gate:

Score / gateHOTA
HMIoU / original52.352
HMIoU / DIoU52.157
IoU / DIoU53.447
Fixed CDA / DIoU53.656
Adaptive CDA / DIoU53.716

Within this control, gating contributes −0.195, score replacement +1.290, fixed fusion +0.209, and adaptation +0.060 points.

With 12,000 paired video-bootstrap resamples, the 95% intervals for mean per-video HOTA differences are [−0.21,+1.00] points for fixed−IoU (P2), and [−0.27,+0.45]/[−0.19,+0.38] for adaptive−fixed (P1/P2). They include zero. Adaptation trades a small pooled HOTA increase for 112 additional P2 ID switches.

The author's supplementary response and newly added experiments have addressed some of the questions regarding data annotation, evaluation protocols, and baseline configuration. However, evidence of long-term identity continuity is still limited, and the actual gain and statistical stability of the CDA module have not been fully demonstrated. Therefore, I tend to maintain the original score (2: Weak Reject).

AuthorsOpenReview ↗

Response to Reviewer qi2K

Thank you for the follow-up comments and for noting that the annotation fixes, protocol details, and baseline configurations are now clear. On the two remaining points, we would like to clarify the exact scope of what our evidence supports:

1. Identity continuity and audit scope. You raise a fair point about what the 20-frame audit covers. The forward (424/427=99.30%424/427 = 99.30\%) and reverse (420/425=98.82%420/425 = 98.82\%) checks are strictly within-clip statistics over 96 sampled clips (Sec. 3.3, App. C.2, Table 13). We added the reverse check specifically to catch fragmentation where multiple original IDs map to a single reannotated ID, and it flagged five cases. We do not treat these numbers as a statistical guarantee across entire videos.

Instead, whole-sequence continuity (mean 34.02 s34.02\text{ s}, Table 1) is enforced by the annotation workflow itself: multi-round continuous-playback review by a 2–3-expert team requiring unanimous sign-off, where any suspected drift or identity switch is traced along the entire trajectory across entry, exit, and occlusion (App. C.2). The independent audit serves as a spot check on top of this pipeline, and the five sequences in long_term_occlusion_cases/ provide qualitative examples showing how reappearance across 25–50-frame (2.5–5 s2.5\text{--}5\text{ s}) gaps is handled. This scope is already noted in Sec. 3.3 and App. C.2, and we will also state it explicitly in the Limitations section for the camera-ready version, alongside releasing our audit lists and checking scripts.

2. CDA attribution and stability. Under the shared normalized-DIoU gate on Protocol 2, fixed CDA (α=0.25\alpha=0.25) changes HOTA from 53.5653.56 to 53.8353.83 (+0.27+0.27) and IDSW from 2,0822,082 to 2,0542,054 (−28-28) compared with pure IoU (Table 6), with the matched control in App. E.4 separating gating (−0.195-0.195), HMIoU→IoU\text{HMIoU}\to\text{IoU} (+1.290+1.290), and fixed fusion (+0.209+0.209). The paired bootstrap 95% CI for this fusion-only increment is [−0.21,+1.00][-0.21, +1.00] and includes zero; we therefore do not claim video-level significance for it, and the camera-ready will state this explicitly.

What we mainly rely on is the consistent improvement over the original trackers: fixed CDA improves TrackTrack by +1.22+1.22 HOTA / −255-255 IDSW on P2 (Table 8) and +1.56+1.56 / −300-300 on P1 (Table 27), improves ByteTrack by +1.36+1.36 / −823-823 (Table 8), and works with or without AFLink (Table 24); the adaptive variant likewise exceeds the baseline in all nine density-grid configurations (53.36–53.9253.36\text{--}53.92 vs. 52.6152.61, Table 25). The offline diagnostic over 190,890190,890 events (Table 26) also shows that CenterSim alone correctly distinguishes 11.01%11.01\% vs. 4.83%4.83\% (2.3×2.3\times) of pairs at displacements 0.5–1.00.5\text{--}1.0, and 5.94%5.94\% vs. 2.31%2.31\% (2.6×2.6\times) above 1.01.0. The adaptive trade-off (+0.09+0.09 HOTA, +112+112 IDSW; Colugo-10 failure case, App. E.4) is already reported as a limitation.

3. Overall. Our core contribution is the VastMAT benchmark (2,9472,947 videos, 337337 categories, 3.66M3.66\text{M} boxes, 22,89522,895 trajectories, Seen/Unseen protocols, and eight baselines), whose value and verification you noted positively. The annotation issues you spotted have been fixed—changing HOTA by only +0.002–+0.085+0.002\text{--}+0.085 (mean +0.0125+0.0125) across 71 stored runs (rebuttal Q1)—leaving all conclusions unchanged. With the specific annotation errors corrected and the scope of both the audit and CDA claims clearly bounded, we hope the remaining concerns can be viewed as disclosed limitations, and we would be grateful for your reconsideration.

Reviewer_A4LGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a dataset for multi-object tracking of animals. It claims to be the biggest animal tracking dataset with a total of 337 animal categories across 2.9k videos at an annotation fps of 10. They propose two different benchmarking protocols based on which categories lie in the training or test split, and propose a lightweight architecture-agnostic module to improve animal tracking across baselines.

Critical strengths
  • The proposed dataset has many more classes than any previous animal multi-object tracking dataset
  • The benchmarking of prior methods on the proposed dataset is solid
  • Dataset statistics are thoroughly studied across all categories and annotation protocol is detailed
Critical weaknesses
  • Why does the CDA module not introduce more ID switches in the tracking of nearby animals? If the scene has multiple animals of the same category their nearby bounding boxes are going to look the same, so I’m kind of confused how IDF1 and IDSW improved with CDA module. Where does CDA fail?
  • from Table 4, the trends of all baselines are similar in protocol 1 and protocol 2 (except performance is lower), it would be nice to point out specific takeaways between the two protocols’ evaluation and how they test different things
  • when measuring the quality of the track annotations, could there be an agreement rate based uncertainty field added to the annotations, which signifies how confident the annotators were. These could be useful for downstream methods.
  • how do the tracks look like when there is occlusion in the scene?
  • are all animals in a scene/video always annotated? If not, are there negative categories for federated training?
Questions for authors

see above

1 comment
AuthorsOpenReview ↗

**Response to Reviewer A4LG**

Thank you for the constructive questions.

Q1. Nearby animals and CDA. CDA combines scale-normalized center distance with IoU, providing an additional geometric cue when displacement reduces overlap between consecutive boxes. It retains the base tracker's other association cues.

P2 configurationHOTA ↑IDF1 ↑IDSW ↓
TrackTrack52.6154.922309
Pure IoU / shared gate53.5656.282082
Fixed CDA53.8356.892054
Adaptive CDA53.9257.062166

Fixed CDA improves HOTA by 1.22 points and reduces IDSW by 255 relative to TrackTrack. Under the same gate, its incremental gain over pure IoU is 0.27 HOTA, with 28 fewer ID switches.

Adaptive weighting adds 0.09 pooled HOTA over fixed weighting but introduces 112 more ID switches. In Colugo-10, adaptive versus fixed gives HOTA 22.77 versus 26.79 and IDSW 16 versus 4. We will use this example to explain the local-ambiguity failure and distinguish fixed fusion from the adaptive weighting trade-off.

Q2. Protocols. P1 evaluates categories represented in VastMAT training; P2 evaluates categories excluded from that training. GT-detection controls reduce TrackTrack's P1–P2 HOTA gap from 12.49 to −0.43, and DiffMOT's from 13.52 to 1.38 points. Detection quality is a major contributor to the observed gap. “Unseen” is defined relative to VastMAT task training.

Q3. Uncertainty. Independent reannotation yields forward/reverse agreement of 99.30%/98.82% within 96 sampled twenty-frame clips. These are sample-level quality statistics. Per-box agreement-based uncertainty is a useful extension requiring additional repeated annotations; the current release provides the audit results.

Q4. Occlusion. Boxes are omitted during fully occluded or out-of-view intervals. The previous ID is retained when the animal reappears and its identity is established. Five training-sequence examples in long_term_occlusion_cases/ illustrate this rule across 25–50 intermediate frames without boxes.

Q5. Coverage. Expert annotators label visible animals whose category and identity can be reliably established. Targets too small or ambiguous for these judgments are excluded, without a fixed pixel-size threshold. Excluded targets have no explicit ignore masks. The release uses single-foreground-class evaluation and does not provide verified-negative category labels for federated training.

These clarifications will be incorporated into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.