acceptodds
Under review as a conference paper at ICLR 2027

The Generator Is the Tracker: Multi-Object Tracking by Painting Persistent Identity Colours

Abstract

Multi-object tracking (MOT) is conventionally decomposed into detection followed by association, with object identity maintained as external state: track buffers, motion models, appearance embeddings. We ask whether a video generator can maintain that state in pixels. We fine-tune a 22B text-to-video diffusion model (LTX-2.3) with a lightweight in-context LoRA to translate an RGB clip into an ID-map clip, a video in which every person is painted a flat, distinct colour that persists over time (same colour, same identity). Long videos are generated as chained windows, each conditioned on the cleaned tail of the previous one; a brief continuation fine-tune teaches the model to extend a given colouring, after which identity flows through the chain with no separately trained detector, no motion model, and no association module. On the DanceTrack test server the system reaches 40.3 HOTA, to our knowledge the first entry whose tracks are read directly off video produced by a pretrained video generator. This is well below today's specialists (66–76 HOTA), but with an error profile no baseline shares: its association score (AssA 44.1) exceeds every tracker of the 2020–22 benchmark suite while detection is the deficit. Controlled comparisons on the validation split show where the identity comes from: windows generated without generation-time state and linked by post-hoc association score 18.2 HOTA against 34.8 for the chain, while re-linking the chain's own windows post hoc changes nothing (34.7), so the colours already carry what overlap association can recover, and motion- and appearance-based associators run on the system's own boxes reach only 22–24 AssA, an ordering that holds again on SportsMOT validation. On 383 mined occlusion events the chained generator re-acquires 42% of the identities it held on both sides of the gap, including gaps longer than its 49-frame context, by keeping the colour alive in pixels through the gap. We release code, checkpoints, and the full pinned experiment log.

Primary area:
generative models
Keywords:
multi-object tracking, video diffusion models, generative models as perception, video-to-video translation, in-context LoRA, identity persistence, occlusion re-acquisition, DanceTrack, SportsMOT

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HXxbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper formulates multi-object tracking as generating videos with persistent identity colours, from which trajectories are decoded. A video generator is adapted through prefix continuation training and chained inference. The main analysis examines the source of identity information, recovery after occlusion and failures in crowded scenes.

Critical strengths
  1. The tracking representation is worth exploring. Encoding identities as persistent colours makes the video generator responsible for both localization and identity preservation, offering a useful alternative formulation to conventional detection-and-association pipelines.
  2. The training design responds to an observed failure. Without continuation training, the model can overwrite the imposed colouring after the overlap. Prefix training addresses this behavior, connecting the proposed intervention to a concrete failure mode.
  3. The controls examine different sources of identity information. The paper compares independent and chained generation, re-links identical generated windows, and runs motion and appearance association on identical detections. These controls help assess the respective roles of generated colours and post-processing. (Table 2)
  4. The evaluation reports useful limitations. Alongside the DanceTrack test result, the paper includes SportsMOT, seed variation, crowded-scene failures and runtime. The unsuccessful transfer of long-gap recovery to SportsMOT is particularly informative about the method’s scope.
Critical weaknesses
  1. The comparison does not isolate generation-time state. Both the checkpoint and generation mode change, while DetA increases from 19.7 to 32.3. Although the paper acknowledges these differences, its subsequent interpretation still attributes the gap rather directly to prefix state. The same-window and identical-detection controls are useful, but do not fully separate additional training, prefix conditioning and changes in detection quality.
  2. The explanation of long-gap identity recovery remains stronger than the evidence. The original colour appears in at least one gap frame for successful recoveries. This is consistent with carry-over, but does not establish propagation through every intervening conditioning prefix or rule out colour reuse in unrelated regions.
  3. The cause of crowded-scene failure is not sufficiently isolated. The paper attributes the failure to latent-token capacity and identifies resolution-matched training as the remedy. However, the 768² experiment also changes the input distribution and reduces detection recall. The results show density-related degradation, but do not distinguish representation capacity from training coverage or resolution adaptation.
Questions for authors
  1. Can the existing results further separate checkpoint changes, prefix conditioning and detection-quality changes in the A/B comparison?
  2. In the seven successful recoveries beyond the context length, does the original colour pass through every intervening prefix, and which regions carry it during the gap?
  3. What evidence specifically distinguishes a token-capacity limit from insufficient training coverage or resolution adaptation?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 3 (1/2)

Q1. Separating checkpoint, prefix and detection. The full 2x2 factorial (Table R1, in our response to R1) varies one factor at a time on identical windows. Detection is separated inside each cell: the Kalman/IoU and DINOv3 rows re-associate that cell's own boxes, so the colour-vs-associator gap at fixed DetA is measured in all four cells.

  • Checkpoint alone does nothing: D-B -1.8 HOTA [-5.1, 0.9]. Chaining alone does nothing: C-B -1.2 [-3.8, 1.1]. Together they add +18.9 over the same checkpoint unchained (A-D, [15.2, 22.9]) and +18.3 over the same chain without prefix training (A-C, [14.9, 22.0]).
  • Detection. DetA under each cell's native link is 32.3 (A), 28.7 (C), 17.4 (D) and 19.7 (B). The low DetA of the independent cells comes from the post-hoc associator's union boxes. Under the raw-colour link D keeps its detections (DetA 33.1 vs 32.3 for A), and A still leads D by +19.8 HOTA [16.3, 23.6] and +29.9 AssA [24.9, 35.3], in all 25 videos.
  • Identical boxes. The colours lead Kalman/IoU and the DINOv3 bank by +15.3 and +12.5 AssA in A, +4.4 and +3.9 in D and +4.2 and +3.1 in B, and trail them by -0.5 and -2.7 in C.

So the A-B gap is generation-time state, present only with the checkpoint trained to continue it, and not detection: with detection equalised (raw-colour link) the A-D gap is as large as under the native links (+19.8 vs +18.9 HOTA). We revised Section 4.2 so the interpretation follows the factorial, not the A/B pair.

Q2. The seven recoveries beyond the context. Per event (gap length; fraction of gap frames with the colour; colour frames in each intervening 9-frame prefix; IoU with the interpolated position and area vs last visible box, over the frames with the colour):

EventGapPresentPrefix framesIoU interp.Area
0010 / id 31021.009, 90.621.44x
0014 / id 11711.0090.581.04x
0073 / id 7501.009, 90.411.79x
0081 / id 191300.889, 6, 30.641.04x
0034 / id 10830.759, 90.204.35x
0014 / id 8540.5720.251.52x
0034 / id 12830.489, 00.072.07x

Four of the seven keep the colour at the hidden person's location (IoU 0.41-0.64, 1.0-1.8x its size) through every prefix; three fill all nine prefix frames and 0081 / id 19 thins to 6 and then 3 of 9. Two keep the colour in every prefix but poorly located: 0034 / id 10 is enlarged 4.35x at IoU 0.20, and 0014 / id 8 appears in only 2 of 9 prefix frames at IoU 0.25. One (0034 / id 12) loses it in a prefix and re-acquires anyway. The two 0034 events carry an enlarged, poorly located colour (area 2.1-4.4x, IoU <= 0.20) that we cannot distinguish from reuse. So propagation through every prefix holds for 6 of 7, and clearly located carry-over for 4 of 7. The revision keeps the count that all 7 show the colour in at least one gap frame, no longer reads it as propagation, and adds this table (Table 7). Over all 38 recoveries whose gap spans a window boundary, the colour is in every intervening prefix in 37.

AuthorsOpenReview ↗

Response to Reviewer 3 (2/2)

Q3. Capacity versus training coverage versus resolution. You are right that the 768^2 probe cannot separate these, and a new probe shows our capacity claim was too strong; we withdraw it. Coverage first: of the 779 49-frame windows we count in the training annotations, 95 average >=14 people per frame and 65 average >=20 (up to 40), so crowds are in training, although the 14-19 band is thinner than in validation (3.7% vs 11.1% of frames). We then rendered 12 training clips that the translation-only checkpoint was fit on, as single windows at the training resolution, with a reading rule fixed before the run:

Training clips (selected by annotated people / frame)GT regions / framePainted regions / frame (max)Mask recall @ IoU 0.5
>=20 people (4 clips)22.9-37.415.7-23.8 (28)0.09-0.30
14-19 people (4 clips)13.9-16.08.5-12.9 (15)0.20-0.35
4-9 people (4 clips)6.0-7.87.4-9.5 (11)0.49-0.94

(GT regions are decoded ground-truth ID-map regions with the same minimum area, which drops small or occluded people, so they can fall below the annotated band.)

Two conclusions follow directly. (a) The model is not limited to 10-12 instances: on seen crowds it paints up to 28 regions per frame, so the count ceiling we reported on validation is not a property of the model at 512^2, and "token-capacity-bound" is withdrawn. A post-hoc count on validation agrees: on the three videos averaging >=14 people per frame, independent windows paint 14-15 regions per frame with either checkpoint, while chaining lowers the count (12-14 with the translation-only checkpoint, 10-12 with the prefix-trained one), so 10-12 is not a token budget. (b) Crowds fail even on clips the model was trained on (recall 0.09-0.30 vs 0.49-0.94 on sparse clips), so training coverage alone does not explain the failure either. The outcome matched neither branch of our pre-registered rule, and we report it as such.

A post-hoc diagnosis (labelled exploratory in the paper) points at spatial resolution. In the >=20-person clips the median dancer is 20-23 px wide at 512^2, under one 32-px latent token (84-97% of people narrower than a token, against 32-46 px in sparse clips), and in one frame per clip 5-7 generated regions merge two or more neighbours, against 0-1 on sparse clips. That fits the 768^2 observation that crowds separate. But density and person size go together in these crowds, so we cannot say whether more tokens per person, more capacity or better optimisation is the fix. The revised Section 4.5 says exactly this: dense scenes fail even on training data, merged neighbours narrower than a latent token are a common error there, and the cause is unresolved.

The factorial experiment (Q1) narrows but does not fully separate generation-time state from checkpoint and detection effects. The long-gap recovery analysis (Q2) actually reduces the evidence for active propagation, with only 4/7 clearly localized cases. The crowded-scene analysis (Q3) withdraws the capacity claim and leaves the mechanism unresolved. I will maintain my rating of 2 (Weak rejection).

Reviewer_FhNEOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a generative formulation of multi-object tracking in which object identities are represented as persistent colours in generated ID-map videos, rather than in an external detector, motion model, appearance embedding, or association module. A 22B video diffusion model is fine-tuned with an in-context LoRA to translate RGB clips into ID-map videos. Long videos are processed through overlapping windows, with cleaned ID-map tails re-injected as prefixes; continuation training teaches the model to preserve an existing colouring across windows. On the DanceTrack test server, the method achieves 40.3 HOTA, 44.1 AssA, and 45.2 IDF1. Although the overall performance is substantially below recent specialized trackers, controlled experiments suggest that generation-time prefix state carries identity more effectively than post-hoc association. The method also demonstrates identity re-acquisition after occlusion and shows a similar association advantage on SportsMOT, although with lower absolute performance.

Critical strengths
  1. Highly novel formulation. The paper presents a genuinely different view of tracking by representing identity directly in generated pixels. This is conceptually interesting and goes beyond incremental modifications to conventional detection-and-association pipelines.

  2. Useful controlled analysis of identity propagation. The comparisons between chained generation, independently generated windows, post-hoc association, and classical associators provide evidence that generation-time state carries identity. The occlusion re-acquisition experiments further support the proposed pixel-level identity mechanism.

Critical weaknesses
  1. The practical performance is substantially below current MOT methods. The method achieves only 40.3 HOTA on DanceTrack and runs at approximately 0.5 FPS end-to-end, while recent specialized trackers achieve much higher accuracy and efficiency. Therefore, the current work should be framed primarily as a proof of concept rather than as a practically competitive tracking system.

  2. The comparison with recent and directly relevant methods is incomplete. Important recent methods such as MOTRv3, CO-MOT, and DualMOT/DualTTrack are not included. MOTRv3 and CO-MOT are particularly relevant because they extend the MOTR-style query-propagation paradigm, while DualMOT/DualTTrack represents recent work on dual-track matching and recovery. Comparisons with recent association-oriented methods such as MOTIP and LA-MOTR would also help clarify the relative value of the proposed pixel-level identity representation. The authors should either include these baselines under matched datasets and extra-data settings or clearly justify their omission.

  3. Robustness is not yet fully established. Most detailed analyses are conducted on the DanceTrack validation set, and the main experiments rely largely on a single sampling seed. In addition, continuation training uses teacher-forced prefixes that differ from the self-generated prefixes used at inference. The authors report additional seed checks, but broader multi-seed and cross-domain validation would strengthen the conclusions.

Questions for authors
  1. Why are MOTRv3, CO-MOT, and DualMOT/DualTTrack omitted from the comparisons? Can the authors report results under matched DanceTrack and SportsMOT protocols, including the use of extra training data?

  2. Can the authors compare the proposed pixel-level identity mechanism with recent ID-prediction and learnable-association methods such as MOTIP and LA-MOTR under comparable detection and evaluation settings?

  3. Given the large gap in absolute HOTA, can the authors provide a more direct comparison of association quality on identical detections, including confidence intervals or per-video results across multiple random seeds?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 2 (1/2)

Proof of concept. Agreed; the abstract, introduction and conclusion now present the system as a proof of concept, with absolute accuracy (40.3 HOTA) and throughput (about 0.4 fps) stated as limitations, not traded off against other claims.

Q1. MOTRv3, CO-MOT, DualMOT/DualTTrack. Table 1 now includes the verified rows we could source; their settings (DanceTrack test):

MethodVenueExtra dataHOTADetAAssAIDF1MOTA
CO-MOTICLR 2025CrowdHuman69.482.158.971.991.2
MOTRv3arXiv 2305.14298none directly; training-time pseudo-labels from detectors trained with extra data (incl. CrowdHuman)70.483.859.372.392.9
LA-MOTRICCV 2025none71.184.459.071.893.5
MOTIPCVPR 2025none (72.0 HOTA with extra data)69.680.460.474.790.6
Ours--none (web-pretrained generator; SAM 3 silhouettes)40.337.644.145.216.2

We could not locate a published method named "DualMOT". We found DualTTrack (Information Processing & Management, 2026) but could not verify its DanceTrack protocol and numbers from the publication available to us, so we do not cite numbers for it; we would be grateful for a pointer. On "extra data": we use no tracking annotations beyond the 40 DanceTrack training videos, but the generator was pretrained at web scale and silhouettes come from SAM 3, which is a different kind of prior from detector pretraining. The revised table states this, and now also marks MOTRv2, whose proposal detector uses CrowdHuman. On SportsMOT, LA-MOTR reports 72.4 and MOTIP 72.6 HOTA on the test split; we report validation only (26.9 HOTA) and do not claim comparability.

Q2. Comparison with MOTIP and LA-MOTR under comparable detections. Their association heads consume their own decoder features, so they cannot be run on our boxes without re-engineering them, and doing so would no longer test their method. We therefore compare association on identical detections against associators that do accept external boxes: Kalman/IoU (motion) and a DINOv3 appearance-and-motion bank. On DanceTrack and on SportsMOT the trained chain's colours lead both (Table 2, Table 4; per seed and per cell in our next comment).

AuthorsOpenReview ↗

Response to Reviewer 2 (2/2)

Q3. Identical detections with CIs, per video, across seeds. Paired per-video bootstrap (25 videos, 10,000 resamples): each cell's native identity link minus the associator on the same boxes. The native link is the colour for the chains (A, C) and, for the independent cells (D, B), the colour within a window plus the pinned overlap link.

BoxesvsHOTAAssAIDF1DetAvideos favouring native (AssA)
A, seed 0Kalman/IoU+9.2 [6.8, 11.7]+15.3 [11.6, 19.1]+13.9 [10.1, 18.1]-0.3 [-0.5, -0.1]24/25
A, seed 0DINOv3 bank+7.5 [5.2, 9.9]+12.5 [9.0, 16.3]+11.3 [7.6, 15.4]-0.0 [-0.2, 0.2]23/25
A, seed 1Kalman/IoU+10.5 [7.9, 13.4]+18.7 [14.4, 23.2]+15.7 [11.5, 20.3]-0.5 [-0.7, -0.4]25/25
A, seed 1DINOv3 bank+8.1 [6.3, 10.1]+14.8 [11.6, 18.2]+12.0 [8.9, 15.5]-0.2 [-0.2, -0.1]24/25
A, seed 2Kalman/IoU+10.5 [7.8, 13.6]+18.3 [13.7, 23.4]+16.3 [11.6, 21.4]-0.3 [-0.4, -0.2]25/25
A, seed 2DINOv3 bank+8.8 [6.3, 11.5]+15.5 [11.3, 20.2]+13.0 [9.2, 17.3]-0.1 [-0.3, -0.0]25/25
CKalman/IoU-0.2 [-1.1, 0.5]-0.5 [-1.7, 0.4]+0.7 [-0.7, 1.9]-0.6 [-0.7, -0.5]13/25
CDINOv3 bank-1.9 [-3.5, -0.7]-2.7 [-5.4, -1.0]-2.3 [-4.8, -0.5]-0.2 [-0.3, 0.0]3/25
DKalman/IoU+2.8 [2.0, 3.8]+4.4 [3.0, 6.1]+3.9 [2.5, 5.6]-0.2 [-0.3, -0.1]25/25
DDINOv3 bank+2.4 [1.5, 3.4]+3.9 [2.3, 5.7]+3.1 [1.9, 4.6]+0.0 [-0.1, 0.1]23/25
BKalman/IoU+2.7 [1.7, 4.0]+4.2 [2.5, 6.4]+3.6 [2.1, 5.4]-0.2 [-0.3, -0.1]24/25
BDINOv3 bank+1.9 [1.1, 2.8]+3.1 [1.7, 4.8]+2.2 [0.8, 3.8]+0.1 [0.0, 0.2]23/25

In the independent cells (D, B) colour carries identity only inside a window, so the lead shrinks, but it stays positive in both. In C (chained without continuation training) it is gone: the model overrides the colouring it is handed, and the colours are no better than Kalman/IoU and worse than the bank. The lead belongs to the trained chain, not to colour as such. Per-video results for the published arms are in Appendix B, and per seed (colours, Kalman/IoU and bank) in Appendix E.

Robustness. Seeds: on all 25 videos HOTA is 34.8 / 37.1 / 37.8 (sd 1.6), and the paper's seed is the lowest; single videos move more (median per-video HOTA range 8.4 points across seeds), which is why every comparison above is paired per video. Cross-domain: the same recipe re-run from the stock backbone on SportsMOT keeps the association ordering on identical boxes (AssA 28.2 vs 20.7 and 17.1) while long-gap re-acquisition does not transfer (369 of 374 gaps there are frame exits). Teacher-forced prefixes: the chain conditions on cleaned (decoded and re-encoded) self-generated tails, which narrows but does not remove the train/inference mismatch. Per-window IDF1 falls from 0.54 (windows 1-5) to 0.49 (windows 16-25) in seed 0 but rises in seeds 1 and 2 (0.49 to 0.54, 0.51 to 0.54), so we see no consistent erosion along the chain. Training on self-generated prefixes is untested, as the Limitations now state.

Reviewer_TCTvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper formulates multi-object tracking as a generation task that maps RGB videos to identity color maps. The authors build on the pretrained 22B-parameter video generation model LTX-2.3 and employ lightweight In-context LoRA fine-tuning to ensure that each pedestrian is assigned a stable and distinct color in the generated video, with the color index directly serving as the tracking ID. For long videos, continuation training is used to achieve identity continuity across windows. The paper achieves 40.3 HOTA on the official DanceTrack test set and explores the feasibility of using a generative model to perform identity association through association comparisons and validation on SportsMOT.

Critical strengths
  1. A Clear and Worthwhile Generative Tracking Paradigm Is Proposed. The paper shifts identity representation from external track states to generated video pixels, uses color as identity encoding, and conveys identity states through prefix conditioning. The technical objective is clear: to use a generative model to directly perform data association.

  2. The Experimental Analysis Is Relatively Comprehensive, and the Authors Adequately Disclose the Limitations. The paper reports not only results on the official test set but also association baselines using the same detection boxes, re-identification analysis after occlusion, dense-scene diagnostics, and validation on SportsMOT. The authors also explicitly acknowledge the low detection recall, insufficient inference efficiency, identity fragmentation, and fluctuations caused by random seeds.

Critical weaknesses
  1. The Incremental Novelty Is Limited, and the Technical Distinction from Existing Work Is Not Sufficiently Clear. The core idea of this paper is to use a generative model to produce videos with persistent identity colors and maintain cross-frame correspondences through color consistency. However, the Related Work section already cites color propagation tracking, Pix2Seq-D for generative video segmentation, and DiffTrack for its study of temporal correspondence in video diffusion models. These works each address some components of the core idea of this paper. The main contribution of this paper is to use a large-scale pretrained video generation model for RGB-to-ID-map conversion and to maintain identity colors through continuation training. This combination has some value, but the paper has not sufficiently clarified the extent of its algorithmic novelty relative to the aforementioned works.

  2. The Control Experiments for the Core Mechanism Do Not Fully Rule Out Checkpoint Differences. In Table 2, the chained generation setup uses the Phase B checkpoint, whereas the independent-window baseline uses the Phase A checkpoint. The authors provide supplementary results from single-video experiments, but have not sufficiently isolated these two factors on the full validation set using fully matched checkpoints and inference settings. Meanwhile, all main results in the paper use a single sampling seed, and the additional three-seed experiments cover only five validation videos. The authors report HOTA fluctuations of several percentage points across different seeds, with the re-identification rate also showing noticeable variation.

Questions for authors

Q1. What is the key algorithmic innovation of this paper relative to existing color propagation tracking, Pix2Seq-D for generative video segmentation, and DiffTrack's study of temporal correspondence in video diffusion models? Could the authors clearly distinguish the core mechanism of this work from existing methods and explain what substantive new capabilities continuation training introduces beyond existing techniques?

Q2. In Table 2, the chained generation setup uses the Phase B checkpoint, whereas the independent-window baseline uses the Phase A checkpoint. Could the authors provide a more rigorous matched comparison between chained generation and independent-window generation using the same checkpoint, window partitioning, decoding rules, and sampling settings? This would help determine the extent to which the performance improvement comes from identity-state propagation during generation, rather than differences in checkpoints or inference settings.

Q3. The paper reports that, in all 86 successful identity recovery events, the original identity color appeared in at least one frame during the intervening interval. Could the authors further quantify the area, duration, and spatial continuity of these color regions, and explain how valid identity-state continuation can be distinguished from generation artifacts or residual color regions? This would help provide a more accurate interpretation of the actual mechanism underlying long-gap identity recovery.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 1 (1/2)

Q1. Novelty relative to colourisation tracking, Pix2Seq-D and DiffTrack. We agree that each ingredient has a precedent; the Related Work acknowledged each, and the revision now states how our mechanism differs from each (blue text). The distinction is in what the generator is asked to output and what it must therefore solve:

  • Tracking by colourisation (Vondrick et al., 2018) learns to copy colours from a reference frame by pointing; at test time it propagates labels given in the first frame. The colour is an input signal, objects must be annotated at t=0, and new objects are never instantiated. In our system the colour is an output identity code: no first-frame masks or detector; the model instantiates every person, assigns a new colour to new entrants, and must keep a colour alive while the person is invisible.
  • Pix2Seq-D generates panoptic masks and, in video, conditions each frame on a few past predicted masks so instance ids persist. That is the ancestor of our decode-and-recondition loop, as we state. It is trained for panoptic segmentation; we fine-tune a pretrained video generator that produces a 49-frame window jointly with 3D attention, so short occlusions are bridged inside the context, and we evaluate it as a multi-object tracker on a test server and under occlusion re-acquisition.
  • DiffTrack shows that video DiTs encode temporal correspondence in their attention and reads it out for zero-shot point tracking. It probes internal features for given query points; it does not instantiate objects or emit identities, and it does not address identity through full occlusion.

What continuation training adds. Chaining a generator across windows means asking it to continue an arbitrary, imposed colouring rather than to produce a plausible one. Without prefix training the model overrides the imposed colours mid-window; with it, the chain carries the colouring across window boundaries and can run to any video length. The factorial (Q2) now measures this at full scale: chained generation with the translation-only checkpoint (cell C) scores 16.8 HOTA / 10.1 AssA / 15.5 IDF1 against 34.8 / 38.3 / 38.3 for the prefix-trained checkpoint (cell A), a paired difference of +18.3 HOTA [14.9, 22.0] and +27.0 AssA [22.0, 32.2], in all 25 videos. Without continuation training, chaining is no better than independent windows (C-B -1.2 HOTA [-3.8, 1.1]), and in C the colours do not beat Kalman/IoU or the DINOv3 bank on the same boxes.

AuthorsOpenReview ↗

Response to Reviewer 1 (2/2)

Q2. Matched comparison. Table R1 gives the full factorial (summarised in the general response, item 1).

Table R1. 2x2 factorial, 25 DanceTrack validation videos, seed 0; each cell under its native identity link, plus AssA of Kalman/IoU and of the DINOv3 bank on that cell's boxes.

CellCheckpointGenerationHOTADetAAssAIDF1AssA, Kalman/IoUAssA, DINOv3 bank
A (paper)prefix-trainedchain34.832.338.338.321.624.3
C (new)translation-onlychain16.828.710.115.510.412.6
D (new)prefix-trainedindependent15.917.415.115.710.711.2
B (paper)translation-onlyindependent18.219.717.717.312.714.3

Paired per-video effects on HOTA (bootstrap 95% CI; videos favouring the first term):

  • chaining, prefix-trained checkpoint (A-D): +18.9 [15.2, 22.9], 24/25;
  • chaining, translation-only checkpoint (C-B): -1.2 [-3.8, 1.1], 12/25;
  • prefix training, chained (A-C): +18.3 [14.9, 22.0], 25/25;
  • prefix training, independent windows (D-B): -1.8 [-5.1, 0.9], 9/25.

Neither factor dominates: each is worth nothing on its own, and the whole gain is their interaction. Chaining pays only when the model has been trained to continue the colouring it is handed. This is the paper's single-video ablation (IDF1 0.17 without prefix training), now confirmed on all 25 videos.

On seeds: with two further seeds on all 25 videos, seeds 0 (the paper's) / 1 / 2 give HOTA 34.8 / 37.1 / 37.8 (mean 36.5, sd 1.6), and the carry-over statistics below replicate in both new seeds (Appendix E). On identical detections the colours beat Kalman/IoU by +15.3 AssA (95% CI [11.6, 19.1], 24/25 videos) and the DINOv3 bank by +12.5 ([9.0, 16.3], 23/25), with DetA within 0.3, and by more in seeds 1 and 2.

Q3. Area, duration and continuity of the carried colour; state versus artifact. Pinned statistics over all 205 conditional gap events of the chain (medians; area, IoU and components over the events with the colour present):

Re-acquired (n=86)Not re-acquired (n=119)
colour present in >=1 gap frame86/86110/119
fraction of gap frames with the colour1.000.90
number of runs / longest run (frames)1 / 14 (gap 16)1 / 10 (gap 20)
area relative to the person's last visible box1.11x0.93x
IoU with the person's interpolated position0.350.19
connected components1.241.21
present in every intervening prefix37/3865/76

In the typical recovery the colour is about person-sized, present in every gap frame as one continuous run (60 of the 86), mostly one component, and it propagates from prefix to prefix. A residual artifact would be fragmentary, intermittent and unanchored; this is none of those. But the colour also survives the gap in 110 of 119 failures, so carry-over is necessary, not sufficient: what distinguishes recoveries, on average, is the colour staying near the hidden person (IoU 0.35 vs 0.19; the distributions overlap). Section 4.3 now says exactly this. Most gaps are occlusions (190 of 205), so the carried colour often lies on another visible person: in a median 62% of its frames for recoveries (32% for failures). A check added after the pinned statistics, and so exploratory, asks whether that person keeps a box of its own colour in those frames: in a median 0.04 of them for recoveries and 0.25 for failures (0.00-0.09 in seeds 1-2). The occluder is mostly painted in the hidden person's colour, so carrying one identity can cost the occluder its own. Appendix D reports this, labelled exploratory.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.