- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper formulates multi-object tracking as generating videos with persistent identity colours, from which trajectories are decoded. A video generator is adapted through prefix continuation training and chained inference. The main analysis examines the source of identity information, recovery after occlusion and failures in crowded scenes.
Critical strengths
- The tracking representation is worth exploring. Encoding identities as persistent colours makes the video generator responsible for both localization and identity preservation, offering a useful alternative formulation to conventional detection-and-association pipelines.
- The training design responds to an observed failure. Without continuation training, the model can overwrite the imposed colouring after the overlap. Prefix training addresses this behavior, connecting the proposed intervention to a concrete failure mode.
- The controls examine different sources of identity information. The paper compares independent and chained generation, re-links identical generated windows, and runs motion and appearance association on identical detections. These controls help assess the respective roles of generated colours and post-processing. (Table 2)
- The evaluation reports useful limitations. Alongside the DanceTrack test result, the paper includes SportsMOT, seed variation, crowded-scene failures and runtime. The unsuccessful transfer of long-gap recovery to SportsMOT is particularly informative about the method’s scope.
Critical weaknesses
- The comparison does not isolate generation-time state. Both the checkpoint and generation mode change, while DetA increases from 19.7 to 32.3. Although the paper acknowledges these differences, its subsequent interpretation still attributes the gap rather directly to prefix state. The same-window and identical-detection controls are useful, but do not fully separate additional training, prefix conditioning and changes in detection quality.
- The explanation of long-gap identity recovery remains stronger than the evidence. The original colour appears in at least one gap frame for successful recoveries. This is consistent with carry-over, but does not establish propagation through every intervening conditioning prefix or rule out colour reuse in unrelated regions.
- The cause of crowded-scene failure is not sufficiently isolated. The paper attributes the failure to latent-token capacity and identifies resolution-matched training as the remedy. However, the 768² experiment also changes the input distribution and reduces detection recall. The results show density-related degradation, but do not distinguish representation capacity from training coverage or resolution adaptation.
Questions for authors
- Can the existing results further separate checkpoint changes, prefix conditioning and detection-quality changes in the A/B comparison?
- In the seven successful recoveries beyond the context length, does the original colour pass through every intervening prefix, and which regions carry it during the gap?
- What evidence specifically distinguishes a token-capacity limit from insufficient training coverage or resolution adaptation?
3 comments
Response to Reviewer 3 (2/2)
Q3. Capacity versus training coverage versus resolution. You are right that the 768^2 probe cannot separate these, and a new probe shows our capacity claim was too strong; we withdraw it. Coverage first: of the 779 49-frame windows we count in the training annotations, 95 average >=14 people per frame and 65 average >=20 (up to 40), so crowds are in training, although the 14-19 band is thinner than in validation (3.7% vs 11.1% of frames). We then rendered 12 training clips that the translation-only checkpoint was fit on, as single windows at the training resolution, with a reading rule fixed before the run:
| Training clips (selected by annotated people / frame) | GT regions / frame | Painted regions / frame (max) | Mask recall @ IoU 0.5 |
|---|---|---|---|
| >=20 people (4 clips) | 22.9-37.4 | 15.7-23.8 (28) | 0.09-0.30 |
| 14-19 people (4 clips) | 13.9-16.0 | 8.5-12.9 (15) | 0.20-0.35 |
| 4-9 people (4 clips) | 6.0-7.8 | 7.4-9.5 (11) | 0.49-0.94 |
(GT regions are decoded ground-truth ID-map regions with the same minimum area, which drops small or occluded people, so they can fall below the annotated band.)
Two conclusions follow directly. (a) The model is not limited to 10-12 instances: on seen crowds it paints up to 28 regions per frame, so the count ceiling we reported on validation is not a property of the model at 512^2, and "token-capacity-bound" is withdrawn. A post-hoc count on validation agrees: on the three videos averaging >=14 people per frame, independent windows paint 14-15 regions per frame with either checkpoint, while chaining lowers the count (12-14 with the translation-only checkpoint, 10-12 with the prefix-trained one), so 10-12 is not a token budget. (b) Crowds fail even on clips the model was trained on (recall 0.09-0.30 vs 0.49-0.94 on sparse clips), so training coverage alone does not explain the failure either. The outcome matched neither branch of our pre-registered rule, and we report it as such.
A post-hoc diagnosis (labelled exploratory in the paper) points at spatial resolution. In the >=20-person clips the median dancer is 20-23 px wide at 512^2, under one 32-px latent token (84-97% of people narrower than a token, against 32-46 px in sparse clips), and in one frame per clip 5-7 generated regions merge two or more neighbours, against 0-1 on sparse clips. That fits the 768^2 observation that crowds separate. But density and person size go together in these crowds, so we cannot say whether more tokens per person, more capacity or better optimisation is the fix. The revised Section 4.5 says exactly this: dense scenes fail even on training data, merged neighbours narrower than a latent token are a common error there, and the cause is unresolved.
The factorial experiment (Q1) narrows but does not fully separate generation-time state from checkpoint and detection effects. The long-gap recovery analysis (Q2) actually reduces the evidence for active propagation, with only 4/7 clearly localized cases. The crowded-scene analysis (Q3) withdraws the capacity claim and leaves the mechanism unresolved. I will maintain my rating of 2 (Weak rejection).
Response to Reviewer 3 (1/2)
Q1. Separating checkpoint, prefix and detection. The full 2x2 factorial (Table R1, in our response to R1) varies one factor at a time on identical windows. Detection is separated inside each cell: the Kalman/IoU and DINOv3 rows re-associate that cell's own boxes, so the colour-vs-associator gap at fixed DetA is measured in all four cells.
So the A-B gap is generation-time state, present only with the checkpoint trained to continue it, and not detection: with detection equalised (raw-colour link) the A-D gap is as large as under the native links (+19.8 vs +18.9 HOTA). We revised Section 4.2 so the interpretation follows the factorial, not the A/B pair.
Q2. The seven recoveries beyond the context. Per event (gap length; fraction of gap frames with the colour; colour frames in each intervening 9-frame prefix; IoU with the interpolated position and area vs last visible box, over the frames with the colour):
Four of the seven keep the colour at the hidden person's location (IoU 0.41-0.64, 1.0-1.8x its size) through every prefix; three fill all nine prefix frames and 0081 / id 19 thins to 6 and then 3 of 9. Two keep the colour in every prefix but poorly located: 0034 / id 10 is enlarged 4.35x at IoU 0.20, and 0014 / id 8 appears in only 2 of 9 prefix frames at IoU 0.25. One (0034 / id 12) loses it in a prefix and re-acquires anyway. The two 0034 events carry an enlarged, poorly located colour (area 2.1-4.4x, IoU <= 0.20) that we cannot distinguish from reuse. So propagation through every prefix holds for 6 of 7, and clearly located carry-over for 4 of 7. The revision keeps the count that all 7 show the colour in at least one gap frame, no longer reads it as propagation, and adds this table (Table 7). Over all 38 recoveries whose gap spans a window boundary, the colour is in every intervening prefix in 37.