acceptodds
Withdrawn from ICLR 2027

QTrack: Learning to Use Temporal Evidence for Reference-Grounded Video Tracking

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Category-level multi-object trackers estimate trajectories for all detected instances, whereas some applications require propagating only targets specified by a user. We study offline, reference-grounded video tracking: given an ordered video clip, a natural-language query, and the reference-frame boxes of the specified targets, a model predicts their subsequent trajectories. We introduce QTrack, a vision-language policy trained with structured trajectory rewards and Temporal Perception-Aware Policy Optimization (TAPO). TAPO contrasts the likelihood of a generated response under the observed clip and a reference-frame repetition, encouraging the output to depend on visual evidence beyond the reference frame. We also introduce RMOT26, a controlled diagnostic benchmark constructed from existing tracking datasets. Under the reported protocol, QTrack obtains a motion-consistency score of 0.30, MOTP of 0.75, center-location error of 44.61 pixels, and normalized distance error of 0.39; its motion-consistency score ties the strongest comparison method. Component ablations suggest complementary effects from the motion reward and temporal contrast, while an exploratory four-clip, 50-frame evaluation examines transfer beyond the primary short-clip setting. The results support further study of temporal-evidence learning in trajectory-generating vision-language models. Code and data are available at https://anonymous.4open.science/r/QTrack-5548.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Object Tracking, Query-Conditioned Tracking, Spatiotemporal Reasoning, Vision-Language Models, Video Understanding, Multimodal Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_aHhqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

QTrack generates target trajectories from a short clip, a text query, and reference-frame boxes. It operates offline, with the boxes identifying the requested targets. The model is initialized from VisionReasoner-3B and trained with GRPO using format, localization, and motion rewards. TAPO contrasts the likelihood under the observed clip with that under a clip repeating the reference frame. Evaluation uses RMOT26, a diagnostic dataset assembled from existing tracking datasets. The paper reports 0.30 MCP, 0.75 MOTP, 44.61-pixel CLE, and 0.39 NDE. It also includes component ablations, a reasoning-output ablation, and an exploratory four-clip evaluation at 50 frames.

Critical strengths
  1. The task definition is explicit about what the model is given: initial boxes specify target membership, and all clip frames are available offline. This distinguishes the task from language-only target discovery and online tracking. A controlled benchmark for propagating these explicitly grounded targets could help assess VLMs that generate coordinates.
  2. The ablations show effects that an overall baseline comparison would miss. In Figure 4, GRPO raises MOTP from 0.44 to 0.65. Adding the MCP reward raises MCP from 0.22 to 0.25 but reduces MOTP to 0.61, and the full combination reaches 0.30 MCP and 0.75 MOTP. Appendix Table 9 removes structured reasoning, directly testing its contribution beyond a correlation between rationale quality and accuracy. These controls are useful, subject to the evaluation concerns below.
Critical weaknesses
  1. The RMOT26 description does not yet support an independent, reproducible test set. Appendix B describes a sequence-level split, but Table 3 lists hundreds of “sequences” from MOT16 and MOT17. These appear to be sampled clips rather than original source videos. Section 6 leaves source-video-disjoint splits to future work. Disjoint clips can still share video content and identities, and MOT17 extends the MOT16 videos. Checking overlap separately by dataset name would therefore be insufficient. This does not establish leakage, but the description does not exclude it. Clip lengths are also inconsistent: Table 4 lists 1,000 test sequences and 4,008 frames, giving 4.008 frames per sequence. Appendix B.1 instead describes a reference frame followed by 5–6 future frames, while Appendix C describes 5–6-frame evaluation. The source-video mapping, frame ranges, and counting convention are needed to interpret the benchmark.
  2. The evaluation MCP definition needs clarification, and the evidence for query and temporal use is incomplete. Main-text Eq. (8) defines the training reward as max(0, A)SP, while Appendix Eqs. (13)–(14) define evaluation MCP as AS. These can serve different purposes. The concern is the [0,1] range stated in Appendix C.1: A is a cosine similarity, so AS can be negative. Under Eq. (7), zero ground-truth displacement also gives A = 0, including when P = 1. Please clarify the evaluator’s handling of negative alignment and stationary targets. The introduction also announces a fixed-box language intervention, but I could not locate its quantitative results in the main paper or appendix. Figure 10 removes the initial box, and Table 9 removes structured reasoning. Neither changes query content while keeping boxes and video fixed. The TAPO training ablation is useful, but sensitivity to the input alone provides limited evidence that motion is used correctly. The current results therefore leave the query-use and temporal-use claims only partly supported.
Questions for authors
  1. Please provide a train/test manifest mapping each RMOT26 clip to its original video and frame range. Were original videos and identities kept disjoint, including duplicates shared by MOT16 and MOT17? How does the stated 5–6-frame protocol reconcile with 4,008 frames across 1,000 test sequences?
  2. Appendix C.1 defines evaluation MCP as AS with a [0,1] range. Is A clamped or transformed in the evaluator, and how are stationary ground-truth intervals scored? Please identify the implementation used for Tables 1–2 and Figure 4. Do the reported evaluation metrics use the Hungarian matching, normalization, and invalid-output penalties described for the rewards in Sections 3.3 and 5? How is the supplied reference frame treated?
  3. Where are the fixed-box language-intervention results announced in the introduction? Is there an evaluation comparing original clips with reference-frame repetition while holding the initial boxes and output protocol fixed? Please identify which experiments have been completed and which are proposed.
  4. Algorithm 1 and Eq. (11) specify a sampled log-ratio estimate of the KL value. Could you provide the corresponding loss or gradient pseudocode? Since the sampling policy depends on the parameters, differentiating the expected KL includes a sampling-distribution term. Please clarify how the implementation handles this dependence, including any score-function, importance-weight, or stop-gradient operations.
Reviewer_RkMoOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

QTrack trains a vision-language model for reference-grounded video tracking using structured rewards and temporal contrast, and introduces the RMOT26 benchmark.

Critical strengths

none

Critical weaknesses

1.Inconsistent metric definition. The MCP formula in Eqs. (13)–(14) can produce negative values, contradicting the claimed range of [0,1]. The precise metric used for the reported results is therefore unclear.

2.Insufficiently specified baseline comparisons. Although specialized trackers are included, equivalent access to reference-box initialization and future frames is not clearly established. This limits the conclusions that can be drawn about QTrack’s advantage over dedicated tracking methods.

Questions for authors

The comparisons do not clearly establish matched initialization and temporal access across methods. How can the reported gains be attributed to QTrack rather than differences in evaluation conditions?

Reviewer_4dtqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces QTrack, an offline video tracking method trained with GRPO, and TAPO, a temporal contrastive objective designed to improve motion understanding. The authors also present RMOT26, a benchmark with 1,000 test instances from seven MOT datasets. QTrack achieves an MCP of 0.30, matching TempRMOT, with additional evaluations covering ablations, traditional trackers, and reasoning quality.

Critical strengths
  1. Clear problem scope. The paper clearly defines the tracking task and its limitations, making the claims easier to evaluate and providing a useful setting to study temporal reasoning in VLMs.

  2. Strong improvement over zero-shot VLMs. QTrack reduces CLE from 94–990 px to 44.6 px using a 3B model, showing that RL with trajectory-based rewards can improve coordinate prediction.

Critical weaknesses
  1. Missing and potentially unfair baselines. The paper lacks key comparisons with standard SOT trackers, SAM2, and a simple reference-box-copy baseline. The existing baselines may use a mismatched protocol, and inconsistent box formats could explain some of their poor results.

  2. Potential train/test leakage. Clips from the same source videos may appear in both splits, and MOT16 and MOT17 share videos. This raises concerns about whether the evaluation measures generalization to unseen videos.

  3. Evaluation metric concerns. MCP is both a training reward and the main evaluation metric, yet QTrack only ties TempRMOT on it. Moreover, MCP is defined inconsistently across equations and experiments.

  4. Incomplete evaluation details. MOTP, CLE, and NDE are not clearly defined, making the results harder to verify and interpret.

  5. Lack of statistical validation. Results are reported from single runs without variance estimates, leaving the reliability of the improvements in the ablation studies unclear.

  6. Table 7 is said to report IDF1 but does not.

Questions for authors
  1. How does QTrack compare with a reference-box copy baseline, per-target SOT trackers, and SAM2 under the same protocol?

  2. Are the train and test clips source-video-disjoint? How are overlapping MOT16/MOT17 videos handled, and can results on a source-video-disjoint split be provided?

  3. What box formats were used for baseline inputs and outputs? Were all predictions converted consistently before evaluation?

  4. Where are the results of the fixed-box language intervention mentioned in the introduction? Does query semantics improve tracking once reference boxes are provided?

  5. How were TempRMOT and DKGTrack adapted to use reference boxes?

  6. Why does zero-shot VisionReasoner achieve much higher HOTA than specialized trackers in Table 7? How were tracker outputs matched to the queried target? Also, why do specialized trackers achieve higher MCP than QTrack, contrary to the claimed improvement in motion consistency?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.