acceptodds
Under review as a conference paper at ICLR 2027

Track-OPD: Learning from Future Consequences for Multi-Object Tracking via On-Policy Distillation

Abstract

Online multi-object tracking associates observations using the identity history created by earlier decisions. Learning from the future consequences of these decisions requires comparing alternative associations, yet each alternative can change which tracks survive and which objects their identities denote. We propose Track-OPD, an on-policy distillation method that makes these consequences comparable under fixed identity bindings. Alternative associations continue from the same tracker state under shared future observations and a frozen policy. Their outcomes are evaluated on the same annotated objects and frames, retaining missing and incorrect predictions in the comparison. The resulting future identity preferences are aligned with current association classes to construct a local teacher. This teacher redistributes probability between the ground-truth reference and verified poorer alternatives while preserving their total probability. On DanceTrack and SportsMOT, Track-OPD improves HOTA over continued training by 1.03–1.36 points across the evaluated tracker adaptations. At 4,000 updates, its target construction gains 0.543 HOTA and 1.184 AssA over unit ground-truth preferences on the same selected records. These results support learning current associations from the identity history they produce while retaining the original online inference structure.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-Object Tracking, On-Policy Distillation, Online Data Association, Future Identity Preferences, Trajectory-Level Supervision

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_doZxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The underlying motivation of this paper is interesting: not all ground-truth annotations are equally important, and the costs of prediction errors are not necessarily equal. In the absence of a perfect solution, a reasonable second-best approach is to prioritize correcting errors with higher costs rather than those with lower costs. However, the proposed method is overly complicated and relies on too many intricate design choices. Moreover, the effectiveness of some of its contributions has not been adequately evaluated, and some of the reported results contain factual errors.

Critical strengths

As stated in our summary, the authors' proposed idea of non-uniform supervision allocation appears to be a feasible approach, and the underlying concept is innovative.

Critical weaknesses
  1. I have serious concerns about the reliability of the experiments presented in this paper. In Table 6, the authors report MOTIP experiments under different conditions, with and without motion prediction. However, to my knowledge, MOTIP does not contain any motion prediction module at all. Moreover, the reported validation score of around 64.0 is close to MOTIP's original validation performance. This makes it difficult for me to understand the purpose and setting of this experiment, and I have serious doubts about the reliability of the reported results.

  2. The computation of future preferences involves numerous complicated and largely heuristic design choices, including the selection and weighting of different evaluation metrics and the construction of four verification channels. However, the authors do not provide individual ablation studies for these channels. This makes it difficult to assess whether all four channels are necessary, how they complement each other, and whether each contributes meaningfully to the final performance. Furthermore, there is considerable overlap among the metrics used across different channels, such as the repeated use of ID-switch penalties and identity correctness. What is the rationale behind this design? Why is such redundancy necessary, and why not consolidate these overlapping metrics into a simpler, unified evaluation function?

  3. Although the proposed method indeed outperforms the baseline under the same number of training steps, the on-policy collection and counterfactual rollouts performed at the beginning of each epoch may introduce substantial additional computational overhead. However, the authors do not provide a detailed analysis of the computational cost or wall-clock time associated with these operations. I believe that a comprehensive evaluation of these additional computational costs is essential for assessing the actual efficiency and practical value of the proposed method.

  4. Statements such as Wall-clock time and GPU-hour measurements are unavailable and The label ‘w/o contrast’ leaves the changed operation unspecified should not appear in a finalized research manuscript, especially the latter. Furthermore, the description of Ordinary Training in Figure 4 is also unclear. These issues raise serious doubts about the completeness and provenance of the submitted manuscript, as well as the extent of the authors’ actual involvement in verifying the work.

  5. On page 6, the authors introduce three auxiliary losses without sufficient justification. Why is KL loss alone insufficient? What are the intuitive motivations and empirical evidence supporting the additional ranking and value losses? The lack of individual ablations further raises concerns about their necessity.

  6. Although this concern may be difficult to address during the rebuttal, I would still like to emphasize that, while the underlying idea is appealing, its implementation is overly complicated, arguably even more so than traditional hand-crafted methods. This seems to contradict the fundamental philosophy of end-to-end MOT research, which aims to minimize manual design and learn directly from data with simpler frameworks.

Questions for authors

Regarding each point raised in the Weaknesses section:

  1. The authors must adequately address this issue. Otherwise, I will lower my rating to 1 due to serious concerns about the authenticity of the reported results.

  2. The authors need to provide appropriate ablation studies to clearly establish the effectiveness of at least the key components.

  3. The authors must provide a comprehensive analysis of the actual computational cost of the entire training pipeline, rather than simply stating that runtime measurements are unavailable.

  4. The authors need to provide point-by-point clarifications for all ambiguously described aspects of the paper.

  5. The authors need to provide the corresponding ablation results or relevant references.

  6. This is a difference in research philosophy, and the authors are not required to respond. However, I hope that other reviewers, ACs, and SACs will take this point into consideration in their overall assessment. I would also welcome further discussion in the discussion section to hear different perspectives on this issue. Ultimately, this is a matter of philosophical judgment rather than factual correctness.

12 comments
AuthorsOpenReview ↗

Responses to Reviewer doZx (Part 1 of 3)

Thank you for the detailed questions about experimental reliability and the training design. We clarify what Table 6 changes, then give the requested cost measurements and comparisons of the scoring and losses.

Question 1 (W1 / Q1): MOTIP has no motion-prediction module. What does Table 6 actually switch, what is the approximately 64 HOTA baseline, and what do the results show?

A1. Table 6 changes trajectory-history augmentation during training; both settings retain MOTIP's architecture. The operations simulate occluded history segments and identity switches. The parameters AUG_TRAJECTORY_OCCLUSION_PROB and AUG_TRAJECTORY_SWITCH_PROB are 0.5/0.5 in the default configuration and 0/0 without augmentation. Trajectory feature processing and the ID decoder remain active.

The experiment tests whether future-informed training benefits both settings. All results use the same 25,508 validation frames; these paired checkpoints are separate from the matched 4k study:

Configuration and starting pointMOTIP HOTA / IDF1Track-OPD HOTA / IDF1
Default; native Continue64.007 / 68.73465.141 / 69.169
Without trajectory augmentation; official checkpoint61.036 / 63.30163.798 / 68.005

The approximately 64 HOTA value in your comment is the default-configuration Continue. The second comparison starts from the official no-augmentation checkpoint. The gains in both comparisons therefore show that future-informed post-training benefits the existing association model under both trajectory-history augmentation settings.

Question 2 (W2 / Q2): Why use four verification channels with overlapping identity metrics? What evidence supports their individual contributions and their use over a simpler unified score?

A2. The four channels contribute to one score, with normalization and reliability determining how observable evidence is combined. Geometry and identity continuity describe different aspects of the same trajectory; interaction and recoverability add evidence when those events are observable. A geometrically plausible continuation that follows another person should still count as an identity error. This is why identity correctness and switch penalties recur across channels.

The aggregate defined in Appendix A.3 is a single function of primitive scores and penalties: repeated identity terms combine into one effective identity weight, and switch terms into one switch weight. The primitive-score coefficients are fixed. Channel availability, score normalization, and reliability determine how they combine for each record. Reliability follows observability and candidate-score separation.

We compared simpler scoring rules at 4k updates with seed 42:

AggregationHOTAAssAIDF1
Track-OPD65.64556.41771.560
Normalized, equal reliability65.54156.35270.105
Equal average of raw scores65.14356.57169.617

The normalized control sets available-channel reliability and aggregate confidence to one; raw averaging additionally removes normalization. Both reapply selection to the same 114 sampled states. The retained supervision is:

AggregationEligible recordsPositive targetsPredictions with nonzero auxiliary gradients
Track-OPD4235398
Normalized, equal reliability3329281
Equal average of raw scores3217

Positive targets count states with a nonzero teacher correction. The last column counts training uses, including repeated uses of a state; native supervision continues on all valid predictions.

Track-OPD has higher HOTA and IDF1 than both controls, while raw averaging has slightly higher AssA. The controls measure calibration together with the supervision it retains. This sample contains observable geometry, identity/appearance, and interaction evidence. These comparisons support normalizing available evidence and accounting for reliability when constructing association targets.

AuthorsOpenReview ↗

Responses to Reviewer doZx (Part 2 of 3)

Question 3 (W3 / Q3): What is the computational cost of the entire training pipeline, including collection, future branching, verification, and optimization? How often is collection repeated?

A3. Future evidence is prepared at frozen snapshots and reused across updates of the existing association model. Histories and verified preferences are reused within an epoch; the teacher is rebuilt from current predictions. The next epoch refreshes the snapshot. With MOTIP's detector frozen, branches share detector features for the same transformed frames.

The single-RTX-3090 profile below uses shared feature/history caches and 40 training sequences. It contains one preparation sweep, with 117 states and 238 branches at a 15-frame horizon, and separately measured 4k optimization:

PhaseContinue (s)Track-OPD (s)
Loading and training setup22.30922.354
Collection and future continuations, one sweep0.000925.352
Future verification and record construction0.0007.940
4k optimization1,397.2221,252.706

The measured optimization phase takes 10.3% less time for Track-OPD. Peak allocated GPU memory is 0.728 GiB during setup, 0.288 GiB during its collection, and 8.792 GiB during optimization for both methods; verification runs on the CPU. Thus, the largest measured GPU-stage peak is the same.

Initial common-cache generation and validation evaluation are separate. Future supervision adds preparation at every snapshot refresh, reused across the subsequent updates; the cached optimization retains the same measured peak memory and is faster in this profile.

Question 4 (W4 / Q4): Please clarify the unfinished runtime and “w/o contrast” statements, and the meaning of Ordinary Training in Figure 4.

A4. Ordinary Training is native training from scratch; Table 8 changes the two operations specified below. A3 gives the measured costs. In Figure 4, the native baseline reaches 43.282 HOTA. All branches end at the same final epoch. Introducing Track-OPD at epochs 0, 2, 4, or 8 yields 28.904, 37.675, 43.751, and 46.139 HOTA. The epoch-8 result is 2.857 higher, supporting introduction after initial association learning.

In Table 8, “w/o contrast” sets the coefficient of that configuration's explicit margin term [m−log⁡p(a+)+log⁡p(a−)]+[m-\log p(a^+)+\log p(a^-)]_+ to zero. Candidate-score centering is a separate teacher-construction operation. “w/o dynamic weight” disables record-specific dynamic weighting. Table 8 compares complete configurations; A5 gives the matched KL-only comparison for the current KL/rank/value objective in Appendix B.5.

Figure 4 therefore examines when to introduce future supervision, while the Table 8 labels identify the training operations being changed.

Question 5 (W5 / Q5): Why use rank and value losses in addition to KL, and what ablations support their contributions?

A5. KL matches the full teacher distribution; rank and value emphasize its pairwise probabilities and confidence. Rank fits the teacher's soft pairwise probabilities, emphasizing verified competitors. For a two-class support {y,j}\{y,j\} with mass M=py+pjM=p_y+p_j, before common normalization and weighting, ∂ℓrank∂zy=py−qyM,\frac{\partial\ell_{\rm rank}}{\partial z_y}=\frac{p_y-q_y}{M}, where the KL gradient is py−qyp_y-q_y. Rank therefore gives the verified pair greater weight when its current support mass is small. Value uses a Huber loss to match tanh-transformed log odds, emphasizing the confidence of the same comparison. These gradients follow from the loss definitions in Appendix B.5. All terms share the fixed point p=qp=q.

The matched 4k, seed-42 comparison is:

ObjectiveHOTAAssAIDF1
KL-only65.07856.40870.374
Track-OPD65.64556.41771.560

The joint objective gains 0.567 HOTA and 1.186 IDF1, with similar AssA. This supports our choice to fit both the distribution and its verified identity comparisons.

AuthorsOpenReview ↗

Responses to Reviewer doZx (Part 3 of 3)

Question 6 (W6 / Q6): The implementation appears more complex than hand-crafted methods. Does this contradict end-to-end MOT's aim of reducing manual design and learning associations through a simpler framework?

A6. Thank you for recognizing the idea's value. We use efficient training to help the existing tracker learn more from its videos. Track-OPD retains the host architecture, parameter count, online inference, and model complexity.

GT defines the reference identity. Score definitions and distillation settings are fixed across the evaluated trackers and datasets, while channel availability and reliability follow each comparison's observable evidence. These rules define the training signal; the association rule is learned by the original network. Detector features are shared across branches when the detector is frozen. Saved histories and verified preferences are reused within an epoch, while the teacher is rebuilt from current predictions. Table 4 and A3 report the resulting quality and optimization cost:

Reported comparisonContinueTrack-OPD
HOTA / AssA / IDF1, Table 464.934 / 56.184 / 69.72765.353 / 56.754 / 70.336
Cached 4k optimization (s), A31,397.2221,252.706
Optimization peak allocated GPU memory (GiB), A38.7928.792

Fewer updates reach higher quality. Cached 4k optimization takes 10.3% less time at the same peak memory; A3 gives preparation costs and the refresh schedule.

We see an important advantage of end-to-end MOT in learning temporal associations directly from well-annotated video. DanceTrack's similar appearances and diverse motion make temporal reasoning especially relevant [1]. MOTIP illustrates how associations can be learned by predicting identities from trajectory history [2]. This motivates us to improve how the model uses the temporal information already present in its training videos.

Even with correct annotations, occlusion and similar appearances make current observations ambiguous. A locally plausible match can contaminate the identity history and cause errors after the objects separate. GT identifies the correct correspondence; future verification reveals how competing decisions affect later identity continuity. Fixed identity bindings and common observable annotations make those consequences comparable. The local teacher then turns them into bounded corrections, helping the original model learn associations that remain correct beyond the ambiguous observation.

Table 1 reports HOTA gains of 1.03–1.36 over native Continue across four tracker–dataset comparisons, with AssA and IDF1 improving throughout. Table 5 reports bidirectional transfer AssA gains of 4.880 and 5.030 with shared coefficients fixed. These results support better use of training evidence and association behavior that transfers across the tested domains.

We see a useful direction here for future unified trackers: a common training principle that turns temporal identity outcomes into supervision across architectures and videos, while keeping deployment simple. Track-OPD takes a step in this direction by making future identity consequences usable for learning within existing end-to-end models.

We will include these analyses and clarify Tables 6 and 8 and Figure 4 in the revision.

References

[1] Peize Sun, Jinkun Cao, Yi Jiang, Zehuan Yuan, Song Bai, Kris Kitani, and Ping Luo. DanceTrack: Multi-object tracking in uniform appearance and diverse motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20993–21002, 2022.

[2] Ruopeng Gao, Ji Qi, and Limin Wang. Multiple object tracking as ID prediction. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 27883–27893, 2025.

Reviewer_CaYZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Track-OPD proposes to improve MOT by exploring alternative association decisions from the same tracker state and learning from their future identity consequences. During training, it compares the resulting continuations using fixed ground-truth identity bindings and the same annotated objects and frames. These comparisons provide relative preferences for refining association targets alongside the tracker's original supervision, while the original online inference computation is retained. The paper reports HOTA gains over continued training of 1.15–1.36 points on DanceTrack across MeMOTR, SambaMOTR, and MOTIP, and 1.03 points on SportsMOT for MOTIP. A controlled comparison using the same selected training records also reports a 0.543 HOTA gain over a control that assigns preference scores of 1 to the GT reference and 0 to competitors within the same target-construction procedure.

Critical strengths

1. The experiments support practical value across multiple modern trackers. Table 1 reports improvements over continued native training for MeMOTR, SambaMOTR, and MOTIP on DanceTrack, and for MOTIP on SportsMOT. HOTA gains range from 1.03 to 1.36 points, with AssA and IDF1 improving in all four comparisons. The bidirectional transfer results in Table 5 provide further support for applicability beyond the training domain. These results demonstrate useful improvements across the evaluated trackers while retaining their original inference computation. This practical contribution is the strongest basis for my positive assessment.

2. The method is promising and has potential to provide deeper insight into MOT training. The paper turns alternative association outcomes into additional supervision through a concrete procedure: compare continuations from the same state under fixed identity bindings, then incorporate the resulting preferences into training alongside the native GT objective. I believe that combining MOT with reinforcement learning is a promising direction that deserves further exploration. The reported improvements across multiple modern trackers provide encouraging evidence for pursuing this direction. They also provide an empirical observation worth understanding: when and why can future-informed supervision improve on ordinary GT training? The method's advantages over related approaches and the explanation of this observation are not yet fully established, but I see potential for a stronger methodological and conceptual contribution beyond the demonstrated practical gains.

Critical weaknesses

1. Why and when the additional supervision helps is not sufficiently explained. The paper provides a reasonable motivation and reports improvements, including a meaningful comparison with fixed GT preferences in Table 2. That control assigns a score of 1 to the GT reference and 0 to competitors on the same selected records while retaining the teacher-construction machinery. As Section 4.3 and Appendix D.1 acknowledge, the future scores affect both target weighting and step selection. Reweighting supervision according to future outcomes may itself be useful, but it remains unclear what these outcome-dependent scores add over a simpler GT-based weighting or target-strength adjustment, and which tracking errors benefit most.

Some gains could also reflect reduced sensitivity to unreliable annotations through the selection and weighting of additional supervision. This would be useful, but the paper does not examine whether improvements concentrate on annotation issues or also occur in correctly annotated but difficult associations. Shared record selection in Table 2 means filtering alone cannot explain the difference there; it does not resolve the possible role of score-dependent weighting. Understanding how persistent identity errors, supervision weighting, and annotation reliability contribute would make the paper's empirical results more informative. These explanations need not be mutually exclusive.

2. The method is conceptually closer to GRPO than conventional OPD, and its methodological advantages need clearer justification. I find the OPD framing insufficiently justified. In the usual OPD formulation, supervision comes from a teacher's conditional predictions on states visited by the student (Agarwal et al., 2024). Here, the guidance originates from scoring alternative continuations and using those scores to reweight the student's current association probabilities. In terms of the source of supervision and the overall learning principle, I find this closer to GRPO: it compares multiple rollouts from the same context by their relative outcomes and uses the resulting signal to adjust action probabilities without a learned critic. This is a comparison of learning principles; the concrete target construction and update rule differ. Calling the resulting distribution a local teacher does not by itself explain why OPD is the most appropriate description. The paper should clarify this distinction and its relationship to reward- or preference-guided policy learning.

The paper discusses LOLS and mentions Xiang et al. (2015), but does not develop the connection to MOT+RL methods such as C-DRL (Ren et al., 2018) and MARLMOT (Rosello and Kochenderfer, 2018). These methods differ in their actions and training procedures, so their relationship cannot be judged from complete-system scores alone. The authors motivate their local teacher's probability and update constraints, but the practical benefits and tradeoffs relative to other reward- or advantage-based updates remain unclear. Explaining these choices would help establish the methodological contribution more firmly.

Questions for authors
  1. What do the future scores add beyond a simpler GT-based weighting or adjustment of target strength? Please explain which tracking errors benefit and how the additional supervision helps. Could part of the improvement come from reduced sensitivity to unreliable annotations, and what evidence distinguishes this from benefits on correctly annotated but difficult associations? I recognize that Table 2 shares selected records, so filtering alone cannot explain that comparison. A focused analysis would help clarify these potentially complementary effects.

  2. I find the method conceptually closer to GRPO-style learning than conventional OPD: it compares alternative rollouts from the same context and learns from their relative outcomes. Why is OPD the more appropriate framing? Please clarify how the GT-constrained teacher and update differ from GRPO and what advantages they offer. I would also like a clearer account of the relationship and tradeoffs relative to earlier MOT+RL approaches such as C-DRL and MARLMOT.

5 comments
AuthorsOpenReview ↗

Responses to Reviewer CaYZ (Part 1 of 2)

Thank you for asking what future outcomes add to GT supervision. We followed your suggestion with a strength-matched GT control and examined persistent identity errors, including difficult clips whose annotations we checked manually.

Question 1 (W1 / Q1): What do future scores add beyond simpler GT-based weighting or target-strength adjustment?

A1. Future scores make the correction specific to the state and the competing identity, with GT defining the correct reference. Confusing two identities can cause a brief deviation or an error that persists after separation. Track-OPD uses that difference in future consequences to set the correction.

For current probabilities pp, aligned future preferences VV, and accepted step η\eta, the target odds within the comparison support are qjql=pjplexp⁡[η(Vj−Vl)].\frac{q_j}{q_l}=\frac{p_j}{p_l}\exp[\eta(V_j-V_l)]. With one competitor, future scores determine a state-dependent correction strength. With several competitors, they also determine the relative probability allocated to each. The values enter both exponential weighting and step selection.

The GT-preference and Track-OPD rows in Table 2 share records, support, confidence, and maturity at 4k updates. The means over seeds 42, 44, and 46 are:

SupervisionHOTAAssAIDF1
Native Continue64.91356.20569.748
Fixed GT preference65.47356.16770.686
Future preference66.01657.35171.395

Future preferences gain 0.543 HOTA, 1.184 AssA, and 0.709 IDF1 over fixed GT. HOTA and AssA improve in all three paired seeds.

To test a simpler strength adjustment, we calibrated one global GT value α≈2.0\alpha\approx2.0 on training data, set competitors to zero, and froze it before evaluation. The control reuses Track-OPD's selected records, comparison support, confidence and maturity, losses, local gradient rule, initialization, and 4k budget; the preference values change. This matches the mean target total variation (TV) of Track-OPD at approximately 0.0843. The seed-42 comparison is:

SupervisionHOTAAssAIDF1
Strength-matched GT65.87356.54071.326
Future preference65.64556.41771.560

Future preferences have 0.234 higher IDF1, while the global control has slightly higher HOTA and AssA. The error analysis identifies the benefit more directly: persistent wrong-identity object–frames decrease from 1,999 to 1,824 at matched average correction strength. Future scores therefore provide state- and competitor-specific information about persistent identity errors beyond a single common preference gap.

Question 2 (W1 / Q1): Which tracking errors benefit, and what distinguishes gains on correctly annotated difficult associations from effects of annotation reliability?

A2. The benefit is also present in difficult clips with verified correct identity annotations. We analyzed 140 fixed, nonoverlapping 30-frame clips from four validation sequences. A persistent error uses the wrong identity for at least two consecutive frames.

Control → Track-OPDCommon GT object–framesPersistent wrong-identity object–framesMissed object–frames
Strength-matched GT → Track-OPD34,8611,999 → 1,8242,178 → 2,056
Fixed GT → Track-OPD34,7961,708 → 1,6722,145 → 2,115

Each row uses the common historically bound GT population for that pair of methods. We then manually verified the GT identities in 4 difficult clips, covering 977 common object–frames:

Control → Track-OPDPersistent wrong-identity object–framesMissed object–frames
Strength-matched GT → Track-OPD12 → 982 → 75
Fixed GT → Track-OPD10 → 982 → 75

The verified-correct clips show fewer persistent errors and missed matches, directly locating a benefit in difficult associations with correct labels. The future signal distinguishes the duration and consequences of competing identity errors. Shared selection in Table 2 holds the retained examples fixed; future scores determine their correction. Figure 5 illustrates the corresponding failure pattern: after overlap, the baseline changes from P6 to P3, while Track-OPD retains P6. The additional supervision helps preserve an identity through an ambiguity whose consequences extend beyond the current frame.

AuthorsOpenReview ↗

Responses to Reviewer CaYZ (Part 2 of 2)

Question 3 (W2 / Q2): Why describe the method as on-policy distillation, and how does its teacher and update relate to GRPO-style learning?

A3. Track-OPD combines outcome-based target construction with on-policy distillation. Algorithm 1 collects histories with the student snapshot, constructs a detached conditional teacher, and distills it alongside native supervision. In GKD [1], a teacher predicts targets on student-generated histories; here, the teacher is constructed from verified future identity consequences.

GRPO [2] is a useful comparison at the level of relative-outcome supervision: it uses group-relative returns in a clipped probability-ratio objective. Track-OPD specifies a constrained target: qk∝pk0exp⁡(ηVk),k∈S.q_k\propto p_k^0\exp(\eta V_k),\qquad k\in S. Here p0p^0 is the detached current distribution. The support contains the GT reference and verified poorer competitors. Its total probability mass and all outside probabilities are preserved, and target TV is bounded by 0.12. These constraints specify which identity probabilities the available evidence should change and by how much. They retain unverified associations in the current distribution and limit the correction to the verified identity comparison. Appendix B.3 derives the target as KL-regularized policy improvement.

The MOT-specific contribution is making the outcomes comparable before this update. Fixed pre-action bindings preserve an identity switch, the common object–frame population retains a lost object, and semantic alignment attaches both consequences to the current identity classes. This construction gives the teacher a consistent identity meaning across collection, evaluation, and learning.

Table 8 already compares the complete target configurations on DanceTrack validation:

TargetHOTAAssAIDF1
Vanilla OPD64.35455.07768.721
OPSD64.05054.44368.449
Track-OPD66.01657.35171.395

Track-OPD improves all three metrics over both alternatives. Table 2 further compares future and unit GT preferences on identical retained records, supporting the combined tracking-specific target design.

Question 4 (W2 / Q2): How does Track-OPD relate to C-DRL and MARLMOT, and what are the practical benefits and costs of its constrained teacher?

A4. Track-OPD post-trains an existing association model while retaining its online computation. The comparison with earlier MOT+RL is:

MethodLearned objectOnline use
C-DRL [3]Collaborative per-object prediction and decision networks, trained with GT status/location rewardsIts learned prediction–decision tracker
MARLMOT [4]A shared TRPO policy for creating, propagating, and terminating Kalman-filter tracks, rewarded for tracking accuracyIts learned track-management policy
Track-OPDConditional association targets for an existing trackerThe host tracker's original online procedure

The design choice is to improve an existing learned association model through a constrained training target. Verified evidence is confined to the identities it evaluates, while native supervision covers all valid predictions. The same teacher construction attaches to MeMOTR, SambaMOTR, and MOTIP, whose online architectures are retained. This retains the host's deployed tracking procedure while adding future continuation and GT verification during training; one measured collection sweep takes 925.352 seconds and verification takes 7.940 seconds on an RTX 3090, with records reused within the epoch. Table 1's four tracker–dataset comparisons show the practical result: 1.03–1.36 HOTA gains over native Continue, with AssA and IDF1 gains throughout.

The practical tradeoff is training-time continuation and verification in exchange for improved associations within the host's original online procedure. We will include these distinctions and the supporting comparisons in the revision.

References

[1] Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations, volume 2024, pp. 21246–21263, 2024.

[2] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models, 2024.

[3] Liangliang Ren, Jiwen Lu, Zifeng Wang, Qi Tian, and Jie Zhou. Collaborative deep reinforcement learning for multi-object tracking. In European Conference on Computer Vision, pp. 586–602, 2018.

[4] Pol Rosello and Mykel J. Kochenderfer. Multi-agent reinforcement learning for multi-object tracking. In Proceedings of the 17th International Conference on Autonomous Agents and Multiagent Systems, pp. 1397–1404, 2018.

AuthorsOpenReview ↗

Follow-up on Our Responses and Manuscript Revisions

Dear Reviewer CaYZ,

Thank you for your thoughtful review. The revised manuscript incorporates the strength-matched GT control and the persistent identity error analysis, including the difficult clips with manually verified GT identities. It also clarifies how future identity outcomes inform the teacher and how this formulation relates to GRPO.

With less than one day remaining before the discussion closes on November 18 (AoE), we would greatly appreciate your feedback on our responses and these updates. Please let us know if any questions remain.

Thank you again for your time and consideration.

Reviewer_JRZROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Track-OPD, an on-policy distillation framework that learns current association decisions from their future consequences in online multi-object tracking. The method generates alternative association trajectories from the same tracker state and evaluates their outcomes under shared future observations and fixed identity bindings. These future identity preferences are then aligned with current association classes to construct a local teacher, which refines association probabilities through bounded posterior distillation while preserving the original online inference procedure. Experiments on DanceTrack and SportsMOT demonstrate improvements in HOTA, AssA, and IDF1 across several tracking architectures. However, the sensitivity to training configurations and the additional computational costs of generating future supervision warrant further investigation.

Critical strengths
  1. The paper addresses an important problem in online multi-object tracking by considering the future consequences of current association decisions. The proposed framework provides a meaningful approach to improving identity consistency by incorporating long-term tracking outcomes into association learning rather than relying solely on immediate correspondence supervision.
  2. The proposed method is technically well motivated and clearly presented. In particular, evaluating alternative associations under fixed identity bindings and common future observations provides a consistent basis for comparing their consequences.
  3. The local teacher provides a controlled way to use future evidence: it changes probabilities only among aligned, verified alternatives while preserving their total probability mass. The method also retains each tracker’s original online inference procedure.
  4. The evaluation goes beyond a single tracker. Results are reported for three tracker adaptations on DanceTrack and for MOTIP on SportsMOT, alongside a matched ground-truth-preference control and analyses of update budget, domain transfer, and motion prediction.
Critical weaknesses
  1. The proposed framework requires generating multiple alternative association trajectories and evaluating their future consequences during training. However, the paper does not provide sufficient analysis of the computational overhead introduced by this process. Although the original online inference procedure remains unchanged, the additional cost of trajectory branching, future verification, and repeated policy collection may be substantial. Reporting total training time, GPU memory consumption, and computational costs relative to conventional training would help clarify the practical efficiency of the proposed method.
  2. The proposed method introduces several mechanisms for constructing and applying future supervision, including semantic alignment, posterior maturity weighting, bounded probability correction, and gradient constraints. While the paper evaluates some of these components, the individual contributions of the future verification channels and their reliability weighting are not sufficiently examined. A more detailed ablation study isolating different verification channels and preference aggregation strategies would help clarify which aspects of future identity information contribute most to the observed improvements.
  3. Table 4 shows that Track-OPD achieves its best performance at 4,000 updates, while extending training to 10,000 updates reduces HOTA from 66.016 to 64.998, AssA from 57.351 to 56.333, and IDF1 from 71.395 to 70.325. Although the authors acknowledge this trend, the underlying reasons for the performance degradation are not sufficiently analyzed. Further investigation into the evolution of future identity preferences and distillation objectives during training would help explain this behavior and clarify the stability of the proposed learning strategy.
Questions for authors
  1. The proposed method applies several eligibility and evidence checks before retaining decisions for supervision, but the paper does not report what proportion of candidate decisions ultimately contributes to training. In addition, future consequences are evaluated using a fixed 15-frame horizon without analyzing sensitivity to this choice. Reporting supervision coverage and evaluating different continuation horizons would help clarify the reliability and applicability of the proposed future supervision, particularly under prolonged occlusion or complex interactions.
  2. Table 5 shows improvements in both transfer directions, but the absolute transferred HOTA remains 42.093 for DanceTrack to SportsMOT and 28.687 in the reverse direction. The discussion should qualify the generalization claim and examine the remaining transfer failures, particularly whether future-verified supervision improves identity association under domain shift or mainly mitigates a weak transferred baseline.
3 comments
AuthorsOpenReview ↗

Responses to Reviewer JRZR (Part 1 of 2)

Thank you for the careful assessment of the future comparisons and local teacher. Below we report the requested cost measurements and analyses of aggregation, training dynamics, and supervision coverage.

Question 1 (W1): What are the total training time, peak GPU memory, and costs of branching, verification, and repeated collection, relative to conventional training?

A1. The additional work is collection and future verification once per frozen snapshot, with the resulting evidence reused throughout that epoch. The tracker reuses saved histories and verified preferences across optimizer updates, while rebuilding the teacher from its current association probabilities. The next epoch refreshes the snapshot and its evidence. With MOTIP's detector frozen, branches also share detector features for the same transformed frames.

We measured the phases on one RTX 3090 using shared feature/history caches and 40 training sequences. This profile contains exactly one preparation sweep, with 117 states and 238 branches at a 15-frame horizon, and a separately measured 4k optimization phase:

PhaseContinue (s)Track-OPD (s)
Loading and training setup22.30922.354
Collection and future continuations, one sweep0.000925.352
Future verification and record construction0.0007.940
4k optimization1,397.2221,252.706

Peak allocated GPU memory is 0.728 GiB during setup and 0.288 GiB during Track-OPD collection; verification runs on the CPU. Both methods reach 8.792 GiB during optimization, the largest measured GPU-stage peak. Optimization takes 10.3% less time for Track-OPD in this profile. Initial common-cache generation and validation evaluation are timed separately. The profile separates reusable supervision preparation from the cost of applying it during optimization.

Question 2 (W2): What do the verification channels and reliability weighting contribute, and how do simpler aggregation strategies perform?

A2. The channels separate observable consequences, and normalization and reliability combine them into one preference. Geometry, identity continuity, interaction, and recoverability share observability across branches. Reliability combines how much evidence is available with how well its scores separate the candidates, so constant-score evidence receives zero weight.

The aggregate in Appendix A.3 can combine repeated identity terms into effective coefficients on the primitive measurements. This unified expression produces exactly the same preferences and teacher. Channel availability, normalization, and reliability determine those coefficients. The comparisons below change that calibration and then reapply selection.

We tested two simpler aggregations on the same sampled states, using seed 42:

AggregationPositive targetsHOTAAssAIDF1
Track-OPD3565.64556.41771.560
Normalized, equal reliability2965.54156.35270.105
Equal average of raw scores265.14356.57169.617

Positive targets count retained states with a nonzero teacher correction. Equal reliability sets available-channel reliability and aggregate confidence to one; raw averaging additionally removes normalization. Both reapply selection.

Track-OPD improves HOTA and IDF1 over both controls. Raw averaging gives slightly higher AssA but retains very few positive targets. Normalization and reliability act on the preferences and the selection of useful comparisons together. This supplementary cohort uses observable geometry, identity/appearance, and interaction evidence. The results support normalized, reliability-weighted evidence for constructing association targets.

AuthorsOpenReview ↗

Responses to Reviewer JRZR (Part 2 of 2)

Question 3 (W3): Why does performance decline after 4k updates? How do future preferences and distillation corrections evolve during training?

A3. The model keeps fitting the training states while the teacher correction becomes smaller. On the same 32 training states:

UpdateMean GT probability over legal candidatesMean teacher TV
4k0.968876430.00636262
7k0.969555810.00212546
10k0.998317320.00014478

Native loss over the preceding 250 updates decreases from 0.00350658 to 0.00192138 and 0.00097470. Predictions receiving a nonzero auxiliary gradient in those intervals decrease from 14 to 10 and 8. By 10k, predictions on these training states concentrate on the GT identity, and the remaining teacher correction is small.

We observe this stronger training fit alongside a smaller validation benefit. Validation in Table 4 peaks at 4k, with HOTA/AssA/IDF1 of 66.016/57.351/71.395, and reaches 64.998/56.333/70.325 at 10k. The gains over equal-budget Continue are 1.103/1.146/1.647 at 4k and 0.064/0.149/0.598 at 10k.

The diagnostic is consistent with increasingly confident training predictions and diminishing auxiliary corrections. The local TV and gradient bounds regulate each correction. The budget study determines how long these corrections improve validation quality as changes accumulate in shared parameters. We use 4k as the best measured post-training budget in this setting.

Question 4 (Q1): How many candidate decisions contribute supervision, and how sensitive are the results to the 15-frame horizon, especially under occlusion?

A4. Future supervision concentrates on ambiguous associations, alongside native supervision on all valid predictions. Across 40 training sequences, 1,092 of 352,541 detection rows meet the ambiguity condition. Sampling yields 114 states: 70 are scorable, 42 pass the evidence checks, and 35 produce positive targets. In 4k training, 398 predictions receive nonzero auxiliary gradients alongside 3,653,538 native supervised predictions.

Within the 114 sampled states, the event counts before and after filtering are:

EventSampled statesEligible recordsPositive targets
Interruption and recovery261715
False new identity241412

The groups can overlap. These counts show how the filters retain these difficult events within the sampled states.

We also tested a 30-frame continuation on the same sampled states. The 15-frame window is its prefix. The paired seed-42 results are:

HorizonPositive targetsHOTAAssAIDF1
15 frames3565.64556.41771.560
30 frames3466.07057.12171.249

The longer horizon improves HOTA and AssA; the default 15-frame horizon has higher IDF1. Both retain similar target coverage.

During occlusion, an annotated object with a missing prediction remains in the common evaluation population and receives zero correctness. Unresolved events at the window boundary follow the same observability rule in every branch. The tested horizons retain similar target coverage, with the reported metric tradeoff; both compare alternatives on the same observable evidence.

Question 5 (Q2): Do the transfer gains reflect improved identity association, and what difficulties remain at the reported absolute performance levels?

A5. Future supervision improves identity association in both transfer directions. Table 5 reports:

Transfer directionAssA: Continue → Track-OPDIDF1: Continue → Track-OPD
DanceTrack → SportsMOT36.980 → 41.86038.031 → 43.831
SportsMOT → DanceTrack15.892 → 20.92225.795 → 32.842

Transferred HOTA reaches 42.093 and 28.687, respectively. DetA changes from 41.172 to 42.405 and from 37.246 to 39.575, while AssA reaches 41.860 and 20.922. The larger AssA gains directly show improved identity association in both directions, with reverse transfer remaining more difficult.

Both transfer directions keep the coefficients selected on DanceTrack validation fixed, with no additional tuning for either transfer direction. Table 1 also reports gains over native Continue in all four tracker–dataset comparisons. Together, Tables 1 and 5 show improved associations both in-domain and under the tested domain shifts.

We will incorporate the cost profile, supervision-coverage and horizon results, training diagnostics, and this transfer interpretation into the revision.

AuthorsOpenReview ↗

Follow-up on Our Responses and Manuscript Revisions

Dear Reviewer JRZR,

Thank you for your careful review and helpful suggestions. The revised manuscript includes measurements of training time and memory use, supervision coverage, and the comparison between 15-frame and 30-frame horizons. We have also expanded the analysis of training dynamics and association performance under domain transfer.

With less than one day remaining before the discussion closes on November 18 (AoE), we would greatly appreciate your feedback on our responses and these updates. We would be happy to clarify any remaining questions.

Thank you again for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final discussion and rebuttal summary

Dear Area Chair and Reviewers,

Thank you for your careful reviews and constructive discussion, which helped us strengthen the evidence and improve the manuscript.

Track-OPD explores how a tracker can learn from the identity history its own decisions create. Alternative associations can change both which predictions survive and which objects the resulting track identities refer to. Fixed identity bindings and common observable GT object-frame pairs make these outcomes comparable. Semantic alignment maps verified future preferences to current association classes, and a bounded local teacher turns them into supervision. The original tracker architecture, parameter count, and online inference remain unchanged.

1. Progress Through Reviewer Discussions

We thank Reviewer CaYZ for recognizing the practical gains across modern trackers and the potential of future-informed supervision for understanding MOT training. We added a strength-matched GT control and persistent-identity-error analysis, including difficult validation clips with manually verified GT identities. We also clarified the relationship to GRPO, C-DRL, and MARLMOT and the scope of Table 8. CaYZ subsequently appreciated these analyses, the channel ablations, KL-only comparison, and manuscript clarifications, stating that they meaningfully strengthened the empirical support and addressed several gaps in the original discussion.

We thank Reviewer JRZR for recognizing the technical motivation, clear presentation, consistent comparison of alternatives under fixed identity bindings, and preservation of original online inference. In response, we supplied phase-by-phase runtime and memory measurements, aggregation controls, training-dynamics diagnostics, supervision-coverage statistics, and a 15/30-frame comparison. We also analyzed AssA and IDF1 in both transfer directions to characterize improvements in identity association.

We thank Reviewer doZx for recognizing the innovative idea of allocating supervision according to the future consequences of errors. We completed four leave-one-channel-out ablations over seeds 42, 44, and 46. Removing any channel reduces the three-seed means of HOTA, AssA, and IDF1, while the full method achieves the highest three-seed mean for all three metrics. We also supplied simpler aggregation and KL-only controls and cost measurements, and clarified the descriptions in Tables 6 and 8 and Figure 4. In the follow-up, doZx described the ablations as a positive step and acknowledged the additional experiments and manuscript revisions.

2. Stronger Experimental Evidence and Manuscript Quality

  • Controlled supervision comparisons. On identical retained records, Table 2 reports three-run mean gains over fixed GT preferences of 0.543 HOTA, 1.184 AssA, and 0.709 IDF1. The strength-matched control and persistent-error analysis further characterize where future-aware supervision helps.
  • Cross-architecture effectiveness. Table 1 reports 1.03–1.36 HOTA gains over native Continue across four tracker-dataset comparisons, with AssA and IDF1 improving in all four.
  • Cross-domain generalization. With shared coefficients and no transfer-specific tuning, gains over Continue are 3.109 HOTA, 4.880 AssA, and 5.800 IDF1 from DanceTrack to SportsMOT, and 4.432 HOTA, 5.030 AssA, and 7.047 IDF1 in the reverse direction.
  • Practical efficiency. On the reported RTX 3090 setup, cached 4k optimization takes 10.3% less time than Continue, with the same peak allocated GPU memory during optimization. Preparation and verification costs are reported separately.

We have revised and checked the experimental descriptions, terminology, and reported values across the main text, captions, and appendix, and identified supplementary experiments separately. We take responsibility for the accuracy and verification of the reported work.

3. Broader Contribution and Research Potential

The central contribution is a shared learning principle that converts future identity consequences into supervision across tracking architectures while retaining their original online computation. The controlled comparisons support future-aware supervision, the multi-tracker experiments demonstrate adaptability, and the bidirectional transfer results support generalization under the tested domain shifts.

Together, these findings motivate fuller use of temporal identity information already present in annotated videos and offer a research direction toward more generalizable end-to-end MOT models that learn from the future consequences of their own decisions.

The revised manuscript is available on OpenReview. Red text marks revisions, and blue text marks additions. If accepted, we will integrate these updates into the camera-ready paper. Thank you again to the Area Chair and all three reviewers for helping us strengthen the evaluation and clarity of the paper.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.