acceptodds
Withdrawn from ICLR 2027

Sequence-Level Post-Training for Visual Tracking

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual trackers are typically trained with frame-level objectives but deployed with sequential inference, where every prediction determines the next search region. This mismatch limits their ability to optimize the behavior encountered at test time. We present Sequence-level Policy Optimization for Tracking (SPOT), a framework for sequence-level post-training. SPOT represents a tracker's native localization output as a stochastic policy, rolls out a group of trajectories under the same prediction-driven cropping mechanism used at inference, and optimizes trajectory-level IoU returns with a critic-free policy gradient. Multiple trajectories provide relative supervision, while frozen-reference KL regularization constrains the updated tracker near its pretrained behavior. Through a unified policy interface, the same framework applies to one-stream, autoregressive, and memory-based trackers without redesigning or modifying their backbone or prediction-head architectures, or changing their test-time inference pipelines. SPOT consistently improves their supervised base trackers across multiple tracking benchmarks. Code will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
visual object tracking, reinforcement learning, sequence-level post-training, policy gradient

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3Hw9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SPOT, a sequence-level post-training framework for visual tracking. It treats native localization outputs as stochastic policies and samples tracking trajectories using prediction-driven search crops. Training combines trajectory-level IoU returns, group-relative advantages, batch-global normalization, and KL regularization against a frozen reference tracker. Only localization-related parameters are updated, leaving the architecture and test-time procedure unchanged. Experiments on OSTrack, ARTrackV2, and MCITrack show consistent improvements across four benchmarks.

Critical strengths
  1. SPOT addresses the mismatch between supervised training and prediction-driven tracking, where localization errors affect subsequent search regions. It improves existing trackers without changing their test-time procedures or adding sampling and post-processing at inference. This makes the approach practical to adopt.
  2. The evaluation covers one-stream, autoregressive, and memory-based trackers, with improvements across all four benchmarks. The controlled SCST comparison and the ablations on crop transitions and trainable parameters provide useful evidence beyond the main benchmark results.
Critical weaknesses
  1. Limited novelty over prior work. SLT already uses prediction-driven rollouts and trajectory-level rewards. SPOT mainly introduces group-relative advantages, global normalization, and reference KL. However, the SCST comparison changes all these components together, making it difficult to identify where the improvement comes from. A comparison with SCST using the same KL and normalization would be helpful.
  2. Lack of a supervised fine-tuning baseline. The paper does not compare SPOT with continued supervised fine-tuning under the same training budget. In particular, an SFT baseline using prediction-driven crops would help clarify whether the improvements come from state-distribution alignment or the proposed sequence-level optimization.
  3. Insufficient analysis of long-term tracking. Although reducing error accumulation is a key motivation, SPOT only trains on short trajectories (8–12 frames). The cumulative-prefix AUC results in Figure 3 do not directly show reduced tracking drift. A more direct analysis of tracking failures or error accumulation would strengthen this claim.
Questions for authors
  1. Does SPOT still outperform SCST when both use the same KL regularization and advantage normalization?
  2. How does SPOT compare with continued supervised fine-tuning using the same checkpoint, data, trainable parameters, and training budget? A version using prediction-driven crops would help separate the benefit of state-distribution alignment from that of the trajectory-level objective.
  3. Can the authors provide more direct evidence that SPOT reduces long-term tracking drift?
Reviewer_chGWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SPOT, a sequence-level post-training framework for visual tracking. It models native tracker outputs as stochastic policies and optimizes prediction-driven trajectories using group-relative advantages and reference-policy regularization. SPOT is applied to OSTrack, ARTrackV2, and MCITrack without changing their test-time inference procedures. Experiments on four tracking benchmarks demonstrate consistent improvements over the corresponding base trackers.

Critical strengths
  1. SPOT unifies heterogeneous native localization outputs asstochastic policies and optimizes prediction-driven trajectories with a critic-free score-function policy gradient on trajectory level IoU returns.
  2. Experiments on three configurations spanning one-stream, autoregressive, and memory-based trackers, validating the effectiveness of SPOT.
Critical weaknesses
  1. The manuscript appears somewhat under-polished and contains several inconsistencies in its presentation. For example, the initial state is denoted as s0s_0 in Fig. 2 but as s1s_1 in the main text. The reference list also contains several method names whose capitalization is inconsistent with that used in the original publications, as well as entries with missing page ranges. The authors are encouraged to carefully proofread the manuscript and correct these inconsistencies.

  2. The manuscript does not provide sufficient information on the training and computational costs. Since the proposed training strategy constitutes a central contribution, its computational overhead should be reported explicitly. In particular, the grouped trajectory rollout requires GG branches to be processed simultaneously and is therefore likely to incur considerable computational and memory costs. The authors should report the wall-clock training time, GPU hours, peak memory consumption, and computational overhead relative to conventional fine-tuning.

  3. The claimed unified nature of the proposed framework is debatable. The method still relies substantially on model-specific designs, as different trackers require customized action distributions, trainable parameter subsets, and state-propagation mechanisms. Therefore, the framework is unified primarily at the level of the optimization objective, whereas its policy parameterization remains architecture-dependent. The authors should clarify the precise scope of this unification and moderate the corresponding claims.

Questions for authors
  1. The manuscript appears somewhat under-polished and contains several inconsistencies in its presentation. For example, the initial state is denoted as s0s_0 in Fig. 2 but as s1s_1 in the main text. The reference list also contains several method names whose capitalization is inconsistent with that used in the original publications, as well as entries with missing page ranges. The authors are encouraged to carefully proofread the manuscript and correct these inconsistencies.

  2. The manuscript does not provide sufficient information on the training and computational costs. Since the proposed training strategy constitutes a central contribution, its computational overhead should be reported explicitly. In particular, the grouped trajectory rollout requires GG branches to be processed simultaneously and is therefore likely to incur considerable computational and memory costs. The authors should report the wall-clock training time, GPU hours, peak memory consumption, and computational overhead relative to conventional fine-tuning.

  3. The claimed unified nature of the proposed framework is debatable. The method still relies substantially on model-specific designs, as different trackers require customized action distributions, trainable parameter subsets, and state-propagation mechanisms. Therefore, the framework is unified primarily at the level of the optimization objective, whereas its policy parameterization remains architecture-dependent. The authors should clarify the precise scope of this unification and moderate the corresponding claims.

Reviewer_SyMjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Sequence-level Policy Optimization for Tracking (SPOT), a sequence-level post-training framework that addresses the mismatch between frame-level training and sequential inference in visual tracking. SPOT models a tracker’s native localization output as a stochastic policy and samples groups of trajectories using prediction-driven cropping. It optimizes trajectory-level IoU returns through critic-free policy gradients, using relative supervision across trajectories and frozen-reference KL regularization. Experiments across multiple tracking benchmarks show consistent improvements over the corresponding supervised baselines.

Critical strengths
  1. This paper addresses the mismatch between tracker training and sequential inference through head-only post-training, without changing the architecture or test-time pipeline.

  2. Experiments show gains across three tracker configurations and four benchmarks under matched inference settings. Additional analysis supports better accuracy over longer tracking horizons.

Critical weaknesses
  1. Table 2 shows that SPOT improves over SCST by only 0.6 AO points, while simultaneously changing the group baseline, global normalization, and reference KL regularization. Table 4 reports an AO of 70.1 without KL, below the Base result of 70.5, suggesting that KL regularization is critical under the current setting. However, the paper lacks closely matched controls, such as SCST with the same KL regularization and variants that change only the normalization scheme or only the group baseline. Thus, the contribution of each SPOT-specific design choice remains unclear.

  2. The paper also lacks a continued supervised fine-tuning baseline using the same data and trainable parameters. Without this comparison, it is difficult to determine whether the observed gains come from SPOT’s sequence-level optimization or from additional training.

Questions for authors
  1. Does SPOT still outperform SCST when both use the same reference KL regularization? Could you isolate the effects of the group baseline and global normalization by changing one component at a time?

  2. How does SPOT compare with continued supervised fine-tuning using the same data, trainable parameters, and training budget? Does its advantage remain after controlling for additional training?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.