acceptodds
Under review as a conference paper at ICLR 2027

TGTP: General Trajectory-Guided Token Pruning for Visual Tracking via Reinforcement Learning

Abstract

Token pruning in visual tracking aims to reduce computational cost by removing redundant tokens, with the key challenge lying in reliable token importance estimation. Existing methods often rely on current-frame cues such as attention responses, token similarity, or static pruning criteria, while making limited use of the temporal continuity inherent to tracking. Since target position, scale, and motion evolve continuously over time, historical trajectories can provide useful priors for token selection. To this end, we propose Trajectory-Guided Token Pruning (TGTP), a general reinforcement learning-based token pruning framework for visual tracking. The core of TGTP is a Trajectory-Guided Reinforcement Learning (TGRL) module, which formulates token pruning as a sequential decision-making problem. TGRL constructs states from recent target trajectories and learns pruning actions through a Trajectory Policy Network, predicting motion priors, motion uncertainty, and token keep ratios for the current frame. The policy is optimized with rewards that balance tracking accuracy, FLOPs, uncertainty regularization, and target-token preservation. As a lightweight plug-in module, TGTP can be integrated into different tracking architectures without redesigning their original structures. Experiments on multiple single-modal and multi-modal baselines show that TGTP substantially reduces search tokens and model FLOPs while maintaining or improving tracking performance, demonstrating its effectiveness and generality for computation-efficient visual tracking.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual Object Tracking, Token Pruning, Reinforcement Learning, Trajectory-Guided Pruning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_iFhXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TGTP, a trajectory-guided token pruning module for visual tracking. A lightweight policy uses recent bounding-box histories to predict motion priors, uncertainty, and token keep ratios. These priors are combined with template–search attention to select search tokens. Experiments across RGB and multimodal trackers show reduced computational cost while generally maintaining or improving tracking accuracy.

Critical strengths

1.A practical design that fits the tracking problem. Historical trajectories provide a useful spatial cue alongside current-frame attention. The policy does not require additional heavy visual features, and the pruning module can be integrated into existing trackers without redesigning their overall architecture. This makes the approach useful beyond a single tracker.
2. Broad evaluation with measurable efficiency gains. The experiments cover several trackers, backbones, and modalities. For SUTrack-B224, FLOPs decrease from 19.2G to 12.9G while LasHeR AUC improves from 59.9 to 62.0. The CPU and GPU speed results in Table 7 also provide evidence of practical acceleration, rather than FLOP reductions alone.

Critical weaknesses
  1. The construction of training trajectories is unclear. The paper does not clearly state whether historical boxes come from annotations, precomputed predictions, or the current model’s sequential predictions. This matters because errors in test-time histories may affect subsequent pruning. Also, the stated objective uses only the current-frame reward, leaving unclear whether training accounts for how pruning affects later tracking results. These details are important for reproducing the method and understanding its sequential decision-making formulation.
  2. The added value of reinforcement learning is not isolated. The ablations examine several design choices, but do not distinguish the benefit of reinforcement learning from that of adding a motion prior. A baseline using simple motion extrapolation with the same attention–prior fusion would help. It is also unclear whether the original trackers receive the same additional fine-tuning budget. Without these controls, some of the accuracy gains remain difficult to attribute.
Questions for authors
  1. Training clarification: How are historical boxes generated during training? Are the model’s own predictions fed into subsequent training frames?
  2. Objective clarification: Is the policy intended to optimize only the immediate reward? How does this choice relate to the sequential decision-making formulation?
  3. Additional evidence: How does TGTP compare with simple motion-extrapolation-based pruning under matched computational and fine-tuning budgets? Please also confirm whether the original baselines receive equivalent additional fine-tuning.
2 comments
AuthorsOpenReview ↗

Responses to Reviewer iFhX (Part 1)

Weakness 1 and Questions 1-2: The source of historical boxes during training and the relationship between the per-frame reward and sequential decision-making need clarification.

Response: We thank the reviewer for raising these questions. We address them separately below.

  • Question 1 (Training trajectories). For video sequences, we sample historical visible frames preceding the current search frame and use their ground-truth boxes to construct the trajectory state. These boxes are transformed into the coordinate system of the current search region. The model's predictions are not fed into subsequent training frames. During inference, the history is instead updated with the tracker's predicted boxes. We have described this construction and the training-inference distinction in Appendix A.2 of the revised supplementary material.

  • Question 2 (Optimization objective). The policy is optimized using the reward from the current frame. Rewards from later frames are not propagated back to earlier pruning decisions. By sequential decision-making, we mean that the policy acts at each frame using a trajectory state updated with previous tracking results, not that it optimizes cumulative future rewards. Errors in those results may affect later token selection. To mitigate this, the policy predicts motion uncertainty to form a soft spatial prior, which is fused with template-to-search attention from the current frame before pruning. Current visual evidence therefore also contributes to token selection when the historical trajectory is inaccurate. We have clarified the per-frame objective in the Method section of the revised manuscript.

Once the paper is accepted, we will release the code and experimental configurations to support reproducibility.

AuthorsOpenReview ↗

Responses to Reviewer iFhX (Part 2)

Weakness 2 and Question 3: The added value of RL is not clearly isolated from that of a simple motion prior. A comparison with pruning based on simple motion extrapolation, using the same fusion of attention responses and motion priors under a matched computational budget, would help clarify this. It is also unclear whether the original trackers receive equivalent additional fine-tuning, making it difficult to attribute the accuracy gains to TGTP.

Response: We thank the reviewer for suggesting controls that distinguish the effects of RL, a simple motion prior, and additional fine-tuning. We address these points separately.

RL vs. simple motion extrapolation. As shown in the table below, linear extrapolation with a fixed keep ratio achieves 61.4 AUC and 85.0 P at 2.02G FLOPs. Combined with a rule-based adaptive ratio, it achieves 61.7 AUC and 85.2 P at 1.94G FLOPs. TGRL jointly learns the motion prior and frame-wise keep ratio, achieving 62.4 AUC and 85.5 P at 1.91G FLOPs. It also runs at 238fps, compared with 230fps for the rule-based strategy. Among the strategies tested, TGRL achieves higher accuracy with less computation, so its gain is not due to a larger token budget. We have added this comparison and analysis to Appendix A.5 of the revised manuscript.

OptimizationMotion priorKeep-ratio strategyUAVDT AUCUAVDT PFLOPs (G) ↓Speed (fps) ↑
Non-RLLinear extrapolationFixed61.485.02.02220
Non-RLLinear extrapolationRule-based adaptive61.785.21.94230
RLTGRL (ours)Learned adaptive62.485.51.91238

Effect of additional fine-tuning. Our primary goal is to reduce FLOPs and improve inference speed through token pruning while preserving tracking accuracy. To control for additional training, we include head-only fine-tuning controls for FERMT and MCITrack. Each FT variant starts from the same pretrained weights as its TGTP counterpart and uses the same training data and 30-epoch fine-tuning schedule. As shown in the table below, FERMT-FT increases LaSOT AUC from 65.1 to 65.4, while its GOT-10k AO decreases from 69.6 to 69.1. MCITrack-FT decreases LaSOT AUC from 75.3 to 74.7 and GOT-10k AO from 77.9 to 77.2. FERMT-TGTP reduces FLOPs from 2.39G to 2.02G, a 15.5% reduction, and increases speed from 181fps to 199fps while reaching 66.7 LaSOT AUC. MCITrack-TGTP reduces FLOPs from 38.5G to 34.5G, a 10.4% reduction, and increases speed from 30fps to 34fps. Its LaSOT AUC of 75.1 is slightly below the original 75.3, while its GOT-10k AO improves from 77.9 to 80.6. These controls show that additional fine-tuning alone leads to only small, inconsistent changes in tracking accuracy and does not produce the efficiency gains. We have added the fine-tuning controls and their analysis to Appendix A.5 of the revised manuscript.

MethodLaSOT AUCLaSOT PNLaSOT PGOT-10k$^\ast$ AOGOT-10k$^\ast$ SR50GOT-10k$^\ast$ SR75LaSOT$_{ext}$ AUCLaSOT$_{ext}$ PNLaSOT$_{ext}$ PFLOPs (G) ↓Speed (fps) ↑
FERMT65.174.669.169.680.163.246.1--2.39181
FERMT-FT65.474.969.369.179.164.046.2--2.39181
FERMT-TGTP66.776.671.471.381.966.846.6--2.02199

MethodLaSOT AUCLaSOT PNLaSOT PGOT-10k$^\ast$ AOGOT-10k$^\ast$ SR50GOT-10k$^\ast$ SR75LaSOT$_{ext}$ AUCLaSOT$_{ext}$ PNLaSOT$_{ext}$ PFLOPs (G) ↓Speed (fps) ↑
MCITrack75.385.683.377.988.276.854.665.762.138.530
MCITrack-FT74.785.182.777.287.576.054.065.161.538.530
MCITrack-TGTP75.185.482.980.690.980.654.765.762.534.534
Reviewer_VPykOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

TGTP exploits short-term target motion trajectories to guide token pruning for efficient visual tracking. A lightweight policy network predicts target displacement, motion uncertainty, and the token retention ratio from recent target states, and combines the resulting spatial prior with template-to-search attention to preserve informative search tokens. The policy is optimized via reinforcement learning by jointly considering localization accuracy, computational cost, and target-relevant token retention. As a plug-and-play module requiring only a single policy forward pass at inference, TGTP reduces computation across multiple backbones and RGB, thermal, depth, and event-based tracking while generally maintaining or improving accuracy.

Critical strengths
  1. Leveraging motion information to guide token pruning is well motivated. By combining trajectory-derived spatial priors with attention scores and dynamically adjusting the token retention ratio, the method provides temporal cues for identifying regions that should be preserved.
  2. The module is lightweight and highly compatible, enabling integration with various existing trackers without redesigning their core architectures. It introduces only a small number of additional parameters and also delivers measurable improvements in practical inference speed.
  3. The experiments are comprehensive, demonstrating a favorable balance between accuracy and efficiency across multiple trackers and diverse tracking modalities.
Critical weaknesses
  1. The necessity of reinforcement learning is not sufficiently justified. The optimization largely relies on per-frame rewards, which does not clearly demonstrate the benefit of long-term decision making, and direct comparisons with simple motion prediction or non-RL pruning strategies are lacking.
  2. The comparison of computational budgets is insufficient. The ablation between fixed retention ratios and the adaptive strategy does not report their respective computational costs, making it difficult to determine whether the accuracy gains of the adaptive strategy stem from increased computation.
Questions for authors
  1. ETBTrack[1] has already optimized token selection using localization-accuracy rewards and supports compression of both template and search tokens under elastic budgets. Why is this closely related work neither cited nor compared against, and what is the key additional contribution of TGTP over ETBTrack? Moreover, multi-template inputs may contain substantial appearance redundancy and irrelevant background, yet TGTP always retains all template tokens. Is this design choice empirically justified?
  2. Can the authors provide efficiency evaluations on mobile or embedded devices, reporting tracking accuracy, end-to-end latency, FPS, and peak memory usage, while accounting for the overhead of the policy network, token sorting, and dynamic indexing operations?
  3. TGTP dynamically selects the token retention ratio from {0.7, 0.75, 0.8}, whereas the compared OSTrack baseline uses a fixed 0.7 retention ratio at all three stages. Since the two settings may differ in the actual number of retained tokens and computational budget, how is the fairness of the speed–accuracy comparison ensured? Can the authors further compare them under a matched average computational budget to disentangle the benefit of dynamic allocation from that of simply retaining more tokens?

[1] Learning Generalized Trackers with Elastic Token Budgets.

4 comments
AuthorsOpenReview ↗

Responses to Reviewer VPyk (Part 1)

Weakness 1: The necessity of reinforcement learning is not sufficiently justified. The optimization largely relies on per-frame rewards, which does not clearly demonstrate the benefit of long-term decision making, and direct comparisons with simple motion prediction or non-RL pruning strategies are lacking.

Response: We thank the reviewer for this suggestion. TGRL builds its state from the recent KK target boxes and predicts a motion prior and token keep ratio at each frame. The current policy is optimized with a per-frame reward, so we do not attribute its gains to long-horizon credit assignment. Reinforcement learning instead allows us to jointly optimize continuous motion parameters and a discrete keep ratio using rewards for tracking accuracy, computational cost, and target-token preservation.

To address the reviewer's concern, we compared TGRL with two non-RL trajectory-based pruning strategies, as shown in the table below. Compared with linear extrapolation and rule-based adaptive pruning, TGRL improves AUC from 61.7 to 62.4, reduces FLOPs from 1.94G to 1.91G, and increases speed from 230 to 238 fps. Among the strategies tested, TGRL achieves the best accuracy-efficiency balance. We have added this comparison to Appendix A.5 of the revised manuscript and clarified the distinction between per-frame reward optimization and sequential tracking in the Method section.

OptimizationMotion priorKeep-ratio strategyUAVDT AUCUAVDT PFLOPs (G) ↓Speed (fps) ↑
Non-RLLinear extrapolationFixed61.485.02.02220
Non-RLLinear extrapolationRule-based adaptive61.785.21.94230
RLTGRL (ours)Learned adaptive62.485.51.91238

Weakness 2 and Question 3: The fixed-versus-adaptive keep-ratio ablation does not report computational costs. Since TGTP selects from {0.7,0.75,0.8}\{0.7,0.75,0.8\} while the compared OSTrack setting uses a fixed ratio of 0.7, the accuracy gains may reflect different token budgets rather than frame-wise adaptation. Could the authors report efficiency for each setting and compare fixed and adaptive strategies under matched average computational cost?

Response: We thank the reviewer for this suggestion. To separate the effect of adaptive keep-ratio selection from that of retaining more tokens, we compare fixed ratios of 0.70, 0.75, and 0.80 with the adaptive strategy on OSTrack-256, reporting average FLOPs and tracking speed in the table below. We also select a fixed ratio r∗r^\ast on validation data to match the adaptive strategy's average FLOPs. At the same average cost of 22.5G FLOPs, the adaptive strategy improves LaSOT AUC from 68.2 to 68.6, PN from 77.6 to 78.1, and P from 73.9 to 74.5, while running at a similar speed. The fixed 0.80 strategy uses more computation (23.7G FLOPs) but obtains a lower AUC of 68.3. These results show that the accuracy gain is not simply due to a larger average computational budget and support the benefit of frame-wise keep-ratio selection. We have added the relevant results and analysis to Appendix A.5 of the revised manuscript.

#Keep-ratio strategyLaSOT AUCLaSOT PNLaSOT PAvg. FLOPs (G) ↓Speed (fps) ↑
1Fixed 0.7067.977.273.521.5143
2Fixed 0.7568.077.373.522.7141
3Fixed 0.8068.377.573.923.7137
4Fixed r∗r^\ast (matched)68.277.673.922.5140
5Adaptive (Ours)68.678.174.522.5141
AuthorsOpenReview ↗

Responses to Reviewer VPyk (Part 2)

Question 1: ETBTrack has already optimized token selection using localization-accuracy rewards and supports compression of both template and search tokens under elastic budgets. Why is this closely related work neither cited nor compared against, and what is the key additional contribution of TGTP over ETBTrack? Moreover, multi-template inputs may contain substantial appearance redundancy and irrelevant background, yet TGTP always retains all template tokens. Is this design choice empirically justified?

Response: We thank the reviewer for the reminder. We had not identified ETBTrack when preparing the original submission, which is why it was omitted from the related work. We have now cited and discussed it in the revised manuscript. Unlike ETBTrack, TGTP uses recent target trajectories to determine both where to retain search tokens and how many to retain in the current frame. Specifically, the policy predicts motion offset and spatial uncertainty from historical target states, fuses the resulting motion prior with current-frame template-to-search attention for token ranking, and selects a frame-wise keep ratio. To the best of our knowledge, TGTP is the first to use explicit historical trajectories as the policy state to jointly learn motion guidance and a frame-wise search-token retention ratio.

Regarding template retention, TGTP prunes only search-region tokens to preserve the original tracker's target reference representation and template-to-search interaction mechanism. This design applies to both single-template and multi-template baselines. Our experiments cover both settings and show that TGTP reduces FLOPs while maintaining competitive tracking performance, supporting the applicability of this design across different template settings. Template compression could be combined with TGTP in future work.

Question 2: Can the authors provide efficiency evaluations on mobile or embedded devices, reporting tracking accuracy, end-to-end latency, FPS, and peak memory usage, while accounting for the overhead of the policy network, token sorting, and dynamic indexing operations?

Response: We thank the reviewer for the suggestion. We evaluated FERMT and FERMT-TGTP under the same settings on an RTX 3090 GPU, an R5300 G5 8358 CPU, and a Jetson TX2. As shown in the table below, FERMT-TGTP improves LaSOT AUC from 65.1 to 66.7 while reducing FLOPs from 2.39G to 2.02G, a 15.4% reduction. On the TX2, tracking speed increases from 22 fps to 25 fps. Speed also increases from 181 fps to 199 fps on the RTX 3090 and from 32 fps to 40 fps on the CPU. The reported GPU latency decreases from 5.52 ms to 5.03 ms per frame, while peak memory decreases from 516 MB to 508 MB. These results show that TGTP reduces computation and improves tracking accuracy while delivering a speedup on the embedded device.

MethodLaSOT AUCLaSOT PNLaSOT PFLOPs (G) ↓GPU peak memory (MB) ↓GPU latency (ms/frame) ↓GPU speed (fps) ↑CPU speed (fps) ↑TX2 speed (fps) ↑
FERMT65.174.669.12.395165.521813222
FERMT-TGTP66.776.671.42.025085.031994025

Thank you for the detailed response and additional experiments. The comparisons with non-RL methods help clarify why the learned policy is useful, and the FLOPs results show that adaptive token retention can improve accuracy without increasing the average computational cost. I also appreciate the Jetson TX2 results and the explanations of how TGTP differs from ETBTrack and why all template tokens are kept.

Overall, my main concerns have been addressed, and I will keep my rating as weak acceptance.

AuthorsOpenReview ↗

Thank You for Your Feedback

Dear Reviewer VPyk,

Thank you for your thoughtful follow-up and for taking the time to examine our responses and additional experiments. We appreciate your positive assessment of our work and your recognition of the non-RL comparisons, matched-budget analysis, and Jetson TX2 results.

We are glad that our responses addressed your main concerns. Thank you again for your careful review and constructive feedback.

Best regards,

The Authors

Reviewer_xJYVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Trajectory-Guided Token Pruning (TGTP) to reduce computation in visual tracking. It assumes that target motion is relatively smooth across frames, so past trajectories can help identify important tokens. The method uses historical target positions and current-frame attention to guide token pruning. An RL policy predicts target motion, uncertainty, and how many tokens to keep. The reward considers tracking accuracy, computation cost, and target-token preservation. Experiments cover several RGB and multi-modal tracking datasets, including LaSOT, GOT-10k, and LasHeR. The authors compare TGTP with original trackers and existing pruning methods such as EViT, ToMe, and UTP. Results show FLOPs reductions of 10.4%–32.8%, with mostly similar or better tracking accuracy. Ablation studies also support the main design choices, although actual inference speed is not reported.

Critical strengths

The paper presents a solution to an important problem in visual tracking by reducing redundant tokens and computation while keeping tracking accuracy.

The method combines historical trajectories with current-frame attention to guide token selection. It also uses RL to adjust the pruning ratio based on target motion.

The experiments cover several tracking models and datasets, including both RGB and multi-modal tracking. Results show consistent FLOPs reductions with mostly maintained or improved accuracy. The ablation studies also provide some support for the design choices.

Critical weaknesses

The need for RL is not well justified. Simpler methods, such as rule-based pruning or dynamic sampling, might achieve similar gains. The paper does not include direct comparisons to show why RL is necessary.

The method offers limited novelty. Historical trajectory guidance and token pruning have already been explored. The proposed approach mainly combines existing techniques without much new technical insight.

The practical efficiency gains remain unclear. TGTP adds extra steps, including RL policy inference, motion prior generation, attention fusion, and Top-K selection. These steps introduce overhead, but the paper only reports FLOPs reduction. Without FPS or latency measurements, it is hard to know whether the method actually runs faster than existing pruning methods.

The experimental controls could be stronger. The authors fine-tune the tracking head along with the RL module. It is unclear how much of the accuracy improvement comes from the proposed method rather than additional training.

The method depends on historical tracking results, which may not always be reliable. Errors from earlier frames could affect later token selection, especially during fast motion or occlusion. The paper provides limited analysis of these cases.

Questions for authors

Could the authors compare TGTP with simpler trajectory-based pruning methods without RL? This would help clarify whether RL is necessary for the reported improvements.

Could the authors report FPS or end-to-end inference latency under the same hardware settings, especially compared with UTP? This would help show whether the FLOPs reduction leads to actual speedup.

Could the authors provide a controlled experiment that fine-tunes the tracking head without TGTP? This would help separate the benefits of additional training from those of the proposed pruning method.

3 comments
AuthorsOpenReview ↗

Responses to Reviewer xJYV (Part 1)

Weakness 1 and Question 1: The need for RL is not sufficiently justified without direct comparisons to simpler trajectory-based pruning strategies that do not use RL, such as rule-based adaptive pruning, which might achieve similar gains.

Response: We thank the reviewer for suggesting this comparison. As shown in the table below, linear extrapolation with a fixed keep ratio achieves 61.4 AUC and 85.0 P at 2.02G FLOPs. Combining linear extrapolation with a rule-based adaptive ratio improves the results to 61.7 AUC and 85.2 P at 1.94G FLOPs. TGRL jointly learns the motion prior and keep ratio, achieving 62.4 AUC and 85.5 P at 1.91G FLOPs. Compared with the rule-based alternative, it improves AUC by 0.7 and P by 0.3, while reducing FLOPs by 0.03G and increasing speed by 8 fps. These results demonstrate the advantage of the RL-based policy over the tested non-RL trajectory-based strategies. We have added this comparison to Appendix A.5 of the revised manuscript.

OptimizationMotion priorKeep-ratio strategyUAVDT AUCUAVDT PFLOPs (G) ↓Speed (fps) ↑
Non-RLLinear extrapolationFixed61.485.02.02220
Non-RLLinear extrapolationRule-based adaptive61.785.21.94230
RLTGRL (ours)Learned adaptive62.485.51.91238

Weakness 2: The method offers limited novelty. Historical trajectory guidance and token pruning have already been explored. The proposed approach mainly combines existing techniques without much new technical insight.

Response: We thank the reviewer for the opportunity to clarify the methodological novelty of TGTP.

  • Trajectory-conditioned hybrid policy. To the best of our knowledge, TGTP is the first visual-tracking token-pruning method to use explicit historical target trajectories as the state of a hybrid-action reinforcement learning policy. Its continuous action predicts motion offset and uncertainty to form a spatial prior, while its discrete action selects a frame-wise keep ratio. The spatial prior is fused with current-frame template-to-search attention to rank search tokens.

  • Joint optimization of token ranking and retention budget. The motion prior affects which tokens rank highly, while the keep ratio determines how many are retained. Although these actions come from separate policy heads, they jointly determine the pruning result and are optimized using rewards for tracking accuracy, FLOPs, motion-prior regularization, and target-token preservation. Spatial guidance and the retention budget are not prescribed independently: both depend on the trajectory state and are optimized for the retained token set they produce together. This joint decision, rather than the individual use of trajectories or token pruning, is the methodological contribution of TGTP. We have clarified this distinction in the Related Work section of the revised manuscript.

Weakness 3 and Question 2: Could the authors report FPS or end-to-end inference latency under the same hardware settings, especially compared with UTP? This would help show whether the FLOPs reduction leads to actual speedup.

Response: We thank the reviewer for the suggestion. We measured the end-to-end tracking speed of OSTrack-256, OSTrack-UTP, and OSTrack-TGTP on the same NVIDIA RTX 3090. As shown in the table below, they run at 133 fps, 136 fps, and 141 fps, respectively. TGTP is therefore 5 fps faster than UTP and 8 fps faster than the original OSTrack, while reducing FLOPs to 22.5G, compared with 24.0G and 29.1G. The speed measurements include policy inference, motion-prior generation, attention fusion, and Top-K selection, showing that the FLOPs reduction translates into faster inference despite these additional operations. We have added the FPS comparison to Tab. 4 of the revised manuscript.

MethodSourceLaSOT AUCLaSOT PNLaSOT PFLOPs (G) ↓Speed (fps) ↑
OSTrack-256ECCV2267.976.973.329.1133
OSTrack-UTPCVPR2667.676.372.924.0136
OSTrack-TGTPOurs68.678.174.522.5141
AuthorsOpenReview ↗

Responses to Reviewer xJYV (Part 2)

Weakness 4 and Question 3: Could the authors provide a controlled experiment that fine-tunes the tracking head without TGTP? This would help separate the benefits of additional training from those of the proposed pruning method.

Response: We thank the reviewer for suggesting this control. As shown in the table below, we fine-tune the tracking heads of FERMT and MCITrack without TGTP and compare them with the full method under the same 30-epoch training configuration. For FERMT, head-only fine-tuning raises LaSOT AUC from 65.1 to 65.4 but decreases GOT-10k AO from 69.6 to 69.1, while FLOPs and speed remain at 2.39G and 181 fps. With TGTP, LaSOT AUC reaches 66.7, FLOPs decrease to 2.02G, and speed increases to 199 fps. For MCITrack, head-only fine-tuning decreases LaSOT AUC from 75.3 to 74.7, while FLOPs and speed remain at 38.5G and 30 fps. With TGTP, LaSOT AUC is 75.1, slightly below the original 75.3 but above the head-only result. Its GOT-10k AO increases from 77.9 to 80.6, while FLOPs decrease to 34.5G and speed increases to 34 fps. These results show that head-only fine-tuning brings small, inconsistent changes in tracking accuracy and no efficiency improvement. TGTP reduces computation and increases inference speed while maintaining competitive tracking performance. We have added this expanded controlled comparison to Appendix A.5 of the revised manuscript.

MethodLaSOT AUCLaSOT PNLaSOT PGOT-10k* AOGOT-10k* SR50GOT-10k* SR75LaSOT-ext AUCLaSOT-ext PNLaSOT-ext PFLOPs (G) ↓Speed (fps) ↑
FERMT65.174.669.169.680.163.246.1--2.39181
FERMT-FT65.474.969.369.179.164.046.2--2.39181
FERMT-TGTP66.776.671.471.381.966.846.6--2.02199

MethodLaSOT AUCLaSOT PNLaSOT PGOT-10k* AOGOT-10k* SR50GOT-10k* SR75LaSOT-ext AUCLaSOT-ext PNLaSOT-ext PFLOPs (G) ↓Speed (fps) ↑
MCITrack75.385.683.377.988.276.854.665.762.138.530
MCITrack-FT74.785.182.777.287.576.054.065.161.538.530
MCITrack-TGTP75.185.482.980.690.980.654.765.762.534.534

Weakness 5: The method depends on historical tracking results, which may not always be reliable. Errors from earlier frames could affect later token selection, especially during fast motion or occlusion. The paper provides limited analysis of these cases.

Response: We thank the reviewer for the suggestion. We considered errors in historical localization when designing TGTP, since they may shift the motion prior and affect which tokens are retained. To reduce this risk, the policy predicts both motion offset and uncertainty, forming a soft spatial prior rather than a hard region defined by the predicted target position. TGTP then fuses this prior with template-to-search attention from the current frame before selecting the Top-K search tokens. The IoU and target-token retention rewards further encourage the policy to preserve target-relevant information.

We summarize the relevant LaSOT attribute results from Fig. 4 in the table below. By fusing a soft motion prior with template-to-search attention, TGTP incorporates current-frame visual evidence into token selection even when historical estimates are less reliable. The results show that it maintains competitive tracking performance under fast motion, occlusion, and out-of-view, where those estimates may be less reliable. In future work, we will assess the reliability of historical predictions to further improve the motion prior in difficult cases. We have added this design rationale and challenge-specific analysis to Appendix A.4 of the revised supplementary material.

MethodALLFast MotionFull OcclusionOut-of-ViewPartial Occlusion
HCAT59.344.750.653.756.4
HiT-Base64.753.456.060.562.0
AsymTrack-B64.751.956.459.562.0
SpikeTrack-S25665.152.357.260.462.8
Ours66.755.357.061.163.9
AuthorsOpenReview ↗

Follow-up on Our Rebuttal to Reviewer xJYV

Dear Reviewer xJYV,

Thank you again for your review. We have posted point-by-point responses, including new non-RL comparisons, same-hardware speed measurements, and head-only fine-tuning controls. If any of your concerns remain unresolved, we would appreciate your feedback during the discussion period and would be glad to clarify.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Brief Summary of the Review-Rebuttal

Dear Area Chairs,

We thank the reviewers and area chairs for their time and constructive feedback. TGTP's central contribution is a tracking-specific token-pruning framework in which a reinforcement-learning policy uses the target's historical trajectory to adapt the keep ratio for each frame. To our knowledge, TGTP is the first to use reinforcement learning for token pruning in visual tracking.

We appreciate that the reviews recognized TGTP's tracking-specific motivation, its applicability across different visual trackers, and its ability to reduce computation while generally maintaining tracking accuracy. To address the remaining concerns, we conducted additional experiments and clarified key design choices, as summarized below:

  • Reviewer xJYV: We added comparisons with non-RL trajectory-based pruning, measured OSTrack, OSTrack-UTP, and OSTrack-TGTP on the same RTX 3090, and introduced head-only fine-tuning controls. We also clarified TGTP's methodological contribution and expanded the analysis of fast motion and occlusion.
  • Reviewer VPyk: We clarified that TGRL uses a current-frame reward rather than a cumulative multi-frame return. At matched average FLOPs, adaptive keep-ratio selection improves LaSOT AUC over the fixed-ratio control. We also added and discussed ETBTrack, explained our template-retention design, and reported FERMT-TGTP efficiency on GPU, CPU, and Jetson TX2.
  • Reviewer iFhX: We specified that training histories use annotated boxes, whereas inference histories use predicted boxes. We clarified how per-frame optimization relates to history-conditioned decisions and provided non-RL and head-only fine-tuning controls under the stated training settings.

Together, these additions distinguish the roles of trajectory history, adaptive keep-ratio selection, and fine-tuning, while strengthening the efficiency comparison. The corresponding changes appear in the revised Method and Related Work sections, Tab. 4, and Appendices A.2, A.4, and A.5.

Upon acceptance, we will release the code and experimental configurations to support reproducibility and future research on visual object tracking.

Thank you for considering our responses.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.