- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes TGTP, a trajectory-guided token pruning module for visual tracking. A lightweight policy uses recent bounding-box histories to predict motion priors, uncertainty, and token keep ratios. These priors are combined with template–search attention to select search tokens. Experiments across RGB and multimodal trackers show reduced computational cost while generally maintaining or improving tracking accuracy.
Critical strengths
1.A practical design that fits the tracking problem. Historical trajectories provide a useful spatial cue alongside current-frame attention. The policy does not require additional heavy visual features, and the pruning module can be integrated into existing trackers without redesigning their overall architecture. This makes the approach useful beyond a single tracker.
2. Broad evaluation with measurable efficiency gains. The experiments cover several trackers, backbones, and modalities. For SUTrack-B224, FLOPs decrease from 19.2G to 12.9G while LasHeR AUC improves from 59.9 to 62.0. The CPU and GPU speed results in Table 7 also provide evidence of practical acceleration, rather than FLOP reductions alone.
Critical weaknesses
- The construction of training trajectories is unclear. The paper does not clearly state whether historical boxes come from annotations, precomputed predictions, or the current model’s sequential predictions. This matters because errors in test-time histories may affect subsequent pruning. Also, the stated objective uses only the current-frame reward, leaving unclear whether training accounts for how pruning affects later tracking results. These details are important for reproducing the method and understanding its sequential decision-making formulation.
- The added value of reinforcement learning is not isolated. The ablations examine several design choices, but do not distinguish the benefit of reinforcement learning from that of adding a motion prior. A baseline using simple motion extrapolation with the same attention–prior fusion would help. It is also unclear whether the original trackers receive the same additional fine-tuning budget. Without these controls, some of the accuracy gains remain difficult to attribute.
Questions for authors
- Training clarification: How are historical boxes generated during training? Are the model’s own predictions fed into subsequent training frames?
- Objective clarification: Is the policy intended to optimize only the immediate reward? How does this choice relate to the sequential decision-making formulation?
- Additional evidence: How does TGTP compare with simple motion-extrapolation-based pruning under matched computational and fine-tuning budgets? Please also confirm whether the original baselines receive equivalent additional fine-tuning.
2 comments
Responses to Reviewer iFhX (Part 2)
Weakness 2 and Question 3: The added value of RL is not clearly isolated from that of a simple motion prior. A comparison with pruning based on simple motion extrapolation, using the same fusion of attention responses and motion priors under a matched computational budget, would help clarify this. It is also unclear whether the original trackers receive equivalent additional fine-tuning, making it difficult to attribute the accuracy gains to TGTP.
Response: We thank the reviewer for suggesting controls that distinguish the effects of RL, a simple motion prior, and additional fine-tuning. We address these points separately.
RL vs. simple motion extrapolation. As shown in the table below, linear extrapolation with a fixed keep ratio achieves 61.4 AUC and 85.0 P at 2.02G FLOPs. Combined with a rule-based adaptive ratio, it achieves 61.7 AUC and 85.2 P at 1.94G FLOPs. TGRL jointly learns the motion prior and frame-wise keep ratio, achieving 62.4 AUC and 85.5 P at 1.91G FLOPs. It also runs at 238fps, compared with 230fps for the rule-based strategy. Among the strategies tested, TGRL achieves higher accuracy with less computation, so its gain is not due to a larger token budget. We have added this comparison and analysis to Appendix A.5 of the revised manuscript.
| Optimization | Motion prior | Keep-ratio strategy | UAVDT AUC | UAVDT P | FLOPs (G) ↓ | Speed (fps) ↑ |
|---|---|---|---|---|---|---|
| Non-RL | Linear extrapolation | Fixed | 61.4 | 85.0 | 2.02 | 220 |
| Non-RL | Linear extrapolation | Rule-based adaptive | 61.7 | 85.2 | 1.94 | 230 |
| RL | TGRL (ours) | Learned adaptive | 62.4 | 85.5 | 1.91 | 238 |
Effect of additional fine-tuning. Our primary goal is to reduce FLOPs and improve inference speed through token pruning while preserving tracking accuracy. To control for additional training, we include head-only fine-tuning controls for FERMT and MCITrack. Each FT variant starts from the same pretrained weights as its TGTP counterpart and uses the same training data and 30-epoch fine-tuning schedule. As shown in the table below, FERMT-FT increases LaSOT AUC from 65.1 to 65.4, while its GOT-10k AO decreases from 69.6 to 69.1. MCITrack-FT decreases LaSOT AUC from 75.3 to 74.7 and GOT-10k AO from 77.9 to 77.2. FERMT-TGTP reduces FLOPs from 2.39G to 2.02G, a 15.5% reduction, and increases speed from 181fps to 199fps while reaching 66.7 LaSOT AUC. MCITrack-TGTP reduces FLOPs from 38.5G to 34.5G, a 10.4% reduction, and increases speed from 30fps to 34fps. Its LaSOT AUC of 75.1 is slightly below the original 75.3, while its GOT-10k AO improves from 77.9 to 80.6. These controls show that additional fine-tuning alone leads to only small, inconsistent changes in tracking accuracy and does not produce the efficiency gains. We have added the fine-tuning controls and their analysis to Appendix A.5 of the revised manuscript.
| Method | LaSOT AUC | LaSOT PN | LaSOT P | GOT-10k$^\ast$ AO | GOT-10k$^\ast$ SR50 | GOT-10k$^\ast$ SR75 | LaSOT$_{ext}$ AUC | LaSOT$_{ext}$ PN | LaSOT$_{ext}$ P | FLOPs (G) ↓ | Speed (fps) ↑ |
|---|---|---|---|---|---|---|---|---|---|---|---|
| FERMT | 65.1 | 74.6 | 69.1 | 69.6 | 80.1 | 63.2 | 46.1 | - | - | 2.39 | 181 |
| FERMT-FT | 65.4 | 74.9 | 69.3 | 69.1 | 79.1 | 64.0 | 46.2 | - | - | 2.39 | 181 |
| FERMT-TGTP | 66.7 | 76.6 | 71.4 | 71.3 | 81.9 | 66.8 | 46.6 | - | - | 2.02 | 199 |
| Method | LaSOT AUC | LaSOT PN | LaSOT P | GOT-10k$^\ast$ AO | GOT-10k$^\ast$ SR50 | GOT-10k$^\ast$ SR75 | LaSOT$_{ext}$ AUC | LaSOT$_{ext}$ PN | LaSOT$_{ext}$ P | FLOPs (G) ↓ | Speed (fps) ↑ |
|---|---|---|---|---|---|---|---|---|---|---|---|
| MCITrack | 75.3 | 85.6 | 83.3 | 77.9 | 88.2 | 76.8 | 54.6 | 65.7 | 62.1 | 38.5 | 30 |
| MCITrack-FT | 74.7 | 85.1 | 82.7 | 77.2 | 87.5 | 76.0 | 54.0 | 65.1 | 61.5 | 38.5 | 30 |
| MCITrack-TGTP | 75.1 | 85.4 | 82.9 | 80.6 | 90.9 | 80.6 | 54.7 | 65.7 | 62.5 | 34.5 | 34 |
Responses to Reviewer iFhX (Part 1)
Response: We thank the reviewer for raising these questions. We address them separately below.
Question 1 (Training trajectories). For video sequences, we sample historical visible frames preceding the current search frame and use their ground-truth boxes to construct the trajectory state. These boxes are transformed into the coordinate system of the current search region. The model's predictions are not fed into subsequent training frames. During inference, the history is instead updated with the tracker's predicted boxes. We have described this construction and the training-inference distinction in Appendix A.2 of the revised supplementary material.
Question 2 (Optimization objective). The policy is optimized using the reward from the current frame. Rewards from later frames are not propagated back to earlier pruning decisions. By sequential decision-making, we mean that the policy acts at each frame using a trajectory state updated with previous tracking results, not that it optimizes cumulative future rewards. Errors in those results may affect later token selection. To mitigate this, the policy predicts motion uncertainty to form a soft spatial prior, which is fused with template-to-search attention from the current frame before pruning. Current visual evidence therefore also contributes to token selection when the historical trajectory is inaccurate. We have clarified the per-frame objective in the Method section of the revised manuscript.
Once the paper is accepted, we will release the code and experimental configurations to support reproducibility.