- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes LoopTrack, a parameter-efficient one-stream Transformer tracker. Instead of stacking many independently parameterized Transformer blocks, LoopTrack reuses a small set of shared Transformer blocks in a recurrent loop. It further introduces Target-Aware Looping (TAL) to use intermediate localization responses for progressive refinement, and Gated Target Memory (GTM) to maintain a compact cross-frame target memory. Experiments on LaSOT, LaSOText, TrackingNet, and GOT-10k show favorable accuracy–parameter trade-offs. LoopTrackThree achieves 69.3% SUC on LaSOT with 6.4M parameters, while LoopTrackOne achieves 66.2% SUC with only 3.4M parameters.
Critical strengths
1.The paper targets a real problem in one-stream Transformer tracking: parameter overhead from stacked blocks. The idea of reusing a small set of shared blocks is intuitive and easy to understand. 2.The reported parameter counts are attractive. LoopTrackThree matches or exceeds several larger trackers while using only 6.4M parameters. LoopTrackOne is especially compact at 3.4M parameters. 3.The ablations on TAL, GTM, spatial response, target appearance update, and memory components help justify the design choices. The gains from TAL and GTM are non-trivial. 4.Applying LoopTrackThree to UTPTrack-O-256 and OSTrack-256 shows consistent parameter reduction with small accuracy drops, suggesting the idea is not tied to only one architecture.
Critical weaknesses
1.The central claim is “parameter-efficient” and “resource-friendly,” but the method still executes the shared blocks K times. Therefore, the reduction in stored parameters does not directly translate into reduced computation, latency, or energy. In Table 2, LoopTrackThree runs at 98 FPS, while UETrack-B runs at 163 FPS with 13.0M parameters, and LiteTrack-B9 runs at 171 FPS with 54.9M parameters. This means LoopTrack is smaller in memory but not faster. Table 14 also shows that adding TAL and GTM reduces throughput from 108 to 98 FPS. For deployment on resource-limited devices, latency and MACs often matter more than parameter count. The paper needs a much clearer and fairer analysis of computation, memory, and latency. 2.Limited novelty over existing looped/shared Transformers. Parameter sharing and looped Transformers have been explored in image recognition, multi-view 3D reconstruction, and universal Transformers. The paper applies this idea to one-stream tracking and adds TAL/GTM, but the architectural novelty is incremental. The paper does not sufficiently distinguish itself from prior looped ViT works beyond the tracking application. 3.The training recipe is excessively heavy and overly complex for a method positioned as a “simple baseline.” It requires two stages of 400 + 280 = 680 epochs, 100k samples per epoch, batch size 128 on 4×RTX 4090 GPUs, and a four-dataset mix including VastTrack. The optimization strategy is also unusually intricate: the loop count is randomly sampled via Beta(1,2) over K∈{6,…,12}; TU is updated only at loops {2,4,6}; auxiliary supervision is applied at loops 4 and 6; multiple auxiliary losses are combined with manually set weights; the second stage resets the optimizer and LR scheduler; and auxiliary branches are disabled at inference. This conflicts with the “simple baseline” claim, increases tuning and reproduction burden, and makes it unclear whether gains come from looped parameter sharing or from the heavy training recipe and extra data/compute. 4.Incomplete and potentially unfair comparison. FPS values for other trackers are taken from original papers and measured on different hardware, while LoopTrack is measured on an RTX 4090. This makes speed comparisons unreliable. The paper also lacks MACs/FLOPs comparison with other efficient trackers, despite claiming resource friendliness. In addition, training data, pretraining, input resolution, and post-processing may differ across methods. The paper should provide unified runtime, MACs, memory, and preferably variance/significance tests. 5.GTM updates its single-token memory using a writing coefficient computed from the peak and spatial concentration of the predicted response map. Although this reduces contamination from false-positive frames, under prolonged occlusion, fast motion, or severe background clutter, a high-confidence but incorrect response to a distractor can still write erroneous appearance into the only memory token. Because the memory consists of a single token, recovery after contamination is difficult. The paper lacks a deeper analysis of cumulative memory error and drift on very long sequences, such as the thousands-of-frames cases in LaSOT.
Questions for authors
- Why is LoopTrackThree slower than UETrack-B and LiteTrack-B9 despite having far fewer parameters? Is the bottleneck the repeated execution of shared blocks, TAL/GTM, or implementation details?
- What is the actual inference-time loop count used in the main results? Can adaptive or input-dependent loop counts reduce computation while preserving accuracy?
- What is the key architectural difference between LoopTrack and prior looped or shared Transformers, such as Universal Transformer, looped ViTs, single-block recurrence, and Deja View?
- How does the single-token memory recover after a high-confidence false response writes distractor appearance into memory?
7 comments
Q3: What are the main architectural differences between LoopTrack and previous recurrent or shared Transformers, such as Universal Transformer, looped ViTs, single-block recurrence, and Déjà View?
A3: Thank you for the comment. LoopTrack addresses the high parameter cost of existing Transformer trackers, which hinders deployment on resource-constrained devices. It reduces the number of independently stored parameters while preserving multiple rounds of template–search interaction. Beyond generic weight sharing, it introduces two mechanisms designed specifically for tracking.
TAL feeds intermediate target localization into subsequent loops. Tracking must stay focused on the instance specified by the initial template, while salient background regions or similar objects can interfere with recurrent interaction. TAL turns the intermediate localization response into spatial feedback and an updated target appearance, both of which are added to the search features of the next loop. Each subsequent round of template–search interaction is therefore explicitly guided by the current estimate of the target's location and appearance. For example, the standard Déjà View architecture recurrently refines its internal states and performs geometric decoding only at the end, whereas LoopTrack uses intermediate target estimates directly to construct the input of the next loop.
GTM adds a cross-frame target state with reliability-dependent updates to the shared recurrence. A fixed initial template cannot capture all subsequent appearance changes, and additional within-frame loops alone provide no recent observations of the target. GTM feeds a historical target memory into the shared module together with the template and search features. It separates within-frame memory interaction from cross-frame memory updates: the final localization response selects the target appearance from the pre-loop search features, and the localization reliability controls how strongly this appearance is written into memory. The shared recurrence can thus exploit historical target information while limiting the influence of unreliable observations.
The ablations quantify the contribution of these tracking-specific designs. In Table 3, the basic shared recurrence, which already performs multiple rounds of template–search interaction, achieves 65.8% LaSOT SUC. Adding TAL and then GTM raises SUC to 68.8% and 69.3%, respectively. Meanwhile, the parameter count increases only from 6.373M to 6.379M (Table 14), and the number and capacity of the shared Transformer blocks remain unchanged. Target feedback and cross-frame state updates therefore make recurrent interaction more effective while keeping the parameter budget compact.
The target-localization feedback and the cross-frame memory are detailed in Sec. 3.2 (L226–L304), and their contributions are quantified in Tables 3 and 14.
Q4: After a high-confidence incorrect response writes distractor appearance into the single memory token, how can the memory be corrected in subsequent frames?
A4: Thank you for the question. GTM allows such errors to be corrected in subsequent frames through three mechanisms:
-
The fixed template preserves the original target reference. The initial template remains unchanged throughout tracking, and its features interact with the search features and the historical memory in every frame. Even if the dynamic memory contains distractor information, the model can still localize the target using the original reference and the current image.
-
New appearance candidates are extracted from the current image. GTM uses the final localization response to extract the target appearance from the current frame's pre-loop search features. Once the model localizes the target correctly again, it obtains a fresh appearance candidate that does not depend on the previously miswritten content.
-
New observations continue to revise the memory. The memory is updated in every subsequent frame. When a new observation of the target is correct and reliable, GU increases its contribution and reduces that of the historical memory in the fusion. Correct target appearance can thus enter the memory and progressively dilute earlier erroneous observations.
The similar-instance distractors in hat-5 and pig-2 and the appearance changes in flag-2 (Appendix C.3, Figure 5) illustrate the benefit of combining the fixed template with dynamic memory for target localization.
The cross-frame memory update is described in Sec. 3.2 (Eqs. (7)–(8), L280–L304), and the related qualitative comparisons are given in Sec. C.3 (Figure 5, L1037–L1057).
Q5: Do the training pipeline, loop-count sampling, and auxiliary supervision conflict with the positioning as a “simple baseline” and increase the tuning and reproduction burden? Are the gains mainly attributable to additional training, data, and computation?
Table C. Applicability to existing Transformer trackers on LaSOT (Table 7 in the paper).
| Tracker configuration | Params | SUC | Normalized precision | Precision |
|---|---|---|---|---|
| UTPTrack-O-256 | 92.5 | 67.40 | 76.30 | 72.50 |
| w/ | 24.0 | 67.04 | 76.36 | 72.21 |
| OSTrack-256 (w/o CE) | 92.52 | 68.70 | 78.10 | 74.60 |
| w/ | 24.1 | 68.26 | 77.82 | 74.21 |
Params are in millions; all tracking metrics are percentages. Each “w/ ” row applies LoopTrack to the tracker in the row immediately above it.
Table D. Training schedules and reported tracking accuracy.
| Method | LaSOT SUC | Training epochs | Samples per epoch |
|---|---|---|---|
| 69.3 | 680 | 100k | |
| CompressTracker-4 | 66.1 | 500 | 60k |
| CompressTracker-6 | 67.5 | 500 | 60k |
| CompressTracker-8 | 68.4 | 500 | 60k |
| MixFormerV2-S | 60.6 | 550 | 60k |
| HiT-Base | 64.6 | 1500 | 60k |
| UETrack-B | 69.2 | 500 | 100k |
SUC is reported as a percentage; k = 1,000. MixFormerV2-S trains for 500 epochs per distillation stage, followed by 50 epochs of score-head training.
A5: Thank you for the comment. By “simple baseline” we mainly mean that LoopTrack builds its shared recurrence from standard Transformer blocks and is trained without additional teachers or cross-model distillation. We clarify this point below with reference to related methods and our experiments.
-
Training pipeline: fewer auxiliary models and no distillation. CompressTracker uses a teacher model, stochastic stage replacement, progressive scheduling, and prediction/feature supervision. MixFormerV2-S relies on multi-stage distillation and progressive pruning. UETrack involves a teacher, a student, and a separately optimized AdaptiveNet. In contrast, both stages of LoopTrack optimize the same shared model, with no additional teacher, no cross-model distillation, and no separate network for distillation decisions.
-
Training schedule and data: comparable to existing methods. As shown in Table D, HiT trains for 1,500 epochs with 60,000 image pairs per epoch, and UETrack trains for 500 epochs with 100,000 samples per epoch and a total batch size of 128. UETrack's unified training uses ten datasets, which cover all four training sources used by LoopTrack, including VastTrack. Multi-dataset training and VastTrack are therefore not data advantages unique to LoopTrack relative to UETrack.
-
Tuning burden: the same configuration transfers across tracking frameworks. In Table C, applying to UTPTrack and OSTrack requires no tracker-specific hyperparameter tuning. In both cases it reduces the parameter count by approximately 74%, with LaSOT SUC decreases of only 0.36 and 0.44 percentage points, respectively. This supports the transferability of the shared recurrent configuration and its low tuning requirements.
-
Sources of improvement: the ablations isolate the contribution of each training strategy. Training with a fixed already achieves 68.9% LaSOT SUC, and random loop-count sampling raises it to 69.3% (Table 9). With the network and feedback structure unchanged, adding only the auxiliary box and quality supervision raises SUC from 68.8% to 69.3% (Table 6). The model therefore remains effective under simplified training, and the additional benefit of each strategy is quantified explicitly. Moreover, the same model supports different inference depths, so an operating point can be selected according to the deployment budget.
We have added the comparison of training schedules and a discussion of cross-framework transfer to the revised supplementary material (highlighted in blue; see Sec. B.7, Table 21, L981–L995).
We thank the reviewer for the careful evaluation and the constructive comments. We address each point below.
A1: Thank you for the question. The additional latency comes mainly from the repeated execution of the shared blocks, not from TAL/GTM.
LoopTrack targets deployment settings where model storage and the parameter budget are limited. Its goal is to retain sufficient template–search interaction, and hence tracking accuracy, within a compact model. Sharing weights across loops reduces LoopTrackThree to approximately 6.4M parameters, but every loop of feature interaction must still be executed. A smaller parameter footprint therefore does not translate into a proportional reduction in computation or latency. This is the trade-off LoopTrack makes to preserve interaction under a tight parameter budget.
In contrast, TAL and GTM add only approximately 0.11% to the MACs (Table 14), so their arithmetic cost is negligible. Table 15 shows how recurrent depth governs runtime: reducing K from 8 to 6 raises the speed from 98 to 121 FPS at a cost of only 0.3 percentage points in SUC. Without changing the parameter count, LoopTrack can thus match its inference depth to the computational budget of the device and offer several accuracy–speed operating points.
Table A. Comparison of fixed inference depths.
Relative to K=8, K=4 and K=6 reduce MACs by 45.54% and 22.77%, with SUC decreases of 1.5 and 0.3 percentage points, respectively. MACs follow the same counting convention as in Table B; percentage reductions are computed from unrounded values.
Table B. Paired re-evaluation of adaptive early exit and fixed-depth inference.
Adaptive MACs are averaged over frames according to the depth at which each frame actually exits. All rows come from the same paired re-evaluation.
A2: Thank you for the question.
Fixed depth. All main experiments use a fixed inference depth of K=8. Because the parameters are shared across loops, the same trained model can run at different depths, trading accuracy for computation without additional parameters or retraining. Extending Table 15, Table A reports the computational cost at each fixed depth. Reducing the number of loops from 8 to 6 lowers MACs by 22.77% and raises the speed from 98 to 121 FPS, with a SUC decrease of only 0.3 percentage points. Reducing the depth further to K=4 lowers MACs by 45.54% and reaches 151 FPS, at a SUC cost of 1.5 points. A single trained model can therefore serve a range of inference budgets.
Adaptive depth. To test whether input-dependent depth can reduce computation while largely preserving accuracy, we further evaluate adaptive early exit without retraining. Every frame first executes four loops. The model then decides whether to exit, based on the quality of the intermediate TAL localization response and its stability across successive loops. If the exit criterion is not met, the frame continues to the sixth loop, where the criterion is checked again; the maximum depth is eight loops. The exit thresholds are calibrated on training sequences and fixed before testing.
Table B reports two policies with different levels of aggressiveness. Policy A reduces the average depth to 6.774 loops and the average MACs by 13.96%, and raises the speed from 98 to 123 FPS, with a SUC decrease of only 0.077 percentage points. It thus largely preserves tracking accuracy in this evaluation. The more aggressive Policy B reduces the average depth to 4.844 loops and the average MACs by 35.93%, reaching 145 FPS with a SUC decrease of 0.520 percentage points.
These results show that LoopTrack can adjust its computational budget either through a fixed inference depth or through input-dependent early exit. Both options reduce computation with only small changes in tracking accuracy and provide additional operating points for computationally constrained applications.
We have added the fixed-depth and adaptive-depth comparisons, together with the corresponding analysis, to the revised supplementary material (highlighted in blue; see Secs. B.5.1–B.5.2, Tables 16–17, L895–L917).