acceptodds
Under review as a conference paper at ICLR 2027

LoopTrack: A Simple Baseline for Parameter-Efficient Transformer Tracking

Abstract

Current Transformer-based tracking methods typically stack multiple Transformer blocks with separate parameters to model interactions between the target template and the search region for target localization. Despite excellent performance, these trackers often incur substantial parameter overhead from stacked blocks, making their deployment on resource-limited devices difficult. To address this, we propose a novel parameter-efficient Transformer tracking framework, dubbed ***LoopTrack***, which repeatedly applies a small set of Transformer blocks with shared parameters to interact features in a looped architecture for tracking, significantly reducing the number of parameters. To further exploit target cues for improving LoopTrack, we present two lightweight designs, including *target-aware looping* (TAL) and *gated target memory* (GTM). The former applies intermediate target information generated by one loop to guide feature interaction in the subsequent loop, enabling progressive feature refinement, while the latter maintains a compact memory across frames, which is incorporated into the loop process to provide long-term information to the tracker, mitigating temporal drift in tracking. Compared to existing Transformer trackers, LoopTrack enables multiple rounds of feature interaction with fewer model parameters, making it resource-friendly for deployment. In extensive experiments on multiple datasets, LoopTrack shows promising results with a favorable accuracy-parameter trade-off. In particular, our , with a single shared Transformer block, achieves 66.2% SUC score on LaSOT with only 3.4M parameters, while , using three shared blocks, achieves 69.3% SUC score with 6.4M model parameters, surpassing existing parameter-efficient tracking methods with comparable or larger model size. With LoopTrack, we aim to establish a simple yet strong baseline for parameter-efficient Transformer tracking. Our code and models will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
visual object tracking, loop transformer

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_ap5nOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes LoopTrack, a parameter-efficient one-stream Transformer tracker. Instead of stacking many independently parameterized Transformer blocks, LoopTrack reuses a small set of shared Transformer blocks in a recurrent loop. It further introduces Target-Aware Looping (TAL) to use intermediate localization responses for progressive refinement, and Gated Target Memory (GTM) to maintain a compact cross-frame target memory. Experiments on LaSOT, LaSOText, TrackingNet, and GOT-10k show favorable accuracy–parameter trade-offs. LoopTrackThree achieves 69.3% SUC on LaSOT with 6.4M parameters, while LoopTrackOne achieves 66.2% SUC with only 3.4M parameters.

Critical strengths

1.The paper targets a real problem in one-stream Transformer tracking: parameter overhead from stacked blocks. The idea of reusing a small set of shared blocks is intuitive and easy to understand. 2.The reported parameter counts are attractive. LoopTrackThree matches or exceeds several larger trackers while using only 6.4M parameters. LoopTrackOne is especially compact at 3.4M parameters. 3.The ablations on TAL, GTM, spatial response, target appearance update, and memory components help justify the design choices. The gains from TAL and GTM are non-trivial. 4.Applying LoopTrackThree to UTPTrack-O-256 and OSTrack-256 shows consistent parameter reduction with small accuracy drops, suggesting the idea is not tied to only one architecture.

Critical weaknesses

1.The central claim is “parameter-efficient” and “resource-friendly,” but the method still executes the shared blocks K times. Therefore, the reduction in stored parameters does not directly translate into reduced computation, latency, or energy. In Table 2, LoopTrackThree runs at 98 FPS, while UETrack-B runs at 163 FPS with 13.0M parameters, and LiteTrack-B9 runs at 171 FPS with 54.9M parameters. This means LoopTrack is smaller in memory but not faster. Table 14 also shows that adding TAL and GTM reduces throughput from 108 to 98 FPS. For deployment on resource-limited devices, latency and MACs often matter more than parameter count. The paper needs a much clearer and fairer analysis of computation, memory, and latency. 2.Limited novelty over existing looped/shared Transformers. Parameter sharing and looped Transformers have been explored in image recognition, multi-view 3D reconstruction, and universal Transformers. The paper applies this idea to one-stream tracking and adds TAL/GTM, but the architectural novelty is incremental. The paper does not sufficiently distinguish itself from prior looped ViT works beyond the tracking application. 3.The training recipe is excessively heavy and overly complex for a method positioned as a “simple baseline.” It requires two stages of 400 + 280 = 680 epochs, 100k samples per epoch, batch size 128 on 4×RTX 4090 GPUs, and a four-dataset mix including VastTrack. The optimization strategy is also unusually intricate: the loop count is randomly sampled via Beta(1,2) over K∈{6,…,12}; TU is updated only at loops {2,4,6}; auxiliary supervision is applied at loops 4 and 6; multiple auxiliary losses are combined with manually set weights; the second stage resets the optimizer and LR scheduler; and auxiliary branches are disabled at inference. This conflicts with the “simple baseline” claim, increases tuning and reproduction burden, and makes it unclear whether gains come from looped parameter sharing or from the heavy training recipe and extra data/compute. 4.Incomplete and potentially unfair comparison. FPS values for other trackers are taken from original papers and measured on different hardware, while LoopTrack is measured on an RTX 4090. This makes speed comparisons unreliable. The paper also lacks MACs/FLOPs comparison with other efficient trackers, despite claiming resource friendliness. In addition, training data, pretraining, input resolution, and post-processing may differ across methods. The paper should provide unified runtime, MACs, memory, and preferably variance/significance tests. 5.GTM updates its single-token memory using a writing coefficient computed from the peak and spatial concentration of the predicted response map. Although this reduces contamination from false-positive frames, under prolonged occlusion, fast motion, or severe background clutter, a high-confidence but incorrect response to a distractor can still write erroneous appearance into the only memory token. Because the memory consists of a single token, recovery after contamination is difficult. The paper lacks a deeper analysis of cumulative memory error and drift on very long sequences, such as the thousands-of-frames cases in LaSOT.

Questions for authors
  1. Why is LoopTrackThree slower than UETrack-B and LiteTrack-B9 despite having far fewer parameters? Is the bottleneck the repeated execution of shared blocks, TAL/GTM, or implementation details?
  2. What is the actual inference-time loop count used in the main results? Can adaptive or input-dependent loop counts reduce computation while preserving accuracy?
  3. What is the key architectural difference between LoopTrack and prior looped or shared Transformers, such as Universal Transformer, looped ViTs, single-block recurrence, and Deja View?
  4. How does the single-token memory recover after a high-confidence false response writes distractor appearance into memory?
7 comments

We thank the reviewer for the careful evaluation and the constructive comments. We address each point below.

Q1: Why is LoopTrackThree\mathrm{LoopTrack}_{\mathrm{Three}} slower despite having substantially fewer parameters than UETrack-B and LiteTrack-B9? Does the overhead come from repeated execution of the shared blocks, TAL/GTM, or implementation details?

A1: Thank you for the question. The additional latency comes mainly from the repeated execution of the shared blocks, not from TAL/GTM.

LoopTrack targets deployment settings where model storage and the parameter budget are limited. Its goal is to retain sufficient template–search interaction, and hence tracking accuracy, within a compact model. Sharing weights across loops reduces LoopTrackThree\mathrm{LoopTrack}_{\mathrm{Three}} to approximately 6.4M parameters, but every loop of feature interaction must still be executed. A smaller parameter footprint therefore does not translate into a proportional reduction in computation or latency. This is the trade-off LoopTrack makes to preserve interaction under a tight parameter budget.

In contrast, TAL and GTM add only approximately 0.11% to the MACs (Table 14), so their arithmetic cost is negligible. Table 15 shows how recurrent depth governs runtime: reducing KK from 8 to 6 raises the speed from 98 to 121 FPS at a cost of only 0.3 percentage points in SUC. Without changing the parameter count, LoopTrack can thus match its inference depth to the computational budget of the device and offer several accuracy–speed operating points.

Q2: What inference depth is used in the main experiments? Can adaptive or input-dependent inference depth reduce computation while maintaining accuracy?

Table A. Comparison of fixed inference depths.

KMACs (G/frame) ↓FPS ↑SUC (%) ↑
45.23815167.8
67.42812169.0
89.6189869.3

Relative to K=8, K=4 and K=6 reduce MACs by 45.54% and 22.77%, with SUC decreases of 1.5 and 0.3 percentage points, respectively. MACs follow the same counting convention as in Table B; percentage reductions are computed from unrounded values.

Table B. Paired re-evaluation of adaptive early exit and fixed-depth inference.

SettingAverage loopsMACs (G/frame) ↓FPS ↑SUC (%) ↑
Fixed K=88.0009.6189869.194
Adaptive Policy A6.7748.27512369.117
Adaptive Policy B4.8446.16214568.674

Adaptive MACs are averaged over frames according to the depth at which each frame actually exits. All rows come from the same paired re-evaluation.

A2: Thank you for the question.

Fixed depth. All main experiments use a fixed inference depth of K=8K=8. Because the parameters are shared across loops, the same trained model can run at different depths, trading accuracy for computation without additional parameters or retraining. Extending Table 15, Table A reports the computational cost at each fixed depth. Reducing the number of loops from 8 to 6 lowers MACs by 22.77% and raises the speed from 98 to 121 FPS, with a SUC decrease of only 0.3 percentage points. Reducing the depth further to K=4K=4 lowers MACs by 45.54% and reaches 151 FPS, at a SUC cost of 1.5 points. A single trained model can therefore serve a range of inference budgets.

Adaptive depth. To test whether input-dependent depth can reduce computation while largely preserving accuracy, we further evaluate adaptive early exit without retraining. Every frame first executes four loops. The model then decides whether to exit, based on the quality of the intermediate TAL localization response and its stability across successive loops. If the exit criterion is not met, the frame continues to the sixth loop, where the criterion is checked again; the maximum depth is eight loops. The exit thresholds are calibrated on training sequences and fixed before testing.

Table B reports two policies with different levels of aggressiveness. Policy A reduces the average depth to 6.774 loops and the average MACs by 13.96%, and raises the speed from 98 to 123 FPS, with a SUC decrease of only 0.077 percentage points. It thus largely preserves tracking accuracy in this evaluation. The more aggressive Policy B reduces the average depth to 4.844 loops and the average MACs by 35.93%, reaching 145 FPS with a SUC decrease of 0.520 percentage points.

These results show that LoopTrack can adjust its computational budget either through a fixed inference depth or through input-dependent early exit. Both options reduce computation with only small changes in tracking accuracy and provide additional operating points for computationally constrained applications.

We have added the fixed-depth and adaptive-depth comparisons, together with the corresponding analysis, to the revised supplementary material (highlighted in blue; see Secs. B.5.1–B.5.2, Tables 16–17, L895–L917).

Q3: What are the main architectural differences between LoopTrack and previous recurrent or shared Transformers, such as Universal Transformer, looped ViTs, single-block recurrence, and Déjà View?

A3: Thank you for the comment. LoopTrack addresses the high parameter cost of existing Transformer trackers, which hinders deployment on resource-constrained devices. It reduces the number of independently stored parameters while preserving multiple rounds of template–search interaction. Beyond generic weight sharing, it introduces two mechanisms designed specifically for tracking.

TAL feeds intermediate target localization into subsequent loops. Tracking must stay focused on the instance specified by the initial template, while salient background regions or similar objects can interfere with recurrent interaction. TAL turns the intermediate localization response into spatial feedback and an updated target appearance, both of which are added to the search features of the next loop. Each subsequent round of template–search interaction is therefore explicitly guided by the current estimate of the target's location and appearance. For example, the standard Déjà View architecture recurrently refines its internal states and performs geometric decoding only at the end, whereas LoopTrack uses intermediate target estimates directly to construct the input of the next loop.

GTM adds a cross-frame target state with reliability-dependent updates to the shared recurrence. A fixed initial template cannot capture all subsequent appearance changes, and additional within-frame loops alone provide no recent observations of the target. GTM feeds a historical target memory into the shared module together with the template and search features. It separates within-frame memory interaction from cross-frame memory updates: the final localization response selects the target appearance from the pre-loop search features, and the localization reliability controls how strongly this appearance is written into memory. The shared recurrence can thus exploit historical target information while limiting the influence of unreliable observations.

The ablations quantify the contribution of these tracking-specific designs. In Table 3, the basic shared recurrence, which already performs multiple rounds of template–search interaction, achieves 65.8% LaSOT SUC. Adding TAL and then GTM raises SUC to 68.8% and 69.3%, respectively. Meanwhile, the parameter count increases only from 6.373M to 6.379M (Table 14), and the number and capacity of the shared Transformer blocks remain unchanged. Target feedback and cross-frame state updates therefore make recurrent interaction more effective while keeping the parameter budget compact.

The target-localization feedback and the cross-frame memory are detailed in Sec. 3.2 (L226–L304), and their contributions are quantified in Tables 3 and 14.

Q4: After a high-confidence incorrect response writes distractor appearance into the single memory token, how can the memory be corrected in subsequent frames?

A4: Thank you for the question. GTM allows such errors to be corrected in subsequent frames through three mechanisms:

  1. The fixed template preserves the original target reference. The initial template remains unchanged throughout tracking, and its features interact with the search features and the historical memory in every frame. Even if the dynamic memory contains distractor information, the model can still localize the target using the original reference and the current image.

  2. New appearance candidates are extracted from the current image. GTM uses the final localization response to extract the target appearance from the current frame's pre-loop search features. Once the model localizes the target correctly again, it obtains a fresh appearance candidate that does not depend on the previously miswritten content.

  3. New observations continue to revise the memory. The memory is updated in every subsequent frame. When a new observation of the target is correct and reliable, GU increases its contribution and reduces that of the historical memory in the fusion. Correct target appearance can thus enter the memory and progressively dilute earlier erroneous observations.

The similar-instance distractors in hat-5 and pig-2 and the appearance changes in flag-2 (Appendix C.3, Figure 5) illustrate the benefit of combining the fixed template with dynamic memory for target localization.

The cross-frame memory update is described in Sec. 3.2 (Eqs. (7)–(8), L280–L304), and the related qualitative comparisons are given in Sec. C.3 (Figure 5, L1037–L1057).

Q5: Do the training pipeline, loop-count sampling, and auxiliary supervision conflict with the positioning as a “simple baseline” and increase the tuning and reproduction burden? Are the gains mainly attributable to additional training, data, and computation?

Table C. Applicability to existing Transformer trackers on LaSOT (Table 7 in the paper).

Tracker configurationParamsSUCNormalized precisionPrecision
UTPTrack-O-25692.567.4076.3072.50
w/ LoopTrackThree\mathbf{LoopTrack}_{\mathbf{Three}}24.067.0476.3672.21
OSTrack-256 (w/o CE)92.5268.7078.1074.60
w/ LoopTrackThree\mathbf{LoopTrack}_{\mathbf{Three}}24.168.2677.8274.21

Params are in millions; all tracking metrics are percentages. Each “w/ LoopTrackThree\mathrm{LoopTrack}_{\mathrm{Three}}” row applies LoopTrack to the tracker in the row immediately above it.

Table D. Training schedules and reported tracking accuracy.

MethodLaSOT SUCTraining epochsSamples per epoch
LoopTrackThree\mathbf{LoopTrack}_{\mathbf{Three}}69.3680100k
CompressTracker-466.150060k
CompressTracker-667.550060k
CompressTracker-868.450060k
MixFormerV2-S60.655060k
HiT-Base64.6150060k
UETrack-B69.2500100k

SUC is reported as a percentage; k = 1,000. MixFormerV2-S trains for 500 epochs per distillation stage, followed by 50 epochs of score-head training.

A5: Thank you for the comment. By “simple baseline” we mainly mean that LoopTrack builds its shared recurrence from standard Transformer blocks and is trained without additional teachers or cross-model distillation. We clarify this point below with reference to related methods and our experiments.

  1. Training pipeline: fewer auxiliary models and no distillation. CompressTracker uses a teacher model, stochastic stage replacement, progressive scheduling, and prediction/feature supervision. MixFormerV2-S relies on multi-stage distillation and progressive pruning. UETrack involves a teacher, a student, and a separately optimized AdaptiveNet. In contrast, both stages of LoopTrack optimize the same shared model, with no additional teacher, no cross-model distillation, and no separate network for distillation decisions.

  2. Training schedule and data: comparable to existing methods. As shown in Table D, HiT trains for 1,500 epochs with 60,000 image pairs per epoch, and UETrack trains for 500 epochs with 100,000 samples per epoch and a total batch size of 128. UETrack's unified training uses ten datasets, which cover all four training sources used by LoopTrack, including VastTrack. Multi-dataset training and VastTrack are therefore not data advantages unique to LoopTrack relative to UETrack.

  3. Tuning burden: the same configuration transfers across tracking frameworks. In Table C, applying LoopTrackThree\mathrm{LoopTrack}_{\mathrm{Three}} to UTPTrack and OSTrack requires no tracker-specific hyperparameter tuning. In both cases it reduces the parameter count by approximately 74%, with LaSOT SUC decreases of only 0.36 and 0.44 percentage points, respectively. This supports the transferability of the shared recurrent configuration and its low tuning requirements.

  4. Sources of improvement: the ablations isolate the contribution of each training strategy. Training with a fixed K=8K=8 already achieves 68.9% LaSOT SUC, and random loop-count sampling raises it to 69.3% (Table 9). With the network and feedback structure unchanged, adding only the auxiliary box and quality supervision raises SUC from 68.8% to 69.3% (Table 6). The model therefore remains effective under simplified training, and the additional benefit of each strategy is quantified explicitly. Moreover, the same model supports different inference depths, so an operating point can be selected according to the deployment budget.

We have added the comparison of training schedules and a discussion of cross-framework transfer to the revised supplementary material (highlighted in blue; see Sec. B.7, Table 21, L981–L995).

Reviewer_PRLvOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

LoopTrack investigates whether recurrent parameter sharing can reduce the model size of Transformer-based visual trackers while preserving accuracy. It reuses a small set of blocks and guides successive refinement with target-localization feedback and memory from previous frames. Evaluation across tracking benchmarks, component ablations, and adaptation to other trackers supports a favorable accuracy–parameter trade-off. Additional experiments examine how changing the number of refinement steps allows a single trained model to operate at different accuracy and speed levels.

Critical strengths
  1. Strong parameter efficiency with practical value for deployment on resource-constrained devices. LoopTrackThree achieves a LaSOT SUC score of 69.3% with only 6.4M parameters, compared with 69.2% achieved by UETrack-B with 13.0M parameters, demonstrating a favorable trade-off between tracking accuracy and parameter count. I agree with the paper’s conclusion regarding parameter efficiency: recurrent parameter sharing reduces the storage requirements for model weights while maintaining competitive tracking accuracy, offering practical benefits for deployment on devices with limited resources.

  2. The parameter-sharing design demonstrates applicability across different tracking frameworks. When applied to UTPTrack and OSTrack, the design reduces parameter counts by approximately 74%, with a decrease in LaSOT SUC of less than 0.5 percentage points for both trackers. These results provide empirical support for its applicability across different Transformer tracking frameworks and strengthen its practical value as a parameter-efficient tracking approach. The ablation studies are well targeted, and the paper also demonstrates flexible inference configurations.

Critical weaknesses
  1. Section 4.1 describes the overall training configuration using multiple datasets. It would be helpful to specify the training data and corresponding model checkpoint used for the GOT-10k results in Table 2, allowing readers to better understand the training protocol and the conditions underlying the comparisons.

  2. The paper and its appendix already describe the memory update mechanism and the associated losses. Further clarification of the temporal sampling of training frames, the number of frames unrolled during training, and how gradients are handled across memory updates would make the description of the training procedure more complete and provide clearer guidance for reproducing the method.

Questions for authors
  1. Clarification: Which training datasets and checkpoint produced the GOT-10k results, and do these match the evaluation conditions of the compared methods?
  2. Clarification: How are training frames sampled and ordered for GTM, when is memory reset, and are gradients propagated through previous-frame updates?
3 comments

We thank the reviewer for the careful evaluation of our work and the constructive comments. Our point-by-point responses are given below.

Q1: Which training datasets and checkpoints were used to produce the GOT-10k results? Are these results obtained under evaluation conditions consistent with those of the compared methods?

A1: Thank you for raising this point. The GOT-10k results are produced by the same model trained jointly on LaSOT, GOT-10k, TrackingNet, and VastTrack, as described in Sec. 4.1. The checkpoint is selected by AO on the GOT-10k validation set, and the GOT-10k test set is never used for checkpoint selection. Our results therefore follow the multi-dataset training protocol, which is also the protocol under which the compared results of UETrack, SUTrack, and MixFormerV2 are reported. We will state this setting explicitly in the revised manuscript.

Q2: How are training frames sampled, ordered, and unrolled for GTM? When is memory reset, and are gradients propagated through the preceding frame's memory update?

A2: Thank you for the question. We clarify the GTM training procedure below.

Sampling and ordering. Each training sample is a clip consisting of one template frame and two consecutive search frames from the same video, all in temporal order and with the target visible. The template frame precedes both search frames. The template image is fixed within the clip, and its features are extracted again each time a search frame is processed.

Unrolling and memory reset. At the start of each clip, the memory is initialized from the target features of the template. The model first processes the first search frame, pools the current target appearance guided by its final localization response, and updates the memory. The updated memory is then used when processing the second search frame. Each clip is initialized independently, so the memory is reset at the start of every clip and is never carried over from one clip to another.

Gradient flow. The losses of both search frames are used for training. The computational graph of the memory update is retained within a clip, so the loss on the second search frame backpropagates through the memory update performed on the first search frame. No graph is shared across clips. We will include these training details in the revised manuscript.

Thank you for the detailed response. The authors have adequately addressed my questions by clarifying the GOT-10k training and checkpoint selection protocol, as well as the temporal sampling, memory initialization, and gradient propagation used for GTM training. These clarifications resolve my main reproducibility concerns.

I believe this paper presents a simple, novel, and effective tracking framework that significantly reduces model parameters while maintaining competitive tracking performance. This is particularly valuable for practical deployment, where computational efficiency and model size are important considerations. Another strength is the generality of the proposed approach, which can be readily integrated into different tracking frameworks. The additional adaptive-depth experiments further demonstrate the flexibility of the method in balancing tracking accuracy and computational efficiency under different deployment constraints.

Overall, the rebuttal satisfactorily addresses my concerns and further strengthens the practical significance of the proposed method. I will increase my rating.

Thank you for recognizing our work. Your constructive feedback and positive comments have been invaluable to us. Once again, thank you for your support and encouragement.

Reviewer_U2zxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LoopTrack, a parameter-efficient Transformer framework for single-object visual tracking. The method repeatedly applies a small number of shared Transformer blocks, thereby decoupling the number of feature-interaction steps from the number of independently parameterized blocks. The authors further introduce Target-Aware Looping (TAL), which uses intermediate response maps to guide subsequent recurrent steps, and Gated Target Memory (GTM), which models cross-frame target appearance changes. Experiments show that LoopTrack achieves competitive tracking performance with a relatively small number of parameters. Overall, the paper is well structured, the motivation is clear, and the experiments cover multiple datasets and ablation settings. The main contribution is an efficient architectural design and engineering integration. However, the paper still has several issues regarding clarity of the method description, the completeness of efficiency analysis, and the fairness of comparisons.

Critical strengths

The paper’s main strength is its simple shared-parameter recurrent design, which enables multiple rounds of Transformer feature interaction with a small parameter budget. In addition, the ablation studies provide evidence that TAL and GTM make complementary contributions to within-frame refinement and cross-frame appearance modeling.

Critical weaknesses
  1. The main concern is that the paper’s central efficiency claim is supported primarily by parameter count, while the recurrent design still requires considerable computation. Without controlled latency and hardware-level evaluations, it is unclear whether LoopTrack provides a meaningful deployment advantage over existing compact trackers.
  2. Moreover, the unclear notation and potentially non-uniform baseline comparisons make the main empirical conclusions difficult to verify.
Questions for authors
  1. Efficiency comparison:
    Can the authors provide controlled latency and throughput comparisons with the main lightweight baselines using the same GPU, input resolution, batch size, and implementation settings? Since LoopTrack repeatedly executes shared Transformer blocks, how does its actual computational cost compare with these baselines beyond parameter count and MACs?

  2. Recurrent feature flow:
    Could the authors clarify the exact feature flow across recurrent steps? In particular, how is the template feature at each recurrent step obtained from the output of the shared Transformer cell, and is the template feature updated at every recurrent step? An explicit expansion of the first two recurrent steps, together with a clearer version of Figure 1, would help determine whether the implementation matches the equations.

  3. Robustness and reproducibility of the reported gains:
    Are the improvements from TAL and GTM consistent across multiple random seeds and across the reported datasets? Could the authors provide standard deviations or confidence intervals for the key ablation results, and release the complete training and evaluation configurations needed to reproduce the comparisons?

4 comments

We thank the reviewer for the careful evaluation of our work and the constructive comments. Our point-by-point responses are given below.

Q1: Can the authors provide latency and throughput comparisons with representative lightweight trackers under the same GPU, input resolution, batch size, and implementation settings, and clarify the actual overhead of repeatedly executing the shared modules?

Table A. Latency and throughput of all methods measured under identical settings (same GPU, input resolution, and implementation).

MethodLatency (ms, B=1) ↓Throughput (images/s, B=8) ↑
FARTrack13.571551.75
MixFormerV2-S3.2361928.24
LiteTrack-B43.5491693.12
LiteTrack-B95.968873.22
HCAT3.9401996.81
AsymTrack-B6.4701353.34
UETrack-T4.5941705.33
UETrack-S6.9851410.44
UETrack-B6.4801204.68
LoopTrackOne\mathbf{LoopTrack}_{\mathbf{One}}6.536710.67
LoopTrackTwo\mathbf{LoopTrack}_{\mathbf{Two}}8.403575.00
LoopTrackThree\mathbf{LoopTrack}_{\mathbf{Three}}10.204487.63

A1: Thank you for this suggestion. Table A reports the forward-pass latency at batch size 1 and the throughput at batch size 8 for all methods under the same settings, which quantifies the actual runtime cost of recurrent parameter sharing.

Repeatedly executing the shared blocks does incur a runtime overhead: LoopTrack has a higher latency and a lower throughput than most of the compared lightweight trackers. In return, it achieves higher tracking accuracy with fewer parameters. For example, compared with UETrack-T, LoopTrackTwo\mathrm{LoopTrack}_{\mathrm{Two}} uses 18.3% fewer parameters and improves the LaSOT SUC by 5.0 percentage points. Its single-frame latency of 8.403 ms still corresponds to about 119 FPS for the model forward pass. LoopTrack therefore targets parameter-constrained deployment, where reusing a small set of parameters for repeated feature refinement offers a favorable trade-off between model size and accuracy, at the cost of additional computation.

This overhead can be further reduced by adapting the inference depth to the input. We additionally evaluate an adaptive early-exit strategy based on TAL responses, which selects 4, 6, or 8 loops for each frame (see our response to Reviewer ap5n, Q2). Compared with a fixed K=8K=8, adaptive policy A reduces the average MACs by 13.96%, at the cost of only a 0.077-point drop in LaSOT SUC.

We have added the controlled latency and throughput comparison, the adaptive-depth results, and the corresponding analysis to the revised supplementary material (highlighted in blue; see Secs. B.5.2–B.5.3, Tables 17–18, L907–L935).

Q2: How are the template features at each loop obtained from the shared cell output? Are the template features updated at every loop? Please expand the feature flow over the first two loops and clarify the corresponding connections in Figure 1.

A2: Thank you for this question. The key is to distinguish initialization across frames from recurrent updates within a frame.

Within a frame: the template features are updated at every loop. At frame tt, the search, template, and memory tokens are concatenated and processed jointly by the shared cell Tθ\mathcal{T}_\theta. The template-token part of the cell output is passed directly, without any feedback, as the template input of the next loop. Only the search tokens receive the TAL feedback Rt(k)\mathbf R_t^{(k)} and the step embedding. The first two loops are

[Xˉt(1);Zˉt(1);mt(1)]=Tθ([Xt(0)+E(1);Z(0);mˉt−1]),[\bar{\mathbf X}_t^{(1)};\bar{\mathbf Z}_t^{(1)};\mathbf m_t^{(1)}] = \mathcal T_\theta\big([\mathbf X_t^{(0)}+\mathbf E^{(1)};\\ \mathbf Z^{(0)};\\ \bar{\mathbf m}_{t-1}]\big),

[Xˉt(2);Zˉt(2);mt(2)]=Tθ([Xˉt(1)+Rt(1)+E(2);Zˉt(1);mt(1)]),[\bar{\mathbf X}_t^{(2)};\bar{\mathbf Z}_t^{(2)};\mathbf m_t^{(2)}] = \mathcal T_\theta\big([\bar{\mathbf X}_t^{(1)}+\mathbf R_t^{(1)}+\mathbf E^{(2)};\\ \bar{\mathbf Z}_t^{(1)};\\ \mathbf m_t^{(1)}]\big),

and all subsequent loops follow the same rule. Hence, the template features are refined by the shared cell at every loop, and Zˉt(k)\bar{\mathbf Z}_t^{(k)} is the template input of loop k+1k+1.

Across frames: the template restarts from the fixed initial template. The initial template image zz and its features Z(0)\mathbf Z^{(0)} are obtained from the first frame and remain fixed throughout tracking. For every new frame, the template branch starts again from Z(0)\mathbf Z^{(0)}; the refined template features Zˉt(K)\bar{\mathbf Z}_t^{(K)} are not carried to the next frame. Cross-frame information is carried only by the GTM memory mˉt\bar{\mathbf m}_t, which is updated after localization in each frame. In short, the initial template is fixed across frames, while its feature representation is progressively refined by the recurrent interaction within each frame.

We have revised Figure 1 to show explicitly how the template and memory features output by each loop are passed to the next loop. Please see Figure 1 in the revised manuscript; the corresponding feature flow is described in Sec. 3.2 (L216–L258).

Q3: Are the gains from TAL and GTM consistent across different training seeds and datasets? Can the authors provide standard deviations or confidence intervals, together with the complete training and evaluation configurations?

Table B. Component ablations over three independent training seeds (mean ± standard deviation, %). ΔTAL and ΔGTM are in percentage points.

Dataset / metricBaseline+ TAL+ TAL + GTMΔTALΔGTM
LaSOT / SUC65.80 ± 0.2168.80 ± 0.1769.30 ± 0.17+3.00+0.50
LaSOT / Normalized precision75.50 ± 0.2078.60 ± 0.1979.20 ± 0.16+3.10+0.60
LaSOT / Precision70.80 ± 0.2274.10 ± 0.1774.70 ± 0.22+3.30+0.60
LaSOText\mathrm{LaSOT}_{\mathrm{ext}} / SUC47.10 ± 0.2249.00 ± 0.2249.50 ± 0.17+1.90+0.50
TrackingNet / SUC81.60 ± 0.1682.40 ± 0.1382.70 ± 0.17+0.80+0.30
GOT-10k / AO71.70 ± 0.1872.60 ± 0.2173.10 ± 0.15+0.90+0.50

A3: Thank you for this suggestion. We repeated the key component ablations with three independent training seeds using the default LoopTrackThree\mathrm{LoopTrack}_{\mathrm{Three}} model (M=3M=3, K=8K=8). Table B reports the mean and standard deviation. ΔTAL is the mean gain from adding TAL to the baseline, and ΔGTM is the additional mean gain from adding GTM on top of TAL. TAL brings positive mean gains on all four benchmarks, and GTM brings further gains on all of them; both gains are larger than the seed-to-seed standard deviations. The two precision metrics on LaSOT show the same trend as SUC.

Regarding the training and evaluation configurations, Sec. 4.1 of the original manuscript specifies the training datasets and their sampling ratio, the pretrained initialization, the input sizes, the optimizer, the learning-rate schedule, and the two-stage training procedure. Sec. 4.2 describes the evaluation benchmarks and metrics, and Appendix A gives the implementation and auxiliary-supervision details of TAL and GTM. All component ablations use the default LoopTrackThree\mathrm{LoopTrack}_{\mathrm{Three}} model with a fixed inference depth of K=8K=8.

We have added the multi-seed ablation results and the corresponding analysis to the revised supplementary material (highlighted in blue; see Sec. B.6, Table 20, L966–L979).

Thanks for the detailed response and the additional experiments. The new runtime results are helpful, and I think they now make the main trade-off reasonably clear. Parameter sharing does reduce model size, but the repeated execution of shared blocks increases MACs and forward latency compared to many lightweight trackers. I appreciate that the authors acknowledge this and also evaluate different loop depths, both fixed and adaptive. That does address my original concern about the lack of a controlled efficiency comparison. Just to be explicit: since the timing excludes loading, cropping, and post-processing, the reported FPS should be read as model forward-pass speed, not end-to-end tracking speed.

The revised equations and Figure 1 improve the feature-flow explanation, especially the distinction between template refinement within a frame and restarting from the fixed initial template across frames. The three-seed ablations are also useful, and the mean gains from TAL and GTM look positive across the tested datasets. The standard deviations are reported, but three seeds are still quite limited for claiming statistical significance.

Overall, the rebuttal addresses my main concerns enough that I will increase my score to 3 (Borderline Accept).

My only remaining comment is that the deployment trade-off should be stated more clearly in the main paper. LoopTrack reduces parameters and memory cost, but it does not necessarily offer lower latency than many lightweight trackers. Making that distinction explicit would strengthen the deployment claim.

Thank you for your positive assessment of our work and your constructive feedback. We also greatly appreciate your careful review of our responses and additional experiments. Following your suggestions, we will further revise the paper to explain the deployment trade-offs more clearly. We will also explicitly state that the reported FPS reflects only the model’s forward-pass speed. Thank you again for your detailed comments, which have helped us further improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.