acceptodds
Under review as a conference paper at ICLR 2027

ReTrack: Learning Recovery Decisions from Trajectory Replay

Abstract

Visual trackers commonly assume limited target displacement across consecutive frames and localize the target within a local search region cropped around the previous prediction. However, the tracker may lose a small target when rapid motion moves it outside this local window or brief occlusion disrupts localization. It may then drift onto background clutter and fail to recover. To recover a lost target, existing trackers combine local tracking with full-frame re-detection, often relying on a separately trained external detector and hand-crafted rules based on current-frame confidence to trigger recovery. Nevertheless, these rules decide whether to switch from current-frame cues, without training on how staying and switching affect the subsequent trajectory. To close this gap, we introduce ReTrack, which learns recovery decisions through trajectory-replay supervision. By rolling out the stay and switch trajectories over subsequent frames and evaluating both against the ground-truth trajectory, we directly measure the relative advantage of switching. At inference, a lightweight policy predicts this advantage from relative geometric, confidence, and temporal cues. It then decides whether to retain the tracker prediction or switch to a full-frame candidate. These candidates are provided by a full-frame branch that shares the same backbone as the tracker, eliminating the need for a separately trained detector. With full-frame candidates and learned recovery decisions, ReTrack achieves 49.07% AUC on UAV-Anti-UAV, surpassing the previous best result by 5.04% AUC, and improves performance on three additional drone tracking benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Re-detection, Trajectory replay, Decision learning, Small object tracking

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bhFxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

ReTrack addresses recovery after target loss in visual object tracking. Conventional trackers typically search the nearby region of the previous prediction and may drift onto background clutter when occlusion or rapid motion causes the target to leave the search region. The proposed framework combines a local tracker, a full-frame proposal head, and a lightweight decision network to enable global recovery. During training, staying with the current prediction and switching to different candidates are rolled out from the same tracker state. Differences in their subsequent mean IoU provide supervision for the decision network. At inference, the network estimates switching benefits to select a recovery candidate or retain the local prediction. Extensive experiments demonstrate the effectiveness of the proposed framework comparing to the conventional trackers.

Critical strengths
  1. The proposed framework is technically sound, and the paired rollout design provides a clear supervision signal for recovery decisions. By comparing staying and switching from the same tracker state, it directly measures the subsequent benefit of different switching decisions and generates labels for each decision using only existing bounding-box annotations. A lightweight decision network predicts these benefits at inference, enabling informed recovery decisions with little additional computational cost.

  2. ReTrack improves over the base tracker across all four main benchmarks and achieves the highest AUC on three. The component ablations further support the contributions of full-frame recovery and the learned decision policy.

Critical weaknesses
  1. The switching behavior and temporal stability of the recovery mechanism are insufficiently characterized. The paper does not report the average number of switches per sequence, the spatial displacement associated with each switch, or the full-sequence trajectory visualizations. These statistics are needed to characterize switching frequency and abrupt trajectory changes during difficult sequences. Additional experiments and analysis would help establish the stability of the recovery mechanism.
Questions for authors
  1. Could the authors provide sensitivity analyses for the confidence threshold and periodic interval used to trigger decision points?

  2. Given that the full-frame proposal head is trained with bounding-box localization and confidence losses, how does it distinguish the designated target from other objects, and what evidence demonstrates target identity recognition beyond reliance on spatial cues?

  3. Could the authors provide full-sequence trajectory visualizations and quantitative analyses of switching behavior, including the average number of switches per sequence and the displacement of each switch, to assess recovery stability and repeated global relocalization?

4 comments
AuthorsOpenReview ↗

Response to Reviewer bhFx (1/2)

We thank the reviewer for the positive assessment and the constructive suggestions, and for finding the framework "technically sound" and the paired rollout design "a clear supervision signal for recovery decisions".

Q3 (W1): Could the authors provide full-sequence trajectory visualizations and quantitative analyses of switching behavior, including the average number of switches per sequence and the displacement of each switch, to assess recovery stability and repeated global relocalization?

The switches are sparse and restore tracking after failures.

(1) Full-sequence visualizations. We added them to §4.4 of the revised paper as Fig. 6. For two long sequences of 5,236 and 7,444 frames, Fig. 6(b, c) plots the per-frame IoU of ReTrack and of the same network without switching. ReTrack recovers the target after most losses, raising the sequence AUC from 21.6% to 57.5% and from 14.5% to 55.3%, whereas the tracker without switching remains lost for most of each sequence.

(2) Switching statistics. The table below summarizes the full run on the 420 evaluation sequences of UAV-Anti-UAV. The policy switches at 4.4% of the decision points, and the mean number of switches exceeds the median because a few difficult sequences account for most switches.

StatisticReTrackReTrack without switching
Decision points (% of frames)28.9–
Switches (% of decision points)4.4–
Median switches per sequence50
Mean switches per sequence14.30
Median switch displacement (fraction of the image diagonal)0.25–
Mean IoU 30 frames after repeated switches (1,159 frames)0.4790.189
AUC (%)49.0741.21

A switch moves the box by a quarter of the image diagonal at the median, well beyond the local search region around a small target. Repeated relocalization appears as several switches within a short time. For every group of switches spaced less than 30 frames apart, we measure the IoU 30 frames after the last switch of the group, at the 1,159 such frames with the target visible. ReTrack achieves a mean IoU of 0.479 at these frames, compared with 0.189 for the same network without switching, indicating that repeated switches converge on the target rather than resulting in drift.


Q1: Could the authors provide sensitivity analyses for the confidence threshold and periodic interval used to trigger decision points?

ReTrack is insensitive to both settings, and AUC stays between 48.78% and 49.09% across a wide range of these settings. The two settings only decide which frames become decision points, and the learned policy then decides whether and to which proposal to switch. The table below varies the confidence threshold csoftc_{\mathrm{soft}} and the period MM of the decision points on UAV-Anti-UAV while keeping the tracker and the policy fixed.

Decision pointsFraction of frames (%)AUC (%)PP (%)PnormP_{\mathrm{norm}} (%)
csoft=0.4c_{\mathrm{soft}} = 0.422.349.0965.8554.19
csoft=0.5c_{\mathrm{soft}} = 0.524.948.9465.6654.03
csoft=0.6c_{\mathrm{soft}} = 0.6, M=10M = 10 (ReTrack)28.949.0765.8454.17
csoft=0.7c_{\mathrm{soft}} = 0.737.248.8065.4653.87
csoft=0.8c_{\mathrm{soft}} = 0.857.848.0764.3053.04
M=1M = 1 (every frame)10046.8462.6851.67
M=5M = 536.948.9065.5953.98
M=20M = 2024.948.9965.7254.08
No periodic trigger20.948.7865.4453.84

From csoftc_{\mathrm{soft}} = 0.4 to 0.7 and from MM = 5 to no periodic trigger, the decision points range from 20.9% to 37.2% of the frames, while AUC changes by at most 0.29%. Even the extreme settings, csoftc_{\mathrm{soft}} = 0.8 with 57.8% of the frames as decision points and a decision at every frame, reach 48.07% and 46.84% AUC. Thus, the reported setting lies within a stable range rather than at a sharp optimum. We added this analysis to §4.4 of the revised paper (Tab. 8).

AuthorsOpenReview ↗

Response to Reviewer bhFx (2/2)

Q2: Given that the full-frame proposal head is trained with bounding-box localization and confidence losses, how does it distinguish the designated target from other objects, and what evidence demonstrates target identity recognition beyond reliance on spatial cues?

The full-frame head proposes target-like locations, and the policy identifies the target among them by comparing each proposal with the current track.

(1) Evidence beyond spatial cues (Tab. 5). The policy compares each proposal with the current track through four groups of relative features: tracker reliability (the tracker confidence and its smoothed value), proposal quality (the proposal score and its rank), the spatial relation (the center distance and the scale ratio), and the time since the last switch. The tracker confidence is derived from matching the initial template with the search region, so it carries the appearance evidence of the target. Removing the tracker confidence lowers AUC from 49.07% to 43.17% (Tab. 5(b)), whereas removing the geometric features lowers AUC to 46.69% (Tab. 5(d)). Therefore, the policy relies more on the appearance evidence of the template than on the spatial cues.

(2) Evidence with similar distractors. UAV-Anti-UAV annotates whether a sequence contains a visually similar distractor. The table below reports the AUC on the two subsets.

MethodAll sequences (420)Similar distractor (222)No similar distractor (198)
LoRAT-ID42.2237.0648.02
ReTrack without switching41.2136.9346.02
Hand-crafted rule45.9740.2452.40
ReTrack49.0743.6055.20

ReTrack achieves the highest AUC on both subsets, and its margin over the hand-crafted rule is larger with a similar distractor (3.36% versus 2.80%), where comparing each proposal with the current track is most critical.

In Fig. 6(a) of the revised paper, the current track has drifted onto a car with a confidence of 0.37, the full-frame head responds most strongly at the drone, and the policy switches to it. Over the next 30 frames with the target visible, ReTrack reaches a mean IoU of 0.80, while ReTrack without switching remains lost.


We hope these responses address the reviewer's questions, and we are glad to discuss any remaining points.

I thank the authors for their response. They have addressed my concerns, and I have no further questions.

AuthorsOpenReview ↗

Thank You to Reviewer bhFx

We thank the reviewer for confirming that our responses address the concerns, and for the positive assessment of ReTrack. The suggestions on switching behavior and on the decision-point settings led to Fig. 6 and Tab. 8 in §4.4 of the revised paper.

Reviewer_CeLYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses target loss in visual object tracking (especially small/fast-moving targets under occlusion or out-of-view in drone scenarios). Standard local-search trackers fail to recover once the target leaves the search region; prior recovery methods rely on a separately trained detector plus hand-crafted confidence rules. ReTrack instead attaches a full-frame head that shares the tracker backbone (eliminating a separate detector) and learns a lightweight policy that decides whether to stay with the local prediction or switch to a full-frame candidate. The key idea is trajectory-replay supervision: from the same tracker state the authors roll out both the “stay” and “switch” trajectories for n frames, measure the difference in mean IoU against ground truth, and treat that difference as a continuous switching-advantage label. The policy is trained on relative geometric/confidence/temporal features only (appearance-free) and is collected via iterative DAgger-style rounds with out-of-fold cross-fitting. Experiments on four anti-UAV benchmarks show a clear gain (especially +5.04 AUC on UAV-Anti-UAV) and the recovery pipeline transfers to three additional base trackers.

Critical strengths
  1. Sharing the backbone for the full-frame head removes the need for a domain-specific detector and keeps extra cost modest (full-frame branch is invoked only at sparse decision points). The policy itself is tiny.
  2. Converting passive box annotations into paired counterfactual rollouts and defining a switching advantage is elegant and directly optimizes the quantity that matters for recovery. This is more principled than pure confidence thresholding or single-action RL rewards used in prior work.
  3. Large gains on the most challenging small-target benchmark, attribute-wise analysis that aligns with the claimed failure modes (fast motion, out-of-view), thorough ablations on components/policy features/replay horizon, and successful transfer across three other trackers.
  4. The recovery pipeline successfully transfers to multiple base trackers (such as OSTrack, LETrack, and UTPTrack).
Critical weaknesses
  1. The manuscript does not mention the availability of code, which may hinder the reproducibility and further research by other scholars.
  2. Although the inference cost is well-controlled (only a 22% increase in MACs), the training pipeline requires complex out-of-fold cross-fitting and multiple offline collection rounds (N=3 rounds, accumulating 98,802 decisions). This multi-stage training framework introduces substantial engineering complexity and high GPU resource consumption (e.g., 8.4 GPU-hours per cross-fitting tracker), which may limit its reproducibility and adoption on broader, larger-scale datasets.
  3. The strategy is entirely based on geometry, confidence, and temporal characteristics, explicitly excluding appearance. The qualitative results of the paper (Figure 5, row 4) have revealed that high-score appearance-similar distractors are wrongly accepted. May I ask: In the case of small target scenarios, is this "appearance-independent" design necessary (because the appearance information itself is extremely weak), or is it merely to maintain lightness? If the simplest template-proposal cosine similarity or lightweight verification head is added, what are the impacts on the failure cases of the distractors and the overall AUC?
  4. The strategy is claimed to be "learning to restore decisions", but the decision points themselves are still determined by fixed confidence threshold + periodic triggering. If the strategy truly learns "when to switch", why not directly use the strategy output (or historical features) to decide whether to activate the full-frame branch? Is the current design essentially still "rule triggering + learning sorting", rather than end-to-end restoration decision learning?
  5. Table 4 shows that a drone-specific detector with a hand-crafted rule reaches 49.75% AUC, while ReTrack reaches 49.07%. The main benefit of ReTrack is avoiding a separate detector, but the paper does not fully demonstrate that its accuracy–efficiency trade-off is superior.
  6. The paper argues that the full-frame branch is cheap because it uses only 240 tokens and runs only at decision points. However, no FPS, latency, FLOPs, or average trigger frequency is reported. Therefore, the claimed efficiency advantage is not empirically established.
Questions for authors
  1. The policy explicitly excludes appearance and relies only on geometry, confidence, and temporal cues. Is this design necessary because appearance information for small targets is inherently weak, or mainly for efficiency? If you add a simple template–proposal cosine similarity or a lightweight verification head, how does it affect the distractor failure cases in Figure 5 and the overall AUC?
  2. Decision points are still triggered by a fixed confidence threshold plus periodic intervals. If the policy truly learns “when to switch,” why not let the policy or historical features directly trigger the full-frame branch? Is the current design essentially rule-based triggering plus learned ranking?
  3. The full pipeline requires cross-fitting, three collection rounds, and 98,802 decisions. Can you report total GPU hours versus standard tracker training, and provide FPS, latency, FLOPs, and average trigger frequency? Does the 22% MACs increase cover the full pipeline or only the full-frame branch?
  4. The replay horizon is fixed to n=10 frames, and full occlusion gains the least. How reliable are the advantage labels when the target disappears for a long time or leaves the view? Have you tried longer or adaptive horizons, and how does ReTrack behave when the target reappears after 30–50 frames?
7 comments
AuthorsOpenReview ↗

Response to Reviewer CeLY (1/4)

We thank the reviewer for the detailed review and specific questions, and for describing the paired counterfactual rollouts as "elegant" and more principled than "pure confidence thresholding or single-action RL rewards".

W1: The manuscript does not mention the availability of code, which may hinder the reproducibility and further research by other scholars.

We respectfully note that the supplementary material provides the code, and the paper states its release. The supplementary material contains five modules of the recovery pipeline.

FileContent
fullframe_branch.pyFull-frame branch with top-K proposals
motion_residual.pyMotion residual
replay.pyCounterfactual stay-and-switch replay and the switching advantage of Eq. 1
decision.pyDecision points, decision representation, and label collection
policy.pyAdvantage-regression policy

A README maps each file to its section of the paper and describes the training order. The Reproducibility Statement also states that the raw prediction files for all reported results will be released alongside the code. We will release the full code after the review process.


Q3 (W2, W6): The full pipeline requires cross-fitting, three collection rounds, and 98,802 decisions. Can you report total GPU hours versus standard tracker training, and provide FPS, latency, FLOPs, and average trigger frequency? Does the 22% MACs increase cover the full pipeline or only the full-frame branch?

All training stages take about 31.0 GPU-hours, 3.8 times the standard training of the final tracker, and the 22% MACs increase covers the full pipeline.

(1) Training cost. The table below summarizes the cost of each stage on two RTX 5090 GPUs.

StageDataGPU-hours
Final tracker (standard training)all training videos8.23
Cross-fitting tracker Aone half of the training videos8.46
Cross-fitting tracker Bthe other half8.41
Collection round 1, forward passes only450 held-out videos per tracker, switching with the hand-crafted rule1.87
Collection round 2, forward passes only450 held-out videos per tracker, switching with the current policy1.95
Collection round 3, forward passes only450 held-out videos per tracker, switching with the current policy2.06
Policy fitting98,802 proposal samples from 32,934 decision pointsabout 10 min on one CPU
Total31.0 (3.8 × the final tracker)

The extra stages use only the existing box annotations and run once per training set. The cross-fitting trackers are discarded after collection, so inference uses only the final tracker and the policy.

(2) Inference cost. We respectfully note that Tab. 7 of the paper reports the computation in MACs. As in fvcore and thop, we count one multiply-accumulate as one FLOP, so the FLOPs equal the MACs. The full-frame branch runs only at decision points, the frames with low tracker confidence and every 10th frame, and encodes the whole frame with 240 tokens, fewer than the 256 tokens of the local search region. The policy, an MLP with fewer than 5K parameters, adds negligible computation. The table below lists the requested metrics.

MetricReTrackNote
Trigger frequency (% of test frames)28.9Decision points, at which the full-frame branch and the policy run
Switch frequency (% of decision points)4.4Switches of the policy
MACs (G)33.72 (+22% over LoRAT-ID)Average over test frames, 27.70 (tracking branch) + 28.9% × 20.84 (full-frame branch)
FLOPs (G)33.72One multiply-accumulate counted as one FLOP
Latency per call (ms)5.64 (tracking branch), 5.79 (full-frame branch)GPU forward pass
Speed (FPS)33.9One RTX 5090 GPU, end to end

The trigger frequency and the latency of the full-frame branch are now in Tab. 7 of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer CeLY (2/4)

W5: Table 4 shows that a drone-specific detector with a hand-crafted rule reaches 49.75% AUC, while ReTrack reaches 49.07%. The main benefit of ReTrack is avoiding a separate detector, but the paper does not fully demonstrate that its accuracy–efficiency trade-off is superior.

Regarding the accuracy–efficiency trade-off, ReTrack stays within 0.68% AUC of the drone-specific detector (49.07% versus 49.75%) with about one-quarter of its additional parameters, 11% fewer FLOPs per call, and no separately trained model.

(1) The detector setting. The detector is a separate YOLOv8s model with an additional stride-4 head, pre-trained on COCO and trained on the same UAV-Anti-UAV training split as ReTrack. The detector processes the full frame at 960 × 544 pixels for the 16:9 frames that form most of the dataset, approximately eleven times the pixel count of the 280 × 168 input of the ReTrack full-frame branch. The hand-crafted rule thresholds the tracker confidence, the detector score, and the displacement.

(2) Accuracy and cost (Tab. 4 and Tab. 7). The FLOPs are counted per call of the proposal source, which runs only at decision points.

ProposalsDecisionAUC (%)PP (%)PnormP_{\mathrm{norm}} (%)Additional parameters (M)FLOPs per call (G)Separately trained model
Drone-specific detectorHand-crafted rule49.7566.2554.7410.6423.45Yes
Drone-specific detectorLearned decision48.9365.7154.0010.6423.45Yes
Shared headHand-crafted rule45.9761.2250.692.7120.84No
Shared head (ReTrack)Learned decision49.0765.8454.172.7120.84No

In terms of efficiency, the shared head adds 2.71M parameters, about one-quarter of the 10.64M of the detector, and requires 20.84 GFLOPs per call, 11% fewer than the 23.45 GFLOPs of the detector. In terms of accuracy, the learned decision closes most of the gap. The hand-crafted rule switches to the top proposal once the thresholds of Appendix D.1 are met, so its AUC drops from 49.75% with the detector proposals to 45.97% with the shared-head proposals. The policy instead compares each proposal with the current track and raises the AUC of the shared head to 49.07%. Therefore, the shared head with the learned decision trades 0.68% AUC for a smaller and cheaper proposal source that requires no separate training.


Q1 (W3): The policy explicitly excludes appearance and relies only on geometry, confidence, and temporal cues. Is this design necessary because appearance information for small targets is inherently weak, or mainly for efficiency? If you add a simple template–proposal cosine similarity or a lightweight verification head, how does it affect the distractor failure cases in Figure 5 and the overall AUC?

The appearance-free design is driven mainly by the weak appearance of small targets rather than efficiency, and adding appearance similarities does not improve the decision. Targets of at most 15 pixels provide the template with little appearance evidence, and ReTrack gains the most on them (Fig. 4(a)). The relative cues are also available in every base tracker, so the design transfers to OSTrack, LETrack, and UTPTrack (Tab. 6).

To test appearance directly, we add two features to every proposal: the cosine similarities between the pooled backbone embedding of the proposal crop and those of (i) the first-frame template and (ii) the current tracked box. The decision points, the nine existing features, and the MLP are unchanged. Both policies are fitted on one collection round that also records the proposal embeddings.

Policy inputAUC (%)PP (%)PnormP_{\mathrm{norm}} (%)AUC on similar distractors (%)
9 relative features (ReTrack)47.0662.4551.8240.96
+ template and track cosine similarities47.0062.3251.7640.84

Each cosine feature alone correlates weakly with the replayed advantage (0.10 and 0.11). The partial correlations further drop to 0.067 and 0.036 when the nine relative features are accounted for, indicating that these features already capture most of the appearance evidence. In the distractor case shown in Fig. 5 (row 4), the same tracker with switching disabled also drifts onto the distractor (16.99% AUC, compared with 16.51% for ReTrack). This indicates that the failure occurs in local tracking before any decision is made, and adding appearance features does not resolve this issue (17.40%). We added this result to the discussion of Fig. 5 in §4.4 of the revised paper and the full table to Appendix E (Tab. A7).

AuthorsOpenReview ↗

Response to Reviewer CeLY (3/4)

Q2 (W4): Decision points are still triggered by a fixed confidence threshold plus periodic intervals. If the policy truly learns "when to switch," why not let the policy or historical features directly trigger the full-frame branch? Is the current design essentially rule-based triggering plus learned ranking?

The trigger only selects the frames at which the policy runs, and the policy decides whether to switch, not only to which proposal.

At inference, ReTrack proceeds as follows: local tracking (box and tracker confidence cc) → decision point if c<csoftc < c_{\mathrm{soft}} or at every MM-th frame → full-frame branch (top-KK proposals) → policy (predicted IoU gain of switching to each proposal over staying) → switch to the best proposal if its gain exceeds τ, otherwise stay. Thus, the trigger only decides whether the full-frame branch runs, while the policy makes the switching decision.

(1) Why the policy does not trigger the full-frame branch. The policy scores the full-frame proposals, which exist only after the full-frame branch has run, so it cannot decide when this branch runs. A trigger learned from historical features would need to predict tracking failures using only tracker signals. For LoRAT, the tracker confidence already detects tracking failures with an AUROC of 0.94 (Tab. A6), and the periodic trigger covers the failures in which the confidence stays high. Moreover, accuracy is insensitive to the trigger setting (Tab. 8 in §4.4 of the revised paper). Varying csoftc_{\mathrm{soft}} from 0.4 to 0.7 and MM from 5 to no periodic trigger keeps AUC between 48.78% and 49.09%, while making every frame a decision point lowers AUC to 46.84%, so evaluating the policy at more frames does not improve accuracy. Learning the trigger with counterfactual replay is a natural extension, as the conclusion notes.

Decision pointsFraction of frames (%)AUC (%)PP (%)PnormP_{\mathrm{norm}} (%)
csoft=0.4c_{\mathrm{soft}} = 0.422.349.0965.8554.19
csoft=0.5c_{\mathrm{soft}} = 0.524.948.9465.6654.03
csoft=0.6c_{\mathrm{soft}} = 0.6, M=10M = 10 (ReTrack)28.949.0765.8454.17
csoft=0.7c_{\mathrm{soft}} = 0.737.248.8065.4653.87
csoft=0.8c_{\mathrm{soft}} = 0.857.848.0764.3053.04
M=1M = 1 (every frame)10046.8462.6851.67
M=5M = 536.948.9065.5953.98
M=20M = 2024.948.9965.7254.08
No periodic trigger20.948.7865.4453.84

(2) Learned switching rather than learned ranking. The policy decides whether to switch at all, not only which proposal ranks first. It predicts the IoU gain of switching relative to staying based on the counterfactual replay of both actions, making zero the natural boundary between the two actions. It switches when the largest predicted gain exceeds τ = 0.05, a small margin above zero, and AUC changes by only 0.34% as τ ranges from 0.05 to 0.30 (Fig. 4(b)). At the same decision points, Tab. 3 changes only the decision. Always switching to the top-scored proposal reduces AUC to 39.27%, indicating that the performance gain stems from deciding whether to switch.

Switching strategy (Tab. 3)AUC (%)PP (%)PnormP_{\mathrm{norm}} (%)
(a) Never switch41.2155.3745.51
(b) Random switch40.7953.8244.87
(c) Always switch39.2752.0343.12
(d) Inverted advantage27.5035.5430.13
(e) Learned advantage (ReTrack)49.0765.8454.17
Reviewer_42rdOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReTrack, a visual tracking framework designed to recover from target loss caused by rapid motion, occlusion, or departure from a local search region. The framework combines a local tracking branch with a full-frame proposal branch that shares the tracker's backbone, along with a lightweight policy that decides whether to retain the current tracking box or switch to a recovery candidate.The main methodological contribution is trajectory-replay supervision. For each decision state, the authors roll out both the stay trajectory and candidate switch trajectories over a future horizon, and define the switching advantage as the difference in their mean IoU with the ground-truth trajectory. A policy is then trained to predict this advantage from relative geometric, confidence, and temporal features. Cross-fitted collection is used to gather decision states from sequences not used to train the corresponding tracker. The method is evaluated on four Anti-UAV benchmarks and is reported to achieve the best AUC on three of them. The paper also includes component ablations, comparisons of alternative switching strategies, transfer experiments with three additional base trackers, and a computational-cost analysis. The results are promising, particularly on small-target and fast-motion scenarios. However, the independent contribution of trajectory-replay supervision, the transferability of the learned policy from cross-fitted trackers to the final tracker, and the practical inference-cost trade-offs could be established more thoroughly.

Critical strengths

The core concept of comparing "maintaining" versus "switching" trajectories—starting from the same tracking state—is well-founded. Unlike approaches that rely on fixed confidence thresholds, the "switching advantage" objective directly measures whether a candidate option can improve subsequent tracking quality. This provides a clear theoretical basis for learning "recovery decisions." The results of ablation studies further confirm this, validating the effectiveness of the proposed method by comparing it against alternative strategies: learning-based advantage estimation, random switching, always switching," and "never switching.

Critical weaknesses

W1: The paper distinguishes ReTrack from prior approaches by directly comparing stay and switch trajectories from the same state and using their IoU difference as supervision. This is a reasonable and potentially useful contribution. However, the broader idea of learning tracking actions or switching decisions has been explored in prior work, including learning-based tracking policies and detector-switching methods discussed in the paper. The current evaluation demonstrates that the complete ReTrack system is effective, but it does not fully establish how much of the gain is specifically attributable to the counterfactual trajectory-replay target, as opposed to the shared full-frame branch, additional in-domain training, or the overall recovery pipeline. More controlled comparisons against existing learned switching policies or alternative supervision targets would strengthen the novelty claim.

W2: The policy is trained using decision states collected by "cross-fitted" trackers, whereas the tracker ultimately used for evaluation is trained on the full training set. Although "cross-fitting" helps avoid collecting states exclusively from the same sequences used to train the corresponding trackers, the resulting distributions of confidence scores, candidate quality, and failure states may differ from those associated with the final tracker. Since the policy relies heavily on confidence scores and geometric features, this distributional mismatch can affect the accuracy of its predicted switching advantages.

Questions for authors

Please refer to W1 and W2.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 42rd (1/2)

We thank the reviewer for the constructive feedback and for finding the comparison of staying and switching from the same tracking state "well-founded".

W1: The current evaluation demonstrates that the complete ReTrack system is effective, but it does not fully establish how much of the gain is specifically attributable to the counterfactual trajectory-replay target, as opposed to the shared full-frame branch, additional in-domain training, or the overall recovery pipeline. More controlled comparisons against existing learned switching policies or alternative supervision targets would strengthen the novelty claim.

Beyond in-domain training, the improvement of ReTrack stems from the learned switching decision supervised by the counterfactual replay target. When the same decision policy is trained on each supervision target, the counterfactual replay target outperforms the supervision of existing learned switching policies and alternative supervision targets. All tables below hold the tracker, the proposals, and the decision points constant. Items (1) and (2) compare supervision targets, while (3) isolates the remaining components via the paper's ablations.

(1) Supervision of existing learned switching policies. We train the same policy, with the same features, decision points, and proposals, on the supervision of four existing learned switching policies. Following the candidate identification of SPLT [1], Siam R-CNN [2], and KeepTrack [3], the policy classifies whether a proposal is the target. Following LTMU [4], it determines whether the current box remains reliable and switches to the top-scored proposal otherwise. All policies are trained on a single collection round and compared with the single-round ReTrack policy (Tab. A2 of the revised paper). Both supervision methods fall below the counterfactual replay target, which compares switching with staying from the same state.

Supervision targetAdopted fromComparison with stayingAUC (%)PP (%)PnormP_{\mathrm{norm}} (%)
Proposal identity (binary)SPLT [1], Siam R-CNN [2], KeepTrack [3]No45.9361.5650.71
Tracking reliability (binary)LTMU [4]No45.6360.1250.21
Counterfactual switching advantageReTrackYes48.2364.6353.20

(2) Alternative supervision targets (Tabs. 3 and 5). Row (h) of Tab. 5 trains the same policy on the IoU of the switch rollout alone, without comparing it with the stay rollout from the same state. The counterfactual comparison with the stay rollout raises AUC by 2.06%, PP by 1.55%, and PnormP_{\mathrm{norm}} by 2.27%. Inverting the paired advantage collapses AUC to 27.50%. These results indicate that the policy learns from comparing switching with staying from the same state, which the counterfactual replay target provides.

Supervision targetAUC (%)PP (%)PnormP_{\mathrm{norm}} (%)
Switch-rollout IoU only, Tab. 5(h)47.0164.2951.90
Inverted paired advantage, Tab. 3(d)27.5035.5430.13
Paired advantage (ReTrack)49.0765.8454.17

(3) In-domain training, full-frame branch, and recovery pipeline (Tabs. 2 and 3). Retraining LoRAT on the training split increases AUC from 37.78% to 42.22%, and the other rows in the table below are built upon this retrained tracker. Adding the shared full-frame branch without switching to its proposals yields 41.21% AUC, whereas switching randomly at the rate of the learned policy or always to the top proposal further lowers AUC. Thus, the full-frame branch and the recovery pipeline improve tracking only through the learned decision of when and to which proposal to switch.

SettingAUC (%)PP (%)PnormP_{\mathrm{norm}} (%)
LoRAT, Tab. 2(a)37.7850.9741.18
+ in-domain training, Tab. 2(b)42.2256.1546.44
+ full-frame branch, never switch, Tab. 3(a)41.2155.3745.51
+ full-frame branch, random switch, Tab. 3(b)40.7953.8244.87
+ full-frame branch, always switch to the top proposal, Tab. 3(c)39.2752.0343.12
+ full-frame branch, learned advantage (ReTrack), Tab. 3(e)49.0765.8454.17

References

[1] B. Yan, H. Zhao, D. Wang, H. Lu, and X. Yang. 'Skimming-Perusal' Tracking: A Framework for Real-Time and Robust Long-Term Tracking. ICCV, 2019.

[2] P. Voigtlaender, J. Luiten, P. H. S. Torr, and B. Leibe. Siam R-CNN: Visual Tracking by Re-Detection. CVPR, 2020.

[3] C. Mayer, M. Danelljan, D. P. Paudel, and L. Van Gool. Learning Target Candidate Association to Keep Track of What Not to Track. ICCV, 2021.

[4] K. Dai, Y. Zhang, D. Wang, J. Li, H. Lu, and X. Yang. High-Performance Long-Term Tracking with Meta-Updater. CVPR, 2020.

AuthorsOpenReview ↗

Response to Reviewer 42rd (2/2)

W2: The policy is trained using decision states collected by "cross-fitted" trackers, whereas the tracker ultimately used for evaluation is trained on the full training set. Although "cross-fitting" helps avoid collecting states exclusively from the same sequences used to train the corresponding trackers, the resulting distributions of confidence scores, candidate quality, and failure states may differ from those associated with the final tracker. Since the policy relies heavily on confidence scores and geometric features, this distributional mismatch can affect the accuracy of its predicted switching advantages.

The evaluation states do differ from the training states, and the policy remains effective under these differences. On four evaluation sets, the mean tracker confidence ranges from 0.47 to 0.78 and the mean proposal score from 0.25 to 0.42, compared with 0.71 and 0.49 on the training states. Despite these differences, the switches of the policy improve tracking over staying on every set. Training the policy on the states of the final tracker instead lowers AUC from 48.23% to 44.40% after one collection round.

(1) Purpose of cross-fitting. Cross-fitting is designed to replicate the evaluation conditions. The cross-fitting trackers have the same architecture and 45-epoch training budget as the final tracker and differ only in the UAV-Anti-UAV training sequences they see, so their confidence and proposal scores are produced in the same way. The final tracker is evaluated on sequences it has never seen, and each cross-fitting tracker likewise collects states only on the half of the training split that it has not seen. As a result, the collected failures occur on unseen sequences, as at evaluation. The two halves alternate sequences by index, together covering the entire training split. Following DAgger, the later collection rounds switch with the current policy, so the training states also include the states that the policy reaches after its own switches.

(2) State distributions on four evaluation sets. At every decision point of the final tracker, we replay the stay and switch rollouts for 10 frames solely to evaluate the trained policy. The policy never sees the states of the three other benchmarks of Tab. 1 during training, and the Training row of the table below lists the states on which the policy is trained.

Decision statesDecision pointsMean tracker confidenceMean proposal scoreSwitchesSwitch rate (% of decision points)IoU gain per switch (%)
Training (cross-fitting trackers, held-out halves)32,9340.710.491,4714.5+17.7
Evaluation (UAV-Anti-UAV)94,5630.470.253,4893.7+12.2
Evaluation (Anti-UAV300)12,2890.660.425384.4+32.8
Evaluation (Anti-UAV410)25,2380.500.267583.0+21.6
Evaluation (DUT-Anti-UAV)2,2040.780.37170.8+35.5

The IoU gain per switch is the replayed advantage of the chosen proposal over staying, measured in 10-frame mean IoU. On every set, the policy switches at a small fraction of the decision points, and these switches raise the 10-frame mean IoU by 12.2% to 35.5% on average. These results show that the policy transfers across mean tracker confidences from 0.47 to 0.78 without retraining.

(3) Training on the states of the final tracker. We also train the policy on the states that the final tracker itself collects on the training split, with one collection round as in Tab. A2 of the revised paper. Both policies are evaluated with the final tracker on UAV-Anti-UAV, and only the states used for training differ.

States used to train the policyAUC (%)PP (%)PnormP_{\mathrm{norm}} (%)
Collected by the final tracker on the training split44.4059.0348.77
Collected by the cross-fitting trackers (ReTrack)48.2364.6353.20

The states of the final tracker yield 44.40% AUC, 3.83% below the cross-fitted states, because the cross-fitting trackers fail on unseen sequences, as the final tracker does at evaluation. Therefore, the cross-fitted states train a better policy than the states of the final tracker.


We hope these responses address the reviewer's concerns, and we are glad to discuss any remaining questions.

Reviewer_42rdOpenReview ↗

Response to the authors

Thank you for the response; the authors have addressed my concerns, so I am changing my score to 3.

AuthorsOpenReview ↗

Thank You to Reviewer 42rd

We thank the reviewer for reading our response and for raising the score. The questions on where the gain comes from and on the states collected by the cross-fitting trackers and the final tracker helped us make the analysis clearer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.