- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Ambiguous Token Refinement (ATR) for Transformer-based visual tracking. It extends the binary template token typing used by trackers such as LoRAT with an additional ambiguous type. A background prototype is formed by averaging template patch embeddings outside the target box, and box-overlapping tokens with high rescaled cosine similarity to this prototype are assigned the ambiguous type. ATR is applied to the initial template; the backbone, prediction head, and training objective are retained, and the augmented trackers are trained under the corresponding baseline protocols. Experiments cover LoRAT, DTPTrack, UTPTrack, and SUTrack across RGB and multimodal benchmarks, including LaSOT, LaSOText, GOT-10k, TrackingNet, VastTrack, UAV123, OTB100, VOT-RGBD22, LasHeR, VisEvent, and TNL2K.
Critical strengths
-
The motivation is practical and clearly stated: spatial overlap with a bounding box does not guarantee that a patch contains reliable target information. ATR makes a small change to template token representation while retaining the baseline backbone, prediction head, and training objective, which makes the idea straightforward to integrate and evaluate.
-
The paired evaluation spans single-template, multi-frame, and multimodal trackers. Table 7 additionally compares ATR with binary token embeddings on two trackers, providing a relevant control. The reported paired results are generally positive, with some metrics unchanged, and the breadth of this evaluation is a substantive strength.
Critical weaknesses
-
The claimed semantic interpretation and the mechanism responsible for the gains are not sufficiently established. Equation (7) compares patch embeddings with a single average background prototype; high similarity to that prototype does not establish that a token is semantically closer to the background than to the target, particularly when target and context share appearance. Table 7 establishes an advantage over the tested BTE variants, but does not isolate whether background-guided selection is responsible for the benefit or whether an additional type embedding and foreground partition could produce similar gains. A simple control that matches the number of retyped tokens, together with direct inspection of selected tokens, would make this distinction clearer. This missing attribution limits the methodological conclusion that the gains arise specifically from resolving semantic ambiguity.
-
The precision and reproducibility of the empirical evidence need clarification. The 4.0-point GOT-10k AO gain for LoRAT-B224 (72.1 to 76.1) is a positive reported result, while many other improvements are only a few tenths of a point and no multi-run uncertainty is reported. There are also two numerical discrepancies: LoRAT-B224 + ATR has LaSOText AUC 50.9 in Tables 1 and 2, versus 50.7 at τ = 0.6 in Table 8; DTPTrack-B224 + ATR has TrackingNet PNorm 90.2 in Table 1 versus 90.1 in Table 5. These differences could reflect distinct runs, settings, or reporting errors and should be reconciled. Section 4.1 and Appendix A refer to baseline training protocols but do not specify the new ambiguous-type embedding’s initialization and trainability or the threshold-selection procedure. These gaps make the smaller gains and the exact implementation difficult to assess. Together with the unresolved mechanism attribution, they motivate my weak-rejection recommendation.
Questions for authors
-
Please specify how the ambiguous-type embedding E_a is initialized and optimized, which parameters are updated, and whether each baseline/ATR pair uses matched pretrained weights, training data, and schedules. For SUTrack, from which modality-specific or fused embeddings are the background prototype and ambiguity scores computed?
-
Can the authors provide a simple third-type control with the same number of reassigned tokens, and inspect whether ATR preferentially selects background-containing patches rather than informative target parts? How is the prototype handled when non-overlapping background tokens are very sparse or absent, or when the surrounding background is heterogeneous?
-
How was τ = 0.6 selected, and which data split was used? Table 8 studies one RGB baseline; please clarify its applicability across the other backbones and modalities, considering differences in cosine-score distributions. Please also reconcile the LaSOText values of 50.9 in Tables 1 and 2 and 50.7 at τ = 0.6 in Table 8.
-
Please reconcile the TrackingNet PNorm values of 90.2 and 90.1 for DTPTrack-B224 + ATR in Tables 1 and 5. Were the paired results obtained from single runs or multiple runs, and what is the run-to-run variability for representative small-margin improvements and the GOT-10k result?
-
Please separate the one-time template-initialization cost from steady-state tracking cost. What are the additional prototype/similarity computation, initialization latency, and parameters, and is subsequent per-frame throughput unchanged under matched measurement conditions?
3 comments
1. is “learned from scratch,” but that only shows it is trainable. What initialization distribution is actually used? Which parameters are updated in each framework? Also, please explicitly confirm whether each baseline/ATR pair really uses matched pretrained weights. For SUTrack, are the background prototype and ambiguity scores computed from RGB, auxiliary-modality, modality-specific, or fused features? Does ATR replace SUTrack’s original soft geometric embedding or add to it? Right now the description and the piecewise formula later do not match, and this needs to be reconciled.
2.Can you add a third-type control that randomly reassigns exactly the same number of in-box tokens as ATR, using the same learnable embedding and training settings? Only then can the gain from “background-guided selection” be separated from the gain from “adding one more token type.” The visualizations help, but they do not show whether the selected patches preferentially contain background. Also, when there are no non-overlapping background tokens, what is the actual fallback?
Response-2 to Reviewer k491
We thank the reviewer for the follow-up questions.
Q1. Implementation Details and Fair Comparisons
(1) Embedding initialization and parameter optimization. Following LoRAT, the newly introduced ambiguous-type embedding adopts the same initialization scheme as the original foreground and background embeddings. Specifically, we initialize its components from PyTorch's default truncated normal distribution with mean 0, standard deviation 0.02, and truncation bounds [-2, 2].
For each baseline tracker (LoRAT, DTPTrack, UTPTrack, and SUTrack), all originally frozen parameters remain frozen, while all originally trainable parameters and the additional ATR embeddings are jointly optimized under the original tracking objective.
(2) Matched pretrained weights and training protocols. We explicitly confirm that each baseline/ATR pair uses identical pretrained weights for all shared components. Both models start training from epoch 0 with identical training data, sampling strategies, data augmentation, optimizers, training schedules, and evaluation protocols. Introducing ATR is the only methodological difference, ensuring controlled comparisons.
(3) Feature source in SUTrack. Background prototypes and ambiguity scores are computed from fused RGB–X visual template features. Specifically, RGB and auxiliary visual inputs are concatenated channel-wise and processed through the first two Fast-iTPN stages, before adding positional and token-type embeddings. Neither separate modality-specific prototypes nor language features are used for ambiguity estimation.
(4) Integration with SUTrack. We directly add ATR into SUTrack rather than replace SUTrack's original soft geometric embeddings. Therefore, ATR preserves SUTrack's geometric prior while introducing complementary semantic ambiguity information, without modifying its backbone, modality processing, or fusion architecture.
Q2. Mechanism Validation and Background Prototype Robustness
(1) Matched-count random assignment. Following the reviewer's suggestion, we conducted an additional controlled experiment. For each template, we randomly select exactly the same number of in-box tokens as ATR and assign them the ambiguous type, while preserving identical embedding configurations, training settings, and evaluation protocols.
| Method | LaSOT AUC | LaSOText AUC | TNL2K AUC |
|---|---|---|---|
| LoRAT-B224 | 71.70 | 50.30 | 58.80 |
| + Random assignment | 71.62 | 50.11 | 58.56 |
| + ATR | 72.70 | 50.70 | 59.20 |
| ATR vs. Random | +1.08 | +0.59 | +0.64 |
Random assignment provides no improvement over the original baseline, whereas ATR consistently improves all three benchmarks.
With identical token-type capacity and matched reassignment counts, ATR outperforms random assignment by 1.08, 0.59, and 0.64 AUC points, respectively. This controlled experiment directly supports the effectiveness of background-guided selection beyond merely introducing an additional learnable token type.
(2) Background content of selected patches. ATR identifies background-like semantics through feature similarity rather than pixel-level foreground/background membership. Fig. 4 provides qualitative evidence that ATR selects background-like regions within the target bounding box.
We acknowledge that quantitatively establishing whether selected patches preferentially contain background pixels requires segmentation annotations. Such annotations are neither assumed nor used by ATR.
Importantly, ATR's objective is to improve template representation rather than recover pixel-level object boundaries. The matched random-assignment experiment directly validates the tracking benefit of its selection mechanism.
(3) Empty-background condition. We have already explained this issue in our previous response(Q2). We use the standard template-cropping protocol that retains a region (e.g. four times) the target bounding-box area, providing surrounding context for prototype construction. Consequently, the empty-background case does not occur in those baselines and our experiments.
When background tokens are available but no in-box token exceeds the ambiguity threshold, ATR still uses Eqs. 5-7 for foreground assignments.
We hope these clarifications address the reviewer's remaining concern. We believe that our method has been sufficiently verified, and we welcome further discussion.
Response to Reviewer k491
Thank you for your thoughtful review and feedback. The main concerns are addressed as follows.
Q1: Implementation Details and Fair Comparisons
(1) Ambiguous-type embedding. Following LoRAT, ATR extends binary template token typing to three categories: 0 (background), 1 (ambiguous), and 2 (foreground). These integers serve as categorical mask indices that select corresponding learnable token-type embeddings, which are added to the visual token features. The embeddings are learned from scratch and optimized jointly with the original trainable tracking components under the unchanged training objective, without additional supervision or losses.
(2) Fair comparisons. As stated in Section 4.1, each baseline/ATR pair retains the original training data, training strategy, and evaluation protocol, with only ATR incorporated into the template representation. The ambiguity threshold is fixed at (\tau=0.6) across all experiments, without dataset-specific tuning.
(3) SUTrack integration. Please refer to our global response (Q3). ATR complements SUTrack's existing geometry-based soft token embeddings with additional ambiguity-aware embeddings, preserving its multimodal fusion strategy, backbone, and training objective.
Q2: Third-Type Control and Background Prototype Robustness
(1) Third-type control. ATR introduces principled background-guided token assignment rather than arbitrary foreground partitioning (Eqs. 5–7). Table 7 demonstrates consistent improvements over binary token representations on both LoRAT and UTPTrack, while the newly added Fig. 4 visualizes the resulting ambiguity scores and token assignments. We acknowledge that a matched three-type control would further isolate the contribution of background-guided assignment from the additional learnable embedding. Nevertheless, these quantitative and qualitative results provide complementary evidence of ATR's effectiveness and intended behavior.
(2) Sparse background tokens. ATR constructs a background prototype using non-overlapping template tokens (Eq. 6). Standard tracking templates typically contain surrounding context, providing background references. However, when no background tokens are available, the prototype cannot be computed directly, and this degenerate case requires explicit handling.
(3) Heterogeneous backgrounds. ATR employs a mean background prototype as a lightweight approximation of the surrounding context. Although heterogeneous backgrounds may reduce its representativeness, ATR preserves the original visual features and assigns uncertain tokens a distinct learnable embedding rather than directly treating them as background. This maintains a representational distinction between ambiguous and background tokens. The consistent improvements across benchmarks support its effectiveness under the evaluated conditions.
Q3: Ambiguity Threshold Selection and Generalization
(1) Threshold selection. The threshold (\tau=0.6) is an empirically selected hyperparameter, rather than a dataset-specific optimized value. No dedicated validation split was used for threshold selection. We consistently apply the same threshold across all trackers, datasets, and modalities, without framework- or dataset-specific adjustments.
(2) Cross-framework generalization. Although Table 8 examines threshold sensitivity on LoRAT-B224, Tables 1–6 demonstrate consistent improvements across multiple tracking frameworks and modalities using the same (\tau=0.6). These results provide empirical evidence that a unified threshold transfers effectively across the evaluated settings, without additional framework-specific optimization.
Q4: Numerical Consistency
We thank the reviewer for identifying these discrepancies. The correct LaSOText AUC for LoRAT-B224 + ATR is 50.7 (with threshold=0.6), and the correct TrackingNet normalized precision for DTPTrack-B224 + ATR is 90.2. The reported values of 50.9 in Tables 1–2 and 90.1 in Table 5 were typographical errors, which have been corrected in the revised manuscript.
Q5: Initialization Overhead and Tracking Efficiency
ATR introduces only 0.23 ms of additional one-time initialization latency for a 112× 112 template, without additional per-frame ATR computation. Please refer to our global response (Q4) for further details.