acceptodds
Under review as a conference paper at ICLR 2027

Ambiguous Token Refinement for Target Representation in Transformer Tracking

Abstract

Transformer-based visual trackers have recently achieved strong performance through effective template–search interaction. To mitigate background interference in template modeling, existing methods commonly introduce token-type embeddings to separate foreground and background tokens, treating all tokens inside the target bounding box as homogeneous foreground. However, such a coarse binary strategy relies solely on geometric overlap and ignores the semantic content within the box, causing background-like tokens inside the target region to confuse the template representation and degrade tracking performance. In this paper, we revisit template modeling in Transformer-based tracking and propose a general ambiguous token refinement method that refines token types inside the target bounding box. It is model-agnostic, and can be readily applied to various tracking tasks, including RGB, RGB+Thermal, RGB+Depth, and RGB+Event tracking. Specifically, unlike the previous coarse binary foreground/background template tokens strategy, our approach automatically assigns an additional ambiguous type to template tokens that are semantically similar to the surrounding background, thereby reducing confusion in template representations. Despite its simplicity, extensive experiments on visible and multi-modal benchmarks demonstrate consistent improvements over strong baselines and outperform many state-of-the-art trackers.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
visual object tracking, deep learning

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_k491OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Ambiguous Token Refinement (ATR) for Transformer-based visual tracking. It extends the binary template token typing used by trackers such as LoRAT with an additional ambiguous type. A background prototype is formed by averaging template patch embeddings outside the target box, and box-overlapping tokens with high rescaled cosine similarity to this prototype are assigned the ambiguous type. ATR is applied to the initial template; the backbone, prediction head, and training objective are retained, and the augmented trackers are trained under the corresponding baseline protocols. Experiments cover LoRAT, DTPTrack, UTPTrack, and SUTrack across RGB and multimodal benchmarks, including LaSOT, LaSOText, GOT-10k, TrackingNet, VastTrack, UAV123, OTB100, VOT-RGBD22, LasHeR, VisEvent, and TNL2K.

Critical strengths
  1. The motivation is practical and clearly stated: spatial overlap with a bounding box does not guarantee that a patch contains reliable target information. ATR makes a small change to template token representation while retaining the baseline backbone, prediction head, and training objective, which makes the idea straightforward to integrate and evaluate.

  2. The paired evaluation spans single-template, multi-frame, and multimodal trackers. Table 7 additionally compares ATR with binary token embeddings on two trackers, providing a relevant control. The reported paired results are generally positive, with some metrics unchanged, and the breadth of this evaluation is a substantive strength.

Critical weaknesses
  1. The claimed semantic interpretation and the mechanism responsible for the gains are not sufficiently established. Equation (7) compares patch embeddings with a single average background prototype; high similarity to that prototype does not establish that a token is semantically closer to the background than to the target, particularly when target and context share appearance. Table 7 establishes an advantage over the tested BTE variants, but does not isolate whether background-guided selection is responsible for the benefit or whether an additional type embedding and foreground partition could produce similar gains. A simple control that matches the number of retyped tokens, together with direct inspection of selected tokens, would make this distinction clearer. This missing attribution limits the methodological conclusion that the gains arise specifically from resolving semantic ambiguity.

  2. The precision and reproducibility of the empirical evidence need clarification. The 4.0-point GOT-10k AO gain for LoRAT-B224 (72.1 to 76.1) is a positive reported result, while many other improvements are only a few tenths of a point and no multi-run uncertainty is reported. There are also two numerical discrepancies: LoRAT-B224 + ATR has LaSOText AUC 50.9 in Tables 1 and 2, versus 50.7 at τ = 0.6 in Table 8; DTPTrack-B224 + ATR has TrackingNet PNorm 90.2 in Table 1 versus 90.1 in Table 5. These differences could reflect distinct runs, settings, or reporting errors and should be reconciled. Section 4.1 and Appendix A refer to baseline training protocols but do not specify the new ambiguous-type embedding’s initialization and trainability or the threshold-selection procedure. These gaps make the smaller gains and the exact implementation difficult to assess. Together with the unresolved mechanism attribution, they motivate my weak-rejection recommendation.

Questions for authors
  1. Please specify how the ambiguous-type embedding E_a is initialized and optimized, which parameters are updated, and whether each baseline/ATR pair uses matched pretrained weights, training data, and schedules. For SUTrack, from which modality-specific or fused embeddings are the background prototype and ambiguity scores computed?

  2. Can the authors provide a simple third-type control with the same number of reassigned tokens, and inspect whether ATR preferentially selects background-containing patches rather than informative target parts? How is the prototype handled when non-overlapping background tokens are very sparse or absent, or when the surrounding background is heterogeneous?

  3. How was τ = 0.6 selected, and which data split was used? Table 8 studies one RGB baseline; please clarify its applicability across the other backbones and modalities, considering differences in cosine-score distributions. Please also reconcile the LaSOText values of 50.9 in Tables 1 and 2 and 50.7 at τ = 0.6 in Table 8.

  4. Please reconcile the TrackingNet PNorm values of 90.2 and 90.1 for DTPTrack-B224 + ATR in Tables 1 and 5. Were the paired results obtained from single runs or multiple runs, and what is the run-to-run variability for representative small-margin improvements and the GOT-10k result?

  5. Please separate the one-time template-initialization cost from steady-state tracking cost. What are the additional prototype/similarity computation, initialization latency, and parameters, and is subsequent per-frame throughput unchanged under matched measurement conditions?

3 comments
AuthorsOpenReview ↗

Response to Reviewer k491

Thank you for your thoughtful review and feedback. The main concerns are addressed as follows.

Q1: Implementation Details and Fair Comparisons

(1) Ambiguous-type embedding. Following LoRAT, ATR extends binary template token typing to three categories: 0 (background), 1 (ambiguous), and 2 (foreground). These integers serve as categorical mask indices that select corresponding learnable token-type embeddings, which are added to the visual token features. The embeddings are learned from scratch and optimized jointly with the original trainable tracking components under the unchanged training objective, without additional supervision or losses.

(2) Fair comparisons. As stated in Section 4.1, each baseline/ATR pair retains the original training data, training strategy, and evaluation protocol, with only ATR incorporated into the template representation. The ambiguity threshold is fixed at (\tau=0.6) across all experiments, without dataset-specific tuning.

(3) SUTrack integration. Please refer to our global response (Q3). ATR complements SUTrack's existing geometry-based soft token embeddings with additional ambiguity-aware embeddings, preserving its multimodal fusion strategy, backbone, and training objective.

Q2: Third-Type Control and Background Prototype Robustness

(1) Third-type control. ATR introduces principled background-guided token assignment rather than arbitrary foreground partitioning (Eqs. 5–7). Table 7 demonstrates consistent improvements over binary token representations on both LoRAT and UTPTrack, while the newly added Fig. 4 visualizes the resulting ambiguity scores and token assignments. We acknowledge that a matched three-type control would further isolate the contribution of background-guided assignment from the additional learnable embedding. Nevertheless, these quantitative and qualitative results provide complementary evidence of ATR's effectiveness and intended behavior.

(2) Sparse background tokens. ATR constructs a background prototype using non-overlapping template tokens (Eq. 6). Standard tracking templates typically contain surrounding context, providing background references. However, when no background tokens are available, the prototype cannot be computed directly, and this degenerate case requires explicit handling.

(3) Heterogeneous backgrounds. ATR employs a mean background prototype as a lightweight approximation of the surrounding context. Although heterogeneous backgrounds may reduce its representativeness, ATR preserves the original visual features and assigns uncertain tokens a distinct learnable embedding rather than directly treating them as background. This maintains a representational distinction between ambiguous and background tokens. The consistent improvements across benchmarks support its effectiveness under the evaluated conditions.

Q3: Ambiguity Threshold Selection and Generalization

(1) Threshold selection. The threshold (\tau=0.6) is an empirically selected hyperparameter, rather than a dataset-specific optimized value. No dedicated validation split was used for threshold selection. We consistently apply the same threshold across all trackers, datasets, and modalities, without framework- or dataset-specific adjustments.

(2) Cross-framework generalization. Although Table 8 examines threshold sensitivity on LoRAT-B224, Tables 1–6 demonstrate consistent improvements across multiple tracking frameworks and modalities using the same (\tau=0.6). These results provide empirical evidence that a unified threshold transfers effectively across the evaluated settings, without additional framework-specific optimization.

Q4: Numerical Consistency

We thank the reviewer for identifying these discrepancies. The correct LaSOText AUC for LoRAT-B224 + ATR is 50.7 (with threshold=0.6), and the correct TrackingNet normalized precision for DTPTrack-B224 + ATR is 90.2. The reported values of 50.9 in Tables 1–2 and 90.1 in Table 5 were typographical errors, which have been corrected in the revised manuscript.

Q5: Initialization Overhead and Tracking Efficiency

ATR introduces only 0.23 ms of additional one-time initialization latency for a 112× 112 template, without additional per-frame ATR computation. Please refer to our global response (Q4) for further details.

1.EaE_a is “learned from scratch,” but that only shows it is trainable. What initialization distribution is actually used? Which parameters are updated in each framework? Also, please explicitly confirm whether each baseline/ATR pair really uses matched pretrained weights. For SUTrack, are the background prototype and ambiguity scores computed from RGB, auxiliary-modality, modality-specific, or fused features? Does ATR replace SUTrack’s original soft geometric embedding or add to it? Right now the description and the piecewise formula later do not match, and this needs to be reconciled.

2.Can you add a third-type control that randomly reassigns exactly the same number of in-box tokens as ATR, using the same learnable embedding and training settings? Only then can the gain from “background-guided selection” be separated from the gain from “adding one more token type.” The visualizations help, but they do not show whether the selected patches preferentially contain background. Also, when there are no non-overlapping background tokens, what is the actual fallback?

AuthorsOpenReview ↗

Response-2 to Reviewer k491

We thank the reviewer for the follow-up questions.

Q1. Implementation Details and Fair Comparisons

(1) Embedding initialization and parameter optimization. Following LoRAT, the newly introduced ambiguous-type embedding EaE_a adopts the same initialization scheme as the original foreground and background embeddings. Specifically, we initialize its components from PyTorch's default truncated normal distribution with mean 0, standard deviation 0.02, and truncation bounds [-2, 2].

For each baseline tracker (LoRAT, DTPTrack, UTPTrack, and SUTrack), all originally frozen parameters remain frozen, while all originally trainable parameters and the additional ATR embeddings are jointly optimized under the original tracking objective.

(2) Matched pretrained weights and training protocols. We explicitly confirm that each baseline/ATR pair uses identical pretrained weights for all shared components. Both models start training from epoch 0 with identical training data, sampling strategies, data augmentation, optimizers, training schedules, and evaluation protocols. Introducing ATR is the only methodological difference, ensuring controlled comparisons.

(3) Feature source in SUTrack. Background prototypes and ambiguity scores are computed from fused RGB–X visual template features. Specifically, RGB and auxiliary visual inputs are concatenated channel-wise and processed through the first two Fast-iTPN stages, before adding positional and token-type embeddings. Neither separate modality-specific prototypes nor language features are used for ambiguity estimation.

(4) Integration with SUTrack. We directly add ATR into SUTrack rather than replace SUTrack's original soft geometric embeddings. Therefore, ATR preserves SUTrack's geometric prior while introducing complementary semantic ambiguity information, without modifying its backbone, modality processing, or fusion architecture.

Q2. Mechanism Validation and Background Prototype Robustness

(1) Matched-count random assignment. Following the reviewer's suggestion, we conducted an additional controlled experiment. For each template, we randomly select exactly the same number of in-box tokens as ATR and assign them the ambiguous type, while preserving identical embedding configurations, training settings, and evaluation protocols.

MethodLaSOT AUCLaSOText AUCTNL2K AUC
LoRAT-B22471.7050.3058.80
+ Random assignment71.6250.1158.56
+ ATR72.7050.7059.20
ATR vs. Random+1.08+0.59+0.64

Random assignment provides no improvement over the original baseline, whereas ATR consistently improves all three benchmarks.

With identical token-type capacity and matched reassignment counts, ATR outperforms random assignment by 1.08, 0.59, and 0.64 AUC points, respectively. This controlled experiment directly supports the effectiveness of background-guided selection beyond merely introducing an additional learnable token type.

(2) Background content of selected patches. ATR identifies background-like semantics through feature similarity rather than pixel-level foreground/background membership. Fig. 4 provides qualitative evidence that ATR selects background-like regions within the target bounding box.

We acknowledge that quantitatively establishing whether selected patches preferentially contain background pixels requires segmentation annotations. Such annotations are neither assumed nor used by ATR.

Importantly, ATR's objective is to improve template representation rather than recover pixel-level object boundaries. The matched random-assignment experiment directly validates the tracking benefit of its selection mechanism.

(3) Empty-background condition. We have already explained this issue in our previous response(Q2). We use the standard template-cropping protocol that retains a region (e.g. four times) the target bounding-box area, providing surrounding context for prototype construction. Consequently, the empty-background case does not occur in those baselines and our experiments.

When background tokens are available but no in-box token exceeds the ambiguity threshold, ATR still uses Eqs. 5-7 for foreground assignments.

We hope these clarifications address the reviewer's remaining concern. We believe that our method has been sufficiently verified, and we welcome further discussion.

Reviewer_96w8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ATR refines template token representations by assigning an additional ambiguous type to box-overlapping patches that resemble the surrounding background. Experiments report improvements across single-template, multi-frame, and multimodal tracking frameworks.

Critical strengths
  1. Simple and well-motivated: ATR addresses the mismatch between bounding-box overlap and semantic foregroundness through a localized change to token-type embeddings.

  2. Broad evaluation: Results across four tracking frameworks, together with binary-versus-ATR ablations, support the usefulness of the overall strategy.

Critical weaknesses
  1. Limited mechanism evidence: Tracking improvements do not directly establish that ATR identifies semantically ambiguous tokens. The ablations also change both the assignment rule and the number of token types, leaving their contributions unresolved.

  2. Some implementation details remain unclear, especially how ATR interacts with SUTrack’s soft token embeddings. Repeated-run statistics are also missing, and numerical discrepancies between Tables 1–2 and 8, and between Tables 1 and 5, complicate the interpretation of small gains.

  3. Unverified efficiency claim: Processing only the initial template avoids recurring computation, but does not establish zero overhead. Initialization latency and runtime measurements are not reported.

Questions for authors

See the weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 96w8

Thank you for your thoughtful review and feedback. The main concerns are addressed as follows.

W1: Mechanism Validation of ATR

ATR estimates potentially ambiguous tokens without pixel-level annotations or additional semantic supervision. Specifically, it exploits explicit background tokens as semantic references (Eq. 6) to refine foreground assignments through background similarity (Eqs. 5–7).

Table 7 demonstrates consistent improvements over binary token representations on LoRAT and UTPTrack. Furthermore, the newly added Fig. 4 visualizes ambiguity scores and token-type assignments, providing direct qualitative evidence that ATR identifies background-like regions within the target bounding box.

We acknowledge that a matched three-type control would further isolate the contribution of background-guided selection from the additional embedding. Nevertheless, the existing ablations and qualitative visualizations provide complementary evidence supporting ATR's effectiveness and proposed mechanism.

W2: Implementation Details and Reproducibility

(1) SUTrack integration. Please refer to our global response (Q3), which clarifies how ATR complements SUTrack's geometry-based soft token embeddings.

(2) Multi-run results. Please refer to our global response (Q2) for the discussion of single-run evaluations and statistical limitations.

(3) Numerical consistency. We thank the reviewer for identifying the discrepancies. The correct LaSOText AUC for LoRAT-B224 + ATR is 50.7, and the reported 50.9 in Tables 1–2 was a typographical error. The TrackingNet normalized precision discrepancy has also been corrected (90.1 → 90.2 in Table 5). These corrections do not affect our conclusions.

We will release our code to facilitate reproducibility.

W3: Initialization Latency and Tracking Efficiency

Please refer to our global response (Q4), which reports only 0.23 ms of additional one-time initialization latency for a 112 × 112 template, with no additional per-frame ATR computation.

Reviewer_CB97OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies a limitation in current Transformer-based visual trackers that use binary foreground/background token typing based solely on bounding-box overlap (e.g., LoRAT). Because a bounding box is a coarse geometric prior, some tokens that overlap the box can be semantically closer to background than to the target. The authors propose Ambiguous Token Refinement (ATR), which constructs a background prototype from explicit non-overlapping tokens and measures cosine similarity of each box-overlapping token against this prototype. Tokens whose similarity exceeds a threshold τ are re-labeled as “ambiguous.” ATR is applied once to the initial template, requires no architectural changes or extra training loss, and incurs no inference overhead. Experiments on single-template (LoRAT), multi-frame (DTPTrack, UTPTrack), and multi-modal (SUTrack) trackers show consistent improvements across RGB, RGB-T, RGB-D, RGB-Event, and RGB-Language settings.

Critical strengths

1.Gains are shown not only on standard RGB single-template tracking but also on multi-frame and multi-modal trackers (RGB-T, RGB-D, RGB-Event, RGB-Language), demonstrating generality. 2.The paper identifies a real limitation of geometry-based binary token typing: box-overlapping tokens are not necessarily semantic foreground. The proposed ambiguous type is intuitive and well illustrated in Fig. 1. 3.Applying ATR once to the initial template is computationally efficient, and the paper explicitly emphasizes this practical advantage. 4.ATR does not modify the backbone, prediction head, or tracking objective, and it can be integrated into different tracking architectures and modalities. This is attractive for practical deployment.

Critical weaknesses

1.Limited technical novelty. ATR is essentially a simple post-hoc re-labeling of existing binary token types using cosine similarity to a mean background prototype. The core idea is close to classic “reliable patch” selection in early trackers and does not introduce new modeling capacity. 2.The improvement margins are modest (typically 0.5–1.5 AUC points). While consistent, they are within the range often obtained by careful hyper-parameter tuning or stronger data augmentation, raising questions about practical significance. 3.The paper lacks deeper analysis: it neither visualizes which tokens are assigned the ambiguous type nor investigates how the ambiguous embedding interacts with attention. 4.The method still relies on the initial bounding box being reasonably accurate. When the first-frame box is noisy or the target is heavily occluded/deformed, the background prototype itself may be contaminated, yet this failure mode is not analyzed. 5.Fixed hard threshold τ=0.6. The threshold is fixed for all models, datasets, and modalities, but the sensitivity study is limited and Table 7 is difficult to parse. If the table indeed shows τ=0.5 outperforming 0.6 on LaSOT, the choice of 0.6 needs justification. A learnable or soft assignment would also be worth discussing.

Questions for authors

1.Can you provide qualitative visualizations showing which tokens inside the box are labeled ambiguous, and whether they indeed correspond to background-like regions? 2.In multi-frame trackers you only refine the first template. Have you experimented with periodically re-applying ATR on updated templates, and does that bring further gains or instability? 3.How sensitive is ATR to the quality of the initial bounding box? What happens when the first-frame annotation is noisy or the target occupies only a small fraction of the box? 4.Can you provide qualitative visualizations of token type assignments and attention maps, and discuss failure cases where the target is semantically similar to the background? 5.Given the small improvements on several benchmarks, what evidence shows that the gains exceed run-to-run variance?

3 comments
AuthorsOpenReview ↗

Response to Reviewer CB97

Thank you for your thoughtful review and feedback. The main concerns are addressed as follows.

W1: Limited Technical Novelty

Please refer to our global response (Q1) for a detailed discussion of ATR's novelty and methodological contributions.

W2: Performance Gains and Hyperparameter Tuning

We respectfully clarify that ATR's improvements are not attributable to stronger data augmentation or changes to baseline training strategies. As stated in Section 4.1 (lines 255–258), we preserve each baseline's original architecture, training strategy, and evaluation protocol, incorporating only ATR. Moreover, we fix (τ\tau=0.6) across all trackers, datasets, and modalities, without dataset-specific tuning.

Importantly, ATR consistently improves four tracking frameworks across multiple benchmarks, including +4.0 AO points on GOT-10k and +1.0 AUC point on LaSOT for LoRAT-B224 (Table 1). These gains are achieved with only 0.23 ms of one-time initialization overhead for a 112× 112 template and no additional per-frame ATR computation, demonstrating its practical value.

Q1 and Q4: Qualitative Visualizations and Failure Cases

We thank the reviewer for this suggestion. We have added Section 4.5 in the revised manuscript. The newly added Figure 4 visualizes ambiguity scores and token-type assignments, illustrating how ATR identifies background-like regions inside the target box while preserving other foreground tokens.

In the failure case (bottom row), the target shares similar visual features with the surrounding background, resulting in high background similarity across the target region. Consequently, some informative target tokens are also assigned the ambiguous type, exposing a limitation of background-similarity-based refinement when foreground and background features are difficult to distinguish. Importantly, ATR assigns these uncertain tokens a distinct learnable embedding rather than treating them as background. This preserves their original visual features and maintains a representational distinction from explicit background tokens, potentially mitigating the adverse effects of inaccurate ambiguity assignments.

Q2: ATR on Updated Templates

ATR deliberately refines only the initial template, whose target bounding box is provided by ground-truth annotation. Our goal is to complement existing temporal modeling rather than modify template-update strategies. Table 5 demonstrates consistent improvements on DTPTrack and UTPTrack while preserving their original update mechanisms. In contrast, updated templates depend on predicted bounding boxes, whose errors could lead to unreliable ambiguity assignments and error propagation. We have not experimentally evaluated periodic re-application, and will investigate its potential benefits and limitations in future work.

Q3: Sensitivity to Initial Bounding-Box Quality

ATR is designed to address the inherent ambiguity of bounding-box-based foreground representation. Even accurate bounding boxes may contain substantial background due to irregular object shapes and poses. ATR refines such regions using background similarity rather than treating all box-overlapping tokens equally.

When the target occupies only a small fraction of the box, ATR can identify background-like tokens and assign them the ambiguous type, potentially reducing foreground contamination. However, severely inaccurate annotations may affect both the background prototype and token assignments. The consistent improvements across benchmarks (Tables 1–6) support effectiveness under standard initialization, while robustness to noisy annotations remains to be investigated.

Q5: Small Performance Gains and Run-to-Run Variance

Please refer to our global response (Q2) for our discussion of single-run evaluation, cross-framework consistency, and statistical limitations.

Thank you for the detailed explanation provided by the author. However, there are still some issues that need to be addressed. Although the authors point out that “LoRAT-B224 + ATR improves GOT-10k by 4.0 AO and LaSOT by 1.0 AUC,” the gains on other trackers and benchmarks remain marginal. In addition, regarding the description of contributions, the paper proposes only a single ATR module; the authors’ core method is in fact just a lightweight module that performs post-hoc token-type relabeling based on a background prototype, cosine similarity, and a fixed threshold, without introducing a new tracking architecture, training objective, or attention mechanism. Its contribution is therefore debatable.

The authors’ rebuttal does not resolve my concerns. For example, although the authors explicitly state that “a unified threshold τ=0.6 is used across all experiments,” they still do not provide a meaningful sensitivity analysis or justification for this choice. The paper does not clearly explain why τ=0.6 is preferred over τ=0.5, nor does it improve the readability of Table 7 or provide trend analysis, and it does not discuss learnable or soft token-type assignment. Therefore, my concerns about the limited contribution and insufficient empirical evidence remain.

AuthorsOpenReview ↗

Response-2 to CB97

We thank the reviewer for the follow-up comments. We respectfully clarify the following four points.

(1) Performance gains and practical significance.

We respectfully disagree that ATR's improvements are marginal. Even recent state-of-the-art trackers achieve modest improvements over strong baselines. For example, Table 5 shows that LoRATv2-L378 outperforms LoRAT-L378 by +1.0 AUC on LaSOT, +0.7 AO on GOT-10k, and +0.1 AUC on TrackingNet.

In comparison, ATR achieves substantial or comparable gains across four distinct tracking frameworks:

BaselineBenchmarkOriginal+ ATRGain
LoRAT-B224GOT-10k72.176.1+4.0
LoRAT-B224LaSOT71.772.7+1.0
UTPTrack-B384LaSOT70.771.9+1.2
UTPTrack-B384LaSOText49.451.9+2.5
UTPTrack-B384TrackingNet84.084.3+0.3
DTPTrack-L378VastTrack46.647.1+0.5
SUTrack-B224LasHeR59.960.5+0.6
SUTrack-B224VisEvent62.763.3+0.6
SUTrack-B224TNL2K65.065.8+0.8

ATR further improves these competitive trackers without stronger data augmentation, framework-specific tuning, or additional per-frame computation. Its contribution should therefore be assessed jointly in terms of accuracy, generalizability, and computational efficiency rather than individual improvement margins on some datasets alone.

(2) Methodological novelty.

We respectfully disagree that ATR is merely a post-hoc relabeling module. ATR challenges the conventional assumption that geometric foreground membership implies semantic foreground reliability, a limitation shared by LoRAT and SUTrack.

ATR introduces an explicit ambiguous token type through background-guided semantic estimation without pixel-level supervision. Its corresponding embeddings are jointly learned during training.Cosine similarity is the implementation tool, not the methodological contribution itself.

The absence of a new architecture, attention mechanism, or training objective is an intentional design choice enabling broad applicability. Methodological novelty should not be equated with architectural complexity.

(3) Threshold selection, sensitivity analysis, and table readability.

We adopt τ = 0.6 as a unified empirical setting rather than claiming it is universally optimal. Because τ = 0.6 outperforms τ = 0.5 on all three benchmarks, achieving +0.6, +0.1, and +0.2 AUC points, respectively.

Table 8 systematically evaluates seven threshold settings across three benchmarks.

τLaSOTLaSOTextTNL2K
0.371.850.358.8
0.471.950.358.8
0.572.150.659.0
0.672.750.759.2
0.772.450.958.9
0.872.050.358.9
1.0 (Binary)71.750.358.8

τ = 0.6 outperforms τ = 0.5 on all three benchmarks. The observed trend reflects the trade-off between excessive reassignment at lower thresholds and insufficient refinement at higher thresholds.

For readability, we reorganize Table 7 as follows:

RepresentationLoRAT-L224 LaSOTLaSOTextUTPTrack-B384 LaSOTLaSOText
Without BTE74.052.470.749.4
Binary embedding74.252.871.050.0
ATR74.853.571.951.9

ATR consistently outperforms binary embeddings by 0.6–1.9 AUC points, supporting the effectiveness of ambiguity-aware representation.

(4) Learnable and soft token-type assignment.

SUTrack employs soft token-type embeddings. Given a patch �0��0� and target bounding box �1��1�, its representation is:

$ r_i=\frac{|P_i\cap B|}{|P_i|}, \qquad E_i^{\mathrm{SU}}=r_iE_f+(1-r_i)E_b. $

For patches entirely inside the bounding box, �2��2�, regardless of their semantic content. Consequently, SUTrack's soft assignment addresses geometric boundary mixtures rather than semantic ambiguity within the target region.

In contrast, ATR introduces an explicit ambiguity-aware representation:

\begin{cases} E_b, & r_i = 0, \\\\ E_a, & r_i > 0,\\ a_i > \tau, \\\\ E_f, & r_i > 0,\\ a_i \leq \tau. \end{cases}$$ **Softness of assignment and semantic awareness are distinct properties.** ATR introduces learnable ambiguity-aware embeddings that complement SUTrack's geometric representation, as validated in Table 6. **Overall, ATR advances template representation beyond conventional hard and soft geometric assignments.** Its consistent improvements across four tracking frameworks and negligible overhead support its novelty and practical significance. We hope these clarifications address the reviewer's remaining concerns and welcome further discussion.
Reviewer_4PFCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Ambiguous Token Refinement (ATR) to improve template representations in Transformer-based visual tracking. Specifically, ATR refines the geometry-derived foreground prior by introducing an additional ambiguous token type for tokens that overlap with the target box but are semantically closer to the surrounding background than the target object. ATR is model-agnostic and can be seamlessly integrated into various tracking frameworks, including RGB, RGB-Thermal, RGB-Depth, RGB-Event, and RGB-Language tracking.

Critical strengths

ATR offers a simple, lightweight design with the consistent performance gains across several tracking frameworks.

Critical weaknesses

The novelty is limited, as ATR mainly extends LoRAT by using cosine similarity to identify background-like template tokens and assigning them an additional ambiguous-type embedding.

Questions for authors

1.This manuscript has limited novelty. ATR builds on LoRAT by adding a module that uses cosine similarity to identify background-like tokens within the template foreground, thereby refining the target representation.

2.How does ATR affect tracking when non-target regions within the template foreground differ in appearance from the surrounding template background? The authors provide neither theoretical analysis nor experimental investigation in this scenario.

3.Although ATR improves the performance of several existing models, the paper lacks a comprehensive comparison with state-of-the-art tracking methods.

4.The authors do not adequately explain why ATR achieves different performance gains across datasets or why the optimal ambiguity threshold varies between datasets.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 4PFC

Thank you for your thoughtful review and feedback. The main concerns are addressed as follows.

Q1: Limited Novelty

Please refer to our global response (Q1) for a detailed discussion of ATR's conceptual novelty and methodological contributions.

Q2: Non-target Regions Within the Template Foreground

(1) ATR does not assume that all in-box non-target regions resemble the surrounding background. Instead, ATR performs selective refinement based on background similarity (Eqs. 5–7). Tokens with high background similarity are assigned the ambiguous type, while those with low similarity retain their original foreground type, preserving the geometric prior.

(2) We acknowledge that non-target regions with substantially different appearances may remain unidentified. However, ATR aims to reduce unreliable foreground assignments rather than recover precise pixel-level object boundaries without additional supervision. Tables 1–7 demonstrate the overall effectiveness of this refinement, although the specific scenario has not been independently evaluated.

Q3: Comparison with State-of-the-Art Methods

We respectfully disagree that the comparisons are insufficient. Tables 2–6 provide extensive comparisons with recent state-of-the-art trackers across three paradigms:

(1) Single-template: LMTrack and MIMTrack (AAAI 2025).

(2) Multi-frame: LoRATv2 (NeurIPS 2025), DTPTrack and UTPTrack (CVPR 2026).

(3) Multimodal: SUTrack (AAAI 2025) and UETrack (CVPR 2026).

Importantly, ATR is a generalizable refinement strategy rather than a standalone tracker. Thus, paired comparisons across four distinct tracking frameworks (Table 1) provide particularly direct evidence of its effectiveness under controlled architectural settings.

Q4: Performance Variations Across Datasets

(1) Performance variations may arise from differences in foreground–background ambiguity across datasets. Different object shapes, poses, and background complexities affect foreground–background similarity distributions, potentially changing the proportion and reliability of tokens identified as ambiguous.

(2) As formulated in Eqs. (5–7), the threshold (\tau) balances identifying background-like tokens and preserving informative foreground tokens. Table 8 shows that (τ\tau=0.6) performs best on LaSOT and TNL2K, while (τ\tau=0.7) performs slightly better on LaSOText. Importantly, we adopt a unified (τ\tau=0.6) across all trackers, datasets, and modalities, without dataset-specific tuning. The consistent improvements in Tables 1–6 support ATR's effectiveness and generalizability under this unified setting.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response to the Area Chair and All Reviewers

We sincerely thank the Area Chair and reviewers for their constructive feedback. Below, we address the major concerns regarding ATR's novelty, effectiveness, and generalizability. Revisions are highlighted in red in the updated manuscript.

Q1: Novelty of ATR (4PFC, CB97, 96w8)

We respectfully disagree that ATR has limited novelty. Our contribution lies not in cosine similarity itself, but in addressing a fundamental limitation of geometry-based foreground representation without pixel-level supervision.

(1) Conceptual novelty. Existing methods, including LoRAT and SUTrack, rely on bounding-box geometry for foreground token-type assignment, overlooking semantic ambiguity within the target region. ATR introduces an explicit ambiguous token type and leverages background similarity to identify potentially unreliable foreground tokens, enabling ambiguity-aware template representation.

(2) Methodological distinction. LoRAT uses hard binary assignment, while SUTrack employs soft embeddings weighted by patch-wise box overlap, primarily addressing boundary mixtures. Both remain geometry-driven without explicitly identifying background-like tokens inside the target region. ATR instead leverages explicit background tokens as references to refine in-box foreground assignments (Fig. 1; Eqs. 5–7).

(3) Extensive validation. Tables 1–6 demonstrate consistent improvements across four tracking frameworks, multiple benchmarks, and modalities, with extensive comparisons against recent state-of-the-art trackers. Table 7 validates ATR against binary representations, while Table 8 examines threshold sensitivity.

Importantly, ATR is a generalizable refinement strategy rather than a standalone tracker. Controlled comparisons (Table 1) show improvements of +1.0 AUC and +1.4 normalized precision points on LaSOT, and +4.0 AO points on GOT-10k for LoRAT-B224. ATR also improves SUTrack-B224 by +0.6 AUC points on LasHeR and VisEvent, and +0.8 points on TNL2K, without modifying baseline training strategies or dataset-specific threshold tuning.

Thus, ATR goes beyond an additional similarity module by extending geometry-based foreground modeling to ambiguity-aware template representation across diverse tracking architectures and modalities.

Q2: Multi-run Results and Reproducibility (CB97, 96w8, k491)

We acknowledge the value of multi-run evaluations. Given their substantial computational cost, we follow each baseline's evaluation protocol and report a single training run per configuration in our experiments.

ATR consistently improves four frameworks across multiple benchmarks and modalities (Tables 1–6), supported by ablation and threshold analyses (Tables 7–8). Baseline/ATR pairs preserve identical training data, augmentation, optimization, and evaluation settings (Section 4.1), with a unified threshold (τ\tau=0.6) across all experiments.

Although multi-run statistics would strengthen reproducibility, these results support ATR's practical effectiveness. We will release our code.

We corrected two typographical errors: LaSOText AUC 50.9 → 50.7 (Tables 1–2) and TrackingNet normalized precision 90.1 → 90.2 (Table 5). These changes do not affect our main conclusions.

Q3: SUTrack Integration (96w8, k491)

ATR applies the same background-guided ambiguity estimation across frameworks (Eqs. 5–7), introducing three token categories—background (0), ambiguous (1), and foreground (2)—with learnable embeddings.

SUTrack's original soft embeddings are learnable, but their mixing weights depend on bounding-box overlap rather than semantic ambiguity. ATR complements SUTrack with additional learnable, background-guided token-type embeddings, preserving its geometric prior, multimodal fusion, backbone, and training objective.

The improvements in Table 6 demonstrate ATR's compatibility with both hard and soft geometry-based token representations.

Q4: Initialization Latency and Tracking Efficiency (96w8, k491)

The newly added Section 4.6 reports only 0.23 ms of additional one-time initialization latency for a 112 × 112 template. The background prototype and similarity scores are computed once, with no additional per-frame ATR computation.

We believe these clarifications and revisions strengthen the evidence for ATR's novelty and effectiveness. We respectfully invite the reviewers to reconsider their initial assessments and welcome further discussion.

AuthorsOpenReview ↗

Global response (2): Mechanism Validation with Random Assignments.

Following the reviewer's suggestion, we conducted an additional controlled experiment. For each template, we randomly select exactly the same number of in-box tokens as ATR and assign them the ambiguous type, while preserving identical embedding configurations, training settings, and evaluation protocols.

MethodLaSOT AUCLaSOText AUCTNL2K AUC
LoRAT-B22471.7050.3058.80
+ Random assignment71.6250.1158.56
+ ATR72.7050.7059.20
ATR vs. Random+1.08+0.59+0.64

Random assignment provides no improvement over the original baseline, whereas ATR consistently improves all three benchmarks.

With identical token-type capacity and matched reassignment counts, ATR outperforms random assignment by 1.08, 0.59, and 0.64 AUC points, respectively. This controlled experiment directly supports the effectiveness of background-guided selection beyond merely introducing an additional learnable token type.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.