acceptodds
Withdrawn from ICLR 2027

A Stable Self-evolving Target Template Helps Test-Time Adaptation in Multimodal Object Tracking

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

RGB-X multimodal trackers have recently achieved remarkable progress by exploiting complementary information from heterogeneous modalities. However, existing methods often overlook performance degradation caused by distribution shifts between training and testing environments. Although recent test-time adaptation approaches have been explored for multimodal tracking, they largely rely on unreliable test-time confidence estimates or blind source-domain alignment, resulting in inaccurate adaptation objectives under shifted environments. To address these challenges, we propose STTA, a stable self-evolving target template-guided test-time adaptation framework. STTA constructs a target-specific template prior and performs stability-aware prior evolution to provide reliable and adaptive test-time optimization objectives. Through a decoupled hierarchical adaptation space with shift-aware space selection, STTA further enables selective updates across different modalities while preserving valuable pre-trained knowledge. Extensive experiments demonstrate that STTA consistently achieves superior performance and significantly improves model robustness and generalization ability.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Object Tracking, Test-Time Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qsXBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes S²TTA, a test-time adaptation method for multimodal object tracking. The method introduces modality-specific prompts into pretrained trackers. It uses the annotation of the first frame to optimize the prompts and construct a multi-level feature statistics prior. During subsequent tracking, the discrepancy between the current frame and the template prior is used to select which prompts to update. Meanwhile, prediction uncertainty is used to determine whether the template prior should be updated.

Critical strengths
  1. This paper uses the first-frame annotation available in object tracking to construct a target-specific template prior and uses it as the optimization objective for test-time adaptation. Compared with methods that directly use prediction confidence or source-domain statistics as adaptation objectives, this idea offers a different approach.
  2. The experimental evaluation is relatively comprehensive. The authors evaluate the method on tracking tasks with different modalities and use two trackers with different architectures as baselines. The paper also includes comparisons with other TTA methods and ablation studies of different components.
Critical weaknesses
  1. The overall network architecture and some important implementation details are not clearly explained. For example, the relationship between the template and search regions in Figure 2, as well as the prompt optimization process in the first frame, lacks sufficient explanation. This makes the proposed method difficult to understand.
  2. Although the paper includes several comparative experiments and ablation studies, it lacks qualitative visualizations of the tracking results. For example, there are no visual comparisons of tracking results before and after applying S²TTA under challenging conditions such as illumination changes, occlusion, and target appearance variations.
Questions for authors
  1. In Figure 2, some modules and labels overlap, such as the Encoder and Layer labels. I suggest improving the layout of the framework diagram.
  2. In the left part of Figure 2, only the template images are provided as input, but the prediction head is still used to predict bounding boxes and compute the tracking loss. How are the target bounding boxes predicted without a search region? In addition, the right part of Figure 2 only shows the search regions, without the original template features. Are the original templates still used for target matching during subsequent tracking? Is this due to a simplification of the framework diagram? I suggest clarifying the complete tracking process.
  3. Appendix A.3 mentions that geometric transformations and color perturbations are used to construct the augmented template pool, but the specific augmentation operations and parameter settings are not provided. I suggest adding these details.
  4. The proposed method performs test-time adaptation by aligning the feature statistics of the current frame with the evolving template prior. However, feature discrepancies may arise not only from environmental changes, such as illumination and weather variations, but also from target deformation or viewpoint changes. In the latter case, why would reducing the feature distribution discrepancy between the current frame and the template prior improve tracking performance? I suggest providing further explanation or experimental analysis.
  5. The paper lacks some intuitive visualization results. I suggest adding visual comparisons of tracking results before and after applying S²TTA under different scenarios. In addition, could the authors provide visualizations of feature responses or feature distributions before and after prompt updates?
Reviewer_ara3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

S²TTA addresses distribution shifts in multimodal object tracking through test-time adaptation guided by a target-specific template prior. The prior evolves when predictions are sufficiently reliable, while discrepancies between current observations and the prior determine which parts of the prompt representation to adapt. Experiments across thermal, depth, and event-based tracking benchmarks report improvements on different backbone trackers, with cross-modality evaluations and component analyses examining adaptation robustness, stability, and computational cost.

Critical strengths
  • The framework improves both backbone trackers across all five benchmarks (Table 1). For reproduced CSTrack, RGBT234 MSR improves from 70.2 to 72.8, and DepthTrack F-score from 65.7 to 66.5; Table 2 additionally demonstrates recovery under cross-modality shifts.
  • Table 3 supports combining template priors with selective adaptation. Appendix A adds meaningful weighting, augmentation, and runtime analyses rather than relying solely on aggregate accuracy.
Critical weaknesses
  • The contribution of online adaptation is not cleanly separated from supervised first-frame prompt fitting (Section 3.3, Eq. 4). Table 3 and Appendix A.3 do not isolate this control; Figure 3 includes 'No Prompt Adaptation' but does not clarify whether it retains the same fitted prompts. Thus, how much improvement specifically requires the proposed online evolution remains uncertain.
  • Central operations remain underspecified: Sections 3.3–4.1 do not state the optimizer, learning rate, or initialization/update step counts, and Appendix A.1 names Fisher weighting without defining its computation here. Figure 3 also omits the dataset, performance metric/normalization, and meaning of shaded bands, limiting verification of the central long-term stability claim.
Questions for authors
  1. Does Figure 3's "No Prompt Adaptation" retain the Eq. 4 fitted prompts? Please report this control on the main benchmarks, keeping first-frame supervision and augmentation identical.
  2. Clarification: please specify the optimization schedule, Fisher-weight computation, and Figure 3 evaluation protocol. Does Table 5 timing include first-frame fitting? Its 26.3-to-11.7 average FPS reduction makes this relevant to the practical tradeoff.
Reviewer_bxGdOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes S2TTA, a target-template-guided test-time adaptation (TTA) framework for RGB-X multimodal object tracking (RGB-T, RGB-D, RGB-E). The core idea is to exploit the annotated template frame — a structural asset unique to tracking that is available at test time and specific to each sequence — and turn it into a sequence-specific, target-specific optimization objective.

Critical strengths

The three claims form a complete argument chain: the template prior answers "what to optimize toward," the hierarchical space answers "in which parameter space," and the entropy-gated evolution answers "when to trust the observation." The paper's ablations (Table 3) cleanly attribute the gains to the template prior and to the adaptation space separately, and Fig. 3 provides mechanism-level analysis of both gating components.

Three auxiliary modalities × five benchmarks × single-/dual-stream architectures × cross-domain settings is a solid experimental footprint for a TTA method

Critical weaknesses

1.Eq. (4) gives the first-frame objective but no λ values; §3.4 gives the alignment objective but no optimizer, learning rate, per-frame gradient steps, or the exact statistics definition (which tokens/spatial dimensions µ/σ are computed over). Besides, I have check the correspondece between the code and equation, I found that some impotant details are missing, such as , Eq. 6 an Eq. 7. I suggest that some trained models should be uploated.

  1. The tech. are the combination of several off-the-shelf methods, such as, entropy, threshold in Eq. 6. how to ahcieve the self-evolving ? if self-evolving is thresold based method, I suggest the title should be changed.

  2. Current template is common approach for TTA.

Questions for authors

What optimizer, learning rate, per-frame gradient steps, and loss weights (λ_iou, λ_L1) are used in the first-frame initialization and the online phase respectively?

How sensitive are results to θ_d, θ_h, and m?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.