acceptodds
Under review as a conference paper at ICLR 2027

CSPTrack: Causal State Prediction with Ridge-Aware Token Pruning for Visual Tracking

Abstract

Current template–search trackers often fail to sufficiently model the temporal evolution of target states, lack effective constraints for maintaining consistent target representations across frames, and densely process a large number of redundant search tokens, thereby limiting both tracking robustness and computational efficiency. To address these issues, we propose CSPTrack, a robust and efficient visual tracking framework with three complementary designs. First, the causal state prediction module leverages current search features and historical hidden states to explicitly predict short-term future-state representations, thereby modeling the temporal evolution of target states. Second, the instance-consistency objective brings the template and search representations of the same target closer while separating the representations of different target instances, enhancing both cross-frame representation consistency and target discrimination. Third, the ridge-aware token pruning strategy evaluates the importance of search tokens based on ridge-reconstruction errors, selectively retains informative search tokens, aggregates information from discarded tokens, and preserves all template tokens. Extensive experiments on six challenging benchmark datasets demonstrate the effectiveness and efficiency of CSPTrack. Notably, without token pruning, CSPTrack-B224 achieves a new state-of-the-art performance of 75.4% AUC on LaSOT under similar settings. Even after reducing the number of search tokens by 30%, CSPTrack-B224 incurs only a marginal 0.2% AUC drop on LaSOT while improving tracking speed by 10%. These results demonstrate that CSPTrack effectively reduces redundant computation while maintaining high tracking accuracy.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
visual object tracking, single object tracking, temporal modeling, causal state prediction, token pruning, efficient tracking

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MedcOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CSPTrack, a robust and efficient visual tracking framework that addresses three key limitations of existing template-search trackers: insufficient temporal state modeling, identity ambiguity under temporal fusion, and redundant token computation. The framework introduces three complementary designs: (1) a Causal State Prediction Module (CSPM) that leverages current search features and historical hidden states to explicitly predict short-term future states; (2) an instance-consistency objective applied during training to align target representations across template and search frames while separating different instances; and (3) a Ridge-Aware Token Pruning (RaTP) strategy that estimates search-token redundancy via ridge-reconstruction errors, selectively retains informative tokens, and aggregates removed tokens without pruning template tokens. Extensive experiments across six benchmarks (including LaSOT, GOT-10k, and TrackingNet) demonstrate competitive performance and improved inference efficiency.

Critical strengths

1.The paper addresses practical issues in tracking: temporal adaptation, distractor discrimination, and computational redundancy. The three proposed components are logically connected and easy to follow. 2.CSPM and RaTP are complementary: CSPM improves feature quality, while RaTP reduces computation. The overall design is sensible for efficient tracking. 3.The paper reports results on GOT-10k, LaSOT, LaSOText, TrackingNet, NFS, and OTB100, and includes ablations on CSPM, IC, RaTP, search frames, templates, and basis-token selection. 4.The LaSOT attribute-wise analysis and qualitative comparisons help demonstrate robustness under motion, occlusion, and appearance changes.

Critical weaknesses

1.The paper reports average results but does not analyze failure cases, especially under severe occlusion, small objects, or fast motion. 2.CSPTrack combines existing ideas in temporal modeling, instance consistency, and token compression, but its methodological novelty is not sufficiently clear. The authors should better distinguish novel contributions from adaptations of prior methods and provide controlled comparisons with relevant baselines under consistent experimental settings. 3.The efficiency claims are not fully supported: retaining 70% of tokens reduces MACs by only about 3.7%, and matched FPS or latency comparisons are missing. Moreover, the ablation results show that removing RaTP slightly improves AUC, raising concerns about its accuracy–efficiency trade-off. 4.The reported improvements are marginal and, in some cases, essentially negligible. For example, the LaSOT AUC gain is only 0.4 points (75.4% vs. 75.0%), the LaSOText gain is only 0.1 point (54.6% vs. 54.5%), the NFS gain is only 0.2 points (71.5% vs. 71.3%), and the OTB100 result is tied at 73.3%. Several attribute-wise gains are also very small, such as 0.1–0.3 points on motion blur, rotation, scale variation, and aspect-ratio change. Moreover, these comparisons are not fully controlled, since the methods differ in backbone resolution, architecture, and computational cost (e.g., B224 vs. B384). Such small margins under mismatched settings do not convincingly demonstrate a robust advantage. 5.The baseline in the component ablation is not clearly defined relative to the adopted MCITrack/CIF backbone, making it difficult to isolate the true contribution of the proposed modules. Most ablations, except the component ablation, are conducted under the pruned setting, and comparisons with concurrent works lack same-backbone or FLOPs-matched controls. The reported efficiency gains from pruning are modest—MACs decrease only from 48.6G to 46.8G at ρ=0.7—which is inconsistent with the claimed 10% speed improvement, and performance degrades noticeably under more aggressive ratios.

Questions for authors

1.Many comparisons mix B224 and B384 models with different architectures and computational costs. Can you provide same-backbone, same-resolution, and FLOPs-matched comparisons with concurrent trackers? Would the reported advantages remain under such matched settings? 2.Most ablations, except the component ablation, are conducted under the pruned setting. Can you provide key ablations under the unpruned setting to isolate the effects of CSPM, IC, search frames, templates, and basis-token selection? 3.CSPTrack combines temporal modeling, instance consistency, and token compression. Can you clarify the essential novel contribution of each component compared with DreamTrack, ARTrackV2, MCITrack, and existing token-pruning methods? Which parts are new designs rather than adaptations? 4.Table 3 shows that removing RaTP gives 75.4% AUC, while the full model gives 75.2% AUC. Can you explain why RaTP should be retained? Is its benefit mainly efficiency, and given the modest MACs reduction, is this trade-off favorable? 5.The paper claims a 10% increase in tracking speed, but MACs decrease only from 48.6G to 46.8G at ρ=0.7, about 3.7%. Can you report actual FPS or latency for different retention ratios, and explain how the 10% figure is measured?

Reviewer_ZXfMOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes CSPTrack, a purely visual single-object tracking framework composed of three components: a Causal State Prediction Module (CSPM), a Target Instance Consistency (IC) constraint, and Ridge-Aware Token Pruning (RaTP). CSPTrack explicitly adopts the Contextual Information Fusion (CIF) module from MCITrack and keeps its internal architecture and hidden-state update mechanism unchanged. CSPM predicts a short-term future-state representation based on the hidden state of CIF; IC aligns the target representations of the template and search frames during training; RaTP estimates search-token redundancy via Ridge reconstruction error and performs retention and aggregation. Experiments cover GOT-10k, LaSOT, LaSOText, TrackingNet, NFS, and OTB100, and include extensive ablation studies.

Critical strengths
  1. RaTP contains some technical detail. It uses Ridge reconstruction error to estimate search-token redundancy, combined with Farthest Point Sampling for basis-token selection and similarity-based aggregation. This is more specific than simple top-k token pruning, and the basis-token selection experiment is also informative.

  2. Broad experimental coverage. The paper evaluates on six tracking benchmarks and provides extensive ablations and attribute-level analysis. The experimental section is generally complete.

  3. Competitive results and clear writing. CSPTrack achieves competitive results on multiple benchmarks, and the paper structure and method description are generally clear.

Critical weaknesses
  1. The motivations of the three modules are not tightly connected, and the method as a whole reads more like an engineering combination of modules from prior tracking methods rather than a coherent framework driven by a unified tracking problem. CSPM targets short-term temporal state prediction, IC targets target discrimination, and RaTP targets inference efficiency. These three correspond respectively to three types of shortcomings in prior trackers, but the paper does not explain why these three modules must be introduced simultaneously, nor how they mutually reinforce each other. This structure of "MCITrack-style contextual propagation + a future-state prediction branch + an InfoNCE regularizer + a token-compression module" makes it difficult for readers to identify the core scientific question of the paper. The authors need to propose a unified tracking failure mode and explain why CSPM, IC, and RaTP must jointly address this problem, rather than presenting three independent motivations that correspond to three existing lines of work.
  2. The necessity of CSPM relative to MCITrack is not established, and its motivation is insufficiently justified. The paper explicitly states that CSPTrack adopts MCITrack's CIF and that the hidden-state update mechanism remains unchanged; CSPM adds future-state queries and next-frame heatmap auxiliary supervision on top of this propagation mechanism. However, the paper does not provide an isolation experiment comparing "MCITrack's original contextual propagation vs. CSPM." Therefore, the gap between CSPM's motivation and MCITrack's existing mechanism is not sufficiently demonstrated.
  3. The efficiency motivation of RaTP relative to token-compression trackers such as ETCTrack is not sufficiently justified. RaTP is similar to ETCTrack in the motivation of "improving tracking efficiency through token compression," and the main difference lies in using Ridge reconstruction error to determine search-token retention and perform aggregation. However, no ablation experiment compares it with ETCTrack's token-compression strategy. More importantly, the paper claims a "10% increase in tracking speed," but RaTP itself also involves Farthest Point Sampling, Ridge regression, matrix inversion, similarity assignment, and aggregation overhead, and the paper does not report end-to-end FPS or latency. Therefore, RaTP's efficiency motivation and its advantage over ETCTrack are not yet sufficiently demonstrated.
Questions for authors
  1. What is the unified tracking problem that the paper attempts to solve? CSPM, IC, and RaTP correspond to three different goals: temporal modeling, discrimination, and efficiency. Can the authors explain the dependencies among the three rather than presenting them as a simple combination of three independent modules?

  2. Can the authors provide an isolation experiment comparing MCITrack's original contextual propagation with CSPM under controlled parameter count and computation?

  3. Can the authors report end-to-end FPS and latency including all RaTP overhead, and compare performance and speed with ETCTrack's token-compression approach?

Reviewer_6qWDOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CSPTrack aims to improve the robustness and efficiency of visual tracking by combining temporal prediction, target-identity consistency, and selective token compression. Future-state supervision encourages features to capture target evolution, while the consistency objective strengthens target discrimination and reconstruction-based pruning removes redundant search tokens while aggregating their information into retained tokens. Experiments across standard tracking benchmarks report competitive accuracy, with component and token-retention ablations examining the contribution of each design and the trade-off between computation and tracking performance.

Critical strengths

The benchmark results are competitive, including 75.4 LaSOT AUC, 54.6 LaSOT-extension AUC, and 78.9 GOT-10k AO. The improvement on LaSOT-extension is particularly encouraging. The component ablations show measurable benefits from the added modules. Token pruning reduces computation from 48.6 to 46.8 G MACs with a 0.2-point LaSOT AUC decrease.

Critical weaknesses

The experiments do not adequately establish the value of the central future-state prediction mechanism. Section 4.1 specifies four search frames per training sample, and Appendix S2 defines supervision using the next sampled frame’s heatmap. However, the temporal ordering and spacing of these frames, together with the relationship between their search-crop coordinate systems, remain unspecified. These details determine what prediction problem the auxiliary objective actually solves. Furthermore, Table 3 removes the entire CSPM, including its four Temporal ViT blocks, so the observed improvement does not isolate the contribution of future supervision from the additional attention capacity. A comparison retaining the same architecture while using only current-frame supervision would directly test the claimed mechanism. Without this control, the evidence supports the utility of the added module but leaves its central forecasting interpretation unresolved.

The evidence does not clearly establish the advance over the inherited tracking framework. Section 3.1 retains MCITrack’s contextual fusion architecture and hidden-state update mechanism unchanged. Yet the baseline in Table 3 obtains 74.0 LaSOT AUC, whereas MCITrack obtains 75.3 in Table 1, compared with CSPTrack’s 75.4. These results may use different settings, but the manuscript does not explain the difference sufficiently to connect the internal ablation gains to improvement over the predecessor. The stronger LaSOT-extension result is valuable, but a matched baseline and a clear account of the configuration differences are needed to establish how much improvement follows from the proposed additions.

Questions for authors
  1. Clarification: How are the four search frames ordered and sampled, and how are their crop coordinates constructed? What happens when the same CSPM architecture is trained with current-frame supervision only?
  2. Clarification: What produces the 74.0 baseline in Table 3, and how does its architecture, training, and inference configuration differ from the 75.3 MCITrack result?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.