acceptodds
Under review as a conference paper at ICLR 2027

Activate Sparse, Predict Continuously: Brain-inspired Ternary Sparse Activation with Predictive Coding for Multi-object Tracking

Abstract

Multi-object tracking (MOT) must detect objects and preserve their identities in continuous video streams under tight computational budgets. Spiking neural networks (SNNs) promise low-power computation, but their sparsity lives along the temporal axis: they encode information by accumulating events over timesteps, which mismatches the instantaneous spatial discrimination and continuous state estimation that online MOT demands, and occlusion-induced event loss destroys the features needed to re-identify a target. To address these issues, this paper proposes an efficient online tracking framework inspired by the sparse coding and predictive cognition mechanisms of biological vision, termed Activate Sparse, Predict Continuously (ASPC). First, we propose a ternary sparse activation network that gates neural responses dynamically and encodes features with ternary sparse activations (-1/0/+1), shifting sparse computation from the temporal domain to a ternary sparse activation-based feature representation space; this mechanism preserves the directional discriminative information of single-frame features while enabling adaptive feature selection. Furthermore, we design a prediction-driven cognitive association module that formulates the tracking process as dynamic inference based on historical state priors, thereby maintaining identity continuity through internal states when visual information is insufficient. Experiments on MOT17, DanceTrack, and SportsMOT show that ASPC matches or surpasses the strongest full-precision trackers in HOTA and AssA while using a single forward pass: on MOT17 it attains HOTA 59.3 and AssA 58.2 at 25.7 mJ per frame, against 1105 mJ for the strongest full-precision baseline, and it exceeds an identically configured spiking counterpart by 29.3 HOTA points on MOT17 and 22.5 points on DanceTrack. Its activation sparsity adapts to scene density (48-59% zero activations), with per-frame inference energy of only 25.7 mJ, 11 mJ, and 19 mJ on the three datasets.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Spiking Neural Networks, Multi-Object Tracking, Sparse Neural Computation, Predictive Representation Learning, Efficient Visual Intelligence

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_gCJLOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This article studies the correlation accuracy and computational energy consumption in multi-target tracking. Based on the CAMELTrack correlation framework, a three value sparse activation and prediction driven correlation mechanism is introduced to reduce computational overhead and maintain cross frame identity continuity. The author conducted experiments on MOT17, DanceTrack, and SportsMOT, compared them with the self constructed LIF-SNN baseline and existing full precision methods, and reported tracking performance, activation sparsity rate, and energy consumption. The current manuscript still lacks sufficient explanation on the specific implementation of SNN baseline configuration, predictive correlation mechanism, and energy consumption statistical caliber. Some theoretical derivations and experimental tables also have consistency issues that need clarification or correction. Supplementing with baseline conditions, independent module ablation, and original evaluation results will help to more accurately assess the contributions of each design and the applicability of the conclusions.

Critical strengths
  1. This method generates sparse features in one forward propagation, reducing the dependence on multi time step pulse accumulation, and is suitable for frame by frame online tracking tasks.

  2. According to the results reported in the article, compared with SNNs with the same architecture, this method improves HOTA by approximately 29.3, 22.5, and 19.8 percentage points on the three datasets, respectively, while also having lower energy consumption.

  3. Triple activation uses \ (\ {-1,0,+1 } ) to represent features, while introducing zero values to reduce computation and preserving the positive and negative information of the features.

  4. The association process combines historical features and current observations to enable the model to still utilize existing identity information for subsequent matching in the event of occlusion or missed detections.

Critical weaknesses

1.The author mentioned in the abstract that the intra frame temporal dynamics of SNN do not match the transient spatial discrimination and continuous state estimation required by MOT. But I think the latter refers to inter frame dynamics, and if the intra frame autoregression speed of SNN is fast enough, this does not constitute a problem. How does the author view this.

2.I am skeptical about the author's statement in the abstract that time information is integrated into the characteristic information of internal pulses, as the actual representation of pulse information is still based on pulse firing rate.

3.There are very obvious prompt word injections at the beginning of each paragraph in the Introduction (traces of automatic compilation using plugin LaTeX, possibly due to warnings and certain errors), indicating that this article may have been completely generated by AI or directly polished by AI, which is completely inconsistent with ICLR rules. I cannot understand why such an article can enter the review process.

4.In the Introduction of this article, it is mentioned that "when an object is occluded, its events stop, the potential leaks away, eliminating the very features needed to re identify the object upon reappearance". For robustness reasons, I believe this is a balanced process. However, this article lacks quantitative experiments to demonstrate that in MOT tasks or on selected datasets, this has a negative impact on the performance of SNN, rather than a positive impact on the robustness improvement of SNN networks caused by membrane potential attenuation factors.

5.In my opinion, the author's several introductions are completely general and do not mention any practical issues. However, considering that this is generated by AI, I think this can be understood.

6.Regarding the four contributions proposed by the author, the first one I believe lacks any insightful analysis, visualization, or other contributing problem analysis; The second one is not proposed in this article, and I can confirm that there have been numerous studies on Ternary Spike [1] [2]. For innovation point 3, many works that have been proposed to introduce MAMBA have done so, and there are also many works in single object tracking that can be summarized without loss by this sentence, such as SeqTrack [3], AQATrack [4], and ARTrack [5] series of works. As of the Introduction, I cannot get any actual design of this article, which I think is unreasonable.

7.The method proposed in this article does not seem to significantly improve the performance of the model, as shown in Table 1. In terms of the four indicators of DetA, IDF1, and MOTA, the method proposed in this article is far inferior to the older work MOTIP. In Table 2, the method proposed in this article did not show any performance improvement in various indicators. In addition, there were some formatting issues in Tables 1 and 2, such as the need to retain decimals with the same number of digits.

8.On DanceTrack, ASPC’s HOTA score of 61.24 is lower than both MOTIP’s 67.5 and DiffMOT’s 62.3, yet the text states that it “trails only MOTIP.” The conclusion that ASPC “matches or surpasses the strongest full-precision methods across all three datasets” should also be qualified.

[1]Guo Y, Chen Y, Liu X, et al. Ternary spike: Learning ternary spikes for spiking neural networks[C]//Proceedings of the AAAI conference on artificial intelligence. 2024, 38(11): 12244-12252. [2]Wang S, Zhang D, Belatreche A, et al. Ternary spike-based neuromorphic signal processing system[J]. Neural Networks, 2025, 187: 107333. [3]Chen X, Peng H, Wang D, et al. Seqtrack: Sequence to sequence learning for visual object tracking[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023: 14572-14581. [4]Xie J, Zhong B, Mo Z, et al. Autoregressive queries for adaptive tracking with spatio-temporal transformers[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024: 19300-19309. [5]Wei X, Bai Y, Zheng Y, et al. Autoregressive visual tracking[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023: 9697-9706.

Questions for authors
  1. Why is there a necessary conflict between the intra frame time step of SNN and the cross frame state estimation of MOT? What pulse encoding is used in the text, and how does time information specifically participate in the expression of identity features?

  2. Is there any quantitative experiment indicating that membrane potential attenuation can impair identity recovery after occlusion? Can the impact on information retention and noise suppression be distinguished by adjusting the leakage coefficient and occlusion duration?

  3. What is the specific innovation of this article compared to existing methods for representing ternary pulses and modeling historical states? Can the contributions of each design be demonstrated through independent ablation?

  4. Has the SNN baseline undergone independent tuning? Can the time steps, leakage coefficient, threshold, and training configuration be made public, and compared for accuracy, latency, and energy consumption after reasonable optimization?

  5. After Lemma 1 fixes two norms\ Why is it still described as a minimization problem when H (m) is already a constant? How does the author explain the issue that the equal amplitude positive component corresponds to the maximum value, and that the threshold \ (E ) only represents the gradient sign boundary?

  6. According to the standard definition of HOTA, multiple groups of HOTA in the SportsMOT table have exceeded the upper limits given by the corresponding DetA and AssA. What is the reason for this? Can you provide the original evaluation output and verify the data sources and transcription accuracy of baselines such as MOTIP?

  7. Why does the main text say that ASPC's HOTA on DanceTrack is "only behind MOTIP" when it is lower than both MOTIP and DiffMOT? Are the existing results sufficient to support the conclusion of 'matching or surpassing the strongest full precision method on three datasets'?

7 comments

Dear authors, the AC has reminded us reviewers to respond to your rebuttal. If there is anything you would like to clarify or discuss, please leave a comment here, and I will respond as soon as possible.

AuthorsOpenReview ↗

Baseline Verification, Component Ablation, and AI-Use Clarification

1.AI assistance was used solely for language polishing and figure-caption drafting, as stated in the AI-use statement. The methods, derivations, experiments, code, and all reported data are the authors’ own work. The raw energy logs and TrackEval outputs will be provided in the supplementary material. In the revised manuscript, we will rewrite formulaic passages in our own words, shorten long sentences, unify terminology, and expand the per-section AI-use statement. We would be happy to provide the code and measurement logs to the AC for verification.

2.We agree that cross-paper differences in detectors and protocols must be made explicit. To verify the quoted numbers, we reproduced MOTIP under our own pipeline; the reproduced values are close to those reported in the original paper. In the revision, every quoted number will be labeled with its corresponding detector and data split, and the reproduction results will be included in the supplementary material.

3.We agree that PAC is a main contribution and must be isolated. We completed the component ablation on DanceTrack; the original CAMELTrack baseline corresponds to the fixed-threshold configuration without ternary gating. The two new runs isolate PAC and TSAN respectively: Component ablation on the DanceTrack validation split (25 sequences, 225,148 GT boxes), with identical detections, splits and evaluation script. The two component runs use the same state cache and were trained separately; the full ASPC row is the main-table run, included for reference. Configuration HOTA↑ AssA↑ DetA↑ MOTA↑ IDF1↑ TSAN-only 61.69 47.88 79.84 88.89 62.23 PAC-only 62.31 52.29 74.91 82.12 65.46 Full ASPC 61.24 47.31 79.63 87.09 62.19

TSAN-only reaches HOTA 61.69 / MOTA 88.89 / IDF1 62.23 at the ternary-activation cost reported in Table 4. PAC-only yields the strongest association-side metrics: AssA 52.29 (+4.41 over TSAN-only), IDF1 65.46 (+3.23), and 500 identity switches versus 3270 for TSAN-only (6.5× fewer), which is the identity-continuity benefit PAC is designed to provide. TSAN-only remains stronger on detection-side metrics (DetA 79.84 vs 74.91, MOTA 88.89 vs 82.12). The two modules act on different failure axes: TSAN improves the representational/computational side, PAC improves association and identity preservation.

4.We have unified all energy measurements to a CPU-measured protocol, with the idle baseline subtracted, and our model and all compared methods are measured under the same protocol. The per-frame energy values under this unified protocol will be provided later in the main text, and the relevant tables will be updated accordingly.

Area_Chair_t9zgOpenReview ↗

Flag For Ethics Review

Dear Reviewer, could you please briefly clarify the ethical concern that led you to flag the paper for ethics review?

Reviewer_j4K6OpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Activate Sparse, Predict Continuously (ASPC), an energy-efficient framework for online multi-object tracking. Built upon the CAMELTrack association pipeline, ASPC introduces two main components:

  1. Ternary Sparse Activation Network (TSAN): It employs Hoyer-based gating to map neural activations into {-1, 0, +1}, enabling sparse computation while preserving signed feature information.
  2. Prediction-driven Cognitive Association (PAC): It formulates track–detection association using a predictive state model inspired by the free-energy principle, aiming to maintain identity continuity when visual observations are missing or unreliable.

Experiments on MOT17, DanceTrack, and SportsMOT report competitive tracking performance with reduced inference energy. ASPC achieves HOTA scores of 59.3, 61.2, and 75.5, respectively, substantially outperforming the authors' spiking baseline while maintaining approximately 48–59% zero activations.

Critical strengths
  1. Practical motivation and efficient sparse representation. The paper addresses the computational cost of online multi-object tracking, which is relevant to resource-constrained deployment. The proposed ternary activation mechanism combines activation sparsity with signed feature representations, providing a potentially useful alternative to conventional binary and spiking activations.

  2. Promising experimental results across multiple benchmarks. ASPC is evaluated on MOT17, DanceTrack, and SportsMOT and consistently outperforms the authors' SNN baseline in tracking accuracy. In particular, Table 4 shows that ternary activation achieves a higher association AUROC (0.9798) than binary activation (0.9694), suggesting that preserving signed information can benefit discriminative association features.

Critical weaknesses

1. Insufficient ablation studies of the core components.

Although Section 4.3 (Table 4) compares different activation mechanisms and Hoyer regularization weights, the paper lacks ablation studies that isolate the contributions of TSAN and PAC. In particular, there is no direct comparison with the original CAMELTrack under identical settings, nor an evaluation of the tracking performance without PAC. Consequently, it is difficult to determine whether PAC genuinely improves object association and how much each component contributes to the overall performance. This weakens the experimental support for the paper's main contributions.

2. Insufficient validation and qualitative analysis of the predictive association mechanism.

The paper claims that PAC maintains identity continuity under occlusion or missing visual observations. However, Section 4.2 mainly relies on aggregate metrics such as ID switches, fragmentation, and AssA, without a dedicated quantitative evaluation of identity recovery under occlusion. Furthermore, the paper provides no qualitative tracking visualizations or comparisons illustrating how PAC behaves differently from the original association method in challenging occlusion scenarios. Therefore, the current evidence does not sufficiently demonstrate the practical benefits of the proposed predictive association mechanism under occlusion.

Questions for authors
  1. Contribution of PAC. Could the authors provide an ablation comparing the original CAMELTrack, TSAN-only, PAC-only, and the complete ASPC under identical experimental settings? In particular, how much does PAC independently improve association accuracy and identity recovery under occlusion?

  2. Free-energy formulation. Could the authors clarify how Eq. (6) is derived from the Gaussian generative model in Eq. (5)? How are the predictive covariance, observation covariance, and state complexity cost computed in practice? Furthermore, how does the soft assignment in Eq. (9) enforce one-to-one matching constraints and handle unmatched detections?

  3. Mathematical consistency of TSAN. Appendix A states that the Hoyer measure is minimized when all positive entries are equal, which appears inconsistent with the stated definition of the measure. Could the authors clarify this derivation? Additionally, Eq. (3) appears to allow nonzero surrogate gradients for some units whose forward activations are zero. How does this support the claim that the backward pass inherits forward activation sparsity?

  4. Fairness of energy evaluation. Do the reported per-frame energy measurements include the detector, ReID extractor, pose estimator, and association module, or only the association network? Could the authors clarify how the operation-level estimates and hardware measurements are distinguished, and provide a comparison against the original full-precision CAMELTrack using identical hardware, input features, and evaluation settings?

1 comment
AuthorsOpenReview ↗

Component Ablation, Free-Energy Derivation, and Energy Evaluation

1.We agree that PAC is a main contribution and must be isolated. We completed the component ablation on DanceTrack; the original CAMELTrack baseline corresponds to the fixed-threshold configuration without ternary gating. The two new runs isolate PAC and TSAN respectively: Component ablation on the DanceTrack validation split (25 sequences, 225,148 GT boxes), with identical detections, splits and evaluation script. The two component runs use the same state cache and were trained separately; the full ASPC row is the main-table run, included for reference. Configuration HOTA↑ AssA↑ DetA↑ MOTA↑ IDF1↑ TSAN-only 61.69 47.88 79.84 88.89 62.23 PAC-only 62.31 52.29 74.91 82.12 65.46 Full ASPC 61.24 47.31 79.63 87.09 62.19

TSAN-only reaches HOTA 61.69 / MOTA 88.89 / IDF1 62.23 at the ternary-activation cost reported in Table 4. PAC-only yields the strongest association-side metrics: AssA 52.29 (+4.41 over TSAN-only), IDF1 65.46 (+3.23), and 500 identity switches versus 3270 for TSAN-only (6.5× fewer), which is the identity-continuity benefit PAC is designed to provide. TSAN-only remains stronger on detection-side metrics (DetA 79.84 vs 74.91, MOTA 88.89 vs 82.12). The two modules act on different failure axes: TSAN improves the representational/computational side, PAC improves association and identity preservation.

2.We agree that aggregate IDSW/Frag/AssA alone are not sufficient to prove the occlusion mechanism. We are currently exploring an implementation path for the qualitative visualization: identity-strip timelines over representative occlusion events, comparing TSAN-only and ASPC. The quantitative analysis will bin occlusion gaps into short (1–15 frames), medium (16–60) and long (>60) intervals and report the fraction of gaps after which the same identity is resumed rather than a new one being spawned. Both the table and the figure will be added in the revision; we will not overstate the occlusion claim until this evidence is included.

3.Derivation of Eq. (6) from Eq. (5). Let q(s)=N(μ, Σpost), p(s|s_prev)=N(s̄, P), p(z|s)=N(z; s, Σ). The variational free energy is F = E_q[log q(s) − log p(z|s) − log p(s)]. Minimizing over μ gives μ = Σpost(Σ⁻¹z + P⁻¹s̄) with Σpost⁻¹ = Σ⁻¹ + P⁻¹. Substituting back gives F = ½‖z − s̄‖²_{(Σ+P)⁻¹} + ½log|Σ+P| + const. Setting C_i = ½log|Σ+P_i| + const yields Eq. (6). The full step-by-step expansion will be added to Appendix A.

4.Practical computation. Σ_j = σ_j²I, with σ² estimated from the residual variance of positive tracklet–detection embedding pairs on the training split. P_i = P_0 + α·a_i, where a_i is the number of frames since the freshest update in track i's feature bank. C_i = c_0 + β·a_i (equivalently ½log|Σ+P_i|); the constant part is absorbed into the calibrated threshold θ.

5.We have unified all energy measurements to a CPU-measured protocol, with the idle baseline subtracted, and our model and all compared methods are measured under the same protocol. The per-frame energy values under this unified protocol will be provided later in the main text, and the relevant tables will be updated accordingly.

Reviewer_PiGZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ASPC, an efficient association framework for online multi-object tracking. Building upon CamelTrack, the method introduces a Ternary Sparse Activation Network (TSAN) to reduce computational costs through sparse, signed feature representations. It also designs a Prediction-driven Cognitive Association (PAC) module that combines historical state priors with current observations for probabilistic association, aiming to improve identity continuity under occlusion. Experiments on MOT17, DanceTrack, and SportsMOT demonstrate competitive tracking performance with relatively low reported energy consumption.

Critical strengths

This paper presents a relatively comprehensive solution to the challenges of computational efficiency and identity continuity in online multi-object tracking, with a clear research motivation. The experiments cover three tracking datasets with different scene characteristics and compare SNN, binary, and ternary activation mechanisms, demonstrating competitive results in terms of tracking accuracy and computational efficiency.

Critical weaknesses
  1. We suggest adding an independent ablation study for the Prediction-driven Cognitive Association (PAC) module. PAC is one of the two main contributions of this paper. We recommend comparing the original CamelTrack, CamelTrack with PAC alone, and the complete ASPC framework. We also suggest evaluating identity recovery performance under different occlusion durations to verify the actual contribution of the Bayesian prediction and free-energy-based association mechanism to identity continuity.
  2. We suggest further strengthening the analysis of SNN limitations. The paper compares ASPC only against a self-implemented LIF-SNN baseline and attributes the performance gap to fundamental limitations of temporal sparsity in SNNs. We recommend including comparisons with more representative SNN methods to better substantiate this claim.
Questions for authors

Regarding the fairness of energy comparisons: As we understand it, the theoretical energy estimation in this paper primarily focuses on the association network, while the energy figures for other tracking methods are taken directly from their respective publications. We would like to clarify whether these comparisons cover the same computational scope, such as whether object detection, ReID, and pose estimation modules are included in all cases. If there are differences, could the authors further clarify the scope of energy measurement for each method, specifically whether the reported values correspond to the entire tracking pipeline or only key components such as the association network?

2 comments
AuthorsOpenReview ↗

PAC Ablation, SNN Baselines, and Energy Scope

1.We agree that PAC is a main contribution and must be isolated. We completed the component ablation on DanceTrack; the original CAMELTrack baseline corresponds to the fixed-threshold configuration without ternary gating. The two new runs isolate PAC and TSAN respectively: Component ablation on the DanceTrack validation split (25 sequences, 225,148 GT boxes), with identical detections, splits and evaluation script. The two component runs use the same state cache and were trained separately; the full ASPC row is the main-table run, included for reference. Configuration HOTA↑ AssA↑ DetA↑ MOTA↑ IDF1↑ TSAN-only 61.69 47.88 79.84 88.89 62.23 PAC-only 62.31 52.29 74.91 82.12 65.46 Full ASPC 61.24 47.31 79.63 87.09 62.19

TSAN-only reaches HOTA 61.69 / MOTA 88.89 / IDF1 62.23 at the ternary-activation cost reported in Table 4. PAC-only yields the strongest association-side metrics: AssA 52.29 (+4.41 over TSAN-only), IDF1 65.46 (+3.23), and 500 identity switches versus 3270 for TSAN-only (6.5× fewer), which is the identity-continuity benefit PAC is designed to provide. TSAN-only remains stronger on detection-side metrics (DetA 79.84 vs 74.91, MOTA 88.89 vs 82.12). The two modules act on different failure axes: TSAN improves the representational/computational side, PAC improves association and identity preservation.

2.We agree that aggregate IDSW/Frag/AssA alone are not sufficient to prove the occlusion mechanism. We are currently exploring an implementation path for the qualitative visualization: identity-strip timelines over representative occlusion events, comparing TSAN-only and ASPC. The quantitative analysis will bin occlusion gaps into short (1–15 frames), medium (16–60) and long (>60) intervals and report the fraction of gaps after which the same identity is resumed rather than a new one being spawned. Both the table and the figure will be added in the revision; we will not overstate the occlusion claim until this evidence is included.

3.We believe that using only a single self-implemented LIF network as the baseline is insufficient to represent SNN methods. We therefore selected the more representative SMTrack and SpikingMOT for comparison and will add them to the table in the main text in a subsequent revision. SMTrack:DanceTrack: HOTA 57.76 / IDF1 58.47 / AssA 42.62; MOT17: HOTA 63.9 / IDF1 78.3 / AssA 64.5; MOT20: HOTA 59.9; T=4 SpikingMOT:DanceTrack: HOTA 56.5 (YL104M); SportsMOT: 74.9; MOT17: 61.4; motion predictor 3.6M; energy −86.7%

4.We have unified all energy measurements to a CPU-measured protocol, with the idle baseline subtracted, and our model and all compared methods are measured under the same protocol. The per-frame energy values under this unified protocol will be provided later in the main text, and the relevant tables will be updated accordingly.

Thanks to the authors for their response and providing additional experiments. I note that although the new ablation results show that PAC can improve identity association, the full ASPC achieves lower HOTA, AssA, and IDF1 than PAC-only, which weakens the contribution of combining the two modules. In addition, the occlusion recovery experiments and unified energy evaluation have not yet been provided. The claimed limitations arising from the temporal sparsity mechanism of SNNs have also not been sufficiently validated in the experiments. Therefore, I maintain my original concerns and rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.