acceptodds
Under review as a conference paper at ICLR 2027

SBMVTrack: Spike-Budgeted Multi-View Learning for Power-Efficient UAV Tracking

Abstract

With sparse and event-driven computation, spiking neural networks show great potential for achieving accurate and power-efficient UAV visual tracking. However, existing SNN-based trackers typically use spike firing rates only for power consumption and lack explicit optimization of actual spike activity. Moreover, regulating spike activity alone does not explicitly encourage stable target representations under partial observations and temporal appearance changes. We propose SBMVTrack, a fully spiking tracking framework that combines spike activity regulation with complementary multi-view representation learning. Specifically, SBMVTrack introduces Energy-Weighted Spike Budgeting (EWSB), which incorporates layer-wise computational costs when regulating spike firing rates and penalizing saturated activations, thereby reducing redundant spike computation. To further improve target representations under the spike budget constraint, we introduce Masked Multi-View Target Modeling (MVTM), which treats the initial template, online template, and search region as temporal views of the same target. By aligning target embeddings between masked and corresponding unmasked views and enforcing cross-view identity consistency, MVTM encourages robustness to missing local cues and temporal appearance changes. Experiments on four UAV benchmarks demonstrate competitive tracking performance with a 24.1% reduction in estimated power consumption relative to the baseline. On VisDrone2018, SBMVTrack achieves a success rate of 70.0%, exceeding SpikeTrack by 9.7 percentage points while reducing estimated power consumption by 45.7%. The source code will be released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Spiking Neural Network, UAV Visual Tracking, Power-Efficient Tracking, Spike Budgeting

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tyyVOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates spiking activity constraints and target representation learning for resource-constrained UAV visual tracking, and proposes the fully spiking SBMVTrack. The training stage comprises EWSB and MVTM: the former constrains the spike firing rate and saturated activity according to the computational cost of each layer, while the latter treats the initial template, online template, and search region as temporal views and learns through masked reconstruction and cross-view identity consistency. During inference, only the original tracking branch is retained. On four UAV tracking benchmarks, the authors report an average precision/success rate of 86.5%/67.4% and a theoretical energy consumption of 4.4 mJ. On VisDrone2018, the corresponding figures are 90.0%/70.0%; compared with SpikeTrack, the success rate is 9.7 percentage points higher and the theoretical energy consumption is 45.7% lower.

Critical strengths
  1. EWSB incorporates the computational costs of different layers into the spiking budget, while MVTM improves the stability of target representations under occlusion and appearance changes. Ablation studies show that combining the two achieves the best tracking performance and theoretical energy consumption.

  2. The paper covers four UAV tracking benchmarks and compares multiple types of tracking methods, while reporting precision, success rate, theoretical energy consumption, and inference speed. It also conducts experiments on modules, loss functions, occlusion strategies, pooling methods, hyperparameters, and cross-model generalization.

Critical weaknesses
  1. Equation (12) appears to introduce non-spike-driven division operations during inference. This would make inference on neuromorphic chips impossible, rendering the energy-consumption calculation meaningless. Moreover, since CC, HH, and WW are calculated separately for each layer, hardwiring them onto the chip for shift operations is also unrealistic. If a customized chip were designed specifically for this purpose, it would greatly limit the practical value of this work.
  2. It is not stated whether fine-tuning is performed based on pretrained weights.
  3. Training uses RGB single-object tracking datasets. Does this affect the fairness of the evaluation? Do the datasets listed in Table 1 provide only test sets without training sets? Are their training strategies all the same as that used in this work?
  4. How was the parameter configuration of EWSB determined? The parameters currently discussed are relatively limited. Even if some of them were selected empirically, I believe this should be explained to some extent.
  5. The proposed SBMVTrack does not seem to perform as well as ORTrack, and it is also slower. As shown in Table 7, this weakens the significance of this work to some extent.
  6. The description of the energy-consumption calculation method is missing. Since many different methods for calculating energy consumption currently exist, clarifying this point is very important for fair comparison. I believe this must be included in the main text.
  7. The authors do not provide the code or other raw data. Although this is not necessary, I am quite certain that these materials would be crucial for me to verify the spike-driven characteristics of this paper; nevertheless, this should not be considered a mandatory requirement. More importantly, if possible, I hope the authors can provide the energy-consumption calculation scripts and the raw data for the spike firing rates.
  8. According to my brief investigation, many SNN-based single-object tracking algorithms have already been proposed and released as open source. I believe this paper should evaluate them fairly, such as the following representative works that I collected: SpikeFET [1] and SDTrack [2]. The authors seem to have deliberately avoided discussing these two prior works.
  9. There is no low-power/high-speed inference validation on neuromorphic chips.
  10. The ablation study of EWSB has not sufficiently demonstrated that energy weighting is superior to simple constraints.
  11. There is no fair comparison with STATrack [3], which is a highly related work. According to my observations, some metrics reported in that work are higher than those in this paper. [1] Yang J, Fan L, Zhang J, et al. Fully spiking neural networks for unified frame-event object tracking[J]. Advances in Neural Information Processing Systems, 2026, 38: 121132–121163. [2] Shan Y, Ren Z, Wu H, et al. SDTrack: A baseline for event-based tracking via spiking neural networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026: 36245–36254. [3] Zhong P, Mo J, Zeng D, et al. Fully Spiking Neural Networks with Target Awareness for Energy-Efficient UAV Tracking[J]. arXiv preprint arXiv:2603.27493, 2026.
Questions for authors

See Weakness. If the authors can resolve my concerns or correct my misunderstandings, I will revise my score.

4 comments

Dear authors, the AC has reminded us reviewers to respond to your rebuttal. If there is anything you would like to clarify or discuss, please leave a comment here, and I will respond as soon as possible.

Q1.Equation (12) is used only during training to compute the EWSB regularization loss. The divisions by BTClHlWlBT C_lH_lW_l and by the sum of layer costs normalize training statistics. Once training is complete, the tracker predicts bounding boxes without evaluating Eq. (12). Thus, deploying the tracker does not require implementing these divisions, hardwiring layer dimensions, or designing a custom chip for EWSB. We will mark Eq. (12) explicitly as a training-only computation.

Q2.We initialized the 10M E-SpikeFormer backbone from the V3_10.0M_1x4 pretrained checkpoint and fine-tuned it for tracking. The initial learning rates were 6×10−56\times10^{-5} for the backbone and 6×10−46\times10^{-4} for the prediction head. We will add the checkpoint source and initialization details to Sec. 4.1.

Q3.As described in Sec. 4.1, we trained SBMVTrack on LaSOT, GOT-10k, COCO, and TrackingNet and used the four UAV benchmarks only for evaluation. Following standard tracking practice, we evaluated the competing trackers using their officially released weights. Many of these trackers, including ORTrack, SpikeTrack, and SGLATrack, use the same main training datasets. For each benchmark in Table 1, we evaluated the trackers on the same sequences using the same precision and success metrics.

Q4.Table 6 evaluates rtarr_{\mathrm{tar}} from 0.08 to 0.16. We use 0.12 as a common operating point across datasets. For example, 0.08 gives higher success on UAV123 (68.8% versus 67.9%) and lower estimated energy, whereas 0.12 gives higher success on UAVDT (64.1% versus 61.9%). Table 8 reports the sensitivity to the budget and saturation loss weights. The remaining settings have an operational interpretation: with D=4D=4, normalized activations take values in {0,0.25,0.5,0.75,1}\{0,0.25,0.5,0.75,1\}, so τsat=0.9\tau_{\mathrm{sat}}=0.9 places the saturation threshold between the two highest levels. Appendix A.3.1 discusses how the budget and saturation loss weights affect tracking accuracy, spike activity, and estimated energy (Table 8).

Q5.ORTrack is faster on the tested CPU and GPU. On VisDrone2018, SBMVTrack achieves a higher tracking success rate (70.0% vs. 66.8%) while running at 118.8 FPS on a conventional GPU. The lightweight SBMVTrack-S also exceeds 30 FPS on the tested CPU (33.3 FPS). Hardware supporting event-driven execution could further reduce SBMVTrack’s inference latency by skipping accumulations when no input spike is emitted.

Q6.We agree that the energy-accounting protocol belongs in the main text. Appendix A.1.2 currently gives the arithmetic-energy assumptions, including 4.6 pJ per MAC and 0.9 pJ per AC under the adopted 45-nm model. We will expand Sec. 4.1 to specify the operators included, how spike activity is measured and aggregated, and how each attention multiplication is counted.

Q7.Appendix Table 12 reports layer-wise firing rates averaged over the evaluation data. To make the energy estimate reproducible, we will provide the evaluation script together with the corresponding per-sequence firing-rate statistics, model configuration, and checkpoint identifier. These materials will make it possible to trace the aggregate estimate back to individual operators.

Q8.Thank you for highlighting these methods. SpikeFET is directly relevant, and our submission already evaluates EWSB on SpikeFET-Tiny using synchronized frame–event inputs on FE108 (Table 11). In that transfer experiment, estimated energy decreases from 18.4 to 14.2 mJ, while AUC changes from 67.9% to 67.6%. We will make this result more visible. SDTrack is also relevant as an event-based SNN tracker. The RGB UAV benchmarks in Table 1 do not supply its native event input, so a direct ranking there would confound tracker performance with input modality. We will discuss both methods explicitly and label the modality and purpose of each comparison.

Q9.We agree that end-to-end latency and energy per frame remain unmeasured on a physical neuromorphic chip. We validated the spiking neuron and recurrent attention computation on Lava’s CPU backend using Loihi fixed-point semantics. In a separate full-network simulation, we compared the outputs of 116 active spiking layers against the reference implementation over 117 tracking frames from three UAVDT sequences. Across all 13,572 comparisons, the maximum observed absolute difference was below 0.0001. Table 7 reports measured CPU and GPU throughput, while the reported energy values are estimates based on operation counts and spike activity.

Q10.Table 4 compares a uniform firing-rate budget with a layer-cost-weighted budget on UAV123. Using the weighted budget alone improves precision from 85.5% to 86.5% and success from 66.9% to 67.4%, while estimated energy increases from 4.8 to 5.1 mJ. Adding the saturation penalty yields the best observed trade-off: 87.1% precision and 67.9% success at 4.4 mJ. Appendix Table 8 further examines sensitivity to the budget and saturation loss coefficients.We will clarify that the joint accuracy and energy improvement is demonstrated for the full EWSB objective.

Q11.At the time of submission, we prioritized peer-reviewed publications when selecting baselines for Table 1. We cited STATrack in Related Work but omitted it from the table because it was then available as an arXiv preprint. We agree that its RGB-only UAV tracking setting makes it a directly relevant baseline, and we will add its author-reported precision and success rates to Table 1. SBMVTrack achieves higher success rates on UAVTrack112 (70.1% vs. 69.7%), VisDrone2018 (70.0% vs. 69.4%), and UAV123 (68.0% vs. 67.6%). STATrack achieves a higher success rate on UAVDT (65.3% vs. 64.8%) and slightly higher precision on UAVTrack112 (83.0% vs. 82.8%). The reported theoretical energy estimates are 4.4 mJ for SBMVTrack and 5.6 mJ for STATrack, corresponding to a 21.4% lower estimate for SBMVTrack.

The authors’ responses have resolved several of my concerns. Regarding my first question, I acknowledge that it was indeed a misunderstanding on my part. However, the authors seem to have misunderstood my point about energy estimation. What I meant is that, even when energy consumption is estimated for 45 nm neuromorphic chips, different calculation methods can be used, such as explicitly counting all MAC and AC operations in the network or calculating FLOPs and applying appropriate weights. To my knowledge, some papers also use other methods. A more detailed description would help readers understand the estimation procedure and make fair comparisons.

In their response to Question 5, the authors state that “hardware supporting event-driven execution can further reduce SBMVTrack’s inference latency by skipping accumulation operations when no input spikes are present.” In my view, although this can certainly reduce the number of computations, achieving an actual speedup requires the proposed network to be deployable on neuromorphic chips that support spike-driven inference. The research community still has reservations about this feasibility. Under these circumstances, assuming that the method can accelerate inference is not justified. To my knowledge, Loihi does not support spike-driven inference either.

I acknowledge that the paper I recommended, “Fully Spiking Neural Networks with Target Awareness for Energy-Efficient UAV Tracking,” is indeed still a preprint. Under these circumstances, omitting a comparison with it is acceptable. Nevertheless, given the various remaining issues in this work, I can only raise my score to 2. I hope the authors will implement all the revisions they have promised, as I consider several of the issues I raised to concern fundamental standards of academic rigor. Thank you for your understanding.

Reviewer_tTXJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SBMVTrack, a fully spiking and energy-efficient tracking framework for UAV visual tracking. The method consists of two core components. First, Energy-Weighted Spike Budgeting weights the firing rates of different layers according to their computational costs during training and introduces a saturation activation penalty to reduce redundant spike computation. Second, Masked Multi-View Target Modeling employs masked-to-clean feature alignment and cross-view identity consistency constraints to improve the stability of target representations under missing local information and appearance variations. In addition, the authors design a TAP module.

The authors evaluate the proposed method on four UAV tracking benchmarks: UAVTrack112, UAVDT, VisDrone2018, and UAV123. Experimental results show that SBMVTrack achieves strong tracking accuracy while maintaining relatively low theoretical energy consumption. The paper further provides ablation studies on EWSB and MVTM, sensitivity analyses of key hyperparameters, inference speed evaluations, layer-wise firing-rate statistics, and generalization experiments that apply EWSB to SpikeFET and SpikeTrack, further demonstrating its effectiveness and transferability.

Critical strengths

EWSB provides a simple yet effective approach to optimizing the energy consumption of spiking neural networks, with some degree of generalizability.Rather than merely measuring the firing rate after training, EWSB directly regularizes spike activity during training by weighting the firing rates of different layers according to their computational costs, while also introducing a saturation penalty. The authors validate EWSB not only within SBMVTrack but also by applying it to SpikeFET and SpikeTrack, achieving approximately a 22% reduction in theoretical energy consumption in both cases while largely preserving tracking performance.

Critical weaknesses
  1. The training objective of MVTM may admit a trivial solution corresponding to representation collapse, but the paper does not explain what mechanism prevents this issue in practice.

  2. MVTM appears to be primarily a combination and adaptation of existing techniques, including masking, representation alignment, and consistency learning, to SNN-based tracking, rather than introducing a clearly novel representation-learning mechanism.

Questions for authors
  1. How does MVTM avoid representation collapse in practice?
    In Eqs. (17)–(19), both the reconstruction loss and the cross-view consistency loss minimize the distance between normalized embeddings produced by a shared projection head. Under the current formulation, if the projection head maps all inputs to the same non-zero constant vector, both losses could be trivially minimized. Could the authors clarify whether the actual implementation employs stop-gradient, constraints on the projection head, or any other collapse-prevention mechanism that is not described in the paper?

  2. Could the authors clarify how the theoretical energy consumption of the matrix multiplications in E-SDSA is computed? Specifically, for QK⊤QK^{\top}and the subsequent matrix multiplication involving VV, are these operations counted as dense MACs or as spike-driven ACs? If they are treated as sparse AC operations, which tensor's firing rate is used to estimate the effective number of operations? Alternatively, how is the joint sparsity of the two operands accounted for when estimating the actual operation count?

1 comment

Q.1 Thank you for pointing this out. In our implementation, the views share a projection head. For the reconstruction loss, we stop gradients through each clean-view target. For the consistency loss, we stop gradients through the initial-template anchor. The projection head has no explicit variance or decorrelation penalty. We will make these gradient paths explicit in Eqs. (17)–(19). We agree that stop-gradient alone does not exclude a constant solution.

To check whether the submitted checkpoint reached that solution, we examined one clean triplet from each of 64 distinct GOT-10k validation sequences. For each search embedding, we ranked all 64 initial-template embeddings by cosine similarity. The correct template ranked first for 51 of 64 queries (79.7%) after projection, compared with 24 of 64 (37.5%) before projection, with no ties for first place after projection. The projected directions were highly concentrated, but they retained enough variation to distinguish the matching template. This rules out the exact constant-output solution on these samples. All three views were jointly encoded, as in the tracker.

Q2. Thank you for asking us to clarify the counting rule. Our estimator treats both attention matrix products as spike-driven ACs. Let F1F_1 and F2F_2 denote the dense MAC counts for A=QK⊤A=QK^\top and AVAV, respectively, and let rQ,rK,rVr_Q,r_K,r_V denote the mean normalized spike activations. For QK⊤QK^\top, we use rQrKr_Qr_K as a proxy for joint activity. This product-of-means approximation does not capture correlations between individual query and key activations. For AVAV, the intermediate matrix AA is not re-quantized, so we use the activity rVr_V of the spiking value tensor.

With T=1T=1 and D=4D=4, the estimated AC count is D(F1rQrK+F2rV)D(F_1r_Qr_K+F_2r_V). We multiply this count by 0.9 pJ per AC to estimate arithmetic energy. We will add this operator-level formula to the manuscript.

Reviewer_JoU1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SBMVTrack, a fully spiking framework for power-efficient UAV tracking, built on two training-time strategies. Specifically, Energy-Weighted Spike Budgeting (EWSB) regulates layer-wise spike activity based on computational cost, while Masked Multi-View Target Modeling (MVTM) improves robustness to partial observations and appearance variations.

Critical strengths

1.Practical spike regulation: EWSB regulates layer-wise spike activity by weighting computational costs. This is a sensible approach to cut theoretical power consumption while keeping the inference architecture unchanged. 2.Robust representation via MVTM: By treating different temporal views as masked inputs, MVTM helps stabilize target representations against partial observations and appearance changes during training. 3.Competitive accuracy-power trade-off: The model shows competitive tracking accuracy on UAV benchmarks at a much lower estimated power than existing trackers.

Critical weaknesses

1.The claim that existing SNN-based trackers generally do not explicitly constrain spike firing rates during training appears overly broad and potentially misleading. Prior studies have incorporated spike firing rates, spike activity, or synaptic operation counts into training objectives or regularization mechanisms to promote sparse neural activity and improve energy efficiency [1, 2]. Related efforts have also explored explicit optimization of spike dynamics in SNN-based object tracking. Therefore, the authors should provide a more comprehensive review of relevant literature and clarify how their proposed method differs from existing approaches.

[1]Zhang J, Zhang M, Wang Y, et al. Spiking neural networks with adaptive membrane time constant for event-based tracking[J]. IEEE Transactions on Image Processing, 2025, 34: 1009-1021. [2]Wang Y, Zhang J, Sun C, et al. SpikeTrack: High-performance and Energy-efficient Event-Based Object Tracking with Spiking Neural Network[C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026: 926-935.

2.The training strategies of the compared methods are not clearly described, making it difficult to determine whether they are evaluated under fair and comparable training settings. Differences in training protocols may affect tracking performance and thus the validity of the comparisons. In addition, the inference settings section refers to calculation details in the supplementary material, but no supplementary material was provided with the submission.

3.The evaluation includes only one SNN-based tracking baseline, making it difficult to assess how the proposed method compares with other SNN-based trackers.

4.The choice of the target firing-rate budget, a key component of EWSB, is not sufficiently justified. In particular, it is unclear why the budget starts at 0.08, and the different results on UAVDT and UAV123 are not adequately explained. The authors should provide a sensitivity analysis to clarify how the budget affects tracking performance and energy consumption across datasets. It would also be interesting to investigate whether the budget could be learned adaptively rather than manually set.

Questions for authors

Please see Critical Strengths and Critical Weaknesses.

1 comment

Q1.We thank the reviewer for highlighting these prior studies and for encouraging us to reassess how we characterized spike regulation in existing SNN-based trackers. In event-based tracking, Zhang et al. [1] adapt the membrane time constant to the input, and Wang et al. [2] use DI-LIF neurons to suppress noise spikes. Both approaches regulate spike activity through neuron dynamics. EWSB takes a complementary approach in RGB UAV tracking by incorporating a target firing rate directly into the training objective. We weight each layer’s firing rate by its computational cost and penalize deviations of the resulting weighted average from a prescribed target. This gives greater weight to spike activity in computationally expensive layers. A separate term penalizes near-saturated activations, and both terms are optimized jointly with the tracking loss. We will revise the Introduction and Related Work to acknowledge these studies and clarify this distinction.

Q2.Following SGLATrack,We trained SBMVTrack on LaSOT, GOT-10k, COCO, and TrackingNet and used the four UAV benchmarks for evaluation. In Table 1, we used the authors’ released weights and evaluated the trackers on the same sequences with the same precision and success metrics. Their checkpoints retain their original training protocols. We will document the checkpoint and training-data sources for the compared methods. The energy calculation is described in Appendix A.1.2, after the references in the manuscript PDF (p. 13, Eqs. 22–23). We will move the essential calculation details into Sec. 4.1.

Q3.At the time of submission, we prioritized peer-reviewed publications when selecting baselines for Table 1. We cited STATrack in Related Work but omitted it from the table because it was then available as an arXiv preprint. We will add STATrack, which uses the same RGB-only UAV tracking setting, and identify its results as author-reported. Across the four shared benchmarks, SBMVTrack has higher success on UAVTrack112 (70.1% vs. 69.7%), VisDrone2018 (70.0% vs. 69.4%), and UAV123 (68.0% vs. 67.6%), while STATrack has higher success on UAVDT (65.3% vs. 64.8%). The reported theoretical energy estimates are 4.4 mJ for SBMVTrack and 5.6 mJ for STATrack, corresponding to a 21.4% lower estimate for SBMVTrack. For frame–event tracking, Table 11 separately evaluates the transfer of EWSB to SpikeFET-Tiny on FE108: estimated energy decreases from 18.4 to 14.2 mJ, and AUC changes from 67.9% to 67.6%. We will make the input modality and purpose of each comparison explicit.

Q4.Given the baseline’s measured average spike firing rate of 0.188, we chose 0.08 as the lower endpoint of our empirical sweep to evaluate tracking under a restrictive budget. Table 6 evaluates five target budgets from 0.08 to 0.16 in increments of 0.02. Across this range, the measured average firing rate increases from 0.131 to 0.153, and estimated energy increases from 4.12 to 4.77 mJ.We selected 0.12 as a common budget across datasets. Increasing the target from 0.08 to 0.12 improves the success rate on UAVDT from 61.9% to 64.1% (+2.2%), while reducing it on UAV123 from 68.8% to 67.9% (−0.9%). At 0.12, success rates on both datasets remain above their respective baselines, and estimated energy falls from the baseline value of 5.80 mJ to 4.42 mJ. We therefore chose 0.12 as an empirical trade-off between tracking accuracy across datasets and estimated energy consumption.

We thank the reviewer for suggesting an adaptive budget. Under the current penalty (rew−rtar)2(r_{\mathrm{ew}}-r_{\mathrm{tar}})^2 , a freely learnable target could reduce the loss simply by matching the observed weighted firing rate, without reducing spike activity. One direction we plan to explore is to impose an overall energy limit and allow the model to allocate layer-wise spike budgets adaptively for each input.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.