acceptodds
Under review as a conference paper at ICLR 2027

Which Internal Neural Representations Should Become Spikes? Task-Relevant Spatiotemporal Threshold Adaptation for Audio-Visual Spiking Neural Networks

Abstract

Spiking neural networks (SNNs), regarded as the third generation of neural networks, exhibit rich spatiotemporal dynamics. Audio-visual SNNs (AV-SNNs) can exploit the temporal nature of spike-based computation and event-driven processing for efficient multimodal learning. However, existing AV-SNN studies often leave a more fundamental question implicit: which internal neural representations should become spikes? Different modalities exhibit different task-relevant spatiotemporal structures, making it necessary to preserve highly task-relevant representations while distinguishing redundant ones. We propose Spatio-Temporal Modality-Adaptive Thresholding (ST-MAT), consisting of three components: Task-Relevant Spatiotemporal Support Estimation (TRSSE) estimates task relevance from continuous pre-spike representations and identifies highly task-relevant locations in the spatiotemporal domain. Semantic–Redundancy Balancing (SRB) measures how task-relevant information is distributed over non-redundant spatiotemporal locations and balances semantic retention against redundancy rejection. Slow Bounded Threshold Adaptation (SBTA) learns neuronal thresholds through slow bounded updates to determine which continuous responses should be converted into spikes. Rather than prescribing how many spikes a modality should emit, ST-MAT changes which internal neuronal representations are converted into spikes. ST-MAT achieves 78.47% on AVE, 79.44% on CREMA-D, and 99.54% on UrbanSound8K-AV, improving ST-MAT with fixed threshold by 5.2%, 3.23%, and 1.14%, respectively. It also surpasses the same-architecture ANN on AVE and CREMA-D. Visualizations and ablation experiments show that ST-MAT converts high-task-value internal neuronal representations into spikes while reducing redundancy, thereby substantially improving AV-SNN performance. ST-MAT introduces no additional inference overhead, making it well suited to brain-inspired edge multimodal fusion.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Spiking Neural Networks, Multi-modal, Audio-visual modality, Neural Representations

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BLxeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ST-MAT for threshold adaptation in audio-visual SNNs. It combines task-relevance estimation, redundancy balancing, and bounded threshold updates to improve spike representation selection. Experiments on AVE, CREMA-D, and UrbanSound8K-AV report improvements over fixed and learnable thresholds.

Critical strengths

The paper addresses an interesting problem in spike generation rather than focusing only on multimodal fusion. The method is well motivated, and the improvements on AVE and CREMA-D are encouraging. The comparisons with fixed and learnable thresholds and the component ablations provide useful evidence of its effectiveness.

Critical weaknesses

The connection between task-relevant spatiotemporal selection and the deployed shared thresholds is not fully explained. In addition, the threshold notation and the order of operations in the adaptation procedure need clarification, which affects the interpretation and reproducibility of the proposed method.

Questions for authors
  1. Eq. (6) updates one threshold per modality and stage; could the authors clarify how location-specific task-relevance estimates during training translate into selective firing at inference, when these estimates are no longer computed?

  2. Section 5.3 refers to θ* as the learned threshold when defining Δp for Figure 3(d), while Eq. (6) defines θ* as the target threshold; please clarify whether the visualization uses the final deployed threshold or the intermediate target.

  3. Algorithm 1 updates network weights in line 4 before computing task relevance in lines 7–8; please clarify whether the calibration step recomputes the forward pass or uses activations obtained before the weight update.

I would consider raising my rating to 3 if the authors can satisfactorily clarify these concerns, particularly the connection between task-relevance estimation and the deployed threshold mechanism.

2 comments
AuthorsOpenReview ↗

Task-Aware Spike Selection and Shared Threshold Optimization

Thank you for your constructive feedback. We expanded the method and theory in Sections 4.1–4.3 and Appendix A, clarified Figure 2, and added controlled results for threshold granularity (main-text Table 3), auxiliary heads (Appendix G), calibration (H), and response selection (K).

Weakness 1 & Question 1

ST-MAT deploys a single scalar θm,l\theta_{m,l} per modality–stage pair, shared across time, tokens, and channels: Sm,l,b,t,n,d=H(Um,l,b,t,n,d−θm,l)S_{m,l,b,t,n,d}=H(U_{m,l,b,t,n,d}-\theta_{m,l}). It does not assign distinct spikes to identical membrane potentials through position-wise gates. Instead, training-time TRSSE computes Ai,d=∣Ui,d∂M/∂Ui,d∣A_{i,d}=|U_{i,d}\partial M/\partial U_{i,d}| and characterizes effective task-relevant spatiotemporal support. SRB constructs semantic mass p+p^+ and suppressible-redundancy mass p−p^- from the task-aware evidence and selects an observed cutoff θ∗\theta^* maximizing Rsem(θ)αRred(θ)1−αR_{\mathrm{sem}}(\theta)^\alpha R_{\mathrm{red}}(\theta)^{1-\alpha}, where Rsem=∑p+1[U≥θ]R_{\mathrm{sem}}=\sum p^+\mathbf1[U\ge\theta] and Rred=∑p−1[U<θ]R_{\mathrm{red}}=\sum p^-\mathbf1[U<\theta]. SBTA updates the deployed scalar toward that target with bounded steps. Fine-grained evidence determines a shared conversion boundary, not per-position inference thresholds.

Table A10 (Appendix K) confirms the intended selectivity. In Stage 3's highest task-relevance quintile (F5), semantic retention increases from 41.85% to 57.63% for audio and 39.14% to 57.92% for video (+15.78%/+18.78%). For the highest suppressible-redundancy quintile, rejection increases from 83.98% to 86.79% and 85.17% to 87.89% (+2.81%/+2.72%). Corresponding high-relevance firing rates are 54.53%/55.16% with ST-MAT versus 38.67%/36.84% for fixed thresholds; high-redundancy firing rates fall to 13.10%/12.00% versus 15.81%/14.47%. These results demonstrate preferential preservation of discriminative responses and suppression of repetitive spikes. Crucially, neuron-wise threshold adaptation attains 76.73%, below ST-MAT's 78.47% on AVE (main-text Table 3), indicating that coherent population-level selection matters more than threshold granularity alone.

Weakness 1 & Question 2

In Figure 3(d), θ∗\theta^{*} denotes an intermediate target threshold.

Weakness 1 & Question 3

The calibration step uses activations obtained before the weight update, without recomputing a forward pass.

Thank you for the clarifications and additional results. While these address several of my concerns, I will maintain my current rating.

Reviewer_LL5iOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies audio-visual spiking neural networks (AV-SNNs) and argues that recognition accuracy depends not only on the number of emitted spikes but also on which internal neuronal responses are converted into spikes. To this end, it proposes Spatio-Temporal Modality-Adaptive Thresholding (ST-MAT) with three components:

(1) Task-Relevant Spatiotemporal Support Estimation (TRSSE) uses a gradient-times-activation score derived from a classification margin to estimate task relevance and measures spatiotemporal redundancy through pairwise similarity.

(2) Semantic–Redundancy Balancing (SRB) defines a multiplicative objective combining semantic retention and redundancy rejection, and searches for a preferred firing threshold.

(3) Slow Bounded Threshold Adaptation (SBTA) periodically moves modality-/stage-specific neuronal thresholds toward the selected values using bounded updates. Only the learned thresholds are retained at inference.

Critical strengths

The paper raises a potentially meaningful question: how the information content of emitted spikes, rather than firing density alone, should influence AV-SNN performance. It evaluates the method on three audio-visual benchmarks, includes matched fixed-threshold and learnable-threshold controls, and reports meaningful gains over fixed-threshold training on AVE and CREMA-D. The fact that the learned thresholds do not add an explicit auxiliary inference branch is also a practical advantage. These positive aspects, however, do not resolve the fundamental methodological and experimental concerns below.

Critical weaknesses
  • W1. Gradient-based relevance signal reliability is insufficiently justified. The core signal A=U×∂M/∂UA=U\times\partial M/\partial U relies on gradient computation, but SNNs typically use surrogate gradients to handle the non-differentiable spiking process, meaning the gradient signal is not the exact gradient of the original discrete network but an approximation [1][2]. Therefore, interpreting this signal directly as the true task contribution of internal neuronal representations lacks sufficient basis. The paper does not discuss or experimentally verify whether gradient estimation errors affect task relevance ranking and, consequently, the reliability of the entire threshold selection mechanism.
  • W2. Mathematical definitions and theoretical assumptions are inconsistent. The task relevance defined in Equation (2) includes negative values, but subsequent normalization formulas and theoretical proofs (e.g., Proposition 1) assume non-negative weights that sum to 1, which are not necessarily consistent. This undermines the justification for the boundedness of spatiotemporal support and raises further questions about the mathematical validity of subsequent semantic retention, redundancy suppression, and threshold optimization objectives. The authors do not explain how negative gradient contributions are handled or prove that the actual computations satisfy the relevant theoretical assumptions.
  • W3. Training costs may be underestimated, and efficiency claims lack sufficient support. The method involves additional task gradient extraction, pairwise similarity computation across spatiotemporal positions, and periodic threshold search, while SNN gradient backpropagation itself already incurs significant computational and memory overhead [3][4]. In particular, pairwise similarity computation has complexity that grows quadratically with the number of spatiotemporal positions. However, the paper primarily relies on small FLOPs differences to indicate computational efficiency, lacking real end-to-end training time, peak memory, and scaling experiments. Therefore, the authors have not sufficiently demonstrated that the additional gradient analysis and threshold calibration do not introduce significant practical training overhead.
  • W4. Core innovation claims are exaggerated, and the actual mechanism does not fully align with the research motivation. The paper's title and motivation emphasize selecting which internal neuronal representations should be converted into spikes based on task relevance, but the actual mechanism only learns shared thresholds at the modality and stage levels, which cannot distinguish between neuronal responses with the same membrane potential but different task relevance. Therefore, the claimed fine-grained task-relevant representation selection is not directly realized in the actual spike generation process. The authors also do not sufficiently explain why the fine-grained relevance estimation is ultimately only used for global threshold calibration and how this design achieves the claimed selective spike generation mechanism.
  • W5. The true source of performance improvement is unclear (mechanism vs. architecture contributions are confounded). Ablation experiments in the appendix show that adjusting only the Stage 3 threshold can achieve the same performance as the full multi-stage ST-MAT, indicating that the necessity of the complete threshold adaptation mechanism has not been sufficiently validated. Furthermore, the paper primarily conducts experiments based on the authors' designed QK/QK/QKV multimodal architecture, lacking fair validation on other standard SNN backbones. Therefore, it remains unclear whether the performance improvement comes from the proposed ST-MAT mechanism or is influenced by specific architecture, cross-modal attention placement, and training configurations, lacking clear experimental attribution.
  • W6. Theoretical analysis lacks direct connection to the core task objective. The theoretical proofs in the appendix mainly focus on boundedness of support, algebraic consistency, and stability of threshold updates under fixed targets, all relying on specific assumptions, without establishing a clear link between the defined task relevance, redundancy metrics, and final classification performance. These properties only show that certain mathematical constructs hold under given conditions, but are insufficient to prove that the selected thresholds can retain more discriminative information or improve audio-visual fusion performance. Therefore, the existing theoretical analysis is insufficient to support the claimed core mechanism advantages.
  • W7. Figures are unclear, and key implementation details lack transparency. The core architecture diagram is overly crowded, with multiple modules, data flows, and threshold update paths intertwined, making it difficult to clearly understand the actual training and inference processes. In particular, the extraction path of task gradients, the training method of the auxiliary classification branch, and the threshold search process lack intuitive explanation, and relevant visualizations do not provide sufficient quantitative evidence. Therefore, the core mechanism of the paper is difficult to accurately understand and independently reproduce.

References

[1] Neftci E. O., Mostafa H., Zenke F. Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-Based Optimization to Spiking Neural Networks. IEEE Signal Processing Magazine, 36(6): 51–63, 2019. https://doi.org/10.1109/MSP.2019.2931595

[2] Gygax J., Zenke F. Elucidating the Theoretical Underpinnings of Surrogate Gradient Learning in Spiking Neural Networks. Neural Computation, 37(5): 886–925, 2025. https://doi.org/10.1162/neco_a_01752

[3] Yu C., Liu L., Wang G., et al. Advancing Training Efficiency of Deep Spiking Neural Networks through Rate-based Backpropagation. NeurIPS, 2024. https://proceedings.neurips.cc/paper_files/paper/2024/hash/d1bdc488ec18f64177b2275a03984683-Abstract-Conference.html

[4] Jiang H., De Masi G., Xiong H., Gu B. NDOT: Neuronal Dynamics-based Online Training for Spiking Neural Networks. ICML, 2024. https://proceedings.mlr.press/v235/jiang24a.html

Questions for authors
  • Q1. Is the gradient relevance signal reliable? Since SNNs use surrogate gradients (SG) for approximate backpropagation, how do the authors ensure that the gradients accurately reflect the task relevance of internal neuronal representations? Are there noise and bias in gradient estimation? Especially if the ST-MAT threshold adaptation strategy is introduced from the early stages of training, when the network has not yet fully converged, could the gradient signal be unstable, leading to inaccurate task relevance estimation, which in turn affects threshold selection and subsequent spike distribution, potentially resulting in error accumulation? Considering that SG itself has approximation bias and is sensitive to surrogate gradient functions, how do the authors ensure the stability and reliability of the entire adaptive threshold optimization process?
  • Q2. Consistency of mathematical definitions and theoretical assumptions. Why can the computed task relevance contain negative values, while Proposition 1 assumes that normalized weights are non-negative and sum to 1? How do the authors explain this contradiction? Is additional non-negativity processing applied in the actual implementation? If not, how can subsequent theoretical conclusions be guaranteed to hold?
  • Q3. Independent contribution to performance improvement. Appendix Table A3 shows that adjusting the threshold only at Stage 3 can achieve the same performance as the full ST-MAT. Is multi-stage threshold adjustment necessary? Can fair experiments be conducted on other standard SNN backbones to distinguish the performance gains brought by the ST-MAT mechanism from those brought by specific network architectures?
1 comment
AuthorsOpenReview ↗

Theoretical Foundations, Robustness, and Efficiency of ST-MAT

Thank you for your constructive feedback. We revised Figure 2 and Sections 4.1–4.3, with additional proofs (Appendix A), resource data (C), stage ablations (E), auxiliary-head controls (G), calibration tests (H), surrogate gradients (I), and spike-selection diagnostics (K).

Weakness 1 & Question 1

Ak=∣Ukgk∣A_k=|U_kg_k|, with gk=∂M~/∂Ukg_k=\partial\widetilde M/\partial U_k, measures relative first-order classification-margin sensitivity, not an exact derivative. Under Uk→(1−λ)UkU_k\to(1-\lambda)U_k, the signed margin change is λUkgk+O(λ2Uk2)\lambda U_kg_k+O(\lambda^2U_k^2), whereas the magnitude of that change is λAk+O(λ2Uk2)\lambda A_k+O(\lambda^2U_k^2). For ∣ek∣≤εk|e_k|\le\varepsilon_k, ∣A^k−Ak∣≤∣Uk∣εk|\hat A_k-A_k|\le|U_k|\varepsilon_k; sufficient sensitivity gaps preserve ranking. Uniform gradient rescaling preserves SRB utility (Appendix A).

Table A9 shows ST-MAT outperforming fixed thresholds with all five surrogates: Sigmoid 78.47%/73.27%, fast sigmoid 73.76%/72.77%, triangular 77.23%/73.51%, tanh 75.37%/71.89%, and erf 75.62%/71.78% (ST-MAT/fixed); see Figures A1–A4. Table A8 yields 78.47% with 64 samples every five epochs versus 75.74% every epoch and 77.72% every ten epochs. All eight settings exceed 73.27%, supporting periodic calibration.

Weakness 2 & Question 2

We define the nonnegative task evidence as follows:

ck=Ukgk,ck+=max⁡(ck,0),ck−=max⁡(−ck,0),Ak=∣ck∣=ck++ck−≥0.c _k = U _k g _k,\quad c _k^{+} = \max(c _k,0),\quad c _k^{-} = \max(-c _k,0),\quad A _k = |c _k| = c _k^{+}+c _k^{-} \ge 0.

For nonzero total evidence, the normalized quantities are defined by:

w _i = \frac{\sum _d A _{i,d}}{S _A},\qquad \pi _{i,d} = \frac{A _{i,d}}{\sum _d A _{i,d}},\qquad p _{i,d}^{+} = \frac{A _{i,d}}{S _A}.$$ These quantities are nonnegative and normalized on their respective supports. Task-weighted descriptors yield: $$C _{ij} = |z _i^{\top}z _j|,\qquad R _{\mathrm{ST}} = \left\lVert \sum _i w _i z _i z _i^{\top} \right\rVert _F^2,\qquad 0 \le s _{\mathrm{ST}} \le 1.$$ The bound on the support measure follows from capacity normalization. Redundancy exposure is defined by: $$\xi _i = \sum _j w _j C _{ij}^2,\qquad p _{i,d}^{-} \propto \xi _i (1-\pi _{i,d}).$$ Accordingly, semantic retention and redundancy rejection are given by: $$R _{\mathrm{sem}} = \sum p^{+}\mathbf{1}[U\ge\theta],\qquad R _{\mathrm{red}} = \sum p^{-}\mathbf{1}[U\lt\theta].$$ Both measures lie in the interval [0, 1]. See Appendix A for proofs and Table A10 for diagnostics. ## Weakness 3 Calibration similarity costs $O(B_c(TN_l)^2D_l)$; $T=6$ and a $32\times32$ early token grid yield **6,144** spatiotemporal positions. It runs on **64 samples**, batch **2**, every **5 epochs**. Over 150 epochs, ST-MAT uses **89.22 min/51.27 G** peak memory versus **73.22 min/27.42 G** for fixed thresholds (Table A1b). Inference remains **32.61 G FLOPs, 1.71 G memory, 1.35 s latency, and 16.35 M parameters**; AVE accuracy improves **5.20%** without inference overhead. ## Weakness 4 SRB determines **one scalar cutoff per modality–stage pair**, without token-wise inference parameters. At Stage 3, highest-relevance F5 audio/visual firing rates are **54.53%/55.16%** versus fixed **38.67%/36.84%**; highest-redundancy F5 rates are **13.10%/12.00%** versus **15.81%/14.47%**. F5 semantic retention improves **15.78/18.78%**, and redundancy rejection **2.81/2.72%** (Appendix K). Shared ST-MAT also exceeds neuron-wise thresholds (**78.47% vs 76.73%** on AVE). Selection follows population task evidence rather than token-wise gates. ## Weakness 5 & Question 3 On matched AVE backbones (Table 2), ST-MAT exceeds fixed thresholds by **2.488/0.995/2.737%** for ResNet/SEW-ResNet/CNN. Full-stage and Stage-3-only adaptation both reach **78.47%** on AVE; however, full adaptation improves Stage 1/2 audio probes to **54.48%/56.67%** versus **49.25%/54.23%** (Table A4). CREMA-D full/Stage-3-only/fixed accuracies are **79.44%/77.02%/76.21%** (Table A5). Thus, gains generalize across backbones, while early adaptation improves unimodal features. ## Weakness 6 Appendix A proves ranking, Gram-support, candidate-selection, and SBTA stability. If $\Gamma > 2\beta_{\ell}$, the optimum is stable; otherwise regret is at most $2\beta_{\ell}$. For two target thresholds applied from the same current deployed threshold, the following bound holds: $|\hat{\theta}_{q+1}-\theta_{q+1}|\leq\eta_{\theta}|\hat{\theta}_{q}^{\ast}-\theta_{q}^{\ast}|$. ## Weakness 7 We have optimized Figure 2 of the paper, highlighting the core mechanism of ST-MAT. A more detailed explanation is provided in the main text under the methods section, and the quantitative experiments' analysis is conducted in Appendix K (Table A10). Training-only auxiliary heads use $L_{\mathrm{task}}=L_f+0.125L_a+0.125L_v$. With auxiliary heads, fixed thresholds improve **72.28%→73.27%**, while ST-MAT reaches **78.47%** (Table A7), isolating the effect of task-informed selection.
Reviewer_wM8ZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets the continuous-to-discrete spike conversion in audio-visual spiking neural networks (AV-SNNs). The authors argue that simply modulating spike firing rates is insufficient; instead, the network should selectively convert internal membrane potentials into spikes based on their task relevance and redundancy. To achieve this, the paper introduces Spatio-Temporal Modality-Adaptive Thresholding (ST-MAT), which consists of three parts: Task-Relevant Spatiotemporal Support Estimation (TRSSE) to evaluate decision-related support, Semantic–Redundancy Balancing (SRB) to trade off semantic retention against redundancy suppression, and Slow Bounded Threshold Adaptation (SBTA) to update thresholds steadily during training. The learned thresholds are fixed at deployment, adding no inference overhead. Experiments on AVE, CREMA-D, and UrbanSound8K-AV demonstrate clear improvements over fixed and gradient-learned threshold baselines.

Critical strengths
  1. Rather than focusing purely on post-spike cross-attention mechanisms, the paper examines the continuous-to-discrete thresholding boundary. The observation that spike count alone does not determine classification performance is reasonable and well-supported by Figure 1.
  2. Because threshold adaptation takes place periodically during training and thresholds are fixed at deployment, the deployed model incurs no additional parameter count, memory footprint, or FLOPs compared to a standard fixed-threshold SNN.
  3. The firing-controlled baseline in Table 2 and Table A2b is informative. By constraining a baseline SNN to match the firing rate of ST-MAT, the experiment effectively shows that the observed performance gains come from selecting discriminative features rather than simply changing overall spike volume.
Critical weaknesses
  1. In Eq. (1) and Eq. (6), the threshold is written as θm,l\theta_{m,l}, which appears to be a single scalar per stage and modality. If θm,l\theta_{m,l} is indeed a scalar, it is difficult to see how adjusting one global threshold selectively discards redundant spatial tokens without uniformly shifting all responses in that layer. The authors should clarify whether θ\theta is a scalar, channel-wise, or spatial-wise parameter, and explain the mechanism behind this selectivity.
  2. In Section 4.1, a "lightweight modality-specific branch readout produces class logits fm,bf_{m,b}" to compute the classification margin Mm,bM_{m,b}. The manuscript does not specify how these branch heads are trained. If they are trained with dedicated classification losses, part of the accuracy improvement might stem from auxiliary supervision / deep supervision rather than threshold adaptation itself.
  3. Table A3 shows that adapting Stage 3 alone reaches 78.47% test accuracy, matching the 78.47% achieved by adapting all three stages. Meanwhile, computing the pairwise similarity matrix CijC_{ij} in TRSSE requires O(Bc(T⋅Nl)2Dl)O(B_c (T \cdot N_l)^2 D_l) operations. In Stage 1 (32×3232 \times 32 tokens, T=6T=6), T⋅Nl=6144T \cdot N_l = 6144, which leads to a relatively large matrix. Given that early-stage adaptation does not appear to provide additional gains over Stage 3 alone, the practical necessity of running TRSSE across earlier stages remains questionable.
Questions for authors
  1. What is the exact tensor dimension of θm,l\theta_{m,l}? If it is a scalar, how does a single threshold separate redundant representations from non-redundant ones when they share similar membrane magnitudes?
  2. Are the modality-specific branch readouts in Section 4.1 trained with separate task losses? If so, did the authors test a baseline that includes these auxiliary heads but uses standard fixed thresholds to isolate the effect of auxiliary supervision?
  3. In Table A3, why does adapting Stage 3 alone achieve the exact same test accuracy (78.47%) as adapting all three stages? Does this imply that adapting Stages 1 and 2 is largely redundant for the final classification result?
  4. The setup uses 64 stratified samples with batch size 2 every 5 epochs. How sensitive is the model to the size of this calibration set (e.g., 16 vs. 64 vs. 128 samples) and the update frequency (e.g., updating every epoch vs. every 10 epochs)?
1 comment
AuthorsOpenReview ↗

Task-Guided Threshold Selection and Stage-Wise Adaptation

Thank you for your constructive feedback. We revised Figure 2 and Sections 4.1–4.3 to clarify ST-MAT; added cross-backbone and neuron-wise comparisons (main-text Tables 2–3); expanded theoretical analysis (Appendix A), resource measurements (Appendix C), stage ablations (Appendix E), auxiliary-head controls (Appendix G), calibration sensitivity (Appendix H), surrogate-gradient tests (Appendix I), and task-informed spike selection (Appendix K).

Weakness 1 & Question 1

ST-MAT deploys one scalar threshold per modality–stage pair, θm,l\theta_{m,l}, shared over time, tokens, and channels; spikes follow S=H(U−θm,l)S=H(U-\theta_{m,l}). It does not use token-wise inference gates. During calibration, TRSSE estimates fine-grained classification-margin sensitivity, Ai,d=∣Ui,d∂M/∂Ui,d∣A_{i,d}=|U_{i,d}\partial M/\partial U_{i,d}|, and characterizes effective task-relevant spatiotemporal support. SRB constructs normalized semantic mass p+p^+ and suppressible-redundancy mass p−p^- from this evidence and searches observed candidate thresholds to maximize J(θ)=Rsem(θ)αRred(θ)1−αJ(\theta)=R_{\mathrm{sem}}(\theta)^\alpha R_{\mathrm{red}}(\theta)^{1-\alpha}, where Rsem=∑p+1[U≥θ]R_{\mathrm{sem}}=\sum p^+\mathbf{1}[U\ge\theta] and Rred=∑p−1[U<θ]R_{\mathrm{red}}=\sum p^-\mathbf{1}[U<\theta]. SBTA then updates the shared threshold with bounded steps.

Table A10 (Appendix K) groups responses into ascending quintiles F1–F5. At Stage 3, F5 semantic retention rises from 41.85% to 57.63% (audio) and 39.14% to 57.92% (visual), gains of 15.78/18.78%. F5 redundancy rejection rises from 83.98% to 86.79% and 85.17% to 87.89% (+2.81/+2.72%). Thus, a shared task-informed cutoff preferentially preserves discriminative responses while rejecting redundancy. Neuron-wise threshold regulation reaches 76.73% on AVE versus 78.47% for ST-MAT (main-text Table 3): finer threshold granularity alone does not replace population-level task-aware selection.

Weakness 2 & Question 2

Modality-specific auxiliary heads provide the logits for margin sensitivity during training and are removed for inference. The task loss is Ltask=Lf+0.125La+0.125LvL_{\mathrm{task}}=L_f+0.125L_a+0.125L_v. Table A7 (Appendix G) isolates their effect: fixed thresholds without/with auxiliary heads yield 72.28%/73.27% on AVE (+0.99%); using ST-MAT with the same heads yields 78.47% (+5.20%). The primary gain therefore comes from using task evidence to select the spike-conversion boundary, not from auxiliary supervision alone.

Weakness 3 & Question 3

On AVE, Stage-3-only and full-stage adaptation both achieve 78.47%, identifying late fusion as a critical control point. Nevertheless, frozen-feature probes (Table A4, Appendix E) show stronger early audio representations with full adaptation: 54.48% vs 49.25% at Stage 1 and 56.67% vs 54.23% at Stage 2. The corresponding high-relevance F5 semantic-retention advantages are 6.86/5.40%. On CREMA-D (Table A5), full-stage ST-MAT achieves 79.44%, versus 77.02% for Stage-3-only and 76.21% for fixed thresholds. Thus, early-stage adaptation improves unimodal representations, while the benefit to final accuracy depends on the dataset and fusion stage.

Question 4

Table A8 (Appendix H) shows that all eight settings exceed the 73.27% fixed-threshold baseline. With a five-epoch interval, 32 samples yield 78.22% and 64 yield 78.47%; using 64 samples, updating every 1/5/10 epochs yields 75.74%/78.47%/77.72%. These results support compact, class-informative calibration sets and a slower, periodic threshold-update schedule rather than recalibration after every mini-batch.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.