acceptodds
Under review as a conference paper at ICLR 2027

Optical-Flow-Free Two-Stream Spiking Networks with Cross-Dimensional Fusion for Action Recognition

Abstract

Spiking neural networks (SNNs) model temporal evolution through recurrent membrane dynamics, while action recognition also depends on clip-level context that varies more slowly than frame-to-frame motion. We propose CF-TS-SNN, an optical-flow-free two-stream SNN for RGB and event-based action recognition. The temporal pathway processes chronologically sampled frames or event windows, whereas the spatial pathway repeatedly encodes a representative clip summary using a structurally matched spiking backbone. A controlled routing study identifies additive interaction after the second backbone stage, followed by final fusion after the third stage, as the most effective configuration. Building on this routing, cross-dimensional fusion derives temporal, channel, and spatial weights from the final spatial feature to modulate the temporal representation. The same spatial feature also produces a threshold map from attention, activity, and normalized spatial entropy, enabling clip-level context to regulate spike generation in the final temporal stage. At 20 time steps, CF-TS-SNN achieves 98.15% accuracy on DVS and 94.05% on UCF-50, improving the temporal-only baseline by 1.25 and 7.75 percentage points, respectively.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Spiking Neural Networks, Action Recognition, Two-Stream Networks, Cross-Dimensional Fusion, Optical-Flow-Free Video Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_87r9OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces CF-TS-SNN, an optical-flow-free two-stream spiking network for action recognition. The temporal pathway processes chronologically sampled video frames or event windows, while the spatial pathway repeatedly encodes a center frame or a full-clip event summary. The pathways interact through intermediate addition and final fusion. Spatial features also generate temporal, channel, and spatial attention weights and regulate neuronal thresholds in the final temporal stage. At 20 time steps, the model achieves 98.15% accuracy on DVS128Gesture and 94.05% on UCF-50. The paper provides fusion-operator comparisons and component ablations, but the experimental protocol, external baseline comparisons, and computational cost analysis remain insufficient to fully establish its contribution.

Critical strengths

The paper presents a clear two-stream design motivated by the complementarity of persistent spatial context and dynamic temporal information, and evaluates the same framework on both event and RGB inputs. Structurally matched spiking backbones facilitate interactions between corresponding stages, and the method avoids additional optical-flow preprocessing. Experiments compare fusion operators, preservation of an independent spatial output, and components of attention and threshold regulation, providing initial support for the design choices. The complete model improves over the temporal-only baseline by 1.25 and 7.75 percentage points on the two datasets, respectively. The larger UCF-50 improvement suggests that spatial context can provide useful complementary information for temporal representations.

Critical weaknesses

Comment 1: The experimental protocol and implementation details are insufficient for reproduction and reliable comparison. Section 4.1 mainly specifies the datasets, time-step count, training hardware, and use of identical settings across variants. It does not clearly specify training, validation, and test splits, optimizer, learning rate, training epochs, batch size, or augmentation. The splitting unit and clip-sampling procedure for UCF-50 are particularly important for assessing whether correlated clips from the same source video appear across splits. The attention mappings, threshold bounds, and basic neuronal parameters also require specification. Stating that settings are identical does not replace a complete experimental protocol.

Comment 2: Comparisons with related methods are limited and do not sufficiently establish competitiveness. The UCF-50 experiments primarily compare internal variants, without direct comparisons against SVFormer, ReSpike, or representative optical-flow-free video methods discussed in the related work. Table 6 on DVS128Gesture also mainly includes earlier methods and has limited coverage. The results establish improvements over the paper's own baselines, but do not adequately establish advantages over related approaches. Key external baselines should be added, with differences in pretraining, input length, and testing strategy explicitly documented.

Comment 3: Single-stream and two-stream comparisons lack matched capacity and computational budgets. The two pathways have independent parameters, so the two-stream model increases both representation capacity and computation relative to the temporal-only baseline. The current comparison does not exclude additional network capacity as a partial explanation for the gains. Although comparisons between simple two-stream fusion and the complete model support the fusion module, a single-stream baseline with matched parameter count or computation would strengthen the analysis. Parameter counts, operation counts, inference latency, and firing rates should also be reported to assess the cost of repeatedly encoding spatial inputs and processing both pathways.

Comment 4: The interpretation and necessity of threshold regulation need stronger evidence. Equation (19) combines activity, spatial entropy, and attention into a threshold multiplier, but the signs and relative scales of these terms are primarily justified through intuition. Removing individual terms supports their empirical usefulness, but does not distinguish context-dependent regulation from general threshold adjustment. Comparisons against tuned fixed thresholds, learnable thresholds, and shuffled spatial guidance maps would help isolate the proposed mechanism. Analysis of threshold distributions, clipping frequency, and firing-rate changes would also be useful. The paper should additionally explain how the threshold map generated from the final spatial feature is mapped to neuronal dimensions across layers within temporal Block 3.

Questions for authors

See Weaknesses.

Reviewer_3C93OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a two-stream spiking network for action recognition that needs no optical flow.

  • Temporal stream: 20 chronologically sampled frames (RGB) or event windows (DVS).
  • Spatial stream: a "clip summary" repeated for all 20 time steps. For RGB this is the center frame; for DVS, all events in the clip accumulated into one image.
  • How the spatial stream steers the temporal one: its Block-2 features are added into the temporal stream; its final features produce attention weights over time, channels and space that rescale the temporal output (cross-dimensional fusion, CDF); the same features set the firing thresholds inside the temporal stream's last block (threshold regulation, TR).
Critical strengths

The novelty is mainly the combination. Two-stream key-frame designs, CBAM and attention SNNs, and adaptive thresholds all exist. What's new is generating the attention and the thresholds from a matched spatial SNN stream.

Critical weaknesses
  1. Major concern: The benchmarks are narrow, and there's no baseline comparison on RGB at all. Only DVS128 Gesture (near-saturated) and UCF-50 are used, with no UCF-101, HMDB-51, Kinetics or SSv2. UCF-50 has no comparison table with any other method, spiking or not. The closest prior work isn't compared. ReSpike and SVFormer are discussed but can't be compared, because the paper uses none of their benchmarks. Table 6 lists only 2017–2021 methods. Spikformer, which the authors cite, reports about 98.2–98.3% on DVS128 at T = 16, compared with this paper's 98.15% at T = 20.

  2. Major concern: Limited technical contribution.

  • The UCF-50 gain isn't attributed to the cross-stream design. The 5-point CDF gain could come from adding CBAM-style attention at all, not from computing it from the spatial stream. No control computes the same weights from the temporal stream itself, or adds attention to the temporal-only baseline.
  • UCF-50 is largely solvable from one frame. The center frame alone gets 85.2%, close to the 86.3% of the 20-frame temporal stream. That says little about modeling motion.
  1. Minor concern: No efficiency numbers despite the SNN motivation. No parameter counts, FLOPs, synaptic operations (SOPs) or energy.

  2. Minor concern: Missing citations. The UCF-50 dataset (Reddy & Shah); BNTT (Kim & Panda), which is used in the backbone; adaptive-threshold spiking neurons such as ALIF or BDETT.

Questions for authors

Refer to weaknesses.

Reviewer_JQNuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose an engineered spikeresnet type architecture for action recognition that maintains a spatial channel and a temporal channel, striving to independently process (and then recombine) spatial object/scene information and motion information, without explicit optical flow computation. Targeted towards both event-based inputs and RGB inputs, the authors propose a specific architecture which has cross-interaction before the two pathways fuse later on, identifying that as the most performant configuration. Experiments on DVSgesture and UCF-50 demonstrate its effectiveness.

Critical strengths
  • The overall idea of separating the information types is valid, although has been also used by multiple works as also acknowledged by the authors.
  • The authors get into the complexity of separation + recombination of the two information types throughout a custom architecture via detailed ablations, although one does wonder if there are data leakage risks.
Critical weaknesses

There are significant issues with this work, both in presentation and substance:

  • The writing is not good. Parts that correspond to describing the method, either in the detailed sections or the overall summary descriptions, are both hard to parse. For instance, the first explanation of the two streams itself is confusing: what does it mean to say that the spatial stream "repeates a center RGB frame"? What is "full-clip event summary" exactly? The construction of the architecture and the fusion itself seems very ad-hoc and dense terminology is outlined without proper motivation or buildup. Proper background is also not provided, both on the spikeresnet architecture and the bntt normalization, and also in general on the structure of the data in these event based and non-event based sensors.
  • The design overview should not contain just a bunch of formulas but also the high-level overview of the approach and the reasoning behind the different elements. At that point things like "cross-dimensional weight generation", "threshold regulation", "adaptive threshold", "cross-dimensional weighting" are all buzzwords and have no meaning to the reader.
  • The fusion part is also not well presented. It is not clear at all how the "basic" operators come into relevant within this architecture, as those four functions are not referenced/mentioned anywhere else in the method. Upon further reading it occured to me that this is just part of an ablation study further down in the experiments, and if that is the case then only the standard option should be reported in the method section and the other alternatives be discussed in the ablation. Also, the terminology has a lot of missing definitions/connective tissues, like F_m is not defined, it is not clear what B_t is, and it is not clear what Fe and Fl represent. Overall, after careful reading what becomes clear is that the method has multiple ad-hoc elemenets w.r.t its architectural choices. It is not clear what C_l exactly represents here.
  • It is also not clearly communicated what importance the cross-dimensional spatial-to-temporal fusion has. g_T, g_C, g_S all are like black box functions and it is not clear from the writing what they represent, the operations in (10-12) seem very arbitrarily set with the choice of average and max and those functions.
  • I mean, the broad idea is quite clear here, that we want to separate spatial information, which is object/scene driven from motion information, and infer with both disentangled separately. As the authors note, this has been investigated multiple times in previous works, and not all of them are optical flow based as well, but ones which avoid explicit flow combination. But from their approach, it is very hard to see how their proposed method is different from all those approaches and why exactly it should do better than them. The method itself has many components with many structural hyperparameters that are just set without proper build-up.
  • Lastly, the experiments themselves only encompass two datasets, and while on the DVS I do see comparison with other approaches, the latest approach is from 2021, If I'm not wrong there should be many other spiking NN based approaches proposed for the DVS gesture event-based dataset since then, and w.r.t the UCF-50 dataset I did not see any comparison beyond architectural ablations unless I missed something.
  • Overall, what bothers me is the below par presentation of the ideaas and discussions of the method on its own and in the context of the wealth of literature that exists in this area, plus the many different elements of the approach, both architectural and functional that make it appear really ad-hoc plus the fact that the experiments only cover two datasets, one for event-based and one for the RGB one. Also, if the authors did use elaborate ablations on any of the datasets to arrive at their "best" configuration, the risk of data leakage needs to be addressed.
Questions for authors

Please see weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.