- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Neural Events, a discrete asynchronous representation that reduces the rate of an event stream by emitting a new token only when its assigned code changes. A patch-local RWKV-7 encoder processes incoming events and maps its outputs to a learned codebook through straight-through Gumbel-Softmax quantization. The encoder is pretrained using time-surface reconstruction together with rate-alignment and latent-straightening losses that encourage stable code assignments. The authors evaluate the representation with synchronous Swin-based and asynchronous DAGr-based task networks. Experiments cover object detection on DSEC-Detection, Gen1, and N-Caltech101, classification on N-Caltech101, several controlled ablations, and hardware measurements. The results show useful detection gains and competitive classification accuracy, with a reported twofold event-rate reduction.
Critical strengths
- The method offers a coherent approach to reducing redundant event updates while retaining a useful learned representation. Patch-local recurrence integrates temporal history, the codebook provides a compact discrete output, and change-triggered emission suppresses repeated code assignments. The reconstruction and temporal regularization objectives are aligned with this goal. This is a meaningful design for the interface between high-rate event sensors and downstream perception networks.
- The evaluation contains substantive evidence for the proposed representation and emission rule. The paper tests both real driving data and N-Caltech101, uses synchronous and asynchronous downstream networks, and includes ablations and hardware measurements. The controlled Gen1 comparison is particularly useful. At the same retained-event fraction of 0.5, code-change emission achieves 49.9 mAP, compared with 38.3 for random removal of learned tokens and 43.7 for pairwise feature averaging. Retaining every quantized code reaches 51.1 mAP, so the proposed rule removes half the outputs with a relatively small accuracy loss in this setting. The main detection results also improve over the reported baselines, including DAGr on DSEC and EVA-L on Gen1. Together, these results support the usefulness of the design in the tested pipelines.
Critical weaknesses
- The main concern is that the connection between the compressed stream and the downstream input is not specified clearly enough to reproduce the synchronous downstream evaluation from the retained tokens. In Section 3.4, the pretraining feature map is constructed by averaging code vectors according to their frequencies in the full encoded sequence. Section 4.1 then describes the synchronous task network as starting from this feature map. However, the retained neural-event stream keeps only code changes. The manuscript does not state whether the counts of discarded events are maintained internally at the encoder, reconstructed from the retained stream, or omitted when constructing the downstream input. Change timestamps alone do not generally recover those event counts. A different aggregation for downstream evaluation could be valid, but its definition and information requirements need to be explicit. Without this clarification, it is difficult to determine whether the reported synchronous results evaluate the claimed compressed-only deployment. The authors should report the exact online aggregation rule and, if it uses statistics from the full encoded sequence, compare it with an implementation that uses only the retained neural-event stream.
- A second concern is that the computational and hardware results do not yet establish the complete efficiency advantage under a consistent operating protocol. The reported seventeen-fold advantage over EVA-L concerns representation construction, using 0.14 versus 2.32 MFLOPs per event. Table 2 also includes full-network costs, with 1,130 MFLOPs per event for TokSwinT-S on Gen1. However, Sections 4 account for synchronous inference by running the full backbone for every incoming event or retained token, whereas Table 6 reports execution on event packets. The paper does not explain how the prediction schedule in the packet measurements relates to the per-event accounting. A comparison at a common output frequency, event window, and hardware configuration is needed to establish practical speed and energy benefits.
Questions for authors
- How is the synchronous downstream feature map constructed at deployment? Does it use code frequencies from the full encoded sequence, only the retained neural events, or additional statistics? Please give the exact rule and report the downstream performance when the task network receives only the information carried by the stated neural-event stream.
- How do the per-event FLOP and energy figures relate to the prediction schedule used in Table 6? Please compare complete pipelines on the same hardware and at a common output frequency, including encoding, task inference, actual energy, and peak memory. In particular, what does the 281 kilobyte encoder-memory allocation include, and does it cover all persistent patch states?
- Can the authors report performance together with the achieved event and byte rates at several compression levels across the evaluated datasets, with repeated-run uncertainty? This would clarify whether the reported twofold reduction and its accuracy cost are stable beyond the single operating point shown in the main ablation.
5 comments
Q2. How do the per-event FLOP and energy figures relate to the prediction schedule used in Table 6? Please compare complete pipelines on the same hardware and at a common output frequency, including encoding, task inference, actual energy, and peak memory. In particular, what does the 281 kilobyte encoder-memory allocation include, and does it cover all persistent patch states?:
We refer to our answer to W1, Q3 by reviewer DAx8 and provide a summary here. To complete Tab. 6 and address the reviewer's question, we compare Neural Events, EVA (ICLR 2026), and Event2vec (ICML 2026) models on the same GH200 Nvidia GPU.
We test these methods in two settings: In the event-by-event setting, individual events are inserted into the encoder and then passed through the task network, updating only the affected activations (Tabs. 1 and 2). In the synchronous setting, we perform a complete forward pass with 100,000 events (Tab. 6). Here, the network requires a complete forward pass to produce an updated prediction. Tab. 6-a, in particular, measures execution time under this setting. Tab 6-b reports the memory (281 KB) to store the encoder weights. This convention explains the higher per-event costs of the synchronous baselines. This constitutes the primary computation comparison with other baselines.
Additionally, in the table below, we report the following quantities. (1–2) Memory required to store the encoder and decoder weights.
(3) Memory required for persistent encoder states or retained event history.
(4–5) Encoder steady-state and peak GPU allocations: memory remaining allocated between updates and the maximum allocation reached during processing, respectively.
(6–7) Decoder steady-state and peak GPU allocations: memory remaining allocated between inference calls and the maximum allocation reached during inference, respectively.
(8–9) Full pipeline steady-state allocations and component peaks.
(10) Encoder throughput with sequential, event-by-event processing.
(11) Encoder throughput with batch processing using parallel-scan.
(12–15) Latencies of the encoder, Swin backbone, YOLOX head with postprocessing, and complete pipeline, respectively. Latencies are expressed in milliseconds, with mean ± standard deviation where reported. Decoder inference starts from the complete input representation.
(16) Gen1 detection accuracy, taken from Table 2.
All measurements are performed using FP32 on an NVIDIA GH200 GPU. Memory is reported in decimal MB. All latencies are in milliseconds, with one downstream prediction per packet. Both throughput measurements refer to the encoder. Gen1 detection accuracies are taken from Table 2. The synchronous decoder operates on a fixed-size representation. Its latency is measured per prediction, independently of the number of raw events used to construct that representation.
We see that neural event-based detection has a complete-pipeline latency of 36.70 ms, compared with 282.71 ms for EVA and 79.20 ms for Event2Vec, corresponding to approximately 7.7× and 2.16× speedups. Table 2 separately reports their Gen1 detection accuracies. Tables 1–2 report MAC-based calculated energy estimates using reference operation costs from Jouppi et al. (2021). These provide a comparison under a common energy model; hardware energy consumption was not measured directly.
| Metric | Neural Events | EVA | Event2Vec |
|---|---|---|---|
| Encoder weights (MB) | 0.281 | 3.487 | 0.288 |
| Decoder weights (MB) | 364.29 | 364.29 | 360.32 |
| Persistent state (MB) | 123.74 | 1,345.51 | 3.20 |
| Encoder steady-state allocation (MB) | 158.94 | 1,386.04 | 39.28 |
| Encoder allocated peak (MB) | 158.99 | 1,386.62 | 613.78 |
| Decoder steady-state allocation (MB) | 546.45 | 399.68 | 394.96 |
| Decoder allocated peak (MB) | 719.51 | 722.16 | 428.59 |
| Full pipeline steady-state allocations (MB) | 705.39 | 1,785.72 | 434.24 |
| Full pipeline peak allocations (MB) | 878.50 | 2,108.78 | 1,042.37 |
| — | |||
| Encoder throughput with event-by-event processing (events/s) | 313 | 55 | 24 |
| Encoder throughput with batch-processing (events/s) | 9.50 x 1e6 | 0.39 x 1e6 | 1.80 x 1e6 |
| — | |||
| Encoder (ms) (mean ± SD) | 10.53 ± 0.52 | 255.57 ± 2.12 | 55.89 ± 2.47 |
| Swin decoder (ms) | 16.80 ± 3.34 | 17.17 ± 0.69 | 13.67 ± 3.08 |
| YOLOX head + postprocessing (ms) | 9.36 ± 0.15 | 9.97 ± 0.11 | 9.64 ± 1.24 |
| Complete pipeline latency (ms) | 36.70 | 282.71 | 79.20 |
| — | |||
| Detection on Gen1 (mAP) | 49.9 | 47.7 | 42.5 |
Thank you for your detailed rebuttal and additional experiments. I appreciate the effort invested in addressing my concerns. The clarification of the compressed-stream interface and the new hardware measurements substantially improve the paper. In particular, the revised memory accounting and same-hardware comparisons provide useful evidence of computational efficiency. However, some concerns remain. The claimed storage reduction has not been sufficiently validated through actual serialized data, energy consumption is still estimated rather than directly measured, and the accuracy–compression trade-off across different operating points remains insufficiently explored. While the additional results strengthen the practical motivation, I do not find the remaining evidence sufficient to justify increasing my rating. I will therefore maintain my original score of Weak Rejection.
We thank reviewer ff4D for the feedback on our submission. We answer in the following to each of the weaknesses and question raised:
W1/Q1. The main concern is that the connection between the compressed stream and the downstream input is not specified clearly enough to reproduce the synchronous downstream evaluation from the retained tokens.... How is the synchronous downstream feature map constructed at deployment? Does it use code frequencies from the full encoded sequence, only the retained neural events, or additional statistics? Please give the exact rule and report the downstream performance when the task network receives only the information carried by the stated neural-event stream.
We refer the reviewer to our answer to Q2 from reviewer DAx8 and summarize the response here. We build the synchronous feature map from Neural Events, retaining only flips. For example, if one patch generates neural event sequence q1,q1,q1,q2,q2,q3, it would only retain flipped events q1,q2,q3 (first occurrences). During pretraining of the encoder, we retain all codes, and thus generate feature maps H with entries at patch j equal to hj=63q1+2q2+q3, following the example. This process is repeated for each patch. During training of the downstream task-network and during deployment, we instead generate feature maps H~ with entries h~j=3q1+q2+q3 , following the example above. The embedding h~j can be constructed in a streaming fashion by only transmitting the timestamp, patch index, and code index of the flipped neural event. To do so, we can hold a rolling buffer of neural events and then compute the above per-patch sum in batch. Alternatively, we can compute it event-by-event by maintaining the number of neural events and their rolling buffer sum at each patch. For each new neural event, we then subtract the code of the neural event leaving the buffer, and add the code of the neural event entering the buffer. To compute h~j, we then divide the sum by the neural event count.
Tab.1-3 all report accuracies in this setting. We study the impact of this dropping mechanism in Tab. 4-c by training a baseline that does not drop neural events. In the example, this baseline would record hj instead of h~j at that patch. This yields slightly higher mAP (last row, achieving 51.1 mAP) than dropping (code-change emission row, achieving 49.9 mAP). To recover the higher performance, one could cheaply transmit the neural event run length (i.e. number of repeated neural events) in addition to the neural event, allowing reconstruction of the full weighted sum.