- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the performance gap between layer-local three-factor learning and backpropagation for event-based dense prediction. The authors argue that spatially averaging the modulatory signal discards the different updates required at different positions. They formulate block averaging as an orthogonal projection and introduce per-pixel modulated three-factor learning (PPM-3F), where each layer obtains a spatially resolved error signal from its own dense readout. Controlled experiments on DDD17 and DSEC-Semantic show that finer modulation improves segmentation accuracy. At the default three-layer setting, PPM-3F reaches 40.1% mIoU on DDD17, compared with 30.5% for uniform modulation and 42.8% for same-architecture BPTT. The authors also study boundary-related errors, block-level modulation, and an additional local rule, and provide post-synthesis estimates for an ASIC implementation.
Critical strengths
-
Understanding the limitations of local learning is important for training SNNs on neuromorphic chips. The paper identifies spatial modulation granularity as a useful design variable for dense prediction and provides an explanation of why spatially collapsed learning signals can lose task-relevant information.
-
The experiments include several useful controls. In particular, keeping the dense readout fixed while changing the backward averaging operator makes the granularity comparison easier to interpret. The frozen-backbone, label-aligned partition, and random-feedback experiments help clarify where the improvement comes from. The main granularity ordering is also reported across three seeds on both segmentation datasets.
-
The paper distinguishes the projection properties established by the analysis from the accuracy ordering measured in the experiments.
Critical weaknesses
-
Writing and organization of this paper are weak. The paper appears to be a crude patchwork of individual points, lacking synthesis and a clear overarching narrative.
-
If I understand Algorithm 1 correctly, the network accumulates spike rates across timesteps and then computes the local readout losses and weight updates. Section 4 applies stop-gradient operations across layers and timesteps, but this alone does not specify how the local derivatives required after rate accumulation are stored or accumulated. The authors should explain the streaming update schedule and report the peak live state, including local derivative or eligibility buffers, BN, readouts, and AdamW moments. This matters because the memory constraint is one of the main reasons for choosing this learning rule.
-
The claim that modulation granularity accounts for most of the gap to backpropagation should remain tied to the particular experimental decomposition. The main contribution appears to be the diagnosis and measurement of the averaging problem, since the authors acknowledge the connection between PPM-3F and existing dense local heads. For this contribution, it is important to distinguish the effects of spatial granularity, cross-layer credit, and temporal credit. In Table 2, the final step changes both the use of multiple local objectives and credit assignment through time. The alternative ordering and deeper-network experiments support the qualitative finding, but the 78% share is still conditional on the chosen path and operating point. Could the authors separate the objective change from temporal credit, or explicitly limit the interpretation of this percentage? The wider PPM–BPTT gap at five and eight layers in Table 7 also deserves emphasis when discussing how closely local learning approaches BPTT.
Questions for authors
-
What are the total training-memory and computational costs of uniform, block-level, and per-pixel modulation under the same implementation and precision? These measurements would show the practical cost of recovering the additional accuracy, beyond the reduction in modulation-state storage.
-
For the ASIC–GPU comparison in Table 5, how is GPU energy measured, and which operations and system components are included on each side? In particular, the paper should clarify the treatment of preprocessing, readout/loss computation, optimizer updates, and idle power. The authors already identify the ASIC results as post-synthesis estimates. A more explicit measurement protocol would help assess the quantitative energy-efficiency ratios.
6 comments
Response to Reviewer Ugbg (2/4)
Response to W2: The local derivatives are not stored as weight-shaped traces. The chip keeps two factors of each derivative per pixel in a 64-pixel eligibility buffer and accumulates them into the weight gradient as soon as the pixel's modulation returns. Our ASIC runs the whole learning step on chip, and its schedule and storage show that the rule meets the memory constraint: the training step of the three-layer network fits in 2.52 MB of SRAM and 0.75 MB of registers.
The chip is built around the structure of PPM-3F. Its layer engine has a forward unit, a backward unit, and a bank of weight-gradient accumulators for each of the 64 output channels, and the three layers run on it in turn. Before a layer's sweep, its input spikes for all five timesteps are in SRAM, and a neuron's membrane at timestep t depends only on its own membrane at t−1 and its convolution input at t, so the chip streams pixels and runs each pixel through all five timesteps. In each cycle, the layer engine computes one timestep of one pixel for all 64 output channels at once: each forward unit sums its 3×3 dilated taps over the 64 input channels, applies the affine normalization, and updates its LIF neuron in the same step. Each pixel's readout, loss gradient, and modulation then run in a pipeline that accepts a new pixel every five cycles. Because a pixel's modulation depends only on its own readout, it returns 119 cycles after the pixel's forward pass, and the 64 backward units and accumulators take one step per cycle on the pixels whose modulation has returned. The normalization constants are loaded with the model, so the chip computes no statistics over the frame. After each sweep, the chip runs the same per-layer gradient clipping and AdamW update as the GPU code, so only one layer's gradient is alive at a time. This order computes exactly the update of Algorithm 1: with the stop-gradients, a layer's forward pass at timestep t reads only the layer below at t and its own membrane at t−1, and its loss depends only on its own forward pass.
The stop-gradients reduce what the backward pass needs to two factors per pixel and timestep. With the per-channel affine normalization, the gradient of the weight from input tap k to output channel c is , whose inner sum is the eligibility trace. The trace itself would be weight-shaped, 36,864 values per pixel, so the ASIC stores its factors instead: the 8-bit pre-reset membrane codes, from which σ′ is read from a table, and the 1-bit input taps, 11,200 bits per pixel over the five timesteps. These factors wait in a 64-pixel buffer only until the pixel's modulation returns from the readout. The backward units then multiply them by and add the products into the layer’s weight-gradient accumulators.
Every buffer on the chip is allocated statically, so the peak live state is the total storage in the table below. Beyond the parameters and their AdamW moments, the storage does not grow with depth. We have added the schedule, the stored factors, and the table to Appendix K.3 of the revised manuscript (Table 16), and Appendix D.1 now points to it from Algorithm 1.
| State | Memory | MB |
|---|---|---|
| Parameters, fp32: convolution weights and biases, readouts, coordinate input, normalization constants, class weights | SRAM | 0.309 |
| AdamW first and second moments of all trained parameters, fp32 | SRAM | 0.617 |
| Gradient of the layer being updated, read by clipping and AdamW | SRAM | 0.148 |
| 8-bit weight codes of the second and third layers | SRAM | 0.125 |
| Current layer's input spikes, 1 bit per channel, five timesteps | SRAM | 0.625 |
| First layer's input, three 8-bit channels, five timesteps | SRAM | 0.234 |
| Spike counts, 3 bits per channel | SRAM | 0.375 |
| Labels, density codes, coordinate-gradient band | SRAM | 0.090 |
| Weight-gradient accumulators of one layer (36,864) | Registers | 0.282 |
| Line buffers and convolution windows | Registers | 0.139 |
| Eligibility buffer (64 pixels × 11,200 bits = 0.085) and pixel sequencing | Registers | 0.094 |
| Channel datapaths (incl. the layer's 8-bit weight codes, A and C), readout working copy, control | Registers | 0.232 |
| Total | SRAM, registers | 2.52, 0.75 |
Response to Reviewer Ugbg (3/4)
Response to W3: We agree that the main contribution is the diagnosis and measurement of the averaging problem. As you asked, a new row splits the last step of Table 2 into the objective change and temporal credit (table below, three seeds). The table therefore distinguishes spatial granularity (first row), cross-layer credit (the two middle rows, the gradient across layers and the objective change), and temporal credit (last row). As in Table 2, the Share column gives each step's share of the distance from uniform modulation to BPTT. Each of the three later steps is smaller than the granularity step on each seed, and Table 2 now includes the new row.
| Step | mIoU reached | Gain (pp) | Share |
|---|---|---|---|
| Per-pixel modulation (PPM-3F) | 40.1±1.3 | 9.7±2.7 | 78% |
| Gradient across layers | 41.2±0.7 | 1.0±0.7 | 8% |
| One objective (new) | 42.6±1.1 | 1.4±1.3 | 11% |
| Temporal credit (BPTT-SNN) | 42.8±0.5 | 0.2±1.4 | 2% |
Granularity takes most of the distance on the reverse path as well, and the paper ties the 78% to three layers and the sequence of Table 2 (abstract, Section 5.3). With full BPTT applied first, uniform modulation reaches 33.8±0.2% mIoU (new run, three seeds), and granularity takes 73.3% of the same distance. At the deeper operating points of Table 10, five and eight layers, granularity takes 66% of the distance. Section 5.3 now gives the reverse path.
As you ask, the abstract and the conclusion now give the gap to BPTT at five and eight layers next to the 2.7 points at three layers. The gap is 6.5 points at five layers and 6.2 points at eight layers (Table 10).
Response to Q1: Under the same implementation and precision, the total training memory is the same for uniform, block-level, and per-pixel modulation (table below). The variants differ only in the modulation state (Section 5.2): one field of 4 MB per layer at 32 bits for per-pixel modulation, and 0.4% of that size with 16×16 blocks. The field is a third of a layer's state while the update is applied (Section 4). On the GPU, one implementation measures the peak and the training time of the four variants of the table (bfloat16, batch size 1), and the peak is 302.1 MB for each. The peak is the same because the coarser variants hold the full field before they average it. Under any implementation, the variants differ by at most the three modulation fields, 12 MB or 4% of the GPU peak.
The computational cost of per-pixel modulation is no higher than that of uniform or block-level modulation under one implementation. Each variant computes the modulation at each position (Section 5.2), and the coarser variants then average it, which adds 3.1 million additions per frame. The averaged value is then multiplied into the update at each position, because the local derivative and the input change from position to position. Accordingly, the measured GPU training time per frame is not longer for per-pixel modulation than for the coarser variants (table below). Under any implementation, a coarser variant saves at most the two steps that use the modulation, forming it and multiplying it into the update. The two steps take 34.6 million multiply–accumulate operations per frame, 0.6% of the 6.18 billion of the forward pass.
Beyond the modulation state, the practical cost of the additional accuracy is therefore at most 0.6% of the forward operations, with the same total training memory. The additional accuracy is 3.1 mIoU points from 16×16 blocks to per-pixel. We have added the table below to Appendix K.4 of the revised manuscript, as columns of Table 17.
| Variant | mIoU (DDD17) | Modulation state per layer, 32 bit | GPU training peak | GPU training time per frame |
|---|---|---|---|---|
| Uniform | 30.5 | 0.0002 MB | 302.1 MB | 6.46 ms |
| 16×16 | 37.0 | 0.016 MB | 302.1 MB | 6.53 ms |
| 8×8 | 38.2 | 0.063 MB | 302.1 MB | 6.55 ms |
| Per-pixel | 40.1 | 4 MB | 302.1 MB | 6.36 ms |
Response to Reviewer Ugbg (1/4)
Dear Reviewer Ugbg,
Thank you for your detailed review and your practical questions. We have answered each point: the streaming update schedule and its peak live state, the split of the gap to BPTT, the cost of each modulation granularity, the energy measurement protocol, and the organization of the paper. Our responses to each item are below. All section, appendix, and table numbers in our responses refer to the revised manuscript. With your comments, the revised paper is clearer, and its hardware claims are easier to check. If any concern remains that affects your score, please let us know and we will address it.
Response to W1: Following your comment, we reorganized the manuscript around one narrative and stated its synthesis explicitly. The narrative is three questions, one per section: what a coarse modulatory signal discards (Section 3), how much accuracy the discarded part costs (Section 5), and what per-pixel modulation costs on a chip (Section 6). The Introduction now lists the three questions in place of its three steps, and the opening of each of the three sections states its question and its answer. For the synthesis, the Introduction now states the contribution in one sentence, the diagnosis and the measurement of the averaging problem. Section 5 now states what its results show together before it lists them, and the abstract and the conclusion now include the hardware result. Section 6 is now titled Hardware Cost of Per-Pixel Modulation. The appendices now follow the order of the sections that they support, and each appendix has one purpose. The claim of the paper is unchanged.