acceptodds
Under review as a conference paper at ICLR 2027

The Averaging Problem in On-Chip Local Learning for Dense Prediction: Most of the Gap to Backpropagation Is Modulation Granularity

Abstract

Layer-local three-factor rules train spiking networks within the constraints of on-chip learning. On classification, their gap to backpropagation is attributed to the credit that cannot cross layers. On dense prediction, where none of these rules has been evaluated, we find a larger term in that gap. Every published rule forms its modulatory signal through a class-level readout that averages the per-pixel error into one vector per image. The backbone trained this way ends within one point of a frozen random backbone. Block-averaging the per-pixel error is an orthogonal projection, and a coarse rule discards the residual, which concentrates at the target's boundaries. When only the averaging operator is varied, accuracy rises monotonically with granularity on two datasets and under two local rules. Granularity is the larger term at every depth within the chip's budget. At three layers, it is 78% of the distance from uniform modulation to backpropagation and the larger term on every seed. Letting the gradient cross layers recovers little, and so does a partition aligned to the ground-truth classes. The loss therefore comes from averaging within a region rather than from missing label information. Blocks of pixels recover 67–77% of the dense gain at of the modulation state. Per-pixel modulation is a per-layer dense readout under the on-chip constraints, and it comes within points of same-architecture backpropagation.

Primary area:
applications to neuroscience & cognitive science
Keywords:
on-chip learning, neuromorphic computing, spiking neural networks, local learning rules, three-factor learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UgbgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the performance gap between layer-local three-factor learning and backpropagation for event-based dense prediction. The authors argue that spatially averaging the modulatory signal discards the different updates required at different positions. They formulate block averaging as an orthogonal projection and introduce per-pixel modulated three-factor learning (PPM-3F), where each layer obtains a spatially resolved error signal from its own dense readout. Controlled experiments on DDD17 and DSEC-Semantic show that finer modulation improves segmentation accuracy. At the default three-layer setting, PPM-3F reaches 40.1% mIoU on DDD17, compared with 30.5% for uniform modulation and 42.8% for same-architecture BPTT. The authors also study boundary-related errors, block-level modulation, and an additional local rule, and provide post-synthesis estimates for an ASIC implementation.

Critical strengths
  1. Understanding the limitations of local learning is important for training SNNs on neuromorphic chips. The paper identifies spatial modulation granularity as a useful design variable for dense prediction and provides an explanation of why spatially collapsed learning signals can lose task-relevant information.

  2. The experiments include several useful controls. In particular, keeping the dense readout fixed while changing the backward averaging operator makes the granularity comparison easier to interpret. The frozen-backbone, label-aligned partition, and random-feedback experiments help clarify where the improvement comes from. The main granularity ordering is also reported across three seeds on both segmentation datasets.

  3. The paper distinguishes the projection properties established by the analysis from the accuracy ordering measured in the experiments.

Critical weaknesses
  1. Writing and organization of this paper are weak. The paper appears to be a crude patchwork of individual points, lacking synthesis and a clear overarching narrative.

  2. If I understand Algorithm 1 correctly, the network accumulates spike rates across timesteps and then computes the local readout losses and weight updates. Section 4 applies stop-gradient operations across layers and timesteps, but this alone does not specify how the local derivatives required after rate accumulation are stored or accumulated. The authors should explain the streaming update schedule and report the peak live state, including local derivative or eligibility buffers, BN, readouts, and AdamW moments. This matters because the memory constraint is one of the main reasons for choosing this learning rule.

  3. The claim that modulation granularity accounts for most of the gap to backpropagation should remain tied to the particular experimental decomposition. The main contribution appears to be the diagnosis and measurement of the averaging problem, since the authors acknowledge the connection between PPM-3F and existing dense local heads. For this contribution, it is important to distinguish the effects of spatial granularity, cross-layer credit, and temporal credit. In Table 2, the final step changes both the use of multiple local objectives and credit assignment through time. The alternative ordering and deeper-network experiments support the qualitative finding, but the 78% share is still conditional on the chosen path and operating point. Could the authors separate the objective change from temporal credit, or explicitly limit the interpretation of this percentage? The wider PPM–BPTT gap at five and eight layers in Table 7 also deserves emphasis when discussing how closely local learning approaches BPTT.

Questions for authors
  1. What are the total training-memory and computational costs of uniform, block-level, and per-pixel modulation under the same implementation and precision? These measurements would show the practical cost of recovering the additional accuracy, beyond the reduction in modulation-state storage.

  2. For the ASIC–GPU comparison in Table 5, how is GPU energy measured, and which operations and system components are included on each side? In particular, the paper should clarify the treatment of preprocessing, readout/loss computation, optimizer updates, and idle power. The authors already identify the ASIC results as post-synthesis estimates. A more explicit measurement protocol would help assess the quantitative energy-efficiency ratios.

6 comments
AuthorsOpenReview ↗

Response to Reviewer Ugbg (1/4)

Dear Reviewer Ugbg,

Thank you for your detailed review and your practical questions. We have answered each point: the streaming update schedule and its peak live state, the split of the gap to BPTT, the cost of each modulation granularity, the energy measurement protocol, and the organization of the paper. Our responses to each item are below. All section, appendix, and table numbers in our responses refer to the revised manuscript. With your comments, the revised paper is clearer, and its hardware claims are easier to check. If any concern remains that affects your score, please let us know and we will address it.

Response to W1: Following your comment, we reorganized the manuscript around one narrative and stated its synthesis explicitly. The narrative is three questions, one per section: what a coarse modulatory signal discards (Section 3), how much accuracy the discarded part costs (Section 5), and what per-pixel modulation costs on a chip (Section 6). The Introduction now lists the three questions in place of its three steps, and the opening of each of the three sections states its question and its answer. For the synthesis, the Introduction now states the contribution in one sentence, the diagnosis and the measurement of the averaging problem. Section 5 now states what its results show together before it lists them, and the abstract and the conclusion now include the hardware result. Section 6 is now titled Hardware Cost of Per-Pixel Modulation. The appendices now follow the order of the sections that they support, and each appendix has one purpose. The claim of the paper is unchanged.

AuthorsOpenReview ↗

Response to Reviewer Ugbg (2/4)

Response to W2: The local derivatives are not stored as weight-shaped traces. The chip keeps two factors of each derivative per pixel in a 64-pixel eligibility buffer and accumulates them into the weight gradient as soon as the pixel's modulation returns. Our ASIC runs the whole learning step on chip, and its schedule and storage show that the rule meets the memory constraint: the training step of the three-layer network fits in 2.52 MB of SRAM and 0.75 MB of registers.

The chip is built around the structure of PPM-3F. Its layer engine has a forward unit, a backward unit, and a bank of weight-gradient accumulators for each of the 64 output channels, and the three layers run on it in turn. Before a layer's sweep, its input spikes for all five timesteps are in SRAM, and a neuron's membrane at timestep t depends only on its own membrane at t−1 and its convolution input at t, so the chip streams pixels and runs each pixel through all five timesteps. In each cycle, the layer engine computes one timestep of one pixel for all 64 output channels at once: each forward unit sums its 3×3 dilated taps over the 64 input channels, applies the affine normalization, and updates its LIF neuron in the same step. Each pixel's readout, loss gradient, and modulation then run in a pipeline that accepts a new pixel every five cycles. Because a pixel's modulation depends only on its own readout, it returns 119 cycles after the pixel's forward pass, and the 64 backward units and accumulators take one step per cycle on the pixels whose modulation has returned. The normalization constants are loaded with the model, so the chip computes no statistics over the frame. After each sweep, the chip runs the same per-layer gradient clipping and AdamW update as the GPU code, so only one layer's gradient is alive at a time. This order computes exactly the update of Algorithm 1: with the stop-gradients, a layer's forward pass at timestep t reads only the layer below at t and its own membrane at t−1, and its loss depends only on its own forward pass.

The stop-gradients reduce what the backward pass needs to two factors per pixel and timestep. With the per-channel affine normalization, the gradient of the weight from input tap k to output channel c is ∑pMp[c]AcT∑tσ′(up,ct)xp,kt\sum_p M_p[c]\frac{A_c}{T}\sum_t \sigma'(u^t_{p,c})x^t_{p,k}, whose inner sum is the eligibility trace. The trace itself would be weight-shaped, 36,864 values per pixel, so the ASIC stores its factors instead: the 8-bit pre-reset membrane codes, from which σ′ is read from a table, and the 1-bit input taps, 11,200 bits per pixel over the five timesteps. These factors wait in a 64-pixel buffer only until the pixel's modulation MpM_p returns from the readout. The backward units then multiply them by MpM_p and add the products into the layer’s weight-gradient accumulators.

Every buffer on the chip is allocated statically, so the peak live state is the total storage in the table below. Beyond the parameters and their AdamW moments, the storage does not grow with depth. We have added the schedule, the stored factors, and the table to Appendix K.3 of the revised manuscript (Table 16), and Appendix D.1 now points to it from Algorithm 1.

StateMemoryMB
Parameters, fp32: convolution weights and biases, readouts, coordinate input, normalization constants, class weightsSRAM0.309
AdamW first and second moments of all trained parameters, fp32SRAM0.617
Gradient of the layer being updated, read by clipping and AdamWSRAM0.148
8-bit weight codes of the second and third layersSRAM0.125
Current layer's input spikes, 1 bit per channel, five timestepsSRAM0.625
First layer's input, three 8-bit channels, five timestepsSRAM0.234
Spike counts, 3 bits per channelSRAM0.375
Labels, density codes, coordinate-gradient bandSRAM0.090
Weight-gradient accumulators of one layer (36,864)Registers0.282
Line buffers and convolution windowsRegisters0.139
Eligibility buffer (64 pixels × 11,200 bits = 0.085) and pixel sequencingRegisters0.094
Channel datapaths (incl. the layer's 8-bit weight codes, A and C), readout working copy, controlRegisters0.232
TotalSRAM, registers2.52, 0.75
AuthorsOpenReview ↗

Response to Reviewer Ugbg (3/4)

Response to W3: We agree that the main contribution is the diagnosis and measurement of the averaging problem. As you asked, a new row splits the last step of Table 2 into the objective change and temporal credit (table below, three seeds). The table therefore distinguishes spatial granularity (first row), cross-layer credit (the two middle rows, the gradient across layers and the objective change), and temporal credit (last row). As in Table 2, the Share column gives each step's share of the distance from uniform modulation to BPTT. Each of the three later steps is smaller than the granularity step on each seed, and Table 2 now includes the new row.

StepmIoU reachedGain (pp)Share
Per-pixel modulation (PPM-3F)40.1±1.39.7±2.778%
Gradient across layers41.2±0.71.0±0.78%
One objective (new)42.6±1.11.4±1.311%
Temporal credit (BPTT-SNN)42.8±0.50.2±1.42%

Granularity takes most of the distance on the reverse path as well, and the paper ties the 78% to three layers and the sequence of Table 2 (abstract, Section 5.3). With full BPTT applied first, uniform modulation reaches 33.8±0.2% mIoU (new run, three seeds), and granularity takes 73.3% of the same distance. At the deeper operating points of Table 10, five and eight layers, granularity takes 66% of the distance. Section 5.3 now gives the reverse path.

As you ask, the abstract and the conclusion now give the gap to BPTT at five and eight layers next to the 2.7 points at three layers. The gap is 6.5 points at five layers and 6.2 points at eight layers (Table 10).

Response to Q1: Under the same implementation and precision, the total training memory is the same for uniform, block-level, and per-pixel modulation (table below). The variants differ only in the modulation state (Section 5.2): one field of 4 MB per layer at 32 bits for per-pixel modulation, and 0.4% of that size with 16×16 blocks. The field is a third of a layer's state while the update is applied (Section 4). On the GPU, one implementation measures the peak and the training time of the four variants of the table (bfloat16, batch size 1), and the peak is 302.1 MB for each. The peak is the same because the coarser variants hold the full field before they average it. Under any implementation, the variants differ by at most the three modulation fields, 12 MB or 4% of the GPU peak.

The computational cost of per-pixel modulation is no higher than that of uniform or block-level modulation under one implementation. Each variant computes the modulation at each position (Section 5.2), and the coarser variants then average it, which adds 3.1 million additions per frame. The averaged value is then multiplied into the update at each position, because the local derivative and the input change from position to position. Accordingly, the measured GPU training time per frame is not longer for per-pixel modulation than for the coarser variants (table below). Under any implementation, a coarser variant saves at most the two steps that use the modulation, forming it and multiplying it into the update. The two steps take 34.6 million multiply–accumulate operations per frame, 0.6% of the 6.18 billion of the forward pass.

Beyond the modulation state, the practical cost of the additional accuracy is therefore at most 0.6% of the forward operations, with the same total training memory. The additional accuracy is 3.1 mIoU points from 16×16 blocks to per-pixel. We have added the table below to Appendix K.4 of the revised manuscript, as columns of Table 17.

VariantmIoU (DDD17)Modulation state per layer, 32 bitGPU training peakGPU training time per frame
Uniform30.50.0002 MB302.1 MB6.46 ms
16×1637.00.016 MB302.1 MB6.53 ms
8×838.20.063 MB302.1 MB6.55 ms
Per-pixel40.14 MB302.1 MB6.36 ms
Reviewer_TwZ4OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates why layer-local learning in spiking neural networks (SNNs) performs worse than backpropagation on dense prediction tasks. The authors argue that spatial averaging of modulation signals, rather than the lack of cross-layer gradients, accounts for most of the performance gap under their on-chip learning constraints. This paper studies how the spatial granularity of learning signals affects on-chip local learning in spiking neural networks (SNNs) for dense prediction. The authors argue that averaging pixel-wise errors into a shared modulation signal causes substantial information loss. They propose PPM-3F, which uses per-pixel modulation and layer-local three-factor updates, and provide a theoretical analysis based on orthogonal projection.

The main contribution is a systematic analysis of this averaging problem. The authors study different modulation granularities and introduce PPM-3F, which applies per-pixel supervision to existing three-factor local learning rules. Experiments on DDD17 and DSEC-Semantic show substantial improvements over uniform modulation, with PPM-3F approaching same-architecture BPTT performance. The paper also provides a theoretical analysis of the information lost through averaging and examines memory and hardware costs through ASIC synthesis.

Experiments on DDD17 and DSEC-Semantic show that finer-grained modulation improves segmentation accuracy. On DDD17, PPM-3F achieves 40.1% mIoU, compared with 30.5% for uniform modulation and 42.8% for same-architecture BPTT. The paper also examines modulation granularity, memory requirements, and hardware feasibility through ASIC synthesis.

Critical strengths

1. The controlled experiments are useful.

The authors do a good job isolating the effect of modulation granularity. They keep the network architecture and training setup fixed and change how the modulation signal is averaged. On DDD17, mIoU improves from 30.5% to 40.1%, and on DSEC-Semantic, from 22.0% to 37.9%. The results are consistent across three random seeds. The authors also test a second local learning rule (S-TLLR) and a stereo disparity task.

I think these experiments provide useful evidence that spatial averaging can substantially hurt dense prediction under the learning constraints considered in this paper.

2. The paper considers the cost of finer-grained modulation.

The authors do not only report accuracy improvements. They also examine the memory trade-off and show that 16×16 block modulation recovers 67–77% of the improvement from per-pixel modulation while requiring only 0.4% of its modulation state. The paper also provides a 28nm ASIC synthesis analysis.

These results are useful for understanding the accuracy–memory trade-off in hardware-constrained SNN training, although the hardware results are still post-synthesis estimates rather than measurements from a fabricated chip.

Critical weaknesses

1. Limited methodological novelty

My first concern is the limited methodological novelty of PPM-3F.

A closely related prior work is InfoPro (Wang et al., ICLR 2021, paper). InfoPro already uses gradient-isolated modules with local auxiliary supervision. It also applies dense prediction heads to semantic segmentation. In particular, Appendix E (Table 11) shows a Cityscapes auxiliary head with a 1×1 convolution producing 19 class scores.

PPM-3F follows a similar training approach: each SNN layer has a dense prediction head, computes pixel-wise cross-entropy, and updates its weights using local errors (Section 3.4 and Algorithm 1).

I recognize that the motivation is different. InfoPro focuses on preserving task-relevant information across modules, while this paper studies spatial averaging of modulation signals. The SNN three-factor update and on-chip constraints are also meaningful differences.

However, it remains unclear what new learning mechanism is introduced beyond adapting existing dense local supervision to SNNs. The authors should better explain what makes this adaptation technically nontrivial.

The paper also considers a specific form of local learning: no cross-layer gradients, O(N) memory, and no weight transport (Section 3.1). These are reasonable constraints, but local learning does not necessarily require all three. Other approaches may use different learning signals or temporal credit assignment without full BPTT. The relevance of these particular constraints to the broader local learning problem needs to be clearer.

I see the empirical analysis of modulation granularity as the paper's main potential contribution, rather than PPM-3F itself. Whether this analysis provides sufficiently new and general knowledge is therefore important to my assessment.

2. Limited generalizability of the experimental findings

The controlled experiments provide useful evidence that spatial averaging can hurt dense prediction. However, I am not convinced that the results support the broader claim that modulation granularity accounts for most of the performance gap between local learning and BPTT.

First, the evaluation is limited to a small and specialized architecture. The main network has three ASPP layers, approximately 77K parameters, five timesteps, and 128×128 inputs (Section 4 and Appendix Table 6). Each layer has its own multiscale feature extractor and local prediction head. This architecture is well suited to independent layer-wise training, but it may not reflect the importance of cross-layer credit assignment in more conventional deep SNNs.

The experiments also focus mainly on six-class DDD17 and eleven-class DSEC-Semantic segmentation. These are valid benchmarks, but the limited model scale and task diversity make it difficult to judge how broadly the findings apply.

Second, the paper's own depth experiments raise concerns about scalability. Appendix Table 7 shows that PPM-3F achieves 39.9%, 41.2%, and 41.5% mIoU at three, five, and eight layers, while BPTT achieves 43.3%, 47.7%, and 47.7% (seed 42). The gap therefore grows from 3.4 to 6.5 and 6.2 percentage points. The reported contribution of modulation granularity also decreases from 73% to 66%.

Granularity remains important in these experiments, but the results suggest that cross-layer credit assignment may become more important as networks grow deeper. This deserves more attention given the paper's central claim.

Third, the main improvement is measured against a spatially uniform modulation baseline. On DDD17, Uniform-3F achieves 30.5% mIoU, only slightly above the 28.3% obtained with a frozen random backbone and trained dense readout. PPM-3F reaches 40.1%.

This is a substantial improvement, and I appreciate that the authors carefully control the modulation operator. However, it mainly shows that spatially uniform feedback performs poorly in this particular dense prediction setting. It does not establish how PPM-3F compares with other spatially informed local learning approaches under comparable resource constraints.

Finally, the reported 78% contribution of modulation granularity comes from a particular sequence of experimental changes. The final step toward BPTT changes both temporal credit assignment and the training objective. I would therefore interpret 78% as a result of this specific experimental decomposition, rather than a general estimate of the relative importance of different learning bottlenecks.

Overall, I find the controlled experiments useful and the improvements convincing within the tested setting. My concern is whether the finding is sufficiently general and significant, given that the underlying training components are largely established techniques.

Questions for authors

Please see my comments in Critical Weaknesses. I have two main questions:

Q1. How is PPM-3F different from applying existing dense local supervision methods, such as InfoPro (ICLR 2021), to an SNN? What is actually new in the learning rule? Also, are all three constraints in Section 3.1 required by the hardware, or are some of them design choices made by the authors?

Q2. Table 7 shows that the gap between PPM-3F and BPTT grows as the network gets deeper. How does this fit with the claim that modulation granularity accounts for most of the gap? What evidence suggests that this conclusion would still hold for other network architectures or local learning methods?

7 comments
AuthorsOpenReview ↗

Response to Reviewer TwZ4 (1/4)

Dear Reviewer TwZ4,

Thank you for your thoughtful review and for seeing the controlled analysis of modulation granularity as a potential contribution of the paper. We have answered each point, from the relation to InfoPro and the on-chip constraints to the generality of the findings, adding experiments where you asked for evidence. Our responses to each item are below. All section, appendix, and table numbers in our responses refer to the revised manuscript. With your comments, the revised paper now states its contribution and scope more clearly. If any concern remains that affects your score, please let us know and we will address it.

AuthorsOpenReview ↗

Response to Reviewer TwZ4 (2/4)

Response to W1: As you point out, InfoPro also uses dense local heads, and we agree that PPM-3F introduces no new learning mechanism (please see our response to Q1 for details). We would like to clarify that the paper contributes new findings, not a new mechanism. The main finding is that modulation granularity takes the largest share of the distance from uniform modulation to BPTT on dense prediction (Section 1). The finding matters to our community because granularity, unlike the cross-layer credit that chips lack, can be refined within the on-chip constraints (Section 7). To make the point explicit, the Introduction now states the contribution as the diagnosis and the measurement of the averaging problem, and Section 3.4 states that PPM-3F is the published three-factor update with one change, the modulation granularity.

The technically nontrivial part of the adaptation is measuring the effect of granularity, because three obstacles hid it. First, the published DECOLLE readout is only 1.4 mIoU points above predicting the most frequent class at every position (Section 5.1). We therefore let every variant learn a per-position readout and average only the modulation field sent to the backbone (Section 5.2). Second, changing the credit path with the granularity would mix the two effects, so all variants keep it inside the layer and differ only in the averaging operator. Third, the share of per-pixel error that a variant discards does not predict its accuracy. A partition with one modulation per ground-truth class discards as much per-pixel error as 16×16 blocks, but its mIoU gain over uniform modulation is 23% of the blocks' gain (Table 3). The discarded-gradient ratio, the share of the update lost after the error passes through the layer's kernel, orders the coarse variants by accuracy instead (Section 3.3, Section 5.4). As you asked, the Introduction now describes the three obstacles and how we overcame each one.

As you say, local learning in general does not necessarily require all three constraints, and the paper focuses on on-chip learning (Section 1). Neuromorphic chips such as Loihi 2 require all three to train layers of the size used in this paper (please see Section 3.1 and our response to Q1). On the broader local learning problem, however, the update lost by averaging does not depend on the three constraints, because the proofs of Section 3.3 use none of them. We also tested the two cases you mention, a different learning signal and temporal credit assignment without full BPTT. Per-pixel modulation outperformed uniform modulation in mIoU even when only the per-pixel variant used fixed random feedback, and when both used S-TLLR inside each layer (Appendix H.2, Section 5.6). To make the relevance clearer, Section 3.1 now states where each constraint comes from in the hardware, and Section 3.3 states that the propositions hold without the three constraints.

On whether the analysis provides sufficiently new knowledge: layer-local rules within the three constraints had been evaluated only on image-level outputs (Section 1). On dense prediction, we found two facts that run against common belief. First, the gap between such rules and backpropagation was attributed to the credit that cannot cross layers (Section 1). Yet at 3 layers, modulation granularity accounts for 78% of the distance from uniform modulation to BPTT (Table 2). Cross-layer gradients account for 8% of that distance when added to per-pixel modulation, and they recover nothing when added to uniform modulation (Table 2, Section 5.3). Second, such rules are known to be competitive with backpropagation in classification. However, a backbone trained for dense prediction through the published DECOLLE readout ends within 1 mIoU point of a frozen random backbone (Section 5.3).

On whether the analysis provides sufficiently general knowledge: both facts come from the update that averaging discards. Our proof assumes only that the update is linear in the modulation field (Section 3.3). Spatial averaging therefore loses part of the update in every three-factor rule of the form in Equation 1 and for every per-position loss. In measurements, performance also improved with finer modulation across two datasets, two local rules, and two tasks (Section 5.2, Section 5.6). The new experiments on other architectures and the depth results that you asked about are in our responses to W2 and Q2.

AuthorsOpenReview ↗

Response to Reviewer TwZ4 (3/4)

Response to W2: Beyond the second task and the larger scale of the submission, the finding now holds in a conventional architecture and on a non-driving dataset. In an encoder–decoder with plain 3×3 convolutions and no multiscale extractor, granularity takes 67.5% of the distance from uniform modulation to BPTT (new, three seeds, our response to Q2). When the main network is scaled to 300K parameters, beyond the parameter budget of the chip, granularity takes 70.5% of the distance (Appendix G). With ten timesteps in place of five, granularity takes 73.7% of the distance (new run, Appendix G). On Oxford-IIIT Pet, a non-driving dataset of still images, PPM-3F reaches 52.9±0.1% mIoU against 36.2±0.9% for uniform modulation and 54.4±0.1% for BPTT-SNN (new, three seeds, Section 5.6 and Table 11). In stereo disparity, a regression task, per-pixel modulation matches a new BPTT reference at a mean absolute error of 1.48 px, against 1.66 px for uniform modulation (Section 5.6 and Table 13).

At each depth of Table 10, the step that lets the gradient cross layers stays smaller than the granularity step, although it grows from three to five layers. Gradients across layers add 1.0 points at three layers (Table 2), and 2.4 and 2.5 points at five and eight layers (Table 10). Our response to Q2 gives the gap and the share of granularity at each depth and the argument for deeper networks. As you ask, Section 5.3 and Appendix G now give the depth result more attention.

Against other spatially informed local approaches, PPM-3F meets the on-chip constraints at a cost of at most 1.4 mIoU points (table below, three seeds, 77K budget). The two approaches are the dense local heads of deep supervision and InfoPro. In deep supervision, each hidden layer has its own loss, and each loss is backpropagated through the layers below it (Lee et al., 2015). With per-pixel losses in our network, deep supervision is the Gradient across layers row of Table 2. The InfoPro head is a new run: it follows the head that InfoPro uses on Cityscapes and trains its 3×3 convolution by backpropagation inside each head. Both heads therefore need a backward pass through hidden layers, which a neuromorphic chip such as Loihi 2 does not have (Section 3.1). The uniform baseline stands for the published rules: Uniform-3F is a strengthened form of the class-level mechanism that DECOLLE, TESS, and ETLP share (Section 5.1). The two heads are now rows of Table 1, and Section 5.1 describes them.

MethodOn-chipmIoU (%)
PPM-3F✓40.1±1.3
Deep supervision✗41.2±0.7
InfoPro head (new)✗41.5±0.8

Granularity takes most of the distance in the reverse sequence as well. With full BPTT applied first, uniform modulation reaches 33.8±0.2% (new run, three seeds), and granularity takes 73.3% of the same distance, against 78% in the sequence of Table 2. A new row of Table 2 also splits the last step into one objective and temporal credit, which add 1.4±1.3 and 0.2±1.4 points, against 9.7±2.7 points for granularity. Section 5.3 now gives the reverse sequence.

The results above make the finding general, and the established components make the comparison controlled. Uniform-3F and PPM-3F share the same training components, so the comparison isolates the averaging operator. The finding matters to the community because granularity, unlike the cross-layer credit that chips lack, can be refined within the on-chip constraints (please see our response to W1).

Reviewer_9QXwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies layer-local three-factor learning for event-based semantic segmentation under three on-chip constraints: layer-local updates, (O(N)) memory, and no weight transport. It argues that published rules form the third factor from a class-level readout, and that on dense targets this spatial average, rather than the missing cross-layer gradient, accounts for most of the gap to backpropagation. Block averaging is treated as an orthogonal projection whose residual concentrates at label boundaries. PPM-3F gives each layer its own (1\times1) per-pixel readout and stops gradients between layers and timesteps. On a shared 3-layer, 77K-parameter network at (128\times128), DDD17 mIoU rises from 30.5% with uniform modulation to 37.0% at (16\times16) blocks, 40.1% with per-pixel modulation, and 42.8% with same-architecture BPTT. The paper assigns 78% of that distance to granularity. The ordering also appears on DSEC-Semantic and under an S-TLLR layer. A 28 nm ASIC is reported from post-synthesis estimates.

Critical strengths

The main comparison is clean and convincing. It changes only the averaging operator. Uniform modulation, (16\times16), (8\times8), and per-pixel modulation share the backbone, the dense readout, and the training recipe. Accuracy rises in that order on every seed of DDD17 and DSEC-Semantic, and a coarser-to-finer order also appears when the within-layer update is replaced by an S-TLLR layer. That is direct, well-controlled evidence for the granularity claim at three layers.

The boundary measurements make the mechanism specific and useful. The gain is concentrated on vehicles, humans, and nature, while the majority classes do not explain it. A class-aligned partition reaches only 32.0%, close to uniform modulation at 30.5% and below (16\times16) blocks at 37.0%. This is a sharp check: the missing signal is spatial detail inside a region, not the class label. Blocks of (16\times16) recover 67% of the dense gain on DDD17 and 77% on DSEC at 0.4% of the modulation state. The practical suggestion is therefore concrete, and narrower than “always use a per-pixel field.”

Critical weaknesses
  1. The 78% in Table 2 is a clean step: uniform modulation at 30.5% to PPM-3F at 40.1%, both layer-local. The next two rows are not. Gradient across layers reaches 41.2% while every layer still has its own dense loss. BPTT-SNN reaches 42.8% by replacing those losses with one objective and turning on temporal credit in the same step. The backbone has no pooling, so every layer stays at 128×128. Granularity at three layers is supported. The comparison with cross-layer credit is not.

  2. PPM-3F is the obvious end of the sweep: a (1\times1) per-pixel readout with the gradient stopped between layers and timesteps, that is, DECOLLE’s local loss without the spatial average. It adds no new factor or credit path. The useful part is the comparison that holds the backbone fixed.

  3. The separation from nearby work is stated more clearly than it is measured. Dense local heads already appear in deep supervision and InfoPro, and a class-level readout that misses shape is standard in weakly supervised segmentation. PPM-3F is not compared with those heads, nor with the off-chip event segmenters cited at 45–55% mIoU. On this 77K backbone, the published-rule rows are the DECOLLE readout and pool-then-classify.

  4. Tables 4 and 5 report joules per frame for PPM-3F only. The claim that (16\times16) blocks recover 67–77% of the dense gain at 0.4% of the modulation state counts vectors, not energy. There is no joules-per-frame entry for uniform, (16\times16), or (8\times8). What the silicon comparison includes is Question 3.

  5. The evidence is one ASPP family: 3 layers, 77K parameters, five timesteps, (128\times128), on two event-camera driving sets. Disparity uses the same scenes and has no BPTT reference. S-TLLR is DDD17 only. Three seeds keep the sign of the PPM–Uniform gap, and they are thin for the 1.0±0.7 and 1.7±1.2 steps in Table 2.

Questions for authors
  1. On DDD17 in Table 2, PPM-3F reaches 40.1%, Gradient across layers reaches 41.2%, and BPTT-SNN reaches 42.8%. From Section 5.3, the first step only lets gradients cross layers, while each layer keeps its own loss. The second step replaces those losses with one objective and turns on temporal credit together. Could you add a row with one final dense loss, cross-layer gradients, and the temporal stop-gradient kept, on seeds 42, 0, and 1? If not, it would help to say which step mixes the three, and to limit the cross-layer claim to what that step actually separates. The 78% itself is only uniform modulation to PPM-3F.

  2. The backbone seems to use neither pooling nor upsampling, so every layer stays at (128\times128). Dilation changes the receptive field, but not the spatial scale, and the added layers are described as parallel feature maps. If feasible, would you repeat the decomposition where depth changes spatial scale, for example with pooling or an encoder–decoder? If that is outside the present scope, would you narrow the title and conclusion to this full-resolution architecture?

  3. Could you say what the ASIC area, power, throughput, and joules per frame include? I was unsure whether they cover the convolutional forward path, normalization, LIF updates, buffering, memory movement, and the AdamW state, and likewise the GPU precision, batch size, measurement procedure, and included operations. The comparison is a 28 nm post-synthesis design at batch size 1 against a 4 nm RTX PRO 6000, so these details would help me read the energy ratio.

7 comments
AuthorsOpenReview ↗

Response to Reviewer 9QXw (1/4)

Dear Reviewer 9QXw,

Thank you for your careful and detailed review. We have answered each weakness and question and added the analyses and measurements you requested. Our responses to each item are below. All section, appendix, and table numbers in our responses refer to the revised manuscript. With your comments, the revised paper now presents the decomposition, the energy comparison, and the scope of our claims more clearly. If any concern remains that affects your score, please let us know and we will address it.

Response to W1: The comparison with cross-layer credit now rests on the two experiments that you requested in Q1 and Q2. With the row of Q1, cross-layer credit becomes a clean step, which is smaller than the granularity step in each of the three seeds. In the encoder–decoder of Q2, whose depth changes the spatial scale, granularity takes 67.5% of the distance from uniform modulation to BPTT. Our responses to Q1 and Q2 give the tables, and Section 5.3 now states the comparison of granularity with cross-layer credit.

Response to W2: As you say, PPM-3F adds no new factor or credit path, and the paper presents it as the per-pixel end of the sweep. We would like to clarify that the value of the paper lies in new findings opposite to what the standard explanation predicts for dense prediction. The standard explanation attributes the gap between layer-local rules and backpropagation to credit assignment (Lillicrap et al., 2020; Kaiser et al., 2020; Quintana et al., 2024). The rules published after DECOLLE, namely TESS, ETLP, and S-TLLR, address credit assignment across layers and time. With the backbone, the factors, and the credit path fixed, removing the spatial average alone gains 9.7 mIoU points, 78% of the distance from uniform modulation to BPTT (Table 2). Under uniform modulation, letting the gradient cross layers recovers nothing (Section 5.3). For dense prediction, the larger term is therefore the granularity that DECOLLE, TESS, and ETLP hold fixed, not the credit that the rules after DECOLLE address. We have extended the comparison that you find useful to other credit paths and to an encoder–decoder (our responses to W1 and Q2). Section 3.4 now states that PPM-3F keeps the published three-factor update and changes only the modulation granularity.

Response to W3: PPM-3F meets the on-chip constraints at a cost of at most 1.4 mIoU points against the dense local heads of deep supervision and InfoPro (table below, three seeds, 77K budget). In deep supervision, each hidden layer has its own loss, and each loss is backpropagated through the layers below it (Lee et al., 2015). With per-pixel losses in our network, deep supervision is the Gradient across layers row of Table 2. The InfoPro head is a new run: it follows the head that InfoPro uses on Cityscapes and trains its 3×3 convolution by backpropagation inside each head. Both heads therefore need a backward pass through hidden layers, which a neuromorphic chip such as Loihi 2 does not have (Section 3.1). PPM-3F gives each pixel its own error signal inside the constraints, and the per-pixel signal is worth 9.7 points over Uniform-3F (Table 2). The comparison therefore supports the main claim of the paper: under the on-chip constraints, modulation granularity sets most of what layer-local learning loses on dense prediction. The two heads are now rows of Table 1, and Section 5.1 describes them.

MethodOn-chipmIoU (%)
PPM-3F✓40.1±1.3
Deep supervision✗41.2±0.7
InfoPro head (new)✗41.5±0.8

Weakly supervised segmentation misses shape because its labels are class-level, while Uniform-3F loses shape with per-pixel labels and a per-pixel readout (30.5±1.4%, Table 1). A new label experiment tests the label side (Appendix H.3): with labels on a random 10% of the pixels, per-pixel modulation still reaches 39.9±0.2%. The shape is therefore lost through the averaged signal, not through the labels.

The distance to the off-chip segmenters separates into the rule and the size of the network. The off-chip segmenters reach 45–55% mIoU with 0.4M to 8.6M parameters and BPTT, and PPM-3F reaches 40.1% with 77K parameters. With BPTT, the same 77K backbone reaches 42.8%, so the rule accounts for 2.7 points (Table 1). With 300K parameters, fewer than their smallest network, BPTT on our backbone is within their mIoU range (Appendix G).

We would like to clarify that the comparison with the published rules rests on Uniform-3F, which is stronger than the DECOLLE readout and pool-then-classify rows. Uniform-3F is DECOLLE's rule with the dense readout kept, a strengthened form of the class-level mechanism that DECOLLE, TESS, and ETLP share (Section 5.1). With the official S-TLLR layer, per-pixel modulation is again above uniform modulation, which is above a pooled readout per layer (Section 5.6). Section 2 now refers to the two heads, and Section 5.4 now reports the label experiment.

AuthorsOpenReview ↗

Response to Reviewer 9QXw (2/4)

Response to W4: Following your point that the 16×16 blocks' 0.4% share of the modulation state counts vectors, not energy, the table below compares all four variants in training joules per frame. Inference has no labels and forms no modulation field, the gradient of the per-position loss, so all four variants run the same inference and the table lists training only.

VariantVectors per layerGPU training J/frame [95% CI]ASIC training J/frame
Uniform11.333 [1.325, 1.340]0.0310*
16×16641.345 [1.332, 1.357]0.0312*
8×82561.342 [1.324, 1.360]0.0314*
Per-pixel16,3841.326 [1.322, 1.330]0.0231

*Estimated from the PPM-3F design.

On the GPU, coarse variants still build the full modulation field before averaging it, so they do all of per-pixel's work and save no energy. The averaging itself adds only a short call, which lengthens the training step slightly. The 95% confidence intervals of the differences in GPU training J/frame among the three coarse variants include 0 (Appendix K.4), so the GPU column does not rank them.

The per-pixel ASIC value comes from the same flow as Tables 4 and 5. The ASIC implements a single layer, made configurable because the first layer's input differs in form from that of the other two, and this one structure runs all three layers of PPM-3F. Power is the post-synthesis power that Synopsys Design Compiler reports at the typical corner. Throughput is taken from a simulation of the full three-layer PPM-3F on the implemented design with Synopsys VCS and Verdi. Training J/frame is the power divided by the throughput. For per-pixel, 6.13 W / 265.5 frames/s = 0.0231 J/frame. The values of the three coarse variants are estimated from the PPM-3F design.

On the ASIC, the only extra arithmetic of a coarse variant is the additions for the average, under 0.1% of the network’s forward MACs. Fewer modulation vectors, however, do not mean less training work. The number of vectors counts distinct modulation values, not operations: every variant still trains its readout at every pixel and applies the modulation to every pixel’s update at every timestep. Because on-chip learning leaves little memory, 2.52 MB of SRAM for the whole network on our chip, the ASIC streams pixels and accumulates each pixel’s gradient as soon as its modulation is known. A block mean is known only after the block’s last pixel, so a block-modulated chip would have to buffer the whole block’s backward inputs. Without this buffer, the block-modulated chip would draw 5–8% less power but need 45% more cycles per training frame. Its estimated training energy therefore rises to about 0.031 J/frame for every coarse variant, 33–38% above per-pixel modulation, while the block-averaging unit itself adds only 26–118 mW. We have added the table and the two analyses above to Appendix K.4 of the revised manuscript (Table 17), and Section 6 now states the result.

AuthorsOpenReview ↗

Response to Reviewer 9QXw (3/4)

Response to W5: The evidence now covers a second architecture and a dataset outside the driving scenes, in addition to the greater depth and width of Appendix G. In an encoder–decoder, granularity takes 67.5% of the distance from uniform modulation to BPTT (our response to Q2). With 300K parameters, granularity takes 70.5% of the distance from uniform modulation to BPTT (Appendix G). Because disparity shares its scenes with segmentation, we added a dataset from other scenes: Oxford-IIIT Pet, still images fed as rate-coded spikes (table below). In the encoder–decoder and on Oxford-IIIT Pet, granularity takes more than half of the distance from uniform modulation to BPTT in each of the three seeds. Both results support the main claim of the paper, that modulation granularity sets most of what layer-local learning loses on dense prediction under the on-chip constraints.

With ten timesteps in place of five, granularity takes 73.7% of the distance from uniform modulation to BPTT (new run, Appendix G). The five timesteps of the paper are the five voxel bins of the input, and 128×128 with 64 channels is the largest layer that fits one Loihi 2 chip (Section 4).

The disparity and S-TLLR experiments now have what you found missing: disparity has a BPTT reference, and S-TLLR also runs on DSEC (table below). In disparity, per-pixel modulation matches the BPTT reference. Under S-TLLR on DSEC, per-pixel modulation is 13.5 mIoU points above uniform modulation. The BPTT value for S-TLLR on DSEC is the same network trained with BPTT.

ExperimentUniformPer-pixelBPTT
Oxford-IIIT Pet, mIoU (%) (new)36.2±0.952.9±0.154.4±0.1
DSEC disparity, error (px)1.661.481.48 (new)
S-TLLR on DSEC, mIoU (%) (new)19.9±1.133.4±0.236.7±0.4

We ran two more seeds for the two steps that you name. In each of the five seeds, both steps stay positive and smaller than the granularity step. Section 5.6 now reports the three rows of the table, and Appendix F gives the values of the two steps per seed (Table 9).

Response to Q1: We have added the row you requested, on seeds 42, 0, and 1 (table below). The row splits the last step of Table 2 into one objective and temporal credit. As in Table 2, the Share column gives each step's share of the distance from uniform modulation to BPTT.

StepmIoU reachedGain (pp)Share
Per-pixel modulation (PPM-3F)40.1±1.39.7±2.778%
Gradient across layers41.2±0.71.0±0.78%
One objective (new)42.6±1.11.4±1.311%
Temporal credit (BPTT-SNN)42.8±0.50.2±1.42%

With the row, the cross-layer claim rests on a clean step, from PPM-3F to the new row. In each of the three seeds, the clean step is smaller than the step from uniform modulation to PPM-3F: 2.5 against 9.7 mIoU points on average (our response to W1). Table 2 now includes the row.

Response to Q2: As you suggest, we repeated the decomposition with an encoder–decoder, whose depth changes the spatial scale. The five layers run at 128×128, 64×64, 32×32, 64×64, and 128×128, with skip connections, at the paper's parameter budget. The table has the columns of Table 2 and starts from uniform modulation at 27.3±1.7%.

StepmIoU reachedGain (pp)Share
Per-pixel modulation (PPM-3F)39.7±0.712.5±1.667.5%
Gradient across layers40.1±2.00.3±1.61.7%
One objective44.3±1.14.3±1.323.1%
Temporal credit (BPTT-SNN)45.8±0.51.4±0.57.6%

Granularity remains the larger term in the encoder–decoder, so we keep the title. In the encoder–decoder, granularity takes 67.5% of the distance from uniform modulation to BPTT, against 66% in the full-resolution backbone of the same depth (five layers, Table 10). The step from uniform modulation to PPM-3F is larger than the three credit steps together in each of the three seeds. The conclusion now states the result for both architectures, and Appendix J.1 includes the table (Table 12).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Responses and Revisions

Dear Area Chair and Reviewers,

Thank you for your time and for your comments, which have improved the paper. As the discussion period is coming to an end, we would like to summarize our responses and the changes in the revised manuscript.

Our responses address each weakness (W) and question (Q) in the three reviews. For each concern, the table below lists the review items that raise it, what the revision adds, and the location in the revised manuscript.

ConcernRaised inAdded in the revisionLocation
Last decomposition step mixes two changes9QXw W1/Q1, TwZ4 W2, Ugbg W3New row that separates themTable 2
78% share depends on step orderTwZ4 W2, Ugbg W3Fully reversed orderSection 5.3
One architecture at full resolution9QXw W1/W5/Q2, TwZ4 W2/Q2Decomposition in an encoder–decoderTable 12
Two driving datasets and five timesteps9QXw W5, TwZ4 W2Oxford-IIIT Pet; ten timestepsTable 11, Appendix G
Gap to BPTT at greater depthTwZ4 W2/Q2, Ugbg W3Depth gap stated and explainedAbstract, Section 5.3, Appendix G
Evidence for other local rules9QXw W5, TwZ4 Q2S-TLLR on DSEC, with BPTT referencesTable 11
Disparity reference and number of seeds9QXw W5BPTT reference for disparity; five seedsTable 13, Table 9
Comparison with dense local heads9QXw W3, TwZ4 W2InfoPro head and deep supervisionTable 1
What is new in the rule9QXw W2, TwZ4 W1/Q1Contribution and difference from InfoPro statedSection 1, Section 3.4
Need for the three on-chip constraintsTwZ4 W1/Q1Hardware source; analysis holds without themSection 3.1, Section 3.3
Energy and cost of each granularity9QXw W4, Ugbg Q1Memory, time, and energy per granularityTable 17
What the hardware figures include9QXw Q3, Ugbg Q2Included items and measurement protocolAppendix K.2, Table 15
Update schedule and peak live stateUgbg W2Schedule and storage of the ASICAppendix K.3, Table 16
Fixed normalization constants on the ASICUgbg follow-upTraining with fixed constantsTable 14a
Writing and organizationUgbg W1Paper organized around three questionsSection 1

The contribution of the paper is the diagnosis and the measurement of the averaging problem in on-chip local learning. On dense prediction, modulation granularity, rather than the credit that cannot cross layers, accounts for most of the distance from uniform modulation to BPTT, 78% at three layers. The new experiments support the main claim of the paper. With the new row of Table 2, cross-layer credit adds 2.5 mIoU points, against 9.7 for granularity. Granularity accounts for 73.3% of the distance with the steps of the decomposition reversed, 67.5% in an encoder–decoder, 73.7% at ten timesteps, and 92.1% on Oxford-IIIT Pet.

Reviewer Ugbg raised the rating from 2 to 3 after our first response and wrote that the new decomposition experiment and the energy-accounting details "substantially address my concerns." We have responded to the reviewer's one remaining question with a new experiment (Table 14a). Reviewer 9QXw also raised the rating from 2 to 3 and wrote that the new experiments "resolve the two questions behind my score." Reviewer TwZ4 wrote that the main remaining concerns are algorithmic novelty and adaptability, and we have responded to both in a follow-up response. We hope that our responses and the revised manuscript have addressed all the concerns raised in the reviews. We will be glad to respond to any further questions before the discussion period ends.

Best,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.