acceptodds
Under review as a conference paper at ICLR 2027

Perturbation Propagation: Scalable Local Learning without Weight Transport

Abstract

The growing energy demands of artificial neural networks on digital hardware increasingly limit the level of intelligence they can reach, motivating the search for potentially far more energy-efficient analog implementations. The brain offers an existence proof that a physical system can learn complex tasks at very low energy cost, but a biologically plausible learning algorithm that trains at scale remains elusive. Backpropagation is not biologically plausible: it relies on exact derivatives from mathematical models of neuron responses and on weight transport, which copies forward weights into the backward pathway. More biologically plausible alternatives struggle to match backpropagation's scalability and algorithmic simplicity. We propose Perturbation Propagation (PerProp), which keeps the forward-backward structure of backpropagation but replaces both operations with local measurements: neuron derivatives with slopes measured by small forward perturbations, and weight transposes with feedback matrices learned from random probes. Because each neuron acts on its own input, one pair of perturbed evaluations measures a whole layer's slopes; because weights are shared across examples, feedback matrices are learned once per update. In Transformer pretraining, PerProp scales at a rate comparable to backpropagation with model size and compute, and at a faster rate with data, batch size, and depth. On GPT and Mamba-2 language models, it outperforms the feedback-alignment and perturbation-based alternatives we evaluate, both in and out of distribution. On spiking, comparator, and probabilistic-bit networks, it rivals or exceeds surrogate-gradient training without hand-designed surrogate derivatives. On analog hardware, PerProp trains physical neural networks directly, learning entirely on the BrainScaleS-2 neuromorphic chip and substantially outperforming reward-modulated STDP. Component-level estimates for analog crossbars further project orders-of-magnitude lower training energy than GPU backpropagation, an advantage that grows with network size. PerProp thus offers a unified approach to scalable, local learning in digital and physical neural networks.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
backpropagation-free learning, physical neural networks, gradient estimation, credit assignment, perturbation propagation, alternatives to backpropagation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_TLPFOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PerProp retains BP's training structure but measures neuron sensitivity using paired perturbations and learns feedback connections from random input–response correlations. It targets physical networks where derivatives and copied feedback weights are difficult to obtain. The authors evaluate Transformers, Mamba, non-differentiable and spiking units, and implement on-chip learning on BrainScaleS-2. Crossbar energy results are projections.

Critical strengths

Three-seed GPT/Mamba evaluations show better in-domain and out-of-domain performance than the evaluated alternatives without access to transposed forward weights, with Mamba loss close to BP. Results across several spiking units support applicability. Actual on-chip updates and improvement over R-STDP under matched BrainScaleS-2 conditions provide evidence beyond simulation.

Critical weaknesses

1.Insufficient comparison with closely related feedback-learning methods. The paper discusses Akrout et al. (2019) but does not evaluate it, leaving the additional benefit unclear. A direct comparison on one representative architecture is needed, with matched training data and tuning budgets, reporting predictive performance and runtime including feedback learning.

2.The contributions of the key modifications remain unclear. BP, Hybrid, and PerProp provide overall controls, but do not isolate the benefits of direct linear probing and coordinate-wise normalization. Controlled ablations should report validation loss and feedback alignment under matched probe and update budgets.

3.The faster-scaling claims are affected by jointly changing training resources. Larger batches process more tokens, while deeper models receive more updates and different learning rates; each configuration uses only one seed. Comparisons at fixed total tokens for batch size and fixed compute for depth, with multiple seeds on representative configurations, are needed.

4.Evidence that a fixed probe budget supports wider networks is insufficient. Feedback estimation error grows with input dimension, and the width sweep shows a widening loss gap to BP. Varying probe count and feedback learning rate at two or three widths, while reporting feedback error, validation loss, and measured runtime, would clarify this limitation.

Questions for authors

Please see the weaknesses above.

5 comments
AuthorsOpenReview ↗

Response to reviewer TLPF

We thank the reviewer for a precise review and for noting that "three-seed GPT/Mamba evaluations show better in-domain and out-of-domain performance than the evaluated alternatives without access to transposed forward weights", that results "across several spiking units support applicability", and that "actual on-chip updates and improvement over R-STDP under matched BrainScaleS-2 conditions provide evidence beyond simulation". We ran each requested comparison.

Several requests concern conventions specific to in-situ learning, so we state three briefly first.

  • (i) The learning signal must be measured on the device, because its derivatives and weights are not available to a computer in closed form; small perturbations and separate probe phases are the standard way to do this (weight and node perturbation; the nudged phase of equilibrium propagation).
  • (ii) On a physical substrate, probes and feedback updates are analog operations of the same array, so we report operation counts and energy alongside simulator runtime.
  • (iii) When devices at the relevant scale do not yet exist, energy is reported from device and circuit models, as is standard in the analog-learning literature (e.g. Yi et al., Nature Electronics 2023).

W1. "Insufficient comparison with closely related feedback-learning methods. The paper discusses Akrout et al. (2019) but does not evaluate it, leaving the additional benefit unclear. A direct comparison on one representative architecture is needed, with matched training data and tuning budgets, reporting predictive performance and runtime including feedback learning."

Response. This request was very useful, and we ran it on two architectures rather than one. We now benchmark both of Akrout et al.'s rules, weight mirror (WM) and Kolen–Pollack (KP), on GPT-24L and Mamba-2-24L (WikiText-103, plus TinyStories out of distribution), 3 seeds, same data and same learning-rate pilot, at matched probe budget and at Akrout et al.'s own configuration. Validation NLL, and full step time including feedback learning (A6000-equivalent):

RuleGPT-24LMamba-2-24LStep ms, GPT / Mamba-2
BP (reference)4.915 ± 0.0224.461 ± 0.017169 / 181
PerProp5.579 ± 0.0234.546 ± 0.019210 / 450
WM, matched probes5.686 ± 0.0234.734 ± 0.022208 / 449
WM, Akrout et al. config (21× probes)5.652 ± 0.0224.678 ± 0.027223 / 601
KP, pilot-selected decay5.669 ± 0.0184.998 ± 0.018207 / 442
KP, Akrout et al. decay 1e-45.599 ± 0.0104.892 ± 0.021207 / 442

PerProp has the lowest loss among the rules without weight transport in every column, also out of distribution (TinyStories, GPT / Mamba-2: PerProp 7.177 / 6.653; full out-of-distribution columns in the general response, First). WM trails by 0.07–0.19 nats even with 21× the probes, which cost +6% / +34% step time. KP ties PerProp on GPT in distribution at Akrout et al.'s decay but trails by 0.35 nats on Mamba-2 and degrades out of distribution (+0.038 on GPT, +0.360 on Mamba-2). Both rules, as published, need analytic derivatives of the neuron nonlinearity, which physical and non-differentiable units do not provide; PerProp measures them with the same probing operation it uses for feedback. Step times are A6000-equivalent, calibrated through the BP/PerProp ratio rather than measured on a dedicated card.

For the spiking (SHD) benchmark in the paper, the Kolen–Pollack baseline uses weight decay 2e-3; we will state this setting in the revision.

AuthorsOpenReview ↗

Response to reviewer TLPF (cont.)

W2. "The contributions of the key modifications remain unclear. BP, Hybrid, and PerProp provide overall controls, but do not isolate the benefits of direct linear probing and coordinate-wise normalization. Controlled ablations should report validation loss and feedback alignment under matched probe and update budgets."

Response. We welcome this request. We ran one-change ablations, with the same probes and the same update budget in every arm, first on language models and then on a spiking network.

Language models (setting of W1; m = 96, ρ = 0.1, one feedback update per step):

  • Normalisation removed: no change in loss or alignment (GPT 5.578 vs 5.579, alignment 0.914 vs 0.913; Mamba-2 4.549 vs 4.546, 0.940 vs 0.939). Our variance analysis predicts this for isotropic Gaussian probes.
  • Direct linear probing replaced by reading after the nonlinearity (the weight-mirror readout): +0.11 nats on GPT and +0.19 on Mamba-2, at alignment 0.910 / 0.924. Direct linear probing carries the result on these architectures.

Spiking network (SHD, recurrent adaptive LIF, 256 units per layer, 60 epochs; 3 seeds at 1–2 layers, 6 at 4 layers), where probe-driven spike differences cannot be isotropic:

ArmTest %, L = 1L = 2L = 4 (runs that train)L = 4 feedback error ‖B − Wᵀ‖/‖Wᵀ‖
PerProp86.1 ± 0.785.3 ± 0.562.3 ± 21.1 (4/6)0.092
one scalar denominator85.6 ± 0.985.1 ± 1.944.4 ± 24.3 (1/6)0.401
read after nonlinearity86.3 ± 0.386.0 ± 0.9——

Normalisation matters here. Without the per-coordinate denominator, feedback error rises 3.6–4.4×, and at 4 layers test accuracy drops by 17.9 points, paired, with PerProp higher on 5 of 6 seeds. Reading after the nonlinearity matches PerProp at 1 and 2 layers on SHD (86.3 vs 86.1, 86.0 vs 85.3) but costs 1.45–1.48× the multiply-accumulates, and on language models it loses 0.11–0.19 nats.

Both components are necessary. Direct linear probing carries the result on language models. Per-coordinate normalisation is inert when probe responses are isotropic, as our variance analysis predicts and the language-model ablation confirms, because it then reduces to a scalar denominator. It becomes essential when responses are not isotropic, as on the spiking network above, and physical devices with mismatched gains produce anisotropic responses in the same way. Each component costs nothing where it is not needed and is required where it is.

W3. "The faster-scaling claims are affected by jointly changing training resources. Larger batches process more tokens, while deeper models receive more updates and different learning rates; each configuration uses only one seed. Comparisons at fixed total tokens for batch size and fixed compute for depth, with multiple seeds on representative configurations, are needed."

Response. We take this concern seriously. Our protocol follows standard practice for single-axis scaling studies. Energy-Based Transformers (Gladstone et al., ICLR 2026), which compare the scaling rates of two training methods along data, batch size, depth and parameters, run the batch-size experiments "for 105k steps", so larger batches also see more tokens. Letting data grow with the scaled resource is also the basis of compute-optimal scaling (Kaplan et al., 2020; Hoffmann et al., 2022). Most importantly, every comparison in our study is paired: at each point PerProp and backprop use the same configuration, data order and token count. Any resource change along an axis applies equally to both, and the quantity we report is their relative scaling.

We also address the concern directly.

  • (a) A refit of the published curves with one shared floor and exponent shows PerProp behaving like backprop at a constant resource multiplier (k = 1.99 in tokens, 2.58 in model size), so the gap narrows with scale.
  • (b) With each method at its own tuned settings, PerProp reaches backprop's loss with 1.23×, 1.25× and 1.22× the FLOPs at 20.8M, 49.6M and 162.9M parameters, a constant overhead of about 1.2× rather than a growing one (general response, Second). Backprop's seed SD at these settings is 0.002–0.004 nats; across both methods it is 0.002–0.009 nats, 4–20× smaller than the tuned gaps.
AuthorsOpenReview ↗

Response to reviewer TLPF (cont.)

W4. "Evidence that a fixed probe budget supports wider networks is insufficient. Feedback estimation error grows with input dimension, and the width sweep shows a widening loss gap to BP. Varying probe count and feedback learning rate at two or three widths, while reporting feedback error, validation loss, and measured runtime, would clarify this limitation."

Response. We agree that the published width sweep used a fixed m = 96, and this explains the widening gap. The feedback error follows a closed-form law in probe count m, input width d_in and rate ρ: ‖B − Wᵀ‖²/‖W‖² ≈ ρ/(2 − ρ)·(d_in − 1)/(m − 2). Measured over predicted is 1.000 (0.997–1.003) on all 20 published runs. So m must grow with width to hold the error fixed; the experiments below test exactly this.

Feedback error, 4 widths × 5 probe counts × 2 rates × 3 seeds: at fixed m/d the error is flat to 0.2% from d = 192 to 1536 (ρ = 0.2: 0.375 at m/d = 1, 0.265 at 2, 0.188 at 4). At fixed m = 96 it rises from 0.53 to 1.51. All 34 configurations match the law (0.993–1.002).

Validation loss. Published, m = 96: gap 0.31 / 0.38 / 0.42 / 0.51 nats at d = 192 / 384 / 768 / 1536. Trained with m = 1536 (m/d = 8 / 4 / 2) and tuned LRs: 0.037 / 0.042 / 0.036 nats at d = 192 / 384 / 768. At d = 384, varying m (16–384) and ρ (0.02–0.5) moves the loss as the law predicts (see fHyc W3 for the table).

Runtime. Feedback arithmetic is 2m/(3T) of a step: 0.20% at m = 96 and 3.1% at m = 1536; the tuned campaign measured 1.03× arithmetic per step. Measured full-step time on language models is in W1. Measured on one dedicated H200 (6 layers, sequence 1024), a PerProp step takes 1.33–1.43× backprop's step time from d = 192 to 1536. Feedback learning is 0.25–1.19% of the step at m = 96 and 1.1–3.7% with m = 4d, so the overhead comes from the slope-measurement path, not from feedback learning. On analog hardware the probe cost is energy: at m = 1536, feedback is 1.7–6.1% of PerProp's projected training energy at 0.1 pJ per product.

We thank the reviewer. Each of the four requests now has a direct experiment: KP and WM on two architectures with runtime (W1), one-change ablations on language and spiking models (W2), tuned scaling and the published protocol in context (W3), and probe count and rate against width with a law that predicts the error to within 1% (W4). We hope this resolves the concerns that led to the current score, and we would be grateful if the reviewer would consider raising it.

References

  • Akrout, M., Wilson, C., Humphreys, P. C., Lillicrap, T., Tweed, D. Deep Learning without Weight Transport. NeurIPS 2019. arXiv:1904.05391.
  • Gladstone, A., et al. Energy-Based Transformers are Scalable Learners and Thinkers. ICLR 2026. arXiv:2507.02092.
  • Kaplan, J., McCandlish, S., Henighan, T., et al. Scaling Laws for Neural Language Models. arXiv:2001.08361, 2020.
  • Hoffmann, J., Borgeaud, S., Mensch, A., et al. Training Compute-Optimal Large Language Models. NeurIPS 2022. arXiv:2203.15556.
  • Yi, S.-i., Kendall, J. D., Williams, R. S., Kumar, S. Activity-difference training of deep neural networks using memristor crossbars. Nature Electronics 6, 45–51 (2023). doi:10.1038/s41928-022-00869-w.
Reviewer_fHycOpenReview ↗
Rating
2
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PerProp, a learning algorithm that measures neuronal activation derivatives through local perturbations and learns feedback weights using normalized input–output correlations. The method preserves the layer-wise error propagation structure while aiming to eliminate the need for weight transport. The authors evaluate PerProp on a variety of tasks and architectures, including language models, vision models, non-differentiable neurons, and neuromorphic hardware.

Critical strengths
  1. The proposed method is broadly applicable to a variety of neuron models and activation functions.
  2. The method is evaluated across a diverse range of architectures, including Transformer, Mamba, CNN, SNN.
  3. The paper provides a theoretical analysis of PerProp's consistency with BP.
Critical weaknesses
  1. The proposed algorithm is a combination of the methods in [1] and [2], and its conceptual novelty seems incremental rather than substantial.

  2. The experiments on the BrainScaleS-2 neuromorphic hardware are conducted at relatively small model and dataset scales. And the paper does not report the direct comparison of measured power consumption against baseline, leaving the practical energy-efficiency benefits unclear.

  3. The paper includes a Hybrid comparison. However, it does not systematically evaluate how the number of probes, probe distribution, perturbation magnitude, or feedback update rate affect performance.

[1] Direct training of snn using local zeroth order method. Advances in Neural Information Processing Systems, 2023.

[2] Deep learning without weight transport. Advances in Neural Information Processing Systems, 2019.

Questions for authors
  1. How to explain "PerProp exhibits steeper scaling slopes than BP along the data, batch size, and depth axes"? In particular, what is the underlying reason for this observation?

  2. How sensitive is PerProp to the number and distribution of probes, perturbation magnitude, and feedback update frequency? How do these choices affect accuracy, feedback alignment, and training time?

8 comments
AuthorsOpenReview ↗

Response to reviewer fHyc

We thank the reviewer for the thoughtful assessment, and we are encouraged that the reviewer finds PerProp "broadly applicable to a variety of neuron models and activation functions", evaluated "across a diverse range of architectures", and supported by "a theoretical analysis of PerProp's consistency with BP". Each answer below starts with the short answer.

W1. "The proposed algorithm is a combination of the methods in [1] and [2], and its conceptual novelty seems incremental rather than substantial."

Response. We appreciate this question, which showed us that our original text did not explain the relation to [1] and [2] clearly enough; we have added a dedicated paragraph to the revision. In short, PerProp shares one thing with [2]: both learn their feedback. Beyond that they have nothing in common: they learn different targets, by different rules. PerProp also obtains its slopes differently from [1], and it obtains slopes and feedback with one operation, which neither method has. Running "[1] + [2]" directly performs worse than PerProp. We take these points in turn; the general response (First) gives the same comparison with the full table.

LocalZO [1] constructs a stochastic surrogate for the spike derivative; PerProp directly measures the implemented nonlinearity's local response. LocalZO evaluates the Heaviside function at randomly displaced inputs, with the perturbation distribution determining the surrogate gradient (Eqs. 5–6). PerProp applies paired displacements to the implemented nonlinearity and reads its finite-difference response, so when the device response departs from the ideal model, the measured slope follows the actual response rather than the assumed model. Adding weight mirror to the original LocalZO rule changes the feedback pathway; it does not by itself replace that threshold-function estimator with a measurement of the implemented device. PerProp additionally probes linear-layer responses to learn scale-normalized feedback mappings.

Weight mirror [2]: both learn feedback, but the target and the rule differ.

  • Target. PerProp's feedback converges in expectation to Wᵀ itself, scale included: E[B] = Wᵀ. Its error follows a closed-form law in probe count, width and EMA rate, which matches every measured run to within 1%. The weight mirror's expected update is E[ΔBₗ₊₁] ≈ η_B σ² Wₗ₊₁ᵀ φ′(bₗ₊₁) (Akrout et al., Eqs. 9–11): Wᵀ with its columns rescaled by the activation derivative at the bias. Only when φ′(bₗ₊₁) ≈ φ′ₛ I (or with bias blocking, Eq. 12) is it a scalar multiple, η_B σ² φ′ₛ Wᵀ, and weight decay then holds B at an unspecified scale.
  • Rule. PerProp injects probes at a layer's input and reads the linear response before the nonlinearity, with per-coordinate normalisation. The weight mirror drives the network with noise in a separate mirror phase and updates B with a Hebbian rule, and its learning step needs the analytic derivative φ′, which a physical device does not provide.

One operation for both slopes and feedback. PerProp measures the slopes and learns the feedback with the same operation: inject a small perturbation at a layer's input and read the forward response. The diagonal response gives the slope; the linear response gives the feedback. It is not a slope estimator stacked on a feedback rule. On an analog chip this means one perturb-and-read primitive on one kind of array, used only in the forward direction: no circuitry for analytic derivatives, no transposed read, and no separate mirror mode with its own noise source and Hebbian rule. Fewer kinds of components simplify design, calibration and chip area, which are the binding constraints of analog chips. Neither [1] nor [2] has this property.

Running "[1] + [2]". To test the combination directly, we kept the same measured slopes and changed only the feedback rule (GPT-24L and Mamba-2-24L, WikiText-103, 3 seeds). Validation NLL, GPT / Mamba-2: PerProp 5.579 / 4.546; weight mirror 5.686 / 4.734 (5.652 / 4.678 with 21× more probes); Kolen–Pollack 5.599 / 4.892. PerProp is best in all four in- and out-of-distribution columns (full table in the general response). The combination of [1] and [2] is therefore not PerProp, and it does not reach PerProp's loss.

AuthorsOpenReview ↗

Response to reviewer fHyc (cont.)

W2. "The experiments on the BrainScaleS-2 neuromorphic hardware are conducted at relatively small model and dataset scales. And the paper does not report the direct comparison of measured power consumption against baseline."

Response. The chip run shows that learning can run fully on the chip; the evidence for scale is GPT, Mamba-2 and ConvNeXt. The general response (Third) describes the size limits of a single BrainScaleS-2 chip, a new frozen-hidden-layer control on the chip, and the energy comparisons on the same chip and for analog crossbars; the points that bear on this question are below.

  • Why the chip run is small. One BrainScaleS-2 chip has only 512 neurons and a 16 KiB on-chip processor. Larger BrainScaleS-2 trainings compute the update on an external host. In our run every update, including feedback learning, runs on the chip. Within this budget, the new control shows that the hidden layer learns on the chip: 62.1 ± 1.8% when PerProp trains it against 58.8 ± 1.6% when it is frozen (paired +3.3 ± 1.6 points, positive in every seed).
  • Measured energy against baselines on the same chip. The chip has no power meter, so energy is chip power × logged chip time. On the same chip and task, PerProp uses about 51× less energy per update than host-in-the-loop training (from our measured 1.85 s round trip). At equal chip time it reaches 62.1 ± 1.8% against 36.9 ± 1.8% for R-STDP.
  • Energy where it matters. BrainScaleS-2 was built for fast emulation, not for low energy. For energy-oriented analog crossbars, our component-level projection at matched perplexity is 16–27× below backprop on an H100; such projections are standard practice before a full training chip is taped out, and our table states its assumptions.
AuthorsOpenReview ↗

Response to reviewer fHyc (cont.)

W3 / Q2. "[The paper] does not systematically evaluate how the number of probes, probe distribution, perturbation magnitude, or feedback update rate affect performance."

Response. We welcome this suggestion and ran it. Perturbation size and probe distribution do not matter. Probe count and EMA rate matter exactly as our closed-form error law predicts. L6 Transformer (49.6M parameters), 2 seeds, one setting changed at a time; validation NLL and alignment cos(B, Wᵀ):

SettingValidation NLLAlignment cos(B, Wᵀ)
default (m = 96, Gaussian, ε = 0.01, ρ = 0.1)4.3750.91
probes m = 16 / 3845.476 / 4.1900.63 / 0.98
Rademacher / uniform probes4.359 / 4.3980.91
ε = 1e-4 / 1e-14.382 / 4.3870.91
ρ = 0.02 / 0.54.418 / 5.3810.98 / 0.64

The measured feedback error matches the law to within 0.4% in every run. Probes cost 0.2% of the arithmetic at m = 96 and 3.1% at m = 1536. The paper used untuned minimal settings; with each method at its own tuned settings, PerProp reaches backprop's loss with 1.23×, 1.25× and 1.22× the FLOPs at 20.8M, 49.6M and 162.9M parameters, an overhead of about 1.2× at every size (general response, Second).

Q1. "How to explain 'PerProp exhibits steeper scaling slopes than BP along the data, batch size, and depth axes'?"

Response. A steeper slope means the gap to backprop shrinks with scale. Both methods approach the same loss floor, and a curve that starts higher above the floor falls faster. A refit with one shared floor and exponent reproduces the published curves when PerProp behaves like backprop with 2.0× fewer tokens or 2.6× fewer parameters. We will state the claim as "the gap to backprop narrows with data and model size".

We hope these new experiments address the reviewer's concerns. If they do, we would be grateful if the reviewer would consider raising the score, and we would gladly provide any further detail.


References

  • Mukhoty, B., Bojković, V., de Vazelhes, W., Zhao, X., De Masi, G., Xiong, H., Gu, B. Direct Training of SNN using Local Zeroth Order Method (LocalZO). NeurIPS 2023.
  • Akrout, M., Wilson, C., Humphreys, P. C., Lillicrap, T., Tweed, D. Deep Learning without Weight Transport. NeurIPS 2019. arXiv:1904.05391.

Comments on the paper

9 comments
AuthorsOpenReview ↗

Response to all

We thank the reviewers for their thoughtful feedback and helpful suggestions. We are glad that the reviewers find PerProp broadly applicable across architectures and neuron models (Reviewers fHyc, tDKE, TLPF), that our on-chip experiment shows learning on real hardware (Reviewers tDKE, TLPF), and that the theoretical analysis (Reviewer fHyc) and the Hybrid ablation (Reviewer tDKE) make PerProp's relation to backprop easy to check.

The primary points raised relate to (1) how PerProp differs from existing feedback-learning methods and how it compares with them empirically, (2) the compute overhead and the scaling of the gap to backprop, and (3) the scale and energy evidence of the on-chip experiment. We address each below, and all new results will be added to the revision.

First, the relation to LocalZO [1] and to the weight mirror and Kolen–Pollack rules of Akrout et al. [2] (Reviewers fHyc, tDKE, TLPF). PerProp shares one property with these methods: like the weight mirror and Kolen–Pollack, it learns its feedback weights. Beyond that, the methods differ in what they compute and in how they compute it.

  • Slopes: LocalZO constructs a stochastic surrogate for the spike derivative; PerProp directly measures the implemented nonlinearity's local response. LocalZO evaluates the Heaviside function at randomly displaced inputs, with the perturbation distribution determining the surrogate gradient (Eqs. 5–6). PerProp applies paired displacements to the implemented nonlinearity and reads its finite-difference response, so when the device response departs from the ideal model, the measured slope follows the actual response rather than the assumed model. Adding weight mirror to the original LocalZO rule changes the feedback pathway; it does not by itself replace that threshold-function estimator with a measurement of the implemented device. PerProp additionally probes linear-layer responses to learn scale-normalized feedback mappings.
  • Feedback: the weight mirror learns a rescaled transpose; PerProp learns the exact Wᵀ. The weight mirror drives layer l with noise of variance σ² in a separate mirror phase and updates Bₗ₊₁ by the Hebbian rule ΔBₗ₊₁ = η_B yₗ yₗ₊₁ᵀ. Its expected update is E[ΔBₗ₊₁] ≈ η_B σ² Wₗ₊₁ᵀ φ′(bₗ₊₁) (Akrout et al., Eqs. 9–11): Wᵀ with its columns rescaled by the activation derivative at the bias. Only under the further assumption φ′(bₗ₊₁) ≈ φ′ₛ I (or with bias blocking, Eq. 12) does this become a scalar multiple, η_B σ² φ′ₛ Wᵀ, and weight decay then holds B at an unspecified scale. The weight mirror therefore learns E[yₗ yₗᵀ] Wₗ₊₁ᵀ φ′(bₗ₊₁) = σ² Wₗ₊₁ᵀ φ′(bₗ₊₁), the transpose multiplied by the noise variance and the activation derivative. PerProp's feedback, in contrast, converges in expectation to Wᵀ itself, and its error follows a closed-form law that matches every measured run to within 1%.
  • One operation for both. PerProp measures the slopes and learns the feedback with the same operation: perturb a layer's input and read the forward response. An analog chip therefore needs only one perturb-and-read primitive on one kind of array, used in the forward direction, which reduces the kinds of components the chip must provide.

Motivated by the reviewers' suggestions, we benchmarked both of Akrout et al.'s rules on GPT-24L and Mamba-2-24L (WikiText-103, with TinyStories as the out-of-distribution set; 3 seeds; same data and learning-rate pilot), both at a matched probe budget and at Akrout et al.'s own configuration. All runs use PerProp's measured slopes, and only the feedback rule changes, so the comparison isolates the feedback rule. PerProp has the lowest validation loss in every column:

RuleGPT-24LMamba-2-24LGPT / Mamba-2, TinyStories
PerProp5.579 ± 0.0234.546 ± 0.0197.177 / 6.653
Weight mirror, matched probes5.686 ± 0.0234.734 ± 0.0227.229 / 6.868
Weight mirror, Akrout et al. config (21× probes)5.652 ± 0.0224.678 ± 0.0277.213 / 6.843
Kolen–Pollack, pilot-selected decay5.669 ± 0.0184.998 ± 0.0187.185 / 6.933
Kolen–Pollack, Akrout et al. decay 1e-45.599 ± 0.0104.892 ± 0.0217.215 / 7.013
AuthorsOpenReview ↗

Response to all (cont.)

Second, compute overhead and scaling (Reviewers tDKE, TLPF, fHyc). We agree that our models are small by current standards and that PerProp does not yet match backprop in absolute loss. Our focus is the scaling behaviour, namely whether the gap to backprop grows or shrinks with scale, which is where backprop alternatives have so far fallen short (e.g. for direct feedback alignment the gap "widens as compute budget and model scale increases", Filipovich et al., 2022). We follow the scaling methodology of Energy-Based Transformers and Kaplan et al. Every run in the paper used the learning rate that is optimal for backprop, while PerProp's own settings (probe count, perturbation size, EMA rate) were left at minimal untuned defaults, including m = 96 probes at every width. This favoured the baseline and explains the ~10× compute gap at the largest size. Following the reviewers' suggestions, we have now tuned these settings:

  • With each method at its own tuned settings, PerProp reaches backprop's loss with 1.23×, 1.25× and 1.22× the FLOPs at 20.8M, 49.6M and 162.9M parameters: the overhead stays constant rather than growing.
  • The feedback error follows a closed-form law, ‖B − Wᵀ‖²/‖W‖² ≈ ρ/(2 − ρ)·(d_in − 1)/(m − 2), so a fixed probe count loses accuracy as width grows. With probes scaled with width, the error stays flat to 0.2% from d = 192 to 1536, and the trained loss gap stays at 0.036–0.042 nats instead of rising from 0.31 to 0.51.
  • With added seeds, the seed-to-seed SD is 0.002–0.009 nats for both methods, 4–20× smaller than the tuned gaps (e.g. tuned L3, 3 seeds: PerProp 4.2100 ± 0.0060, backprop 4.1706 ± 0.0044).
  • On a dedicated H200, a PerProp step takes 1.33–1.43× backprop's step time, and feedback learning accounts for at most 3.7% of the step.

We have also revised "steeper scaling slopes" to "the gap to backprop narrows with data and model size", which a refit with a shared loss floor supports.

Third, the chip experiment and energy (Reviewers fHyc, tDKE). One BrainScaleS-2 chip has 512 neurons and a 16 KiB on-chip processor, and larger BrainScaleS-2 trainings compute the update on a host. In our run every update, including feedback learning, runs on the chip, so we present it as a demonstration of fully on-chip learning; the evidence for scale is the GPT, Mamba-2 and ConvNeXt experiments in the paper.

  • As kindly suggested by Reviewer tDKE, we ran a frozen-hidden-layer control on the chip itself: 58.8 ± 1.6% with the hidden layer frozen against 62.1 ± 1.8% when PerProp trains it (paired +3.3 ± 1.6 points, positive in every seed). On a calibrated model of the same network at 40 epochs the hidden layer adds 5.5 points (79.5 ± 0.4% vs 74.0 ± 2.1%).
  • On the same chip and task, PerProp uses about 51× less energy per update than host-in-the-loop training (estimated from our measured host round trip), and at equal chip time it reaches 62.1% against 36.9% for R-STDP.
  • For energy-oriented analog crossbars, our component-level projection at matched perplexity is 16–27× below backprop on an H100. Reporting such projections before a full training chip is taped out is standard practice in the analog-learning literature, and our table states its assumptions.

We hope these experiments and clarifications address the reviewers' main concerns. We thank the reviewers again for their constructive suggestions, which have strengthened the paper, and we would be happy to incorporate any further suggestions or experiments.


References

  • Mukhoty, B., Bojković, V., de Vazelhes, W., Zhao, X., De Masi, G., Xiong, H., Gu, B. Direct Training of SNN using Local Zeroth Order Method (LocalZO). NeurIPS 2023.
  • Akrout, M., Wilson, C., Humphreys, P. C., Lillicrap, T., Tweed, D. Deep Learning without Weight Transport. NeurIPS 2019. arXiv:1904.05391.
  • Filipovich, M. J., Cappelli, A., Hesslow, D., Launay, J. Scaling Laws Beyond Backpropagation. NeurIPS 2022 Workshop. arXiv:2210.14593.
  • Gladstone, A., et al. Energy-Based Transformers are Scalable Learners and Thinkers. ICLR 2026. arXiv:2507.02092.
  • Kaplan, J., McCandlish, S., Henighan, T., et al. Scaling Laws for Neural Language Models. arXiv:2001.08361, 2020.
AuthorsOpenReview ↗

Response to reviewer tDKE

We thank the reviewer for a clear and constructive review, and for noting that "the same learning rule works across many architecture classes" and "runs end-to-end on a physical neuromorphic chip... with no redesign for each architecture or manually chosen surrogate" (S1), and that the Hybrid ablation "makes it easy to check where PerProp's gap to backprop comes from" (S2). We ran every experiment the reviewer suggested that our compute allowed.

W1. "Transformers are tested only up to 0.4B parameters, with five single-seed points. At the largest tested scales, PerProp needs about 10× more compute than backprop... Also, the loss gap to backprop widens with width... It would help if the authors provided a width sweep with more probes per layer, the compute gap reported at each scale, and multiple seeds."

Q2. "If the number of probes is scaled with layer width, does the loss gap to backprop stop growing? And does the ~10× compute gap change across model sizes?"

Response. We are grateful for these helpful suggestions. We agree that 0.4B parameters is not large by the standards of recent foundation models, and that PerProp still trails backprop in absolute loss. What our study is designed to measure is the scaling behaviour, namely how the gap to backprop changes with scale, rather than absolute performance at a single size. This is where backprop alternatives have so far fallen short: for direct feedback alignment, for example, "the gap with BP widens as compute budget and model scale increases" (Filipovich et al., 2022). We ask whether PerProp's gap grows in the same way. Below we first explain how we study scale, then answer the three requests: the compute gap at each scale (about 1.2× at every size once each method uses its own tuned settings), width-scaled probes (the loss gap stops growing), and added seeds. The general response (Second) summarises the same results.

How we study scale. For this question we follow established standards. Our scaling study follows the standard of Energy-Based Transformers (Gladstone et al., 2025), which compares two training methods by their scaling rates: EBT trains several model sizes and compares "scaling rate[s]" along data, batch size, depth, parameters, FLOPs and width. Mamba (Gu and Dao, 2024) compares architectures the same way, by scaling curves over model sizes from about 125M to 1.3B parameters. EBT's scaling method follows Kaplan et al. (2020), who vary depth, width and data and find that loss "scales as a power-law with model size, dataset size, and the amount of compute", and the survey of Li et al. (2025), which shows that a scaling conclusion can change with small changes to the setup. EBT's answer to that fragility, and ours, is to measure many sizes along each axis and compare trends. A single large run cannot give a slope. This matters for methods without weight transport, as the DFA result above shows.

Compute gap at each scale. The reviewer's arithmetic is right for the submitted runs (backprop 163M at 1.457e18 FLOPs against PerProp 408M at 14.79e18). Every published run kept PerProp's own settings at untuned minimal defaults, including a fixed m = 96 probes at every width, while the learning rate was backprop's optimum. With these untuned settings, the FLOPs multiple to reach the same perplexity is 3.19× (L12 d384), 6.36× (L12 d768) and 10.87× (0.4B). With each method at its own tuned settings, it is 1.23×, 1.25× and 1.22× at 20.8M, 49.6M and 162.9M parameters. So the ~10× gap comes from the untuned settings. Tuned, the overhead stays near 1.2× and does not grow with size. At L12 it splits into 1.18× more steps and 1.03× more arithmetic per step. We will add the tuned 0.4B point in the revision.

Probes scaled with width: the gap stops growing. The feedback error follows a closed-form law, ‖B − Wᵀ‖²/‖W‖² ≈ ρ/(2 − ρ)·(d_in − 1)/(m − 2). Measured over predicted is 1.000 (range 0.997–1.003) on all 20 published runs. So at fixed m the error must grow with width. At m = 96 it rises from 0.37 (d = 192) to 1.04 (d = 1536), and the loss gap rises with it, 0.31 → 0.51 nats. The law also says how to stop this: scale m with d. We measured the error at 4 widths × 5 probe counts × 2 rates × 3 seeds. At fixed m/d it is flat to 0.2% from d = 192 to 1536 (ρ = 0.2: 0.265 at m/d = 2, 0.188 at m/d = 4). In trained models with m = 1536 (m/d = 8, 4, 2), the loss gap is 0.037, 0.042 and 0.036 nats at d = 192, 384 and 768, against 0.31–0.42 nats at m = 96. The gap no longer grows. We will add a trained run at d = 1536 with scaled m in the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.