- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether equilibrium propagation (EP), a local, energy-based learning rule compatible with neuromorphic / in-memory hardware, can train a reinforcement-learning locomotion controller. The control architecture is taken unchanged from prior work (the two-stage CPG–RES controller of Seto et al. 2025, built on CPG-RL of Bellegarda & Ijspeert 2022). Stage 1 trains a CPG policy (63-D obs → 12 Hopf-oscillator parameters) on flat ground. Stage 2 freezes it and trains a residual postural policy (87-D obs → 12 residual joint-angle derivatives) on randomized box terrain, using a simulated Unitree A1 in MuJoCo. The paper replaces the backprop-trained MLPs of PPO with layered EP networks (1024-D input, 2×768 hard-sigmoid hidden layers, one free phase plus ±β nudge phases) for both the actor and the critic. The log-std vector is updated with an analytic gradient.
The technical contribution is a PPO-derived nudging force for the actor's output layer:
- Per-relaxation-step ratio gating. The PPO probability ratio is recomputed from the evolving output state at every nudged relaxation step, and the nudge is masked when the ratio leaves an allowed range.
- Two-sided clipping (a "reverse" clip coefficient ε_rev). This bounds the ratio in the "away-from-target" direction that one of the two symmetric nudge phases always takes.
- 1/σ instead of 1/σ² scaling of the output-layer force.
- iDCT expansion of the observation (63/87-D → 1024-D), which shortens relaxation.
Validation. There are 5 seeds per stage at 10⁸ samples each. Training curves (reward, value MSE, per-update KL) are compared with a feed-forward BP-PPO baseline. A walking test (500 episodes × 3 target speeds × 6 terrain heights × 5 seeds) reports success rate, speed, actuator power and body roll/pitch statistics. A GPU-memory comparison of EP against BPTT through the same recurrent network gives 3,110 MB vs 13,486 MB (≈4.3×). Ablations (Appendix E) cover ε_rev ∈ {0.3, 0.7, 1.0}, 1/σ vs 1/σ², a static vs dynamic mask, and with/without iDCT. Appendix D.3 analyses how the nudged ratio r_nudging depends on the advantage. Code (C++/LibTorch) is in the supplement.
Critical strengths
S1. First end-to-end EP training of both policy and value networks on a contact-rich, continuous-action locomotion task, with careful empirical diagnostics. Prior EP-RL work (Kubo et al. 2022) kept a BP-trained critic and used much simpler tasks. Here the full PPO loop (actor, critic, log-std) is BP-free, and training is stable across 5 seeds in both stages. Stage-1 learning curves (Fig. 2a) essentially match BP-PPO, and the walking test shows high success rates up to H_max = 0.08 m. This matters for my assessment because it shows the energy-based/neuromorphic learning community that EP can train a controller in a non-trivial embodied RL setting, beyond supervised benchmarks. The released code makes this a reusable starting point.
S2. An honest, mechanistic look at the failure mode of naïve EP-PPO. The paper does not just report a working configuration. It documents that the straightforward nudging (Eq. 9) diverges, traces the cause to the unbounded "away-from-target" nudge (Appendix C, D.3, Figs. 5–6), and shows the corresponding KL spikes, rollbacks and a NaN collapse at ε_rev = 1.0 (Fig. 7). This kind of failure analysis is valuable and uncommon, and it is the most informative part of the paper.
Critical weaknesses
W1. The proposed nudging rule is a heuristic stabilizer whose relation to the PPO gradient is not established. The root cause of the instability is analysable, and principled alternatives are not explored.
What the instability is. In the released implementation, the output layer is linear, has no leak, and is updated by fixed-point iteration. The nudged output update is therefore an affine map of the output state with slope β·Â_t/σ_i (β·Â_t/σ_i² with the original scaling): s_out ← c − β·Â_t·(a_t − s_out)/σ_i. The iteration diverges monotonically when that slope is ≥ 1, which is the "unbounded away-from-target" drift described for (β>0, Â>0). It oscillates without converging when the slope is ≤ −1, which matches the authors' remark that ξ_out and r_nudging "keep oscillating." Equivalently, for every sample, one of the two symmetric nudge phases adds a concave term β·|Â|·(a−ξ)²/(2σ²) to the energy. The nudged energy is then well-posed only if that curvature stays below the output neuron's unit self-coupling.
With β = 0.1 and normalized advantages up to |Â| ≈ 4, the bound is violated once σ_i drops below ≈0.4 (1/σ scaling) or ≈0.63 (1/σ² scaling). This explains without further assumptions why 1/σ² is less stable than 1/σ (Fig. 8) and why instability grows as exploration noise shrinks. The paper does not state this condition. Instead it introduces a reverse clip that switches the force off once the ratio leaves (1−ε_rev, 1+ε). That turns a divergent iteration into a gated, possibly limit-cycling one.
Why this matters for soundness.
- (a) No equilibrium is guaranteed. Under gating, the "nudged equilibria" used in Eq. (7) need not exist. The gradient is then computed from the state after a fixed 20 (positive) and 10 (negative) relaxation steps. This is also an asymmetric budget that breaks the symmetric-difference estimator. EP's theoretical guarantee (equivalence to the loss gradient as β→0 at equilibrium) therefore does not apply, and the paper never measures what is actually being followed.
- (b) The method relies on finite-β bias. As β→0, the nudged ratio stays at its free-phase value, so the "dynamic" mask converges to the "static" mask. The claimed benefit of per-step ratio gating over the BP-style mask can only exist because β is finite. The method's success is therefore tied to a deviation from the EP limit, not to faithful PPO gradient estimation.
- (c) Key comparisons are missing. There is no ablation over β. There is no measurement of the alignment between the EP-PPO update and the true gradient of the clipped surrogate for the same network (e.g., cosine similarity against BPTT or recurrent backprop, which the authors already implement for the memory test). This is standard practice in the EP literature (e.g., Laborieux et al. 2021). Simple principled alternatives are also not compared: β small enough to satisfy the stability condition, advantage clipping, a leaky/saturating output layer, or holomorphic EP (Laborieux & Zenke 2022), which the paper cites.
- (d) The static-vs-dynamic ablation is confounded. As defined in Eq. (A-32), the static mask uses the one-sided PPO mask and the 1/σ² scaling, while the "dynamic" arm is the full method (two-sided, 1/σ). Fig. 9 therefore does not isolate per-step ratio gating, which is presented as contribution 2. Likewise, the ε_rev ablation is run only in stage 1 and the σ-scaling ablation only in stage 2.
This weakness weighs heavily because contribution 2 (ratio-gated nudging) is the paper's only algorithmic novelty. The architecture, task, reward and evaluation protocol are inherited from Seto et al. (2025). Without evidence about what the rule estimates, or that it is needed beyond choosing a well-posed β, the contribution reduces to a tuned configuration.
W2. The evidence does not support the strength of the "comparable performance" and "efficiency" claims, nor the significance claimed in the motivation.
- Comparability is overstated, and the baseline is under-specified.
- Body attitude. In Fig. 3(d–e), EP's mean |roll| is ≈0.14–0.16 rad, roughly 3–4× BP's ≈0.04 rad, at every H_max, with very large across-seed spread. Mean |pitch| is also roughly 1.5–2× higher. This looks like a persistent posture bias in some EP seeds, and the text quietly excludes these two metrics from the "comparable" statement.
- Hardest terrain. At H_max = 0.12 m, EP's success rate is below BP's at all three speeds (as far as I can read: ≈76/72/57% vs ≈87/85/63%), and EP's speed at 0.5 m/s is lower (≈0.38 vs ≈0.43 m/s).
- Stage-2 training. EP is markedly less sample-efficient in stage 2 (Fig. 2c: BP plateaus near 3×10⁷ samples, EP near 8×10⁷) and has several-fold higher value MSE (Fig. 2d). The text mentions only the stage-1 parity.
- Confounded comparison. Because EP and BP use different architectures, preprocessing (iDCT vs none) and an unspecified BP configuration, any gap or parity cannot be attributed to the learning rule. The cleanest control is missing: the same EP network trained with BPTT or implicit differentiation.
- The efficiency evidence does not address the claimed benefit.
- Wrong reference point. The 4.3× memory saving is relative to BPTT through the relaxation dynamics. That figure is mainly a function of the number of unrolled steps, and it is not the relevant alternative: the deployed BP baseline is a feed-forward MLP, which needs no unrolling at all. Implicit/recurrent backprop (Almeida–Pineda) also has O(1) memory in the number of steps.
- Costs not reported. Wall-clock training time, FLOPs per update (each actor mini-batch requires 30+20+10 relaxation steps on 2×768 networks), and inference latency at the 100 Hz control rate are not reported.
- Inconsistent batch size. The reported memory-test mini-batch (32,768) differs from the configured one: Table 6 and
parameters.hgive 262,144 / 4 = 65,536.
- The motivation (on-robot adaptation) is not tested, and the "feasibility" question is narrower than framed.
- Training regime. Training uses 10⁸ samples per stage with 1,024–2,048 parallel environments and up to 200 GB of host memory.
- Non-local components. The PPO loop still relies on non-local operations: GAE over full rollouts, global advantage normalization, full-dataset KL evaluation, parameter snapshots for rollback, and KL-adaptive learning rates spanning 10⁻⁶ to 10.
- No adaptation experiment. No adaptation experiment is reported, not even in simulation (e.g., pre-train, then adapt online with EP to a payload change, friction shift or actuator degradation). Such an experiment would directly test the paper's motivation.
- Narrow evaluation. The evaluation uses only the training terrain family, with H_max at most the training value. There are no out-of-distribution terrains, perturbations, other robots or tasks.
- How much the learner contributes is unclear. The CPG prior constrains the problem strongly, and the paper does not show how much the learned policies contribute beyond it (e.g., a stage-1 CPG policy without RES on uneven terrain, or a linear RES policy).
- Feasibility of BP-free locomotion is already established. BP-free training of locomotion policies has been shown by evolution strategies (Salimans et al. 2017), random search with linear policies (Mania et al. 2018), and evolved Hebbian plasticity (Najarro & Risi 2020, including quadruped locomotion). These are not discussed. The relevant question is therefore not whether a BP-free rule can train locomotion, but what EP specifically offers, and that requires the hardware-relevant metrics above.
Questions for authors
Clarifications whose answers could change my assessment:
- Stability condition and β. Do you agree that, for your linear, leak-free output layer, the nudged iteration is stable iff |β·Â_t|/σ_i < 1 (with 1/σ scaling)? Could you report training with (i) a sweep over β (e.g., 0.01, 0.03, 0.1, 0.3), and (ii) the standard one-sided PPO mask with β or advantage clipping chosen to satisfy this bound? If such a configuration trains well, the two-sided ratio gating would be unnecessary. If it does not, that would substantially strengthen contribution 2.
- Gradient alignment. For the same EP network, what is the cosine similarity (per layer, over training) between the EP-PPO weight update and the gradient of the PPO clipped surrogate computed by BPTT or implicit differentiation? Does the nudged phase actually converge, or does it oscillate? Please report the fraction of samples that converge in the nudge phases (Fig. 10 shows free-phase convergence for the policy only). Why are the positive and negative nudge budgets asymmetric (20 vs 10 steps)?
- Static-mask ablation. Please rerun Fig. 9 with a static mask that uses the same 1/σ scaling and two-sided range as the proposed method, so the comparison isolates per-step gating.
- BP baseline specification. What are the BP-PPO architecture, activations, optimizer, learning rate, mini-batch size, and KL early-stop/rollback settings? Was it tuned? Did it use iDCT inputs? Can you add the EP network trained with BPTT (which you already implement) as a learning-curve and walking-test baseline?
- Roll/pitch gap. What causes EP's ≈3–4× larger mean |roll| (Fig. 3d) and its large across-seed variance? Is it concentrated in particular seeds, i.e., a static body tilt? Please give per-seed walking-test numbers.
- Compute and memory. Please report wall-clock training time and GPU memory for (a) BP-PPO with the feed-forward baseline, (b) EP, and (c) BPTT, all at the same mini-batch size, plus inference latency per control step. Also, which mini-batch size was used in Sec. 4.2.1 (32,768 vs the configured 65,536)?
- Evaluation settings. Table 5 fixes swing clearance g_c = 0.1 m at test time, while stage-2 training samples g_c ∈ [0.15, 0.20] m. Why? Is the evaluation intentionally outside the stage-2 training distribution, and do the EP vs BP conclusions hold at in-distribution g_c?
- Safeguard reliance. How often do KL early-stop and rollback trigger for EP vs BP (Fig. 4 suggests far more often for EP)? Does EP-PPO still train without rollback, or with a narrower learning-rate range? This bears on how much of the stability comes from the nudging rule versus external trust-region machinery.
- iDCT. The iDCT followed by per-dimension normalization is a fixed linear map, so it does not change the function class of the first layer, only its parameterization and conditioning. Is the benefit equivalent to a larger first-layer learning rate or a different initialization scale? A random Gaussian projection to 1024-D would be an informative control.
Suggestions (future work, not required for my assessment):
- An in-simulation online-adaptation experiment (e.g., payload or friction change after deployment, with EP updates from a small data budget) would directly support the paper's motivation.
- A discussion and comparison with other BP-free RL approaches (ES/ARS, evolved Hebbian rules, e-prop-based RL) and with holomorphic EP would help position the contribution.
- Discuss the weight-symmetry requirement (the code uses W and Wᵀ) with respect to the targeted neuromorphic hardware.
Minor / presentation: reconcile Eqs. (5)–(6) with the implemented discrete update. Define r_nudging^(k) and the relaxation index earlier. Add a results table. Fix the inconsistent ratio-mask intervals between the main text and Appendix C. Fix typos ("MoJoCo", "the final statistics is").