acceptodds
Withdrawn from ICLR 2027

Neuromorphic Reinforcement Learning for Quadruped Locomotion Control on Uneven Terrain

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning (RL) has enabled robust quadruped locomotion over complex terrain, but most learned controllers are trained offline with backpropagation in massively parallel simulation and deployed as fixed policies, limiting adaptation to terrain variation, payload changes, actuator wear, and other real-world conditions under onboard power constraints. Local learning provides a potential path toward energy-aware on-robot adaptation by replacing global backpropagation graphs with updates driven by local neural states, making the learning rule more compatible with neuromorphic and in-memory computing substrates. This work proposes an equilibrium-propagation (EP)-based proximal policy optimization (PPO) framework for uneven-terrain quadruped locomotion. The controller combines a bio-inspired central pattern generator (CPG) policy with a residual postural adjustment policy, while replacing conventional backpropagation-trained policy and value networks with EP-enabled local learning. To train stochastic continuous-control policies with EP, we derive an EP-compatible PPO output-nudging signal and introduce a two-sided ratio clipping mechanism that stabilizes policy updates during relaxation. Experiments on a 12-DoF A1 quadruped show that the proposed controller achieves stable policy convergence in a two-stage uneven terrain locomotion task. Its locomotion performance is comparable to a backpropagation-trained PPO baseline in success rate, velocity tracking, actuator power, and body stability, while improving GPU memory efficiency by 4.3\(\times\) compared with backpropagation through time (BPTT). These results suggest that local equilibrium-based learning can support high-dimensional embodied locomotion and provide an algorithmic foundation for low-power on-robot adaptation and fine-tuning.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Neuromorphic computing, Quadruped locomotion, Deep reinforcement learning, Local learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nfyzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether equilibrium propagation (EP), a local, energy-based learning rule compatible with neuromorphic / in-memory hardware, can train a reinforcement-learning locomotion controller. The control architecture is taken unchanged from prior work (the two-stage CPG–RES controller of Seto et al. 2025, built on CPG-RL of Bellegarda & Ijspeert 2022). Stage 1 trains a CPG policy (63-D obs → 12 Hopf-oscillator parameters) on flat ground. Stage 2 freezes it and trains a residual postural policy (87-D obs → 12 residual joint-angle derivatives) on randomized box terrain, using a simulated Unitree A1 in MuJoCo. The paper replaces the backprop-trained MLPs of PPO with layered EP networks (1024-D input, 2×768 hard-sigmoid hidden layers, one free phase plus ±β nudge phases) for both the actor and the critic. The log-std vector is updated with an analytic gradient.

The technical contribution is a PPO-derived nudging force for the actor's output layer:

  1. Per-relaxation-step ratio gating. The PPO probability ratio is recomputed from the evolving output state at every nudged relaxation step, and the nudge is masked when the ratio leaves an allowed range.
  2. Two-sided clipping (a "reverse" clip coefficient ε_rev). This bounds the ratio in the "away-from-target" direction that one of the two symmetric nudge phases always takes.
  3. 1/σ instead of 1/σ² scaling of the output-layer force.
  4. iDCT expansion of the observation (63/87-D → 1024-D), which shortens relaxation.

Validation. There are 5 seeds per stage at 10⁸ samples each. Training curves (reward, value MSE, per-update KL) are compared with a feed-forward BP-PPO baseline. A walking test (500 episodes × 3 target speeds × 6 terrain heights × 5 seeds) reports success rate, speed, actuator power and body roll/pitch statistics. A GPU-memory comparison of EP against BPTT through the same recurrent network gives 3,110 MB vs 13,486 MB (≈4.3×). Ablations (Appendix E) cover ε_rev ∈ {0.3, 0.7, 1.0}, 1/σ vs 1/σ², a static vs dynamic mask, and with/without iDCT. Appendix D.3 analyses how the nudged ratio r_nudging depends on the advantage. Code (C++/LibTorch) is in the supplement.

Critical strengths

S1. First end-to-end EP training of both policy and value networks on a contact-rich, continuous-action locomotion task, with careful empirical diagnostics. Prior EP-RL work (Kubo et al. 2022) kept a BP-trained critic and used much simpler tasks. Here the full PPO loop (actor, critic, log-std) is BP-free, and training is stable across 5 seeds in both stages. Stage-1 learning curves (Fig. 2a) essentially match BP-PPO, and the walking test shows high success rates up to H_max = 0.08 m. This matters for my assessment because it shows the energy-based/neuromorphic learning community that EP can train a controller in a non-trivial embodied RL setting, beyond supervised benchmarks. The released code makes this a reusable starting point.

S2. An honest, mechanistic look at the failure mode of naïve EP-PPO. The paper does not just report a working configuration. It documents that the straightforward nudging (Eq. 9) diverges, traces the cause to the unbounded "away-from-target" nudge (Appendix C, D.3, Figs. 5–6), and shows the corresponding KL spikes, rollbacks and a NaN collapse at ε_rev = 1.0 (Fig. 7). This kind of failure analysis is valuable and uncommon, and it is the most informative part of the paper.

Critical weaknesses

W1. The proposed nudging rule is a heuristic stabilizer whose relation to the PPO gradient is not established. The root cause of the instability is analysable, and principled alternatives are not explored.

What the instability is. In the released implementation, the output layer is linear, has no leak, and is updated by fixed-point iteration. The nudged output update is therefore an affine map of the output state with slope β·Â_t/σ_i (β·Â_t/σ_i² with the original scaling): s_out ← c − β·Â_t·(a_t − s_out)/σ_i. The iteration diverges monotonically when that slope is ≥ 1, which is the "unbounded away-from-target" drift described for (β>0, Â>0). It oscillates without converging when the slope is ≤ −1, which matches the authors' remark that ξ_out and r_nudging "keep oscillating." Equivalently, for every sample, one of the two symmetric nudge phases adds a concave term β·|Â|·(a−ξ)²/(2σ²) to the energy. The nudged energy is then well-posed only if that curvature stays below the output neuron's unit self-coupling.

With β = 0.1 and normalized advantages up to |Â| ≈ 4, the bound is violated once σ_i drops below ≈0.4 (1/σ scaling) or ≈0.63 (1/σ² scaling). This explains without further assumptions why 1/σ² is less stable than 1/σ (Fig. 8) and why instability grows as exploration noise shrinks. The paper does not state this condition. Instead it introduces a reverse clip that switches the force off once the ratio leaves (1−ε_rev, 1+ε). That turns a divergent iteration into a gated, possibly limit-cycling one.

Why this matters for soundness.

  • (a) No equilibrium is guaranteed. Under gating, the "nudged equilibria" used in Eq. (7) need not exist. The gradient is then computed from the state after a fixed 20 (positive) and 10 (negative) relaxation steps. This is also an asymmetric budget that breaks the symmetric-difference estimator. EP's theoretical guarantee (equivalence to the loss gradient as β→0 at equilibrium) therefore does not apply, and the paper never measures what is actually being followed.
  • (b) The method relies on finite-β bias. As β→0, the nudged ratio stays at its free-phase value, so the "dynamic" mask converges to the "static" mask. The claimed benefit of per-step ratio gating over the BP-style mask can only exist because β is finite. The method's success is therefore tied to a deviation from the EP limit, not to faithful PPO gradient estimation.
  • (c) Key comparisons are missing. There is no ablation over β. There is no measurement of the alignment between the EP-PPO update and the true gradient of the clipped surrogate for the same network (e.g., cosine similarity against BPTT or recurrent backprop, which the authors already implement for the memory test). This is standard practice in the EP literature (e.g., Laborieux et al. 2021). Simple principled alternatives are also not compared: β small enough to satisfy the stability condition, advantage clipping, a leaky/saturating output layer, or holomorphic EP (Laborieux & Zenke 2022), which the paper cites.
  • (d) The static-vs-dynamic ablation is confounded. As defined in Eq. (A-32), the static mask uses the one-sided PPO mask and the 1/σ² scaling, while the "dynamic" arm is the full method (two-sided, 1/σ). Fig. 9 therefore does not isolate per-step ratio gating, which is presented as contribution 2. Likewise, the ε_rev ablation is run only in stage 1 and the σ-scaling ablation only in stage 2.

This weakness weighs heavily because contribution 2 (ratio-gated nudging) is the paper's only algorithmic novelty. The architecture, task, reward and evaluation protocol are inherited from Seto et al. (2025). Without evidence about what the rule estimates, or that it is needed beyond choosing a well-posed β, the contribution reduces to a tuned configuration.

W2. The evidence does not support the strength of the "comparable performance" and "efficiency" claims, nor the significance claimed in the motivation.

  • Comparability is overstated, and the baseline is under-specified.
    • Body attitude. In Fig. 3(d–e), EP's mean |roll| is ≈0.14–0.16 rad, roughly 3–4× BP's ≈0.04 rad, at every H_max, with very large across-seed spread. Mean |pitch| is also roughly 1.5–2× higher. This looks like a persistent posture bias in some EP seeds, and the text quietly excludes these two metrics from the "comparable" statement.
    • Hardest terrain. At H_max = 0.12 m, EP's success rate is below BP's at all three speeds (as far as I can read: ≈76/72/57% vs ≈87/85/63%), and EP's speed at 0.5 m/s is lower (≈0.38 vs ≈0.43 m/s).
    • Stage-2 training. EP is markedly less sample-efficient in stage 2 (Fig. 2c: BP plateaus near 3×10⁷ samples, EP near 8×10⁷) and has several-fold higher value MSE (Fig. 2d). The text mentions only the stage-1 parity.
    • Confounded comparison. Because EP and BP use different architectures, preprocessing (iDCT vs none) and an unspecified BP configuration, any gap or parity cannot be attributed to the learning rule. The cleanest control is missing: the same EP network trained with BPTT or implicit differentiation.
  • The efficiency evidence does not address the claimed benefit.
    • Wrong reference point. The 4.3× memory saving is relative to BPTT through the relaxation dynamics. That figure is mainly a function of the number of unrolled steps, and it is not the relevant alternative: the deployed BP baseline is a feed-forward MLP, which needs no unrolling at all. Implicit/recurrent backprop (Almeida–Pineda) also has O(1) memory in the number of steps.
    • Costs not reported. Wall-clock training time, FLOPs per update (each actor mini-batch requires 30+20+10 relaxation steps on 2×768 networks), and inference latency at the 100 Hz control rate are not reported.
    • Inconsistent batch size. The reported memory-test mini-batch (32,768) differs from the configured one: Table 6 and parameters.h give 262,144 / 4 = 65,536.
  • The motivation (on-robot adaptation) is not tested, and the "feasibility" question is narrower than framed.
    • Training regime. Training uses 10⁸ samples per stage with 1,024–2,048 parallel environments and up to 200 GB of host memory.
    • Non-local components. The PPO loop still relies on non-local operations: GAE over full rollouts, global advantage normalization, full-dataset KL evaluation, parameter snapshots for rollback, and KL-adaptive learning rates spanning 10⁻⁶ to 10.
    • No adaptation experiment. No adaptation experiment is reported, not even in simulation (e.g., pre-train, then adapt online with EP to a payload change, friction shift or actuator degradation). Such an experiment would directly test the paper's motivation.
    • Narrow evaluation. The evaluation uses only the training terrain family, with H_max at most the training value. There are no out-of-distribution terrains, perturbations, other robots or tasks.
    • How much the learner contributes is unclear. The CPG prior constrains the problem strongly, and the paper does not show how much the learned policies contribute beyond it (e.g., a stage-1 CPG policy without RES on uneven terrain, or a linear RES policy).
    • Feasibility of BP-free locomotion is already established. BP-free training of locomotion policies has been shown by evolution strategies (Salimans et al. 2017), random search with linear policies (Mania et al. 2018), and evolved Hebbian plasticity (Najarro & Risi 2020, including quadruped locomotion). These are not discussed. The relevant question is therefore not whether a BP-free rule can train locomotion, but what EP specifically offers, and that requires the hardware-relevant metrics above.
Questions for authors

Clarifications whose answers could change my assessment:

  1. Stability condition and β. Do you agree that, for your linear, leak-free output layer, the nudged iteration is stable iff |β·Â_t|/σ_i < 1 (with 1/σ scaling)? Could you report training with (i) a sweep over β (e.g., 0.01, 0.03, 0.1, 0.3), and (ii) the standard one-sided PPO mask with β or advantage clipping chosen to satisfy this bound? If such a configuration trains well, the two-sided ratio gating would be unnecessary. If it does not, that would substantially strengthen contribution 2.
  2. Gradient alignment. For the same EP network, what is the cosine similarity (per layer, over training) between the EP-PPO weight update and the gradient of the PPO clipped surrogate computed by BPTT or implicit differentiation? Does the nudged phase actually converge, or does it oscillate? Please report the fraction of samples that converge in the nudge phases (Fig. 10 shows free-phase convergence for the policy only). Why are the positive and negative nudge budgets asymmetric (20 vs 10 steps)?
  3. Static-mask ablation. Please rerun Fig. 9 with a static mask that uses the same 1/σ scaling and two-sided range as the proposed method, so the comparison isolates per-step gating.
  4. BP baseline specification. What are the BP-PPO architecture, activations, optimizer, learning rate, mini-batch size, and KL early-stop/rollback settings? Was it tuned? Did it use iDCT inputs? Can you add the EP network trained with BPTT (which you already implement) as a learning-curve and walking-test baseline?
  5. Roll/pitch gap. What causes EP's ≈3–4× larger mean |roll| (Fig. 3d) and its large across-seed variance? Is it concentrated in particular seeds, i.e., a static body tilt? Please give per-seed walking-test numbers.
  6. Compute and memory. Please report wall-clock training time and GPU memory for (a) BP-PPO with the feed-forward baseline, (b) EP, and (c) BPTT, all at the same mini-batch size, plus inference latency per control step. Also, which mini-batch size was used in Sec. 4.2.1 (32,768 vs the configured 65,536)?
  7. Evaluation settings. Table 5 fixes swing clearance g_c = 0.1 m at test time, while stage-2 training samples g_c ∈ [0.15, 0.20] m. Why? Is the evaluation intentionally outside the stage-2 training distribution, and do the EP vs BP conclusions hold at in-distribution g_c?
  8. Safeguard reliance. How often do KL early-stop and rollback trigger for EP vs BP (Fig. 4 suggests far more often for EP)? Does EP-PPO still train without rollback, or with a narrower learning-rate range? This bears on how much of the stability comes from the nudging rule versus external trust-region machinery.
  9. iDCT. The iDCT followed by per-dimension normalization is a fixed linear map, so it does not change the function class of the first layer, only its parameterization and conditioning. Is the benefit equivalent to a larger first-layer learning rate or a different initialization scale? A random Gaussian projection to 1024-D would be an informative control.

Suggestions (future work, not required for my assessment):

  • An in-simulation online-adaptation experiment (e.g., payload or friction change after deployment, with EP updates from a small data budget) would directly support the paper's motivation.
  • A discussion and comparison with other BP-free RL approaches (ES/ARS, evolved Hebbian rules, e-prop-based RL) and with holomorphic EP would help position the contribution.
  • Discuss the weight-symmetry requirement (the code uses W and Wᵀ) with respect to the targeted neuromorphic hardware.

Minor / presentation: reconcile Eqs. (5)–(6) with the implemented discrete update. Define r_nudging^(k) and the relaxation index earlier. Add a results table. Fix the inconsistent ratio-mask intervals between the main text and Appendix C. Fix typos ("MoJoCo", "the final statistics is").

Reviewer_45gTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper trains PPO with equilibrium propagation (EP) instead of backpropagation (BP), motivated by the potential to be useful for low power on-robot learning with neuromorphic hardware. They train a Unitree A1 in MuJoCo with first a CPG policy on flat ground, then a residual policy on box terrain, with forward velocity commands up to 0.5 m/s. To make EP work with PPO, the authors recompute the PPO ratio at every relaxation step to gate the nudging force, and add multiple tricks to make it more stable. The EP policies reach a similar reward as the BP baseline, and the EP update needs about 4× less VRAM than BPTT.

Critical strengths

I was previously not aware of equilibrium propagation and it seems like the first time it was applied to a setting of that complexity. The authors go in detail through all the tricks they had to do to make it work with PPO and show in detail all the ablations for it. I think this is a pretty useful study for everyone working on EP and wanting to combine this with typical policy gradient methods.

Critical weaknesses

The authors motivate this work through on-robot learning and the necessary efficiency for it. In the end, the paper trains on 1-2k parallel environments with 100 million samples and 200 GB RAM needed for training. The improved VRAM savings are nice but this does not make it usable for training on a typical Jetson Orin which you find on a Unitree A1. Also the 4.3× memory saving is w.r.t. BPTT through the relaxation dynamics, but the relevant comparison would with BP PPO with a feed-forward network, and for that one memory, compute and wall-clock time are not shown. EP needs 60 relaxation steps per sample for the policy update and 30 already for inference, so I would guess it is a lot more expensive on a GPU. In my opinion the paper should either be framed as a study of EP in complex RL settings in simulation, or show at least one adaptation experiment in simulation (e.g. pretrain, then adapt with EP to a payload or terrain change with few samples!), as this is the actual use case. Otherwise the story does not match the experiments at all.

Furthermore, the task is pretty simple as its only forward walking with a CPG. So the CPG probably does a lot of heavy lifting as other on-robot learning papers like "Gait in Eight: Efficient On-Robot Learning for Omnidirectional Quadruped Locomotion" do, which actually learns with RL and a CPG on a real robot, so this should also be added to the related work, as there are clearly works that do this, although without EP. This makes it also unclear if EP PPO would also work for the usual joint position control that people do in locomotion learning. An investigation, even with a negative result, would be interesting. Also, EP and BP seem to differ in their architecture, inputs (1024-D iDCT vs. normalized observations) and update rule, and the architecture and hyperparameters of the BP baseline are not given at all. "Comparable performance" is also a bit generous as BP converges clearly faster in stage 2, EP hits the KL early stop a lot, and in Fig. 3d/e the EP policies have about 3–4× larger mean roll and about 2× larger mean pitch angles with a very large spread, which the text does not talk about.

Questions for authors

What are the architecture, optimizer and hyperparameters of the BP PPO baseline?

What are the GPU memory and wall-clock time per update for BP PPO vs. EP PPO?

Reviewer_DDZjOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether equilibrium propagation (EP), a local learning rule, can train a quadruped locomotion controller. It adopts the two-stage CPG–RES architecture of Seto et al. (2025) on a simulated 12-DoF Unitree A1 in MuJoCo. Stage 1 trains a CPG-parameter policy on flat ground. Stage 2 freezes that policy and trains a residual joint-angle policy on randomized box terrain. In both stages, the policy and value networks are two-hidden-layer (768 units) energy-based recurrent networks trained with three-phase EP (one free phase and ± nudge phases). The inputs are iDCT-expanded 1024-D observations. The log-std vector is updated analytically with Adam. To make PPO work with EP, the authors inject the PPO clipped-surrogate gradient as the output-layer nudging force (Eq. 9–10). They recompute the probability ratio at every relaxation step, and they make three modifications that the experiments show are needed for convergence: (i) a two-sided ratio mask with a reverse clip coefficient ε_rev (Eq. 11), (ii) 1/σ instead of 1/σ² scaling, and (iii) the iDCT preprocessing. They also use KL early stopping, KL rollback and adaptive learning rates (Appendix F).

With 1e8 samples per stage and 5 seeds, EP-PPO reaches final training rewards similar to a feed-forward BP-PPO baseline (Fig. 2). In a walking test (6 terrain heights × 3 target speeds × 500 episodes × 5 versions, Fig. 3), it reaches broadly similar success rate, velocity, power and angular-velocity metrics. The authors also report 3,110 MB vs. 13,486 MB of GPU memory for EP vs. BPTT gradient estimation on the same recurrent network (Sec. 4.2.1). Ablations in Appendix E cover ε_rev, gradient scaling, a static vs. dynamic mask, and iDCT. The paper frames itself as a feasibility study. It does not evaluate on-robot learning, neuromorphic hardware, or energy use.

Critical strengths
  1. A careful first demonstration in a hard setting, with seeds. As far as I know, this is the first demonstration in which both the actor and the critic of a continuous-control PPO agent are trained with EP, on a contact-rich, domain-randomized locomotion task. Prior work cited in the paper (Kubo et al., 2022) kept a BP-trained critic. The comparison with BP-PPO uses the same controller, reward, observations and sample budget, and it reports means and standard deviations over 5 independently trained versions (Fig. 2, Fig. 3, Fig. 4). The walking test is large (500 episodes per condition per version) and stratifies difficulty by H_max and target speed.
  2. Useful analysis of an EP-specific failure mode. The paper shows why a naive EP-PPO nudge fails. With positive advantage in the positive nudge phase, the force pushes the output away from the sampled action, and the one-sided PPO mask cannot bound this (Appendix C.3, Eq. A-19–A-21). The measurements in Appendix D.3 (Fig. 5–6) show that r_nudging ranges from about 1e-5 to 1e3 within a nudge phase, even though the free-phase ratio stays near 1. ε_rev = 1.0 removes the bound, yields markedly lower extreme log-ratios, and training then diverges (Fig. 7, including one NaN run). The static-mask ablation (Fig. 9) supports per-step ratio gating as a necessary ingredient, not decoration. Practitioners who try to plug EP into policy-gradient methods will find this insight useful.
Critical weaknesses
  1. Limited algorithmic novelty and no theoretical grounding for the modified nudge. The method applies the PPO output gradient as the EP nudging term, plus three empirically tuned fixes: the two-sided mask, 1/σ scaling and iDCT inputs. Standard EP estimates gradients for a scalar cost added to the energy (Eq. 5). The state-dependent indicator gating in Eq. 11 makes the output force discontinuous and dependent on the evolving state. It is unclear whether this force derives from any scalar function, or which objective the resulting EP update (Eq. 7) approximates. The paper never characterizes how the update relates to the true clipped-surrogate gradient, for example by measuring cosine similarity with the BP or BPTT gradient on the same network. Fig. 4 shows that EP repeatedly hits the KL early-stop threshold. Appendix D.2 attributes this to a "non-linear response" but does not substantiate it. Without such analysis, the result reads as an engineering recipe validated on a single task.
  2. The feasibility claim is narrow, and some "comparable" statements are selective. All evidence comes from one robot, one architecture inherited from prior work (the CPG prior constrains the stage-1 action space), and evaluation on new samples of the same box-terrain family used in training (Sec. 4.2). There are no standard continuous-control benchmarks to show that EP-PPO works beyond this structured setting. In stage 2, which is the uneven-terrain stage, EP improves more slowly and its value MSE stays visibly higher than BP's (Fig. 2c, 2d). Sec. 4.2 claims comparability on success, velocity, power and angular velocity, but omits roll/pitch angles. In Fig. 3(d),(e), the error bars on those angles are very large for some configurations. The paper should state which method this concerns and discuss it. The figures are too small to read the per-condition differences, and no table of numbers is given.
  3. Baseline specification and computational claims are incomplete. The BP-PPO baseline's architecture, width and depth, optimizer and hyperparameters are not reported; Table 6 lists EP settings only. The baseline also uses different inputs (no iDCT) and a different network class, so the capacity match is unknown. The 4.3× memory figure (Sec. 4.2.1) compares EP with BPTT on the same recurrent network. It does not compare EP with the feed-forward BP-PPO actually used as the performance baseline. It is a single measurement, with no wall-clock, FLOP or relaxation-step cost. EP needs 30/20/10 relaxation steps for the policy and 25/15/10 for the value network per mini-batch (Table 6), and the policy needs iterative relaxation at every 100 Hz control step. This cost matters for both latency and the claimed low-power motivation. The title ("Neuromorphic") and the motivation stress on-robot, low-power adaptation, but nothing is run on neuromorphic or analog hardware or simulated as such. The Discussion acknowledges this, but the framing overstates what is shown.
  4. Presentation issues hinder verification. Key details sit only in the appendices, including the walking-test protocol, network architecture and full algorithm. The main-text ablation summary (Sec. 4.3) gives no figure. Several figures (Fig. 3, Fig. 4) are illegible at print size. Some hyperparameters are ambiguous in Table 6: the misaligned rows make values such as v_x,upper, η_policy,upper and the learning-rate adaptation coefficient hard to assign.
Questions for authors
  1. What is the BP-PPO baseline's architecture (width, depth, activation), optimizer, learning-rate schedule and KL-control setup? Did you tune it with an effort comparable to EP-PPO? Would BP-PPO with the same iDCT inputs, or with BPTT on the same recurrent EP network, change the comparison?
  2. Can you quantify how well the EP update with the gated, 1/σ-scaled nudge aligns with the true clipped-surrogate gradient? For example, report the cosine similarity with BPTT gradients on the same network over training. Can you characterize which objective the update optimizes, given that the force in Eq. 11 is state-gated?
  3. Please give numeric results for the walking test (a table of mean ± std per H_max and target speed), including roll/pitch angles. Which method has the large error bars in Fig. 3(d),(e)? Did you run any statistical test for the claimed comparability?
  4. What are the wall-clock training time and the per-step inference cost (number of relaxation steps, FLOPs) of EP-PPO compared with BP-PPO and BPTT? Is the 30-step free-phase relaxation compatible with the 100 Hz control loop, as in the target on-robot setting?
  5. Does the recipe (ε_rev, 1/σ scaling, iDCT) transfer without retuning to other continuous-control tasks or to out-of-distribution terrain? The answer determines whether the contribution is a general EP-RL method or a task-specific demonstration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.