acceptodds
Under review as a conference paper at ICLR 2027

Learning a Synergy Basis from Reward Alone

Abstract

Musculoskeletal control involves action spaces where muscles far outnumber the mechanical degrees of freedom they drive, creating a redundancy that challenges standard reinforcement learning (RL). A natural remedy drawn from motor neuroscience is to act through a smaller set of muscle synergies. Most existing synergy-based RL methods obtain their basis from a separate discovery phase, a dynamics model, or physiology rather than from the task reward. We introduce Plexus, a latent policy learning method that constrains actions to non-negative combinations of synergy vectors and makes the basis a free parameter of the actor. Crucially, this basis is learned end-to-end, from reward alone, at the same time as the actor. Because the decoder is learned and otherwise unconstrained, it is generically non-orthogonal and distorts volume, so latent and ambient entropy no longer represent the same quantity. This makes the basis lose rank and the policy steer a subspace narrower than its latent size advertises. We remedy this by measuring entropy on the induced action manifold, through a Gram determinant correction to the log-density. We show that when the decoder is jointly learned, letting this quantity flow through the decoder keeps the basis well conditioned. Across six MyoSuite tasks on two musculoskeletal models, Plexus matches or exceeds full-dimensional baselines on every task and is competitive with synergy-based methods at matched latent width, while yielding a single state-independent basis that can additionally be initialized or constrained with biomechanical priors, or inspected for interpretability.

Primary area:
reinforcement learning
Keywords:
muscle synergies, maximum-entropy reinforcement learning, musculoskeletal control, latent action spaces

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_8ffMOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work presents a way to learn a fixed synergy matrix directly from the reward signals. This is important, as a musculoskeletal system is extremely redundant, and learning the right synergies yields a proper gait or muscular activation patterns. Thus, the contribution is valuable for the field, adding the comparison with some existing solutions.

Critical strengths

• By constraining the W matrix into a learning scheme directly from the rewards, the synergies are well defined and learned through interaction, conversely to other approaches that need a two-stage process. • Very interesting to have the entropy bonus while maintaining action diversity. • It allows information priors on the synergy matrix.

Critical weaknesses

• Hyfydy and scone-based simulators are not referenced in the related work. It is mentioned in the conclusion. • It is not clear why DEP-RL is not tested as a baseline to show the action diversity discovery (maybe because it is difficult to port to Myosuit), but leaves the quality of the action diversity open. In this sense, The det(J^TJ) is supposed to provide action diversity, but this could be better shown in the results. • Not clear what is meant by "solved” as a function of performance. I would think that better synergies should also imply less effort of the muscles and better, for instance, gait patterns. • It would be nice to have a comparison against biological synergies taken from data (for instance, in walking and analyse the similarities.

Questions for authors

• Authors state that W is not state-dependent but J is; thus, what is the impact of imposing priors on the regularizer? • How are the synergies found in comparison with the biological counterpart?

2 comments

We thank the reviewer for the positive review, and for highlighting that W is learned from reward in a single stage while preserving action diversity, and for emphasizing the flexibility of the architecture. Below we answer the two questions and the remaining points.

Q1 - Priors and a state-dependent J

The Jacobian factorizes as J(z) = D(z) W, with D(z) = diag((1 + Wz)^−2) positive and diagonal. The state only changes how strongly each muscle's row is scaled: the directions available to the policy are the columns of W at every state, and rank J = rank W. A prior on W therefore acts on the regularizer at all states at once, and the regularizer keeps the decoder well-conditioned within whatever structure the prior allows:

  • A mask on W (which muscles a synergy may load) would restrict which rows contribute to each column's volume and the regularizer spreads the columns only within that pattern.
  • Tying the two legs makes the decoder block-diagonal, each leg on its own slice of z. The Gram matrix is then block-diagonal and its log-determinant is the sum of a left and a right term. Both send gradient into the same shared W, evaluated at the two legs' different gait phases, so the shared basis is kept well-conditioned over the whole cycle.
  • An initialization (e.g. DynSyn's grouping) only sets where W starts; the regularizer is unchanged and the reward decides how far W moves.

Priors are imposed on W and the regularizer is computed through them so nothing in the correction needs to change.

Q2 - Biological synergies

Please see our answer to Reviewer DesD (W2). In short, the learned synergies load one to a few muscles each, in line with synergies measured in human walking and hand EMG. Additionnaly, the muscle activity of our compressed walking policy correlates positively with human EMG, above the reward-trained DynSyn and DEP-RL yet still below the imitation-trained KINESIS.

On the related point that better synergies should mean less muscle effort, we measured the mean muscle activation (in [0, 1]) of the trained policies. On the hand, Plexus at K = 12 uses about a third of Latent SAC's activation at a similar success rate: 0.021 vs 0.070 on Die Reorient and 0.076 vs 0.204 on Baoding. This follows from the correction: the Jacobian shrinks as a muscle's activation grows, so measuring entropy on the actions implicitly rewards keeping muscles away from saturation, which Latent SAC's latent entropy does not. On the leg, Plexus uses 4–15% less activation than Latent SAC at K = 10 and 34–37% less at K = 50: Latent SAC's activation rises with K while Plexus's stays nearly constant. Importantly, effort is not part of any reward used for training, so these differences are directly attributable to the method itself. Gait, in contrast, is largely prescribed by the default MyoLegWalk reward, which fixes the cadence (cyclic hip term), the target speed and the hip adduction and rotation angles.

Other

Hyfydy and SCONE (Geijtenbeek, 2019) will be added to the simulator paragraph in related work.

DEP-RL addresses a different problem: it injects correlated exploration in full muscle space at collection time but learns no synergy basis. We did test it on MyoLeg in preliminary runs, where it brought no clear improvement over SAC while being highly sensitive to its many hyperparameters. Lattice is the more relevant baseline: it also produces structured, correlated muscle exploration, but as a stochastic policy with a proper density trained with SAC's entropy term, so it isolates exactly what our correction changes. It was also developed and evaluated on the same MyoSuite tasks.

On action diversity, the volume term can be measured directly on the trained policies: ½ log det(JᵀJ) is the entropy the exploration noise gains or loses when the decoder maps it from latent to muscle space. At the same K, Plexus's noise covers far more of muscle space than Latent SAC's: per latent dimension (geometric mean over directions), it spreads over a 5–18 times larger range of muscle activity. We will add this measure to the appendix.

"Solved" is the per-step success flag of each MyoSuite hand task, and a policy's score is the fraction of the episode spent solved. Hand Pose is solved when the norm of the joint-angle error (23 joints) is below 0.8 rad; Baoding when both balls are within 1.5 cm of their targets and none has dropped; Die Reorient when the die is within 2.5 cm of the target position and 0.26 rad (15°) of the target orientation, without dropping. We will state this in the environments appendix.

Reviewer_8ffMOpenReview ↗

Nothing to add

As I said, I like the work, and thanks so much for the detailed response. I maintain my original score. I was asking about Hyfydy because it is a pity that we can only use this method in MyoSuite.

Reviewer_DesDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a method that learns muscle synergies in locomotion and manipulation using reinforcement learning. In the paper, the synergy structure is represented by a non-negative latent action z and a non-negative matrix W that encodes relationships among muscles, followed by a nonlinear transformation into muscle activations. Unlike previous works that typically learn W separately, the paper proposes to learn the latent policy and W simultaneously in the same training process. To achieve this, the paper allows exploration in z-space and derives the entropy of the induced action distribution, considering both the exploration distribution of z and the learnable W. With this modification, an entropy-based method like SAC can be correctly applied to train the policy.

Critical strengths
  • the paper derives the formulation of entropy on the non-negative latent manifold, which enables the application of entropy-based methods like SAC to this problem.

  • the results shows the proposed framework improves performance over the previous SAC baselines without the entropy correction in almost all tested task and latent-width configurations.

  • the main paper is clearly written and generally easy to follow.

Critical weaknesses
  • the derived entropy correction is evaluated only within SAC. It is not clear how the framework can be extended to other RL methods, such as PPO or model-based RL [1]. Additionally, the paper does not compare with other types of RL methods.

  • while the proposed method ensures non-negative decomposition of the control, how the learned W quantitatively reflects real or experimentally measured muscle synergies is not discussed.

[1] [Feng et al. 2023. MuscleVAE: Model-Based Controllers of Muscle-Actuated Characters]

Questions for authors

None.

2 comments

We thank the reviewer for the supportive review, and for highlighting the derivation of the entropy on the non-negative latent manifold, the consistent improvement over Latent SAC across tasks and latent widths, and the clarity of the paper. Below we address the extension to other RL methods and the quantitative comparison of the learned synergies with measured ones.

W1 - PPO, model-based RL and other methods

The correction is not specific to SAC. It changes one thing, the log-density of a policy that samples a low-dimensional latent and acts through a decoder, so that its entropy is measured on the actions it actually produces. Any algorithm that rewards or constrains the entropy of such a policy needs it. What does depend on the algorithm is how the decoder W receives reward:

  • Off-policy actor-critic (SAC, this paper): The critic's gradient flows through the decoder into W, thus directly shaped by the reward and by the volume term together.

  • On-policy methods (PPO): PPO improves the policy by reweighting the log-probability of the latents it sampled, never by differentiating through the action. That log-probability does not involve W, so W receives no reward gradient. The entropy correction carries over unchanged, but a learned decoder then needs an additional pathwise value term to keep receiving reward (Limitations).

  • Model-based methods: Such methods usually train their controller through a learned, differentiable world model with a pathwise actor gradient: a learned W would receive reward through the model, and the policy's entropy term can be replaced by its corrected version. However, MuscleVAE (Feng et al. 2023) itself does not need the correction, because its regularizer is a KL over the latent, and it never measures the entropy of muscle commands. In their limitations, authors note that muscles are controlled individually, without coordination structure. The correction only becomes necessary when a model acts through such synergistic decoders and must keep its muscle commands diverse.

  • Other RL methods as baselines: We kept every arm on the same SAC learner on purpose, so that differences come from the action representation and not from the learning algorithm.

W2 - W vs measured synergies

The submission inspects the learned synergies qualitatively in the Biomechanical priors paragraph (Fig. 4) and in the case studies at small K (App. D.7). We add here a quantitative comparison with measured muscle activity.

W composition: Considering muscles above 10% of the column's peak, at K = DoF/2, each learned synergy loads only a few of them: 1.2–1.5 on Die Reorient and Baoding, 2.8 on Hand Pose and 5–7 out of 80 on the leg tasks. This is in line with synergies measured in humans. In walking, four modules account for most of the variability of muscle activation of healthy gait (Clark et al. 2010) each of them dominated by two or three muscles. Restricted to the ten leg muscles usually recorded with surface EMG, our Leg Walk synergies load about two each. For the hand, synergies extracted from hand and forearm EMG include synergies dominated by single intrinsic hand muscles, attributed to the demands of fractionated finger movement (Yarossi et al. 2022), consistent with the one or two muscles per synergy we find on the manipulation tasks.

Comparison with human EMG: We reproduced the pipeline of Simos et al. (2026, KINESIS), which correlates a policy's stride-averaged muscle activity with human surface EMG (Wang et al. 2023; 9 subjects, 9 leg muscles), using the cross-subject correlation (0.73) as an upper bound. Our compressed (K = DoF/2) case study policy reaches a mean correlation of 0.21±0.05. This is positive and above other muscle-RL baselines (DynSyn and DEP-RL), which are reported at or below zero. It however stays behind KINESIS itself (0.36±0.08), a motion-imitation method trained on human motion-capture demonstrations that we do not use. Agreement is strongest on the plantar flexors (soleus 0.72, medial and lateral gastrocnemius 0.78 and 0.59), while the more proximal muscles agree less. One should also consider that quantitative EMG comparison is inherently limited: for the hand it is largely infeasible (surface EMG of the small intrinsic muscles is crosstalk-dominated), and for the leg only a few superficial muscles are accessible, so most modeled muscles have no experimental reference.

References

[Clark et al. 2010. Merging of Healthy Motor Modules Predicts Reduced Locomotor Performance and Muscle Coordination Complexity Post-Stroke. Journal of Neurophysiology]

[Simos et al. 2026. KINESIS: Motion Imitation for Human Musculoskeletal Locomotion. ICRA]

[Wang et al. 2023. A Wearable Real-Time Kinetic Measurement Sensor Setup for Human Locomotion. Wearable Technologies]

[Yarossi et al. 2022. Similarity of Hand Muscle Synergies Elicited by Transcranial Magnetic Stimulation and Those Found During Voluntary Movement. Journal of Neurophysiology]

I thank the authors for the clarification. I think the paper makes an inspiring contribution and have no further major concerns, though I would keep my original score given the framework-specific nature of the proposed approach.

Reviewer_jJGsOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Plexus, a synergy-motivated policy architecture integrated into SAC. The policy produces nonnegative latent coefficients, and a learned, state-independent nonnegative matrix maps them to muscle pre-activations before a bounded output transformation. The decoder and policy are optimized jointly, without a separate discovery phase. The central algorithmic contribution is to regularize entropy on the induced action manifold using a Gram-determinant correction, and to propagate its gradients through the decoder. This addresses the possibility that high latent entropy coexists with a nearly collapsed action representation.

Critical strengths

The paper identifies a meaningful mismatch between latent diversity and actual action diversity. Although the geometric correction itself draws on established ideas, its interaction with a decoder trained jointly through the RL objective is a useful contribution.

Critical weaknesses

The evaluation does not clearly establish when the core objective produces an advantage because of compressed control. The comparisons with Latent SAC support the usefulness of Equation (11), particularly for mitigating decoder degeneration. However, in figure 2 the best performance of Plexus are always obtained at (K=A), where there is no dimensionality reduction. The paper has not clearly identified a proper application in which the proposed correction makes a learned low-dimensional controller practically preferable. A controlled task with known effective control dimension would help distinguish objective limitations from the representational limitations of a fixed nonnegative decoder.

Minor weaknesses: Some figures have super small font sizes, making important messages hard to read (Figure 1, 3, 4).

Questions for authors
  1. What is the strongest demonstrated advantage when K<A? Could you distinguish the evidence for reduced dimensionality and sample efficiency (if not overall performance)?

  2. How is “comparable performance” defined in the compression analysis? Could you report confidence intervals for performance differences and distinguish absence of a significant difference from evidence that any degradation is practically small?

3 comments

We thank the reviewer for the careful and constructive review. Below we answer the two questions about the advantage of compressed control, and how comparable performance is defined.

Q1 - Advantage of compressed control

We agree with the reviewer about that, on every task, Plexus performs best at K = A (which is itself a learned reparametrization of the action space) and we do not claim that compressing the controller improves final performance but rather that Eq. (11) makes it possible without the decoder degenerating. Yet, on our benchmark, compression still bring advantages, clearly observable on Die Reorient:

First, performance falls much more slowly than the number of control dimensions. Keeping 31% of them (K = 12, half the number of DoF) retains 87% of Plexus's K = A score (95% CI [72%, 102%]); keeping 59% (K = 23) retains 88% [84%, 92%], and 79% (K = 31) retains 96% [91%, 101%]. At every compressed width Plexus also stays above the best full-dimensional baseline: respectively +16%, +17% and +27% at K = 12, 23 and 31. At the same widths, Latent SAC's decoder collapses to an effective rank about three times lower than Plexus's.

Second, compressed controllers learn faster, and the fewer the dimensions, the faster. To reach 80% of the best baseline's final score, Plexus needs 3.3M, 3.9M and 4.8M environment steps at K = 12, 23 and 31, against 5.1M at K = A, 13.8M for Lattice and 20.1M for SAC; to reach half of it, K = 12 needs 1.8M steps against 4.8M at K = A. Every compressed width is faster than K = A and three to six times faster than Lattice or SAC.

These benefits are task dependent and gains at K < A on the other tasks are smaller. However, on every task, compression makes the controller readable. Because W is a single state-independent matrix, a small K gives a handful of columns, each loading a few muscles, which can be read as muscle groups and followed over an episode or a gait cycle (Fig. 4 and case study appendix). At K = A the same matrix is 39×39 or 80×80 and cannot be read this way.

Q2 - Comparable performance

As highlighted, the absence of a significant difference is not necessarily evidence that the gap is indeed small. In the submission, each marker of Figure 2 reports a one-sided Welch t-test against the strongest full-dimensional baseline at p < 0.05 (up or down triangle when Plexus is significantly above or below, circle otherwise). The table below gives, for each task, the smallest latent width we read as comparable (circle), with its difference to the strongest full-dimensional baseline and the 95% confidence interval, in absolute terms and relative to the baseline.

TaskAsmallest KPlexusbest full-dim.difference [95% CI]relative [95% CI]
Die Reorient39120.7570.654 (Lattice)+0.10 [−0.03, +0.23]+16% [−4, +36]
Baoding39230.6410.629 (Lattice)+0.01 [−0.21, +0.24]+2% [−34, +38]
Hand Pose39310.6140.653 (Lattice)−0.04 [−0.13, +0.05]−6% [−19, +7]
Leg Walk805021,16922,071 (SAC)−902 [−2,212, +407]−4% [−10, +2]
Rough Terrain805017,67718,942 (Lattice)−1,265 [−2,964, +433]−7% [−16, +2]
Stair Terrain802011,41312,253 (Lattice)−840 [−4,588, +2,909]−7% [−37, +24]

On Hand Pose, Leg Walk and Rough Terrain the largest deficit compatible with the data is 10–19%; on Baoding and Stair Terrain the intervals span about ±35%, so neither a large gain nor a large loss can be excluded. We therefore use "comparable" in a practical sense: not significantly different, with a mean within 7% of the strongest full-dimensional baseline, while the two full-dimensional baselines themselves differ by up to 18%. We will add the full table and state this definition explicitly in the text.

Other

The font in the figures 1, 3, 4 as well as their appendix counterparts will be enlarged to improve readability.

I thank the authors for their detailed reply. My concerns over performance evaluation has been properly resolved.

About the advantage of compressed dimensions: I understand that plexus learns faster and also retains performance when dimensions decreases. But these benefits seem universal for all dimension-reduction methods. e.g. in Figure 2, Plexus doesn't seem to have a generally smaller performance drop speed compared to latent SAC. Should this be claimed a unique contribution of Plexus?

We thank the reviewer for the comment and question.

Plexus and Latent SAC indeed follow similar trends as the latent space is compressed, and we agree that retaining performance or learning faster at smaller K is not unique to Plexus. What is specific to Plexus is the performance at each K: by ensuring the policy uses all the dimensions it is given, the correction places Plexus above Latent SAC in 23 of the 24 configurations (by a median of 25%). Equivalently, Plexus reaches Latent SAC’s best performance with K between DoF and (A+DoF)/2 on all tasks. We will revise the compression paragraph accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.