- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work presents a way to learn a fixed synergy matrix directly from the reward signals. This is important, as a musculoskeletal system is extremely redundant, and learning the right synergies yields a proper gait or muscular activation patterns. Thus, the contribution is valuable for the field, adding the comparison with some existing solutions.
Critical strengths
• By constraining the W matrix into a learning scheme directly from the rewards, the synergies are well defined and learned through interaction, conversely to other approaches that need a two-stage process. • Very interesting to have the entropy bonus while maintaining action diversity. • It allows information priors on the synergy matrix.
Critical weaknesses
• Hyfydy and scone-based simulators are not referenced in the related work. It is mentioned in the conclusion. • It is not clear why DEP-RL is not tested as a baseline to show the action diversity discovery (maybe because it is difficult to port to Myosuit), but leaves the quality of the action diversity open. In this sense, The det(J^TJ) is supposed to provide action diversity, but this could be better shown in the results. • Not clear what is meant by "solved” as a function of performance. I would think that better synergies should also imply less effort of the muscles and better, for instance, gait patterns. • It would be nice to have a comparison against biological synergies taken from data (for instance, in walking and analyse the similarities.
Questions for authors
• Authors state that W is not state-dependent but J is; thus, what is the impact of imposing priors on the regularizer? • How are the synergies found in comparison with the biological counterpart?
2 comments
Nothing to add
As I said, I like the work, and thanks so much for the detailed response. I maintain my original score. I was asking about Hyfydy because it is a pity that we can only use this method in MyoSuite.
We thank the reviewer for the positive review, and for highlighting that W is learned from reward in a single stage while preserving action diversity, and for emphasizing the flexibility of the architecture. Below we answer the two questions and the remaining points.
The Jacobian factorizes as J(z) = D(z) W, with D(z) = diag((1 + Wz)^−2) positive and diagonal. The state only changes how strongly each muscle's row is scaled: the directions available to the policy are the columns of W at every state, and rank J = rank W. A prior on W therefore acts on the regularizer at all states at once, and the regularizer keeps the decoder well-conditioned within whatever structure the prior allows:
Priors are imposed on W and the regularizer is computed through them so nothing in the correction needs to change.
Please see our answer to Reviewer DesD (W2). In short, the learned synergies load one to a few muscles each, in line with synergies measured in human walking and hand EMG. Additionnaly, the muscle activity of our compressed walking policy correlates positively with human EMG, above the reward-trained DynSyn and DEP-RL yet still below the imitation-trained KINESIS.
On the related point that better synergies should mean less muscle effort, we measured the mean muscle activation (in [0, 1]) of the trained policies. On the hand, Plexus at K = 12 uses about a third of Latent SAC's activation at a similar success rate: 0.021 vs 0.070 on Die Reorient and 0.076 vs 0.204 on Baoding. This follows from the correction: the Jacobian shrinks as a muscle's activation grows, so measuring entropy on the actions implicitly rewards keeping muscles away from saturation, which Latent SAC's latent entropy does not. On the leg, Plexus uses 4–15% less activation than Latent SAC at K = 10 and 34–37% less at K = 50: Latent SAC's activation rises with K while Plexus's stays nearly constant. Importantly, effort is not part of any reward used for training, so these differences are directly attributable to the method itself. Gait, in contrast, is largely prescribed by the default MyoLegWalk reward, which fixes the cadence (cyclic hip term), the target speed and the hip adduction and rotation angles.
Hyfydy and SCONE (Geijtenbeek, 2019) will be added to the simulator paragraph in related work.
DEP-RL addresses a different problem: it injects correlated exploration in full muscle space at collection time but learns no synergy basis. We did test it on MyoLeg in preliminary runs, where it brought no clear improvement over SAC while being highly sensitive to its many hyperparameters. Lattice is the more relevant baseline: it also produces structured, correlated muscle exploration, but as a stochastic policy with a proper density trained with SAC's entropy term, so it isolates exactly what our correction changes. It was also developed and evaluated on the same MyoSuite tasks.
On action diversity, the volume term can be measured directly on the trained policies: ½ log det(JᵀJ) is the entropy the exploration noise gains or loses when the decoder maps it from latent to muscle space. At the same K, Plexus's noise covers far more of muscle space than Latent SAC's: per latent dimension (geometric mean over directions), it spreads over a 5–18 times larger range of muscle activity. We will add this measure to the appendix.
"Solved" is the per-step success flag of each MyoSuite hand task, and a policy's score is the fraction of the episode spent solved. Hand Pose is solved when the norm of the joint-angle error (23 joints) is below 0.8 rad; Baoding when both balls are within 1.5 cm of their targets and none has dropped; Die Reorient when the die is within 2.5 cm of the target position and 0.26 rad (15°) of the target orientation, without dropping. We will state this in the environments appendix.