- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper tackles whole-body musculoskeletal control in simulation, where an agent has to carry out arm tasks while keeping its balance. Inspired by human motor control, the authors propose a modular framework that splits the policy across body segments (arm, torso and legs), with each module driven by its own network and all of them trained concurrently on the same task reward.
Moreover, they introduce a nested observation scheme and impose posture as a constraint rather than a reward term. Compared to the baselines such as SAC and QFLEX, the resulting controllers substantially outperform a single policy over all muscles on manipulation tasks and remain robust to unseen perturbations.
Critical strengths
- Simple, well-motivated idea that drops into existing off-policy algorithms without modification.
- The nested observations and the asymmetric penalty (arm optimizes the task alone) are a neat way to keep the arm from backing off the task.
- Posture as a constraint rather than a reward term is well argued, and Figure 3 nicely supports it: all reward-based variants, including the Lagrangian one, fail outright.
- The paper shows large gains on manipulation across three algorithms, plus a zero-shot robustness evaluation.
Critical weaknesses
- The ablations don’t resolve where the gain comes from. The method bundles three changes (disjoint action heads with separate critics, nested observations, penalty only on torso/leg), but only the postural constraint is ablated. I would like to see permutation controls rather than just removals: randomly partition the muscles into three groups of the same sizes, scramble the observation nesting (e.g., leg < arm < torso), give every module the full observation, and charge the penalty to all modules. Without these, “human-inspired modularity” cannot be told apart from “smaller action heads learn faster.” The idea is interesting, but the results are not substantial and convincing enough to be published.
- Separating trunk/postural control from limb control is a long-standing design in neuromechanical modelling, e.g., reflex-based walking models with a dedicated trunk-balance controller [1] and CPG/reflex models that keep trunk balance as a separate module [2] (I only cited two papers here, but please see other works by Geyer and Ijspeert Labs). The authors should position MM-PoCo against this line of work and state precisely what is novel.
- Section 3.2 makes testable predictions that the authors never circle back to: do torso/leg activations precede arm activation when a new target appears (anticipatory postural adjustment)? Does the CoM stay within the base of support? Showing even one of these would turn the motivation into a result. Relatedly, no biological evidence is given for the specific one-way information flow arm < torso < leg.
- The content of the observation vector is never described, and no videos are provided, so it is hard to tell whether the policy stands naturally or holds a stiff pose that happens to satisfy the 0.15 m criterion.
Overall, I find the idea interesting and the execution on the right track, but the results are not yet substantial or convincing enough for publication.
References
[1] H. Geyer and H. Herr. A muscle-reflex model that encodes principles of legged mechanics produces human walking dynamics and muscle activities. IEEE Trans. Neural Systems and Rehabilitation Engineering, 18(3):263–273, 2010.
[2] A. Di Russo, D. Stanev, A. Sabnis, S. M. Danner, J. Ausborn, S. Armand, and A. Ijspeert. Investigating the roles of reflexes and central pattern generators in the control and modulation of human locomotion using a physiologically plausible neuromechanical model. Journal of Neural Engineering, 20(6):066006, 2023.
[3] C. Ma, Y. Wei, C. Zuo, C. Zhang, and Y. Sui. Bipedal balance control with whole-body musculoskeletal standing and falling simulations. CoRL, 2025.
Questions for authors
- Which component drives the improvement? Could you run the permutation controls above?
- What goes into each module’s observation, and could you provide videos of the learned behaviour, including failure cases?
4 comments
Response to Reviewer W2R1 (2/2)
W3: Evidence for APA-like behavior
Thank you for suggesting additional experiments to better support the motivation of our work. Since MM-PoCo was inspired by human anticipatory postural adjustments (APAs), we conducted two additional analyses to examine whether similar patterns emerge in the learned controller. Here, APAs refer to postural adjustments made before movement to maintain balance, rather than reactive recovery after balance is lost [4].
To examine whether APA-like patterns emerge in the learned controller, we analyzed trunk muscle activation around target changes in the drawing task. The torso muscles co-contract within 60 ms before the new target is observed. The end effector stays near the reached target until the new target is observed and moves away only afterward, so the trunk activation precedes the arm movement toward the next target. Consistent with findings from human studies [5], this pattern emerges even though the agent cannot predict the direction of the next target, since its observation contains only the error to the current target.
We additionally analyzed whether the controller maintains balance during arm movement by measuring the CoM relative to the base of support in the draw task. We evaluated the final models over 200 episodes per seed and found that the CoM was outside the base of support in only 0.2% of the steps. This indicates that the postural modules maintain posture and equilibrium while supporting arm movement.
Together, these analyses show that the agent trained with MM-PoCo learns to maintain balance and exhibits APA-like patterns similar to those observed in humans, rather than relying on reactive balance recovery. We have added a new experimental section (Section 5.5, Figure 5) presenting these analyses to strengthen the empirical support for our work (the additions are highlighted in red).
Regarding the one-way information flow arm < torso < leg, as described in our response to W1, the revised MM-PoCo gives every module the full observation and therefore no longer imposes this flow.
[4] J. Massion. Movement, posture and equilibrium: interaction and coordination. Progress in Neurobiology, 1992.
[5] D. Piscitelli, A. Falaki, S. Solnik, and M. L. Latash. Anticipatory postural adjustments and anticipatory synergy adjustments: Preparing to a postural perturbation with predictable and unpredictable direction. Experimental Brain Research, 2017.
W4, Q2: Observation details and supplementary videos
We have provided videos of the learned behavior and additional details on the observation space of each module (Appendix B, Table 4).
As shown in the videos, MM-PoCo maintains upright posture and equilibrium while swaying naturally within the feasible region, rather than holding a stiff pose. The videos also demonstrate how the agent maintains balance under external perturbations, including a moving platform and ball strikes. In failure cases, the agent leans and falls, ends up on one leg and falls, or keeps standing but misses the target.
For the observation space, all modules receive the same full observation as the monolithic baseline, including task information, joint positions and velocities, and balance-related quantities. The detailed observation keys and their dimensions are provided in Table 4 of Appendix B (the additions are highlighted in red).
Thank you for the detailed response and for running all of the controls I suggested, plus the compute-matched baseline. The new related work, the observation details, and the CoM and trunk-activation analysis all make the paper clearer. However, the new results raise a few concerns of their own.
The observation ablation shows that observation design barely matters. Every variant reaches roughly 95-99%, including all six nesting orders, separate observations and full observations. The original submission motivated nesting by the body's chain ("a module can support the modules above it only if it knows where they are"). The revision instead motivates full observation by saying that the postural modules need information about the other modules. But with separate observations, the postural modules can't see the arm at all and still reach 96.9%, so the results support neither argument. Moving to full observation is a reasonable engineering choice, but it drops the information-flow part of the human-inspired framing of the paper. What's left is a hand-picked arm/torso/leg split and the way the penalty is assigned. When the arm also gets the terminal penalty, success falls from 99.4% to 1.6% (Fig. 7). That makes me think most of the gain comes from the penalty assignment rather than from the biological structure.
I also have some concerns about the baselines. The single policy stays near zero on myosuite-draw with all three algorithms, even when compute-matched. A simple sanity check would help, for example showing that your single-policy SAC can solve the original arm-only MyoSuite reaching task. The QFLEX results suggest that something may not be working as intended. On ms-human-stand, episodes last only 6 to 10 steps (about 0.1 to 0.2 seconds), and QFLEX + MM-PoCo actually gets worse during training on myosuite-draw and myosuite-stand (Fig. 2). Additionally, in Fig. 2, MM-PoCo is already well ahead at the first point of training, even though all methods start from random networks.
I was also surprised not to see PPO, which is the usual choice for full-body musculoskeletal control. MuscleMimic (Li et al., 2026), which you cite, trains a single PPO policy on a 416-muscle full-body model to imitate a wide range of motions while staying balanced. Imitation is a different setting, but it shows that a single policy can handle this kind of model. That goes against the claim that the difficulty "lies in the structure of the policy rather than in the learning algorithm."
I'm glad you tested for anticipatory postural adjustments in Section 5.5, but I don't think the analysis shows anticipation yet. The next target appears as soon as the current one is reached, so the trunk activation before the switch happens while the arm is still finishing its previous reach (Fig. 5c). Comparing trunk onset with arm-muscle onset, as human APA studies do, would be more convincing. The trunk muscles also stay at around 0.4 activation throughout, which suggests a stiff, co-contracted trunk rather than natural standing.
Overall, I appreciate the work that went into the revision, and I still think the idea is interesting. But the new results have weakened the human-inspired motivation, and I'm not convinced the baselines give a fair comparison. Therefore, I will keep my score.
Thank you for your thoughtful feedback. We have carefully reviewed your concerns.
Human-inspired framing and modular training
We agree that the observation design barely matters. However, our human-inspired framing rests on the three principles stated in Section 3.3 (Motivation): divide the body by function, let posture serve the movement, and treat posture and equilibrium as a feasible region.
Divide the body by function. Ours is divided into the arm for movement, and the other body parts for posture, and each module has its own critic.
Let posture serve the movement. All modules work together to support the arm movement. The reward depends on the distance between the end effector of the arm and the target, so the postural modules optimize for what the arm achieves. This is why the torso receives a reward determined by the position of the arm rather than a separate torso and leg reward term.
Treat posture and equilibrium as a feasible region. To adopt the concept of posture in the RL framework, we use a constraint rather than a reward for a vertical torso. The agent maximizes the task reward as long as it stays inside the feasible region. This matches human balance, staying dynamically within a region rather than holding a fixed posture.
We acknowledge that our original claim about the nested observation was overstated. It was not the core of our argument, and the revision no longer presents it as human-inspired.
Baselines
One point we can state clearly: in an arm-only environment with the same draw task, SAC converges to a success rate close to 1 within 4M steps, and so does QFLEX. If helpful, we can add these curves to the appendix, together with videos.
We agree that QFLEX collapses. Although not reported in the paper, the modular approach also worked well with other algorithms (e.g., TD3), but the architecture design mattered. They needed to adopt an 8-layer network with LayerNorm and ensemble critics. Since RL is highly sensitive to hyperparameters, we tried to minimize their effect by applying nearly identical settings to every base algorithm. For QFLEX, however, the use of CrossQ is its defining feature, and we respected it. QFLEX is not well suited to our setting, but we included it because we believe the most recent baseline should be reported.
PPO
We agree that PPO is the usual baseline for musculoskeletal models, and it was the first algorithm we considered. Our goal was to test whether the torso and legs can support the arm as it moves quickly and precisely over a wide region in front of the body, so we widened the target range box and set the draw task to reach seven random targets in sequence. This sequential task is considerably harder than the single-target reaching task of MyoSuite. PPO does solve it in the arm-only setting, but its sample efficiency is far lower, so we restricted the base algorithms to off-policy methods.
Anticipatory postural adjustments
Inspired by APAs, our goal was to keep the CoM inside the BoS without any reward term that drives the CoM toward a stable region. What our trunk anticipates is the following. Given the one-step target distance, it produces an action that keeps the body balanced at the next step. We regard this as anticipatory, but we do not claim that it is identical to human APAs. We are grateful for the human APA studies you suggested to us. We will lower the claims of Section 5.5 and complete it accordingly, and we are willing to show what you suggested in future work.
Response to Reviewer W2R1 (1/2)
Thank you for taking the time to review our paper. We sincerely appreciate your feedback. Below, we provide detailed responses to your concerns.
W1, Q1: Permutation controls
To examine how each design choice contributes to the performance improvement of MM-PoCo, we ran all four controls you proposed. Each experiment was conducted with 3 seeds, and the details are provided in Appendix D.
(a) Random actuator partitioning: We compared a random partitioning with the same group sizes against our proposed human-body-based partitioning (arm, torso, and leg). The random partitioning achieved a success rate of only 0.6%, highlighting the importance of anatomically informed modularization.
(b) Alternative observation nesting: We compared all six possible nesting orders and found that all achieved success rates above 94%, indicating that performance is not sensitive to the nesting order.
(c) Full observation: Giving all modules the full observation resulted in slower learning than the previous nested observation but achieved the highest success rate of 99.2%.
(d) Penalty for all modules: Applying the same postural penalty to all modules, including the arm, resulted in a success rate of only 1.6%, with the agent barely learning the task.
These results highlight the importance of anatomically informed modularization and module-specific penalty assignment (Appendix D, Figures 6–8). All observation settings maintained a modular structure and substantially outperformed the monolithic single policy (0.1%), with full observation performing best. This suggests that modularization itself plays a more central role in improving performance than the choice of a specific observation design.
Based on these additional experimental results, we adopted full observation as the final observation design for MM-PoCo. Accordingly, we reran the main experiments and all other experiments in the paper under the updated setting, and revised the figures and manuscript (the revisions are highlighted in red). The other core design elements of MM-PoCo, including separate actor-critics, the shared task reward, and the postural penalty on the torso and leg modules only, remain unchanged.
W2: Position against prior neuromechanical models
Thank you for sharing these works. We agree that they are closely related to MM-PoCo.
These neuromechanical studies divide the control of human locomotion into modules inspired by the human neuromuscular system, including a module dedicated to balance [1, 2, 3]. MM-PoCo shares this philosophy of modular control. The main difference is that MM-PoCo is a learning-based method that trains each module as a neural network with RL, and we apply it to whole-body standing and manipulation tasks.
We have added these works to the Related Work section and clarified the methodological contribution of MM-PoCo relative to prior neuromechanical control approaches (the revisions are highlighted in red).
[1] H. Geyer and H. Herr. A muscle-reflex model that encodes principles of legged mechanics produces human walking dynamics and muscle activities. IEEE TNSRE, 2010.
[2] A. Di Russo et al. Investigating the roles of reflexes and central pattern generators in the control and modulation of human locomotion using a physiologically plausible neuromechanical model. Journal of Neural Engineering, 2023.
[3] S. Song and H. Geyer. A neural circuitry that emphasizes spinal feedback generates diverse behaviours of human locomotion. The Journal of Physiology, 2015.