acceptodds
Under review as a conference paper at ICLR 2027

Human-Inspired Modular Reinforcement Learning of Movement and Posture for Musculoskeletal Control

Abstract

Controlling upper-body movement while maintaining bipedal balance is a long-standing challenge in high-dimensional musculoskeletal models. Although recent reinforcement learning (RL) algorithms are capable of handling large action spaces, they still struggle here because balancing requires the rest of the body to continuously compensate for arm motion. In human physiology, upright posture is maintained not by a single rigid block but by sets of muscles that link body modules from the feet to the head, counteracting internal perturbations and keeping the center of mass within a narrow base of support. Inspired by human motor control, we propose a modular RL framework that coordinates movement, posture, and equilibrium across body segments. Specifically, we divide the body into arm, torso, and leg modules and train them jointly, with posture imposed as a constraint on the whole-body controller rather than as a reward. This lets the whole-body policy maximize task reward while staying within a stable postural region. In experiments on MyoSuite and MS-Human-700, with control expanded from the arm to the whole body, the resulting policy performs standing and manipulation tasks that demand both stability and mobility. These findings suggest that structuring control along biomechanical lines makes learning in neuromuscular models substantially more tractable.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, musculoskeletal control, whole body control, postural control, human motor control, constrained reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_W2R1OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tackles whole-body musculoskeletal control in simulation, where an agent has to carry out arm tasks while keeping its balance. Inspired by human motor control, the authors propose a modular framework that splits the policy across body segments (arm, torso and legs), with each module driven by its own network and all of them trained concurrently on the same task reward.

Moreover, they introduce a nested observation scheme and impose posture as a constraint rather than a reward term. Compared to the baselines such as SAC and QFLEX, the resulting controllers substantially outperform a single policy over all muscles on manipulation tasks and remain robust to unseen perturbations.

Critical strengths
  1. Simple, well-motivated idea that drops into existing off-policy algorithms without modification.
  2. The nested observations and the asymmetric penalty (arm optimizes the task alone) are a neat way to keep the arm from backing off the task.
  3. Posture as a constraint rather than a reward term is well argued, and Figure 3 nicely supports it: all reward-based variants, including the Lagrangian one, fail outright.
  4. The paper shows large gains on manipulation across three algorithms, plus a zero-shot robustness evaluation.
Critical weaknesses
  1. The ablations don’t resolve where the gain comes from. The method bundles three changes (disjoint action heads with separate critics, nested observations, penalty only on torso/leg), but only the postural constraint is ablated. I would like to see permutation controls rather than just removals: randomly partition the muscles into three groups of the same sizes, scramble the observation nesting (e.g., leg < arm < torso), give every module the full observation, and charge the penalty to all modules. Without these, “human-inspired modularity” cannot be told apart from “smaller action heads learn faster.” The idea is interesting, but the results are not substantial and convincing enough to be published.
  2. Separating trunk/postural control from limb control is a long-standing design in neuromechanical modelling, e.g., reflex-based walking models with a dedicated trunk-balance controller [1] and CPG/reflex models that keep trunk balance as a separate module [2] (I only cited two papers here, but please see other works by Geyer and Ijspeert Labs). The authors should position MM-PoCo against this line of work and state precisely what is novel.
  3. Section 3.2 makes testable predictions that the authors never circle back to: do torso/leg activations precede arm activation when a new target appears (anticipatory postural adjustment)? Does the CoM stay within the base of support? Showing even one of these would turn the motivation into a result. Relatedly, no biological evidence is given for the specific one-way information flow arm < torso < leg.
  4. The content of the observation vector is never described, and no videos are provided, so it is hard to tell whether the policy stands naturally or holds a stiff pose that happens to satisfy the 0.15 m criterion.

Overall, I find the idea interesting and the execution on the right track, but the results are not yet substantial or convincing enough for publication.

References

[1] H. Geyer and H. Herr. A muscle-reflex model that encodes principles of legged mechanics produces human walking dynamics and muscle activities. IEEE Trans. Neural Systems and Rehabilitation Engineering, 18(3):263–273, 2010.

[2] A. Di Russo, D. Stanev, A. Sabnis, S. M. Danner, J. Ausborn, S. Armand, and A. Ijspeert. Investigating the roles of reflexes and central pattern generators in the control and modulation of human locomotion using a physiologically plausible neuromechanical model. Journal of Neural Engineering, 20(6):066006, 2023.

[3] C. Ma, Y. Wei, C. Zuo, C. Zhang, and Y. Sui. Bipedal balance control with whole-body musculoskeletal standing and falling simulations. CoRL, 2025.

Questions for authors
  1. Which component drives the improvement? Could you run the permutation controls above?
  2. What goes into each module’s observation, and could you provide videos of the learned behaviour, including failure cases?
4 comments
AuthorsOpenReview ↗

Response to Reviewer W2R1 (1/2)

Thank you for taking the time to review our paper. We sincerely appreciate your feedback. Below, we provide detailed responses to your concerns.

W1, Q1: Permutation controls

To examine how each design choice contributes to the performance improvement of MM-PoCo, we ran all four controls you proposed. Each experiment was conducted with 3 seeds, and the details are provided in Appendix D.

(a) Random actuator partitioning: We compared a random partitioning with the same group sizes against our proposed human-body-based partitioning (arm, torso, and leg). The random partitioning achieved a success rate of only 0.6%, highlighting the importance of anatomically informed modularization.

(b) Alternative observation nesting: We compared all six possible nesting orders and found that all achieved success rates above 94%, indicating that performance is not sensitive to the nesting order.

(c) Full observation: Giving all modules the full observation resulted in slower learning than the previous nested observation but achieved the highest success rate of 99.2%.

(d) Penalty for all modules: Applying the same postural penalty to all modules, including the arm, resulted in a success rate of only 1.6%, with the agent barely learning the task.

These results highlight the importance of anatomically informed modularization and module-specific penalty assignment (Appendix D, Figures 6–8). All observation settings maintained a modular structure and substantially outperformed the monolithic single policy (0.1%), with full observation performing best. This suggests that modularization itself plays a more central role in improving performance than the choice of a specific observation design.

Based on these additional experimental results, we adopted full observation as the final observation design for MM-PoCo. Accordingly, we reran the main experiments and all other experiments in the paper under the updated setting, and revised the figures and manuscript (the revisions are highlighted in red). The other core design elements of MM-PoCo, including separate actor-critics, the shared task reward, and the postural penalty on the torso and leg modules only, remain unchanged.

W2: Position against prior neuromechanical models

Thank you for sharing these works. We agree that they are closely related to MM-PoCo.

These neuromechanical studies divide the control of human locomotion into modules inspired by the human neuromuscular system, including a module dedicated to balance [1, 2, 3]. MM-PoCo shares this philosophy of modular control. The main difference is that MM-PoCo is a learning-based method that trains each module as a neural network with RL, and we apply it to whole-body standing and manipulation tasks.

We have added these works to the Related Work section and clarified the methodological contribution of MM-PoCo relative to prior neuromechanical control approaches (the revisions are highlighted in red).

[1] H. Geyer and H. Herr. A muscle-reflex model that encodes principles of legged mechanics produces human walking dynamics and muscle activities. IEEE TNSRE, 2010.

[2] A. Di Russo et al. Investigating the roles of reflexes and central pattern generators in the control and modulation of human locomotion using a physiologically plausible neuromechanical model. Journal of Neural Engineering, 2023.

[3] S. Song and H. Geyer. A neural circuitry that emphasizes spinal feedback generates diverse behaviours of human locomotion. The Journal of Physiology, 2015.

AuthorsOpenReview ↗

Response to Reviewer W2R1 (2/2)

W3: Evidence for APA-like behavior

Thank you for suggesting additional experiments to better support the motivation of our work. Since MM-PoCo was inspired by human anticipatory postural adjustments (APAs), we conducted two additional analyses to examine whether similar patterns emerge in the learned controller. Here, APAs refer to postural adjustments made before movement to maintain balance, rather than reactive recovery after balance is lost [4].

To examine whether APA-like patterns emerge in the learned controller, we analyzed trunk muscle activation around target changes in the drawing task. The torso muscles co-contract within 60 ms before the new target is observed. The end effector stays near the reached target until the new target is observed and moves away only afterward, so the trunk activation precedes the arm movement toward the next target. Consistent with findings from human studies [5], this pattern emerges even though the agent cannot predict the direction of the next target, since its observation contains only the error to the current target.

We additionally analyzed whether the controller maintains balance during arm movement by measuring the CoM relative to the base of support in the draw task. We evaluated the final models over 200 episodes per seed and found that the CoM was outside the base of support in only 0.2% of the steps. This indicates that the postural modules maintain posture and equilibrium while supporting arm movement.

Together, these analyses show that the agent trained with MM-PoCo learns to maintain balance and exhibits APA-like patterns similar to those observed in humans, rather than relying on reactive balance recovery. We have added a new experimental section (Section 5.5, Figure 5) presenting these analyses to strengthen the empirical support for our work (the additions are highlighted in red).

Regarding the one-way information flow arm < torso < leg, as described in our response to W1, the revised MM-PoCo gives every module the full observation and therefore no longer imposes this flow.

[4] J. Massion. Movement, posture and equilibrium: interaction and coordination. Progress in Neurobiology, 1992.

[5] D. Piscitelli, A. Falaki, S. Solnik, and M. L. Latash. Anticipatory postural adjustments and anticipatory synergy adjustments: Preparing to a postural perturbation with predictable and unpredictable direction. Experimental Brain Research, 2017.

W4, Q2: Observation details and supplementary videos

We have provided videos of the learned behavior and additional details on the observation space of each module (Appendix B, Table 4).

As shown in the videos, MM-PoCo maintains upright posture and equilibrium while swaying naturally within the feasible region, rather than holding a stiff pose. The videos also demonstrate how the agent maintains balance under external perturbations, including a moving platform and ball strikes. In failure cases, the agent leans and falls, ends up on one leg and falls, or keeps standing but misses the target.

For the observation space, all modules receive the same full observation as the monolithic baseline, including task information, joint positions and velocities, and balance-related quantities. The detailed observation keys and their dimensions are provided in Table 4 of Appendix B (the additions are highlighted in red).

Thank you for the detailed response and for running all of the controls I suggested, plus the compute-matched baseline. The new related work, the observation details, and the CoM and trunk-activation analysis all make the paper clearer. However, the new results raise a few concerns of their own.

The observation ablation shows that observation design barely matters. Every variant reaches roughly 95-99%, including all six nesting orders, separate observations and full observations. The original submission motivated nesting by the body's chain ("a module can support the modules above it only if it knows where they are"). The revision instead motivates full observation by saying that the postural modules need information about the other modules. But with separate observations, the postural modules can't see the arm at all and still reach 96.9%, so the results support neither argument. Moving to full observation is a reasonable engineering choice, but it drops the information-flow part of the human-inspired framing of the paper. What's left is a hand-picked arm/torso/leg split and the way the penalty is assigned. When the arm also gets the terminal penalty, success falls from 99.4% to 1.6% (Fig. 7). That makes me think most of the gain comes from the penalty assignment rather than from the biological structure.

I also have some concerns about the baselines. The single policy stays near zero on myosuite-draw with all three algorithms, even when compute-matched. A simple sanity check would help, for example showing that your single-policy SAC can solve the original arm-only MyoSuite reaching task. The QFLEX results suggest that something may not be working as intended. On ms-human-stand, episodes last only 6 to 10 steps (about 0.1 to 0.2 seconds), and QFLEX + MM-PoCo actually gets worse during training on myosuite-draw and myosuite-stand (Fig. 2). Additionally, in Fig. 2, MM-PoCo is already well ahead at the first point of training, even though all methods start from random networks.

I was also surprised not to see PPO, which is the usual choice for full-body musculoskeletal control. MuscleMimic (Li et al., 2026), which you cite, trains a single PPO policy on a 416-muscle full-body model to imitate a wide range of motions while staying balanced. Imitation is a different setting, but it shows that a single policy can handle this kind of model. That goes against the claim that the difficulty "lies in the structure of the policy rather than in the learning algorithm."

I'm glad you tested for anticipatory postural adjustments in Section 5.5, but I don't think the analysis shows anticipation yet. The next target appears as soon as the current one is reached, so the trunk activation before the switch happens while the arm is still finishing its previous reach (Fig. 5c). Comparing trunk onset with arm-muscle onset, as human APA studies do, would be more convincing. The trunk muscles also stay at around 0.4 activation throughout, which suggests a stiff, co-contracted trunk rather than natural standing.

Overall, I appreciate the work that went into the revision, and I still think the idea is interesting. But the new results have weakened the human-inspired motivation, and I'm not convinced the baselines give a fair comparison. Therefore, I will keep my score.

Thank you for your thoughtful feedback. We have carefully reviewed your concerns.

Human-inspired framing and modular training

We agree that the observation design barely matters. However, our human-inspired framing rests on the three principles stated in Section 3.3 (Motivation): divide the body by function, let posture serve the movement, and treat posture and equilibrium as a feasible region.

Divide the body by function. Ours is divided into the arm for movement, and the other body parts for posture, and each module has its own critic.

Let posture serve the movement. All modules work together to support the arm movement. The reward depends on the distance between the end effector of the arm and the target, so the postural modules optimize for what the arm achieves. This is why the torso receives a reward determined by the position of the arm rather than a separate torso and leg reward term.

Treat posture and equilibrium as a feasible region. To adopt the concept of posture in the RL framework, we use a constraint rather than a reward for a vertical torso. The agent maximizes the task reward as long as it stays inside the feasible region. This matches human balance, staying dynamically within a region rather than holding a fixed posture.

We acknowledge that our original claim about the nested observation was overstated. It was not the core of our argument, and the revision no longer presents it as human-inspired.

Baselines

One point we can state clearly: in an arm-only environment with the same draw task, SAC converges to a success rate close to 1 within 4M steps, and so does QFLEX. If helpful, we can add these curves to the appendix, together with videos.

We agree that QFLEX collapses. Although not reported in the paper, the modular approach also worked well with other algorithms (e.g., TD3), but the architecture design mattered. They needed to adopt an 8-layer network with LayerNorm and ensemble critics. Since RL is highly sensitive to hyperparameters, we tried to minimize their effect by applying nearly identical settings to every base algorithm. For QFLEX, however, the use of CrossQ is its defining feature, and we respected it. QFLEX is not well suited to our setting, but we included it because we believe the most recent baseline should be reported.

PPO

We agree that PPO is the usual baseline for musculoskeletal models, and it was the first algorithm we considered. Our goal was to test whether the torso and legs can support the arm as it moves quickly and precisely over a wide region in front of the body, so we widened the target range box and set the draw task to reach seven random targets in sequence. This sequential task is considerably harder than the single-target reaching task of MyoSuite. PPO does solve it in the arm-only setting, but its sample efficiency is far lower, so we restricted the base algorithms to off-policy methods.

Anticipatory postural adjustments

Inspired by APAs, our goal was to keep the CoM inside the BoS without any reward term that drives the CoM toward a stable region. What our trunk anticipates is the following. Given the one-step target distance, it produces an action that keeps the body balanced at the next step. We regard this as anticipatory, but we do not claim that it is identical to human APAs. We are grateful for the human APA studies you suggested to us. We will lower the claims of Section 5.5 and complete it accordingly, and we are willing to show what you suggested in future work.

Reviewer_c31kOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MM-PoCo, a modular RL framework for whole-body musculoskeletal control. The action space is split into arm, torso and leg modules, each with its own actor-critic, nested observations, and the shared task reward. Posture is imposed as a constraint via early termination when an upper-body landmark leaves a 0.15 m tolerance ball, with the terminal penalty charged only to the postural modules. On four whole-body tasks built from MyoSuite and MS-Human-700, MM-PoCo improves SAC, DynSyn, and QFLEX in 10 of 12 settings. Ablations show early termination outperforms reward-based and Lagrangian alternatives, and the policy remains robust to unseen perturbations.

Critical strengths
  1. Well-motivated and algorithm-agnostic design. Splitting hundreds of muscles into cooperating functional modules is an interesting and natural way to tame whole-body musculoskeletal RL. The movement decomposition and posture-as-feasible-region are grounded in motor-control literature, and only change the policy structure and termination condition, so they plug into existing RL algorithms without modification.

  2. Large gains on hard whole-body manipulation. On tasks where single policies almost completely fail (<=2.1% on draw, <=18.5% on manipulation), MM-PoCo reaches 88--93% (SAC/DynSyn) and 45--69%, across two musculoskeletal models.

  3. Informative ablation on the postural constraint. The comparison of six specifications clearly shows that early termination is essential, and the zero-shot perturbation tests add evidence of robustness.

Critical weaknesses
  1. Gains may partly come from extra capacity and compute. Each module has its own actor-critic and is updated separately, so MM-PoCo uses about 2--3x the networks and gradient updates of the single-policy baseline (Tables 6, 9).

  2. Evaluation is not rigorous enough. Table 1 is computed from training episodes with only 3 seeds, and the draw success rate is defined differently in Sec. 5.1 (all seven targets reached) and Appendix B (fraction of targets reached).

  3. Novelty and scope are limited. Decomposing whole-body control into jointly trained modules has been explored before (e.g., FALCON) but is not discussed, and all tasks only require the torso and legs to hold a static posture.

Questions for authors
  1. Compute-matched ablation. Please compare MM-PoCo against (a) a single policy matched in total parameters and gradient updates, (b) a shared-trunk policy with separate arm/torso/leg action heads, and (c) a non-anatomical partition of the muscles. This would show whether the gain comes from the functional decomposition itself.

  2. Independent evaluation and metric clarification. Please report Table 1 results with deterministic evaluation on a fixed set of episodes at the final checkpoint (ideally with more seeds), and clarify which definition of the draw success rate is used.

  3. How does MM-PoCo differ from prior modular whole-body control (e.g., FALCON) beyond the musculoskeletal setting, and which components are new?

  4. Can the authors add a dynamic task where the postural modules must also move to show that the movement decomposition still holds?

2 comments
AuthorsOpenReview ↗

Response to Reviewer c31k

Thank you for taking the time to review our paper. We sincerely appreciate your feedback. Below, we provide detailed responses to your concerns.

W1, Q1: Compute-matched ablations

To test whether the gain comes from the functional decomposition itself, we ran all three ablations you proposed:

(a) Compute-matched single policy: A single policy matched to MM-PoCo in total parameters and gradient updates (success rate 0.5%).

(b) Shared-trunk policy: A shared-trunk policy with separate arm, torso, and leg action heads (success rate 0.1%).

(c) Non-anatomical partition: A non-anatomical partition of the muscles (success rate 0.6%).

None of these variants comes close to MM-PoCo (99.4%; Appendix D, Figure 6). This indicates that the gain comes from the anatomically informed functional decomposition rather than from extra capacity or compute.

W2, Q2: Evaluation protocol and draw success metric

We agree with your concern and have clarified the evaluation process and metrics accordingly.

To improve the rigor of the evaluation, we re-evaluated all results in Table 1 using deterministic evaluation at the final checkpoint and increased the number of training seeds from three to five. We also corrected an inconsistency in the definition of the draw success rate. The correct metric is the fraction of the seven targets reached (Appendix B), rather than the fraction of episodes in which all seven targets are reached. Section 5.1 and the related descriptions have been revised accordingly (all revisions are highlighted in red). We apologize for the confusion and thank you for the constructive feedback.

W3, Q3: Difference from prior modular whole-body control (e.g., FALCON)

FALCON [1] and MM-PoCo are similar in that both decompose whole-body control into multiple modules and train them jointly. However, they differ in their learning objectives and how they handle postural stability.

FALCON assigns manipulation and locomotion rewards to the upper-body and lower-body agents, respectively, optimizing separate task objectives. In contrast, MM-PoCo shares a common task objective across all modules while applying postural constraints differently to each module. This allows the postural modules to support arm movement toward the same task goal rather than optimizing separate tasks.

These rewards also include postural terms such as torso orientation. MM-PoCo instead imposes posture as an early-termination constraint, and this constraint outperformed reward penalties in our experiments (Section 5.3, Figure 3).

We have added FALCON to the Related Work section and clarified the methodological contributions of MM-PoCo (the revisions are highlighted in red).

[1] Y. Zhang et al. FALCON: Learning force-adaptive humanoid loco-manipulation. L4DC, 2026.

Q4: Extension to dynamic tasks

Thank you for the suggestion. Extending MM-PoCo to dynamic tasks where postural modules must also move actively would be a meaningful direction to demonstrate its broader applicability.

In this work, we focused on maintaining posture during standing and goal-directed arm movements. Whole-body standing is itself a challenging control problem [2], and arm movements make it more difficult by introducing perturbations that must be compensated for while maintaining balance. We demonstrated the effectiveness of MM-PoCo in these challenging settings.

As future work, we plan to extend MM-PoCo to dynamic tasks, such as manipulation requiring larger torso movements or goal-directed arm movements during locomotion.

[2] C. Ma, Y. Wei, C. Zuo, C. Zhang, and Y. Sui. Bipedal balance control with whole-body musculoskeletal standing and falling simulations. CoRL, 2025.

Dear Reviewer c31k,

Thank you again for your thoughtful review. As the discussion period is ending soon, we wanted to briefly follow up.

To address your concerns, we (1) added the compute-matched ablations you proposed to clarify where the gain comes from, with parameters and gradient updates matched to MM-PoCo (Appendix D, Figure 6), (2) re-evaluated all results in Table 1 using the final checkpoint over five seeds and unified the definition of the draw success rate, (3) added FALCON to the Related Work section and discussed how MM-PoCo differs from it and which components are new, and (4) shared our view on your suggestion of dynamic tasks. All changes are highlighted in red in the revised PDF.

We respectfully invite you to revisit our response. If there are any remaining questions or points that would benefit from further clarification, we would be happy to address them.

Thank you again for your time and consideration.

Reviewer_H2AcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Drawing inspiration from human movement such as modularity and postural control, the authors propose a method to learn whole body control for musculo-skeletal systems in simulation. At the core, the method decomposes a muscle-actuator controls of a humanoid whole-body (the entire kinematic chain) into disjoint action spaces and train separate policies for each subspace using existing reinforcement learning (RL) algorithms . Acting together, these separate policies outperform a single policy in most tested configurations and the strongest results in manipulation. Further, the authors argue that imposing posture as a constraint helps boost the robustness of the learned motion. The results show substantial improvement on two whole body task categories - standing and standing + manipulation over a single policy baseline. An ablation on answer the question - what is the most effective way to impose postural constraint? adds an interesting dimension to this empirical study.

Critical strengths
  1. This paper is a good empirical study of one approach which can scale to learning coordinated whole body motion for neuro-muscular systems. A major strength of the paper is that it is a simple formulation which works with existing RL algorithms.
  2. The improvement over baseline is substantial - especially because other variables such as task rewards and early terminations are kept constant. Section 5.3, also an interesting study about how to incorporate postural constraint into learning whole body control - perhaps something which can be carried over for other more-complex movements.
  3. On a related note, the authors do a good job of designing thoughtful experiments and ablations.
Critical weaknesses
  1. The experiments do not justify the modularization of the action space. Neither does it justify the nested observation design choice. To make the contribution stronger, a reader needs to be convinced that the chosen modularization of the action space is backed by evidence. It might be useful to compare against - a different actuator partitioning or full observation for all modules
  2. "Anticipatory postural adjustments" framing has no backing. I did not find experimental evidence in the learned controller that postural actions or forces precede arm movements for either tasks.
  3. The paper appears to be missing a relevant related work - Peng et al FACMAC: Factored multi-agent centralized policy gradients https://arxiv.org/pdf/2003.06709 and Lee et al "Learning Humanoid Arm Motion via Centroidal Momentum Regularized Multi-Agent Reinforcement Learning" https://arxiv.org/abs/2507.04140. It might strengthen the paper to position itself relative to relevant MARL approaches.
Questions for authors
  1. Why use independent module critics instead of established centralized training approaches?
  2. On the observation space - the paper is missing details on what exactly does the observation space for each policy contain. I noticed numbers in the appendix of the size of observation tensor - but what information is in there is important. Especially because of the choice to go with nested observations.
  3. For balance, arms can play an important role especially under external perturbation - think about when humans are pushed unexpectedly, we use arms to regain balance. So, it surprises me that arms were kept passive for the standing task. Do you expect the balancing ability to improve if arms were included?
  4. Including supplementary video would vastly strengthen the paper.
3 comments
AuthorsOpenReview ↗

Response to Reviewer H2Ac (1/2)

Thank you for taking the time to review our paper. We sincerely appreciate your feedback. Below, we provide detailed responses to your concerns.

W1: Justification of modularization and observation design choices

Thank you for the constructive comment. We ran two additional ablations, one for each design choice.

(1) Body-part modularization is essential.

We replaced our partitioning of the action space by body section (arm, torso, leg) with a random actuator partitioning into the same number of modules. With random partitioning, the policy fails to learn the task at all (near-zero success rate). Our body-part partitioning reaches above 94% in every observation setting we tested, while the monolithic single policy also fails (0.1%). Naively splitting the action space is therefore not enough. The gain comes from partitioning it along the structure of the body (Appendix D, Figure 6).

(2) Nested observation is not required.

We compared full observation for all modules with nested observation in all six orders (3 seeds each). The previous nested observation learns faster early in training, but full observation matches or slightly exceeds it in final performance: full observation reaches 99.2%, and the nested order used in the original draft (arm < torso < leg) reaches 98.2% (Appendix D, Figure 8). Based on this result, we now use full observation as the default design of MM-PoCo. We reran the main experiments and all other experiments in the paper, updated every figure, and revised the method section to match. What MM-PoCo relies on is the body-part modularization and a separate critic for each module, with one task reward shared by all modules and the postural penalty on the torso and leg modules only.

W2: Experimental evidence for APA-like behavior

Since MM-PoCo was inspired by human anticipatory postural adjustments (APAs), we conducted two additional analyses to examine whether similar patterns emerge in the learned controller. Here, APAs refer to postural adjustments made before movement to maintain balance, rather than reactive recovery after balance is lost [1].

To examine whether APA-like patterns emerge in the learned controller, we analyzed trunk muscle activation around target changes in the drawing task. The torso muscles co-contract within 60 ms before the new target is observed. The end effector stays near the reached target until the new target is observed and moves away only afterward, so the trunk activation precedes the arm movement toward the next target. Consistent with findings from human studies [2], this pattern emerges even though the agent cannot predict the direction of the next target, since its observation contains only the error to the current target.

In the same episodes, the CoM stayed within the narrow region and was outside the base of support in only 0.2% of the steps, so the postural modules keep balance during the arm movement rather than recovering it after it is lost.

Together, these analyses show that the agent trained with MM-PoCo learns to maintain balance and exhibits APA-like patterns similar to those observed in humans, rather than relying on reactive balance recovery. We have added a new experimental section (Section 5.5) presenting these analyses to strengthen the empirical support for our work (the additions are highlighted in red).

[1] J. Massion. Movement, posture and equilibrium: interaction and coordination. Progress in Neurobiology, 1992.

[2] D. Piscitelli, A. Falaki, S. Solnik, and M. L. Latash. Anticipatory postural adjustments and anticipatory synergy adjustments: Preparing to a postural perturbation with predictable and unpredictable direction. Experimental Brain Research, 2017.

AuthorsOpenReview ↗

Response to Reviewer H2Ac (2/2)

W3, Q1: Comparison with related MARL approaches

Thank you for pointing out these relevant works. Our approach is also closely related to MARL, as it treats different body components of a human musculoskeletal model as separate agents. We would like to clarify how MM-PoCo differs from these prior approaches.

FACMAC [3] uses a shared team reward with a centralized factored critic and joint policy optimization to coordinate agents. In contrast, MM-PoCo maintains a shared task reward while assigning different postural responsibilities to each module. Specifically, the arm receives only the task reward, while the torso and legs receive the same task reward with an additional postural penalty. Lee et al. [4] also use separate actor-critic structures and module-specific rewards, but assign distinct task objectives to the arms and legs, namely CAM regulation and locomotion tracking. In contrast, MM-PoCo maintains a common task objective across all modules while assigning postural constraints only to the torso and legs.

We also found that applying the same task reward and postural penalty to all modules causes training to fail (Appendix D, Figure 7). MM-PoCo therefore gives each module its own critic: the arm module receives the task reward alone, and the torso and leg modules receive the task reward plus the postural penalty, so each critic is trained independently on its own module's reward.

We have added the suggested works to the Related Work section and clarified the differences between MM-PoCo and existing MARL approaches (the revisions are highlighted in red).

[3] B. Peng, T. Rashid, C. Schroeder de Witt, P.-A. Kamienny, P. Torr, W. Böhmer, and S. Whiteson. FACMAC: Factored multi-agent centralised policy gradients. NeurIPS, 2021.

[4] H. J. Lee, S. H. Jeon, and S. Kim. Learning humanoid arm motion via centroidal momentum regularized multi-agent reinforcement learning. IEEE Robotics and Automation Letters, 2025.

Q2: Table of observation details

To clarify its contents for each task, we have added a table to Appendix B (Table 4) listing the observation keys and their dimensions (the additions are highlighted in red). The observation space provides information about the task, whole-body joint states, and balance-related quantities.

Q3: Rationale for passive arms in the standing tasks

As you pointed out, arms play an important role in maintaining balance, and we expect that actively using the arms could further improve balancing performance, particularly under unexpected external perturbations.

In this work, however, we designed the standing tasks to examine whether stable standing could be learned using only the torso and leg modules, without active arm control. To this end, we set the arm muscle activations to zero, allowing the arms to hang passively (Appendix B). Unlike the manipulation tasks, which require both arm movement and posture maintenance, the standing tasks focus on posture maintenance alone.

Whole-body standing is itself a challenging control problem [5], and we showed that MM-PoCo can learn stable standing even under this restricted setting. As future work, we plan to extend the standing tasks to include active arm control, particularly to improve balancing performance under external perturbations.

[5] C. Ma, Y. Wei, C. Zuo, C. Zhang, and Y. Sui. Bipedal balance control with whole-body musculoskeletal standing and falling simulations. CoRL, 2025.

Q4: Supplementary videos

We have added videos to the supplementary material. Thank you for the suggestion.

Dear Reviewer H2Ac,

Thank you again for your thoughtful review. As the discussion period is ending soon, we wanted to briefly follow up.

To address your concerns, we (1) added ablations showing that the body-part partition is essential (a random partition fails) and that nested observation is not required, and revised the method so that every module receives the full observation, (2) added Section 5.5 to provide evidence that our method keeps balance in an anticipatory manner rather than by recovering it after it is lost, (3) added FACMAC and Lee et al. to the Related Work section and clarified how MM-PoCo differs from these MARL approaches, and (4) added a table of the observation contents (Appendix B, Table 4) and supplementary videos. All changes are highlighted in red in the revised PDF.

We respectfully invite you to revisit our response. If there are any remaining questions or points that would benefit from further clarification, we would be happy to address them.

Thank you again for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear reviewers,

We sincerely appreciate the time and effort you have dedicated to reviewing our paper. Your insightful comments have been incredibly valuable in refining and improving our work.

As reviewers highlighted, our paper presents a simple, algorithm-agnostic, and well-motivated design for whole-body musculoskeletal control (H2Ac, c31k, W2R1). In controlled experiments, MM-PoCo outperforms single policies with a simple formulation that works with existing RL algorithms (H2Ac, c31k, W2R1). Treating posture as a constraint is well argued and supported by the ablation (H2Ac, c31k, W2R1) and the zero-shot robustness evaluation (c31k, W2R1). These results are complemented by clear writing and thoughtful experiments (H2Ac, c31k).

In response to your feedback, we have conducted additional experiments that clarify where the gains of MM-PoCo come from, and updated all experiments in the revised draft accordingly (changes highlighted in red). Below are the key updates:

  • Permutation experiments isolating each component of MM-PoCo: (a) random muscle partition, (b) full, nested, and separate observations, and (c) the posture penalty on all modules (Appendix D, Figures 6–8; H2Ac, c31k, W2R1).
  • Revised method that treats nested observation as a design choice, with all experiments rerun so that every module receives the full observation (Sections 3–5, Table 1, Figures 2–5; H2Ac, W2R1).
  • Experimental evidence of APAs, as distinct from reactive correction of CoM deviations (Section 5.5, Figure 5; H2Ac, W2R1).
  • Compute-matched ablation in parameters and gradient updates (Appendix D, Figure 6; c31k).
  • Additional references on multi-agent RL, modular whole-body control, and neuromechanical models (Section 2; H2Ac, c31k, W2R1).
  • Expanded appendix and supplementary videos (Appendices B–D, Supplementary material; H2Ac, W2R1).

We believe that MM-PoCo can be a useful addition to the ICLR community, and your feedback has helped us improve the clarity and depth of our work. Thank you once again for your constructive reviews and support.

Authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.