acceptodds
Under review as a conference paper at ICLR 2027

Analytic Policy Gradients for Contact-Rich Loco-Manipulation in Soft Robots

Abstract

Analytic policy gradients are widely considered impractical for contact-rich manipulation, where contact discontinuities induce bias and stiff or chaotic dynamics can produce high-variance estimates and exploding gradients over long horizons. Contrary to this view, we demonstrate contact-rich, soccer-inspired loco-manipulation in soft robots with up to 2,420 actuated springs, trained end-to-end with first-order gradients through simulation and control. On one commodity GPU, a single agent learned to locate, pursue, dribble, and kick the ball through the goal in 100 gradient steps—less than a minute of wall-clock time. In multi-agent settings, cooperative "passing" emerged strictly from task loss without explicit behavioral shaping, and in a competitive setting a defender tasked with preventing goals reduced its opponents' scoring rate by up to 60%. Gradient analysis reveals that sustained contact events inflate gradient energy by up to two orders of magnitude and leave per-window gradients largely misaligned, yet batch aggregation yields a reliable training signal. We also find that gradient energy is skewed away from the wide output layer that drives spring actuation, starving it of the sparse contact-mediated learning signal under plain stochastic gradient descent (SGD). With per-layer rescaling, SGD nearly matches Adam without per-coordinate adaptation. We release our simulator and task environment as a scalable, fully differentiable foundation for contact-rich, multi-agent learning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
soft robots, analytic policy gradients, differentiable simulation, loco-manipulation, contact-rich manipulation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_92NxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows that analytic policy gradients can train closed-loop, contact-rich loco-manipulation in high-dimensional simulated soft robots, and examines the roles of batch aggregation and layer-wise update scaling. A deterministic neural controller actuates individual springs in a differentiable mass–spring simulator. Training backpropagates through 500-step simulation windows, with gradients truncated at window boundaries and no learned critic or zeroth-order gradient estimates. The objective combines approach, ball transport, boundary, and actuation costs. The main experiment compares four optimizers on a quadruped over 20 random seeds. Additional experiments investigate batch size and contact-gradient statistics, extend the approach to biped and hexapod morphologies with a separate policy trained for each, and explore cooperative and competitive settings involving quadruped robots.

Critical strengths

The work is well motivated by the challenges of using analytic gradients for contact-rich control, particularly in soft robots with high-dimensional action spaces. The experiments provide convincing evidence that direct, windowed analytic policy gradients can successfully train controllers in the studied setting. The main optimizer comparison uses 20 random seeds, while separately trained controllers across three morphologies and extensions to multi-robot tasks broaden the demonstrated applicability. It also provides informative gradient analysis and optimizer comparison. Limitation is also discusse.

Critical weaknesses
  1. Limited comparison evidence with previous methods (e.g., SHAC) in same environmental settings.
  2. Unclear characterization of several training factors, e.g., sensitivity to window length, the contribution of batch aggregation versus increased simulation budget, and robustness to contact stiffness and damping.
Questions for authors

To support the comparison with SHAC’s update requirements, can you provide a same-environment comparison reporting overall performance? If this is not feasible, please clarify the limitations of the cross-method comparison.

How sensitive is performance to the backpropagation window length? Since gradients are truncated without a terminal critic, a window-length ablation reporting performance and computational cost would clarify the dependence on the chosen 500-step horizon.

Does the advantage of larger batches persist under a matched total simulation-transition budget, with smaller batches receiving more updates?

3 comments

Response to Reviewer 92Nx (1/3)


We thank the reviewer for their time and thoughtful assessment. We appreciate that the reviewer found the experiments to provide “convincing evidence that direct, windowed analytic policy gradients can successfully train controllers in the studied setting,” which is the paper’s central claim. We also appreciate that they found the morphologies and multi-robot tasks to broaden the demonstrated applicability and the gradient analysis and optimizer comparison informative.

Below we address the stated weaknesses (W1 −- W2) and questions (Q1 −- Q3), one by one, also referencing how our manuscript was modified in response. The updated manuscript has been uploaded to OpenReview with additions marked in red text. We believe all changes have significantly improved the paper. In particular, investigating the reported batch advantage under equal simulation budget (W2) has produced new results that we believe provide a broader, more informative perspective on the sufficiency of analytic policy gradients in this setting. We are grateful to the reviewer for prompting this study.


W1. “Limited comparison evidence with previous methods (e.g., SHAC) in same environmental settings.”

We agree this is a limitation of our work. Our claims are limited to the sufficiency of analytic policy gradients, and we do not claim superiority over alternative first-order methods such as SHAC. One sentence in the original Introduction did draw a quantitative contrast with SHAC. But because the task environments fundamentally differ, and because a “gradient step” is not a shared unit—a SHAC policy update simulates shorter rollouts, and its many critic updates reuse rollouts—this comparison may be uninformative or misleading, and we have removed it:

Despite the added complexity of our setting—soft bodies in place of rigid, an order of magnitude larger action space, and an extension to contact-rich loco-manipulation—we found the critic unnecessary, training with 20x fewer policy updates and, without the critic's > 10^5 updates, three orders of magnitude fewer gradient steps in total.

We consider a same-environment SHAC comparison beyond the scope of this paper, and we appreciate the reviewer acknowledging its possible infeasibility (Q1). To the best of our knowledge, SHAC has not been demonstrated on soft body loco-manipulation, so a credible baseline would require adapting and tuning it for this setting, and a poorly tuned baseline would be uninformative. To clarify this, as well as the limitations of a cross-method comparison with SHAC, we added this to the Discussion:

Finally, we did not compare against critic-based short-horizon methods such as that of Xu et al. (2021b) in our environment. Although our setting is more complex in body, action dimension, and task, rigid-body dynamics pose difficulties of their own for first-order methods, and a detailed understanding of when analytic gradients suffice and when a critic is required remains to be established.


Response to Reviewer 92Nx (2/3)


W2. “Unclear characterization of several training factors, e.g., sensitivity to window length, the contribution of batch aggregation versus increased simulation budget, and robustness to contact stiffness and damping.”

To remedy this we conducted the following experiments:

  1. We tested sensitivity to window length HH and reported results in an addition to Appendix A.2.2 (pasted below). To summarize, the final scoring rates were 0.000.00 (H=50H = 50), 0.000.00 (125125), 0.52±0.060.52 \pm 0.06 (250250), and 0.84±0.050.84 \pm 0.05 (500500).

    We tested additional window lengths (Sect. 2.5), H∈{50,125,250}H \in \lbrace 50, 125, 250 \rbrace alongside the baseline H=500H = 500, training the quadruped with Adam and scaling the number of gradient steps by 500/H500/H to match the baseline’s total simulation steps (Tab. 4). At H≤125H \le 125, policies collapsed to inaction and never contacted the ball. These windows were too short—less than one gait cycle of the controller—to reduce the approach loss through locomotion toward the ball, so the effort penalty dominated and actuation collapsed. At H=250H = 250 (≈1.8\approx 1.8 gait cycles), locomotion was learned comparably to the baseline but manipulation was poor (scoring rate 0.520.52 vs. 0.840.84). Total simulated steps were matched by construction, and shorter windows required less wall clock per update. However, the fixed overhead of each forward and backward pass accumulated over more updates, so total training time rose modestly as windows shortened (4848 to 7070 s).

    We also added to Section 3.1 and the Discussion to clarify sensitivity to window length:

    Performance depended strongly on window length: when the gradient horizon was shorter than about one gait cycle the policy collapsed to inaction, and with intermediate windows (250 steps) policies reached the ball as quickly as the baseline (2.972.97 vs. 3.423.42 s) but scored far less often (0.520.52 vs. 0.840.84; Appx. A.2.2).

    Window length was a prerequisite: below roughly one gait cycle no learning occurred (Appx. A.2.2).

  2. We tested the advantage of large batches when the total simulation budget was constant and reported results in Section 3.2:

    The fixed-update sweep (Fig. 5), however, gave smaller batches fewer total simulation steps and a larger learning rate per sample. With a matched simulation budget and the learning rate scaled to batch size, every B≤128B \le 128 reached final scoring rates of 0.920.92–0.970.97, exceeding the 100100-update B=256B = 256 baseline (Appx. A.2.2). A reliable manipulation signal thus could be obtained by aggregating a sufficient number of per-window gradients, whether within a large batch or, consistent with the linear scaling rule (Goyal et al., 2017), across many smaller updates.

    And Appendix A.2.2:

    We retrained B∈{32,64,128}B \in \lbrace 32, 64, 128 \rbrace for 800800, 400400, and 200200 updates, matching the baseline’s (B=256B = 256) simulation budget, first at the original learning rate (Appx. A.1.3) and then with the learning rate scaled down by batch size (Tab. 3). Matching the simulation budget alone recovered B=128B = 128, which exceeded the baseline’s scoring rate. At B≤64B \le 64, increasing the number of updates alone was insufficient, but scaling down the learning rate brought both (B∈{32,64}B \in \lbrace 32, 64 \rbrace) above the baseline (0.930.93 and 0.920.92 compared to 0.840.84), with no seed below 0.700.70, compared with a worst seed of 0.100.10 for the baseline. Smaller batches—paired with a greater number of small steps—thus produced more stable training and erased the batch size advantage reported in Figure 5. However, consistent with Figure 5, every configuration learned locomotion, with time to contact ranging from 2.92.9 to 4.44.4 s. After scaling learning rates, B∈{32,64}B \in \lbrace 32, 64 \rbrace bettered the baseline's 3.423.42 s, while the unscaled-learning-rate configurations lagged by about 11 s. Because smaller batch sizes required more gradient steps they were 44-7×7\times slower in wall clock over the same simulation budget.

    In light of these results, we updated the manuscript (Abstract, Introduction, Sect. 3.2, and Discussion) to reflect that batch aggregation is not the operative factor per se. What the manipulation signal requires is sufficient aggregation of per-window gradients, which can come from larger batches or from additional updates.

We also appreciate the reviewer’s interest in sensitivity to contact stiffness and damping. The original Discussion named the single, penalty-based contact model as a limitation and stiffness as a direction for future work. We have now added damping to that sentence. We believe sweeping these parameters would be most informative once the simulator is calibrated to a physical system for real-world transfer, which is left for future work. As additional characterization of training factors, we ablated the cosine learning-rate schedule—results are reported in Appx. A.2.2, Table 5.


Response to Reviewer 92Nx (3/3)


Q1. “To support the comparison with SHAC’s update requirements, can you provide a same-environment comparison reporting overall performance? If this is not feasible, please clarify the limitations of the cross-method comparison.”

Please see our response to W1.


Q2. “How sensitive is performance to the backpropagation window length? Since gradients are truncated without a terminal critic, a window-length ablation reporting performance and computational cost would clarify the dependence on the chosen 500-step horizon.”

Please see our response to W2.


Q3. “Does the advantage of larger batches persist under a matched total simulation-transition budget, with smaller batches receiving more updates?”

Please see our response to W2.


We thank the reviewer for their consideration of our response and gladly welcome any further questions or comments.


Reviewer_CZQZOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper demonstrates soccer-inspired contact-rich loco-manipulation of soft robots trained using analytic policy gradients through a differentiable simulation. Policies are trained in single-agent, cooperative multi-agent, and competitive multi-agent settings. The legged robot morphologies span biped, quadruped, and hexapod with high-dimensional action spaces and soft-body dynamics. Large batch sizes and adaptive learning rate rescaling are used to handle large gradient spikes and misalignment under sustained robot-ball contact interactions.

Critical strengths

Demonstrating the practical use of analytic policy gradients in a multi-agent setting is novel. The particular problem setting is also high-dimensional, with up to ~2400 actuated springs in soft robot loco-manipulation, with single-agent policies trained in less than a minute of wall-clock time using under 100 gradient steps.

Critical weaknesses

The presentation of the experimental results and the main contribution of the paper is unclear, which makes it difficult to evaluate the work.

  • The authors show that vanilla analytic policy gradients (APG) is sufficient to learn policies in the problem setting, but there are no comparisons to zero-order policy gradients (REINFORCE), or more sophisticated first-order methods that use short-horizon rollouts or a learned terminal critic. It would be helpful to understand why APG by itself is sufficient to train policies in this problem setting. My current understanding is that moderate batch sizes (32-256 parallel envs with 500 time steps with a long horizon rollout) effectively reduces the variance. Would using 4096 parallel envs with 32 time step short horizon rollouts be equivalent?

  • The introduction of SGD-layer in Section 3.2 is not clear, when the following multi-agent experiments in Section 3.3 just use Adam anyway. The performance also appears to be weaker than Adam in the single-agent setting. Section 2.8 also mentions using a cosine learning rate scheduler, which is not ablated against.

  • The authors demonstrate both single-agent contact-rich loco-manipulation with different robot morphologies, and multi-agent cooperative and competitive settings. The writing is unclear for readers about the primary contributions of the paper, whether different morphologies, single-agent, multi-agent, or gradient analysis and SGD-layer optimizer.

Questions for authors
  • Figure 2: Does the simulator support scenes other than a circular field? Or is the circular field necessary for the reward function definition?

  • The experiments use 100 to 600 gradient steps, which only requires a few minutes on a single consumer GPU. Does performance plateau, or would performance keep improving with 1e4 or 1e6 gradient steps?

  • Section 3.3: The multi-agent experiments use two or three agents. Is it possible to train 10 or even 100 agent tasks, or is there a GPU memory limitation?

  • In the cooperative pair setting, what is the “shared policy” architecture? Does the neural network output actions for both robots together? Or is the neural network run twice? It would be interesting to understand how different multi-agent learning formulations compare when using analytic policy gradients.

  • The biped, quadruped, and hexapod robots seem to all be the same style of square-like soft robot. Does the simulation support different morphologies? For the multi-agent setting, could the scene support different morphologies as well like biped, quadruped, and hexapod in the same scene?

8 comments

Response to Reviewer CZQZ (1/5)


We thank the reviewer for their time and detailed feedback, and we appreciate that they view “the practical use of analytic policy gradients in a multi-agent setting” as “novel,” as it is a primary contribution of our work. We also appreciate the reviewer naming the scale and efficiency of the control problem as a strength of the paper.

Below we address the stated weaknesses (W1 −- W3) and questions (Q1 −- Q5), one by one, also referencing how our manuscript was modified in response. The updated manuscript has been uploaded to OpenReview with additions marked in red text. We believe all changes have significantly improved the paper. In particular, the reviewer’s comments prompted an ablation of the cosine learning rate scheduler (W2), which revealed an important factor in the success of training with SGD and the nuances of training with analytic policy gradients. We are grateful to the reviewer for this.


Response to Reviewer CZQZ (2/5)


W1. “The authors show that vanilla analytic policy gradients (APG) is sufficient to learn policies in the problem setting, but there are no comparisons to zero-order policy gradients (REINFORCE), or more sophisticated first-order methods that use short-horizon rollouts or a learned terminal critic. It would be helpful to understand why APG by itself is sufficient to train policies in this problem setting. My current understanding is that moderate batch sizes (32-256 parallel envs with 500 time steps with a long horizon rollout) effectively reduces the variance. Would using 4096 parallel envs with 32 time step short horizon rollouts be equivalent?”

We agree that our work demonstrates the sufficiency of bare analytic policy gradients in this setting, and our claims are limited to sufficiency. We do not claim superiority over zeroth-order methods or over first-order methods that use short-horizon rollouts or a learned terminal critic (e.g., SHAC), and our original submission did not compare against them. One sentence in the original Introduction did draw a quantitative contrast with SHAC. But because the task environments fundamentally differ, and because a “gradient step” is not a shared unit—a SHAC policy update simulates shorter rollouts, and its many critic updates reuse rollouts—this comparison may be uninformative or misleading, and we have removed it:

Despite the added complexity of our setting—soft bodies in place of rigid, an order of magnitude larger action space, and an extension to contact-rich loco-manipulation—we found the critic unnecessary, training with 20x fewer policy updates and, without the critic's > 10^5 updates, three orders of magnitude fewer gradient steps in total.

We also consider a same-environment SHAC comparison beyond the scope of this paper. To the best of our knowledge, SHAC has not been demonstrated on soft body loco-manipulation, so a credible baseline would require adapting and tuning it for this setting. We added this as a limitation in the Discussion, also clarifying the limitations of a cross-method comparison:

Finally, we did not compare against critic-based short-horizon methods such as that of Xu et al. (2021b) in our environment. Although our setting is more complex in body, action dimension, and task, rigid-body dynamics pose difficulties of their own for first-order methods, and a detailed understanding of when analytic gradients suffice and when a critic is required remains to be established.

To address the missing zeroth-order baseline, we implemented a REINFORCE estimator, which we tested in place of the simulator’s analytic policy gradient. We swept the learning rate and policy exploration parameter σ\sigma at batch size B=256B = 256 and 100100 gradient updates, identical to the settings in Section 3.1, and later extended training to 10310^3 then 10410^4 gradient updates with the same batch size. Finally, we again tested 10410^4 updates with the largest batch B=1792B=1792 that could fit on our GPU. In short, no REINFORCE configuration produced a policy that ever came into contact with the ball, even when provided 700×700\times the simulation budget. We report complete setup and results in Appendix A.2.2, which is now referenced in Section 3.1 and the Introduction.

Finally, we appreciate the reviewer’s question of whether batch size and window length HH are interchangeable. They are not, for two reasons. First, windows resume episodes from states sampled uniformly from their predecessor window (Sect. 2.5), so an episode advances an expected H/2H/2 steps per update. With H=32H=32, our 100100-update budget would cover less than 55 s of an episode. By contrast, H=500H=500 covers the full 2020 s training horizon in 2727 updates. We clarified this with an addition to Section 2.5:

Because resume states were sampled uniformly, an episode advanced an expected H/2H/2 steps per update, so window length determined how quickly—in training steps—episode states advanced.

Second, the window must be long enough for actuation to produce displacement and reduce the approach loss (Sect. 2.6). We ran the proposed configuration (H=32H=32, B=4096B=4096) for 100100 and for ~15001500 gradient steps, the latter well beyond the ~420420 updates needed to reach the 2020 s episode cap. In both cases the policy collapsed to inaction across all ten seeds—mean squared actuation fell from 0.400.40 at initialization to 0.060.06 and remained there. For reference, successful policies had mean squared actuation near 0.60.6. The robot also never scored. The approach term’s gradient must propagate through body deformation to center-of-mass motion, and with an H=32H=32 window (0.1 s) that motion has not yet occurred. Larger batches can do nothing to fix this failure. To investigate sensitivity to window length further, we performed a systematic sweep of HH at fixed batch size—see our response to Reviewer 92Nx W2 (Appx. A.2.2, Table 4).


Response to Reviewer CZQZ (3/5)


W2. “The introduction of SGD-layer in Section 3.2 is not clear, when the following multi-agent experiments in Section 3.3 just use Adam anyway. The performance also appears to be weaker than Adam in the single-agent setting. Section 2.8 also mentions using a cosine learning rate scheduler, which is not ablated against.”

We appreciate the reviewer pointing out that the introduction of SGD-layer was unclear. We apologize for this and for any confusion around its motivation, usage, or relative performance. To clarify, SGD-layer is not a proposed method or a contribution in its own right. The gradient analysis (Sect. 3.2) found that gradient energy was distributed across layers out of proportion to their parameter counts (Fig. 14). While Adam renders the imbalance irrelevant through per-coordinate normalization, an optimizer with a single global learning rate is bound to it. SGD-layer, which rescales each tensor’s update to match its share of parameters and changes nothing else, was introduced simply to evaluate whether this imbalance explains the poor performance of plain SGD and SGD-norm. While SGD-layer did substantially improve over the other two SGD variants, the reviewer is correct in stating that it was weaker than Adam in the single-agent setting. We therefore used only Adam in the multi-agent experiments.

To clarify these points we made the following changes:

  1. Section 2.8 now describes SGD-layer as a diagnostic and refers forward to its motivation:

    SGD-layer, a diagnostic variant (Sect. 3.2) …

  2. Section 2.8 now states why multi-agent training used Adam:

    Multi-agent policies used Adam, the best-performing optimizer in the single-agent comparison (Sect. 3.1; Figs. 4, 8)

  3. Section 3.2 now describes SGD-layer as a diagnostic:

    Under Adam the skew grew much further—the input layer’s share reached ≈ 65% and the output layer’s ≈ 10%—but per-coordinate normalization made its updates indifferent to the raw allocation. This suggested that the allocation, which SGD and SGD-norm inherit through a single learning rate, may explain their poor performance. To test this hypothesis, we introduced SGD-layer, which, within each tensor, steps in the same direction as SGD-norm, rescaled to the tensor’s parameter share, leaving the total step norm unchanged …

  4. Section 3.2 now states the performance gap of Adam and SGD-layer explicitly:

    This reallocation alone raised the final scoring rate from 0.57±0.07 to 0.81±0.04, within standard error of Adam’s final scoring rate (0.84 ± 0.05) under the cosine schedule (Appx. A.2.2); however, SGD-layer still underperformed Adam in reliability, with 11 out of 20 seeds scoring above 80% compared to Adam’s 16.

Regarding the learning rate scheduler, we realize the language in our submission may have been ambiguous—suggesting that the scheduler was tuned in one setting and then used throughout. In reality, the hyperparameters of the learning rate schedule were fixed, and each optimizer then swept for an initial suitable learning rate. This ensured each optimizer was in a well-tuned setting given the schedule. We have updated Section 2.8 to remove any ambiguity:

Learning rates followed a cosine schedule with 5% linear warmup and decay to 1% of the initial value. Initial values for each optimizer were selected by sweep in the single-agent setting and reused elsewhere.

In response to the reviewer's observation that the scheduler was not ablated, we explicitly tested its contribution to performance under each optimizer. We reported results in Appendix A.2.2 (Table 5). In summary, final scoring rates with a constant learning rate were 0.87 (Adam), 0.68 (SGD-layer), 0.41 (SGD-norm), and 0.08 (SGD), versus 0.84, 0.81, 0.57, and 0.53 with the schedule. This passage was added to Section 3.2 with forward reference to full details in Appendix:

To test whether learning rate annealing (Sect. 2.8) mitigated the effect of these gradient properties, we replaced it with a constant learning rate (Appx. A.2.2). Adam was unaffected, presumably owing to its per-coordinate adaptation, while every SGD variant declined and plain SGD collapsed, indicating that with a single global learning rate the annealed schedule played an important role in mediating gradient magnitude and variance.

We also added the qualifier “under the cosine schedule” to the SGD-layer vs. Adam comparison in Section 3.2 (item 4 above).


Reviewer_PqbZOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work shows that direct e2e differentiation through physics and neural controllers, following the approach established by systems such as DiffTaichi, can efficiently train soft robots with thousands of actuated springs to perform contact-rich locomotion and manipulation, such as ball pursuit, dribbling, kicking, and multi-agent cooperation (which is not surprising given recent progresses in differentiable physics). Using short-horizon backpropagation without a learned critic, single-agent policies learn to score in approximately 100 gradient updates, requiring less than a minute on a single GPU. The paper also shows that large batches recover useful learning signals from poorly aligned contact gradients, while proper scaling of updates across network layers substantially improves optimization stability.

Critical strengths

In general this is an interesting paper showing that the capability of a simple differentiable simulator is actually stronger than I thought, especially with frequent contacts. Everything works almost out of the box, with some optimization tricks the authros adopted which will inpsire future research.

I believe the application complexity and optimization tricks in this work opens up future possibilities in more complex differentiable simulations.

Critical weaknesses

The work is a natural extension and solid step over existing results in differentiable soft body simulation yet is not a groundbreaking work. It would be nicer if the author shows more use cases of the system. Moreocer, the current results depends on a manually designed loss function which may harm the generalizability of the system.

Questions for authors

None.

1 comment

Response to Reviewer PqbZ (1/1)


We thank the reviewer for their time and careful reading of our paper. We appreciate that the reviewer found our work interesting and a potential source of inspiration for future research.

Below we address the stated weaknesses (W1 −- W3), one by one. We also reference, where applicable, how our manuscript was modified. The updated manuscript has been uploaded to OpenReview with additions marked in red text. We believe all changes have significantly improved the paper, and we thank the reviewer for prompting them.


W1. “The work is a natural extension and solid step over existing results in differentiable soft body simulation yet is not a groundbreaking work.”

We appreciate the reviewer noting that our work advances differentiable soft body simulation. We also agree that our work builds on many prior studies. To clarify exactly what our contributions are, we added the following summary to the end of the Introduction:

To summarize, the contributions of this work are as follows. First, we provide the first demonstration that bare analytic policy gradients—without a learned critic, stochastic smoothing, or zeroth-order estimates—are practical for closed-loop, contact-rich loco-manipulation in soft robots. Second, we characterize the gradients underlying this result, showing, most importantly, that per-window contact gradients are large and mutually misaligned, yet aggregate into a usable signal with a sufficient number of windows. In addition, the failure of SGD can be attributed to a layer-wise skew in gradient energy and thus largely corrected through per-layer rescaling. Finally, we demonstrate cooperative and competitive multi-agent play in the same environment, which represents—to the best of our knowledge—the first instance of multi-agent learning in a differentiable, contact-rich setting.


W2. “It would be nicer if the author shows more use cases of the system”

We agree the paper demonstrates a single environment. Within this environment, our original submission showed three use cases: single-agent loco-manipulation across three morphologies (Sect. 3.1), in which locomotion and contact-rich manipulation are learned and measured as distinct skills (Figs. 4, 5); cooperative play with a shared policy (Sect. 3.3); and adversarial play with a defender trained on a distinct objective (Sects. 2.6, 3.3).

The revised text now states additional factors that can facilitate future use cases:

  1. The simulator is not restricted to the cubic voxel robots studied here. Addition to Section 2.2:

    The simulator (Sect. 2.4) accepts any numerically stable mass–spring configuration; cubic voxels served only as a convenience in constructing the bodies studied here.

  2. The task environment is not restricted to a circular field. Addition to Section 2.1:

    The field was not a simulated object: robots and the ball moved on an unbounded plane (Sect. 2.4), and the field entered only through the objective (Sect. 2.6), observations (Sect. 2.3), and the episode distribution (Sect. 2.5). A circle was chosen so that the boundary reduced to a single radial distance; alternative field geometries may be realized through corresponding changes to those components.


W3. “the current results depends on a manually designed loss function which may harm the generalizability of the system”

It is true that the loss functions used in our paper were manually designed. We note, however, that the paper already trains with two objectives, attacker and defender (Sects. 2.6, 3.3), and that the same attacker objective generalized to three distinct morphologies (Sect. 3.1) and to a jointly trained cooperating pair, both in isolation and against a defender (Sect. 3.3). We are excited to test new objectives and continue investigating the range of behaviors that can be realized in our system.


We thank the reviewer for their consideration of our response and gladly welcome any further questions or comments.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.