- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper shows that analytic policy gradients can train closed-loop, contact-rich loco-manipulation in high-dimensional simulated soft robots, and examines the roles of batch aggregation and layer-wise update scaling. A deterministic neural controller actuates individual springs in a differentiable mass–spring simulator. Training backpropagates through 500-step simulation windows, with gradients truncated at window boundaries and no learned critic or zeroth-order gradient estimates. The objective combines approach, ball transport, boundary, and actuation costs. The main experiment compares four optimizers on a quadruped over 20 random seeds. Additional experiments investigate batch size and contact-gradient statistics, extend the approach to biped and hexapod morphologies with a separate policy trained for each, and explore cooperative and competitive settings involving quadruped robots.
Critical strengths
The work is well motivated by the challenges of using analytic gradients for contact-rich control, particularly in soft robots with high-dimensional action spaces. The experiments provide convincing evidence that direct, windowed analytic policy gradients can successfully train controllers in the studied setting. The main optimizer comparison uses 20 random seeds, while separately trained controllers across three morphologies and extensions to multi-robot tasks broaden the demonstrated applicability. It also provides informative gradient analysis and optimizer comparison. Limitation is also discusse.
Critical weaknesses
- Limited comparison evidence with previous methods (e.g., SHAC) in same environmental settings.
- Unclear characterization of several training factors, e.g., sensitivity to window length, the contribution of batch aggregation versus increased simulation budget, and robustness to contact stiffness and damping.
Questions for authors
To support the comparison with SHAC’s update requirements, can you provide a same-environment comparison reporting overall performance? If this is not feasible, please clarify the limitations of the cross-method comparison.
How sensitive is performance to the backpropagation window length? Since gradients are truncated without a terminal critic, a window-length ablation reporting performance and computational cost would clarify the dependence on the chosen 500-step horizon.
Does the advantage of larger batches persist under a matched total simulation-transition budget, with smaller batches receiving more updates?
3 comments
Response to Reviewer 92Nx (2/3)
W2. “Unclear characterization of several training factors, e.g., sensitivity to window length, the contribution of batch aggregation versus increased simulation budget, and robustness to contact stiffness and damping.”
To remedy this we conducted the following experiments:
-
We tested sensitivity to window length and reported results in an addition to Appendix A.2.2 (pasted below). To summarize, the final scoring rates were (), (), (), and ().
We tested additional window lengths (Sect. 2.5), alongside the baseline , training the quadruped with Adam and scaling the number of gradient steps by to match the baseline’s total simulation steps (Tab. 4). At , policies collapsed to inaction and never contacted the ball. These windows were too short—less than one gait cycle of the controller—to reduce the approach loss through locomotion toward the ball, so the effort penalty dominated and actuation collapsed. At ( gait cycles), locomotion was learned comparably to the baseline but manipulation was poor (scoring rate vs. ). Total simulated steps were matched by construction, and shorter windows required less wall clock per update. However, the fixed overhead of each forward and backward pass accumulated over more updates, so total training time rose modestly as windows shortened ( to s).
We also added to Section 3.1 and the Discussion to clarify sensitivity to window length:
Performance depended strongly on window length: when the gradient horizon was shorter than about one gait cycle the policy collapsed to inaction, and with intermediate windows (250 steps) policies reached the ball as quickly as the baseline ( vs. s) but scored far less often ( vs. ; Appx. A.2.2).
Window length was a prerequisite: below roughly one gait cycle no learning occurred (Appx. A.2.2).
-
We tested the advantage of large batches when the total simulation budget was constant and reported results in Section 3.2:
The fixed-update sweep (Fig. 5), however, gave smaller batches fewer total simulation steps and a larger learning rate per sample. With a matched simulation budget and the learning rate scaled to batch size, every reached final scoring rates of –, exceeding the -update baseline (Appx. A.2.2). A reliable manipulation signal thus could be obtained by aggregating a sufficient number of per-window gradients, whether within a large batch or, consistent with the linear scaling rule (Goyal et al., 2017), across many smaller updates.
And Appendix A.2.2:
We retrained for , , and updates, matching the baseline’s () simulation budget, first at the original learning rate (Appx. A.1.3) and then with the learning rate scaled down by batch size (Tab. 3). Matching the simulation budget alone recovered , which exceeded the baseline’s scoring rate. At , increasing the number of updates alone was insufficient, but scaling down the learning rate brought both () above the baseline ( and compared to ), with no seed below , compared with a worst seed of for the baseline. Smaller batches—paired with a greater number of small steps—thus produced more stable training and erased the batch size advantage reported in Figure 5. However, consistent with Figure 5, every configuration learned locomotion, with time to contact ranging from to s. After scaling learning rates, bettered the baseline's s, while the unscaled-learning-rate configurations lagged by about s. Because smaller batch sizes required more gradient steps they were - slower in wall clock over the same simulation budget.
In light of these results, we updated the manuscript (Abstract, Introduction, Sect. 3.2, and Discussion) to reflect that batch aggregation is not the operative factor per se. What the manipulation signal requires is sufficient aggregation of per-window gradients, which can come from larger batches or from additional updates.
We also appreciate the reviewer’s interest in sensitivity to contact stiffness and damping. The original Discussion named the single, penalty-based contact model as a limitation and stiffness as a direction for future work. We have now added damping to that sentence. We believe sweeping these parameters would be most informative once the simulator is calibrated to a physical system for real-world transfer, which is left for future work. As additional characterization of training factors, we ablated the cosine learning-rate schedule—results are reported in Appx. A.2.2, Table 5.
Response to Reviewer 92Nx (3/3)
Q1. “To support the comparison with SHAC’s update requirements, can you provide a same-environment comparison reporting overall performance? If this is not feasible, please clarify the limitations of the cross-method comparison.”
Please see our response to W1.
Q2. “How sensitive is performance to the backpropagation window length? Since gradients are truncated without a terminal critic, a window-length ablation reporting performance and computational cost would clarify the dependence on the chosen 500-step horizon.”
Please see our response to W2.
Q3. “Does the advantage of larger batches persist under a matched total simulation-transition budget, with smaller batches receiving more updates?”
Please see our response to W2.
We thank the reviewer for their consideration of our response and gladly welcome any further questions or comments.
Response to Reviewer 92Nx (1/3)
We thank the reviewer for their time and thoughtful assessment. We appreciate that the reviewer found the experiments to provide “convincing evidence that direct, windowed analytic policy gradients can successfully train controllers in the studied setting,” which is the paper’s central claim. We also appreciate that they found the morphologies and multi-robot tasks to broaden the demonstrated applicability and the gradient analysis and optimizer comparison informative.
Below we address the stated weaknesses (W1 − W2) and questions (Q1 − Q3), one by one, also referencing how our manuscript was modified in response. The updated manuscript has been uploaded to OpenReview with additions marked in red text. We believe all changes have significantly improved the paper. In particular, investigating the reported batch advantage under equal simulation budget (W2) has produced new results that we believe provide a broader, more informative perspective on the sufficiency of analytic policy gradients in this setting. We are grateful to the reviewer for prompting this study.
W1. “Limited comparison evidence with previous methods (e.g., SHAC) in same environmental settings.”
We agree this is a limitation of our work. Our claims are limited to the sufficiency of analytic policy gradients, and we do not claim superiority over alternative first-order methods such as SHAC. One sentence in the original Introduction did draw a quantitative contrast with SHAC. But because the task environments fundamentally differ, and because a “gradient step” is not a shared unit—a SHAC policy update simulates shorter rollouts, and its many critic updates reuse rollouts—this comparison may be uninformative or misleading, and we have removed it:
We consider a same-environment SHAC comparison beyond the scope of this paper, and we appreciate the reviewer acknowledging its possible infeasibility (Q1). To the best of our knowledge, SHAC has not been demonstrated on soft body loco-manipulation, so a credible baseline would require adapting and tuning it for this setting, and a poorly tuned baseline would be uninformative. To clarify this, as well as the limitations of a cross-method comparison with SHAC, we added this to the Discussion: