acceptodds
Under review as a conference paper at ICLR 2027

Dual-Agent Co-Design of Terrain Curricula and Reward Functions for Mixed-Terrain Quadrupedal Locomotion

Abstract

Robust quadrupedal locomotion over mixed terrain requires the robot to preserve stable motion as contact, posture, and momentum demands change between ter- rain segments. These transitions couple two training decisions: the terrain cur- ricula used for policy training and the reward functions used to guide policy op- timization. Optimizing them separately can misalign reward incentives with the behaviors required by the training terrains. We introduce a policy-mediated dual- agent framework in which a Terrain Curriculum Agent and a Reward Hypothe- sis Agent co-design executable mixed-terrain training arenas and arena-informed reward programs. Transition-frontier feedback computed from rollouts of the in- duced policy identifies unresolved transitions and links them to local geometry and robot–terrain interactions, enabling both agents to address the same locomo- tion bottleneck. Candidate designs are grounded by proximal policy optimization in physics simulation and selected through hierarchical behavioral validation that combines reward screening with gait-constrained cross-arena evaluation. Exper- iments on an independently constructed mixed-terrain test set show gains over terrain-evolution, reward-search, hand-designed, and task-curriculum baselines in full-course success and traversal progress. The advantage is larger on closely spaced terrain compositions than on isolated components, and ablations show that terrain evolution and explicit designer feedback complement reward optimization, particularly on difficult courses. Additional evaluations on three public terrain suites support the transfer of the learned policies across terrain generators.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Mixed-terrain quadruped locomotion, Terrain–Reward Co-Design, Large Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_anxnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a dual-agent framework for designing terrain curricula and reward functions for mixed-terrain quadrupedal locomotion. One LLM agent generates terrain programs, while another proposes rewards conditioned on the terrain and feedback from the current policy. PPO training, within-arena reward screening, and gait-constrained cross-arena selection determine which policy is retained for the next round. Transition-frontier feedback summarizes the policy's competence boundary, frequent failure intervals, and associated statistics. The framework is evaluated on simulated Go1 locomotion, including a mixed-terrain benchmark, component-to-composition diagnostics, and fixed-policy tests on three public terrain suites.

Critical strengths
  1. I get the motivation. Being able to cross individual obstacles does not necessarily mean a policy can handle them in sequence. Adjusting both the terrain and reward around the same failures is a reasonable direction, and the component-to-composition experiment makes the motivation more concrete.

  2. The reported gains look good. The main benchmark reaches 82.5% success versus 62.5% for Eureka, and the fixed-policy tests on public terrain suites provide some evidence beyond the authors' own benchmark. The appendix also gives substantial detail about the design loop.

Critical weaknesses
  1. I am still not sure why this needs two agents. What happens if one GPT generates both terrain and reward with the same feedback and budget? The pipeline currently feels more like task-specific LLM-guided search than a clearly established methodological advance. There may be a useful decomposition here, but the comparison needed to establish it is missing.

  2. The ablations help, but they do not quite isolate the mechanisms emphasized in the paper. Removing all explicit feedback does not tell me whether transition-frontier diagnosis is better than ordinary progress feedback. Reward screening and cross-arena selection remain active, so their individual contributions are also unclear. With search, gait constraints, inheritance, and selection all affecting training, I find it hard to tell where the gain comes from.

Questions for authors
  1. Why are two agents needed? How does a single agent that jointly generates terrain and reward perform under the same feedback, candidate count, and training budget?

  2. Does transition-frontier feedback outperform ordinary success-rate or progress feedback? The current feedback ablation removes several kinds of information together.

  3. What happens if policies are selected only on their own arena rather than across arenas? Can the contribution of within-arena screening also be separated from the rest of the search procedure?

  4. How is the training budget kept comparable when reward candidates are reduced from eight to one in the no-reward-optimization ablation? Are additional trained batches and failed runs included in the reported budget?

Reviewer_6yiWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work proposes an iterative framework which jointly generates terrain curricula and reward functions for quadrupedal locomotion on rough terrain. Two LLM agents generate terrain curricula and candidate rewards using feedback from the current policy's performance and failure locations. For each newly generated terrain, policies are briefly trained with PPO under different candidate rewards. The best-performing policy that meets minimum foot-contact requirements is selected for further training. The resulting policies are then evaluated across all child arenas in the current generation, and the best eligible policy is retained for the next round.

Simulation experiments with a Go1 robot report improved rough-terrain traversal over human-designed training, Eureka, Eurekaverse, and CurricuLLM. Additional experiments test traversal of isolated obstacles and obstacle compositions, performance in publicly available terrain suites, and the contributions of individual framework components through ablations.

Critical strengths
  1. The method reaches 82.5 +- 0.7% success rate compared with the 62.5 +- 5.5% success rate for the strongest baseline in Table1. The improvement over the baselines also remains on higher difficulty levels. The nominally matched PPO training budgets and the three training seeds strengthen the evidence that the full method can improve performance in this specific setting.

  2. The explicit distinction between individual component competence and successful composition is useful. Evaluating isolated obstacles alongside ordered combinations addresses a meaningful locomotion question: being able to traverse individual obstacles does not imply being able to negotiate their transitions.

Critical weaknesses
  1. The contribution beyond existing methods is insufficiently isolated. Eurekaverse evolves terrains for quadrupedal parkour and already evaluates candidate policies across multiple generated environments before selecting a policy for subsequent training. Cross-environment selection alone therefore cannot distinguish the proposed method from Eurekaverse. The potentially novel contribution lies in using failure-location and difficulty feedback to coordinate terrain and reward generation, together with the specific reward-screening and gait-constrained selection procedure. Although the authors cite relevant prior work, they should position these contributions more precisely within the existing literature and isolate their benefits through controlled ablations. The current ablations establish that removing major components reduces performance. However, they do not isolate the benefit of the proposed feedback and coordination mechanism. Removing all explicit feedback does not ablate the benefit of ordinary performance feedback from that of localized failure information. Likewise, freezing terrain or reward generation does not isolate the benefit of coordinating their updates through the proposed feedback mechanism. The reward-freezing ablation also reduces the number of candidates from eight to one, without sufficiently clear accounting of the resulting budget allocation.

  2. The evaluation protocol leaves open questions about model selection and the scope of the reported gains. In Section 4.1 the authors state that for each method and seed, they report the evaluated policy with the highest overall success rate on the benchmark. This appears to use the test benchmark for model selection. Such selection can inflate reported performance and affect methods differently if different numbers of policies or checkpoints are considered. The authors should clarify the selection procedure and report results for policies selected without consulting the test benchmark, either using each method’s final output or a separate validation set. Furthermore, the strongest gains occur on the mixed-terrain benchmark, which was designed by the authors. Results on publicly available suites are less consistently favorable, including lower overall performance than Eurekaverse on Robot Parkour Ground and a near tie with Eureka at high difficulty on ETH confined terrain. The conclusions should therefore distinguish improvements in the targeted mixed-terrain setting from broader improvements in rough-terrain locomotion.

Questions for authors
  1. Evaluation clarification: Which policies or checkpoints were eligible for the benchmark selection described in Section 4.1, and how many were evaluated per method and seed? Can you report the same metrics for the final policy returned by each method, without using benchmark performance to select the reported policy?

  2. Budget clarification: In the fixed-reward ablation, how is the budget redistributed when the candidate count decreases from eight to one? Were any reward-candidate batches retried because no policy passed the gait checks? If so, were the PPO iterations spent on unsuccessful batches included in the reported 160,000-iteration budget?

  3. Can you provide a comparison that retains both terrain and reward search and the same selection procedure, but replaces localized feedback with aggregate performance feedback?

  4. Novelty clarification: What precisely distinguishes the proposed cross-arena selection from Eurekaverse’s existing cross-environment evaluation?

1 comment

Thank you for the additional experiments and clarifications. My concerns about final policy evaluation and budget fairness are addressed.

However, I am still concerned that the methodological contribution beyond Eurekaverse is incremental. The results support the full framework. However, the clear and big gains of the proposed method are achieved on the authors' mixed-terrain benchmark. On the public benchmarks the results are still competitive but show smaller and less consistent improvements.

In my opinion, clear evidence that specifically the added algorithmic changes lead to benefits is missing. The effects of the particular coordination and selection choices are insufficiently isolated. Therefore, it is hard to asses if the extensions lead to a substantial methodological advance.

I maintain my weak-rejection recommendation.

Reviewer_JTbEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The method proposes automated new terrain and reward program generation with LLMS. Experiments compare this to previous curriculum learning based methods, and point out improvement over human expert design, and state of the art LLM reward and curriculum generation. The terrain agent regenerates complete programs from parent programs and the feedback, and reward agent uses the new arena and historical feedback to propose candidate rewards. The policy optimization loop then trains candidates, selects a transferable policy and then use the rollouts to guide the generation of the terrains.

Critical strengths
  1. Coordination between the agents is coherent and the agents access the latest evidence and results from the policy.

  2. The experiments demonstrate promising results across various suites in simulation extensively.

  3. The paper is written clearly, and it's easy to follow.

  4. The paper's appendices give thresholds, PPO settings, complete selected reward modules and prompt templates, including explicit limitations of feedback and gait checks.

Critical weaknesses
  1. Since the practical value of locomotion sim training depends on deployment on the physical robots, the lack of real-world experiments leaves unclear whether the reported gains survive the perception errors, imperfect dynamics, and hardware constraints encountered outside simulation. That is why the original Isaac environments are build for, to close this gap for locomotion.

  2. It seems that the computational overhead relative to Eureka and other baselines is unclear: the paper reports nominal PPO budgets but not LLM token usage, curriculum-generation time, or total wall-clock time to reach comparable performance.

  3. The experiments show the complete system helps without fully isolating which specific design choices cause the improvement. The phrase was too broad without that qualification. For example the ''without explicit feedback'' ablation removes several things together so it's performance drop supports the usefulness of that combined feedback package, but cannot establish whether the gain comes specifically from transition-frontier localization, reward diagnostics, or simply giving the LLM more performance information.

  4. Even though the design loop is automated, it relies on task-specific terrain constraints, prompting, and gait rules so the human effort needed to configure these components or transfer the framework to a new task hasn't been evaluated. And it's not clear how much effort more/less than Eureka is needed for this purpose.

Questions for authors

See and address weaknesses please.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.