acceptodds
Under review as a conference paper at ICLR 2027

TRACE-RL: Trajectory-Guided Reward Adaptation and Collaborative Evolution for Robotic Skill Learning

Abstract

Learning robotic skills through reinforcement learning offers a promising path toward embodied intelligence, yet designing rewards that continually and reliably guide high-dimensional robotic control remains difficult. Large language models can reduce this burden by generating executable dense reward programs. Existing LLM-based methods, however, generally revise reward guidance at outer-loop boundaries while keeping each candidate reward fixed during the corresponding policy-training cycle. This design becomes restrictive as an improving policy reaches new contact configurations and near-success behaviors that a fixed reward may no longer guide adequately. We introduce **T**rajectory-Guided **R**eward **A**daptation and **C**ollaborative **E**volution for **R**obotic Skill **L**earning (**TRACE-RL**), a framework that maintains effective reward guidance as the policy-induced trajectory distribution evolves. In the inner loop, TRACE-RL combines an LLM-generated task-prior reward with a trajectory-calibrated reward learned using an executable trajectory-scoring function, and uses an evaluation function to support reliability-aware fusion. In the outer loop, the LLM jointly refines the three executable reward, score, and evaluation functions using structured policy-training feedback, while validated reward-model and policy states are retained across successive outer iterations. Across eighteen robotic tasks spanning locomotion, whole-body control, and dexterous manipulation, TRACE-RL achieves the best or tied-best mean maximum training score on every task, with ablation studies supporting the effectiveness of its key components.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Skill Acquisition, Dexterous Manipulation, Automatic Reward Design

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_eVAPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TRACE-RL extends LLM-based reward design by adapting the reward during policy training. In each outer iteration, an LLM writes three kinds of programs: dense reward functions (the task prior R_p), trajectory-scoring functions S and evaluation functions E. In the inner loop, PPO training is split into stages by a plateau detector. At each stage boundary, fresh on-policy segments are labeled pairwise by S, and a reward model R_t is trained on these labels. R_t is then fused with R_p using a weight set by how well each reward's ranking agrees with the environment task score, and a small task reward R_s is added. Across outer iterations, the LLM revises all three programs, and the policy and reward model of the global-best candidate are inherited by the next iteration. On 18 Isaac Gym / Bi-DexHands tasks (GPT-5.6-Terra), TRACE-RL obtains the best or tied-best maximum training score (MTS) on every task, with large gains on contact-rich manipulation. Ablations on 3 tasks support each component.

Critical strengths
  1. A sensible idea. Recalibrating a learned reward on current-policy trajectories, using pairwise labels from an LLM-written scorer, addresses a real limitation of methods that keep each candidate reward fixed during training. The single-iteration result (Fig. 4) suggests that the inner loop itself contributes a large part of the gain.

  2. A broad benchmark with large gains on hard tasks, e.g., DoorOpenInward, PushBlock, and extensive appendix ablations.

Critical weaknesses
  1. The comparison is not fair in compute or information, so it is unclear where the gains come from. (a) Only PPO epochs per candidate are matched. Because of policy inheritance, the best lineage can accumulate up to 5×3000 epochs, whereas each Eureka candidate is trained from scratch for 3000 epochs. Removing inheritance costs 10–16%. (b) The environment interaction spent on calibration (about 4M transitions per stage), the reward-model training and the extra LLM calls are all excluded from the budget. (c) The evaluation metric (the task score) is used heavily during training: for plateau detection, screening S and E, the fusion weight, candidate selection and best-checkpoint retention, while Eureka uses it only for outer-loop selection. (d) MTS is the running maximum of a noisy training score, not an evaluation of the returned policy, and it favors methods with more candidates and longer lineages. (e) The re-run baselines look weak. For example, Eureka scores 0.000 on CatchOver2, CatchAbreast and CatchUnderarm, contrary to the original Eureka paper.

  2. The method is complex and hard to reproduce, and its dependence on the LLM is under-tested. It relies on many hand-set constants (fusion gain, evidence gates, EMA, R_s cap, stage budgets), with no sensitivity analysis. The ablations report no standard deviations. The LLM is proprietary, and no code is released. The backbone-LLM study (Table 10) tests only GPT-family models, only on 3 locomotion tasks under a reduced budget, and only for TRACE-RL, not for the baselines. It is therefore unknown whether the method works with open-weight or other-vendor LLMs, and whether its advantage persists when all methods share the same backbone.

Questions for authors
  1. What is the total number of PPO epochs on TRACE-RL's best lineage compared with the baselines? Can you compare against baselines that also inherit policies, or run TRACE-RL without inheritance under the same total budget?
  2. Please report total environment steps, LLM calls/tokens and wall-clock time for all methods.
  3. How does TRACE-RL perform if the task score is used only for candidate selection (no task-score-based fusion or screening, and no R_s)?
  4. Why do the re-run Eureka results differ from the original paper? Did all LLM-based methods use the same backbone and the same number of samples?
  5. Can you compare all LLM-based methods under a shared backbone, including at least one open-weight model (e.g., Qwen or DeepSeek) and one model from another vendor (e.g., Claude or Gemini), on some manipulation tasks, with standard deviations?
Reviewer_ENaFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The LLM-generated reward literature mostly focuses on running the LLM in the outer loop, which limits its efficiency and efficacy, since the proposed reward function generally stays detached from the inner-loop policy updates and performance. In this work, the authors propose a hybrid approach in which they combine an LLM-generated reward with a learned trajectory-calibrated reward in the inner loop, while the LLM edits reward candidates in the outer loop. The LLM in the outer loop proposes a reward function, an evaluation function, and a scoring function. In the inner loop, a reward model is trained to predict the reward of each state-action pair coming from the current policy. The learned reward and the LLM reward are fused based on their alignment with the environment score and the evaluation function (proposed by the LLM in the outer loop). In addition to the LLM and learned rewards, an environment reward is also used. The learned reward model and policy parameters are kept after each outer-loop iteration rather than being trained from scratch. The method is evaluated on 18 tasks, showing significant improvements over the baselines.

Critical strengths
  • It is very nice that the learned reward and policy continue from their previous weights in each new outer-loop iteration.
  • The experiments on inner-loop adaptation clearly show how the inner-loop learned reward and adaptation help policy learning.
  • The performance on all 18 tasks is strong, and the experiments are extensive.
Critical weaknesses
  • “fϕ aggregates its transition-level outputs”: This is very unclear in the main text and is explained only in the Appendix. Please explain it briefly in the main text and refer to the Appendix, as is done elsewhere in the paper.
  • “within each reward group, we estimate the reliability of both rewards”: Although this is explained in the Appendix, neither Q nor A is formally defined or well explained. Please define and explain them more clearly, and properly refer to the Appendix in the main text.
  • Using an environment reward is problematic, since the whole idea is to eliminate such privileged reward functions. Why is such an engineered, privileged reward used? It is also very unfair and makes the results questionable if such privileged information is not provided to the baselines’ policies.
  • The budget is defined in terms of PPO epochs. Since the proposed approach also trains a reward model and queries a huge LLM, can you provide results under a matched compute budget?
  • The proposed method uses GPT-5.6 as the LLM. Are the baselines evaluated using the same LLM?
  • The maximum training score (MTS) is used as the main metric, but it is the maximum over training and is therefore biased. Please report the test scores achieved after training ends for all methods, which is a fairer metric.
Questions for authors

See Weaknesses.

Reviewer_P3pGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on reward learning / designing for robotic control and introduces TRACE-RL, aiming to maintain and adapt reward guidance as policy behavior evolves. Rather than improving the reward signals simply by an outer loop, this paper proposes to also incorporate inner loop updating to recalibrate the reward signals as the policy improving. It shows better or tied performance on 18 robotic tasks, with ablations validating key components.

Critical strengths
  • This paper is well-motivated. The problem is meaningful and sensible. Recalibrating and revising a reward function on on-policy trajectories is important to make sure the policy is improving continually in the right direction. Not only in robot control learning, but also in other domains.
  • This paper conducts experiments on 18 tasks across locomotion and bimanual dexterous manipulation, as well as thorough ablations, demonstrating the effectiveness of the proposed approach.
Critical weaknesses
  • It may be a little confusing and hard to understand this paper. Some of the terms and expressions used in the text are not very intuitive and are difficult to understand.
  • The definitions of reward function, score function and evaluation are not explained well in this paper, which makes the method design confusing from my perspective. Don't "reward," "score," and "evaluation" all refer to the same thing?
  • The paper appears to lack a comparison of training costs and environmental interaction costs across different methods. Such comparisons would help readers better understand the pros and cons of each approach.
Questions for authors
  • Across line 74-76, the authors claim: "they adapt reward guidance by selecting or revising reward programs rather than recalibrating rewards from newly collected trajectories during policy training". Isn't revising reward programs based on newly collected trajectories essentially a form of recalibration? What is the difference between revising and recalibrating.
  • Could the authors explain their definitions of reward function, score function and evaluation function more clearly? From my perspective, this way of categorizing things is confusing, as the terms "reward," "score," and "evaluation"—as used in the existing literature—all refer to the same thing.
  • In the Inner-loop of the proposed method, the RM is trained using the trajectories scored by the score function. In this case, why do we need to train an RM? What are the benefits compared to using a score function directly?

Comments on the paper

1 comment
Reviewer_ENaFOpenReview ↗

Response to Rebuttal

Thanks to the authors for the detailed response. It is now clear how fϕf_{\phi} aggregates transition-level outputs (W1), how Q and A are defined (W2), and that the baselines use the same LLM backbone (W5). The environment-reward ablation (W3) is meaningful, but I would like to see it on all tasks rather than two. Without the added RsR_s, TRACE-RL and ROSKA perform similarly on Ant given the standard deviations (12.707 ± 0.211 vs. 12.192 ± 0.607). Regarding compute-matched results (W4), TRACE-RL uses substantially more API calls than every baseline (90 vs. 30–65) and more tokens (11–14% more than PoRSE, and roughly 2–5× more than Eureka and ROSKA). I would really prefer to see GPU-hours rather than wall-clock time, API calls, or token usage, since the reward-model training and additional calibration rollouts are not reflected in those. Regarding test scores (W6), I would like to see results for all baselines and all tasks, rather than only the proposed method on three tasks. Given these points, I prefer to keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.