TRACE-RL: Trajectory-Guided Reward Adaptation and Collaborative Evolution for Robotic Skill Learning

Learning robotic skills through reinforcement learning offers a promising path toward embodied intelligence, yet designing rewards that continually and reliably guide high-dimensional robotic control remains difficult. Large language models can reduce this burden by generating executable dense reward programs. Existing LLM-based methods, however, generally revise reward guidance at outer-loop boundaries while keeping each candidate reward fixed during the corresponding policy-training cycle. This design becomes restrictive as an improving policy reaches new contact configurations and near-success behaviors that a fixed reward may no longer guide adequately. We introduce **T**rajectory-Guided **R**eward **A**daptation and **C**ollaborative **E**volution for **R**obotic Skill **L**earning (**TRACE-RL**), a framework that maintains effective reward guidance as the policy-induced trajectory distribution evolves. In the inner loop, TRACE-RL combines an LLM-generated task-prior reward with a trajectory-calibrated reward learned using an executable trajectory-scoring function, and uses an evaluation function to support reliability-aware fusion. In the outer loop, the LLM jointly refines the three executable reward, score, and evaluation functions using structured policy-training feedback, while validated reward-model and policy states are retained across successive outer iterations. Across eighteen robotic tasks spanning locomotion, whole-body control, and dexterous manipulation, TRACE-RL achieves the best or tied-best mean maximum training score on every task, with ablation studies supporting the effectiveness of its key components.