- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
TRACE-RL extends LLM-based reward design by adapting the reward during policy training. In each outer iteration, an LLM writes three kinds of programs: dense reward functions (the task prior R_p), trajectory-scoring functions S and evaluation functions E. In the inner loop, PPO training is split into stages by a plateau detector. At each stage boundary, fresh on-policy segments are labeled pairwise by S, and a reward model R_t is trained on these labels. R_t is then fused with R_p using a weight set by how well each reward's ranking agrees with the environment task score, and a small task reward R_s is added. Across outer iterations, the LLM revises all three programs, and the policy and reward model of the global-best candidate are inherited by the next iteration. On 18 Isaac Gym / Bi-DexHands tasks (GPT-5.6-Terra), TRACE-RL obtains the best or tied-best maximum training score (MTS) on every task, with large gains on contact-rich manipulation. Ablations on 3 tasks support each component.
Critical strengths
-
A sensible idea. Recalibrating a learned reward on current-policy trajectories, using pairwise labels from an LLM-written scorer, addresses a real limitation of methods that keep each candidate reward fixed during training. The single-iteration result (Fig. 4) suggests that the inner loop itself contributes a large part of the gain.
-
A broad benchmark with large gains on hard tasks, e.g., DoorOpenInward, PushBlock, and extensive appendix ablations.
Critical weaknesses
-
The comparison is not fair in compute or information, so it is unclear where the gains come from. (a) Only PPO epochs per candidate are matched. Because of policy inheritance, the best lineage can accumulate up to 5×3000 epochs, whereas each Eureka candidate is trained from scratch for 3000 epochs. Removing inheritance costs 10–16%. (b) The environment interaction spent on calibration (about 4M transitions per stage), the reward-model training and the extra LLM calls are all excluded from the budget. (c) The evaluation metric (the task score) is used heavily during training: for plateau detection, screening S and E, the fusion weight, candidate selection and best-checkpoint retention, while Eureka uses it only for outer-loop selection. (d) MTS is the running maximum of a noisy training score, not an evaluation of the returned policy, and it favors methods with more candidates and longer lineages. (e) The re-run baselines look weak. For example, Eureka scores 0.000 on CatchOver2, CatchAbreast and CatchUnderarm, contrary to the original Eureka paper.
-
The method is complex and hard to reproduce, and its dependence on the LLM is under-tested. It relies on many hand-set constants (fusion gain, evidence gates, EMA, R_s cap, stage budgets), with no sensitivity analysis. The ablations report no standard deviations. The LLM is proprietary, and no code is released. The backbone-LLM study (Table 10) tests only GPT-family models, only on 3 locomotion tasks under a reduced budget, and only for TRACE-RL, not for the baselines. It is therefore unknown whether the method works with open-weight or other-vendor LLMs, and whether its advantage persists when all methods share the same backbone.
Questions for authors
- What is the total number of PPO epochs on TRACE-RL's best lineage compared with the baselines? Can you compare against baselines that also inherit policies, or run TRACE-RL without inheritance under the same total budget?
- Please report total environment steps, LLM calls/tokens and wall-clock time for all methods.
- How does TRACE-RL perform if the task score is used only for candidate selection (no task-score-based fusion or screening, and no R_s)?
- Why do the re-run Eureka results differ from the original paper? Did all LLM-based methods use the same backbone and the same number of samples?
- Can you compare all LLM-based methods under a shared backbone, including at least one open-weight model (e.g., Qwen or DeepSeek) and one model from another vendor (e.g., Claude or Gemini), on some manipulation tasks, with standard deviations?
Response to Rebuttal
Thanks to the authors for the detailed response. It is now clear how fϕ aggregates transition-level outputs (W1), how Q and A are defined (W2), and that the baselines use the same LLM backbone (W5). The environment-reward ablation (W3) is meaningful, but I would like to see it on all tasks rather than two. Without the added Rs, TRACE-RL and ROSKA perform similarly on Ant given the standard deviations (12.707 ± 0.211 vs. 12.192 ± 0.607). Regarding compute-matched results (W4), TRACE-RL uses substantially more API calls than every baseline (90 vs. 30–65) and more tokens (11–14% more than PoRSE, and roughly 2–5× more than Eureka and ROSKA). I would really prefer to see GPU-hours rather than wall-clock time, API calls, or token usage, since the reward-model training and additional calibration rollouts are not reflected in those. Regarding test scores (W6), I would like to see results for all baselines and all tasks, rather than only the proposed method on three tasks. Given these points, I prefer to keep my score.