acceptodds
Under review as a conference paper at ICLR 2027

ReLoop: Progress Supervision from LLM-Generated Reward Program Execution

Abstract

Large language models (LLMs) can produce executable reward programs from task descriptions, but the downstream reward interface typically exposes only each program's scalar return. Distances and contact predicates computed inside the program may reveal differences in task progress even when scalar rewards coincide. We introduce ReLoop, which records these runtime values and maps them to a progress potential and trace state. Potential changes supply a training signal, while a frontier bonus and memory of observed success use past traces to provide additional signals. ReLoop leaves the generated reward logic and policy observations unchanged and requires no further LLM calls. Across eight manipulation tasks, it improves mean final success by 17.1 percentage points over direct optimization of the same programs and by 7.0 points over a control matched for reward scale and success feedback. A control built from the scalar return favors the structured interface on average. On three tasks, the potential term retains the endpoint advantage over the matched control without the two memory terms. ReLoop thus turns intermediate values from a generated reward program into progress supervision during policy training.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, reward shaping, reward design, program instrumentation, robotic manipulation, language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_B1EiOpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
3
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies reward shaping for LLM-generated programs in robotic environments. The paper introduces Ground Process and online memory to shape the reward. The Grounder maps the reward signals into four sources: Reach, Interact, Goal, and Stable. The mapping provides a finer signal for training. Online memories record visit frequency and encourage exploration. The proposed method, ReLoop, on Meta World and ManiSkill 2. The evaluation results show that ReLoop improves policy learning and training efficiency. The ablation studies support the design of Online memories.

Critical strengths

In robotic learning, previous paradigms try to learn directly from the reward function using SAC or PPO. This paper introduces an in-training reward-function modification method to show there is room to construct a finer training signal and improve the LLM-generated reward function. The LLM-generated reward function may be imperfect and hard for humans to modify. The proposed method can make the LLM-generated reward function more training-efficient and help it converge to better performance.

Critical weaknesses

The largest problem with this paper is soundness. The proposed method does not compare with any known baselines. This makes the results hard to evaluate. The paper evaluates only some of the design choices. The justification for the design choice is unclear: the loss function introduces five coefficients, and each component should be evaluated carefully.

Questions for authors
  1. Your supplementary contains much more detail; it is better to put it in the Appendix and use hyperlinks for better understanding.
  2. In Table 2, are FrozenProram, Matched, and ReLoop all from your method and its ablations? What is the performance of other open-source baselines like Eureka [1]?
  3. Do the selected tasks have a reference reward function in the environment? If so, I'd like to see the learning curve for the oracle reward function. If possible, please share the original prompt used to generate the reward function. This information tells us how granular the reward function is. Your experiments suggest the robotic reward needs to be more fine-grained. If you prompt the reward function to be more fine-grained, will ReLoop still improve performance?
  4. The ablation studies are not enough to justify all designs. For example, you introduce five coefficients in the modified loss function; in a new environment, how should I tune these hyperparameters? Also, two key designs are Ground Process and Online Memory; your ablation studies justify Online Memory, but they do not include Ground Process.
  5. As introduced in the related work, your method is closely related to reward shaping and exploration. You should compare it to some of the related methods.
  6. The predicate and the discretization are not fully discussed in the paper.

[1] Ma, Y. J., et al. (2023). Eureka: Human-Level Reward Design via Coding Large Language Models. arXiv:2310.12931.

Reviewer_TsojOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ReLoop takes an LLM-generated reward program (Text2Reward-style Python), instruments it so that intermediate values (distances, grasp booleans, etc.) are logged at each step, and maps those values to four fixed semantic channels (reach ≺ interact ≺ goal ≺ stable) via a lexical rule on variable names. Channels are normalized to [0,1], gated by a sigmoid on preceding channels, averaged into a potential Φ, and binned into a discrete trace state z. The shaped reward is rₜ = 0.3·rᵖₜ + 3.0·ΔΦₜ + 0.1·Fₜ + 0.2·(γ_V·V(zₜ) − V(zₜ₋₁)) + 1·[successₜ], where Fₜ is a frontier bonus for exceeding the episode's previous maximum potential and V is a tabular success-occupancy score backed up at episode end. Evaluated on 2 ManiSkill2 + 6 Meta-World tasks, 3 seeds, against FrozenProgram (raw program reward), Matched (0.3·r + success bonus, no trace terms), ReturnOnly (same machinery but state built from the scalar), Flat (no gating, 3 tasks), PotentialOnly (ΔΦ only, 3 tasks).

Critical strengths

Matched control: Authors isolate scale + success bonus and still report a (smaller) gain. That’s the right experimental design even if the resulting number is modest. It’s consequential because it’s what makes the +7.0 interpretable at all.

Critical weaknesses

The paper’s own ablation shows the simpler PotentialOnly variant matches or beats the full method on every tested task (Table 3; +13.7 on SweepInto), and Flat shows no benefit from gating. So the method as presented — frontier bonus, success-occupancy memory, tabular backups, ordered gating — is mostly unsupported.

Questions for authors

1.Given Table 3, why keep the frontier and V terms? Is there any task or metric where Full > PotentialOnly outside noise? If not, would you reframe the method? 2. How does ReLoop compare to Meta-World’s built-in dense rewards and to a potential built from the same four quantities read directly from simulator state (no LLM program)?

Reviewer_rgUYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ReLoop reuses intermediate values from a fixed LLM-generated reward program as progress supervision. Variable names are mapped to ordered manipulation channels, producing a potential and trace state. The final reward combines program return, potential changes, frontier/memory terms and a success bonus. On eight manipulation tasks, the paper reports gains over the frozen program and a control matching reward scale and the success bonus.

Critical strengths
  1. Reusing computations already present in reward programs is a practical idea that adds no LLM calls and does not change policy observations. Shared coefficients across tasks also reduce the need for task-specific tuning.

  2. The experimental controls are thoughtfully chosen. In particular, the Matched control separates scaling and success-bonus effects, while ReturnOnly, Flat and PotentialOnly diagnostics expose which parts of the method may actually matter. The write-up states limitations and program provenance candidly.

Critical weaknesses
  1. The empirical support is small and concentrated. Eight tasks, one archived program per task and three seeds do not establish robustness to variation in generated reward programs. Much of the aggregate gain over Matched comes from PickCube, and diagnostic tasks were chosen using pilot outcomes. Per-task results and uncertainty are needed to show whether the average reflects a broadly useful improvement rather than a few favorable cases.

  2. The evidence does not justify the full method's complexity or isolate the source of the gain. PotentialOnly matches or exceeds the full method on the tested diagnostics, and Flat does not establish a benefit from ordering. A hand-designed reach/interact/goal/stable prior may account for part of the improvement. The lexical grounding is also sensitive to variable naming. These observations favor a simpler potential-shaping interpretation unless additional controls support the frontier, memory and ordering terms. I currently lean toward weak rejection.

Questions for authors
  1. How do per-task gains and their uncertainty compare with Matched, especially with and without PickCube? Can the diagnostics be extended beyond the pilot-selected tasks and to independently generated programs?

  2. Where do frontier and memory terms improve over PotentialOnly? How does a hand-written staged potential over the same environment quantities compare, and what evidence specifically supports channel ordering?

  3. How often does lexical grounding misclassify quantities, and what happens under variable renaming or equivalent program rewrites? Please also characterize failure cases from the history-dependent shaping and undiscounted potential differences rather than assuming policy invariance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.