acceptodds
Under review as a conference paper at ICLR 2027

Task-Directed Reward Shaping with Weight-Conditioned Policies

Abstract

Reward shaping has been used successfully in large-scale reinforcement learning systems across a wide-range of domains. Yet poorly chosen reward weights can direct learning toward shaping signals at the expense of the task objective. Selecting these weights often requires manual iteration, repeated policy training, or bilevel optimization. We introduce Mix-Max Policy Search (MMPS), which takes a multi-objective reinforcement learning perspective to tune a fixed library of auxiliary rewards for a given task. MMPS trains a shared, weight-conditioned actor–critic across reward mixtures and uses its predicted primary-task value to select the mixture for each rollout. The selected mixture guides both on-policy data collection and the PPO update, sharing learning across weights while directing training toward behaviors predicted to yield high task return. Across nine IsaacGym tasks, MMPS achieves higher task performance than existing weight-tuning methods under matched environment-interaction budgets, with its largest gains on difficult manipulation tasks, achieving the score of policies trained with expert-tuned reward weights.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, reward shaping

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HisiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed MMPS method which is a reward weight tuning method which takes a multi-objective reinforcement learning perspective to tune a fixed library of auxiliary rewards for a given task. MMPS trains a shared, weight-conditioned actor–critic across reward mixtures and uses its predicted primary-task value to select the mixture for each rollout. MMPS achieves higher task performance than existing weight-tuning methods under matched environment-interaction budgets, with its largest gains on difficult manipulation tasks, achieving 2.26× the score of policies trained with expert-tuned reward weights.

Critical strengths
  1. The empirical performance is strong which outperforms BO, PBT and BiPaRS

  2. The framework is simple and easy to implement.

Critical weaknesses
  1. The key component of proposed method relies how can task critic can accurate rank the candidate reward weights. While the paper do evaluate random select vs MMPS, the random select itself is a weak baseline. The result do not directly validate wheather it can rank weight correctly. As a result, it remains unclear whether the gains stem from accurate task-directed weight selection or from the richer reward-conditioned actor-critic architecture and multi-head value-function representation.

  2. For the theoretical part, in Theorem J1, auther frezz the selector which select same weight did not really reflect the actual method. In actual method, it can choose different weight every rollout and updates. The objective may becoming a running target. The theory therefore provides limited insight into the stability and convergence behavior of the algorithm used in practice.

Questions for authors
  1. Line 78, auther cite the "... are auxiliary shaping rewards available during training (Ng et al., 1999). " However, the Ng 1999 mainly talks about shaped reward in PBRS format which is not related with the reward type of auther proposed in this paper. Can auther clearify why the PBRS reward shaping format is mentioned here? or what is the relationship between proposed method with PBRS?

  2. The proposed method required much larger network ("MMPS agent has approximately 0.81M–1.12M") compare withe default PPO baseline (" 0.13M–0.25M "). Can auther use parallel critic to bring default PPO the same scale at MMPS and compare?

  3. Can auther provide analysis on how V0V_0 is accurate?

Reviewer_aNAzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors want to tune shaping weights without training a separate RL agent for every choice. MMPS uses one PPO policy conditioned on the weights, together with a critic that predicts returns for each reward component. Before a rollout, it tries several candidate weights and picks the one with the highest predicted task return. PopArt and component-wise advantage normalization handle reward scales. The method is tested on nine modified IsaacGym tasks.

Critical strengths
  • This is a problem people run into when training RL agents.
  • The selector looks at task return, so it doesn't simply favor a mixture because that mixture makes its own shaping reward easy to collect.
  • There are quite a few useful controls: random selection, split networks, the task-only vector critic, normalization ablations, and different conditioning architectures.
Critical weaknesses
  • The BO setup bothers me most. Each candidate gets only 393k environment steps, while MMPS keeps training its shared learner over 500 million steps. I agree that total interactions should be counted, and BO has to pay for separate learners. But there is still a choice between trying many weights briefly and trying fewer weights properly. On the harder tasks, these short runs may mostly tell us which weights make early progress. I'd want to see longer trials, a multi-fidelity search, or a search-then-retrain comparison before trusting the size of the gap.

  • A weight that gives good task return now isn't necessarily the one that will help learning most next. Greedy selection could drop a mixture that is currently weak but useful for exploration. Warm-start may help with this, but the paper doesn't directly check whether task-value ranking predicts learning progress or whether useful mixtures get abandoned.

  • There are still other differences mixed into the comparison. MMPS has about 0.81M-1.12M parameters, compared with 0.13M-0.25M for default PPO, and it uses component-wise normalization. The existing controls help. Still, I'd like a fixed-weight learner with the same actor capacity and normalization, so we can see what the online selection adds.

  • Appendix J uses a state-wise mixture over weights, but the actual method chooses one weight and keeps it for the rollout. Those generally give different trajectory distributions. I don't see the argument connecting the two. Keeping the weight and rollout boundary in an augmented state might be the cleaner route. Also, the local alignment condition doesn't tell us that the learned selector actually finds aligned updates.

Questions for authors
  1. Why give BO exactly 393,216 steps per trial? What happens with fewer, longer trials, or if the best weights are retrained for a full policy-training budget?
  2. How well does the critic rank weights against actual task returns? Is it less reliable for weights that are rarely picked?
  3. How much do the sparse-task results depend on the 25% warm-start? I'd like to see its effect separated from the later task-value selection.
  4. What exactly is capacity-matched in the task-only control? Can you add a fixed-weight control with the same actor size and normalization?
  5. How does Appendix J account for keeping a weight fixed through a rollout? Also, how many runs were completed or excluded for each method and task?
Reviewer_VWfVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how to weight a set of hand-designed auxiliary rewards (distance, grasp, action penalties, etc.) so that a PPO policy ends up doing well on a fixed, often sparse task reward. The authors propose Mix-Max Policy Search (MMPS): a single actor and critic both take the weight vector as input, so one network represents the whole family of shaped behaviors, and the critic has one head per reward component. The task-reward head serves as a cheap proxy for how good the behavior under mixture w\mathbf{w} is. At every rollout, they draw 32 random weight directions plus the task-only weight, keep the one with the highest predicted task value, collect data with it, and update the shared networks. Components of different magnitudes are handled by normalizing each component’s value targets and minibatch advantages before the weight combines them. Empirically, on nine modified IsaacGym tasks, MMPS shows promising performance improvement, with the gains concentrated on the three sparse manipulation tasks.

Critical strengths
  1. The research question is important and practical. Choosing weights across reward terms is a tedious part of reward engineering, and a method that avoids separate learners per candidate and avoids differentiating through PPO updates would be welcome.
  2. The method is simple and intuitive. The selection rule is one argmax over sampled candidates, and the remaining pieces are combined in a sensible way.
Critical weaknesses
  1. The presentation of this paper could be largely improved, and the main body lacks the details needed to fully understand what was done.

a. The mechanism ablation (Section 4.3) does not say what the two controls actually are. For Split Networks, the number of partitions, how weight space is partitioned, whether each sub-network is still w-conditioned within its region, and how the 500M-step budget is divided are all unstated in detail. For Random-candidate, does e₀ remain in the candidate set? Section 4.3 also never references Table 3.

b. The path from the value vector to the scalar PPO advantage, which is the core of the update, is deferred to the Appendix. Relatedly, Section 3.3 attributes scale control to sampling w on the ℓ₂ sphere, but the sphere alone cannot fix reward units. I inferred that what does is the per-component PopArt and the per-minibatch advantage standardization. The main text should say this.

c. Much of the main body is spent on the same result: Figure 1(a), Table 1, Figure 1(b), Figure 2, and Table 2 all report terminal performance, and Table 1 duplicates the values already printed on the bars of Figure 1(a), while Figure 1(c) is a panel for two numbers that appear verbatim in Section 4.4 and Section 6. Meanwhile, Table 2 has no Default PPO or random-policy column, so the normalization in Eq. 1 cannot be reconstructed.


2. The soundness of the empirical comparison is also my concern.

a. Many related methods are discussed, but few are benchmarked, and the most relevant one is missing. ORSO [1] is described in Section 5 as the candidate-selection approach closest to MMPS. MORSE [2], cited as the successor to BiPaRS, is also not compared. Without these, it is hard to assess the significance of the claimed gains.

b. The baselines that are run are trained under different conditions, not just with different methods. For example, MMPS and default-weight PPO use 16,384 parallel environments (Appendix D). But BO uses 512 environments (Appendix B), while PBT uses 2,048 environments shared across 16 agents. In addition, BO and PBT also use different PPO settings (minibatch, epoch, environment steps, learning rates and value coefficients). Therefore, are those baselines being compared by a fair setup?

c. What is the motivation for sampling those candidate weights instead of optimizing them via gradient method? Since Vϕ,0(s0,w)V_{\phi, 0}(s_0, w) is conditioned on w, gradient ascent on it over the sphere (or sampling followed by a few refinement steps) is a natural alternative. Also, there is no ablation on the number of sampling K either.


References:

[1] ORSO: Accelerating Reward Design via Online Reward Selection and Policy Optimization (Zhang et al., 2025)

[2] Automatic Reward Shaping from Multi-Objective Human Heuristics (Xie et al., 2025)

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.