acceptodds
Under review as a conference paper at ICLR 2027

Undiscounted Potential Based Reward Shaping

Abstract

Reinforcement learning usually trains with a discount factor for stability, even when the task's objective is the undiscounted return. In this paper, we compare two forms of a frequently used reward shaping method, which adds a shaping term to immediate rewards to speed up learning. Potential-based reward shaping (PBRS) provides a framework which guarantees that the shaping term does not change the discounted optimization objective. However, when training uses , the undiscounted objective does not necessarily coincide with the discounted objective. This means that PBRS cannot shape rewards towards an undiscounted optimum when the discounted objective differs from the undiscounted one. To provide stable training in these cases, we examine undiscounted potential-based reward shaping (PBRS), which drops the discount factor from the shaping term but still uses discounted targets when training reinforcement learning agents. Though PBRS drops the guarantee to be objective-preserving, we show that when the potential function corresponds to the optimal state value function, the PBRS-shaped rewards correspond to the action advantage function of the undiscounted objective and reproduce an optimal policy. As in practice we do not have optimal values, we further present an error bound based on the estimation bias of the state value function. In our experiments, PBRS beats PBRS and the correction term shaping with the same learned potential in several settings.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Reward Shaping, Discounting

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xFVgOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

I am putting my entire review here, as a sort of protest against the review form, which just makes it harder to be clear, in my honest opinion.

Paper 35262: Undiscounted Potential Based Reward Shaping

This paper explores the utility of using undiscounted potential-based reward shaping (PBRS) together with discounting RL algorithms to solve undiscounted episodic MDPs. It is a bit of an odd combination, as more usual would be have all three (shaping, algorithms, problem) discounted or all three undiscounted. In the paper, it is taken as normal to use discounted RL algorithms to solve undiscounted problems (even though this is known to be theoretically unsound). The paper does not challenge this supposedly normal but unsound practice, but instead argues that, if you are going to do it, and you want to do PBRS, then you should use undiscounted shaping. That is, the shaping should match the problem, not the RL algorithm. There are five claimed contributions about how undiscounted shaping might be better in this case, but all the contributions are weak (obvious mathematical observations and soft claims about working better on some problems).

I do not recommend accepting this paper because the results are weak and not placed in a meaningful context, and because the writing is poor. Weak, non-meaningful results together with poor presentation is a clear reject for me.

The reason the writing is so poor in this paper is presumably the heavy reliance on LLMs, as acknowledged at the end of paper. The paper is full of loose statements, statements that do not clearly mean anything, or that have ambiguous relationships between subjects and verbs. The first sentence of the abstract illustrates the latter. It is "Reinforcement learning usually trains..." Of course, RL is a field, or a kind of learning, and as such it is not something that "trains". Every sentence of the paper is a bit off like that. I also attribute the use of LLMs to the style of the paper, which never comes down to the key ideas, just dances around them with imprecise statements, changing perspectives constantly.

A second general problem for the paper is that PBRS, at least for this reviewer, is a questionable focus. PBRS has always seemed a complicated way to talk about an initial value function. Eric Wiewiora (2003) proved that the updates made by PBRS are exactly the same, step by step, as those by the ordinary algorithms if they were started with the potential function as their initial estimate of the value function. This makes sense: the "potential function" was always meant as an approximation of the correct value function, so it makes sense to initialize the approximate value function that way. It is the first thing you would think of. Thinking of it as a "shaped" reward is actually a complex, contrived way of thinking about the initial approximate value function. This still holds if \gamma=1. The Wiewiora paper is listed in this paper's references, but is never cited in this paper. As far as I am concerned, PBRS has already received far more attention than it deserves. None of the five weak contributions justifies giving it more.

Critical strengths

I don't find this part of the review form helpful.

Critical weaknesses

see review above.

Questions for authors

I don't find this part of the review form helpful.

1 comment

We thank the reviewer. The review makes four points, and we answer each.

1. "PBRS has always seemed a complicated way to talk about an initial value function" (Wiewiora, 2003), also at γ = 1. The Wiewiora initialisation claim holds for PBRS, and for PBRS₁ at γ = 1. It does not hold for PBRS₁ at γ < 1, the case the paper studies. Wiewiora's argument, applied to PBRS₁, shows that a PBRS₁ learner started at Q₀ makes exactly the updates of an unshaped learner started at Q₀ + Φ and trained on the reward r + (1−γ)Φ(s′). Initial values do not move a γ-discounted fixpoint. The added reward term does: with Φ = V∗₁ it moves the fixpoint to the undiscounted optimum at every γ < 1 (Theorem 2), which no initialisation can do. In this reading our 2×2 grid crosses initialisation at Φ with the level term (1−γ)Φ(s′), and PBRS₁ is significantly better than either factor alone (Table 3). At γ = 1 the level term vanishes and PBRS₁ equals PBRS. The revision cites Wiewiora (2003) in §2.

2. Training a discounted learner on an undiscounted problem is unsound, and the paper does not challenge it. The paper starts from exactly this unsoundness. §1 states that a learner trained below the Blackwell limit optimises a different objective than the undiscounted one. The paper then shows which shaping repairs this mismatch between the training discount and the objective. The combination is not specific to us. κ-PI, HuRL and reward tweaking (§2) pair a low training discount with a longer-horizon or undiscounted target by design. Training can benefit from a smaller discount factor, which reduces variance (§1: "estimates average over less randomness and have lower variance"), but it is hurt by a discount below the Blackwell limit (§1: "training will optimise a different objective"). A shaped reward can then move the learner closer to the undiscounted objective.

3. The contributions are obvious observations and soft claims without context. The review identifies no result as previously known and no proof step or experiment as incorrect. On context: standard PBRS puts the learner's γ into the shaping term. Practitioners drop it anyway and describe this as a sacrificed guarantee or an open question (§2), and PyBullet and Isaac Gym ship the undiscounted form in their progress rewards (§5). The theorems say when dropping γ is correct and what an inexact potential costs. On softness: the paper tests eight comparisons (PBRS₁ against PBRS and against the level term, with every shaping at its best γ and at its worst γ, under two potentials) on 24 environments with PQN and PPO, six discounts, 50 or 100 seeds and matched budgets. All eight of our claims are significant after Holm correction (Table 3). Could the reviewer name the result that is already known, with a reference, or the proof step or experiment that is incorrect?

4. Writing. The review gives one example, the first sentence of the abstract. We agree, and it now reads "Reinforcement learning agents are usually trained with a discount factor γ < 1". The review names no other passage, so we cannot tell which other statements it finds loose. The key idea does come across: the review's own summary states it correctly in one sentence ("the shaping should match the problem, not the RL algorithm"), and the summaries of the other two reviews state the contributions correctly as well. We will revise any specific passage the reviewer points to.

Reviewer_b24wOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies undiscounted potential-based reward shaping while retaining a discount factor during training. It shows that using the optimal undiscounted value as the potential recovers an undiscounted-optimal policy, and bounds the loss when the potential is approximate. Experiments compare the shaping variants across 24 environments and several training discounts.

Critical strengths
  • The paper gives a useful account of when dropping the discount from the shaping term can help. The optimal-potential result and the approximation bound make this more than an empirical recommendation.

  • The experiments provide a substantial comparison across environments and discounts, using shared potentials and matched training budgets. This supports the practical value of the proposed shaping choice.

Critical weaknesses
  • The discussion of dense-reward reconstruction is broader than Theorem 1 supports. The theorem requires an informative potential, but Remark 6 and the later discussion suggest that PBRS cannot reconstruct dense rewards in general. That conclusion does not hold without the additional condition. The distinction should be kept clear when describing the contribution.
  • Though the authors acknowledge the gap between theory and experiments, it leaves open how much the theoretical explanation accounts for the observed gains.
Questions for authors
  • Under what conditions does the dense-to-delatyed construction satifsy the informative-potential assumptions?
  • How sensitive are the learned-potential results?
2 comments

Thank you for the careful reading. Everything below is in the revised PDF.

W1. The dense-reward claim is broader than Theorem 1. Agreed. Remark 6, the paragraph after it, the Introduction, the third contribution, the Discussion and the Conclusion stated without condition what Theorem 1 proves only for an informative potential. All six now carry the condition.

Q1. When is the accumulated reward informative? When Φ is the reward accumulated so far on the delayed task, Definition 1 holds exactly when two reachable episodes from the same start have the same length and the same total reward but different γ-discounted dense returns (new App. A.8). Two such episodes that earn their total at different times satisfy this at all but finitely many γ. Example: with a fixed episode length, as in Freeway, it holds as soon as a reward can be collected earlier or later with the same total. It fails when the episode length fixes the reward sequence, for example a constant reward per step as in MountainCar or CartPole. There Theorem 1 is silent, and Remark 6 now says so.

W2. How much does the theory explain? The theorems cover an exact or ε-close potential: with Φ = V∗₁ the PBRS₁ optimum is the undiscounted optimum at every γ (Theorem 2), and an error ε in Φ costs at most 2ε per step (Theorem 3). They motivate the prediction that a badly chosen γ hurts PBRS₁ less than PBRS, also when Φ is an estimate of unknown accuracy. The experiments test this, and also whether PBRS₁ beats PBRS when each shaping is trained at its own best γ. Both hold after Holm correction (Table 3).

Q2. Sensitivity to the learned potential. App. F tests it: 3 or 4 source policies of increasing return per environment, one Φ fitted to each, in 14 environments at γ = 0.99 (50 environment-potential cells, 50 seeds per shaping). PBRS₁ against PBRS: 27 clear wins, 8 clear losses, 15 not separable. With the weakest potential PBRS₁ is ahead in 10 of the 14 environments (6 clear wins, 2 clear losses). The losses concentrate in two games, SpaceInvaders at all three of its potentials and flight_runner at two of four.

We hope this resolves the points raised. If anything remains open, we are glad to address it during the discussion.

Thanks for providing more details and addressing my concerns. I will keep my original positive score of 3.

Reviewer_fEYmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies potential-based reward shaping (PBRS) when the desired objective is the undiscounted return, but RL training uses a discount factor γ<1\gamma<1 for stability. Standard PBRS preserves the optimal policy of the discounted MDP, but cannot in general recover the undiscounted optimum when the two objectives disagree. The paper studies PBRS1\text{PBRS}_1, which removes the discount factor from the shaping term while retaining discounted training targets. Although PBRS1\text{PBRS}_1 is no longer generally objective-preserving, the authors show that, when the potential equals the optimal undiscounted value function, the shaped rewards correspond to the undiscounted advantage and recover an optimal policy. The paper further relates this issue to the Blackwell limit and empirically compares PBRS1\text{PBRS}_1 with standard PBRS and level-based shaping, including robustness to the training discount.

Critical strengths
  • Addresses an important limitation of established PBRS when the target objective is undiscounted but γ<1\gamma<1 is desirable for training.
  • Provides a rigorous theoretical characterization of PBRS1\text{PBRS}_1 and its relationship to the undiscounted optimum.
  • Clearly separates objective discounting from training discounting, which is conceptually important.
  • The experiments directly test the central claims, including performance relative to PBRS and sensitivity to γ\gamma.
Critical weaknesses
  • The theoretical setting appears largely restricted to finite state/action spaces. It is unclear how directly the guarantees extend to continuous spaces and function approximation.
  • The presentation is dense, with some important arguments deferred to the appendix. The experimental section could also be streamlined.
Questions for authors
  • Which theoretical results fundamentally require finite state/action spaces, and what prevents extensions to continuous spaces or function approximation?
  • Are the experiments also conducted on domains with finite state/action spaces, or do they cover continuous domains?
4 comments

Thank you for the precise questions. Everything below is in the revised PDF.

W1 / Q1. Which results need finite spaces? None. The submission assumed finite state and action spaces for simplicity, to keep the Setup easy to read. No proof uses finiteness itself. The revised Setup (§3) replaces it by the four conditions that Theorems 2 and 3 need, listed in the second bullet below. This question led us to state the results in the generality the proofs support, and the paper is stronger for it. The experiments already go beyond the finite case: 12 of the 24 environments have continuous states, and nine of these also have continuous actions (Q2).

  • Proposition 1 and Theorems 1 and 4 are identities along single trajectories. They hold in any state and action space. With continuous states, Theorem 1 is read almost surely (App. A.3).
  • Theorems 2 and 3 need four conditions: bounded rewards and Φ, an expected episode length bounded uniformly over policies, the Bellman optimality equations, and a maximising action in every state (for example compact action sets under the usual continuity conditions, Hernández-Lerma & Lasserre, 1996). Finite spaces imply all four. Beyond them the proofs use only a γ-contraction and sup-norm bounds.
  • With function approximation there are two sources of error. The first is the potential, which may differ from V∗₁ by up to ε. Theorem 3 already covers this error. The second is the learner, whose Q-function may differ from the optimal Q-function of the shaped problem by up to δ. The revision states the bound for both errors together: the policy that acts greedily on the learned Q-function loses at most 2(ε + δ) per step, and this holds at every γ. The statement follows Theorem 3 and the proof is in App. A.5. How small δ becomes depends on the learning algorithm, as it does in every bound of this kind.
  • Only the Blackwell limit of §1 needs finiteness, because with infinitely many actions a Blackwell limit may not exist. Then no training discount below 1 recovers the undiscounted optimum, and PBRS₁ matters more. Example (App. A.9): in s₀ an action a ∈ [0,1) pays a and ends the episode. The action a = 1 pays 0 and leads to s₁, where the single action pays 1 and ends. Undiscounted, a = 1 is optimal. At every γ < 1 it is worth γ, less than any a ∈ (γ,1). PBRS₁ with Φ = V∗₁ selects a = 1 at every γ (shaped values 0 against a − 1).

No theorem statement and no proof argument changes.

Q2. Finite or continuous domains? Both, twelve each, all with neural-network learners. Continuous states: CartPole, Acrobot, MountainCar and nine Brax tasks (Brax also has continuous actions). Discrete: four MinAtar and eight Octax games. Clear wins / clear losses of PBRS₁ against PBRS per environment (Table 3 split by domain type, added to App. I):

learned Φ, worst γlearned Φ, tuned γaccumulated reward, worst γaccumulated reward, tuned γ
continuous (12)10 / 13 / 112 / 06 / 3
discrete (12)9 / 08 / 111 / 06 / 0

W2. Presentation. We used the tenth page of the discussion phase for the arguments the review missed in the main text. §3 now gives the proof idea of Theorem 1 directly after the theorem, and it states the bound for an approximate learner directly after Theorem 3. The experimental section is organised by the three questions stated at its start (§4).

We hope this resolves the concerns behind the rating. If it does, we would be grateful if you reconsidered your score. If anything remains open, we are glad to address it during the discussion.

Dear Reviewer fEYm, the discussion phase ends on November 18. We would be grateful to know whether our reply and the revised Setup address your question on finite state and action spaces, or whether anything remains open.

I thank the authors for the clarifications, revision, and addressing my concerns. I will update my score after discussing with other reviewers.

Thank you for letting us know. We are glad to answer any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revisions

We thank all three reviewers. A revised PDF is uploaded. No theorem statement, number or experimental result has changed, and no experiment was added. The numbering of theorems, remarks, tables, figures and appendices is unchanged. The main text has 10 pages.

Theory

  • §3 Setup: finite state and action spaces are no longer assumed. The Setup states the four conditions the proofs use (fEYm).
  • After Theorem 3: the bound 2(ε + δ)·E[T] for a learner whose Q is δ-accurate, with proof in App. A.5 (fEYm).
  • App. A.3: almost-sure reading of Theorem 1 for continuous states. App. A.9: an example without a Blackwell limit (fEYm).
  • §2 cites Wiewiora (2003) for the equivalence of PBRS and Q-value initialisation (xFVg).
  • Remark 6, the paragraph after it, §1, the third contribution, §5 and §6: the statement that PBRS cannot rebuild the dense returns now carries the condition of Theorem 1. App. A.8 proves when the accumulated reward meets it (b24w).
  • A proof idea follows Theorem 1 directly in the main text (fEYm).

Experiments, re-presentation of existing results only

  • App. F: PBRS₁ against PBRS on the Φ-quality ladder (b24w).
  • App. I: Table 3 split into continuous and discrete environments (fEYm).

The appendix is now part of the main PDF and replaces the one in the supplementary material and the first sentence of the abstract is corrected (xFVg).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.