- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
I am putting my entire review here, as a sort of protest against the review form, which just makes it harder to be clear, in my honest opinion.
Paper 35262: Undiscounted Potential Based Reward Shaping
This paper explores the utility of using undiscounted potential-based reward shaping (PBRS) together with discounting RL algorithms to solve undiscounted episodic MDPs. It is a bit of an odd combination, as more usual would be have all three (shaping, algorithms, problem) discounted or all three undiscounted. In the paper, it is taken as normal to use discounted RL algorithms to solve undiscounted problems (even though this is known to be theoretically unsound). The paper does not challenge this supposedly normal but unsound practice, but instead argues that, if you are going to do it, and you want to do PBRS, then you should use undiscounted shaping. That is, the shaping should match the problem, not the RL algorithm. There are five claimed contributions about how undiscounted shaping might be better in this case, but all the contributions are weak (obvious mathematical observations and soft claims about working better on some problems).
I do not recommend accepting this paper because the results are weak and not placed in a meaningful context, and because the writing is poor. Weak, non-meaningful results together with poor presentation is a clear reject for me.
The reason the writing is so poor in this paper is presumably the heavy reliance on LLMs, as acknowledged at the end of paper. The paper is full of loose statements, statements that do not clearly mean anything, or that have ambiguous relationships between subjects and verbs. The first sentence of the abstract illustrates the latter. It is "Reinforcement learning usually trains..." Of course, RL is a field, or a kind of learning, and as such it is not something that "trains". Every sentence of the paper is a bit off like that. I also attribute the use of LLMs to the style of the paper, which never comes down to the key ideas, just dances around them with imprecise statements, changing perspectives constantly.
A second general problem for the paper is that PBRS, at least for this reviewer, is a questionable focus. PBRS has always seemed a complicated way to talk about an initial value function. Eric Wiewiora (2003) proved that the updates made by PBRS are exactly the same, step by step, as those by the ordinary algorithms if they were started with the potential function as their initial estimate of the value function. This makes sense: the "potential function" was always meant as an approximation of the correct value function, so it makes sense to initialize the approximate value function that way. It is the first thing you would think of. Thinking of it as a "shaped" reward is actually a complex, contrived way of thinking about the initial approximate value function. This still holds if \gamma=1. The Wiewiora paper is listed in this paper's references, but is never cited in this paper. As far as I am concerned, PBRS has already received far more attention than it deserves. None of the five weak contributions justifies giving it more.
Critical strengths
I don't find this part of the review form helpful.
Critical weaknesses
see review above.
Questions for authors
I don't find this part of the review form helpful.
We thank the reviewer. The review makes four points, and we answer each.
1. "PBRS has always seemed a complicated way to talk about an initial value function" (Wiewiora, 2003), also at γ = 1. The Wiewiora initialisation claim holds for PBRS, and for PBRS₁ at γ = 1. It does not hold for PBRS₁ at γ < 1, the case the paper studies. Wiewiora's argument, applied to PBRS₁, shows that a PBRS₁ learner started at Q₀ makes exactly the updates of an unshaped learner started at Q₀ + Φ and trained on the reward r + (1−γ)Φ(s′). Initial values do not move a γ-discounted fixpoint. The added reward term does: with Φ = V∗₁ it moves the fixpoint to the undiscounted optimum at every γ < 1 (Theorem 2), which no initialisation can do. In this reading our 2×2 grid crosses initialisation at Φ with the level term (1−γ)Φ(s′), and PBRS₁ is significantly better than either factor alone (Table 3). At γ = 1 the level term vanishes and PBRS₁ equals PBRS. The revision cites Wiewiora (2003) in §2.
2. Training a discounted learner on an undiscounted problem is unsound, and the paper does not challenge it. The paper starts from exactly this unsoundness. §1 states that a learner trained below the Blackwell limit optimises a different objective than the undiscounted one. The paper then shows which shaping repairs this mismatch between the training discount and the objective. The combination is not specific to us. κ-PI, HuRL and reward tweaking (§2) pair a low training discount with a longer-horizon or undiscounted target by design. Training can benefit from a smaller discount factor, which reduces variance (§1: "estimates average over less randomness and have lower variance"), but it is hurt by a discount below the Blackwell limit (§1: "training will optimise a different objective"). A shaped reward can then move the learner closer to the undiscounted objective.
3. The contributions are obvious observations and soft claims without context. The review identifies no result as previously known and no proof step or experiment as incorrect. On context: standard PBRS puts the learner's γ into the shaping term. Practitioners drop it anyway and describe this as a sacrificed guarantee or an open question (§2), and PyBullet and Isaac Gym ship the undiscounted form in their progress rewards (§5). The theorems say when dropping γ is correct and what an inexact potential costs. On softness: the paper tests eight comparisons (PBRS₁ against PBRS and against the level term, with every shaping at its best γ and at its worst γ, under two potentials) on 24 environments with PQN and PPO, six discounts, 50 or 100 seeds and matched budgets. All eight of our claims are significant after Holm correction (Table 3). Could the reviewer name the result that is already known, with a reference, or the proof step or experiment that is incorrect?
4. Writing. The review gives one example, the first sentence of the abstract. We agree, and it now reads "Reinforcement learning agents are usually trained with a discount factor γ < 1". The review names no other passage, so we cannot tell which other statements it finds loose. The key idea does come across: the review's own summary states it correctly in one sentence ("the shaping should match the problem, not the RL algorithm"), and the summaries of the other two reviews state the contributions correctly as well. We will revise any specific passage the reviewer points to.