acceptodds
Under review as a conference paper at ICLR 2027

Target Value as Potential: Reward Shaping in Deep Reinforcement Learning by Transferring Prior Knowledge

Abstract

We propose Target Value as Potential (TAP), a simple reward shaping method that accelerates learning in deep reinforcement learning (RL) by transferring prior knowledge. Although reward shaping provides a natural mechanism for leveraging prior knowledge, theoretical analysis and empirical results show that its effectiveness depends on how this knowledge is represented and that priors constructed using existing methods may not reliably improve performance. To address this challenge, TAP introduces a new approach to constructing priors by using only the critic’s target value as the shaping potential within the classical Potential-Based Reward Shaping (PBRS) framework. Consequently, TAP requires neither additional representational structures nor extra hyperparameters. Our analysis and experiments show that TAP accelerates convergence and improves cumulative returns over standard DRL baselines, including TD3 and D4PG. Across tasks in the DeepMind Control Suite, TAP consistently outperforms existing reward shaping methods, such as Dynamic Potential-Based Reward Shaping (DPBRS), and more general methods for incorporating prior knowledge into RL, such as Heuristic-Guided Reinforcement Learning (HuRL). Furthermore, initial results integrating TAP with MR.Q, a representation-learning method for pixel-based observations, also results in improvements across two visual control environments. These results suggest that TAP remains effective even when the shaping potential is derived from high-dimensional learned representations rather than true environment states.

Primary area:
reinforcement learning
Keywords:
reward shaping, deep reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GSeiOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes TAP, a simple potential‑based reward‑shaping technique that repurposes target critic values as shaping potentials for DRL.

Critical strengths
  1. This paper proposes a new potential-based reward-shaping method.
  2. The paper validates its effectiveness through multiple experiments.
Critical weaknesses

The experiments are insufficient: they lack more visual-control benchmarks, do not compare against a sufficient number of baselines, and provide no hardware-efficiency analysis of the proposed method.

Questions for authors
  1. The authors note that both existing potential-based reward-shaping schemes increase complexity. It is recommended to conduct a complexity analysis, or a hardware-efficiency analysis (FLOPs, throughput, memory consumption, etc.), comparing the proposed method with baseline methods, including state-of-the-art potential-based reward-shaping methods.
  2. What would be the effect of directly using the online critic, rather than the target critic, as the potential function? Why must the target critic be used as the potential function?
  3. Two visual-control tasks are too few; it is recommended to increase the number to at least five visual-control tasks.
  4. The comparison between the original DRL algorithms and the Base-Visual algorithm is too limited, and the compared algorithms are all too old. It is recommended to additionally compare against five state-of-the-art DRL algorithms and several visual DRL algorithms.
  5. The baselines from related methods in the same field (same-type reward shaping / prior-knowledge transfer methods) are also too few and mostly outdated. It is recommended to compare against at least five state-of-the-art same-type reward-shaping / prior-knowledge transfer methods. In addition, reward shaping includes other types of methods besides potential-based ones; it is recommended to add several other types of reward-shaping methods as comparisons.
  6. If the prior knowledge is very poor, for example a source critic of very low quality, is the proposed method still effective?
Reviewer_poDqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes TAP within potential-based reward shaping. TAP uses only the critic’s target value to construct the shaping potential. The potential is actually a state-action value, but because TAP evaluates the target policy’s action, it becomes a state-only potential, making it simple in construction and easy to implement.

The theoretical intuition is that TAP is like a useful value initialization that aids learning. Theorem 1 states that TAP converges to an optimal policy and preserves potential-based reward-shaping policy invariance. Theorem 2 shows that the shaped iteration has a tighter worst-case contraction bound than iteration initialized from zero.

The experiments evaluate TAP-Self, TAP-Same, and TAP-Cross. For TAP-Same and TAP-Cross, the paper demonstrates target-task adaptation improvement, but not necessarily total sample efficiency. The paper also provides qualitative exploration results and comparisons against HuRL, DPBRS, and warm-start initialization.

Critical strengths
  • The core idea could be elegant, avoiding extra representation parameters.
  • The paper is well-written and clearly states its contributions.
  • The discussion does not claim beyond the paper, and the limitations are clearly written.
Critical weaknesses
  • A big weakness is that there is no formal statement or intuition for why TAP-Self would converge. TAP-Self’s potential changes during training, and it is still unclear whether it retains policy invariance over the overall learning process or whether we get the actual optimal policy.
  • I am also not convinced of the faster-convergence claim, as that would require proving (ns<nnsn_s<n_{ns}). Theorem 2 only gives a worst-case or upper-bound convergence relation through point 3.
  • The experimental comparisons and implementation are insufficiently clear. For TAP-Same and TAP-Cross, the paper demonstrates target-task adaptation improvement, not necessarily total sample efficiency. The paper should report both target-only and source-plus-target environment steps and compute.
  • The TAP-Self implementation should be explicitly presented as pseudocode or an algorithm.
  • Details are also missing for HuRL and DPBRS datasets, tuning, checkpoint selection, and implementation, making it difficult to determine whether all methods received comparable prior information and tuning effort.
  • The exploration claim is only qualitative, and the warm-start comparison is unclear.
Questions for authors
  • How does using a noisy learning critic avoid the misalignment problem present in prior work? Could the circularity of using an unconverged critic produce an incorrect dense signal, amplify estimation errors, or cause the algorithm to exploit early and get stuck in bad regions of the state space?
  • For TAP-Self, is the potential calculated inside no grad with stopped gradients? Is there a feedback loop from the potential, or is a separate critic used? Please explicitly present the TAP-Self implementation as pseudocode or an algorithm.
  • Since TAP-Self’s potential changes during training, under what assumptions does it preserve policy invariance over the overall learning process? Do we get the actual optimal policy?
  • How does Theorem 2 establish faster convergence? Point 3 compares the upper bounds, but it does not appear to prove (ns<nnsn_s<n_{ns}).
  • Please also explicitly state all assumptions, including boundedness of the iterates, the policy being greedy with respect to the shaped Q-value, (0<Qθ′≤Q∗0<Q_{\theta'}\leq Q^*), and the norm condition.
  • For TAP-Same and TAP-Cross, what are the total source-plus-target environment steps and computational costs?
  • How were the HuRL and DPBRS datasets, hyperparameters, and checkpoints selected and tuned?
  • Can the exploration claim be supported with quantitative metrics such as coverage, entropy, novelty, or visitation diversity? The current evidence is only qualitative. Check Metra (Park et. al.) or CSF (Zheng et al.) for ideas on quantifying the exploration claim.
  • See weaknesses.
Reviewer_otkBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work presents Target Value as Potential (TAP), a reward-shaping method that uses an actor-critic target network’s state value as the potential in potential-based reward shaping. The paper studies three variants: TAP-Self updates the potential from the current run, TAP-Same freezes a critic trained on the same task, and TAP-Cross transfers a critic from an easier related task. A fixed-potential analysis argues that an informative critic preserves optimal policies and reduces the residual value that must be learned. Experiments across DeepMind Control Suite with TD3, D4PG, DPBRS, HuRL, and preliminary MR.Q experiments report faster learning, higher returns, and lower variance, especially in humanoid tasks.

Critical strengths
  • TAP reuses the target critic already maintained by off-policy actor-critic methods, requires no auxiliary reward model or shaping-specific scale, and changes mainly the reward used in the Q-target. The residual-value interpretation gives a clear reason why an informative critic can accelerate learning.
  • The paper evaluates 20 proprioceptive DMC tasks, TD3 and D4PG, 10 seeds per task, same-task and cross-task transfer, DPBRS and HuRL baselines, a warm-start control, and initial pixel-based MR.Q integration. The reported gains on difficult humanoid tasks are substantial and the implementation details are relatively complete.
Critical weaknesses
  • Classical PBRS invariance assumes a fixed potential, while TAP-Self changes the target critic throughout training. The paper calls this quasi-static because Polyak updates are slow, but gives no finite drift rate, approximation error, or nonstationary-potential guarantee. Theorem 2 also assumes exact Bellman updates and an informative potential; it therefore explains an idealized fixed-potential regime rather than the deep actor-critic experiments.
  • TAP-Same uses a critic from a separate full run on the same task, but the source-run sample cost is not counted, so its target-task acceleration does not by itself demonstrate overall sample efficiency. TAP-Cross is evaluated on four hand-paired, state/action-compatible source-target tasks without broad mismatch or negative-transfer analysis. Most evaluation is dense-reward DMC; the two MR.Q experiments are preliminary, and the paper reports per-task learning curves with shaded confidence ranges, but no aggregate performance-profile summary or significance tests across the full suite.
Questions for authors
  • Can you provide a finite-sample or stochastic-approximation analysis for TAP-Self with a continuously evolving potential, including the effect of Polyak drift on policy invariance and convergence?
  • When total source-task plus target-task samples are counted, how do TAP-Same and TAP-Cross compare with baseline training? Can negative-transfer pairs and state/action mismatch be systematically evaluated?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.