acceptodds
Under review as a conference paper at ICLR 2027

Probing Reward Beliefs in Reinforcement Learning

Abstract

Driven by a decade of rigorous research, deep neural policies can now navigate complex MDPs, learn how to reason and strategize in high dimensional environments. While reinforcement learning is now a cornerstone of varied domains, spanning from language model mathematical and scientific reasoning to sensitive settings such as medical and finance, a prominent line of research seeks to infer reward functions by observing how an optimal policy behaves. This approach is predicated on the claims that reconstructing the underlying objective leads to policies more closely aligned with intended outcomes than those defined by hand. In this paper, we analyze the underpinnings of learning with reward beliefs in high-dimensional state representation MDPs. We demonstrate that this premise has fundamental shortcomings and standard deep reinforcement learning yields more resilient and value-aligned policies when compared to learning from the behaviour of other policies in MDPs with complex state representations.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
reward beliefs, standard reinforcement learning, value inflation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_EdBxOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies the adversarial robustness of imitation learning policies, specifically the IQ-learn policy. The paper provides a proof that the IQ-learn policy is untrained outside the expert subspace, leaving the policy random once it drifts off expert data manifold. The authors frames this as "value inflation" and "alignment illusion".

The authors conducted experiment in the ALE environments, showing that the imitation policy's performance decreases with added random noise and the learner policy's reward prediction's correlation with environment reward also decrease.

Critical strengths
  • This paper provides an interesting proof for the IQ-learn policy being untrained outside the expert manifold using a linear parameterization.
  • Experiment results are generally consistent with what the theory expects.
Critical weaknesses
  • I think this paper runs a major risk of anthropomorphizing and over-interpreting the results. While this paper provides an interesting and novel analysis specifically for IQ-learn, offline/off-policy training and distribution shift from expert manifold are well known problems in imitation learning [1] and the phenomena observed by the authors are exactly consistent with what's already known. I don't know if the observations qualify as "overfit to the expert’s beliefs". Furthermore, in practice one aught to evaluate the imitation policy in scenarios of varying difficult. Generally the performance of these offline algorithms like behavior cloning and IQ-learn with expert-only data drop quickly in these scenarios. So there wouldn't be an "alignment illusion" easily.
  • On line 376, which part of the results show decrease in the accuracy of the predicted rewards?
  • On line 411, I see the initial negative predicted performance drop. But I am skeptical of it being noise rather than signal, since your theory is value function outside expert manifold should be random which I agree. Is there any way to show this phenomenon will hold generally rather than idiosyncratically in the current set of experiments?
  • On line 413, you state "small deviations cause the policy to form an inaccurate view of the rewards it will obtain". Again your theory says value should be random outside expert manifold. So they aren't really forming any inaccurate view. I actually think the randomness explains decreased correlation with environment reward -- is this the right way to understand this?

[1] Spencer, J., Choudhury, S., Venkatraman, A., Ziebart, B., & Bagnell, J. A. (2021). Feedback in imitation learning: The three regimes of covariate shift.

Questions for authors

Minor: I think there are some typos.

  • On line 105, I think its logsumexp(Q).
  • In eq. (1) you are missing a minus sign between the two terms. Also what's ϕ\phi?
Reviewer_Ce99OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies robustness of policies learned from expert demonstrations, using IQ-Learn as the imitation/IRL method and DDQN as the reward-trained RL baseline. The authors perturb policies either through transferred adversarial observation directions or by forcing occasional random/worst actions, and measure performance degradation and the correlation between IQ-Learn's inferred reward and environment reward. Under the tested perturbations, IQ-Learn often degrades much more than DDQN, and very small trajectory deviations can substantially reduce the correlation between inferred and environment rewards.

Critical strengths

The empirical brittleness result is large and quite surprising. Specifically across the tested Atari settings, the author s show that transferred perturbations produce much larger normalized degradation for IQ-Learn than DDQN, and the inferred reward becomes poorly correlated with environment return after small deviations

Another thing I liked is that the paper probes the learned reward representation rather than only reporting attacked return. This stems from the observation that measuring both task performance and inferred-reward correlation reveals that clean policy performance can coexist with severe off-trajectory instability in the learned reward representation.

Critical weaknesses

*First axis: Theory *

  1. The theoretical explanation underlying the paper’s main claim is not valid as written, and multiple steps fail independently. First, Lemma 3.1 contains a false inequality. The proof uses max⁡aQ(s′,a)≥1∣A∣∑aQ(s′,a)2\max_a Q(s',a)\ge \frac{1}{\sqrt{|A|}} \sqrt{\sum_a Q(s',a)^2} which is not generally true without the assumption that Q>=0.
  2. Second, even setting this error aside, the paper does not derive the property needed for the subsequent theory. Lemma 3.1 concerns projection onto the span of the linear Q parameters, whereas Definition 3.2 assumes that every state in every expert trajectory lies entirely inside a strict subspace W. The authors explicitly “posit” this stronger condition. Thus the central off-subspace argument relies on an additional restrictive assumption rather than following from expert trajectories having high value.
  3. Third, Corollary 3.4 does not follow from Proposition 3.3 as written. The Proposition only establishes that the learned Q-function remains at initialization for a state s∈W⊥s\in W^\perp. In the corollary, however, the proof also replaces the Q-values of successor states s′∼P(⋅∣s,a)s'\sim P(\cdot|s,a) by their initialization values. This requires the transition to remain in W⊥W^\perp but no such transition-closure assumption is made in the corollary.
  4. Even if this was repaired, the corollary establishes only that $ \mathbb E_{\theta^{(0)}}[r_\theta(s,a)] $ is independent of ss. (ev is over random initializations). It does not imply that, for one trained network, the inferred reward is uncorrelated with the environment reward across visited states or trajectories. The paper explicitly treats the empirical near-zero Pearson correlations as verification of Corollary 3.4.
  5. The theorem 3.5 is valid only as a constructed sufficient condition under these strong assumptions; it does not establish a general or intrinsic cost of learning from demonstrations because it obtains catastrophic failure only after assuming that a perturbation sends the learner into W⊥W^\perp, that it can never return to W, that the imitation policy is untrained there, and that the optimal policy nevertheless receives the same return there as on the expert manifold.

Second axis (experiments): The experiments demonstrate IQ-Learn brittleness, but they do not establish that “lack of exploration” is responsiblel which is a much broader claim; and several reported metrics are not sufficient for the alignment conclusions drawn from them.

I believe the empirical phenomenon itself is interesting specifically since the tested IQ-Learn policies degrade strongly under the chosen perturbations, and their inferred cumulative rewards become poorly correlated with environment returns. However, my concern is largely around the core comparison which is IQ-Learn versus DDQN. These methods differ simultaneously in their objective, supervision, state distribution, amount/type of environment interaction, learned representations, and optimization procedure. So these experimetns cannot distinguish between i) IQ-Learn is unusually fragile or ii) learning from demonstrations without exploration is intrinsically fragile. I couldn’t find any evidence in the paper where they directly manipulate state coverage or exploration while keeping the learning method otherwise fixed. Take for example, there is no IQ-Learn experiment with progressively increasing learner-generated/off-expert data, nor an online/improved-coverage version showing robustness recovering as the proposed mechanism predicts. There are also no additional imitation/IRL methods demonstrating that the phenomenon generalizes beyond IQ-Learn. Nevertheless, the manuscript repeatedly generalizes from this single algorithm to imitation learning, inverse RL, “reward-free algorithms,” and even RLHF/alignment.

The reward-correlation argument is also overstated. Table 3 shows that Pearson correlation between cumulative inferred reward and environment return can become small or negative after perturbation. That is evidence that this particular inferred reward representation behaves differently off-distribution. It does not imply, as the paper states, that “without the correlation … the decisions of the policy will be random.” A policy's action probabilities are determined by its learned Q-function; zero correlation between two trajectory-level scalar quantities does not mathematically imply a random policy.

The novelty framing also appears overstated. The paper claims to be the first to study adversarial vulnerabilities of deep policies learned without predefined rewards, including imitation and inverse RL. However, Wang et al. (2022), Adversarially Robust Imitation Learning, already explicitly studies deep imitation policies under adversarial state perturbations, and prior imitation-learning work has long studied failure after distributional shift away from expert trajectories (e.g., DAgger). More recently, Caliskan et al. study inverse RL specifically under state-adversarial perturbations. The reward-belief analysis here is more specific, but I think the paper should revise its novelty claims.

Per my read, the paper has some interesting ideas but I do not find the current theoretical or experimental evidence sufficient to support the authors main message which is why I recommend rejection.

Questions for authors
  1. Can the authors provide an experiment that directly tests the proposed causal mechanism? I gave an example in the weakness section, please refer to the above.
  2. Do the main conclusions hold for other imitation/IRL methods, rather than only IQ-Learn?
  3. In Lemma 3.1 did you intend to assume non-negativity? If not, how would you repair it?
Reviewer_LPEkOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper compares the robustness of IQ-Learn and DDQN under transferred observation perturbations and occasional action interventions. It examines both environment returns and inferred rewards, and proposes a linear-subspace explanation for the observed vulnerabilities. Atari experiments show substantial degradation of IQ-Learn and reduced correlation between inferred and environment rewards.

Critical strengths

1.Relevant diagnostic question.

Studying inferred rewards alongside actual returns helps expose failure modes that policy-performance measurements alone may miss.

2.Substantial empirical effects.

The transfer-attack experiments show large IQ-Learn–DDQN robustness gaps, with Gaussian-noise controls and standard errors over ten runs.

Critical weaknesses

1.The central theoretical argument contains substantive gaps.

Lemma 3.1 incorrectly assumes max⁡aQa≥∥Q∥2/∣A∣\max_a Q_a\geq\|Q\|_2/\sqrt{|\mathcal A|}, which fails for Q=(1,-10). Corollary 3.4 applies Proposition 3.3 to successor states without assuming they remain in W⊥W^\perp. Moreover, state-independent expectations over initialization do not establish reward decorrelation for a trained policy. These issues undermine the claimed theoretical explanation.

2.The experiments do not establish the proposed causal or general alignment conclusions.

Comparing IQ-Learn with DDQN does not isolate exploration from differences in objectives, reward access, and training data. Reduced reward correlation also does not imply random decisions or general value misalignment. The extrapolation from Atari results to RLHF and language-model safety substantially exceeds the evidence.

Questions for authors

1.How can Lemma 3.1 and Corollary 3.4 be repaired, and which subsequent conclusions remain valid under the necessary additional assumptions?

2.What controlled evidence isolates exploration as the cause of the robustness gap? What justifies interpreting reward decorrelation as policy misalignment?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.