acceptodds
Desk rejected at ICLR 2027

When Should an Agent Trust a Reward? Trajectory-Supported Learning under Corrupted Reward Observations

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning typically treats observed rewards as direct supervision, yet deployed agents may receive feedback corrupted during reward observation or transmission. Without structural assumptions, such corruption cannot be distinguished from rare but valid rewards using trajectories alone. We study a setting in which observed rewards equal clean environment feedback plus sparse or transient corruption that is weakly coupled to transition dynamics. We introduce ReConRL, which estimates trajectory support for each observed reward and modulates its influence on value learning rather than attempting to reconstruct an unobservable true reward. Support is estimated using temporally separated return evidence and transition-conditioned signals obtained through distinct estimation pathways, reducing dependence on self-consistency from a single critic. We construct matched cases that separate rare-but-valid rewards, frequent corrupted rewards, and equally unusual rewards with different trajectory support, directly testing whether rarity is confused with unreliability. Evaluation measures corrupted-reward acceptance, genuine-reward rejection, and policy return. This formulation studies when an observed reward is sufficiently supported by trajectory evidence to guide learning without assuming that unusual feedback is necessarily untrustworthy.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, reward corruption, robust reinforcement learning, trajectory consistency, reliable learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nzhzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method to improve reinforcement learning performance when reward corruption is weakly related to the context. They trained a model to predict a trust value, and used it to weight the loss function. The empirical results showed that their method improve the RL tasks performance under corruption.

Critical strengths

1)The statement of the logic of this paper is clear to me.

2)The assumption of this paper and the high-level idea of the corresponding method make sense to me.

Critical weaknesses

1)To my understanding, you use mostly average information from the context for the temporal part. But I’m worried that much information belonging to the MDP could be lost. Therefore, I doubt that by only using this information in the temporal part, the predicted trust value could be not that informative.

2)I hope the author can clarify the novelty of their method more clearly, as stated in my question part.

Questions for authors

1)To my understanding, you tried to train a model to judge if an (action, state_t, reward, state_t+1 ) pair follows(at least to some extent) the real local distribution of the MDP. I think it’s something that could be highly motivated when people study DRL problems under reward pollution or non-stationary MDP settings. I wonder if there are some close previous work that use similar methods?

Reviewer_UQbqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RECONRL addresses corrupted reward observations by learning per-transition trust from reward-trajectory association rather than reward rarity. A temporally separated context branch and a local transition branch use matched-versus-shuffled reward discrimination, followed by monotone fusion. The resulting weights modify only the backbone-native critic loss without reconstructing rewards. Experiments cover matched diagnostic MDPs, continuous control and Atari across five backbones, with corruption-severity sweeps, frozen-support mechanism shifts, and held-out-task design transfer.

Critical strengths
  1. The focus on potentially unreliable rewards from the environment is interesting.

  2. The experiments provide broad coverage of different environments and algorithms.

Critical weaknesses

I would first like to clarify that my familiarity with conventional RL algorithms and environments is limited, and my knowledge is mainly of some RLHF methods used with LLMs. The following questions therefore reflect my perspective as an LLM researcher, and I am not certain how well they apply across conventional RL settings.

  1. The paper focuses on the unreliability of rewards returned by the environment. While this can indeed be an issue in real-world settings, in my view, a more important and direct approach would be to repair the reward function rather than filter potentially problematic rewards. Even when the reward function cannot be repaired, the nature of its errors would still depend on the environment. Addressing these errors would therefore seem to call for environment-specific remedies on a case-by-case basis, rather than a general anomaly-detection approach.

  2. The authors assume that some rewards are noisy, then detect and downweight these rewards during RL training. They train a predictor to estimate whether a reward is trustworthy. Why not use the same training data to train a reward model directly? Its predictions might generalize to cases in which the observed rewards are unreliable, it should serve as a baseline.

  3. The reward noise constructed in this paper is largely random. In practice, however, reward noise is often systematic: it arises from recurring problems in the environment under particular conditions, rather than random deviations from the true reward. This extends my first concern. In such settings, it seems more appropriate to design different noise-robust strategies for the specific scenarios involved.

  4. I would like to see concrete real-world examples illustrating when the rewards returned by a reward function become noisy and what form that noise takes. If the evaluation merely adds random noise, I find that it provides little guidance for practical RL applications.

Questions for authors

Please refer to the weakness

Reviewer_WWZjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Authors proposed a new method that assigns each observed reward a trust weight based on how well it is supported by its surrounding trajectory context.

Critical strengths

The main critical strength of the paper is Proposition 1 that unrestricted reward corruption cannot be distinguished from clean rewards using trajectories alone. After saying that the paper carefully scoping its claims to sparse or transient corruption which is still valuable.

Critical weaknesses
  1. The paper claims monotone fusion because stronger evidence can never lower the trust weight. But that monotonicity holds for the evidence score, not for what the critic. Which is actually cares about how far the observed reward sits from the true latent reward. Importantly the two branch scores are proxies. In my understanding the property is just a fact about the gate, and it only means anything if you accept the untested assumption that corruption is contextually mismatched.

  2. The second issue is that the method uses a single context based scoring to understand the trust. In my understanding this means it cannot distinguish a reward that is wrong from a reward that is correct. I mean what if the reward simply does not reflected the trajectory context? This can make a sparse real rewards get down weighted. I see the discussion on this in the paper, but I think it's an obvious design weaknesses of the paper that need to be fixed. In experiments every corruption is made context-agnostic, so the method is only ever tested under the exact assumption it depends on.

Questions for authors

See weaknesses above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.