acceptodds
Under review as a conference paper at ICLR 2027

Equivalent Rewards, Unequal Interfaces: Reward Coordinates in History-Conditioned RL

Abstract

History-conditioned reinforcement learning agents use rewards both to define an objective and as inputs for online adaptation. Does preserving the objective also preserve adaptation? We distinguish objective, information, and policy-interface equivalence, and study their separation using interventions that alter only the rewards visible to the actor. Across five independently trained AMAGO policies in each of two continuous-control task families, positive-affine reward transformations alter actions and original-task returns. HalfCheetah exhibits a non-monotone, training-coordinate-specific response; Ant-Fwd-Back reproduces closed-loop sensitivity and a direct, reversible reward-to-action pathway. Teacher-forced replay localizes effects before environmental feedback, while HalfCheetah activation patching identifies structured, action-relevant changes not captured by the decoded task axis. Recurrent RL agents also exhibit strong coordinate sensitivity, whereas Decision Transformer controls show weaker sensitivity consistent with their weaker reliance on trajectory-specific reward evidence. Zero and donor interventions further connect coordinate sensitivity to reward reliance. These results demonstrate reward-interface dependence in learned adaptation: equivalent objectives and information need not produce equivalent behavior when reward is part of the policy interface.

Primary area:
reinforcement learning
Keywords:
Meta-Reinforcement Learning, In-Context Reinforcement Learning, Reward Transformations, History-Conditioned Policies

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UNw9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates reward-coordinate sensitivity in history-conditioned reinforcement learning. The authors distinguish three notions of reward equivalence: objective equivalence, information equivalence, and policy-interface equivalence. They demonstrate that positive-affine reward transformations, despite preserving the underlying optimization objective and information about the task, can change the behavior of a fixed, trained policy.

The paper presents extensive experiments on AMAGO policies in HalfCheetah and Ant environments, using closed-loop evaluations, teacher-forced replay, and activation patching to characterize the effects of reward transformations. Additional experiments on RL² and Decision Transformers investigate the relationship between reward-coordinate sensitivity and reliance on trajectory-specific reward evidence.

Critical strengths
  • Extensive and carefully designed experiments. The paper investigates reward-coordinate sensitivity from multiple angles, including different transformations, intervention schedules, training coordinates, architectures, and environments. The experiments are generally well controlled, with multiple independently trained policies and additional robustness checks.

  • Thorough empirical characterization. Beyond demonstrating that reward transformations affect performance, the authors investigate their direct effects on actions, reversibility, and changes in internal representations. In particular, teacher-forced experiments help separate direct policy responses from environmental feedback, while activation-patching experiments provide additional insights into the representations involved.

  • Interesting empirical observations. The non-monotonic response to reward scaling, dependence on training coordinates, and differences across policy architectures are interesting findings that may inform the design and evaluation of history-conditioned RL agents.

Critical weaknesses
  1. Unclear novelty and practical implications of the central phenomenon.

My main concern is that the significance of the central phenomenon is not sufficiently established. A history-conditioned policy is explicitly trained to select actions based on previous reward observations. Therefore, modifying these observations would naturally be expected to produce different actions, particularly when the reward scale differs from the one encountered during training.

Although the transformations preserve the objective and information mathematically, the policy is not informed of the transformation and has not necessarily learned to interpret different reward coordinates equivalently. It is therefore unclear what makes the observed sensitivity distinctive beyond general neural-network sensitivity to changes in input distributions, or why reward-coordinate invariance should necessarily be expected.

The practical implications also remain somewhat unclear, particularly in settings where reward-coordinate consistency could be maintained through preprocessing or calibration.

  1. Limited interpretation of single-token and pulse experiments.

The single-token experiment demonstrates that modifying one reward observation directly changes the next action. However, changing a reward observation can naturally provide different evidence about the underlying task, even if the modified value remains within the original reward distribution. Therefore, this result seems largely expected for a history-conditioned policy.

Similarly, the pulse experiments introduce temporary reward-coordinate changes without informing the policy. It is unclear why the policy should maintain equivalent behavior under such interventions. Moreover, a temporary transformation does not necessarily preserve the trajectory-level objective equivalence guaranteed by a consistent positive-affine transformation.

While these experiments establish direct sensitivity and reversibility, their implications for the broader claim of reward-coordinate invariance could be clearer.

  1. Limited systematic analysis of matched training and evaluation coordinates.

Section 4.1 presents an interesting experiment showing that models trained with transformed reward coordinates are substantially less sensitive to those coordinates during evaluation. This suggests that the observed phenomenon may be closely related to training–inference coordinate mismatch.

However, this direction is not systematically explored across the different interventions. Further comparisons between policies trained and evaluated with matched versus mismatched coordinates could help clarify whether the phenomenon represents a fundamental limitation of history-conditioned adaptation or primarily a calibration issue.

  1. Confounding factors in cross-architecture comparisons.

Section 6 suggests that reward-coordinate sensitivity tracks reliance on trajectory-specific reward evidence. However, AMAGO, RL², and Decision Transformers differ in multiple aspects beyond reward reliance, including architecture, training procedure, task, and reward-input representation.

Consequently, the cross-architecture comparisons provide suggestive evidence, but do not fully isolate reward reliance as the explanation for the observed differences in sensitivity.

  1. Presentation and organization.

Although the paper contains substantial experimental detail, I found the organization somewhat difficult to follow. The main sections are organized primarily around experimental conclusions, while the underlying settings, evaluation protocols, and model configurations change across and within sections.

As a result, it is sometimes difficult to track which model, training coordinate, intervention schedule, and evaluation metric correspond to each result. A clearer overview of the experimental configurations would improve readability.

Questions for authors
  1. Motivation and implications. Why should a history-conditioned policy trained under one reward coordinate be expected to behave equivalently under another coordinate at inference time, especially when it is not informed of the transformation? Is such invariance necessarily desirable? Could the authors further clarify what makes the observed phenomenon distinctive and provide concrete practical scenarios where this sensitivity poses a meaningful problem?

  2. Interpretation of interventions. For the single-token and pulse experiments, what conclusions should be drawn beyond the fact that the policy responds to changes in its reward inputs? In particular, since these interventions can change the evidence available for task inference, how should we distinguish undesirable coordinate sensitivity from an expected adaptive response?

  3. Training-coordinate experiments. Could the authors more systematically investigate policies trained and evaluated under matched versus mismatched reward coordinates, including reverse transformations? Do policies trained under different but equivalent reward coordinates converge to comparable task performance and adaptation behavior? It would also be interesting to extend these comparisons to the static, switch, and pulse experiments. I acknowledge the augmentation experiments in Appendix H, although they address a somewhat different question.

  4. Cross-architecture comparisons. Could the authors further clarify how strongly the results in Section 6 support the proposed relationship between trajectory-specific reward reliance and coordinate sensitivity, given the differences between the architectures and training settings? A more controlled comparison using the same architecture and task with varying degrees of reward reliance might strengthen this conclusion.

  5. Presentation. Could the authors provide a consolidated overview of the experimental configurations, including the models, training coordinates, interventions, and evaluation metrics? This would make it easier to understand how the different experiments relate to one another.

4 comments

Thank you for the detailed response and additional experiments. I appreciate the new matched-versus-mismatched training-coordinate comparison, which provides stronger evidence for learned reward-coordinate calibration. The clarifications regarding the single-token and pulse experiments also address my concerns about their interpretation.

I appreciate the acknowledgment of confounding factors in the cross-architecture comparisons, although this concern remains partially addressed without a more controlled comparison.

I have one remaining question regarding the motivation and significance of the work. I acknowledge that I am not particularly familiar with the literature on history-conditioned RL, so I may be missing some important context. My understanding is that a fixed network would naturally be sensitive to changes in its reward inputs, especially when the transformation is unknown. The improvement from matching training and evaluation coordinates also seems consistent with this intuition. Could the authors help clarify why reward-coordinate sensitivity is an important problem in practice? In what scenarios would such mismatches arise and be difficult to address through calibration? More broadly, what new insights does this work provide beyond the expected sensitivity of neural networks to input-distribution shifts?

I would appreciate further clarification on this point, as it is important to my assessment of the paper.

Thank you for clarifying the remaining question and for recognizing the additional experiments. We agree that a fixed network can be sensitive to an unknown input recoding.

Practical relevance. Reward-scale heterogeneity is an established concern in multi-task meta-RL, including AMAGO-2. Our study examines its role at the frozen policy's input. For a policy that does not consume rewards, changing reward preprocessing has no direct effect on action selection. For a history-conditioned policy, it also changes the feedback used for task inference and control.

A concrete source of mismatch is transferring a checkpoint to an evaluation pipeline whose reward-normalization state is reinitialized or differs from training. Gymnasium's documentation notes that normalization depends on past trajectories and can scale rewards incorrectly after the wrapper is newly instantiated. If original rewards are retained for scoring, the evaluation objective can stay fixed while the actor receives differently scaled inputs. This provides a concrete motivation; our experiments quantify controlled coordinate changes, rather than their frequency in deployed systems.

When calibration suffices. If the transformation is known and invertible, inverse calibration restores the native input and is an appropriate solution. Correctly preserving preprocessing with the checkpoint can therefore prevent this problem. Calibration is less direct when the reference normalization state is unavailable or scales differ across unlabeled tasks: the correspondence to the training coordinate must then be identified. Since reward also conveys task information, normalization estimated from a short history need not recover that correspondence. These broader calibration settings remain outside our experiments.

What the experiments add. General input-shift sensitivity establishes the possibility of a response; our results characterize its consequences and constrain its explanation:

  • Learned calibration and control cost. With optimization and scoring rewards fixed, the preferred test coordinate reverses with training. Mismatch costs 240.1 and 199.5 original-return points in the two directions, with positive losses in every seed pair (Section 4.4, Table 2). The wider scaling panel is non-monotone, so distance from the native scale alone does not describe the response.
  • A localized pathway and a constrained mechanism. Replay places the action change before environmental feedback, while pulses establish reversibility. HalfCheetah patching further shows that restoring the readily decoded task axis recovers little of the action effect; structured subspaces recover substantially more (Section 5). This constrains a specific task-representation explanation without claiming a complete mechanism.
  • An evaluation criterion for adaptation. Coordinate stability must be assessed alongside competence and reward use: an agent that ignores feedback can appear invariant while failing to adapt. The audit separates direct action responses from original-task return consequences and checks both against native capability.

These findings support treating reward preprocessing as part of an adaptive policy's specification and evaluating its stability together with task competence. Known calibration is one way to maintain that specification; our evidence establishes why maintaining it matters in the tested agents.

Thank you for the further clarification. I find the reward-normalization example a convincing motivation, and I now better understand the practical relevance of the problem and the contributions of the empirical analysis. It might be helpful to include this example in the paper, particularly for readers who are familiar with RL but less familiar with reward-coordinate sensitivity.

Given the additional experiments and clarifications throughout the rebuttal, I will raise my overall rating to 3. I will keep my confidence at 2, as I am less familiar with this particular subfield and remain somewhat uncertain about my assessment of the contribution's significance.

Thank you for the constructive discussion and for suggesting this addition. We will incorporate the reward-normalization example, together with its supporting reference, into the introduction to make the practical motivation clearer. Your feedback has helped us sharpen both the presentation and the scope of the paper.

Reviewer_A47GOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how reward transformations affect history-conditioned RL policies. The authors distinguish objective, information, and policy-interface equivalence, and argue that the first two do not guarantee the third. They evaluate affine reward transformations on AMAGO agents trained on HalfCheetah and Ant-Fwd-Back, examining changes in task performance and action selection. They also use teacher-forced replay and activation patching to investigate these changes, and examine reward dependence in RL2 and Decision Transformer agents.

Critical strengths
  1. The paper separates changes in task performance from the direct effects of reward transformations on action selection through controlled experiments.
  2. The experiments cover multiple reward transformations, two continuous-control task families, and different policy architectures, including AMAGO, RL2, and Decision Transformer.
Critical weaknesses
  1. The main claim has limited novelty. Objective equivalence concerns the ordering of policies, while information equivalence only guarantees that the original rewards can be recovered through an inverse transformation. Neither implies that a learned policy should behave identically when its reward inputs are transformed. Proposition 2.4 therefore does not seem to establish much beyond what follows directly from these definitions. The empirical contribution is also limited. The evaluated agents are trained to use reward histories for task inference and adaptation. Changing the reward scale or offset at evaluation time changes the inputs on which this behavior was learned, so the resulting changes in actions and performance are largely expected. Although the paper examines non-monotonic responses and internal representations, it remains unclear what these findings establish beyond the sensitivity of learned policies to changes in their input representation.

  2. The paper is difficult to follow due to its dense writing and the way the argument is developed across sections. The explanation is often more complicated than necessary, making the central contribution difficult to identify. Additionally, the relationship between history-conditioned RL and the meta-RL settings used in the experiments could be explained more explicitly.

Questions for authors
  1. How does reward-interface dependence differ from the general problem of input-distribution shift?
  2. What do the additional findings, particularly the non-monotonic responses and activation-patching results, tell us about history-conditioned RL beyond the observed reward-coordinate sensitivity?
Reviewer_eg4DOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper studies the question of whether reward shaping that perserve the objective also preserve the behavior of frozen policies that receive the reward as part of their input (e.g., Decision Transformers). The authors describe three types of equivalence: objective, information and policy-interface. The paper mostly deals with what happens when the first two hold and the third does not.

In their analysis, the authors use several degrees of transformation to the reward the actor "sees" (but not to the environment itself). Their experiments are run with AMAGO Transformers, RL2 GRUs, and Decision Transformers. They claim the following findings: 1) The reward modificaitons change actions and returns; 2) Sesitivity follows the training coordinate; 3) The effects are reversible after pulses; 4) DTs are less sensitive to the reward modifications.

The contributions claimed by the authors are:

  1. A formal definition/separation of the three equivalences mentioned above
  2. The definition of a causal link between reward modification/shaping and its effect on actions taken by the model
  3. Cross-carrier evidence that reward-interface dependence is organized by a policy’s reliance on trajectory-specific reward evidence.
Critical strengths
  1. An interesting question and (potentially) useful answers - the authors tackle a question that (to my knowledge) was not addressed before. The effect reward shaping has on performance is potentially important, and the authors point it out in a convincing way.

  2. The experiments are extensive (algorithm-wise) and well executed - several models, the use of competence gates, interventions are paired, etc.

  3. The pulse experiments were an interesting addition to the evalutaion

Critical weaknesses
  1. Limited novelty and scope - while interesting, the core phenomenon is to be expected, and the size of its effect (at least on relatively small datasets on which it was evaluated) is not great. Pointing out a problem in existing approaches is always important, but one would expect a solution (the best option) or a wide-ranging experimental setup that can quantify the problem on a range of problems (second best). This paper has neither.

  2. Lack of "real world", practical demonstration of the problem's impact - the authors don't present any of the realistic scenarios they mention in the introduction: wrappers, nomalization conventions, or learned reward models. Presenting results for these use cases exactly would strengthen the paper significantly. Two examples: 1) AMAGO on tasks with different reward scales; 2) dealing with natural drift between training and testing

  3. Lack of statistical rigor - the uncertainty is reported over 5 seeds, but there are no statistical significance tests or confidence intervals. Given the relatively small number of seeds, and the sometimes small differences in results, the lack of these analyses weakens the results.

Questions for authors
  1. The authors are welcome to address the weaknesses I wrote earlier.

  2. How many seeds were trained for each family, and how many failed the competence gate? Did the gate thresholds change after seeing any results?

  3. Can you show a realistic deployment scenario where the actor-visible coordinate changes unintentionally? for example, through normalization drift or multi-task scale heterogeneity?

Comments on the paper

1 comment

Thanks for the clarification. The response explains the scope of the analysis, but my concern about its contribution beyond the expected sensitivity to changes in reward inputs remains. I will maintain my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.