- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates reward-coordinate sensitivity in history-conditioned reinforcement learning. The authors distinguish three notions of reward equivalence: objective equivalence, information equivalence, and policy-interface equivalence. They demonstrate that positive-affine reward transformations, despite preserving the underlying optimization objective and information about the task, can change the behavior of a fixed, trained policy.
The paper presents extensive experiments on AMAGO policies in HalfCheetah and Ant environments, using closed-loop evaluations, teacher-forced replay, and activation patching to characterize the effects of reward transformations. Additional experiments on RL² and Decision Transformers investigate the relationship between reward-coordinate sensitivity and reliance on trajectory-specific reward evidence.
Critical strengths
-
Extensive and carefully designed experiments. The paper investigates reward-coordinate sensitivity from multiple angles, including different transformations, intervention schedules, training coordinates, architectures, and environments. The experiments are generally well controlled, with multiple independently trained policies and additional robustness checks.
-
Thorough empirical characterization. Beyond demonstrating that reward transformations affect performance, the authors investigate their direct effects on actions, reversibility, and changes in internal representations. In particular, teacher-forced experiments help separate direct policy responses from environmental feedback, while activation-patching experiments provide additional insights into the representations involved.
-
Interesting empirical observations. The non-monotonic response to reward scaling, dependence on training coordinates, and differences across policy architectures are interesting findings that may inform the design and evaluation of history-conditioned RL agents.
Critical weaknesses
- Unclear novelty and practical implications of the central phenomenon.
My main concern is that the significance of the central phenomenon is not sufficiently established. A history-conditioned policy is explicitly trained to select actions based on previous reward observations. Therefore, modifying these observations would naturally be expected to produce different actions, particularly when the reward scale differs from the one encountered during training.
Although the transformations preserve the objective and information mathematically, the policy is not informed of the transformation and has not necessarily learned to interpret different reward coordinates equivalently. It is therefore unclear what makes the observed sensitivity distinctive beyond general neural-network sensitivity to changes in input distributions, or why reward-coordinate invariance should necessarily be expected.
The practical implications also remain somewhat unclear, particularly in settings where reward-coordinate consistency could be maintained through preprocessing or calibration.
- Limited interpretation of single-token and pulse experiments.
The single-token experiment demonstrates that modifying one reward observation directly changes the next action. However, changing a reward observation can naturally provide different evidence about the underlying task, even if the modified value remains within the original reward distribution. Therefore, this result seems largely expected for a history-conditioned policy.
Similarly, the pulse experiments introduce temporary reward-coordinate changes without informing the policy. It is unclear why the policy should maintain equivalent behavior under such interventions. Moreover, a temporary transformation does not necessarily preserve the trajectory-level objective equivalence guaranteed by a consistent positive-affine transformation.
While these experiments establish direct sensitivity and reversibility, their implications for the broader claim of reward-coordinate invariance could be clearer.
- Limited systematic analysis of matched training and evaluation coordinates.
Section 4.1 presents an interesting experiment showing that models trained with transformed reward coordinates are substantially less sensitive to those coordinates during evaluation. This suggests that the observed phenomenon may be closely related to training–inference coordinate mismatch.
However, this direction is not systematically explored across the different interventions. Further comparisons between policies trained and evaluated with matched versus mismatched coordinates could help clarify whether the phenomenon represents a fundamental limitation of history-conditioned adaptation or primarily a calibration issue.
- Confounding factors in cross-architecture comparisons.
Section 6 suggests that reward-coordinate sensitivity tracks reliance on trajectory-specific reward evidence. However, AMAGO, RL², and Decision Transformers differ in multiple aspects beyond reward reliance, including architecture, training procedure, task, and reward-input representation.
Consequently, the cross-architecture comparisons provide suggestive evidence, but do not fully isolate reward reliance as the explanation for the observed differences in sensitivity.
- Presentation and organization.
Although the paper contains substantial experimental detail, I found the organization somewhat difficult to follow. The main sections are organized primarily around experimental conclusions, while the underlying settings, evaluation protocols, and model configurations change across and within sections.
As a result, it is sometimes difficult to track which model, training coordinate, intervention schedule, and evaluation metric correspond to each result. A clearer overview of the experimental configurations would improve readability.
Questions for authors
-
Motivation and implications. Why should a history-conditioned policy trained under one reward coordinate be expected to behave equivalently under another coordinate at inference time, especially when it is not informed of the transformation? Is such invariance necessarily desirable? Could the authors further clarify what makes the observed phenomenon distinctive and provide concrete practical scenarios where this sensitivity poses a meaningful problem?
-
Interpretation of interventions. For the single-token and pulse experiments, what conclusions should be drawn beyond the fact that the policy responds to changes in its reward inputs? In particular, since these interventions can change the evidence available for task inference, how should we distinguish undesirable coordinate sensitivity from an expected adaptive response?
-
Training-coordinate experiments. Could the authors more systematically investigate policies trained and evaluated under matched versus mismatched reward coordinates, including reverse transformations? Do policies trained under different but equivalent reward coordinates converge to comparable task performance and adaptation behavior? It would also be interesting to extend these comparisons to the static, switch, and pulse experiments. I acknowledge the augmentation experiments in Appendix H, although they address a somewhat different question.
-
Cross-architecture comparisons. Could the authors further clarify how strongly the results in Section 6 support the proposed relationship between trajectory-specific reward reliance and coordinate sensitivity, given the differences between the architectures and training settings? A more controlled comparison using the same architecture and task with varying degrees of reward reliance might strengthen this conclusion.
-
Presentation. Could the authors provide a consolidated overview of the experimental configurations, including the models, training coordinates, interventions, and evaluation metrics? This would make it easier to understand how the different experiments relate to one another.
4 comments
Thank you for clarifying the remaining question and for recognizing the additional experiments. We agree that a fixed network can be sensitive to an unknown input recoding.
Practical relevance. Reward-scale heterogeneity is an established concern in multi-task meta-RL, including AMAGO-2. Our study examines its role at the frozen policy's input. For a policy that does not consume rewards, changing reward preprocessing has no direct effect on action selection. For a history-conditioned policy, it also changes the feedback used for task inference and control.
A concrete source of mismatch is transferring a checkpoint to an evaluation pipeline whose reward-normalization state is reinitialized or differs from training. Gymnasium's documentation notes that normalization depends on past trajectories and can scale rewards incorrectly after the wrapper is newly instantiated. If original rewards are retained for scoring, the evaluation objective can stay fixed while the actor receives differently scaled inputs. This provides a concrete motivation; our experiments quantify controlled coordinate changes, rather than their frequency in deployed systems.
When calibration suffices. If the transformation is known and invertible, inverse calibration restores the native input and is an appropriate solution. Correctly preserving preprocessing with the checkpoint can therefore prevent this problem. Calibration is less direct when the reference normalization state is unavailable or scales differ across unlabeled tasks: the correspondence to the training coordinate must then be identified. Since reward also conveys task information, normalization estimated from a short history need not recover that correspondence. These broader calibration settings remain outside our experiments.
What the experiments add. General input-shift sensitivity establishes the possibility of a response; our results characterize its consequences and constrain its explanation:
- Learned calibration and control cost. With optimization and scoring rewards fixed, the preferred test coordinate reverses with training. Mismatch costs 240.1 and 199.5 original-return points in the two directions, with positive losses in every seed pair (Section 4.4, Table 2). The wider scaling panel is non-monotone, so distance from the native scale alone does not describe the response.
- A localized pathway and a constrained mechanism. Replay places the action change before environmental feedback, while pulses establish reversibility. HalfCheetah patching further shows that restoring the readily decoded task axis recovers little of the action effect; structured subspaces recover substantially more (Section 5). This constrains a specific task-representation explanation without claiming a complete mechanism.
- An evaluation criterion for adaptation. Coordinate stability must be assessed alongside competence and reward use: an agent that ignores feedback can appear invariant while failing to adapt. The audit separates direct action responses from original-task return consequences and checks both against native capability.
These findings support treating reward preprocessing as part of an adaptive policy's specification and evaluating its stability together with task competence. Known calibration is one way to maintain that specification; our evidence establishes why maintaining it matters in the tested agents.
Thank you for the further clarification. I find the reward-normalization example a convincing motivation, and I now better understand the practical relevance of the problem and the contributions of the empirical analysis. It might be helpful to include this example in the paper, particularly for readers who are familiar with RL but less familiar with reward-coordinate sensitivity.
Given the additional experiments and clarifications throughout the rebuttal, I will raise my overall rating to 3. I will keep my confidence at 2, as I am less familiar with this particular subfield and remain somewhat uncertain about my assessment of the contribution's significance.
Thank you for the constructive discussion and for suggesting this addition. We will incorporate the reward-normalization example, together with its supporting reference, into the introduction to make the practical motivation clearer. Your feedback has helped us sharpen both the presentation and the scope of the paper.
Thank you for the detailed response and additional experiments. I appreciate the new matched-versus-mismatched training-coordinate comparison, which provides stronger evidence for learned reward-coordinate calibration. The clarifications regarding the single-token and pulse experiments also address my concerns about their interpretation.
I appreciate the acknowledgment of confounding factors in the cross-architecture comparisons, although this concern remains partially addressed without a more controlled comparison.
I have one remaining question regarding the motivation and significance of the work. I acknowledge that I am not particularly familiar with the literature on history-conditioned RL, so I may be missing some important context. My understanding is that a fixed network would naturally be sensitive to changes in its reward inputs, especially when the transformation is unknown. The improvement from matching training and evaluation coordinates also seems consistent with this intuition. Could the authors help clarify why reward-coordinate sensitivity is an important problem in practice? In what scenarios would such mismatches arise and be difficult to address through calibration? More broadly, what new insights does this work provide beyond the expected sensitivity of neural networks to input-distribution shifts?
I would appreciate further clarification on this point, as it is important to my assessment of the paper.