acceptodds
Withdrawn from ICLR 2027

Monitoring Emergent Reward Hacking During Generation via Internal Activations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Fine-tuned large language models can exhibit reward-hacking behavior arising from emergent misalignment, which is difficult to detect from final outputs alone. While prior work has studied reward hacking at the level of completed responses, it remains unclear whether such behavior can be identified during generation and whether activation-based monitors remain valid as models are further adapted. We propose an activation-based monitoring approach that detects reward-hacking signals from internal representations as a model generates its response. Our method trains sparse autoencoders on residual stream activations and applies lightweight linear classifiers to produce token-level estimates of reward-hacking activity. Across multiple model families and fine-tuning mixtures, we find that internal activation patterns reliably distinguish reward-hacking from benign behavior, generalize to held-out mixed-policy adapters, and exhibit model-dependent temporal structure during generation. We further show that monitor validity under model post-training updates is direction- and model-dependent: complete monitors can transfer from SFT to RL without requiring component replacement, while reverse RL-to-SFT transfer can benefit from updating downstream monitor components. Finally, we evaluate the frozen monitor on an external single-turn EvilGenie setting, comparing internal activation scores against an independent behavioral judge without retraining or recalibration. Together, these results suggest that activation-based monitoring can provide a complementary signal of emergent misalignment while remaining useful across adapter variation, post-training model updates, and external evaluation settings.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Activation Monitoring, Chain-of-Thought, AI Safety, Emergent Misalignment

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_hMTiOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether rh behavior in language models can be detected through internal activations during generation. The authors propose an activation-based monitoring framework that uses SAEs to extract features from residual stream activations, followed by PCA and linear classifiers to produce token-level rh scores.

The paper evaluates the proposed approach across different model families and LoRA adapters trained with varying proportions of reward-hacking data. The experiments examine the sensitivity of activation-based signals to rh supervision, their temporal dynamics during cot generation, and their ability to predict externally judged rh behavior, including on the EvilGenie benchmark. The authors further investigate monitor transferability between SFT and RL trained models through component wise replacement experiments.

Critical strengths
  1. This is an important and practically research question. The paper investigates an interesting research question: whether reward hacking behavior can be detected from internal activations during language model generation. This is a meaningful direction for AI safety, as activation based monitoring could potentially provide additional signals beyond output based evaluation.
  2. A complete and coherent monitoring framework. The paper presents a relatively complete activation based monitoring framework, covering activation collection, feature extraction using SAEs, and reward hacking prediction through linear classifiers.
Critical weaknesses

The experiments do not sufficiently demonstrate the advantages of activation based monitoring over output based evaluation. One of the main motivations of the paper is that reward hacking behavior can be difficult to detect from model outputs, as the generated responses may appear plausible even when the model's internal computations shift toward exploiting the reward objective. However, the proposed activation based monitor is primarily evaluated against behavioral labels obtained from external judgments of generated responses. They do not establish whether the proposed method can detect reward hacking that would be missed by output based monitoring. Furthermore, the paper lacks a direct comparison with output based monitoring baselines, making it difficult to assess the additional value of using internal activations.

Questions for authors

I understand that obtaining reliable ground truth labels for reward hacking behavior can be challenging, especially given the paper's motivation that output based evaluations may fail to detect certain forms of misalignment. However, could the authors consider using more objective or independently validated methods to determine whether reward hacking has actually occurred? For example, in coding tasks, could programmatic checks or executable evaluations be used to identify reward hacking behavior. Such evaluations could potentially provide a more reliable behavioral reference than relying solely on LLM-based judgments of generated responses.

Reviewer_qQPcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

In the paper the authors propose an activation-based monitoring method to detect and classify reward-hacking behavior in language models during inference time. So the method computes internal activations from transformer layers and passes them through a SAE, PCA, and a linear classifier to estimate reward-hacking scores which are then averaged across tokens and layers to get the final reward hacking score which is a scaler from 0-1.

The authors then use this method to study how these activation signals behave under different conditions. The paper mainly has three experiments, first, they examine how do the reward-hacking signals change as the percentage of reward-hacking data used during fine-tuning increases. Second, they experimented how these signals evolve throughout generation and with different reasoning lengths. Finally, the authors study whether monitors trained on SFT models continue to work on RL models and viceversa.

Critical strengths
  • The paper is well written and easy to follow.
  • The experimental section is clean and experiments cover all the questions the paper promises in the introduction. Particularly the evidence from the results are enough to draw the right conclusions that paper has shown like SFT to RL does not require to update the signals much but do in other way, etc.
Critical weaknesses
  • The main criticism that I have from the paper is that it lacks novelty. I do understand that it is an empirical paper, and it is fine to not have novelty in the method, but the paper lacks novelty in interesting findings. For examples, the finding that reward hacking signals do not change from SFT to RL but they do in reverse is because when doing RL on model we are not changing the model but mostly eliciting already existent behaviour (Yue et al.), while in SFT we are changing the model considerably using some offline data.

  • Similarly, I think for the insight that if we increase the percentage of reward hacking data, the amount of reward hacking behavior increases, also does not come to me as a surprise.

  • Overall my main criticism is that the three main experiments are lacking in terms of providing interesting insights and also the questions themselves does not seem coherent enough to be under the same study.

Yue et al., Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?. Neurips 2025

Questions for authors

Why do you average the reward hacking scores for all tokens, i am not sure if this is the best way. I do think that it may reduce a lot of information.

Reviewer_56dgOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes an activation-based framework for monitoring reward-hacking behavior during language model generation. The method uses sparse autoencoders (SAEs), PCA, and linear classifiers to extract token-level monitoring scores from internal activations. The authors consider two types of readouts: a provenance readout trained to distinguish Control and Hack adapters, and a behavior readout trained to predict externally judged reward-hacking behavior. Experiments investigate sensitivity to reward-hacking supervision, temporal activation patterns during chain-of-thought generation, generalization to the EvilGenie benchmark, and monitor transfer across SFT and RL updates.

Critical strengths
  1. The paper addresses an important problem in language model safety. Detecting reward-hacking behavior through internal activations is a potentially useful direction, particularly when output-based monitoring may be insufficient.
Critical weaknesses
  1. The methodological novelty of the proposed framework is limited. The monitoring pipeline essentially combines existing techniques, including SAEs, PCA, and linear classification, without introducing a new representation learning objective or detection mechanism. Moreover, the provenance readout is trained simply to distinguish activations from two adapters fine-tuned on different datasets (Alpaca and SRH). This is a relatively straightforward binary classification problem rather than a method specifically designed to identify reward-hacking behavior. Although the behavior readout uses behavioral supervision, it also relies on standard supervised linear probing. Overall, the methodological contribution beyond existing activation-based monitoring approaches remains unclear.
  2. The roles and relationship of the two readouts are unclear and confusing. The provenance readout distinguishes adapters trained on different datasets and is used for the main sensitivity and temporal analyses (Figures 2–3), whereas the behavior readout predicts actual reward-hacking behavior and is used for external generalization and SFT/RL transfer experiments. The experiments do not sufficiently rule out the possibility that provenance merely captures adapter-specific characteristics. Moreover, it is unclear what advantages provenance offers over behavior, whether it serves as a proxy, or why both are necessary. This fragmented experimental design makes the methodology and interpretation confusing, and raises further doubts about whether both readouts are genuinely generalizable detectors of reward-hacking behavior.
  3. The scientific significance of the temporal dynamics analysis is unclear. The main finding in Figure 3 is that different model families exhibit different temporal patterns in their activation scores. However, the paper does not explain why these differences arise, establish how they relate to actual reward-hacking behavior, or demonstrate how the observed patterns can improve monitoring. The proposed final monitoring decision simply averages token-level scores rather than exploiting the identified temporal characteristics. Thus, the analysis remains largely descriptive, and its implications for understanding or detecting reward hacking are insufficiently established.
Questions for authors
  1. What is the main technical contribution of the proposed monitoring pipeline beyond existing SAE-based representation learning and linear probing? In particular, what makes the provenance classification objective specific to reward-hacking detection rather than general adapter discrimination?
  2. Why are the central sensitivity and temporal analyses conducted using the provenance readout instead of the behavior readout, which is more directly aligned with the stated detection objective? Can the authors compare the two readouts on the same generated responses and determine whether their temporal patterns are consistent?
  3. What actionable or scientific insight follows from the observed model-dependent temporal patterns? Can these patterns be connected to realized reward-hacking behavior or used to improve monitoring performance?
  4. Since the proposed monitoring framework emphasizes token-level signals during generation, could the authors report detection performance using only partial generation prefixes? This would help clarify the practical value of monitoring internal activations before the full response is available.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.