- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates whether rh behavior in language models can be detected through internal activations during generation. The authors propose an activation-based monitoring framework that uses SAEs to extract features from residual stream activations, followed by PCA and linear classifiers to produce token-level rh scores.
The paper evaluates the proposed approach across different model families and LoRA adapters trained with varying proportions of reward-hacking data. The experiments examine the sensitivity of activation-based signals to rh supervision, their temporal dynamics during cot generation, and their ability to predict externally judged rh behavior, including on the EvilGenie benchmark. The authors further investigate monitor transferability between SFT and RL trained models through component wise replacement experiments.
Critical strengths
- This is an important and practically research question. The paper investigates an interesting research question: whether reward hacking behavior can be detected from internal activations during language model generation. This is a meaningful direction for AI safety, as activation based monitoring could potentially provide additional signals beyond output based evaluation.
- A complete and coherent monitoring framework. The paper presents a relatively complete activation based monitoring framework, covering activation collection, feature extraction using SAEs, and reward hacking prediction through linear classifiers.
Critical weaknesses
The experiments do not sufficiently demonstrate the advantages of activation based monitoring over output based evaluation. One of the main motivations of the paper is that reward hacking behavior can be difficult to detect from model outputs, as the generated responses may appear plausible even when the model's internal computations shift toward exploiting the reward objective. However, the proposed activation based monitor is primarily evaluated against behavioral labels obtained from external judgments of generated responses. They do not establish whether the proposed method can detect reward hacking that would be missed by output based monitoring. Furthermore, the paper lacks a direct comparison with output based monitoring baselines, making it difficult to assess the additional value of using internal activations.
Questions for authors
I understand that obtaining reliable ground truth labels for reward hacking behavior can be challenging, especially given the paper's motivation that output based evaluations may fail to detect certain forms of misalignment. However, could the authors consider using more objective or independently validated methods to determine whether reward hacking has actually occurred? For example, in coding tasks, could programmatic checks or executable evaluations be used to identify reward hacking behavior. Such evaluations could potentially provide a more reliable behavioral reference than relying solely on LLM-based judgments of generated responses.