acceptodds
Under review as a conference paper at ICLR 2027

Replication Is Not Specificity: Negative Controls for Representation-Based Reward-Hacking Detection

Abstract

Representation-based detectors of reward hacking are validated mainly by accuracy and by replication across tasks, architectures or models, and only in part by controls like those we apply here. Replication shows that a statistic is stable, not that it tracks reward hacking rather than training progress, text style or ordinary change. We apply three controls to our own significant results: a clean-run placebo, a benign-onset control in which a genuine capability unlocks on the exploit's schedule, and a scorer swap that holds the text fixed and changes the model reading it. In nine toy environments, activation anisotropy shifted at hacking onset in 8 of 9, and as often in runs with no hacking. Against benign onsets it separates only one exploit type, a false self-report (AUROC 1.00 and 0.98 under an MLP and a GRU; 0.37–0.69 elsewhere), and there the raw observation stream separates the arms as well. In three LLM families, base models that never saw reward-hacking data reproduce 97.7% of the anisotropy gap between hacking and control completions. An onset detector fires on 8/8 hacking and 0/3 clean runs in two models, yet on re-collections in three models, clean fine-tunes and untuned base models reading the same text reproduce 97–110% of the rise. A hack-versus-clean linear probe on raw activations keeps a small model-state component under the scorer swap, but in two of three models a benign change of fine-tuning data leaves one too, and a bag-of-words reader of the same text comes within 0.021 AUROC of the probe. A planted model-state signal on matched inputs, which the scorer swap passes by construction, separates from benign runs at AUROC 0.57–0.63 at the smallest dose and 0.81–0.97 at the largest. We release the suite and all 357 runs.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
reward hacking, negative controls, AI safety, representation-based monitoring, activation probing, interpretability, specification gaming

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_JHAEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper asks whether representation-based reward-hacking detectors actually detect hacking itself, or if they just pick up on normal training progress, harmless behavior changes, or properties of the generated text. To test this, the authors use three baselines: a clean training run (a placebo), a benign capability change that matches the timing and reward jump of an exploit in toy settings, and a "scorer swap" where the text is held constant but activations are pulled from a different model.

Critical strengths

Great Research Angle Reward hacking is hard to monitor, frontier labs tried to solve this via agents monitoring & researcher baby sitting. If a model-internal representation-based metric can work for such goal, this would be ground breaking.

Useful findings The experimental results are helpful for future research on the same direction.

Critical weaknesses

Ineffectiveness of the method From what I read, Table 2 shows the proposed method is effectly not useful? The proposed probe will also activate on Clean training runs, which makes the paper of less meaning.

Hard to follow Writing needs to be significantly improved.

Questions for authors

What is the main contribution of this paper? Some clarifications would be great, thanks.

Reviewer_3rfJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that the prevailing validation practice for representation-based reward-hacking detectors — reporting accuracy and showing that a statistic replicates across tasks, architectures or models — establishes stability rather than specificity. A statistic that moves at hacking onset may equally be tracking training progress, a benign mid-training change, or the surface form of the generated text. To make the distinction operational, the authors specify three negative controls and apply them to five of their own previously significant results: a clean-run placebo, which scores well-specified runs containing no hacking and splits them at the matched onset; a benign-onset control, in which a genuine capability unlocks on the exploit's schedule and payout profile; and a scorer swap, which holds the scored text fixed and changes the model reading it, instantiated against both the pre-fine-tuning base model and a clean sibling at the matched checkpoint. A planted positive control spikes a synthetic unit of known size into benign runs' hidden streams, giving the benign-onset control an operating characteristic.

The testbed comprises nine toy environments spanning three architectures and six exploit mechanisms, and three LLM families (Llama-3.1-8B, Qwen3-8B and OLMo-3-7B) fine-tuned by a third party on the School of Reward Hacks dataset, with LoRA adapters additionally trained on Qwen2.5-1.5B, SmolLM2-1.7B and Qwen2.5-7B. Four of the five results each fail one control: the toy anisotropy sign claim fails the clean-run placebo, which reaches the same 8/9 on runs with nothing to detect; the LLM anisotropy gap fails the base-model scorer swap, which reproduces 97.7% of it; and an onset detector firing on 8/8 hacking and 0/3 clean runs fails the scorer swap, with untuned readers reproducing 97–110% of the rise. The fifth, a false self-report in environment H2, passes the placebo and the benign-onset control but fails a toy scorer swap, since the raw observation stream separates the arms as well (AUROC 1.00). The authors release 357 runs, an audit log of how the withdrawn results were produced, six disclosed implementation defects and a pre-registered text-only analysis.

Critical strengths

S1. The methodological point is correct, is underappreciated in this literature, and is operationalised rather than asserted.

Appendix B, Table 4 is careful about priority: each control has precedents, and Nghiem et al. (2026) apply analogues of all three to a single monitor. What the paper adds is the benign arm matched to the exploit's onset time and payout schedule, and a behaviour-matched planted signal run through the same controls to establish what they can accept. Both additions are well chosen, and the paper then specifies the controls concretely enough to run and runs them on its own results, retracting four of its own reported findings and qualifying a fifth. This is the costly form of the argument and is more persuasive than a critique aimed at other groups' work would have been. The audit log in Appendix A — which records a mis-specified one-sided test, an exploit counter saturating against its own threshold, and a result an earlier draft had read as LLM-scale corroboration of the toy finding — gives the reader the provenance needed to judge the claims. This matters to my assessment because it establishes that the negative findings are not an artifact of a carelessly built pipeline.

S2. The planted positive control anticipates the standard objection to a null result.

A paper reporting that no statistic survives its controls faces an immediate question: are the controls too strict to pass anything? The planted control answers it empirically. The planted-versus-benign AUROC rises with dose from 0.57–0.63 at rho = 0.1 to 0.81–0.97 at rho = 1, and a one-sided rank test separates the arms from rho = 0.5 upwards in three settings. Reporting a detection suite's power rather than only its verdict is uncommon in this subarea and should be adopted more widely. My reservation concerns the geometry of the planted signal (W2), not the principle.

Critical weaknesses

W1. The LLM-scale conclusion is largely determined by how hacking was induced, so it does not support the frame the paper sets around it.

At LLM scale, hacking is produced by supervised fine-tuning on the School of Reward Hacks dataset, whose hacking completions a bag-of-words classifier separates from controls at 0.96–0.98 AUROC. Once the two arms differ that sharply in surface form, the central findings follow closely from the construction: a detector reading activations elicited by lexically distinctive text will largely be reading that text, whichever model performs the reading. The scorer swap is well designed for diagnosing exactly this, but on these stimuli its verdict is close to predetermined. Three aspects compound the concern.

The configuration that motivates representation-based monitoring is the one left untested. Section 6 identifies it precisely: RL-discovered hacking whose text looks benign is "the case motivating representation-based monitors" and is untested here, and under that condition "a swapped reader at chance leaves any own-model separation to model state." The paper's own framing therefore locates the decisive experiment outside its evidence.

Scale is untested in the same direction. The toy policies have a few thousand parameters and the language models span 1.5B–8B, whereas internal monitoring is proposed as a safeguard at frontier scale. I accept the authors' account of why a matched benign arm has no cheap recipe at 70B+ scale; the difficulty does not change what the present experiments license.

The scoping of the claim is uneven across the paper. The TL;DR is careful ("every representation-based reward-hacking result of ours that had replicated"), and the body is scrupulous throughout. The title, however, names representation-based reward-hacking detection without qualification, and the abstract opens on how such detectors are validated as a class. A reader arriving through either will take the demonstrations to speak to the general case, when they rest on a lexically marked fine-tuning organism and nine toy environments.

This is the pivot of my recommendation. It cannot be addressed by revising the text; it requires at least one setting in which the hacking and control arms are not separable from the text alone.

W2. Both ends of the comparison rest on stand-ins, which bounds the negative result further than the framing admits.

The paper's force depends on two surrogates: the detectors it tests against, and the signal it plants to show the suite can accept something. Each is weaker than the role it is asked to play.

No original implementation of any prior detector is run. Appendix Q substitutes a supervised contrastive direction fit in the paper's own four derived statistics, and Appendix S applies the control suite to it. The paper is explicit that the object under test is not Wu & Tang's concept direction in raw activation space, Wilhelm et al.'s classifier on token activations, or Wang et al.'s gradient geometry, and that none of those pipelines is run. I take the disclosure at face value; it nonetheless leaves the field-level reading of the result untested. Appendix S also concedes an asymmetry in the baseline's favour — its direction is fit per environment on held-out labelled hacking seeds, an advantage the label-free original does not have — which makes the comparison informative about the control's cheapness but not about relative detector quality.

The comparison has too little power to order the detectors in any case. Appendix S reports a smallest attainable p of 0.031 at n = 6 under the exact sign-flip null, with no pairwise difference significant (p = 0.938, 0.812, 0.562), and the authors accordingly claim no ordering. That is the honest reading, but it means the appendix cannot settle the question it is raised to address.

The planted positive control is aligned with the statistic it is used to validate. It appends a synthetic unit whose stream is the projection on the layer's dominant direction, scaled to variance rho times the layer's total, while anisotropy is lambda_max over the sum of lambda. The planted signal therefore lives in precisely the coordinate the detector reads, so the control establishes that the suite accepts a model-state effect of a favourable geometry, not that genuine reward hacking would leave a comparably detectable one. The authors say as much — the positive control "tests mainly the benign-onset control" — but the consequence is that the demonstrated power is specific to aligned signals, and the paper's concluding recommendation that detection claims report "a planted positive control of the size it claims to detect" inherits the same limitation.

Taken together, the two stand-ins bound the result to the following: four statistics derived from one activation covariance spectrum, applied to these constructions, did not survive these controls. That is a defensible and useful finding, and narrower than the title and abstract convey.

Questions for authors

Text separability. Given that a bag-of-words reader separates the fine-tuning data's two arms at 0.96–0.98 AUROC, how do you distinguish the LLM-scale conclusion from a property of the stimulus? Concretely, what does the scorer swap show where the text is not separable — a MacDiarmid-style fixed-trigger design, in which the scored input is identical across arms by construction, or the least lexically marked subset of School of Reward Hacks? A single such setting would considerably strengthen the claim.

Geometry of the planted signal. Does the suite accept a planted signal that is not aligned with the top eigendirection — an isotropic perturbation, or one confined to an orthogonal subspace? If power falls substantially under misalignment, the planted control bounds the benign-onset control only for signals of a particular geometry, and that limitation belongs alongside the dose-response curve.

Minimum detectable effect. Under the unit of analysis of Appendix S (n = 6, smallest attainable p = 0.031), what difference between two detectors could the design have resolved? A stated minimum detectable effect would let readers calibrate "we claim no ordering", which at present cannot be distinguished from a design with too little power to order anything.

The placebo arm's behaviour. In Table 2 the regenerated clean arm reaches 8/9 and 9/9 while the hacking arm reaches 7/9 and 8/9. What drives the within-run sign statistic on runs where there is nothing to detect, and does the clean arm's higher rate have a mechanism, or is it within the variability of a nine-environment binomial?

Carryover above unity. Scorer-swap carryover is reported at 97–110%; above 100% the untuned base model reproduces more than the fine-tuned model's own rise. Is there a mechanism, or is this noise at five runs per arm? Appendix G attributes 15–33% of the gap on the re-collection to length; does the excess above 100% survive that adjustment?

Intended scope. Section 4.2 notes that effective rank and anisotropy are functions of the same spectrum. How far do you intend the conclusion to transfer to detectors reading different objects — probes on SAE latents, gradient fingerprints, or reward-model latent spaces? A sentence delimiting this in the abstract would bring the framing into line with Appendix S's own statement.

Suggestions for future work (not requests for the rebuttal)

One hacking organism discovered by reinforcement learning rather than installed by fine-tuning, which would place the controls in the regime that motivates representation-based monitoring.

A planted positive control whose geometry is chosen adversarially with respect to the statistic under test, reported as a power curve over alignment as well as over dose.

At least one original detector implementation carried through the suite, which would convert Appendix S from a stand-in result into a statement about a published method.

Variance over training seeds for the toy suite, and more than five runs per arm at LLM scale, since several conclusions currently rest on increments the paper itself notes cannot be bounded at that sample size.

Reviewer_q4UPOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies the effectiveness of reward-hacking probes. It is hard to understand exactly what the takeaways/implications are due to the extremely defensive and convoluted writing style, but as far as I understand, the results show that the reward hacking probes that they trained do not generalize very well, and are typically picking up on indicators such as word choice, or idiosyncracies of the particular environment.

Critical strengths
  • The paper is comprehensive about what it did, and describes in great detail what was carried out. Many careful statistical measures are used to ensure that the results which are presented are statistically significant.
  • The area is important. Reward hacking is an increasingly-relevant topic as we are continuing to train powerful LLMs with RL and delegating many important tasks to them.
Critical weaknesses
  • The paper is extremely hard to follow and comprehend what the takeaways are. Even the introduction mostly just obfuscates what's going on instead of actually explaining. At a high level, the work seems to examine several reward hacking trials. At each trial, it studies the activation anistotropy, a measure of dispersion of activations. I am unsure how this connects back to the motivation of trying to detect reward hacking (RH) from the activations--for this motivation I would expect some sort of study of a common policy, such as an LLM, on several different RH envs, identifying the common signal in the activations, and exploring generalization to new envs. Instead the work dives into this anisotropy statistic with extremely little motivation.
  • I believe the authors have falsely filled out their 'AI use statement', so the paper should be rejected for ethical concerns in any case.
Questions for authors
  • Can you clearly state the takeaways from this work?
  • Can you clearly state how the takeaways would transfer to cases of real-world RL reward hacking, either for LLMs or real robotics tasks?
  • Can you explain why your paper contains a very high number of signs of being written by AI, while your AI use statement claims you did not use AI to write any of the paper?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.