W1. The LLM-scale conclusion is largely determined by how hacking was induced, so it does not support the frame the paper sets around it.
At LLM scale, hacking is produced by supervised fine-tuning on the School of Reward Hacks dataset, whose hacking completions a bag-of-words classifier separates from controls at 0.96–0.98 AUROC. Once the two arms differ that sharply in surface form, the central findings follow closely from the construction: a detector reading activations elicited by lexically distinctive text will largely be reading that text, whichever model performs the reading. The scorer swap is well designed for diagnosing exactly this, but on these stimuli its verdict is close to predetermined. Three aspects compound the concern.
The configuration that motivates representation-based monitoring is the one left untested. Section 6 identifies it precisely: RL-discovered hacking whose text looks benign is "the case motivating representation-based monitors" and is untested here, and under that condition "a swapped reader at chance leaves any own-model separation to model state." The paper's own framing therefore locates the decisive experiment outside its evidence.
Scale is untested in the same direction. The toy policies have a few thousand parameters and the language models span 1.5B–8B, whereas internal monitoring is proposed as a safeguard at frontier scale. I accept the authors' account of why a matched benign arm has no cheap recipe at 70B+ scale; the difficulty does not change what the present experiments license.
The scoping of the claim is uneven across the paper. The TL;DR is careful ("every representation-based reward-hacking result of ours that had replicated"), and the body is scrupulous throughout. The title, however, names representation-based reward-hacking detection without qualification, and the abstract opens on how such detectors are validated as a class. A reader arriving through either will take the demonstrations to speak to the general case, when they rest on a lexically marked fine-tuning organism and nine toy environments.
This is the pivot of my recommendation. It cannot be addressed by revising the text; it requires at least one setting in which the hacking and control arms are not separable from the text alone.
W2. Both ends of the comparison rest on stand-ins, which bounds the negative result further than the framing admits.
The paper's force depends on two surrogates: the detectors it tests against, and the signal it plants to show the suite can accept something. Each is weaker than the role it is asked to play.
No original implementation of any prior detector is run. Appendix Q substitutes a supervised contrastive direction fit in the paper's own four derived statistics, and Appendix S applies the control suite to it. The paper is explicit that the object under test is not Wu & Tang's concept direction in raw activation space, Wilhelm et al.'s classifier on token activations, or Wang et al.'s gradient geometry, and that none of those pipelines is run. I take the disclosure at face value; it nonetheless leaves the field-level reading of the result untested. Appendix S also concedes an asymmetry in the baseline's favour — its direction is fit per environment on held-out labelled hacking seeds, an advantage the label-free original does not have — which makes the comparison informative about the control's cheapness but not about relative detector quality.
The comparison has too little power to order the detectors in any case. Appendix S reports a smallest attainable p of 0.031 at n = 6 under the exact sign-flip null, with no pairwise difference significant (p = 0.938, 0.812, 0.562), and the authors accordingly claim no ordering. That is the honest reading, but it means the appendix cannot settle the question it is raised to address.
The planted positive control is aligned with the statistic it is used to validate. It appends a synthetic unit whose stream is the projection on the layer's dominant direction, scaled to variance rho times the layer's total, while anisotropy is lambda_max over the sum of lambda. The planted signal therefore lives in precisely the coordinate the detector reads, so the control establishes that the suite accepts a model-state effect of a favourable geometry, not that genuine reward hacking would leave a comparably detectable one. The authors say as much — the positive control "tests mainly the benign-onset control" — but the consequence is that the demonstrated power is specific to aligned signals, and the paper's concluding recommendation that detection claims report "a planted positive control of the size it claims to detect" inherits the same limitation.
Taken together, the two stand-ins bound the result to the following: four statistics derived from one activation covariance spectrum, applied to these constructions, did not survive these controls. That is a defensible and useful finding, and narrower than the title and abstract convey.