- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
Central studied problem: activation monitors might rely on spurious correlations. Authors compare different intervention-based tests that should help tell apart whether an activation monitor relies on a spurious feature or not. The three studied tests either don't make sense or are based on completely unrealistic assumptions. The central contribution is the evaluation of these three intervention tests for activation monitors. The only test that is shown to work well by the authors assumes information that is never available in practice, and is only available because of their synthetic data.
Critical strengths
None
Critical weaknesses
This paper is unreadable. It has undefined and not introduced terms. It completely lacks coherency and semblance of a motivation or a story. It is almost impossible to understand, what even is being monitored. It gives off all signs of being AI-generated throughout, including both experimental design, execution and paper writing. Most experiments are unfounded, unnecessary or do not make sense.
I tried making sense of the authors contributions, and found that they are severely lacking:
- The authors do not establish that "held-out accuracy cannot show whether a monitor reads the behaviour itself or a correlated feature" and just state it plainly. Without any empirical or conceptual proof - the paper completely lacks motivation. Relatedly, there is no motivation for studying specifically intervention-based tests, rather, for example, testing OOD generalization.
- "Floor test" - introducing noise into the activations does not make any sense as a way of establishing whether an activation monitor is confounded. No sense at all.
- "Matched erasure" results are trivial. If you remove the information that the activation monitor reads from the activations - it will perform much worse. There was no sense at all in running this experiments.
- The test that allegedly should help understand if a monitor relies on a spurious correlation - attribute reversal - only works in the synthetic data the paper conveniently introduced. You can't have a pure, unconfounded representation of the spurious feature, if you assume that in your data it is always appearing together with the monitoring behavior. Again, this just does not make sense.
Questions for authors
None
3 comments
Changes to the submitted analyses
While preparing the rebuttal, we audited the submitted analyses. Four findings change what the submission can claim:
- The 1,117-probe panel used selection tied to reversal. We withdraw its reversal and erasure classification counts and reader–rider rankings.
- Our audit of all 1,599 candidates in the original constructed panels found source-question overlap between fitting and evaluation. Results not otherwise withdrawn will be treated as descriptive, and the associated generalization claims will be removed.
- We exclude 148 cells with incorrect text–activation pairings. We do not use affected submitted results as evidence; corrected results are not yet available.
- The universal site maximum in Appendix B.8 is incorrect. We withdraw it, the site-normalized results, and conclusions that depend on that maximum.
The new internally trained probes use separate source-question partitions for probe fitting, eraser fitting, and evaluation. Threshold calibration uses other architectures but shares questions; it is not fully question-held-out. The new experiments do not rely on the withdrawn maximum. Their scripts and records are listed under "Analyses added after submission" in the README of the anonymous repository: https://anonymous.4open.science/r/iclr_paper1-4DEF/
AI use
For the submission, the authors determined the research questions, experimental design, analyses, and scientific conclusions. We used Claude Opus for prose editing, appendix restructuring, LaTeX formatting, and boilerplate code. We did not list appendix restructuring in the submitted disclosure, and we will add it.
Rebuttal response
The author's comments do not remotely address my concerns. I'm retaining my score. P.S. Pangram flags 100% of these two comment as AI generated.
Thank you for the review. We did not explain the question or the experiments clearly enough. The question is whether a deception probe uses deception information or an associated feature, such as answer length. We will use this example to introduce the paper.
The new experiments use Liars' Bench instructed deception. In that setting, a positive example is an answer given under a lie instruction and judged false by the benchmark.
Why held-out accuracy is not enough
We trained a probe on response-length labels, with deception balanced in its training data. On test records where length and deception labels agreed about 90% of the time, the probe achieved deception AUROC 0.90. When that association was reversed, its AUROC fell to 0.08.
This example shows that high held-out deception AUROC can occur for a probe trained on a non-target attribute, and can fail when the association changes. It does not prove that the probe contains no deception information. It shows why accuracy alone is insufficient to establish reliance.
We agree that out-of-distribution evaluation is relevant and added a cross-dataset baseline. With the same calibration target, transfer accepted 18/90 length-trained evaluations and erasure accepted 10/90. Both accepted all 90 deception-trained evaluations. The paired interval included zero, so this comparison did not establish a difference between the methods. The 10/90 erasure result uses the original eraser fitted at the observed attribute–label agreement. The 82/90 result below uses an eraser fitted at 90% agreement.
Balanced perturbation
For conditional mean-zero perturbations, the mean-score-change statistic loses the linear response. It therefore does not establish target reliance. The cancellation follows from a Taylor expansion, and we gave it too much emphasis. We will shorten this section rather than present it as a main contribution.
What the erasure experiment tests
We agree that removing information used by a probe can reduce its score. The question is whether that reduction tells us which information the probe uses.
The erased direction is fitted from behaviour-labelled data. If length is correlated with those labels, the fitted direction may contain length information too. We tested this while keeping the probes and evaluation records fixed.
We used 60 probes and changed only the eraser-fitting records, keeping their number and class balance unchanged. As length–deception agreement increased from 50% to 90%, length-trained evaluations above a frozen threshold increased from 0/90 to 82/90. These are 30 probes evaluated on three test sets, not 90 independent probes.
The result shows that a probe can receive a different diagnostic decision because the eraser-fitting data changed. A score reduction alone therefore does not establish that the probe uses deception.
Controlled data construction and reversal
We did not generate new examples for these experiments. We selected subsets of existing Liars' Bench, Among Us, and Anthropic sycophancy records to vary the attribute–label association. For sycophancy, our pipeline also appends the dataset-key answer letter.
These are controlled constructions. We do not claim that the same associations occur at the same rates in deployment.
Reversal requires records where the candidate attribute and behaviour label disagree. If they never disagree in the available records, we cannot apply the test and report no verdict. It is not a solution for unknown confounds.
We will organize the revised paper around the fixed-probe experiment and explain each dataset's label before presenting its results.
The reply below lists corrections to the submitted analyses and the limits of the new evaluation splits.