acceptodds
Under review as a conference paper at ICLR 2027

Balanced Perturbation and Subspace Erasure Do Not Identify What an Activation Monitor Reads

Abstract

Activation monitors score behaviours such as deception from language-model hidden states, but held-out accuracy cannot show whether a monitor reads the behaviour itself or a correlated feature. We analyse two intervention diagnostics for this distinction. Under a balanced perturbation with conditional mean zero, positive and negative first-order score responses cancel in expectation. Its mean score change therefore reflects local curvature rather than target reliance. For linear monitors under elliptical perturbations, variance, mean absolute change, and flip rate are also non-identifying. Matched erasure compares the score loss after removing a fitted class-separating direction with the loss from an equal-size control edit. A monitor that reads a correlated attribute can share the same alignment with this direction as a target reader. In panels with sealed construction labels, matched erasure ranks readers above confound-riders, but no threshold separates them, including after site-level normalization. A margin calibrated on one corpus passes of riders in a corpus-matched follow-up; all fail when their attribute is reversed while the behaviour is held fixed. Two released deception probes pass in only of benchmark cells. Reversing a named candidate attribute classifies of monitors correctly on held-out records. We recommend reporting this test with its full operating curve rather than treating one intervention result as evidence of target reliance.

Primary area:
interpretability and explainable AI
Keywords:
activation monitoring, mechanistic interpretability, causal interpretability, representation interventions, concept erasure, AI safety, probing, reliability evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DLHQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Central studied problem: activation monitors might rely on spurious correlations. Authors compare different intervention-based tests that should help tell apart whether an activation monitor relies on a spurious feature or not. The three studied tests either don't make sense or are based on completely unrealistic assumptions. The central contribution is the evaluation of these three intervention tests for activation monitors. The only test that is shown to work well by the authors assumes information that is never available in practice, and is only available because of their synthetic data.

Critical strengths

None

Critical weaknesses

This paper is unreadable. It has undefined and not introduced terms. It completely lacks coherency and semblance of a motivation or a story. It is almost impossible to understand, what even is being monitored. It gives off all signs of being AI-generated throughout, including both experimental design, execution and paper writing. Most experiments are unfounded, unnecessary or do not make sense.

I tried making sense of the authors contributions, and found that they are severely lacking:

  1. The authors do not establish that "held-out accuracy cannot show whether a monitor reads the behaviour itself or a correlated feature" and just state it plainly. Without any empirical or conceptual proof - the paper completely lacks motivation. Relatedly, there is no motivation for studying specifically intervention-based tests, rather, for example, testing OOD generalization.
  2. "Floor test" - introducing noise into the activations does not make any sense as a way of establishing whether an activation monitor is confounded. No sense at all.
  3. "Matched erasure" results are trivial. If you remove the information that the activation monitor reads from the activations - it will perform much worse. There was no sense at all in running this experiments.
  4. The test that allegedly should help understand if a monitor relies on a spurious correlation - attribute reversal - only works in the synthetic data the paper conveniently introduced. You can't have a pure, unconfounded representation of the spurious feature, if you assume that in your data it is always appearing together with the monitoring behavior. Again, this just does not make sense.
Questions for authors

None

3 comments

Thank you for the review. We did not explain the question or the experiments clearly enough. The question is whether a deception probe uses deception information or an associated feature, such as answer length. We will use this example to introduce the paper.

The new experiments use Liars' Bench instructed deception. In that setting, a positive example is an answer given under a lie instruction and judged false by the benchmark.

Why held-out accuracy is not enough

We trained a probe on response-length labels, with deception balanced in its training data. On test records where length and deception labels agreed about 90% of the time, the probe achieved deception AUROC 0.90. When that association was reversed, its AUROC fell to 0.08.

This example shows that high held-out deception AUROC can occur for a probe trained on a non-target attribute, and can fail when the association changes. It does not prove that the probe contains no deception information. It shows why accuracy alone is insufficient to establish reliance.

We agree that out-of-distribution evaluation is relevant and added a cross-dataset baseline. With the same calibration target, transfer accepted 18/90 length-trained evaluations and erasure accepted 10/90. Both accepted all 90 deception-trained evaluations. The paired interval included zero, so this comparison did not establish a difference between the methods. The 10/90 erasure result uses the original eraser fitted at the observed attribute–label agreement. The 82/90 result below uses an eraser fitted at 90% agreement.

Balanced perturbation

For conditional mean-zero perturbations, the mean-score-change statistic loses the linear response. It therefore does not establish target reliance. The cancellation follows from a Taylor expansion, and we gave it too much emphasis. We will shorten this section rather than present it as a main contribution.

What the erasure experiment tests

We agree that removing information used by a probe can reduce its score. The question is whether that reduction tells us which information the probe uses.

The erased direction is fitted from behaviour-labelled data. If length is correlated with those labels, the fitted direction may contain length information too. We tested this while keeping the probes and evaluation records fixed.

We used 60 probes and changed only the eraser-fitting records, keeping their number and class balance unchanged. As length–deception agreement increased from 50% to 90%, length-trained evaluations above a frozen threshold increased from 0/90 to 82/90. These are 30 probes evaluated on three test sets, not 90 independent probes.

The result shows that a probe can receive a different diagnostic decision because the eraser-fitting data changed. A score reduction alone therefore does not establish that the probe uses deception.

Controlled data construction and reversal

We did not generate new examples for these experiments. We selected subsets of existing Liars' Bench, Among Us, and Anthropic sycophancy records to vary the attribute–label association. For sycophancy, our pipeline also appends the dataset-key answer letter.

These are controlled constructions. We do not claim that the same associations occur at the same rates in deployment.

Reversal requires records where the candidate attribute and behaviour label disagree. If they never disagree in the available records, we cannot apply the test and report no verdict. It is not a solution for unknown confounds.

We will organize the revised paper around the fixed-probe experiment and explain each dataset's label before presenting its results.

The reply below lists corrections to the submitted analyses and the limits of the new evaluation splits.

Changes to the submitted analyses

While preparing the rebuttal, we audited the submitted analyses. Four findings change what the submission can claim:

  1. The 1,117-probe panel used selection tied to reversal. We withdraw its reversal and erasure classification counts and reader–rider rankings.
  2. Our audit of all 1,599 candidates in the original constructed panels found source-question overlap between fitting and evaluation. Results not otherwise withdrawn will be treated as descriptive, and the associated generalization claims will be removed.
  3. We exclude 148 cells with incorrect text–activation pairings. We do not use affected submitted results as evidence; corrected results are not yet available.
  4. The universal site maximum in Appendix B.8 is incorrect. We withdraw it, the site-normalized results, and conclusions that depend on that maximum.

The new internally trained probes use separate source-question partitions for probe fitting, eraser fitting, and evaluation. Threshold calibration uses other architectures but shares questions; it is not fully question-held-out. The new experiments do not rely on the withdrawn maximum. Their scripts and records are listed under "Analyses added after submission" in the README of the anonymous repository: https://anonymous.4open.science/r/iclr_paper1-4DEF/

AI use

For the submission, the authors determined the research questions, experimental design, analyses, and scientific conclusions. We used Claude Opus for prose editing, appendix restructuring, LaTeX formatting, and boilerplate code. We did not list appendix restructuring in the submitted disclosure, and we will add it.

Reviewer_DLHQOpenReview ↗

Rebuttal response

The author's comments do not remotely address my concerns. I'm retaining my score. P.S. Pangram flags 100% of these two comment as AI generated.

Reviewer_Rs8sOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper first shows two representation-level interventions to diagnose whether a probe is relying on spurious features do not work: 1) zero-mean perturbation and then observing probe scores only reflects local curvature (which is zero for linear probes), 2) erasing probe direction, refitting the probe, and observing performance drop also do not separate spurious from causal features. The authors show that for linear probes, the performance drop depends only on the probes' cosine similarity with the erased direction in the whitened space. The paper then proposes an attribute reversal test where probes are evaluated on test data where a predefined possible confound is anticorrelated with the label.

Critical strengths
  • The idea behind the attribute reversal test is reasonable, in which one selects test data to break possible confounds.
  • The empirical evaluation is seemingly large, although the exact empirical settings are not clearly described
Critical weaknesses
  • This paper is extremely difficult to read and borderline unreadable. It doesn’t seem like this paper is written with human readers in mind.
  • The floor test and matched erasure test seem fundamentally misguided. There is no a priori reason to expect a random perturbation to reveal whether a probe relies on spurious features. If so, the authors should state that hypothesis early and clearly. As written, the paper sets up the perturbation only to show that it fails, which is unsurprising. Added to the fact that the theoretical results are rudimentary algebra, the random perturbation results do not deserve to be in an ICLR paper. There is likewise no reason to expect erasure to separate confounds from causal features. The extensive empirical analysis quantifies failures that should be expected from the beginning.
  • Attribute reversal as a practical method is severely limited by requiring 1) knowing and labeling the potential confounds and 2) the existence of data where the potential confounds are broken
Questions for authors
  • What is the motivation and hypothesis behind the perturbation tests and the erasure tests? Why do the authors expect it to work at all?
  • The most standard way of testing for spurious correlation in white-box probing is cross-dataset generalization and causal steering. The paper should at least include cross-dataset to table 2 as a baseline.
  • Please specify the source of all datasets. What is the "pooled sycophancy" corpus, where 167/168 failed probes are from?
2 comments

Thank you for the detailed review. The paper does not explain the setup clearly enough. We will rewrite the introduction and setup around one example: a probe trained to detect deception may instead use answer length.

Clarification about erasure

We do not retrain the probe after erasure. Within each evaluation, we fit the probe before erasure, then use that same probe to score the original and edited activations.

Q1 - Why study these tests?

Matched erasure tests the following hypothesis: if a probe uses the behaviour of interest, removing a direction fitted to separate the behaviour labels should reduce its score more than an equally large control edit.

Removal-based checks have been used to assess probes; for example, Boxo et al. use INLP as a correlated-feature check. But the fitted direction can also contain information about attributes correlated with the label. Our question is whether the matched control is enough to distinguish these cases.

For balanced perturbation, we asked whether behaviour-trained and attribute-trained probes would differ in their mean response to small perturbations. For the mean score change, conditional mean zero cancels the linear response. This is an elementary result, and we will shorten that part of the paper.

Removal-based tests also have known limitations, including those studied by Kumar, Tan and Sharma (2022) for frozen classifiers. We will cite that work and make the narrower purpose of our experiments clear.

Our new experiment holds the probes and test records fixed and changes only the data used to fit the eraser:

  • We used 60 probes: 30 trained on deception and 30 on prompt or response length. Each was evaluated on three test sets.
  • Changing length–deception agreement in the eraser-fitting records from 50% to 90% moved length-trained evaluations from 0/90 to 82/90 above a frozen threshold.
  • For two released deception probes, balancing prompt length in the eraser-fitting data reduced their erasure scores from 0.635 to 0.435 and from 0.693 to 0.475.

The released-probe result shows that the diagnostic is sensitive to the eraser-fitting data. It does not show that those probes rely on prompt length.

W3 - Limits of attribute reversal

We agree with both limitations. Reversal requires a named candidate attribute and records where that attribute disagrees with the behaviour label. If those records are unavailable, the test gives no verdict. It also cannot rule out attributes we have not tested.

Q2 - Cross-dataset baseline

We added cross-dataset evaluation. Probes trained on instructed deception were applied unchanged to the convincing-game dataset, which contains 750 records from 12 distinct questions.

For transfer, acceptance means that the probe's AUROC exceeds a calibrated threshold. Both methods' thresholds were chosen on the other two architectures to accept at most 10% of length-trained evaluations in the calibration data. We then froze the thresholds and applied them to the evaluated architecture.

MethodDeception-trained acceptedLength-trained accepted
Cross-dataset transfer90/9018/90
Matched erasure90/9010/90

The matched-erasure row uses the baseline eraser from the new experiment, fitted on records at their observed attribute–label agreement. The 82/90 result above uses erasers fitted at 90% agreement.

For length-trained acceptance, the paired 95% interval for erasure minus transfer is −12.2 to +3.3 percentage points. We used 2,000 bootstrap draws, resampling source and destination questions. The interval includes zero, so this experiment does not establish a difference between the methods.

We will add the baseline to the comparison table. We did not test steering effects on generated behaviour. The reply below gives the dataset sources and identifies corrections to the submitted analyses

Q3 - Dataset sources

The datasets use different positive labels. An Impostor role, a deception-inducing prompt, and a judged false answer are not interchangeable measures of deception. We will make these distinctions clear and add dataset revisions and preprocessing details to Appendix F.

DatasetSourceInput construction and positive label
Instructed deceptionLiars' Bench, Cadenza-Labs/liars-benchSubsets selected by attribute–label agreement. Positive: an answer given under a lie instruction and judged false by the benchmark.
Convincing gameLiars' BenchAs the transfer target in the new experiments, used without the attribute–label subset construction; the original panels also used constructed subsets. Positive: an answer judged to contradict the model's belief.
Pooled sycophancyAnthropic model-written evaluations: NLP survey, PhilPapers, political typologyOur pipeline appends an answer letter from the dataset key. Positive: the letter matches the persona's view.
Among Us7vik/AmongUs transcriptsSubsets selected by attribute–label agreement. Positive: the speaker's assigned role is Impostor.
MASKcais/MASKThe model answers and self-judges. Positive: it judges that its answer contradicts the true statement.
DeceptionBenchPKU-Alignment/DeceptionBenchPrompt inputs. Positive: a deception-inducing prompt is present.
RefusalAdvBench and JailbreakBench harmful promptsPrompt activations, with labels assigned from separately sampled responses. Positive: the model refuses in most samples.
AuditBench organismsThird-party Qwen3-14B buildPrefill attack, with concealment judged by Claude Sonnet 4.6.

We evaluated activations from Llama, Gemma, Mistral, and Qwen models for the original instructed-deception, convincing-game, sycophancy, and Among Us experiments. MASK and DeceptionBench used Llama-3.1-8B, Gemma-2-9B, Mistral-7B, and Llama-3.3-70B. Refusal used Llama-3.1-8B; the AuditBench organism experiments used Qwen3-14B. The new experiments use Llama-3.1-8B, Gemma-2-9B, and Llama-3.3-70B.

What "pooled sycophancy" means

This is the Perez et al. persona set. Each record contains a persona and a survey question. Our pipeline appends an answer letter from the dataset key; no model generates that answer, and no judge labels it. The Ethics Statement incorrectly calls this corpus "judge-elicited," and we will correct that description.

This corpus supplied 167 of the 168 attribute-trained probes in the preregistered panel. That imbalance prevents a clean comparison with behaviour-trained probes from other corpora. The new controlled experiment uses the same corpus for both groups.

Changes to the submitted analyses

While preparing the rebuttal, we audited the submitted analyses. Four findings change what the submission can claim:

  1. The 1,117-probe panel used selection tied to reversal. We withdraw its reversal and erasure classification counts and reader–rider rankings.
  2. Our audit of all 1,599 candidates in the original constructed panels found source-question overlap between fitting and evaluation. Results not otherwise withdrawn will be treated as descriptive, and the associated generalization claims will be removed.
  3. We exclude 148 cells with incorrect text–activation pairings. We do not use affected submitted results as evidence; corrected results are not yet available.
  4. The universal site maximum in Appendix B.8 is incorrect. We withdraw it, the site-normalized results, and conclusions that depend on that maximum.

The new internally trained probes use separate source-question partitions for probe fitting, eraser fitting, and evaluation. Threshold calibration uses other architectures but shares questions; it is not fully question-held-out. The new experiments do not rely on the withdrawn maximum. Their scripts and records are listed under "Analyses added after submission" in the README of the anonymous repository: https://anonymous.4open.science/r/iclr_paper1-4DEF/

Reviewer_yLHXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper analyzes two interventions for determining whether an activation monitor relies on the target behavior: (i) a balanced perturbation with conditional mean zero, where they show positive and negative first-order responses cancel. So the mean score change just captures curvature. (ii) matched erasure which compares the score loss after removing a fitted class-separating direction with the loss from an equal-size control edit. There, they show a monitor that reads a correlated attribute have the same alignment with the erased direction but no fixed threshold separates them.

Finally, the paper systematically tests these identification failures, studies the effect of confound strength, and evaluates whether a decision threshold transfers across datasets and models.

Critical strengths

I would say the strength are (i) nice problem formulation (ii) clean (and super simple) theoretical results (iii) extenstive and systematci experiments: authors examine multiple monitor architectures, language models, datasets, and confound strengths.

Critical weaknesses

The balanced-perturbation result follows almost immediately from a Taylor expansion. Also, balanced perturbation is not a widely used method. So limited novelty and applicability of the first part.

For the second approach, the criterion for success may be too demanding. The y argue that no fixed threshold perfectly separates readers from riders.. How about other statistical tests?

The paper ultimately leaves open the central question of how to determine what information a monitor relies on when the confounds are unknown.

Questions for authors

No questions

1 comment

Thank you for the review. We agree that perfect separation should not be the only way to assess these tests.

W1 - Balanced perturbation

The cancellation follows directly from a Taylor expansion. We gave this result too much emphasis, especially because the test has limited use in existing work. We will shorten Section 3 and move its experiments to the appendix.

We also found that Gu et al., which we cited as a precedent, does not satisfy our conditional mean-zero assumption. We will correct the citation discussion and the corresponding count in Appendix A.

W2 - Other statistical tests

We tested a rule that requires erasure loss to be significantly greater than control loss, but does not require a minimum effect size. We kept the class-separation test and Holm correction unchanged.

We applied both rules to 1,423 evaluations from the original experiments. This set excludes the 148 cells with mismatched activations and does not use the reversal-based selection described below. Source questions overlap between fitting and evaluation, so these results are descriptive rather than an independent test of generalization.

Probe training labelOriginal ruleSignificance-only rule
Target behaviour213/339 accepted230/339 accepted
Non-target attribute206/1,084 accepted355/1,084 accepted

Removing the margin accepted more probes in both groups. It did not resolve the ambiguity in these evaluations.

We also ran a new experiment to test whether the result depends on the data used to fit the erased direction. We fixed 60 probes on Liars' Bench instructed deception: 30 trained on deception and 30 trained on prompt or response length. Each probe was evaluated on three test sets, giving 90 evaluations per group, not 90 independent probes.

We kept the probes and test records unchanged. We changed only the eraser-fitting records, holding their number and class balance fixed. We varied the percentage of those records whose binary length label agreed with the deception label. Thresholds were calibrated on the other architectures and frozen before this comparison.

Agreement in eraser-fitting recordsLength-trained evaluations above threshold
50%0/90
90%82/90

All 90 deception-trained evaluations remained above threshold across the tested agreement levels.

Thus, the same probe on the same test records can receive a different decision because the eraser was fitted on different data. This is the main finding of the revised experiments. We will report acceptance rates at different thresholds rather than use perfect separation as the sole criterion.

W3 - Unknown confounds

We do not solve the unknown-confound problem. We tried a text-based search for candidate attributes, but only 12 of 362 probes had enough suitable records for testing. We will report this as a limited feasibility check, not a general solution.

Changes to the submitted analyses

While preparing the rebuttal, we audited the submitted analyses. Four findings change what the submission can claim:

  1. The 1,117-probe panel used selection tied to reversal. We withdraw its reversal and erasure classification counts and reader–rider rankings.
  2. Our audit of all 1,599 candidates in the original constructed panels found source-question overlap between fitting and evaluation. Results not otherwise withdrawn will be treated as descriptive, and the associated generalization claims will be removed.
  3. We exclude 148 cells with incorrect text–activation pairings. We do not use affected submitted results as evidence; corrected results are not yet available.
  4. The universal site maximum in Appendix B.8 is incorrect. We withdraw it, the site-normalized results, and conclusions that depend on that maximum.

The new internally trained probes use separate source-question partitions for probe fitting, eraser fitting, and evaluation. Threshold calibration uses other architectures but shares questions; it is not fully question-held-out. The new experiments do not rely on the withdrawn maximum. Their scripts and records are listed under "Analyses added after submission" in the README of the anonymous repository: https://anonymous.4open.science/r/iclr_paper1-4DEF/

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.