acceptodds
Under review as a conference paper at ICLR 2027

When Do Deception Probes Transfer? Wrongness Controls and Source Specificity

Abstract

A deception probe may transfer better with more source labels because it learns deception-specific information, or because it becomes better at detecting ordinary wrong answers. We distinguish these explanations by inducing five wrong-answer policies while holding the task, model and item difficulty fixed. Each transfer curve is compared with a wrongness control that scores the same direction on honest errors. On ARC with Llama-3.1-8B, difference-of-means transfer from a hidden-objective source rises faster than the control. The supervised contrasts have the same sign but are unresolved after full refitting. A question-disjoint Gemma-2-9B replication resolves the scheming contrast for both estimators. On MMLU, by contrast, transfer and wrongness rise together. The diagnosis changes across task–model configurations, including configurations using the same benchmark. Even at larger label budgets, targets are usually detected best by source-matched directions. These results motivate reporting a powered wrongness control when interpreting probe-transfer curves.

Primary area:
interpretability and explainable AI
Keywords:
deception detection, linear probes, activation monitoring, sandbagging, probe transfer, label efficiency, scheming, AI safety, language models, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_gjz2OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether activation probes trained on one form of deliberately wrong answering generalize to others, and whether gains from more training data reflect more than ordinary error detection.

Experiments across models and tasks use honest mistakes as a control and show mixed results: some gains exceed this control, while others do not. Probes generally perform best on the behavior they were trained on.

Critical strengths

The comparison with honest errors addresses a confound in activation-probe evaluation: detecting deliberately wrong answers may simply reflect ordinary error detection. Evaluating the same probes under both conditions makes the transfer results more interpretable and provides a useful control for future studies.

Critical weaknesses

The main comparison does not sufficiently isolate deliberate deception.

The transfer targets and honest-error control differ in question selection and decoding. The within-question analysis also does not establish separation for scheming. This leaves alternative explanations for the central positive result unresolved.

Questions for authors
  • The transfer targets and honest-error control differ in question selection and decoding. What evidence shows that these differences do not explain the positive results, particularly given the inconclusive within-question comparison for scheming?

  • How do the uncertainty estimates account for shared questions and repeated splits? After accounting for these dependencies and the relevant multiple-comparison corrections, how strong is the support for the source-matched advantage and the claim that the Gemma replication supports both estimators?

2 comments

Thank you for identifying the matching and dependence issues. We have added a new comparison and revised the uncertainty analysis. The new analyses cited below are in the revised supplementary material (supplementary_tables.pdf in the zip file).

W1/Q1 - Question selection and decoding

The earlier analysis did not establish same-question separation for scheming.

The new experiment matches the question, wrong answer letter and decoding settings, and excludes probe-training questions. Its design was fixed before generation (supplementary Section S10). It includes only questions that produce both error types with a shared wrong letter, so its conclusions apply to that subset.

On 132 matched questions, the difference-of-means (DoM) probe ranks the deliberate error above the honest error with probability 0.644 [0.567, 0.711] at 10 labels per class and 0.762 [0.672, 0.836] at 150. The supervised values are 0.659 and 0.716. Across five budgets and two estimators, all ten pointwise (not simultaneous) 95% intervals are above 0.5 (supplementary Table S4c).

The statistic averages all error pairs within each question-letter cell, with ties counted as one half, then weights letters and questions equally.

We also checked response length. Restricting pairs to lengths within 20 tokens retains 72 scheming questions. DoM ranking at 150 labels is 0.713 [0.606, 0.816]. This is a post hoc sensitivity analysis, not a confirmatory test (supplementary Table S5c).

The prespecified signed scaling contrast is +0.032 per doubling [0.009, 0.054], BH q=0.006 and Holm-adjusted p=0.010. It compares the slopes for each error type against the same honest-correct responses (supplementary Table S3b).

When the honest-error control counts separation in either direction, the scaling contrast is inconclusive: +0.002 [-0.019, 0.025]. The prompts also remain different. In a separate replay test, re-encoding the same scheming transcripts under the honest prompt reduces hidden-objective DoM transfer from 0.591 to 0.511 (supplementary Table S9b).

The new experiment therefore supports separation between the matched response groups. It does not establish intent detection or independence from prompt context.

Q2 - Shared questions, splits and multiple comparisons

The submitted bootstrap did not keep shared questions together across pools and split seeds.

We now assign each question group one bootstrap weight wherever it occurs. We also use common source splits, group duplicate normalized question texts, and score every compared probe on the same held-out questions. Each full-refit replicate re-estimates the fitted quantities, including standardization and source directions. Results are averaged across split seeds; seeds and training draws are not treated as independent question samples.

The updated results reflect both the revised evaluation design and the corrected bootstrap.

The submitted 22/24 count described positive pairwise differences. Under the corrected full-refit analysis, 15/24 comparisons survive BH and 13/24 survive Holm. Against the strongest transferred direction, selected again in each resample, only 1/8 comparisons survives BH.

For the question-disjoint Gemma replication, supervised scheming has a nominal interval above zero but does not survive BH or Holm correction over six contrasts (BH q=0.067; Holm-adjusted p=0.133). We withdraw the multiplicity-corrected both-estimator replication claim.

DoM instructed and scheming survive both corrections, each with BH q=0.020 and Holm-adjusted p=0.040. These estimates use 300 question-bootstrap replicates with full refitting, so the tail probabilities remain coarse.

In the revision, Appendix A will describe the question-group bootstrap, and line 363 will report the corrected counts. We will also remove the both-estimator Gemma claim from the Abstract (lines 017-018), lines 303-304 and Appendix G.2.

Thanks, I've increased my score. Good luck.

The new matched experiment and revised uncertainty analysis substantially address my main concerns. I particularly appreciate the clear acknowledgment of the earlier limitations and the willingness to withdraw claims that are no longer supported. These improvements make me more positive about the paper.

The comparison on 132 matched questions provides stronger evidence that the probes distinguish the two response groups within the selected subset.

However, the interpretation remains limited: the scaling contrast becomes inconclusive when the honest-error control counts separation in either direction, and the replay experiment indicates substantial sensitivity to prompt context. I therefore view the new results as evidence of response-group separation, with the explanation for the transfer gains still partly unresolved. The distinction from intent detection should remain explicit.

The corrected analysis also supports a more qualified account of source specificity. Some pairwise advantages remain, but only one of eight comparisons against the strongest transferred direction survives BH correction, which provides limited support for a general source-matched advantage.

Similarly, the Gemma replication supports the difference-of-means results more clearly than the supervised results after correction. I appreciate the planned revisions to reflect these distinctions.

My remaining reservations concern the breadth and strength of the conclusions, while the rebuttal meaningfully improves the evidence and its presentation.

Reviewer_zeEzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

Summary This paper follows on idea that a model can act to behave as complying or competent, but actual response or behaviour was otherwise, resulting in wrong answer. To solve for this idea, author propose to train linear probe on internal activations and try to see if that learned direction transfers deceptive behaviour from one source to another. As part o author's approach, four sources were induced through prompts applied to same base model, and fifth one being a trigger-conditioned backdoor installed through fine-tuning, Activations were used from final generated answer token, taken after model completes the final Answer: line. For cross-source transfer protocol, authors estimate a direction from labelled responses of source A and score responses of source B with it. To classify honest and deceptive responses, authors used two estimators, one being supervised L2 regularized logistic-regression probe and other being difference between mean activation vector of two classes. For identifying scenarios where Label scaling is interpretable, authors illustrate that more labels is not equivalent to more variety, and ARC diagnosis does not generalize to every task. Authors also applied their protocol across eight task-model configs with target source specific evaluation resulting with some transfer curves improve beyond their wrongness controls, while others being limited by error sesnitivity or insufficient class support. Four source-specific structure, authors find that a probe is strongest on its training source, and a shared direction built from all sources improve transfer to an unseen source. Author also concludes that high AUROC does not make a probe reliable deployment monitor.

Contributions

  • Experimental framework and protocol for isolating deception from wrongness
  • Establish a diagnostic criteria to compare label-scaling slope of cross-source transfer against a control defining transfer gains as deception-specific only when transfer slope outpaces control slope.
  • Paper uses a 4x4 cross source-transfer matrices and layer-wise cosine similarity analyses to show that linear deception directions contains substantial source-specific components.

Validations

  • authors validate their label-scaling protocol across eight task-model configurations using both l2l_2-regularized logistic regression (supervised) and difference-of-means estimators, calculating confidence intervals via full probe refitting across 500 sets of training draws.
  • authors validate their baseline wrongness control by sampling at temperature to generate a powered pool of 262 honest errors across 136 questions, and also run a same-question lie-versus-mistake diagnostic ranking deliberate errors directly against honest errors on identical items.
  • authors validate against two honest-prompt confidence proxies—no-reasoning answer margin and agreement across repeated honest responses alongside confidence-matched class splits and comparisons against emitted token entropy and log-probability.
  • authors test whether probes rely on surface prompt features using preregistered neutral paraphrases, 10-bin response-length matching, and linear projection removing eight paraphrase-versus-honest directions.
  • authors validate that directions trained on model-selected prompt sources (hidden objective, incentive cue, and scheming) detect a weight-finetuned trigger backdoor above the fine-tuned model's wrongness control across two training seeds
Critical strengths
  1. When a model lies on a test, it also gives a wrong answer, making it hard to tell if a probe is learning to catch lies or just spotting mistakes. The authors solve this with a clean setup: they keep question difficulty fixed, train probes within the same prompt so the probe cannot just memorize the instructions, and compare results against the model's honest mistakes. This matters because it proves that rising transfer scores on benchmarks like MMLU and OpenBookQA only happen because the probe gets better at flagging ordinary errors. It also exposes major flaws in popular baselines: a standard honesty probe (Zou et al., 2023) drops from 0.995 accuracy on its own data to a coin flip (0.51) on these deception tasks, and a chain-of-thought text judge flags honest mistakes at 0.80 AUROC. Together with their five-step checklist, this gives the field a clear standard to stop confusing error detection with deception detection.
  2. Many safety studies train probes by explicitly telling a model to lie, assuming the probe will also catch a model that decides to scheme on its own. By comparing five ways of getting wrong answers on the exact same questions, this paper shows that assumption fails. A probe trained on direct instructions to lie works well on that exact task (0.83 AUROC), but drops to random guessing (0.53) when the model schemes on its own, with probes trained on the exact target behavior winning in 22 of 24 tests. The authors also show that at a 5% false-alarm rate, even the best probes catch less than a third of lies. This gives a clear, practical takeaway for AI safety: probes trained on forced lies miss self-driven scheming, and current probes should only be used to flag suspicious cases for human review rather than to block outputs automatically.
Critical weaknesses
  1. The authors claim their probe beats the error control across all three ARC targets, but Table 2 shows only 1 of 6 tests (scheming) survives statistical correction. Worse, the very last sentence of the paper admits this single surviving result depends on random sampling rather than greedy decoding. Finally, when tested on the exact same question, the probe separates lies from honest mistakes 100% of the time (48/48) on instructed lies, but drops to a coin flip (~50%) on scheming. This weakens the paper's core positive claim: if the probe cannot beat a coin flip on the same question, its cross-question gain on scheming looks like a sampling artifact rather than true lie detection.
  2. In Table 3, the authors claim 4 of 8 prompt-control tests "pass" (score below 0.58) because their AUROC scores are 0.35 to 0.41. However, 0.50 is random guessing, so 0.35 separates prompts just as strongly as 0.65 with the opposite sign. Measuring true distance from 0.50 puts all 8 tests in the "partial" failure band (0.58–0.65), matching a later result where simply rewording a prompt drops transfer by 0.19 (nearly to chance). Also, despite arguing that transfer scores require error controls, the main transfer table (Table 4) and shared-direction test (Section 5.2) omit an error-control column. The text reveals that Table 4's best transfers (0.79–0.80) come from probes that also flag honest mistakes at 0.63 and 0.71, meaning the main transfer matrix is still heavily confounded by ordinary errors.
Questions for authors
  1. Across questions in Table 2, difference-of-means on scheming is the only primary ARC contrast (1 of 6) that survives Benjamini-Hochberg correction. However, on the exact same question, the hidden-objective probe ranks a deliberate lie above an honest mistake 100% of the time (48/48) for instructed lies, but drops to chance (~50%) for scheming. In addition, the final sentence of the Limitations section notes that the difference-of-means scheming estimates depend on sampled decoding. Why does the probe fail to beat chance on the same question for scheming, and what is the scheming slope contrast when evaluated under greedy decoding?
  2. Table 3 classifies 4 of 8 prompt-presence cells as "passing" (<0.58< 0.58) based on raw AUROC scores of 0.35, 0.36, 0.40, and 0.41. Because 0.50 represents no separability, these scores reflect folded separability (max⁡(AUROC,1−AUROC)\max(\text{AUROC}, 1 - \text{AUROC})) of 0.59 to 0.65, placing all 8 raw cells into the "partial" band (0.58–0.75). Can you update Table 3 to evaluate folded deviations from 0.50 and clarify how this affects your claim that prompt presence does not drive transfer, especially given that casually restyling the hidden-objective prompt drops instructed transfer by 0.19 (Section 6)?
  3. While Section 5.1 notes in prose that the incentive-cue and scheming directions score ordinary honest mistakes at 0.71 and 0.63 AUROC, Table 4 omits a wrongness-control column and never states the full-budget control score for the hidden-objective direction. Similarly, Section 5.2 reports a 0.753 transfer AUROC for the leave-one-source-out rank-1 shared direction without reporting its wrongness-control score or its mathematical definition. What are the powered wrongness-control AUROCs for the hidden-objective row in Table 4 and the rank-1 shared direction in Section 5.2?
  4. On Gemma-2-9B, overlapping questions between the training and target pools inflated the wrongness control slope (+0.012+0.012 per doubling), whereas a preregistered question-disjoint split flattened the control and resolved the scheming contrast. Which of the eight task-model configurations in Section 4.3 use overlapping questions between source and target, and which are strictly question-disjoint?
3 comments

Thank you for the detailed questions. The new analyses cited below are in the revised supplementary material (supplementary_tables.pdf in the zip file).

W1/Q1 - Greedy decoding and the same-question result

The decoding caveat in our Limitations (Appendix D.4) concerns transfer from the scheming-trained direction. It does not concern the hidden-objective-trained scaling result in Table 2. The Table 2 targets already use greedy decoding.

With corrected question-group refitting, the original difference-of-means (DoM) scheming contrast is +0.0315 per doubling [0.0131, 0.0484]. It survives BH and Holm correction. DoM incentive survives BH (q=0.048), but not Holm. We will make this distinction explicit at lines 248-249.

The earlier same-question test had 20 scheming questions and did not establish separation. We ran a new experiment with the question, wrong letter and decoding settings matched. Its design was fixed before generation (supplementary Section S10). Because the matching procedure and eligible questions changed, it is not simply a larger rerun of the old test.

On 132 eligible questions, DoM ranks a deliberate error above an honest error with the same wrong letter with probability:

  • 0.644 [0.567, 0.711] at 10 labels per class;
  • 0.762 [0.672, 0.836] at 150 labels per class.

The statistic averages response pairs within each question-letter cell, then gives equal weight to letters within a question and to questions. Ties count as one half.

The prespecified signed scaling contrast is +0.032 per doubling [0.009, 0.054], BH q=0.006. A post hoc check that counts honest-error separation in either direction is inconclusive: +0.002 [-0.019, 0.025]. We therefore distinguish the positive matched ranking result from the unresolved scaling result under that alternative control.

W2/Q2 - Table 3 orientation

You are right that below-chance AUROC also indicates separation. We withdraw the claim that four raw cells pass.

Using max⁡(A,1−A)\max(A, 1-A) for raw AUROC AA, the scores are:

  • Supervised: 0.648, 0.631, 0.638 and 0.588.
  • DoM: 0.599, 0.603, 0.586 and 0.582.

In instructed, hidden-objective, incentive-cue and scheming order, all eight fall in the partial band, 0.58 to 0.75 (supplementary Table S11a).

We also withdraw the claim that these controls rule out prompt-context explanations. Removing the measured paraphrase directions changes transfer by at most 0.029, but restyling and transcript replay show that some transfer scores remain sensitive to context.

W2/Q3 - Wrongness controls

The powered control scores the same fitted direction on honest errors versus honest-correct responses, sampled under the honest prompt at temperature 1.0.

For the submitted Table 4 fits, the hidden-objective control was 0.527 for the supervised probe and 0.412 for the difference-of-means (DoM) probe. Under the corrected held-out design, the controls are:

Training sourceSupervisedDoM
Instructed0.4400.412
Hidden objective0.5160.409
Incentive cue0.6810.588
Scheming0.6290.516

Supplementary Table S2d gives the intervals. Control questions belonging to source-training groups are excluded within each split.

For reference, the submitted transfer matrix's strongest transferred scores, 0.79 and 0.80, came from incentive and scheming directions whose honest-error controls were 0.71 and 0.63. The submitted hidden-objective supervised row scored 0.68 to 0.74 off-diagonal, against a control of 0.53.

Shared direction

The submitted analysis used instructed, hidden-objective and scheming sources. We had not documented or prespecified the exclusion of incentive cue. The submitted direction scored honest errors at 0.544.

We also found a coordinate-conversion error. The corrected procedure is:

  1. Fit a logistic probe on source-standardized activations.
  2. Convert its coefficient wsw_s to raw coordinates, dividing by the feature standard deviations σs\sigma_s, and normalize: us=(ws/σs)/∥ws/σs∥2u_s = (w_s/\sigma_s)/\lVert w_s/\sigma_s\rVert_2.
  3. For target tt, stack the non-target directions as rows and take their leading right singular vector, v−tv_{-t}.
  4. Fit a one-dimensional logistic model on each source's training projections. For source ss and target tt, the score of an activation hh is αs,t v−t⊤h+cs,t\alpha_{s,t}\, v_{-t}^\top h + c_{s,t}.

The basis and calibration use outer-training questions, not target labels. Evaluation uses held-out questions.

Across six ordered source-target pairs and five splits, corrected mean transfer is 0.763, compared with 0.701 for single-source ridge. The paired difference is +0.062 [0.017, 0.078]. The held-out powered control is 0.535 [0.474, 0.569]. Both intervals include refitting of the basis and calibration.

This comparison does not isolate rank reduction: the shared basis uses labels from two non-target sources, while each ridge comparator uses one.

Including all four sources gives transfer of 0.745 versus 0.664 for ridge across twelve ordered pairs, a difference of +0.081 [0.067, 0.093]. Its powered control is 0.618 [0.579, 0.657]. This analysis evaluates a different set of source-target pairs, and its intervals hold the fitted probes fixed. We therefore report it separately as a sensitivity analysis rather than compare its gain directly with the three-source result (supplementary Table S6).

Q4 - Question overlap

Only the original ARC-Gemma and ARC-Qwen cells share questions between source training and targets: 547 and 508 shared IDs, respectively. The other six cells and the separate Gemma replication have no shared IDs or normalized question texts (supplementary Table S11b).

The Gemma replication also changed the training pool. We therefore cannot attribute the change in its control slope to overlap alone.

In the revision, lines 248-259 will separate nominal from multiplicity-corrected evidence and report the matched experiment. We will withdraw the raw pass claim in Table 3 and its discussion (lines 339-345) and add the powered controls to Table 4. An overlap table will follow line 304. Section 5.2 (lines 403-412) and Appendix H.1 will disclose the three-source restriction and the coordinate correction. The conclusion and limitations (lines 447-449 and 471-472) will be revised to match.

Reviewer_zeEzOpenReview ↗

Thank you for the transparent audit and revisions (Maintaining Weak Accept)

i would like to thank the authors for their detailed rebuttal. The new analyses, bug fixes, and claim retractions directly answer all my questions and resolve my concerns regarding the technical correctness of the reported numbers. I am raising my Soundness score to 3.

However, I am keeping my overall rating at 3 (Weak Accept). While the paper's core diagnostic message, that deception probes often just track ordinary errors remains a useful contribution to the field, the rebuttal resolves several of my concerns primarily by weakening or retracting the positive claims.

  • First, once Table 3 is evaluated using folded separability around 0.50, none of the eight raw prompt-presence cells pass (all land in the 0.582–0.648 partial band), and the authors have withdrawn the claim that these controls rule out prompt-context sensitivity.
  • Second, while the new 132-question matched ranking result for scheming is encouraging (0.762 at 150 labels), the authors note that counting honest-error separation in either direction leaves the corresponding scaling contrast inconclusive (+0.002 [-0.019, 0.025]).
  • Third, the audit of Section 5.2 revealed a coordinate bug, an undocumented exclusion of the incentive-cue source, and a baseline mismatch: the shared direction combines two non-target sources while the ridge baseline uses only one, which does not isolate rank reduction. When all four sources are included, the wrongness control rises to 0.618, showing substantial error confounding.
  • Finally, two of the eight configurations (ARC-Gemma and ARC-Qwen) had over 500 overlapping questions between source and target, and the Gemma replication changed the training pool alongside the split design, making it hard to isolate the effect of question overlap.

Taken together, the paper is a solid methodological warning on probe evaluation and deserves to be accepted, but given the narrower positive findings and the extent of the revisions needed across Sections 4 and 5, Weak Accept feels like the right calibration.

Reviewer_sfDQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies the extent to which probes learned to predict deception from LLMs' activations are picking up on deception, instead of certain spurious correlations.

It is somewhat hard to understand the headline results of the paper, since they are not stated clearly in a results section or contributions section. However, as I understand it, the paper looks at a setup where a linear probe is trained on a certain 'source' distribution, containing a particular type of deception, such as that from instructed deception, and evaluates on a 'target' distribution where the type of deception is different, such as from self-motivated scheming. The paper finds that in the particular setup of analyzing ARC-Challenge, there is a significant to severe degradation when moving from the source training distribution to the target eval distribution.

Critical strengths
  • The paper is comprehensive and, I am fairly confident in saying, accurately describes the work that the authors performed. The experimental setup is sound insofar as it's measuring the considerations that are introduced in the text.
  • The study of monitoring techniques for AI deployment is an extremely timely topic and I'm glad to see that work is being done on this area. Several recent security incidents illustrate that current AI systems have the ability to cause substantial harm, and guardrails such as probes have the potential to prevent this harm.
Critical weaknesses
  • My main critique of the paper is that the setup is quite contrived and it is unclear how the results would transfer to anything approaching a real-life deployment of probes. A high level (and somewhat uncharitable) characterization of this work would be that it introduces a dataset with a spurious correlation, and shows that training a model (probe) on the dataset suffers from spurious correlations. Of course there are important nuances here, such as the fact that we're looking at activations and it's unclear to what extent they would transfer a-priori. But previous work such as [Goldowsky-Dill et al] and many others have already looked at this question, back in 2025 (with more examples than 150). If the paper is to say anything about considerations for real deployment, it needs to either exhibit a very clear takeaway that would likely still be exhibited at the 400B+ scale, or actually probe at that scale.

  • Similar to the above weakness, the paper is quite unclear about what the actual takeaways are. There is no clear section for results or contribution. The paper spends most of the space describing in great detail exactly what was done, without really explaining why or what the takeaways are.

Questions for authors
  • Can you please summarize in a few sentences what the clear takeaways are from your work?
  • Can you explain why the models used in this work are so old?
  • Can you explain why you believe your takeaways will transfer to the deployment motivations that you discuss in the introduction, with much larger models likely trained on significantly larger, diverse corpora with control for confounders such as the wrong answer?
3 comments

Thank you for the review. We apologize that the paper did not make its main result clear enough. The new analyses cited below are in the revised supplementary material (supplementary_tables.pdf in the zip file).

We ask whether adding labels improves transfer for a reason beyond ordinary-error detection. To test this, we track transfer and honest-error detection using the same fitted probe.

Q1 - Main takeaways

Better transfer alone is not enough. On MMLU, supervised transfer to the instructed target and honest-error detection improve at almost the same rate. Their slope difference is +0.001 AUROC per doubling.

Training on instructed errors does not reliably cover the other sources we tested. On ARC-Llama, the supervised instructed probe scores 0.53 on scheming and 0.45 on incentive-cue responses. Source-matched probes do better, although some also respond to honest errors.

The reported hidden-objective transfer curves have limited recall. At a 5% false-positive rate, they detect fewer than one third of deceptive responses.

The new matched experiment provides a positive but limited result. Both estimators distinguish deliberate from honest errors on the eligible scheming questions after matching the question and wrong letter. This does not establish intent detection or independence from prompt context.

W1 - Spurious correlations and prior work

Correctness is an unavoidable confound in our setup: every deliberately wrong answer is also incorrect. We do not claim that this concern, or large-model probing, is new.

Our contribution is to test whether the improvement from additional labels exceeds the same probe's improvement on honest errors. This comparison is a diagnostic, not a way to divide the probe's signal into separate amounts of wrongness and deception.

Goldowsky-Dill et al. (ICML 2025) study Llama-3.3-70B and use responses to general Alpaca queries for false-positive evaluation in Section 3.2.3. We additionally evaluate probes on the model's honest mistakes. The new experiment compares deliberate and honest errors with the same question and wrong letter.

The 10 to 150 range refers to training responses per class for each probe fit. It is not the total number of generated responses or evaluation examples.

Q2 - Model choice

Our model set limits what we can say about frontier systems. The experiments require activation access, repeated generation, and enough correct and incorrect responses under each source prompt.

We used Llama-3.3-70B for the released-detector comparison because those detectors were trained on that model. Only its scheming source provided enough examples in both classes for that evaluation.

We also ran a newer model, Qwen3-32B. Its hidden-objective source produced 1,005 incorrect responses and only 13 correct responses after filtering. This supports the smallest training budget, but not a scaling curve (supplementary Table S9c). We report this unsuccessful attempt rather than treating it as evidence of transfer.

Q3 - Relevance to deployment and larger models

We have not tested whether these transfer failures persist at 400B+ scale, and we will remove wording that suggests otherwise.

The methodological concern is broader than the models tested: transfer improvement may accompany improved error detection. How much this matters must be measured for each model and setting.

Our 70B evaluation is only a limited check. In their native views, the released instructed-pairs and roleplaying detectors score 0.26 and 0.17 AUROC below a source-matched scheming probe. This does not establish the same result for other sources or deployed systems.

In the revision, we will add a short summary of the main findings after line 059 and at the start of Section 4 (line 212). We will also narrow the Abstract (lines 016-018) and the monitoring discussion around line 419, so that neither suggests results beyond the models we tested.

Thanks for your replies. I appreciate that you'll change the paper to remove over-reaching language suggesting applicability at the frontier scale. However, given that there's no substantial updates to the experiments themselves in reply to my questions, and the greater methodological/scope limitations with the paper therefore persist, I will maintain my score.

Thank you for your reply. Here is a quick breakdown of how our new experiments in the supplementary material address each of your points:

On W1 (the correctness confound), we agree that since every deliberate error is also an incorrect answer, a probe could look like it detects deception just by picking up on wrong answers. To test this directly, we collected 22,176 new responses across 792 held-out ARC questions that were never used in probe training (Sections S1 and S10) and paired deliberate and honest errors on the exact same question with the exact same wrong answer letter, so both answers are equally wrong. Across the 132 eligible scheming questions, the difference-of-means (DoM) probe ranks the deliberate error above the matched honest error with probability 0.644 at 10 labels per class and 0.762 at 150 labels, while the supervised probe gets 0.659 and 0.716 (Table S4c). This shows the probes do separate deliberate from honest errors even when the wrong answer itself is held fixed, though whether that separation grows faster with label count than honest-error detection is still inconclusive under our folded control (+0.002, 95% CI [-0.019, 0.025]).

On Q2 (testing a newer model), we ran the full protocol on Qwen3-32B (Table S9c), but its hidden-objective prompt produced 1,005 wrong answers and only 13 right ones, which was too imbalanced to fit the label-scaling curve. While the scheming prompt was well balanced (423 wrong and 415 right), using it as the training source would have forced us to split the evaluation set and leave only around 25 incentive-cue errors for the transfer test, so we decided not to claim a Qwen3 scaling result and instead note this openly as a limitation.

On Q3 (frontier scale), we completely agree that our experiments do not cover frontier-scale models, and we are removing any wording from the paper that implies they do. Our main takeaway is really about the evaluation method itself: whenever a probe is evaluated by cross-source transfer, its honest-error detection rate should be measured right alongside it, which only requires access to the probe and a set of natural mistakes.

Reviewer_7v3vOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper investigates the transferability of deception probes across different sources of deceptive behavior, with a particular focus on distinguishing deception-specific information from sensitivity to ordinary incorrect answers (wrongness). By comparing cross-source transfer curves against wrongness controls, the authors find evidence of deception-specific transfer in certain settings, but the results depend substantially on the task, model, and probe estimator. The findings also suggest considerable source specificity in the learned representations.

Critical strengths
  1. The paper provides a valuable evaluation perspective by highlighting that improved transfer performance with more training data does not necessarily imply learning more general deception representations. Comparing transfer and wrongness-control slopes offers a principled framework for interpreting probe generalization.

  2. The experimental design is systematic, incorporating within-prompt contrasts, difficulty matching, and controls for wrongness, confidence, and prompt context. These efforts help mitigate potential confounding factors and strengthen the validity of the cross-source transfer analysis.

  3. The empirical evaluation is relatively rigorous, covering multiple models, tasks, and deception sources, with bootstrap confidence intervals, full-refit analyses, multiple-comparison corrections, and independent replications. The authors also transparently report statistically insignificant or inconsistent findings.

Critical weaknesses
  1. The experiments do not fully isolate deception-specific signals from other behavioral differences. Deliberate errors and honest mistakes may still differ in item distributions, generation processes, and internal states. Therefore, transfer improvements beyond the wrongness control do not necessarily establish that the probes capture representations specifically associated with deceptive intent.

  2. The statistical evidence supporting the main findings is somewhat inconsistent. In the primary ARC experiments, only one difference-of-means contrast remains significant after full refitting and multiple-comparison correction, while none of the supervised contrasts does. The substantial variation across tasks and models further limits the generalizability of the conclusions.

  3. The source-specificity findings primarily establish differences in predictive performance rather than distinct underlying representations. Different deception sources may induce different prompt contexts, reasoning patterns, or behavioral strategies, and the existing controls do not fully rule out their influence on the learned probe directions.

Questions for authors
  1. Could the authors construct more strictly matched comparisons between deliberate errors and honest mistakes, ideally controlling for the question, incorrect answer, and generation conditions? This would help clarify whether the observed transfer gains genuinely reflect deception-specific information.

  2. How should the disagreement between supervised probes and difference-of-means directions be interpreted, particularly when only the latter shows statistically significant deception-specific scaling? Does this reflect meaningful representational differences or sensitivity to the statistical properties of the activation space?

  3. To what extent does the observed source specificity reflect genuinely distinct deception representations rather than differences in elicitation contexts or behavioral strategies? Could additional controlled experiments help distinguish these explanations?

2 comments

Thank you for the review. We ran a new experiment to address your request for a more closely matched comparison. The new analyses cited below are in the revised supplementary material (supplementary_tables.pdf in the zip file).

W1/Q1 - Comparing deliberate and honest errors

In the new experiment, we compare errors on the same question, with the same wrong answer letter and the same decoding settings. All evaluation questions are excluded from probe training. The design was fixed before generation; supplementary Section S10 records the protocol.

The comparison includes only questions and wrong letters observed under both prompts. It therefore tests separation within this subset, not across all scheming responses.

Supplementary Tables S4c and S3b report the ranking and scaling results, respectively:

  • Direct comparison of errors. On 132 scheming questions, the difference-of-means (DoM) probe ranks a deliberate error above an honest error with the same wrong letter with probability 0.644 at 10 labels per class and 0.762 at 150. The supervised probe gives 0.659 and 0.716. Across the five budgets and two estimators, all ten pointwise 95% intervals are above 0.5. These are not simultaneous intervals.
  • Change with training labels. The prespecified test compares how quickly each error type separates from the same honest-correct responses. For DoM, the difference in slopes is +0.032 AUROC per doubling [0.009, 0.054], with BH q=0.006 and Holm-adjusted p=0.010.

We also checked whether this scaling result holds when the honest-error control counts separation in either score direction. This post hoc comparison is inconclusive: +0.002 [-0.019, 0.025]. Thus, the direct error comparison is positive, but we cannot claim faster scaling than this alternative control.

The prompts still differ. These results show that the probes can distinguish the two response groups under the matched conditions; they do not establish that the probes detect intent.

W2 - Statistical consistency

The evidence is narrower than a general claim about deception-probe transfer.

After correcting the analysis of the original ARC pools, DoM scheming survives both BH and Holm correction. DoM incentive survives BH (q=0.048), but not Holm.

In the new matched experiment, three of the six primary comparisons survive BH: DoM scheming and both instructed comparisons. The instructed errors remain much shorter than honest errors, which limits what we can conclude from those two results.

We will distinguish these findings from the inconclusive comparisons and retain the variation across tasks, models and estimators as part of the result.

Q2 - Why the estimators disagree

Both estimators distinguish deliberate from honest errors in the matched scheming subset. Their scaling results differ.

We examined whether covariance weighting contributes to this difference. Replacing DoM with regularized LDA reduces the original scheming slope contrast from +0.0315 to +0.0058 per doubling (supplementary Table S9a).

This is a descriptive comparison. Its intervals use the original uncertainty procedure, not the corrected question-group procedure, so we do not use them to test the difference between estimators. The comparison shows sensitivity to covariance weighting; it does not identify the responsible features or establish different kinds of deception representation.

W3/Q3 - What source specificity means

The submitted 22/24 count described positive pairwise differences, not 22 significant results. With the corrected evaluation design and full refitting, 15/24 comparisons survive BH. Against the strongest transferred direction, selected again within each resample, only 1/8 target-estimator comparisons survives BH.

We also tested how the prompt affects the measured activations. We kept each probe fixed and re-encoded the same completion after the honest prompt (supplementary Table S9b). Instructed-to-hidden-objective transfer changes from 0.763 to 0.283. On scheming, the hidden-objective DoM score changes from 0.591 to 0.511, while the supervised score changes from 0.720 to 0.701.

Some directions therefore depend strongly on the context in which the transcript is encoded. The replay creates prompt-transcript combinations that were not generated naturally, so it cannot tell us how much of the original separation came from context rather than behavior.

We will describe source specificity as a difference in predictive performance, not evidence of distinct causal representations.

In the revision, we will narrow the Abstract (lines 016-018) and replace the earlier same-question comparison at lines 255-259 with the matched experiment. We will restate the source-comparison result at line 363 with the corrected counts. The Limitations (lines 456-472) will state that the matched result depends on prompt context and question selection.

Thank you for the additional experiments, particularly the matched error comparison, covariance-weighting analysis, and prompt-context replay experiment. These additions strengthen the empirical evidence. I appreciate the authors’ transparency and more careful framing of the conclusions. I am raising my rating to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.