- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper examines whether activation probes trained on one form of deliberately wrong answering generalize to others, and whether gains from more training data reflect more than ordinary error detection.
Experiments across models and tasks use honest mistakes as a control and show mixed results: some gains exceed this control, while others do not. Probes generally perform best on the behavior they were trained on.
Critical strengths
The comparison with honest errors addresses a confound in activation-probe evaluation: detecting deliberately wrong answers may simply reflect ordinary error detection. Evaluating the same probes under both conditions makes the transfer results more interpretable and provides a useful control for future studies.
Critical weaknesses
The main comparison does not sufficiently isolate deliberate deception.
The transfer targets and honest-error control differ in question selection and decoding. The within-question analysis also does not establish separation for scheming. This leaves alternative explanations for the central positive result unresolved.
Questions for authors
-
The transfer targets and honest-error control differ in question selection and decoding. What evidence shows that these differences do not explain the positive results, particularly given the inconclusive within-question comparison for scheming?
-
How do the uncertainty estimates account for shared questions and repeated splits? After accounting for these dependencies and the relevant multiple-comparison corrections, how strong is the support for the source-matched advantage and the claim that the Gemma replication supports both estimators?
2 comments
Thanks, I've increased my score. Good luck.
The new matched experiment and revised uncertainty analysis substantially address my main concerns. I particularly appreciate the clear acknowledgment of the earlier limitations and the willingness to withdraw claims that are no longer supported. These improvements make me more positive about the paper.
The comparison on 132 matched questions provides stronger evidence that the probes distinguish the two response groups within the selected subset.
However, the interpretation remains limited: the scaling contrast becomes inconclusive when the honest-error control counts separation in either direction, and the replay experiment indicates substantial sensitivity to prompt context. I therefore view the new results as evidence of response-group separation, with the explanation for the transfer gains still partly unresolved. The distinction from intent detection should remain explicit.
The corrected analysis also supports a more qualified account of source specificity. Some pairwise advantages remain, but only one of eight comparisons against the strongest transferred direction survives BH correction, which provides limited support for a general source-matched advantage.
Similarly, the Gemma replication supports the difference-of-means results more clearly than the supervised results after correction. I appreciate the planned revisions to reflect these distinctions.
My remaining reservations concern the breadth and strength of the conclusions, while the rebuttal meaningfully improves the evidence and its presentation.
Thank you for identifying the matching and dependence issues. We have added a new comparison and revised the uncertainty analysis. The new analyses cited below are in the revised supplementary material (supplementary_tables.pdf in the zip file).
W1/Q1 - Question selection and decoding
The earlier analysis did not establish same-question separation for scheming.
The new experiment matches the question, wrong answer letter and decoding settings, and excludes probe-training questions. Its design was fixed before generation (supplementary Section S10). It includes only questions that produce both error types with a shared wrong letter, so its conclusions apply to that subset.
On 132 matched questions, the difference-of-means (DoM) probe ranks the deliberate error above the honest error with probability 0.644 [0.567, 0.711] at 10 labels per class and 0.762 [0.672, 0.836] at 150. The supervised values are 0.659 and 0.716. Across five budgets and two estimators, all ten pointwise (not simultaneous) 95% intervals are above 0.5 (supplementary Table S4c).
The statistic averages all error pairs within each question-letter cell, with ties counted as one half, then weights letters and questions equally.
We also checked response length. Restricting pairs to lengths within 20 tokens retains 72 scheming questions. DoM ranking at 150 labels is 0.713 [0.606, 0.816]. This is a post hoc sensitivity analysis, not a confirmatory test (supplementary Table S5c).
The prespecified signed scaling contrast is +0.032 per doubling [0.009, 0.054], BH q=0.006 and Holm-adjusted p=0.010. It compares the slopes for each error type against the same honest-correct responses (supplementary Table S3b).
When the honest-error control counts separation in either direction, the scaling contrast is inconclusive: +0.002 [-0.019, 0.025]. The prompts also remain different. In a separate replay test, re-encoding the same scheming transcripts under the honest prompt reduces hidden-objective DoM transfer from 0.591 to 0.511 (supplementary Table S9b).
The new experiment therefore supports separation between the matched response groups. It does not establish intent detection or independence from prompt context.
Q2 - Shared questions, splits and multiple comparisons
The submitted bootstrap did not keep shared questions together across pools and split seeds.
We now assign each question group one bootstrap weight wherever it occurs. We also use common source splits, group duplicate normalized question texts, and score every compared probe on the same held-out questions. Each full-refit replicate re-estimates the fitted quantities, including standardization and source directions. Results are averaged across split seeds; seeds and training draws are not treated as independent question samples.
The updated results reflect both the revised evaluation design and the corrected bootstrap.
The submitted 22/24 count described positive pairwise differences. Under the corrected full-refit analysis, 15/24 comparisons survive BH and 13/24 survive Holm. Against the strongest transferred direction, selected again in each resample, only 1/8 comparisons survives BH.
For the question-disjoint Gemma replication, supervised scheming has a nominal interval above zero but does not survive BH or Holm correction over six contrasts (BH q=0.067; Holm-adjusted p=0.133). We withdraw the multiplicity-corrected both-estimator replication claim.
DoM instructed and scheming survive both corrections, each with BH q=0.020 and Holm-adjusted p=0.040. These estimates use 300 question-bootstrap replicates with full refitting, so the tail probabilities remain coarse.
In the revision, Appendix A will describe the question-group bootstrap, and line 363 will report the corrected counts. We will also remove the both-estimator Gemma claim from the Abstract (lines 017-018), lines 303-304 and Appendix G.2.