acceptodds
Under review as a conference paper at ICLR 2027

A Probe Direction Is a Property of Its Prompt

Abstract

The prompt used to build an activation probe for evaluation awareness decides the sign of the scaling result that the probe reports. Such a probe contrasts a model's activations on prompts framed as a test with its activations on the same prompts framed as ordinary use, and scores how well the resulting direction separates held-out items. We keep the task text byte-identical and cross a set of test framings with a set of deployment framings, on four series of open-weight models, each released at several sizes, up to the current Qwen3.5 generation. On every series some pairs of framings reproducibly make the score rise with model size and others make it fall, and this one design spans both published scaling results on this benchmark, one positive and one flat. The model being measured accounts for a point estimate of at most 13.1% of the variance in the score, while the way each model responds to each pair of framings contributes at least twice as much as the way it responds to each evaluation item. A reliable comparison between models therefore averages the score over many pairs of framings rather than adding items, and we measure how many are needed by checking that averages over disjoint sets order the models alike. A direction that carries no information about evaluation also reproduces much of each published score, because the benchmark's two classes differ in surface form, so every score should be read against that baseline. Code: https://anonymous.4open.science/r/probe-direction-384B/

Primary area:
interpretability and explainable AI
Keywords:
evaluation awareness, linear probes, measurement validity, generalizability theory, model comparison, interpretability, activation probing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_J3yeOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper shows that prompt choices can reverse conclusions about how evaluation awareness scales with model size. It investigates this sensitivity across four model families and evaluates whether averaging across prompts makes model comparisons more reliable.

Critical strengths
  1. The paper addresses an important measurement problem: whether reported differences between models reflect stable model properties or choices made when constructing probes.

  2. The crossed design varies both evaluation and deployment framings while keeping task text byte-identical, allowing their effects and interactions to be examined systematically.

  3. Experiments across multiple model families, split-half analyses, and pooling controls provide substantial evidence that wrapper sensitivity is reproducible rather than merely item-sampling noise.

Critical weaknesses
  1. Averaging across wrappers makes scores more consistent, but they may still reflect surface features rather than evaluation awareness. Testing on a format-controlled dataset would help resolve this concern.

  2. The sign-flip baseline may retain part of the original concept direction. Calling it “content-free” therefore needs stronger justification.

  3. The two disjoint wrapper subsets can still share individual framings. Their agreement does not show whether the results extend to new framings.

  4. The recommendation to add wrappers rather than items is based on the current benchmark. New, format-balanced items could address problems that wrapper averaging cannot.

Questions for authors
  1. Do the main findings hold when prompt format is controlled? Does wrapper averaging also improve agreement with behavioral measures of evaluation awareness?

  2. How similar are the sign-flip directions to the original probe directions? Would other random-direction baselines give the same results?

  3. Do the two wrapper subsets still agree when they share no framings on either side? How does Equation (1) account for the separate effects of the two framing arms?

  4. Does the sensitivity persist when the wrappers express the same scenario in different words?

4 comments
AuthorsOpenReview ↗

Response to Reviewer J3ye (1/2)

Thank you for the review, and for calling this "an important measurement problem". Most of your questions became experiments.

W1.

Averaging across wrappers makes scores more consistent, but they may still reflect surface features rather than evaluation awareness. Testing on a format-controlled dataset would help resolve this concern.

We ran both. On byte-identical tasks the input embeddings already orient the framings, and agreement with behaviour follows scale and wording.

Format-controlled items (Section 11, Appendix R).

Byte-identical tasksResult
Trained direction orients unseen framings77 to 100%
Input embeddings do so96 to 100%
Model's share of varianceat most 4.0%

So this test cannot tell a computed representation from the framings' words. Alignment trends take both signs on every ladder, as registered, but no more often than under random framing partitions (12th to 79th percentile). Nor does a wrapper's sign reproduce across disjoint test framings beyond that null.

Behaviour (Section 9, Figure 1e, Appendix S). We registered a comparison with SAD stages-oversight accuracy, the task our scoring set comes from (278 of 400 items shared on Qwen3, Qwen3.5 and OLMo-2). We ran it with its authors' code on all 19 non-thinking checkpoints. The primary variant (sp_large) met all three preconditions: split-half reliability 0.94, between-model SD 0.084, all 19 models included.

Spearman with SAD accuracyValueStatus
36-wrapper average+0.63 [+0.40, +0.72], p = 0.0009registered: "supports"
100 geometry-matched rotations, 95th percentile+0.37 (the average beats all 100)registered null
Single wrapper, median+0.36 (30 of 36 positive)descriptive
Same words, shuffled labels, median+0.60 (35% of draws match +0.63)descriptive, post hoc
Log parameter count+0.89 (partial of the average, given size: -0.12)descriptive, post hoc

Brackets: 95% item-bootstrap interval. p: one-sided within-ladder permutation test. Averaging does improve agreement, descriptively. The two controls computed after the registered result show that scale and wording produce it. This extends the paper's conclusion from probe scores to behavioural anchors. Other registered readings:

  • Orientation: "supports" (+0.43 [+0.26, +0.53]).
  • Signed band statistic: anti-aligned (-0.24 [-0.39, -0.05]), as the registration expected from its falling trend with size on Qwen3.
  • Plain variant: too little spread between models to be informative (SD 0.030, below the registered 0.05), as anticipated.

W2.

The sign-flip baseline may retain part of the original concept direction. Calling it "content-free" therefore needs stronger justification.

We agree, and renamed it the label-permuted floor. "Content-free" now names a floor that is content-free by construction, and it gives the same picture.

New floor (Appendix N, Table 5): random directions matching the released direction's layer-to-layer autocorrelation.

Four Qwen2.5 checkpointsValues
Floor (SD 0.037 to 0.048)0.203, 0.184, 0.216, 0.253
Published values0.211, 0.291, 0.292, 0.190
z of the published value+0.2, +2.9, +1.8, -1.3

One model of four clears 2 SD. The floor reaches 63 to 133 percent of the published values (sign-flip floor: 70 to 116 percent).

Sign-flip draws (released 204-item construction). Their median ∣cos⁡∣|\cos| with the rebuilt direction is 0.06 to 0.18. But per-state medians reach 0.60, single draws at single states reach 0.97 to 0.99, and the statistic maximises over states. The draws are also nearly collinear with each other (0.84 to 0.98). So the floor's SD understates content-free spread, and its mean may be biased. On the framing ladders the construction is degenerate (median ∣cos⁡∣|\cos| 0.98 to 0.99 with the real direction), so the ladder analyses use other nulls.

Other nulls (Appendix N, Table 4). Task-permutation directions sit further from the real ones (mean ∣cos⁡∣|\cos| 0.26 to 0.37) than two real wrappers do from each other (0.38 to 0.43). The nulls disagree. Per-wrapper slopes are more reproducible than 98 to 100 percent of isotropic draws on Qwen2.5 and Qwen3, and less than matched rotations on Qwen3.5. The 36-wrapper trend beats matched rotations on Qwen3, Qwen3.5 and OLMo-2, and sits inside the task-permutation null everywhere. That null keeps the framings' words, so we read the trend against it. The appendix reports every null.

AuthorsOpenReview ↗

Response to Reviewer J3ye (2/2)

W3.

The two disjoint wrapper subsets can still share individual framings. Their agreement does not show whether the results extend to new framings.

We agree: shared framings inflated the agreement we reported (Section 9). Splitting each arm's six framings into halves, in all 200 ways, gives two 3×33 \times 3 subsets that share nothing. Their agreement is lower than for random disjoint subsets of the same size on every ladder (Kendall τ\tau 0.83 against 0.90 on OLMo-2, 0.02 against 0.23 on Qwen2.5). It never exceeds the 95th percentile of the task-permutation null. With four or five models that test sits at a ceiling. On OLMo-2 the real median Spearman and the null's 95th percentile are both 1.0, and on Qwen3.5 with thinking both are 0.9.

Framing-disjoint halvesSame scaling sign
OLMo-298.5% of splits
Qwen3.598.5% (99.5% with thinking)
Qwen382.5%
Qwen2.520%

Equation (1) treats the 36 wrappers as exchangeable, pooling the two arms. Section 6 and Appendix B analyse the design as crossed, where the flat form is optimistic (0.629 against 0.257).

Wrappers for Eρ2=0.80E\rho^2 = 0.80CrossedFlat
Qwen3.536 (6 framings per arm)12
OLMo-264 (8 per arm)10
Qwen3no number suffices248
Qwen2.5no number sufficesnever

Table 1 now prints both. R2 now follows the crossed analysis, and R3 the byte-identical tasks.

W4.

The recommendation to add wrappers rather than items is based on the current benchmark. New, format-balanced items could address problems that wrapper averaging cannot.

We agree, and R3 now says so. On the byte-identical set the largest facet is the pairing of training and test framings. A reliable design must therefore sample the framings of the scored items as well as the wrappers.

Q1.

Do the main findings hold when prompt format is controlled? Does wrapper averaging also improve agreement with behavioral measures of evaluation awareness?

Answered with W1 above.

Q2.

How similar are the sign-flip directions to the original probe directions? Would other random-direction baselines give the same results?

Answered with W2 above.

Q3.

Do the two wrapper subsets still agree when they share no framings on either side? How does Equation (1) account for the separate effects of the two framing arms?

Answered with W3 above.

Q4.

Does the sensitivity persist when the wrappers express the same scenario in different words?

Yes for the folded score, and rewording reorders the models. The signed value and the orientation stay stable.

Two paraphrases of each of the 12 framings keep every scenario fact. A script checked that each shares no content 4-gram with the original and stays within 10 percent of its length. We extracted 11 rungs (Qwen3-0.6B to Qwen3-32B, Qwen3.5-0.8B to Qwen3.5-27B).

Variance shareQwen3Qwen3.5
Rewording0.230.18
Scenario0.190.32
Model ×\times wrapper ×\times wording0.310.27
Model ×\times wrapper0.160.10

Rewording carries a share comparable to the scenario's. The registered decomposition asked whether the scenario carries the model-by-wrapper term. It does not: the three-way term, where a wrapper's ordering of the models changes with its wording, is larger.

Under rewording, the folded score reproduces only at Spearman 0.19 to 0.66, and the signed interior-band value at 0.68 to 0.95. Orientation survives on all 11 models (0.75 to 0.998, each above its registered baseline). On the six smaller rungs it is already 0.91 to 0.99 at the paraphrased framings' embeddings. The scaling sign sits in between, following partly the scenario and partly the words:

Wrappers of 36 keeping their signQwen3Qwen3.5
Second paraphrase2929
Random re-pairings, 95th percentile2527
First paraphrase2624
Random re-pairings, 95th percentile2628

The results are labelled descriptive. The registered gate reran the original framings and asked each wrapper's statistic to reproduce within 0.01. It passed on the five largest rungs (largest change 0.004) and failed on three of the six smaller ones (0.030, 0.018 and 0.0115). Two failures came from the change of GPU and software stack, one from batch composition on the same machine. An amended gate, written after that failure and before any paraphrase number was computed, still failed on two. All versions come from one extraction. Stored and rerun statistics agree at Spearman 0.988 to 1.000, far above the 0.19 to 0.66 from rewording (Appendix T).

Sections 6, 9 and 10, Table 1, the new Section 11, the recommendations, the limitations and the appendices are revised accordingly. We hope the answers bear out your reading of the paper's importance.

We are happy to discuss any of these points further.

Reviewer_J3yeOpenReview ↗

Rebuttal

The author resolved most of my concerns. I will maintain my score.

AuthorsOpenReview ↗

Reply to Reviewer J3ye

Thank you for reading our response and for letting us know. If any concern remains open, could you tell us which one? We will answer it before the discussion closes.

Reviewer_5WB2OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper evaluates the effectiveness of activation probes (difference-of-means vectors) fit from a single pair of prompt framings (with multiple prompts per framing) in the setting of evaluation awareness. They study whether probe performance on a held out set of examples for evaluation vs. deployment corresponds with model size. They find that the correlation can be set by choosing different prompt framings, and that there isn't a general trend. They also show that averaging probe scores across multiple framings gives a more consistent correlation. They identify substantial confounding in the held-out evaluation dataset: differences between evaluation and deployment examples that can be found with a simple bag-of-words classifier or a probe with random labels. The authors validate this on four model families (Qwen2.5, Qwen3, Qwen3.5, and OLMo-2), using 36 combinations of evaluation/deployment framings.

Critical strengths

None

Critical weaknesses

The paper's primary finding, that difference-of-means probes are sensitive to prompt choice, is already well established in prior interpretability research. Hua et al. (2026), has investigated and used multiple evaluation/deployment prompt pairs to construct more robust activation directions. The authors quantify this sensitivity across model sizes, but do not establish what causes particular prompt pairs to produce positive versus negative scaling correlations. Furthermore, the prompts describe meaningfully different situations (e.g., red-teaming evaluations, monitored oversight, benchmarks), which may genuinely elicit different representations. It is therefore unclear that these directions should be expected to measure an identical underlying concept.

More importantly, the authors demonstrate in section 10 that their evaluation benchmark is highly confounded by superficial task-format differences, yet continue to rely on it for their main experiments. A bag-of-words classifier achieves near perfect performance, suggesting that the reported scaling correlations may reflect distinctions unrelated to evaluation awareness. It is unclear if the authors are making a claim about using multiple prompts pairs to construct a robust direction, or if the claim is that the dataset is untenable for evaluation awareness research.

Hua, T. T., Qin, A., Marks, S., & Nanda, N. (2026). Steering Evaluation-Aware Language Models to Act Like They Are Deployed. International Conference on Learning Representations.

Questions for authors

How does this work extend prior research on prompt sensitivity in activation probing, particularly Hua et al. (2026), which already uses multiple prompt pairs for evaluation awareness?

Since section 10 demonstrates substantial task-format confounding in the SAD oversight benchmark (0.993 bag-of-words AUROC), it would be useful to evaluate the probe directions on a controlled dataset with identical underlying tasks and varied evaluation/deployment framings. Without this control, is it possible to distinguish genuine evaluation-awareness representations from superficial differences in prompt format?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 5WB2 (1/2)

Thank you. The revised abstract now answers your central question: on every family, the framings' words alone can produce the averaged trend.

W1.

The paper's primary finding, that difference-of-means probes are sensitive to prompt choice, is already well established in prior interpretability research. Hua et al. (2026), has investigated and used multiple evaluation/deployment prompt pairs to construct more robust activation directions. The authors quantify this sensitivity across model sizes, but do not establish what causes particular prompt pairs to produce positive versus negative scaling correlations. Furthermore, the prompts describe meaningfully different situations (e.g., red-teaming evaluations, monitored oversight, benchmarks), which may genuinely elicit different representations. It is therefore unclear that these directions should be expected to measure an identical underlying concept.

Hua, T. T., Qin, A., Marks, S., & Nanda, N. (2026). Steering Evaluation-Aware Language Models to Act Like They Are Deployed. International Conference on Learning Representations.

Prior work shows that probes vary with prompts. We show that this variation decides a published model comparison. On every ladder the prompt pair reverses the sign of the published scaling result, and the model explains at most 13.1 percent of the score's variance. We measure this share and give an audit that detects it (W2). Both studies we reanalyse made their scaling claims on this benchmark, so we audit them where they were made.

Hua et al. (2026)This paper
Prompt pairs16, hand-written, mean difference at the colon tokencrossed framings, byte-identical task text
Modelssteers one (Llama 3.3 Nemotron Super 49B)probe scores on 19 checkpoints, four ladders, two developers
Spread across pairs16-pair average steers better than most single pairs (their Appendix G.3)can reverse a scaling conclusion

Devbunova (2026) shows that probes track benchmark format. We hold the task text byte-identical and show what remains (Section 2).

Source of the sign.

  • The statistic. Unfolding it reverses the averaged trend on every ladder. At the state where its maximum falls, 32 to 88 percent of model-by-wrapper cells separate the classes the wrong way round. That state is often hidden state 0, the input embeddings (58 percent of wrappers on OLMo-2-1B).
  • The layer set. On Qwen3.5, seven of the nine falling wrappers without thinking, and all five with thinking on, fall only because the last hidden state enters the maximum.

A registered test, committed before it ran, rules out a simpler account: that a wrapper's sign follows its alignment with the scoring set's class axis.

Different situations are what the model-by-wrapper term measures. On three of four ladders it is no larger for the real directions than for label-shuffled ones with the same words. Reworded with the scenario kept, a wrapper's ordering of the models depends more on wording than on scenario (11 rungs of Qwen3 and Qwen3.5, descriptive).

Presentation. Terms are now defined in plain words where they first appear. One running example, the two Qwen2.5 wrappers coloured in Figure 1, runs from the introduction to the Conclusion.

AuthorsOpenReview ↗

Response to Reviewer 5WB2 (2/2)

W2.

More importantly, the authors demonstrate in section 10 that their evaluation benchmark is highly confounded by superficial task-format differences, yet continue to rely on it for their main experiments. A bag-of-words classifier achieves near perfect performance, suggesting that the reported scaling correlations may reflect distinctions unrelated to evaluation awareness. It is unclear if the authors are making a claim about using multiple prompts pairs to construct a robust direction, or if the claim is that the dataset is untenable for evaluation awareness research.

It is the second. Many prompt pairs can make the comparison reliable, and reliable is not the same as valid. On OLMo-2 and Qwen3.5, averages from disjoint framings agree on the scaling sign in 98.5 percent of splits. The new audit tests validity on stored activations. On every ladder the averaged trend falls inside the task-permutation null, which keeps the framings' words and shuffles their labels. With the signed statistic where the maximum falls, it reverses on every ladder. A single pair's comparison also changes with the pair and the layer set, and each wrapper's score moves under rewording.

We agree that the classes differ in surface form, as Section 10 measures. Section 11 runs your controlled experiment: the same tasks, byte for byte, under evaluation and deployment framings, on 19 checkpoints.

Unseen framing pairs oriented correctlyShare
Evaluation-minus-deployment direction77 to 100%
Input embeddings alone (the lexical floor)96 to 100%

So yes, this control distinguishes the two, and it attributes the orientation signal to the framings' words. The same audit tells any study on a benchmark whose classes differ in wording what its probe measures.

Q1.

How does this work extend prior research on prompt sensitivity in activation probing, particularly Hua et al. (2026), which already uses multiple prompt pairs for evaluation awareness?

Answered with W1 above.

Q2.

Since section 10 demonstrates substantial task-format confounding in the SAD oversight benchmark (0.993 bag-of-words AUROC), it would be useful to evaluate the probe directions on a controlled dataset with identical underlying tasks and varied evaluation/deployment framings. Without this control, is it possible to distinguish genuine evaluation-awareness representations from superficial differences in prompt format?

Answered with W2 above.

The revised abstract, Sections 2 and 6, the new Section 11, the inline definitions, the running example and the recommendations reflect your review. We would be glad to know whether the reframed claim addresses your concern. If it does, we would appreciate it if you took the revision into account in your final score.

AuthorsOpenReview ↗

Follow-up for Reviewer 5WB2

Following your review, we ran new analyses that sharpen the paper's contribution. We crossed framings over byte-identical task text on 19 checkpoints and four ladders. The choice of prompt pair reverses the sign of the published scaling result on every ladder, while the model explains at most 13.1%13.1\%% of the variance. This separates our result from Hua et al. (2026), who average pairs to steer one model. The paper now also includes an audit that detects the effect. We would be glad to hear whether these results resolve your concerns, and if so, we would be grateful if you would consider revising your rating.

Thank you for the clarifications and additional experiments. I agree that the paper provides a useful audit of prior scaling claims, and the rebuttal addresses my concern about its intended contribution. While this strengthens the paper, the contribution remains narrow, and my overall assessment remains unchanged.

Reviewer_cKQ5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper applies Generalizability Theory (G-theory) to audit contrastive difference-in-means probes for evaluation awareness. Crossing 6 evaluation framings with 6 deployment framings (36 wrappers) around byte-identical task text across four model scaling ladders (Qwen2.5, Qwen3, Qwen3.5, OLMo-2), the authors decompose variance in max⁡ℓ∣AUROCℓ−0.5∣\max_\ell |\text{AUROC}_\ell - 0.5| across models, wrappers, items, and implementation choices, and evaluate scores against a label-permuted content-free floor.

Critical strengths
  1. The 6×66 \times 6 crossed wrapper design with byte-identical task text cleanly isolates prompt framing from item content, showing that wrapper choice reverses the sign of scaling correlations (r∈[−0.99,+0.94]r \in [-0.99, +0.94]) on every model ladder.
  2. The G-theory variance decomposition and the derivation of the flat-analysis degrees-of-freedom bias (Appendix B) provide a rigorous framework for measuring benchmark reliability (Eρ2E\rho^2), supported by pre-committed out-of-sample predictions on Qwen3.5.
  3. The paired sign-flip control d(s)=n−1∑isi(ai−bi)d(s) = n^{-1}\sum_i s_i(a_i - b_i) and the analytical derivation of arbitrary-direction separation demonstrate that surface-form differences in stages_oversight (bag-of-words AUROC=0.993\text{AUROC} = 0.993) account for 70%–116%70\%–116\% of published scores.
Critical weaknesses
  1. Sections 4–9 decompose variance by scoring directions on stages_oversight, which Section 10 shows is dominated by surface-form confounds. It is unclear how much of the large model×wrapper\text{model} \times \text{wrapper} interaction (σmw2\sigma_{mw}^2) is driven by wrappers randomly aligning with the surface-form difference of stages_oversight versus genuine differences in how models represent evaluation framing.
  2. The proposed remedy averages the folded scalar statistic max⁡ℓ∣AUROCℓ−0.5∣\max_\ell |\text{AUROC}_\ell - 0.5| over kk wrappers. Folding discards polarity (treating AUROC=0.2\text{AUROC} = 0.2 and 0.80.8 identically), and scalar averaging does not test whether averaging the unit-normalized direction vectors dˉℓ=1k∑wd^ℓ,w\bar{d}_\ell = \frac{1}{k}\sum_w \hat{d}_{\ell,w} in activation space cancels wrapper-specific subspaces (the 71.4%71.4\% arm-specific direction variance in Section 5).
  3. Qwen2.5 directions pool all tokens while other ladders drop BOS and whitespace tokens (Table 3). In addition, the prose is very compressed, which makes transitions between fixed-item and random-item Eρ2E\rho^2 (Section 9) hard to follow.
Questions for authors
  1. If we average the contrastive vectors dˉℓ=1k∑w=1kdℓ,w\bar{d}_\ell = \frac{1}{k}\sum_{w=1}^k d_{\ell,w} across kk wrappers before scoring on the test set, does Eρ2E\rho^2 rise faster with kk than averaging the scalar statistics?
  2. What are the G-theory variance components (σm2,σmw2\sigma_m^2, \sigma_{mw}^2) if directions from wrapper w1w_1 are scored on held-out byte-identical items wrapped in w2w_2 (w1≠w2w_1 \neq w_2) instead of stages_oversight?
  3. Do the Qwen2.5 variance shares in Table 1 change if recomputed with the BOS-and-whitespace-dropped convention used for the other ladders?
2 comments
AuthorsOpenReview ↗

Response to Reviewer cKQ5

Thank you for the review, and for noting that the crossed design "cleanly isolates prompt framing from item content". Every number you reviewed stands, and every analysis you requested is in the revised PDF (Q2 with a substitute statistic, explained below).

W1.

Sections 4–9 decompose variance by scoring directions on stages_oversight, which Section 10 shows is dominated by surface-form confounds. It is unclear how much of the large model ×\times wrapper interaction (σmw2\sigma^2_{mw}) is driven by wrappers randomly aligning with the surface-form difference of stages_oversight versus genuine differences in how models represent evaluation framing.

On three of four ladders the wording drives it, and the revised Section 6 now says so.

Percentile of the real model ×\times wrapper shareTask-permutation null (same words, shuffled labels)Matched rotations
Qwen362nd1st (87th with thinking)
Qwen3.514th0th
OLMo-221st2nd
Qwen2.596th37th

Rotations drop the words and, without thinking, produce a larger share than the real directions. Qwen2.5 is the exception, and there the real share also sits at the 99th percentile of framing relabellings. The interaction is real and reproducible, and it belongs to the prompt.

W2.

The proposed remedy averages the folded scalar statistic max⁡ℓ∣AUROCℓ−0.5∣\max_\ell |\mathrm{AUROC}_\ell - 0.5| over kk wrappers. Folding discards polarity (treating AUROC = 0.2 and 0.8 identically), and scalar averaging does not test whether averaging the unit-normalized direction vectors dˉℓ=1k∑wd^ℓ,w\bar d_\ell = \frac{1}{k} \sum_w \hat d_{\ell,w} in activation space cancels wrapper-specific subspaces (the 71.4% arm-specific direction variance in Section 5).

We agree on folding, and Table 1 now prints both versions. Unfolded where its maximum is read, the statistic points the wrong way in 32 to 88 percent of model-by-wrapper cells. The averaged trend then reverses on every ladder, and on every ladder except Qwen3.5 under the released states 1 to L−1L-1.

Averaging scalars does better on three of four ladders (Table 8, same random partitions, non-thinking renderings).

Eρ2E\rho^2 at k=12k = 12Vectors averagedScalars averaged
Qwen30.520.36
Qwen3.50.300.62
OLMo-20.330.61
Qwen2.50.000.11

The paper keeps the scalar average, whose measured values the D-study predicts within 0.03.

W3.

Qwen2.5 directions pool all tokens while other ladders drop BOS and whitespace tokens (Table 3). In addition, the prose is very compressed, which makes transitions between fixed-item and random-item Eρ2E\rho^2 (Section 9) hard to follow.

On pooling, the conclusions do not change (Table 9, Appendix Q).

Qwen2.5All tokens (submitted)BOS, whitespace dropped
Model share0.8%1.6%
Model ×\times wrapper38.3%38.8%
Eρ2E\rho^2 (k=1k = 1)0.0180.034
kk for 0.80nevernever
r>0r > 021 of 3621 of 36

Per-wrapper slopes correlate at 0.975 across conventions (4 of 36 change sign). The validated arrays of all four rungs reproduce the submitted all-token arrays bit for bit, and so does a fresh re-extraction of the three smaller rungs.

We agree on the prose. Section 9 now opens by naming the fixed-items and random-items readings and what each resamples.

Q1.

If we average the contrastive vectors dˉℓ=1k∑w=1kdℓ,w\bar d_\ell = \frac{1}{k} \sum_{w=1}^{k} d_{\ell,w} across kk wrappers before scoring on the test set, does Eρ2E\rho^2 rise faster with kk than averaging the scalar statistics?

Answered with W2 above.

Q2.

What are the G-theory variance components (σm2\sigma^2_m, σmw2\sigma^2_{mw}) if directions from wrapper w1w_1 are scored on held-out byte-identical items wrapped in w2w_2 (w1≠w2w_1 \neq w_2) instead of stages_oversight?

Both components are small on byte-identical items. The folded statistic is degenerate there: every direction, random ones included, separates two framings perfectly at some layer. So we score a direction's alignment with the held-out framing difference, registered as an amendment before computing.

Share of varianceAs computedShared-framing cells imputed
Model0.1 to 1.4%up to 4.0%
Model ×\times wrapper0.4 to 1.3%2.8 to 7.4%
Pairing of training and test framings77 to 97%38 to 46%

On the standard set, model by wrapper takes 10 to 38 percent of the folded statistic's variance (Section 11, Appendix R).

Q3.

Do the Qwen2.5 variance shares in Table 1 change if recomputed with the BOS-and-whitespace-dropped convention used for the other ladders?

Answered with W3 above.

The abstract, Sections 3, 6 and 9, the new Section 11, Table 1 and the appendices are revised accordingly. We are happy to discuss any of these points further.

AuthorsOpenReview ↗

Follow-up for Reviewer cKQ5

We ran the two analyses you asked for, and both are now in the paper. Matched rotations show that the wording drives the model ×\times wrapper interaction on three of four ladders (Section 6). The unfolded statistic, now reported beside the folded one, reverses the averaged trend on every ladder (Table 1). Together they make the paper's central claim more direct. We would be glad to hear whether these results resolve your concerns, and if so, we would be grateful if you would consider revising your rating.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to all reviewers (1/2): main results

We thank the three reviewers. All three asked for an experiment our crossed design makes possible, and we ran it. The main results:

  • The prompt pair sets the sign. On the benchmark both published scaling studies use, changing the prompt pair reverses the sign of the scaling result. Our design isolates the prompt pair: byte-identical task text, six evaluation framings crossed with six deployment framings, four ladders from two developers. This design is what we add to Hua et al. (2026) and Devbunova (2026).
  • The statistic hides direction. Unfolding it reverses the averaged trend on every ladder (Qwen3: +0.80 to -0.76). On each ladder, 32 to 88 percent of model-by-wrapper cells separate the classes the wrong way round.
  • Two tools at no extra compute. Both run on stored activations: the number of prompt pairs a comparison needs, and a three-part validity audit.
  • New: the audit applied to behaviour. The 36-pair average tracks the SAD stages-oversight accuracy (Laine et al., 2024) of 19 models at Spearman +0.63 [+0.40, +0.72], above every geometry-matched rotation. Directions with the same words and shuffled labels reach a median of +0.60, and model size alone +0.89. Scale and wording produce this agreement, so a behavioural anchor needs the same controls before it can validate a probe.

Reliability and validity. All three reviews ask about it, and 5WB2 put it most directly. Averaging many prompt pairs makes a comparison between models reproduce, which the G-study measures as reliability. Whether the comparison measures evaluation awareness is validity, a separate question. The new audit answers it on this benchmark:

CheckResult
Task-permutation null (same words, shuffled labels)Averaged trend inside it on every ladder (12th to 78th percentile)
Signed statistic where the maximum fallsAveraged trend reverses on every ladder
Orientation at the input embeddings96 to 100% of unseen framing pairs on byte-identical tasks (trained direction: 77 to 100%)

Orientation is therefore a lexical check, and the paper reports it as one. A wrapper average is reliable on OLMo-2 and Qwen3.5, where averages from disjoint framings agree on the scaling sign in 98.5 percent of splits.

Other results.

  • Each wrapper's scaling slope reproduces across item halves (Spearman-Brown 0.75 to 0.90). Task-permutation directions reverse just as reproducibly, so the reversal is no evidence about evaluation content.
  • Rewording moves the folded score by a share comparable to a change of scenario, and reorders the models (11 rungs of Qwen3 and Qwen3.5, up to 32B, descriptive). The signed value and the orientation stay stable.
AuthorsOpenReview ↗

Response to all reviewers (2/2): correction, registration and changes to the PDF

Correction to the layer description. The submitted Section 3, Table 3 and Appendix J said the maximum runs over 16 to 64 layers, "as the released implementations take it". On the ladders it runs over all 17 to 65 hidden states, including the embedding output and the last hidden state. The released per-layer files, like our Section 10 controls, cover states 1 to L−1L-1. Every reported number used the code as released, and the revision corrects Section 3, Table 3 and Appendix J.

Layer setScaling signs
Without both states: Qwen2.5, Qwen3, OLMo-2Both signs remain
Without the last state: Qwen3.534 of 36 wrappers rise (all 36 with thinking on)
Interior band: every ladderBoth signs remain

On Qwen3.5 most falling wrappers come from the last hidden state.

Registration and disclosures.

  • Registered in docs/ before computing: the controlled-transfer, interior-band, paraphrase and sign-mechanism analyses, and the behavioural comparison.
  • Labelled descriptive: the October audit (matched nulls, and a first framing-disjoint split scored with Kendall τ\tau) and the later checks in docs/post-submission-checks.md. The registered framing-disjoint test adds the task-permutation null.
  • Under the registered interior band, the averaged trend stays inside the task-permutation null on every ladder (17th to 68th percentile). It is distinguishable from rotations only on Qwen3 (above them), and from framing relabellings only on OLMo-2 (below them, 2nd percentile).
  • Two registered statistics did not work as designed, and the paper reports both. Separability on byte-identical tasks ties at its ceiling for every direction, so an amendment replaced it with orientation and alignment before those were computed. The paraphrase reproduction gate passes on the five largest models and failed on three of the six smaller ones (reply to J3ye).
  • The registered test of what sets a wrapper's sign was negative.
  • From our own audit: the Qwen2.5 ladder uses the released 400-item split, whose testing half is SAD's test_oversight file. The other three ladders drew theirs from SAD's oversight file. The two share 78 of 200 testing items. Results within a ladder are unaffected. Statements that compare ladders compare different testing items, and the revision says so where they appear.

Changes to the PDF (Blue marks results and content added in this revision. Changes of wording are not marked.):

  • Abstract: the decision-study projection, and the audit's result that the framings' words alone can produce the averaged trend on every family.
  • Introduction and a short new Conclusion: reliable separated from valid, in the same words.
  • Section 2: the paper positioned against Hua et al. (2026) and Devbunova (2026).
  • Section 3, Table 3 and Appendix J: the corrected layer description.
  • Section 6: the read layer, with its distribution in Appendix O.
  • Section 9: a plainer opening, and a new reading of disjoint wrapper sets.
  • Section 11, new: "What holds on byte-identical tasks", one page. About a page of Sections 5 and 7 moves to the appendix.
  • Table 1: the signed statistic beside the folded one, and the flat and crossed wrapper counts.
  • Appendices: the matched nulls by null, layer sets, null geometry, Qwen2.5 pooling, vector versus scalar averaging, framing-disjoint subsets, paraphrases.
  • Recommendations: run the validity audit first. Score the autocorrelation-matched floor beside the label-permuted one. Sample the framings of the scored items as well as the wrappers. Read k from the crossed analysis once a comparison passes the audit.
  • Presentation: each term defined in plain words where it first appears, and one running example from Figure 1 to the Conclusion.

Unchanged. Every number in the submission's Sections 6 to 11 (now Sections 6 to 10 and 12). Four readings change:

  • Section 6 no longer reads the model-by-wrapper term as models differing in how they represent evaluation.
  • Section 9 says that shared framings inflated the agreement between disjoint wrapper sets.
  • R2: analysed crossed, no number of wrappers reaches Eρ2=0.80E\rho^2 = 0.80 on Qwen3 or Qwen2.5. The submitted R2 said "6 to more than 18".
  • Section 10 reads the sign-flip floor beside a new autocorrelation-matched floor, content-free by construction, which reaches 63 to 133% of the published values: the same picture.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.