acceptodds
Under review as a conference paper at ICLR 2027

Content-Matched Is Not Lexically Matched: Text and Untrained-Network Baselines for Evaluation-Awareness Probing

21 views

Abstract

Linear probes are increasingly read as evidence that language models represent whether they are being evaluated. Content-matched designs, which hold a task fixed and switch one cue at a time, are meant to make that evidence strong. The assumption is that a probe separating the two versions has detected the model recognising the cue, rather than the cue itself. We test that assumption under a pre-registered protocol on 17 open models. Matching content is not matching words: a bag-of-words classifier given the probe's labels and splits separates all eight cues of EvalAwareBench (median AUROC 0.89–1.00); replacing "Marcus Chen" with "John Smith" is the manipulation, and a word counter sees it. The standard per-cue probe test does not need learned weights either: 13 untrained networks pass it on every cue and also separate a mixed-source evaluation/deployment set. Probe success on such benchmarks therefore cannot, by itself, show that a model represents being evaluated. Two results survive both baselines. Activations add a small but robust increment beyond the words, larger in trained than in untrained networks. And when both conditions use unseen words, probes on trained networks, like pretrained sentence encoders, still separate real from placeholder entities, while word models and untrained networks do not: they read a learned property of the cue, not evaluation. We release lexcheck and recommend a same-split text baseline and an untrained-network baseline for every probe claim.

Primary area:
interpretability and explainable AI
Keywords:
evaluation awareness, linear probes, interpretability, probing baselines, untrained networks, lexical baselines, measurement validity, pre-registration, large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yE5WOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether probes can show that an LLM knows it is being evaluated. The authors use prompt pairs that ask for the same task but differ in features such as names or formatting. They fit probes to distinguish these versions using the LLM’s hidden states. They compare the results with text classifiers and probes on untrained networks. Both baselines often succeed, showing that simple input differences can explain successful probing. Trained LLMs provide some additional predictive value. Their probes also generalize to unfamiliar placeholders. However, recognizing placeholders does not establish that the LLM realizes it is being tested. The paper argues that evaluation awareness requires evidence beyond probe accuracy.

Critical strengths
  1. The controls directly challenge the interpretation of probe success.
  2. The paper shows that removing familiar positive cues does not remove all shortcuts. Words that disappear from the opposite class can still reveal the answer.
  3. The paper makes its findings more reliable by testing different data splits, preventing information leakage between training and testing, and checking whether combining results produces misleading scores.
Critical weaknesses
  1. The lexically matched test (B1′) does not remove all surface clues. Although it introduces new entity words, shared character patterns still help a simple classifier distinguish the groups. This makes it harder to determine how much of the probe’s success comes from recognizing meaning rather than spelling patterns.
  2. The lexically matched test (B1′) focuses only on realistic versus placeholder entities. Its results do not establish similar generalization for other cues, such as coherence, completeness, or stakes.
  3. The random-network controls use only two small models with 0.5B and 1B parameters. They do not include random versions of the 7B–8B models. Therefore, the trained-versus-untrained comparison remains untested at larger model sizes.
Questions for authors
  1. Does the probe retain its advantage on B1′ when spelling patterns, capitalization, punctuation, and token lengths are also balanced between realistic entities and placeholders?
  2. Can a probe trained on explicit evaluation disclosure generalize to other cues, such as benchmark-style formatting, or does successful classification require a separate probe for each cue?
1 comment
AuthorsOpenReview ↗

Response to Reviewer yE5W

Thank you for your review and for appreciating how our controls challenge the reading of probe success. Your points helped us understand our own work better while providing useful directions to conduct experiments. We updated the paper and the supplementary code with the same rigour and transparency as before.

Key changes in our revision related to your review (details in our general comment): Each of your mentioned weaknesses led to a new experiment. As a result B1$'$ is withdrawn as evidence (W1). A new matched test on another cue holds (W2). Untrained 7B networks are now included (W3). We address your weaknesses and questions in order.

W1 and Q1: surface clues in B1$'$

You were right: on B1$'$ the probe's advantage does not survive these surface features. We thank you for this observation.

  1. What we ran: a model on six counts per prompt (tokens, words, capital letters, punctuation, digits and single capital letters), trained on the original contrast and tested on B1$'$ over 50 splits.
  2. What we found: the counts separate B1$'$ (median AUROC 0.963). Single capital letters alone reach 0.966, consistent with placeholders such as "Person Q". The probe beats every text model, these counts included, in only 9 of 50 splits (registered prediction E1-P1, not supported). So we withdrew B1$'$ as evidence (Section 7).
  3. What holds instead: we did not rebuild B1$'$ with these features balanced (planned for future work, Appendix B.2). The new matched test in W2 keeps word, character and count models near chance. There probes on trained networks still separate the pair, also without capital letters.

W2: other cues

We agree that the revision adds a matched test for another cue, so the matched evidence still covers a single cue.

  1. What we ran: a matched test for explicit evaluation disclosure (F8; Section 7). Its answer-format endings are replaced by new ones that share no word with them (drafted with AI assistance).
  2. What we found: probes on all 11 trained networks still separate them from neutral remarks (medians 0.815 to 1.000). Untrained networks do not. Our further checks suggest the probe mainly reads the style of an instruction, not evaluation.
  3. Other cues: for coherence and completeness a word-matched version is not defined, because these cues add or remove whole passages. For stakes a synonym rewrite stayed readable by the word model (0.778).

W3: untrained controls only at 0.5B and 1B

We added untrained 7B networks. They behave like the small ones.

  1. What we ran: untrained copies of Qwen2.5-7B (three seeds) and Mistral-7B (one), through the same pipeline (Section 5, Appendix B.2).
  2. What we found: every one passes the per-cue test on all eight cues, while the trained Mistral passes four. The extra signal beyond the words is learned at this size: the trained 7B's increment exceeds each untrained seed's in every split for F2 and F5. On the new matched test the untrained 7B networks do not separate the pair. All registered predictions of the 7B experiment are supported. We did not run an untrained 8B.

Q2: does a direction for one cue work on another?

For your references, mostly yes in trained networks. Across all cues, each cue needs its own direction.

  1. What we ran: each cue's direction scored on the other seven cues, in every network (E2, registered).
  2. What we found: the F8 direction scores 0.663 to 0.999 on task format (F7) in 21 trained networks below 7B and 0.996 at 7B (means over 50 splits). The word model reaches 0.551. Across all pairs of cues, transfer beat the word model clearly (by at least 0.05) in only 13 of 21 networks, short of the 17 we registered (Appendix B.2).

If you have any further suggestions or questions, please let us know.

Reviewer_1htSOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper audits the evidentiary value of linear probes used to claim that language models internally represent “being evaluated.” Using EvalAwareBench, a factor-controlled benchmark that flips one of eight evaluation cues at a time while holding task content fixed, the authors run two baselines alongside standard residual-stream probes on 17 open models (0.5B–8B): (1) a bag-of-words TF-IDF classifier given the same labels and splits as the probe, and (2) untrained copies of the same networks (plus trained-embeddings/random-layers hybrids). The findings: the word model separates all eight cues (median AUROC 0.89–1.00), so content matching does not mean lexical matching; all 13 untrained networks pass the field’s standard per-cue probe test on all eight cues and fall within the trained range on a mixed-source evaluation/deployment set; what survives both baselines is a small (+0.03–0.07 AUROC) but robust incremental contribution of trained activations on two or three cues, and a newly constructed lexically matched test (B1′) in which both conditions use words unseen in training — there, word models drop to chance while probes on trained networks (and pretrained sentence encoders) still separate real from placeholder entities, and untrained networks mostly do not. The authors conclude probes read a learned property of the cue (real-vs-placeholder entities), not “evaluation,” release a tool (lexcheck), and propose a reporting protocol. The confirmatory analyses were pre-registered with a dated amendment trail; 21 of 40 predictions supported, all reported.

Critical strengths

The methodological rigor is impressive. Pre-registration with 11 dated amendments, each demonstrably preceding the data it governs; 40 predictions scored, with 14 failures reported prominently rather than buried — several of which narrow the authors’ own claims (e.g., the ABSTRACT classification on B1′ holds in only 12/50 splits and is reported as split-dependent; one untrained seed weakly reads B1′, so the clean trained/untrained dichotomy is softened). The positive control (B2, held-out animal/tool words) validates that the pipeline can detect word-independent signal, which inoculates against the objection that the baselines are just too weak. Bit-for-bit activation reproduction, checksummed prompt sets, a released tool with two-minute reproduction of the core table, and 50-split robustness everywhere. The scientific writing is unusually precise about what each result does and does not license. The practical payoff (Table 3’s reporting protocol plus lexcheck) is immediately usable by the field, and the central negative results have direct consequences for published steering and suppression work built on these directions.

Critical weaknesses

Single benchmark, single readout. Everything rests on EvalAwareBench (plus one mixture set), one token position (last prompt token), and linear readouts. The conclusions are explicitly scoped to this, but the title’s general claim (“content-matched is not lexically matched”) is established in detail for one benchmark family; the full-factorial result and the Noël-dataset lexcheck run help, but a second factor-controlled benchmark would have made the generality claim much stronger. The lexically matched test covers only one cue (F1), as the limitations section concedes — so the paper’s most interesting constructive result (probes read a learned entity-realness property) is demonstrated for a single cue.

Novelty The individual tools are not new — text baselines, untrained-network controls, incremental validity, and lexically matched test sets all exist. The novelty is in the combination and the target: no prior evaluation-awareness probe study ran either baseline, the two-sided lexical-signature observation is a nice useful methodological insight, and the B1′ construction (new vocabulary on both sides) is an empirical modest contribution that goes beyond keyword-held-out designs. The finding that untrained networks pass the full three-part per-cue test that published studies use piecemeal is a sharp negative result extended from SAE metrics to this probing protocol. The novelty is therefore adequate-to-good for a methods/analysis paper: not a new technique, but a new and consequential application with one new test construction.

Questions for authors

See the weaknesses

1 comment
AuthorsOpenReview ↗

Response to Reviewer 1htS

Thank you for your detailed review and for appreciating the rigour of our work. Your points showed us what else to test and where our claims needed limits. We updated the paper and the supplementary code with the same rigour and transparency as before. Your questions refer to the weaknesses, so we answer them together, in order.

Key updates in our revision related to your review (details in our general comment)

  1. B1$'$ no longer holds. New tests showed that six surface counts (tokens, words, capitals, punctuation, digits, single capitals) separate it (median 0.963 over 50 splits). The LR probe beats every text model in only 9 of 50 splits (E1-P1, not supported). So the entity-realness result does not hold beyond surface form. The matched test that holds is now a new one (Weakness 2).
  2. Untrained copies at 7B. In the submission only the 0.5B and 1B networks had untrained copies. The revision adds Qwen2.5-7B (three seeds) and Mistral-7B. All pass the per-cue test on all eight cues.
  3. Mixture wording. Untrained networks fall within or just below the trained range, not all within it. This was our wording. The revision corrects it.

Weakness 1: single benchmark, single readout

The depth on one benchmark was a trade-off: every control on every cue rather than fewer controls on more benchmarks. We kept the title and scoped the text instead: Section 10 now says we test one benchmark, a second paired design and English prompts only. Based on this weakness we did three experiments.

Your concernWhat we ranWhat we found
One benchmarkE5: the paired framings of Noël (2026), 720 promptsUntrained networks reach a median DoM AUROC of 1.000. Text models also reach 1.000 with whole framings held out. So trained probes cannot show anything beyond text there.
Last token onlyE7: mean over the prompt's tokensUntrained networks still pass 7 of 8 cues (every seed). At 7B the LR increment on F5 remains (+0.033+0.033 [+0.008+0.008, +0.059+0.059]). The DoM increment does not.
Linear readoutsE6: an MLP probeThe learned increment stays: the trained twin is above each untrained seed in 50 of 50 splits.

In short, the main results mostly hold under mean pooling and an MLP probe. Two registered B1$'$ predictions under these readouts failed (E6-P1, E7-P3). The second design is paired, not factor controlled. So the generality claim still rests on one factor-controlled benchmark.

Weakness 2: the matched test covers one cue

With B1$'$ withdrawn, the matched test that holds is again a single cue. It is a new test on a different cue.

1.What we ran: E4 replaces the answer-format endings of explicit evaluation disclosure (F8) with 25 new ones that share no word with them. Probes and text models are trained on the original contrast. They are tested on the new endings against neutral remarks.

ModelMedian AUROC on the new pair
Word, character and six-count models0.516, 0.531, 0.513
F8 LR probe, 11 trained networks (4 registered, 7 exploratory)0.815 to 1.000
Untrained networks0.408 to 0.540 (4 registered); up to 0.613 (exploratory)
Sentence encoders (exploratory)0.704, 0.815

Trained networks separate the new pair. Untrained networks and simple text models do not. Exploratory checks suggest the direction mainly reads instruction style, not evaluation. Its registered confirmation on fresh endings (E4e) was withdrawn before any data. For F2 and F4 a word-matched version is not defined, because they add or remove whole passages. For F5 a synonym rewrite stayed readable by the word model (0.778).

Weakness 3: novelty

We agree that the tools are known. As you also mentioned, the contribution lies in the target and the combination. Also Noël (2026) reports a bag-of-words baseline but notes that its design cannot compare it fairly with the probe. Our comparison uses the same labels and splits. Concurrent work has since added text baselines. Based on our suvey and research, we know of none with an untrained network.

We think the work remains useful & relevant to the field. The untrained-network result you called sharp now holds from 0.5B to 7B. The reporting protocol (Table 3) and lexcheck, which is "immediately usable", remain in the paper. The revision adds a matched test that untrained networks fail. So the paper now offers a positive check as well as a warning. The B1$'$ construction stays in the paper as a record. It no longer counts as evidence.

If you have any further suggestions or questions, please let us know.

Reviewer_4TMnOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This study question an increasing popular claim that LLM know when they are being evaluated. Specifically, this paper shows that the probing technique used to identify such ability, often just reflect differences in the context. For example, they show that word counting and untrained networks can separate the prompts with this type of method.

After controlling for some of these issues, the authors show that only a small amount of learned signal about the cues seem to remain.

Critical strengths
  • The study is transparent and well controlled: its main tests, thresholds, and predictions are claimed to be pre-registered (although I didn't go check), amendments were dated, and failed predictions were reported rather than hidden.

  • The paper provides a useful lexical audit of EvalAwareBench: content-matched prompts appear to be identifiable from added and removed words. Simple word-level controls are seem essential before interpreting probe results. In addition, its matched-vocabulary analysis shows that trained models encode a general entity distinction.

Critical weaknesses

Contribution:

The paper’s central methodological warning is already well established in the probing literature. The contribution is thus not the discovery that probes can exploit wording (or work in untrained networks), but that on a specific benchmark, EvalAwareBench, probe results can still suffer from the same long-standing confounds.

Overall, while this appears to me as a rigorous methodological study with a useful warning, the contribution is narrow and largely confirmatory. That makes it arguably better suited to a workshop or a benchmarks/negative-results venue.

Questions for authors

Given that (text and untrained-network) baselines are already standard probing controls, what is the contribution of the present paper?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 4TMn

Thank you for your review and your comments on the transparency and rigour of our work. We read your points thoroughly. Your question on the contribution were one of the many concerns for us to test a second paired design and to state more clearly what the work adds. We kept the same standard as before: each new test was registered before it ran and its code is in the supplementary material.

Key updates in our revision related to your review (details in our general summary comment)

  1. New tests showed that the matched-vocabulary result (B1$'$) does not hold. Simple counts such as the number of capital letters already separate it. We no longer count it as evidence. The revision also adds a new matched test that holds: probes on trained networks pass it and probes on untrained networks fail it (Section 7).
  2. Untrained 7B networks also pass the standard probe checks (Section 5).

Weakness: the contribution is narrow and confirmatory

We agree on both points. First, text and untrained-network baselines are standard in probing and we do not claim they are new (Section 2 cites this work). Second, the narrow scope is a trade-off we chose given our resources: one benchmark tested in depth with every control, plus a second paired design. We only count a result if it beats both baselines.

We think the work's relevance is mainly where these baselines are used. As your summary notes, the claim that models know when they are being evaluated is increasingly popular. Probes are one of the things used as evidence for it. In our survey of these probe studies in September, none compared its probe with these baselines on the same data. Only text baselines have appeared since. We know of none with an untrained network. Together, the two baselines test what a single one cannot: whether a probe shows more than the text and more than an untrained network. In our tests:

  • A simple word model, tested on tasks it has not seen, tells apart the two versions of the prompts for all eight cues.
  • All 17 untrained networks in our pipeline, from 0.5B to 7B parameters, pass the checks that studies use to show a probe found something real.

We also have reasons to believe the work is not only confirmatory: the new matched test is a positive result (Section 7).

Question: given that these baselines are standard, what is the contribution?

We believe the contribution is not the baselines themselves rather what they show when applied to evaluation-awareness probes:

  1. The usual probe checks for evaluation awareness pass without any learned weights.
  2. The benchmark's matched prompts can be told apart from their words alone.
  3. A matched test that trained networks pass and untrained networks fail. Our further checks suggest it mainly reads the style of an instruction, not evaluation. It covers one cue.
  4. A simple released check (lexcheck) and a recommendation: report a text model and an untrained copy next to every probe result.

As studies of evaluation awareness grow, we hope these baselines help readers judge what a probe result does and does not show and use the rigor to properly infer such results.

If you have any further suggestions or questions, please let us know.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary: Post reviews

We thank all three reviewers for their careful and detailed reviews. We are glad that the reviews noted the rigour of the study: the registered tests, the controls and the reporting of failed predictions. Our aim was to do meaningful and useful research for which we believe readers need to see what failed as well as what worked.

After the reviews we studied each point and ran new experiments (where required) to the same standard as the submitted study. We ran eight new analyses (Appendix B.2, Table 10). Each was registered in a dated amendment before it ran (Appendix A). Three of the new registered predictions failed. Table 5 lists them with all the others. The registrations, code and logs are in the supplementary material (folder discussion_2026-11/; VERIFY.md explains how to check them).

What we found post reviews

  • **New tests showed that one earlier result does not hold (B1$').∗∗InB1).** In B1'$ both sides use new words. We had not tested counts such as capital letters or punctuation on it. When we did, six simple counts separated its two sides (median AUROC 0.963, where 0.5 is chance). So B1$'$ cannot show more than surface form. We no longer count it as evidence (Section 7).
  • A new matched test that only trained networks pass (E4). We replaced the answer-format endings of one cue, explicit evaluation disclosure, with 25 new endings that share no word with the old ones. They were drafted with AI assistance. Simple text models (words, characters, counts) do not separate them from neutral remarks. Probes on all 11 trained networks we tested do. Probes on untrained networks do not. Further checks suggest the probe mainly reads the style of an instruction, not evaluation. This test covers one cue (Section 7).
  • Untrained networks pass the standard checks at 7B too. Untrained copies of Qwen2.5-7B and Mistral-7B pass the per-cue test on all eight cues. The small extra signal of trained networks beyond the words also holds at this size (Sections 5 and 6).
  • The same problem appears on a second set of paired prompts. On the prompts of Noël (2026), untrained networks and word models both separate the two classes perfectly (Appendix B.2).
  • Other probe types give the same picture. When the probe reads an average over all tokens, untrained networks still pass on 7 of 8 cues. A non-linear probe keeps the extra signal of trained networks (Appendix B.2).
  • A probe for one cue does not replace the others. By our registered test, each cue needs its own probe direction (Appendix B.2).

How the paper is updated

  • The abstract, the introduction and Sections 5 to 7 report these results.
  • Section 10 states the scope: one benchmark, a second paired design and English prompts only.
  • Appendix A adds the new amendments, deviations and timeline. Appendix B.2 holds all new analyses. Appendix E explains how the new endings were built.
  • Smaller corrections. For example, on the mixed evaluation/deployment set untrained networks fall "within or just below" the trained range.
  • Some tables and figures moved into the sections that discuss them. The submitted Tables 10 and 11 are now Tables 12 and 13.

Updates in Future work: B1$'$ rebuilt with spelling, capitals, punctuation and token length balanced; matched tests for more cues; causal tests.

If reviewers suggest further changes, we are happy to address them and list them in brief before the discussion ends.

Edit (Nov 16, ~10:40 AoE): We uploaded the same revision with new/modified text shown in blue (for more convenience). Apart from the colour, the PDF is identical to the revision we uploaded earlier. Minor changes inside figures such as some table cells and the references are not coloured.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.