acceptodds
Under review as a conference paper at ICLR 2027

Looking Like a Test: What Format Does to Evaluation Awareness in Language Models

22 views

Abstract

Language models often recognize when they are being evaluated, which can make benchmarks unreliable. Yet it remains unclear why a model identifies a conversation as an evaluation: whether format plays a part, and whether changing it changes behavior. To study this, we use judge models from eight families to compare pairs of conversations from evaluations and from deployment and choose the one that seems more like an evaluation. A pairwise preference model turns these choices into a single scale of evaluation-likeness, on which the judges largely agree. We then find what drives these choices by marking each conversation for hand-defined format cues and ranking the cues by how far they move the score. A question that comes with answer options looks like a test to the models, while a conversation with several back-and-forth user turns looks like real use. Because this ranking shows only an association, we rewrite hazardous-knowledge questions into versions that change format one rewriting step at a time. As a result, the question wrapped in exam framing scores higher than any evaluation conversation, while the same question inside a realistic work conversation scores lower than nearly every deployment conversation. Finally, we instruct models in the system prompt to underperform, then give them the hazardous and benign questions rewritten with each format cue. With refusals set aside, the drop in accuracy tracks the evaluation-likeness score of each version in most settings, including models that never served as judges. Together, these results suggest that format shapes how evaluation-like a conversation appears to models, and that presenting evaluation questions in realistic conversations may bring model behavior on benchmarks closer to behavior in deployment.

Primary area:
interpretability and explainable AI
Keywords:
evaluation awareness, sandbagging, LLM evaluation, format sensitivity, AI safety

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_JfnROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

The paper analyses whether LLMs can distinguish evaluation conversations from deployment conversations based on presentation cues. To study this, the paper rewrites benchmark questions (e.g. from MMLU and a biology subset of WMDP) into different formats (MCQA to more open formats), and uses fourteen instruction-tuned judges from eight families to compare pairs of conversations stemming from evaluations and from deployment. Then, a pairwise preference model is used to turn these choices into a single scale of evaluation-likeness. The key findings is that format shapes how evaluation-like a conversation is.

Critical strengths

The paper is easy to follow (however, see note below on anthropomorphism and evaluation setup).

The experimental breadth is good (e.g., in terms of number of judges and benchmarks).

Critical weaknesses

The paper mainly tests whether models recognize different question formats - this makes sense, and results are quiet straightforward. However, there is a big gap to "awareness of evaluation", I am not convinced the experimental evaluation tests what it claims to test.

A key concern is that the rewrites are not single-feature interventions so that the results mix several things. For example, the second prompt mixes in personas, secret keys etc - making it hard to properly disentangle and understand what the results measure.

The rewriting steps are only presented in the appendix, despite being the key experimental manipulation. The main paper (line 251) only reports "Each step applies one rewriting operation (Appendix F.1)."

The feature regression identifies option lists as strongly correlated with evaluation-likeliness. But the setup covers many diverse datasets and topics (genres). To make such a claim a more controlled experiments (for topic) would be needed.

The writing style of the paper is hugely antlophomorphising systems. “Language models often recognize when they are being evaluated” sounds anthropomorphic. Importantly, "recognize" implies awareness that the paper's experiments do not directly establish (it tests evaluation likeliness).

The exact number of items and their origins from the 60 benchmarks remains unclear to me, Appendix A is difficult to follow, also the main text, where the 976 items come from.

Questions for authors

Why did you use this particular set of rewrite steps, and not try to tease apart single factors? (see comment above)

Can you clarify the origin of each of the 976 items?

3 comments

Thank you for the detailed review and comments. We have addressed each point and uploaded a revised version of the paper, with all changes marked in red.

W1 / W5: The gap to “awareness of evaluation” and anthropomorphic wording.

Our definition of evaluation awareness builds on Needham et al. [1], whose corpus we use, where it is measured as a model's ability to classify whether a transcript comes from an evaluation or from a deployment scenario. Other work uses the same term, such as Li et al. [2], though they detect it with chain-of-thought monitoring. In our case, we measure it as evaluation-likeness, through pairwise choices between conversations. In the revised version, we state this operational definition explicitly in the introduction, and we changed the first sentence of the abstract from "recognize when they are being evaluated" to "classify conversations as evaluations". The scale also correlates with behaviour, since the drop in answered accuracy under an instruction to underperform tracks the scale, even for two held-out models (Section 4).

W2 / Q1: The rewrites are not single-feature interventions.

We agree that some renderings are modified by more than one cue, and this is now explained in Section 3.1. However, our key comparisons are based on renderings that change only one feature at a time. For example, Bare-MCQ to Bare-Open removes only the option list, and Bare-Open to Tools-Open adds only a tool list. Exam-MCQ is built using multiple cues on purpose, since it was designed as an upper bound for evaluation-likeness, and the chat renderings necessarily add a speaking voice along with the extra user turns, mimicking deployment-like conversations. We chose these steps to cover the whole scale with both answer interfaces, following the strongest cues from Section 2.

W3: The rewriting steps are only in the appendix.

We agree, and we have moved the table of renderings (each rendering, its parent and the rewriting step) into Section 3 (Table 1).

W4: Option lists may be confounded with topic.

Since our pools are defined by topic, provenance and shape, we ran an additional analysis using the topic and provenance labels as controls in the feature regression, and the option-list weight stays clearly positive, going from +0.37 to +0.33 with topic and +0.29 with both. In general, the directions of the main feature weights are preserved with the controls. Moreover, when we compare only evaluation conversations within each topic that has multiple-choice items, those with an option list are picked as the evaluation far more often (81% against 58% in safety, 81% against 54% in ethics and 84% against 43% in capability). We have added this analysis to Appendix D. Finally, the rewriting experiment has a fixed topic, since it uses the same WMDP questions, and removing only the option list still moves the accuracy substantially.

W6 / Q2: The origin of the 976 items.

All 976 conversations come from the corpus of Needham et al. [1]: 537 evaluation conversations from 41 datasets and 439 deployment conversations from 19 datasets, 60 datasets in total. They curated samples from each dataset. We extended this by organising the dataset into pools. For example, the Hazardous MCQ pool (E4) holds 11 WMDP items and the Knowledge MCQ pool (E16) 12 MMLU items. We have added the number of conversations per source category to Table 2 and simplified Appendix A.

References

[1] J. Needham, G. Edkins, G. Pimpale, H. Bartsch and M. Hobbhahn. Large Language Models Often Know When They Are Being Evaluated. arXiv:2505.23836, 2025.

[2] C. Li, T. J. Zhang, J. Zhang, Z. Jin, F. Tramèr, S. Abdelnabi and M. Andriushchenko. Decomposing and Measuring Evaluation Awareness. arXiv:2605.23055, 2026.

Thanks for the updated manuscript, the added details on the manipulations, the data and topic-based results. This addresses most of my questions and I updated my score (2->3).

Regarding anthropomorphic wording, thanks for acknowledging it and changing the wording. I think this is important (and personally think the original title of Needham's is too anthropomorphic, and we should reduce that going forward). I recommend to add a footnote on your stance towards anthropomorphic word usage.

Thank you for raising your score and for your helpful comments. We agree with your point on anthropomorphic wording, and we will add a footnote about it in the final version of the paper.

Reviewer_xSCCOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper evaluated and scored thirty-three conversation pools using pairwise comparisons and a Bradley-Terry model. They found, using their evaluation-likeness scale, that surface format cues significantly influence how test-like an interaction appears to a model. By rewriting benchmark questions through these formatting steps, they found that the perceived evaluation-likeness of a task can be shifted from a test-like exam to a realistic chat.

Critical strengths
  1. the authors create a metric for how much a conversation looks like an evaluation. this is a contribution that many in the safety/alignment community would find useful.

  2. the authors are very comprehensive in their study setup, and the study is large in scale (33 conversation pools, 8 families of judges, >158,000 trials).

Critical weaknesses
  1. please increase the size of your figures and figure labels. they are very difficult to read.

  2. familiarity with the benchmarks is a stated confounder. it would be interesting to design a control around this.

Questions for authors

how do you anticipate more subtle or adversarially optimized prompt rewrites might impact evaluation awareness?

4 comments

Thank you for the review and for the positive assessment. We have uploaded a revised version of the paper, with all changes marked in red.

W1: Figures.

We have made all figures and labels bigger in the revised version, and split the pool-tournament figure into separate panels.

W2: Familiarity with benchmarks as a confound.

We agree that it is important to rule out alternative explanations, and we designed the experiments with this in mind. All renderings have the same base questions, which are embedded word for word, and change only the way they are presented, so memorisation of the questions alone cannot explain the differences between them.

An alternative explanation might be that models think cues such as multiple-choice options look like tests simply because they appear so often in actual tests in the training data. However, Chat-MCQ keeps the question and options word for word, yet inside a work conversation it falls on the deployment side. In the behavioural test, the same WMDP questions also lose more answered accuracy in the more test-like renderings, so we can see that behaviour does change depending on the modifications made according to the features ranking. It is true that a more expensive but more direct control would render newly written or paraphrased questions, which the judges have not seen during training, and this is a natural next step (which could go inside future work).

Q1: How would more subtle or adversarially optimised rewrites affect evaluation awareness?

Our simple rewrites already reach the deployment end of the scale (Chat-Open scores below all but one real deployment pool). We expect optimised rewrites to push items further in either direction, although there may be less room at the top, since Exam-MCQ, which combines several of the strongest evaluation-like cues, already scores above every evaluation pool.

This has two uses. Evaluators could make benchmarks look like real use, and the same method could produce matched test-like and deployment-like versions of the same item, for example as contrastive pairs for activation steering and more in-depth analysis.

One open question is whether rewritten prompts generalise to models we have not tested. Across the paper we can see evidence that it could generalise, since the judges largely agree with each other about what looks like a test, and the sandbagging results on two held-out models that never served as judges (Section 4.3) suggest that our renderings transfer. Optimised rewrites would need the same check.

Dear Reviewer xSCC,

We hope our rebuttal has addressed your concerns, and we would be happy to clarify anything further.

Best regards, The Authors

We thank the authors for addressing our comments. We would recommend increasing the figure labels even further (to at least the size of the text.

Thank you for the follow-up. We have enlarged the figure labels further in the updated PDF, so that they are now close to the size of the main text.

Reviewer_E19hOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the mechanisms that make LLMs recognize they are being evaluated. The authors extract evaluation-likeness scores from pairwise comparisons from eight model families, and investigate the importance of hand-defined format cues that drives these choices by how far they move the score when used to rewrite each conversation. The authors find that conversations wrapped in exam framing are more likely to be detected as an evaluation than any other evaluation conversation, while the same question wrapped in a realistic framing is less likely to be detected than nearly every deployment conversation. They also show that, when told to underperform when they believe they are being evaluated, accuracy drops significantly in three of four settings, and this largely depends on format.

Critical strengths

The paper is clear, easy to follow, and well organized. The methodology is sound and well-scoped, and the results seem novel. The authors investigate a relevant question through an interesting approach.

Critical weaknesses
  • The claims are limited by the scope: the binary features seem to be hand-designed instead of induced from the data, and may not encompass every aspect that may be associated with the model's awareness of an evaluation. For example, the authors acknowledge the limitation that the current setup doesn't separate whether the conversation resembles a familiar benchmark or the model believes it's being tested.
  • Telling the model it's an "expert evaluation assistant" may be most of what's inflating the evaluation-likeness score of Exam-MCQ.
  • Figures are very small, and shared axes are difficult to align visually.
Questions for authors
  • How were the 22 binary features derived? Were they induced from the data or entirely human-generated?
  • It isn't very clear why the pool grouping is necessary, and if it also groups similar evaluation-likeness. Is there a reason beyond reducing the number of pairs to evaluate?
4 comments

Thank you for the review. We have uploaded a revised version of the paper, with all changes marked in red. Our answers are below.

W1 / Q1: How were the features derived?

We manually extracted/induced the 22 features from the data, instead of introducing them ourselves into the conversations transcript at the first experiment. It is only in the second experiment (Section 3) when we manually edit the conversations adding additional features into the WMDP items (which was done to isolate the effect of the cues, starting all edits from the same base conversation). For example, we attributed to MMLU and WMDP transcripts the features of a lettered option list, a system persona, and a required answer format, among others. We are aware that the list cannot cover the whole cue space, since there are many possible cues, and that the regression only shows associations. This is also the reason why in Section 3 we test the strongest cues directly on fixed WMDP items, to see how it shifts the ranking.

W2: The persona in Exam-MCQ.

Agreed. However, this was intended, since Exam-MCQ is designed to set an upper bound for evaluation-likeness, where we added strong evaluation-like cues obtained in the analysis, which also appear in the initial 976 conversations (including the “You are an expert … assistant” framing), and our format comparisons rest mainly on renderings one step apart, such as Bare-MCQ and Bare-Open.

W3: Figures.

We have made all figures and labels bigger, and split the pool-tournament figure into separate panels.

Q2: Why pools?

We grouped conversations into pools mainly because we think it is a clever way to reduce cost and to stratify, since comparing all 976 conversations pairwise would mean 475,800 pairs per model. With 33 pools there are 528 pool pairs, each with at least 150 trials, which is enough to estimate every pool’s score. As stated in Section 2, the conversations in each pool share three properties: provenance, conversational shape and topic. Pools also allow us to focus on the most interesting and hardest comparisons (evaluation against deployment of the same shape) and to place the rewritten items on the same scale.

We did an additional analysis, which shows that pools also group conversations that look alike to the judges. If we score each conversation by how often it was picked as the evaluation, pool membership explains 67% of the variation in this score, against at most 13% for random groups of the same sizes. Conversations in a pool therefore score alike but not identically, which is why we run the feature analysis per conversation.

Dear Reviewer E19h,

We hope our rebuttal has addressed your concerns, and we would be happy to clarify anything further.

Best regards, The Authors

I thank the authors for the rebuttal, addressing my main concerns, and updating the PDF. I'd still suggest adding horizontal cues to figures 3 and 4 where they share the Y axis.

I'm convinced that this paper presents valid contributions to the community and will update my score (3->4).

Thank you for raising your score and for the helpful suggestion. We have added horizontal guide lines to Figures 3 and 4 where the panels share the y-axis, so each row is easier to follow across panels.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the rebuttal

Dear Area Chair and reviewers,

Thank you for the helpful reviews and the constructive discussion. Your comments made the paper clearer and the analysis stronger. Here is a short summary of the rebuttal and of what changed in the revised PDF, where all changes are marked in red.

New results

  1. Topic and provenance controls (JfnR). With topic and provenance added as controls in the feature regression, the option-list weight stays clearly positive (+0.37, +0.33 with topic, +0.29 with both), and the main feature directions are preserved.
  2. Within-topic comparison (JfnR). Within the same topic, evaluation conversations with an option list are picked as the evaluation much more often than those without one, for example 81% against 58% in safety.
  3. Pools (E19h). Pools also group conversations that look alike to the judges. Pool membership explains 67% of the variation in per-conversation scores, against at most 13% for random groups.

Changes to the paper

  1. The introduction now states our operational definition of evaluation awareness, and the abstract uses less anthropomorphic wording.
  2. The table of rewriting steps is now in Section 3, with a note on which steps change one cue and which change several.
  3. Section 2, Table 2 and Appendix A explain where all 976 conversations come from.
  4. Appendix D adds the topic and provenance analysis.
  5. All figures are larger, and Figures 3 and 4 have guide lines and repeated labels where the panels share an axis.

Thank you again for your time. We would be glad to answer any further questions.

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.