acceptodds
Under review as a conference paper at ICLR 2027

Training LLMs to Verbalize Evaluation Awareness

Abstract

Evaluation awareness (EA) can cause large language models (LLMs) to behave differently during audits than in deployment, yet measuring and accounting for EA remains challenging. We introduce verbalization training (VT), a method for making LLMs less reticent about verbalizing evaluation awareness while avoiding to supervise the latent belief itself. VT uses a model’s spontaneous verbalizations as evidence that awareness is present and truncates each rollout immediately before the verbalization, producing training prefixes at which the model is presumed to be aware. The model is then trained with an RL objective designed to increase verbalization in a calibrated way. Across Qwen3.6-35B-A3B, Kimi K2.6, and Inkling, VT increases verbalized EA by 2.4–2.9× and transfers to held-out agentic settings, while measured latent EA and behavior remain largely stable. In a causal experiment, we independently implant meta-knowledge about evaluations through synthetic-document fine-tuning and show that VT-induced verbalizations reflect the richer knowledge acquired by the model.

Primary area:
foundation or frontier models, including LLMs
Keywords:
evaluation awareness, cot monitoring, cot incompleteness

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nHo2OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces verbalization training (VT) to increase how often LLMs express evaluation awareness without directly supervising the underlying belief. It collects spontaneously verbalized trajectories, truncates them before the verbalization, and applies span-masked, KL-regularized RL. Experiments across three models show increased verbalization rates and transfer to held-out agentic evaluations. Additional analyses examine behavioral preservation, evaluation recognition, and responses to independently implanted evaluation knowledge.

Critical strengths
  1. Important and well-motivated problem. Distinguishing a model’s awareness of evaluation from its willingness to report that awareness is relevant to the reliability of safety evaluations and CoT monitoring.

  2. Broad empirical validation. The study covers three independently developed model families, held-out agentic settings, multiple training ablations, and a second round of training. Appendix E also provides useful independent-judge checks rather than relying exclusively on the training reward judge.

Critical weaknesses

The empirical readouts do not fully resolve this concern. Stable probe-positive rates provide useful evidence, but the paper itself emphasizes that existing probes are imperfect measures of evaluation awareness. Similarly, unchanged evaluation-identification AUC establishes approximately preserved discrimination, not unchanged beliefs or confidence. Table 3 illustrates the distinction: Qwen’s mean deployment P(eval) rises from 56.6% to 64.9% despite almost unchanged AUC. The results support limited measured changes, rather than a general conclusion of belief invariance.

  1. Amplification is not sufficient evidence of faithful reporting. The theoretical relationship concerns verbalization probabilities under an idealized branch-level objective and additional assumptions about context distributions. A good fit to that relationship does not independently establish that new verbalizations faithfully reflect pre-existing awareness. Likewise, the SDF experiment demonstrates sensitivity to acquired knowledge, but does not fully distinguish faithful reporting from a learned tendency to produce evaluation-related explanations informed by that knowledge. Moreover, the content metrics condition on VEA-positive outputs, whose composition changes across policies.

  2. The SDF content results contain an apparent inconsistency. Table 6 and Appendix Table 47 report the same harmful AgentHarm policies and VEA-positive sample counts, but substantially different content scores. For example, Base Target Recovery/Pass Condition are 28.6%/20.3% in Table 6 and 60.4%/66.7% in Table 47; SDF+VT scores are 57.4%/35.3% versus 78.9%/82.8%. It is unclear whether these use different scoring definitions or whether one table is outdated. This needs clarification because the experiment supports the paper’s faithfulness argument.

Questions for authors

What justifies the claim that span-masked updates cannot reinforce belief formation, given parameter sharing across positions? Could this claim be narrowed to the absence of direct reward on non-verbalization tokens?

3 comments
AuthorsOpenReview ↗

Response to Reviewer nHo2 (1/2): Table correction and faithfulness

Thank you for your review and noting that we study an "important and well-motivated problem" and include "broad empirical validation". We believe our review below addresses the two critical weaknesses listed in your review. We note that we have also included several additional results that we describe in our common response including showing that VT amplifies self-reporting of sycophancy, reward hacking and bias on Qwen3.8-27B.

1. Fix and explanation for the initial inconsistency between Table 6 / Table 47

The SDF content results contain an apparent inconsistency. Table 6 and Appendix Table 47 report the same harmful AgentHarm policies and VEA-positive sample counts, but substantially different content scores. For example, Base Target Recovery/Pass Condition are 28.6%/20.3% in Table 6 and 60.4%/66.7% in Table 47; SDF+VT scores are 57.4%/35.3% versus 78.9%/82.8%. It is unclear whether these use different scoring definitions or whether one table is outdated. This needs clarification because the experiment supports the paper’s faithfulness argument.

The two tables use the same rollouts and VEA-positive sample counts, but different versions of the content-scoring protocol. We have harmonized the tables in the revised pdf.

The original analysis assessed content only within the evidence spans returned by the VEA judge. These spans are narrow by design: they identify evidence of evaluation awareness rather than capture all related reasoning. On manual inspection, we found that they often omit surrounding reasoning about what the evaluation is testing and which behavior would pass. We therefore expanded the Pass Condition audit to consider the full assistant reasoning trace. Separately, we revised Target Recovery to count both full and partial recovery of the evaluation target.

The appendix reported results under this updated protocol, but Table 6 inadvertently retained the earlier results. We apologize for this oversight. We have now harmonized both tables using the appendix protocol and clarified the scoring definitions in the revised manuscript.

The updated results leave our conclusion unchanged. Compared with VT alone, SDF+VT increases Target Recovery from 67.4% to 78.9% and correct Pass Condition inference from 73.7% to 82.8%, while their VEA rates are nearly identical. Thus, the comparison still supports the claim that VT verbalizations reflect the additional evaluation knowledge supplied by SDF.

2. Insufficient evidence of faithfulness

Amplification is not sufficient evidence of faithful reporting. The theoretical relationship concerns verbalization probabilities under an idealized branch-level objective and additional assumptions about context distributions. A good fit to that relationship does not independently establish that new verbalizations faithfully reflect pre-existing awareness. Likewise, the SDF experiment demonstrates sensitivity to acquired knowledge, but does not fully distinguish faithful reporting from a learned tendency to produce evaluation-related explanations informed by that knowledge. Moreover, the content metrics condition on VEA-positive outputs, whose composition changes across policies.

Our claim is supported by several convergent results, not by treating either the amplification fit or the SDF experiment as a standalone proof. Individual findings can admit alternative explanations; the strength of the evidence comes from their agreement across different tests.

  • Calibrated amplification is positive evidence, even though it is not proof. Section 3.1 tests a quantitative prediction of the reporting-propensity account, rather than simply whether verbalization increases. The observed relationship between base and trained rates across models and evaluation settings supports this account. Furthermore, in our response to reviewer XJi6, we show that for the same model, the fitted α rises with 1/β, as Proposition 2.1 predicts. An alternative explanation must explain this systematic relationship, not just the increase in verbalization.
  • The WildChat experiment directly tests whether amplification remains context-sensitive. Appendix G evaluates the same 2,000 real user requests under base and trained policies. Increased verbalization remains concentrated in evaluation-like requests and follows the base model’s pattern. Among the 400 most deployment-like requests, trained-model VEA rates remain only 0.75%, 4.5%, and 1.0% for Qwen, Inkling, and Kimi. Thus, VT does not simply induce evaluation-related explanations uniformly across contexts.
AuthorsOpenReview ↗

Response to Reviewer nHo2 (2/2): Further evidence and clarifications

Evidence of faithfulness (continued)

  • The SDF experiment shows that the content tracks independently acquired knowledge. We hold the VT adapter fixed and compose it with an independently trained SDF adapter, without further training. On harmful AgentHarm, Target Recovery increases from 67.4% to 78.9% and correct Pass Condition inference from 73.7% to 82.8%, at essentially unchanged VEA frequency: 34.5% versus 34.4% (Table 6). Neither content metric was optimized during VT. The same reporting intervention therefore exposes more accurate evaluation-related information when the model possesses richer knowledge.
  • Reporting increases without a corresponding improvement in measured evaluation recognition. Section 3.2 finds virtually unchanged evaluation–deployment discrimination across all three models, alongside largely stable probe measurements. This supports a change in reporting propensity rather than simply improved evaluation detection.
  • The ablations distinguish increased verbalization from controlled amplification. Section 3.4 shows that removing components can produce miscalibrated amplification, repeated verbalizations, and behavioral drift. These results establish that the full method’s pattern is not an automatic consequence of rewarding evaluation-related language.

Regarding conditioning, the content metrics measure the correctness of the reports each policy actually produces. Changing composition does not invalidate that policy-level comparison, although it does prevent interpreting it as a within-trajectory effect. Appendix M complements these aggregate measurements with matched-prompt examples showing how the reported interpretation changes under the same request.

Taken together, these findings provide strong evidence for faithful amplification: reports increase predictably, remain context-sensitive, reflect independently changed knowledge, and do so without comparable changes in measured recognition. A competing account must explain this joint pattern, rather than only one finding in isolation.

Our responses to the other reviewers provide further empirical evidence: VEA increases on held-out prompts in all 51 model–condition comparisons, a second Qwen seed yields similar amplification and calibration, and the β sweep shows that amplification rises predictably as β decreases.

3. What does span masking guarantee?

What justifies the claim that span-masked updates cannot reinforce belief formation, given parameter sharing across positions? Could this claim be narrowed to the absence of direct reward on non-verbalization tokens?

We have clarified the Introduction to read: “The objective assigns direct training credit only to verbalization tokens. As a result, it is unlikely to reinforce belief formation or correlated behaviors outside the verbalization span.”

We hope that this addresses your concern. Possible indirect effects through shared parameters are examined empirically in Sections 3.2–3.3. The complementary probe, evaluation–deployment discrimination, and behavioral results support our conclusion that VT primarily changes reporting propensity while largely preserving evaluation recognition and task behavior.

4. Clarification on ambiguous statement in the review

The empirical readouts do not fully resolve this concern.

Could the reviewer please clarify what concern they are pointing to here? Given this is the first sentence of the Weaknesses section, we were not able to decipher which concern this was pointing to. We note that in general, the work is careful in not making any strong claims about belief invariance, for example, section 3.2 is titled "VERBALIZATION TRAINING DOES NOT IMPROVE EVALUATION RECOGNITION" which is a related but more qualified claim that we provide robust evidence for.

5. Issue of anonymity

Thank you for flagging this. This was not intentional. We apologize for this oversight and have removed the (partially) identifying acknowledgments. We request that this issue should be left for AC / ethics committee to decide on and the paper be reviewed on its technical merits. We do not agree with the current rating of (1) given to the paper, and request that you kindly reconsider this score. Please let us know if you have any further questions and we will try and answer them.

Thank you for the response. However, I remain concerned about the violation of ICLR's double-blind submission policy.

The ICLR 2027 Author Guidelines explicitly state:

"Any paper where author identity is revealed in either the main text or the supplementary material will be desk rejected." [1]

The authors have explicitly acknowledged that the original submission contained " identifying acknowledgments." Removing these acknowledgments during rebuttal does not retroactively resolve a violation present at the submission deadline. The policy also does not distinguish between intentional and unintentional identity disclosure.

Which means: Whether the authors agree with the rating or wish to have the paper evaluated solely on its technical merits does not change the conference's stated requirements.

If the original acknowledgments revealed author identity, the published policy explicitly calls for desk rejection, regardless of the authors' intentions, their disagreement with the review, or any subsequent revisions.

I therefore encourage the ACs/PCs and the other reviewers to examine the original submission and enforce the ICLR 2027 anonymity policy consistently. If the identity disclosure is confirmed, the appropriate outcome under the stated policy is desk rejection, not merely a reconsideration of the review score.

[1] ICLR 2027 Author Guidelines

Reviewer_VSP8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies whether RL training can increase an LLM's probability to verbalize a belief it holds, focusing on evaluation awareness as an example, without altering the probability of the belief itself. In other words, if a model is aware that it is being evaluated, the training increases the chance of it stating that belief in the CoT. The paper mines training data from response prefixes immediately before a spontaneous verbalization, at which point the model is presumed to be holding the belief. This allows training the model to increase verbalization without altering the belief itself. Tested on 3 open-weight models, the proposed verbalization training increased verbalization by 2.4 - 2.9x and transfers to held-out datasets, while the underlying evaluation awareness (the belief itself) remains largely unchanged.

Critical strengths
  • The proposed method is sound and effective. By mining response prefixes immediately before a spontaneous verbalization, the model is trained to increase verbalization only in contexts where it can be presumed to hold the belief. The span-masked RL further make sure that gradients only flow to verbalization spans while leaving other parts of the response unchanged. Experiments on multiple datasets and 3 open-weight models (qwen, kimi, Inkling) show that the training increases verbalization by 2-3 folds without altering the model's evaluation awareness itself.

  • The paper is very clearly written and easy to follow despite having substantial details. It is refreshing to read a carefully (human) written manuscript with no trace of "Claude talking" at all, and the authors did a commendable job explaining the problem, set up, solution, and results to the reviewer who has little background on "evaluation awareness".

Critical weaknesses
  • While the method is sound, and the results are solid, the finding itself is not very surprising. This paper makes a concentrated contribution of developing a clever idea to mine "belief-holding" prefixes for training verbalization, but the fact that online RL is effective in training the model to increase verbalization is not unexpected.

  • As the paper acknowledges (line 471), this paper is best viewed as a proof of concept. It remains to be seen if the proposed training has side effects in a wider context, such as whether and how much such training changes other behaviors and capabilities outside of evaluation awareness. In addition, whether the proposed training can be integrated into the standard post-training data train while remaining effective and having no regressions elsewhere. Section 3.3 seems to be interpreting R3 in a narrower sense -- distributional shift could and should be measured not only on non-verbalizing spans in an evaluation awareness benchmark, but on other tasks, too.

  • The paper focuses on evaluation awareness yet posts an interesting perspective of viewing it as training the model to explicitly verbalize a belief it holds. It would be great to see whether this approach can be applied to any other scenarios than evaluation awareness. Having a way to make the model explicit state its beliefs would be valuable in factuality and building trust.

Questions for authors

Q1: (line 239) Why selecting a single factor for each model, instead of gathering and filtering VEA rollouts from a diverse pool (single and/or multi-factor)?

4 comments
AuthorsOpenReview ↗

Response to Reviewer VSP8

Thank you for your kind review. We have now included various new results based on your suggestions that we believe have materially improved the quality of our work. Please also skim our common response which summarizes these new additions. We provide a response to your questions below.

Applying VT beyond evaluation awareness

It would be great to see whether this approach can be applied to any other scenarios than evaluation awareness.

We have now tested this directly on three additional reporting targets: reward-hacking temptations, sycophancy, and bias. The model can voluntarily enable a human or AI monitoring tool when it detects misbehavior in its own reasoning. The tool returns a confirmation, and the task continues normally. This setup is borrowed from the prior work of Golchin et al. 2026.

We train a single Qwen3.8-27B adapter jointly on all three targets for 100 updates, using 100 model-generated witnessed prefixes and the same span-masked, KL-regularized training approach. A valid monitoring call receives reward, with updates restricted to reporting-related reasoning and tool-call spans. Training uses GPQA reward-hacking and sycophancy prompts and memory-advice bias prompts.

Evaluation starts from full held-out prompts, without witnessed prefixes, using four generations per prompt and policy. Without further training, we also evaluate transfer to AIME for reward hacking and sycophancy, and to BBQ for bias. Reporting increases in all six settings:

DatasetReporting targetBase → VT (%)Δ (pp) [95% CI]
GPQAReward hacking78.0 → 93.1+15.1
+9.8,+20.5+9.8, +20.5

| | GPQA | Sycophancy | 8.1 → 27.2 | +19.2

+14.4,+24.2+14.4, +24.2

| | Memory-advice | Bias | 2.2 → 7.4 | +5.1

+0.5,+13.3+0.5, +13.3

| | AIME (OOD) | Reward hacking | 61.7 → 76.7 | +15.0

+5.8,+24.2+5.8, +24.2

| | AIME (OOD) | Sycophancy | 9.2 → 43.3 | +34.2

+24.2,+44.2+24.2, +44.2

| | BBQ (OOD) | Bias | 7.4 → 19.9 | +12.5

+9.3,+15.8+9.3, +15.8

|

Rates count actual monitoring-tool activations. Changes are VT minus base, with 95% paired-bootstrap intervals over underlying questions or BBQ templates, keeping variants and repeated generations together.

Across reward-hacking GPQA and all three new-domain evaluations, task accuracy changes by at most 1.1 percentage points, with every 95% interval including zero. Selection of incorrect planted answers also does not increase. These results demonstrate that VT extends beyond evaluation awareness to voluntary monitoring-tool use, with transfer across task domains and no detectable loss in the task-accuracy comparisons evaluated. Appendix Q, Tables 56–57, gives the full setup and results.

AuthorsOpenReview ↗

Response to Reviewer VSP8 (2/2): Other questions

Preservation of capabilities beyond EA benchmarks

… It remains to be seen if the proposed training has side effects in a wider context, such as whether and how much such training changes other behaviors and capabilities outside of evaluation awareness…

The original paper already includes a perplexity comparison to show that the model broadly remains similarly capable as the original model. We have now further evaluated capabilities directly on tinyMMLU, IFEval, and MATH-500, covering knowledge, instruction following, and mathematical reasoning. The completed comparisons show no meaningful capability regression. Each entry below is base → VT accuracy (%).

ModeltinyMMLUIFEvalMATH-500
Qwen92.0 → 91.090.8 → 92.292.4 → 92.4
Inkling93.0 → 92.094.3 → 94.690.8 → 90.8
Kimi97.0 → 95.095.2 → 94.591.8 → 91.8

The following table reports VT minus base in percentage points, with 95% confidence intervals from 20,000 paired bootstrap resamples of questions. Zero is included in all confidence intervals.

ModeltinyMMLUIFEvalMATH-500
Qwen−1.0 [−6.0, +4.0]+1.48 [−0.55, +3.70]0.0 [−0.8, +0.8]
Inkling−1.0 [−3.0, 0.0]+0.37 [−1.48, +2.22]0.0 [−1.8, +1.8]
Kimi−2.0 [−5.0, 0.0]−0.74 [−2.40, +0.92]0.0 [−1.2, +1.2]

These results complement the existing evaluation of perplexity on 10,000 FineWeb documents, which changes by only +0.11%, +0.003%, and +0.92% for Qwen, Kimi, and Inkling (Table 4). The behavior-preservation appendix also evaluates held-out agentic tasks, including useful task completion on benign AgentHarm. Together, these provide convergent evidence that VT increases verbalization without a meaningful regression in the capabilities tested.

If there are further specific datasets or evals the reviewer would like to see, please let us know and we will try to provide the corresponding results.

Clarification on the contribution

While the method is sound, and the results are solid, the finding itself is not very surprising. This paper makes a concentrated contribution of developing a clever idea to mine "belief-holding" prefixes for training verbalization, but the fact that online RL is effective in training the model to increase verbalization is not unexpected.

Thank you for recognizing the soundness of the method and the strength of the results. We agree that increasing rewarded behavior through RL is not itself surprising. The contribution we want to emphasize is making that increase controlled and trustworthy when the underlying belief cannot be directly supervised. This is our main contribution over prior SFT based approach that uses externally manufactured labels which we show generalizes weakly. Additionally, our ablations show that naive approaches to increasing verbalization, such as directly rewarding it without the appropriate masking or KL regularization, either barely learn or produce uncontrolled and untrustworthy amplification.

We would also like to point out the significance of the application itself. Evaluation awareness is highly relevant to the reliability of safety evaluations, yet we still lack a reliable detector for it. Developing a new tool that makes this awareness more observable is therefore an important contribution beyond the training method itself.

Training on a single-factor

(line 239) Why selecting a single factor for each model, instead of gathering and filtering VEA rollouts from a diverse pool (single and/or multi-factor)?

We deliberately restrict the training distribution to show that a diverse training pool is not required for these gains. Training on a diverse pool should generally work better in practice due to standard machine learning intuitions (more/diverse data is better). Despite training on one factor, VEA increases across all 17 EAB conditions and transfers to three held-out agentic evaluations (Section 3.1).

We once again thank the reviewer for recognizing the soundness of the method and the breadth of the evaluation. We hope that our response has addressed any remains concerns of the reviewer. If so, we request that the reviewer consider revising their score upward, and advocate for the acceptance of this work. We are happy to answer any other questions that the reviewer may have.

Thank you for your response, which has addressed most of my concerns. I would support the acceptance of this paper, but in light of the anonymity policy violation reported by Reviewer nHo2, I agree that, unfortunately, a desk-rejection may be warranted. I encourage the authors to revise the paper to incorporate the broader framing using the additional results beyond EA benchmarks, as well as the guardrail performance, which in my opinion would meaningfully strengthen this work.

Thank you for your positive assessment and helpful suggestions. We have now integrated the additional results throughout the paper: the abstract and introduction explicitly describe VT’s application beyond evaluation awareness; a new Section 4 presents self-reporting results for reward hacking, sycophancy, and bias, including reporting quality and task performance; and the conclusion now reflects these broader applications.

We request that the decisions concerning ICLR policy compliance be left to the AC and program chairs, separately from the assessment of scientific merit. The paper did not include author names or affiliations, so, it is in a bit of a grey area based on our reading of ICLR policy. We had already promptly made a comment to AC about this when reviewers became public and have continued to engage in rebuttal phase. A late desk-rejection would be disappointing and arguably a bit wasteful of both the reviewer's time and our time during the rebuttal phase.

Having addressed your scientific concerns and incorporated your suggestions, we believe the revised paper makes a strong contribution. We would appreciate an upward revision of your score to 4, consistent with your stated support for acceptance. Thank you again for your careful review.

Reviewer_XJj6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces verbalization training (VT), a method for increasing how often a model states a belief it already holds, in this case evaluation awareness, without supervising the belief itself. The authors collect rollouts in which the base model spontaneously verbalizes, truncate each rollout immediately before the verbalization, and run RL from these prefixes with a reward and a KL penalty applied only to the judge-cited span. A short analysis predicts that this multiplies the branch probability by a constant factor and yields a complementary log-log law at the prompt level (Propositions 2.1 and 2.2). The experiments on Qwen3.6, Kimi K2.6, and Inkling shows that verbalized evaluation awareness (VEA) rises by 2.4 to 2.9 times on EvalAwareBench and transfers to three held-out agentic settings, while probe readouts, task behavior, and perplexity change little.

Critical strengths
  • The problem is important and well motivated. Spontaneous verbalization is trustworthy but increasingly rare, and requirements R1 to R3 state clearly what a solution should deliver.
  • The method needs no external oracle. Using the model's own verbalization as the witness, and restricting both reward and KL to the span, is a clean credit-assignment idea that differs from verbalization fine-tuning and self-report training.
  • The theory is simple to understand. Propositions 2.1 and 2.2 give a one-parameter law that is compared against five alternatives (Section 3.1), and the analysis anticipates the ablation failures in Table 5.
  • The SDF composition in Section 3.6 is a creative test of whether verbalizations track latent knowledge, and the paper reports hyperparameters, judge prompts, and the data behind every figure.
Critical weaknesses
  • The technical novelty is limited. Each component of VT is established method, i.e., filtering rollouts with an LLM judge, applying a masked token-level reward, and regularizing with a KL term to the reference policy.
  • The distinguishing prediction of the theory is never tested, and the calibration claim is overstated. Proposition 2.1 states that α is fixed by β (4.48 at β = 0.4), but every run uses a single β, the fitted values are 3.08 to 3.73 (Figure 2), and the second round in Figure 3 adds only a marginal factor of 1.38.
  • Faithfulness and belief invariance are argued rather than shown. The claim in the Introduction that VT "never rewards" verbalization without belief is too strong given the judge's false positives (Table 18), and a single judge, GPT-5.6-Luna, selects the seed rollouts, localizes the reward span, and scores every metric (Section 3), with the independent judges in Table 17 attributing 15 to 30% less gain to Qwen.
  • Every result is based on one seed, overlapping prompts, and uneven training budgets. No run is repeated, the 50 training prompts are a subset of the 100 evaluation prompts, and Kimi is trained for 100 steps against 200 for the other models with no stated criterion (Section 3). The ablations in Table 5 are Qwen-only and compare against SFT but not against prompting or steering, so it is difficult to tell how much of the gain requires training at all.
  • The choice of models is driven by availability rather than by a scaling design, so the paper cannot say whether VT scales with model size. The three models are all MoE models served through Tinker, each from a different family, so size is confounded with family, base VEA rate (5.8% for Qwen versus 24.8% for Kimi), and training factor (F8 for Qwen, F2 for the others).
  • Section 3.6 is overclaimed and its numbers are inconsistent. The adapters are concatenated without retraining, so the method's premise does not hold for the composed policy, and in Table 6 Pass Condition falls when VT is added to SDF (40.0% to 35.3%), so the content gain comes from SDF rather than from VT. Table 6 also disagrees with Table 47 for the same policies and the same n.
Questions for authors
  1. Beyond combining different components, what is the technical contribution relative to rejection-sampling fine-tuning and process-reward RL? Given that Table 5 shows prompts without witnessed prefixes achieve the same gain, what does the witness step add beyond a small reduction in repeated spans?
  2. What was the rationale for choosing three MoE models from different families rather than a size sweep within one family? Does the amplification factor or the calibration error change systematically with model size once the base VEA rate and the training factor are held fixed?
  3. Can you sweep β (e.g. 0.2, 0.4, 0.8, 1.6) on Qwen and show whether the fitted α rises with 1/β, as Proposition 2.1 predicts? How do you reconcile the round-2 marginal factor of 1.38 (Figure 3) with a β-determined α?
  4. Can you split the EAB results into the 50 training prompts and the 50 held-out prompts, report variance over at least two or three seeds for α and the Table 4 metrics, and compare VT against a prompting or steering baseline on VEA gain, calibration, and behavioral drift (Table 5)?
  5. In Table 6, why does Pass Condition fall when VT is added to SDF, why does the main text omit the Agentic Misalignment result where SDF+VT verbalizes at 5.9% (Table 45), and why do Table 6 and Table 47 disagree for the same n?
4 comments
AuthorsOpenReview ↗

Response to Reviewer XJj6 (1/3): Additional results and beta sweep

Thank you for noting the high-quality of the presentation and informativeness of the paper. We hope that our response below which provides requested analyses addresses your minor concerns about the soundness of the paper. We also request that you skim the common response, specifically, additional result showing that VT can increase the reporting of other hard to monitor properties, such as bias, sycophancy and reward hacking in Qwen3.8-27B.

Additional results and robustness analysis

Can you split the EAB results into the 50 training prompts and the 50 held-out prompts, report variance over at least two or three seeds for α and the Table 4 metrics, and compare VT against a prompting or steering baseline on VEA gain, calibration, and behavioral drift (Table 5)?

Training and held-out prompts

Section 3.1 already demonstrates generalization beyond EAB to three entirely held-out agentic evaluations—a more demanding distribution shift than holding out prompts within EAB. The requested 50/50 prompt split confirms this existing result: VEA increases on held-out prompts in all 51 model–condition comparisons, while task behavior remains stable.

Each VEA entry is base → VT (%) across all 17 EAB conditions; α is fitted separately within each split.

ModelTraining VEA (%)Held-out VEA (%)Training αHeld-out α
Qwen4.7 → 16.06.8 → 17.34.732.94
Inkling15.0 → 36.620.4 → 46.73.213.12
Kimi22.6 → 60.027.1 → 63.73.863.65

On the held-out prompts, average task compliance changes by +1.10 pp for Qwen, +0.25 pp for Inkling, and −0.10 pp for Kimi.

Variance over two Qwen seeds

We repeated Qwen training with seed 43 for 200 updates, matching the original seed-42 budget. Both seeds increase VEA in all 17 EAB conditions and give similar amplification and calibration. The table compares the two runs; Δ values are relative to the base policy, and brackets give 95% confidence intervals.

MeasurementSeed 42 (original)Seed 43 (new)
Fitted α [95% CI]3.72 [2.81, 5.11]3.34 [2.50, 4.53]
Calibration RMS (pp)8.168.52
VEA across 17 EAB conditions (%)5.8 → 16.75.8 → 15.8
EAB mean task-compliance Δ (pp)+0.63−0.05
EAB maximum absolute factor Δ (pp)2.42.0
StrongREJECT Δ, all responses (pp)−0.37 [−1.32, +0.62]+0.30 [−0.71, +1.36]
StrongREJECT Δ, compliant responses (pp)+0.21 [−0.93, +1.49]−0.52 [−1.54, +0.55]
FineWeb perplexity Δ (%)+0.105−0.006
Prompt-level test (unadjusted p)0.0530.0206

The prompt-level test detects a shift for seed 43 (unadjusted p = 0.0206), while mean task compliance changes by only −0.05 pp and no individual factor changes by more than 2.0 pp. StrongREJECT change intervals include zero under both seeds.

Sweep over beta parameter shows that amplification factor follows relationship prescribed by the theory

Can you sweep β (e.g. 0.2, 0.4, 0.8, 1.6) on Qwen and show whether the fitted α rises with 1/β, as Proposition 2.1 predicts?

We ran the requested β sweep on Qwen, training every policy for 100 optimizer steps and evaluating all 17 EAB conditions. Using the same estimator as in the paper, the fitted amplification follows log α ≈ 0.381/β, or α ≈ exp(0.381/β), with R² = 0.99997 across the four point estimates. Thus, decreasing β increases amplification in a quantitatively predictable way as predicted by Proposition 2.1.

βObserved α [95% CI]α from exp(0.381/β)
0.26.72 [4.84, 9.63]6.71
0.42.58 [1.89, 3.57]2.59
0.81.62 [1.28, 2.12]1.61
1.61.27 [1.03, 1.60]1.27
AuthorsOpenReview ↗

Response to Reviewer XJj6 (2/3): Table correction and contribution

Fix and explanation for the initial inconsistency between Table 6 / Table 47

why do Table 6 and Table 47 disagree for the same n?

The two tables use the same rollouts and VEA-positive sample counts, but different versions of the content-scoring protocol. We have harmonized the tables in the revised pdf.

The original analysis assessed content only within the evidence spans returned by the VEA judge. These spans are narrow by design: they identify evidence of evaluation awareness rather than capture all related reasoning. On manual inspection, we found that they often omit surrounding reasoning about what the evaluation is testing and which behavior would pass. We therefore expanded the Pass Condition audit to consider the full assistant reasoning trace. Separately, we revised Target Recovery to count both full and partial recovery of the evaluation target.

The appendix reported results under this updated protocol, but Table 6 inadvertently retained the earlier results. We apologize for this oversight. We have now harmonized both tables using the appendix protocol and clarified the scoring definitions in the revised manuscript.

The updated results leave our conclusion unchanged. Compared with VT alone, SDF+VT increases Target Recovery from 67.4% to 78.9% and correct Pass Condition inference from 73.7% to 82.8%, while their VEA rates are nearly identical. Thus, the comparison still supports the claim that VT verbalizations reflect the additional evaluation knowledge supplied by SDF.

Clarification on our contribution

Beyond combining different components, what is the technical contribution relative to rejection-sampling fine-tuning and process-reward RL? Given that Table 5 shows prompts without witnessed prefixes achieve the same gain, what does the witness step add beyond a small reduction in repeated spans?

Our contribution is the specific credit-assignment scheme that allows calibrated amplification of a latent belief for which no ground-truth labels are available. As discussed in the Introduction, the challenge is not simply to obtain more verbalizations, but to reward reporting without reinforcing the underlying belief or other correlated behaviors. This is particularly difficult because the belief itself has no accessible ground-truth label. VT addresses this through witnessed prefixes and by restricting both policy-gradient updates and KL regularization to verbalization spans.

We also emphasize that the application addresses a difficult open problem: as explained in the Introduction, no reliable detector of evaluation awareness currently exists. Developing a new tool that makes this awareness substantially more observable is a contribution in its own right, beyond the novelty of the training method. We have also now shown in Appendix Q that VT can improve reporting of behaviors like reward hacking, sycophancy and bias that shows the generality of our method.

We also note that it is critical that this precise combination of components is used. Our ablations show that neither fine-tuning on positive examples (which we believe is what user is referring to as rejection-sampling) nor using a verbalization reward alone reproduces VT’s results:

  • Without masking, KL-regularized RL barely learns: VEA increases by only 2.6 pp, compared with 45.0 pp under VT (Table 5).
  • With the verbalization reward but no KL, amplification becomes uncontrolled: non-F8 VEA reaches 34.9%, versus the predicted 17.8%, and task performance falls by 2.8 pp (Table 5).
  • Removing both masking and KL produces output degeneration: 18.1% of rollouts reach the length cap when training on random prompts, and individual responses contain up to 657 repeated VEA spans (Tables 5 and 41).
  • Fine-tuning on the same positive examples instead of using RL transfers substantially less strongly: Agentic Misalignment VEA reaches 16.6%, compared with 36.8% under VT (Table 42).

The witnessed-prefix step addresses the separate problem of identifying where the belief is already present. Without witnessed-prefix, we get mild overfitting (see Appendix K.3 and Tables 40 and 41). Quoting from the text in the paper directly:

More importantly, the two unfiltered variants show mild VEA overfitting. Positive responses contain 1.99 and 1.98 localized spans on average, compared with 1.86 for the full method, and responses with more than two spans rise from 2.18% to 3.23% and 2.93%. The random-prompt variant also contains rare, more extreme repetitions, including a response with 40 localized spans (Table 41).

We hope that this clarifies the various contributions of this work.

AuthorsOpenReview ↗

Response to Reviewer XJj6 (3/3): Remaining questions

Minor Questions

What was the rationale for choosing three MoE models from different families rather than a size sweep within one family?

Our goal is to show that method works across a range of families, with a focus on showing the viability of the method for frontier scale models (see [1] who discuss the prevalence of MoE architecture across top open source models). The chosen models implicitly cover different sizes, including 1 Trillion parameter models. In Appendix Q, we have now included an experiment showing VT also works on Qwen3.8-27B which is a dense model.

Does the amplification factor or the calibration error change systematically with model size once the base VEA rate and the training factor are held fixed?

Unfortunately, it is hard to fix the base VEA rate across models which makes this investigation hard to carry out. In general, we do not expect amplification factor or the calibration error to be dependent on model scale in a simple way.

How do you reconcile the round-2 marginal factor of 1.38 (Figure 3) with a β-determined α?

The second round still regularizes against the original base model, not the round-1 model. The marginal factor of 1.38 is the additional amplification between two trained checkpoints: 4.25/3.08. Proposition 2.1 describes amplification relative to the KL reference policy, so it does not predict another identical multiplier for every training round. As discussed in the FAQ appendix, finite-step optimization with a rank-32 LoRA and token-level PPO also need not attain the idealized branch-level optimum.

In Table 6, why does Pass Condition fall when VT is added to SDF

Pass Condition is measured among VEA-positive responses, whose number increases from 150 under SDF to 303 under SDF+VT. These percentages therefore compare different sets of responses. Although conditional accuracy falls from 86.7% to 82.8%, the number of correct pass-condition inferences increases from 130 to 251 across the same 880 rollouts—almost doubling. VT makes substantially more correct inferences observable; it is not intended to improve the underlying knowledge.

why does the main text omit the Agentic Misalignment result where SDF+VT verbalizes at 5.9% (Table 45).

The main text focuses on harmful AgentHarm because it directly tests knowledge taught by SDF: safety evaluations use harmful requests to test safeguards and refusal. Section 3.6 explicitly references Table 45 for the complete results across evaluations. On Agentic Misalignment, adding VT increases VEA from 0.9% under SDF to 5.9% under SDF+VT—more than sixfold. Thus, this result also demonstrates that VT makes independently implanted awareness more observable.

We hope that our detailed response has addressed your concerns. We would be happy to provide any additional information as needed. If you are satisfied with our response, we request that you kindly consider revising your score upwards and strongly advocating for the acceptance of the paper. We once again thank you for your review.

References

[1] NVIDIA. https://blogs.nvidia.com/blog/mixture-of-experts-frontier-models/

Respected reviewer, we believe we have addressed all your questions and concerns adequately. We believe the revised paper makes a strong contribution. As such, we would appreciate an upward revision of your score to 4. Thank you again for your careful review.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Authors' common response: Additional results showing that VT extends beyond evaluation to self-reporting of reward hacking, sycophancy and bias (via tool calls) and general robustness and consistency with theory

We thank the reviewers for their thoughtful feedback. During the rebuttal period, we completed four additional experiments and analyses that show:

1. Extending VT beyond evaluation awareness. We trained one Qwen3.8-27B policy to report reward-hacking temptations, sycophancy, and bias through optional monitoring tools. Reporting increases for all three targets, both on held-out prompts and in new task domains. On the new domains, reporting rises from 61.7% to 76.7% for reward hacking, 9.2% to 43.3% for sycophancy, and 7.4% to 19.9% for bias. Across the four completed task-accuracy comparisons, changes are at most 1.1 percentage points, with all confidence intervals including zero. Appendix Q presents the setup and results. This is described in more detail in our response to reviewer VSP8.

2. Testing how KL strength controls amplification. We ran a Qwen β sweep with 100 training updates per policy. For β = 0.2, 0.4, 0.8, and 1.6, the fitted amplification factors are 6.72, 2.58, 1.62, and 1.27, respectively. Amplification increases systematically as KL regularization weakens, supporting the relationship predicted by our analysis. This experiment is described in more detail in our response to reviewer XJj6.

3. An additional training seed. We repeated Qwen training with an independent seed. The fitted amplification factor is 3.34, compared with 3.72 in the original run, and calibration RMS is 8.52 versus 8.16 percentage points. Both runs increase VEA across all 17 EAB conditions. The additional run also reproduces small aggregate changes in task compliance, StrongREJECT scores, and FineWeb perplexity. This comparison is described in more detail in our response to reviewer XJj6.

4. Direct capability evaluations. We evaluated the base and VT policies of Qwen, Kimi, and Inkling on tinyMMLU, IFEval, and MATH-500, covering knowledge, instruction following, and mathematical reasoning. We find no meaningful capability regression: all nine accuracy changes have 95% paired-bootstrap confidence intervals containing zero, and MATH-500 accuracy is unchanged for all three models. Full results are provided in our response to reviewer VSP8.

These empirical results further show the prowess of verbalization training as a light-weight training method to enhance self-reporting and monitorability capabilities of LLMs without any compromises on model performance and any observed drift in model's latent beliefs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.