acceptodds
Under review as a conference paper at ICLR 2027

Language models recognize dropout and Gaussian noise applied to their activations

Abstract

We provide evidence that language models can detect, localize and, to a certain degree, verbalize the difference between a-semantic perturbations applied to their activations. More precisely, we either (a) apply *dropout* or (b) add *Gaussian noise* to them, at a target sentence. We then ask a multiple-choice question such as *Which of the previous sentences was perturbed?* or *Which perturbation was applied?*. We test eight models from the Gemma, Llama, Mistral, Olmo, Qwen and Seed families, with sizes between 8B and 36B, all of which can easily detect and localize the perturbations, often with perfect accuracy. In studying the mechanisms behind this capacity, we find that models develop it already at pretraining, and they do not simply track the log-likelihood of the perturbed sentences. To investigate the mechanisms, we find that Olmo models develop this capacity during pre-training, and we rule out the hypothesis that perturbed sentences simply have lower log-likelihood. Notably, the zero-shot accuracy of Qwen3-32B and Qwen3.8-27B in identifying *which* perturbation was applied correlates with the perturbation strength. While this task proves hard for the remaining models, most of them can learn, when taught in context, to distinguish between dropout and Gaussian noise. Interestingly, Qwen3-32B and Qwen3.8-27B learn worse when supervised with wrong labels, again suggesting a prior towards the correct ones. Because dropout is used as a training-regularization technique, while Gaussian noise is sometimes added during inference, we discuss the possibility of a data-agnostic *training awareness* signal and the implications for AI safety.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Language models, introspective awareness, interpretability, AI safety

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_wUyWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper investigates whether large language models can detect, localize, and distinguish between dropout and Gaussian noise applied to their internal activations. Experiments across eight models show that all tested models can detect and localize sufficiently strong perturbations, while most can learn to distinguish the two perturbation types through in context examples. The authors discuss potential implications for model introspection and training awareness, but the mechanisms underlying these capabilities remain unclear.

Critical strengths
  1. The paper evaluates eight models across multiple perturbation strengths and conducts control experiments to rule out several simple alternative explanations.
  2. The study explores whether models can verbalize information about non semantic changes to their internal activations, raising interesting questions about the relationship between internal representations and learned semantic concepts.
Critical weaknesses
  1. The paper demonstrates perturbation recognition but provides limited mechanistic insight into why it occurs. Simple statistical differences between dropout and Gaussian noise may explain the results, but this possibility is not systematically investigated.
  2. The broader significance of the findings remains unclear. In particular, the connection between recognizing specific perturbations and training awareness is speculative, with insufficient evidence that the observed ability generalizes to realistic training and inference settings.
Questions for authors
  1. The authors should compare model performance against statistical feature-based classifiers and linear probes, and investigate which activation features and layers enable perturbation recognition. This would help determine whether the results arise from simple statistical cues or more complex internal representations.
  2. The authors should evaluate additional perturbation types and test generalization to unseen perturbations. They should also clarify the distinction between perturbation recognition and training awareness to better substantiate the broader implications of their findings.
3 comments

Thank you for your review of our paper!

Most importantly, we would like to better understand why the reviewer reports significant concerns about the soundness of our paper, and do our best to assuage those concerns. The weaknesses highlighted in the review (limited mechanistic insight and lack of probe baselines, both discussed below) do not, to our minds, bear on the soundness of our methodology. We would appreciate any further information about these concerns.

Now, on to the concrete points raised. The reviewer correctly points out that there are simple statistical tests to differentiate dropout and noise, and no doubt this is what makes the task possible. Even though such patterns exist, it does not follow that LMs can connect these statistical patterns to the tokens “dropout” and “noise”, especially zero-shot. To quote our conclusions (lines 466-470):

Let us emphasize a key point: it should be straightforward to distinguish dropout from noise with a simple probe on the activations. The mystery here is not how the signal can exist, but rather how a model is able to verbalize the difference and connect it to its semantic knowledge. That is, how does semantic understanding of the concepts of dropout and Gaussian noise, instilled by training, end up aligned with the models’ (first) direct “experience” of having these perturbations applied to their activations?

We are confident that probes can do this task, but we are happy to double-check this if it is important to the reviewer.

The authors should evaluate additional perturbation types and test generalization to unseen perturbations.

This is an insightful question (shared with reviewer TdaN). First, we remark that success at our setup does require a form of generalization: at test time both the sentences and the perturbation vectors will differ from the ICL examples by randomness. Still, the reviewer suggests an interesting experiment. Here are two variants of our ICL task where the perturbation strength changes at test time.

  1. Among our evenly spaced bins, sample perturbation strengths for teaching pairs at random; at test time, select among unseen perturbation strengths.
  2. Provide an in-context curriculum of increasing perturbations, and test on a larger perturbation strength.

We ran both these experiments (see A.18). We observe that the overall accuracy in both cases mostly decreases relative to our original setup. Nonetheless, two models out of eight still learn and generalize under setup (1), and five learn and generalize under setup (2). Notably, Mistral-Small-24B generalizes better under setup (2) than in the original, and Gemma4-31B performs comparably well. Overall, we interpret this as: (i) a corroboration that our calibration allows for a fair comparison of the perturbation strengths (in fact, many models struggle to learn when taught with pairs of comparable magnitude); (ii) nonetheless, some models can exploit higher order features of dropout and Gaussian noise to distinguish them.

The broader significance of the findings remains unclear. In particular, the connection between recognizing specific perturbations and training awareness is speculative, with insufficient evidence that the observed ability generalizes to realistic training and inference settings. They should also clarify the distinction between perturbation recognition and training awareness to better substantiate the broader implications of their findings.

We thank the reviewer for pushing us on this front. Training awareness and its consequences have recently become a prominent concern for AI safety, as the Hugging Face incident and the system cards of many frontier and open-weight models corroborate (for example, the system card of Claude Mythos 5.2 dedicates 5 pages to this). With models becoming increasingly prone to reason about being under training, and changing their behavior accordingly (a prediction we advanced in our Conclusions, §7), any hint, such as the presence of dropout, could directly leak this information to the model. We remark that dropout is still sometimes used at scale in modern LMs, e.g., in Mai-thinking-1 and MAI-Base-2. At the reviewer’s suggestion, we expand on this in the newly added footnote 2.

That said, training awareness is just one motivation; our main point is to provide evidence about a new (and, to us, surprising) capability of even small, open-weight models: that of locating, detecting, and recognizing perturbations with no semantic content, even at comparable magnitudes, and without having seen them at training. In doing so, our approach has the benefit of acting as a control for related claims about “model’s introspection”, where a semantic confounder is present. Let us take this occasion to highlight another direct implication of this capacity for AI safety: LMs may be able to detect “fuzzing” (noise added to weights or activations), a technique used to mitigate phenomena such as exploration hacking and sandbagging.

Thank you for the detailed response and additional experiments.

I appreciate the clarification that the main contribution lies in connecting internal perturbation signals to semantic concepts, rather than merely detecting statistical differences. I also find the new generalization experiments in Appendix A.18 helpful.

I agree that my original soundness assessment was overly harsh, as my concerns primarily relate to the interpretation and generality of the findings rather than fundamental methodological flaws.

However, the underlying mechanisms remain insufficiently understood, and the generalization experiments are still limited to unseen perturbation magnitudes. The connection to training awareness also remains speculative without direct empirical evidence.

Overall, the response has addressed some of my concerns, but not sufficiently to change my overall recommendation. I therefore maintain my rating, while revising my soundness assessment from 1 to 2.

AuthorsOpenReview ↗

New experiments: generalizing to unseen perturbations

We thank the reviewer for answering our questions, explaining their rationale, and updating the soundness score. We ultimately agree that the mechanisms behind models’ success at our zero-shot and few-shot experiments remain elusive.

At the reviewer’s suggestion, we ran a new experiment further testing generalization to unseen perturbations. We provide in-context examples of two perturbations among three: dropout, noise, or steering (by “steering” we mean sampling a random unit vector uu for each sentence and, at every layer, adding α⋅∥h∥2⋅u\alpha \cdot \lVert h \rVert_2 \cdot u to the activation hh of every sentence’s token, with varying α∈R\alpha \in \mathbb{R}). At test time, we might apply any of the three perturbations.

We ran two variants of this experiment. After observing in-context examples of perturbations xx and yy, models are asked if a test perturbation is (i) “xx, yy, or an unseen perturbation”, or (ii) “xx, yy, or zz”. Our results follow (every cell reports variant (i) / variant (ii)):

Accuracy (%, chance 33%33\%, SE ≤\leq 0.4)

Modelk=0k=0k=1k=1k=3k=3k=5k=5k=7k=7k=9k=9
Llama3.1-8B34 / 3434 / 3434 / 3434 / 3434 / 3434 / 34
Qwen3-14B34 / 3635 / 3837 / 3637 / 3737 / 3738 / 37
Qwen3-32B35 / 3833 / 3836 / 4036 / 4038 / 4138 / 41
Olmo3.1-32B32 / 3133 / 3234 / 3334 / 3335 / 3435 / 34
Qwen3.8-27B36 / 4334 / 4138 / 4640 / 4742 / 4743 / 48
Seed-OSS-36B36 / 4734 / 4535 / 5036 / 5238 / 5139 / 50
Mistral-Small-24B34 / 3934 / 4036 / 4137 / 4238 / 4239 / 42
Gemma4-31B32 / 3334 / 3852 / 5259 / 5762 / 5863 / 59

AUROC (%, {x,y}\lbrace x,y \rbrace vs zz, chance 50%50\%, SE ≤\leq 0.5)

Modelk=0k=0k=1k=1k=3k=3k=5k=5k=7k=7k=9k=9
Llama3.1-8B51 / 5151 / 5152 / 5151 / 5151 / 5152 / 51
Qwen3-14B52 / 5354 / 5555 / 5555 / 5455 / 5455 / 55
Qwen3-32B51 / 5655 / 5856 / 5755 / 5757 / 5756 / 57
Olmo3.1-32B49 / 4750 / 5052 / 5052 / 5152 / 5153 / 52
Qwen3.8-27B57 / 6157 / 6259 / 6360 / 6460 / 6460 / 64
Seed-OSS-36B56 / 6657 / 7065 / 7166 / 7066 / 6865 / 66
Mistral-Small-24B52 / 5856 / 5857 / 5958 / 5859 / 5857 / 58
Gemma4-31B50 / 4959 / 5768 / 6570 / 6771 / 6871 / 68

This task proves harder than both our original setup (§6) and the additional one the reviewer originally suggested (§A.18). Still, all models except Llama improve in the number of examples in variant (i), and all except Llama and Qwen3-14B in variant (ii), with Gemma notably reaching ~63%63\% accuracy (AUROC of seen vs unseen 71). This tests generalization to just one unseen perturbation for now, in part because of compute costs (the range of every new perturbation must be calibrated via our localization experiment and its controls, §4.1, §4.2).

Summarizing, our in-context learning experiments now test generalization to unseen sentences, unseen samples of any given perturbation, unseen magnitudes, and unseen perturbations. We hope this resolves the reviewer’s concern about the generality of our findings. Finally, it was not our intention to directly test training awareness, and we are happy to soften any wording that might suggest otherwise.

Thank you again for your engagement with our paper, and the useful feedback!

Reviewer_TdaNOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors test whether language models can detect perturbations with their own activations. Eight models are given dropout or Gaussian noise when processing a given sentence, then asked to specify the sentence or type of perturbation. Localization works well for all models. Type detection without examples is substantially weaker and depends on the labels of the answer options. Examples in context help most models. The authors also demonstrate the emergence of localization ability during pretraining via OLMO checkpoints.

Critical strengths

The study demonstrates interesting behavior without specifically retraining the model for the task. The experiments cover eight models, varying intervention strengths, and several query formulations. Tests with permuted answer options, synonyms, unrelated control labels, and intentionally incorrect labels are particularly useful. Evaluating OLMO checkpoints also helps understand when localization capability emerges. It's very interesting and, most importantly, easy to read, as the study explores its topic from multiple angles.

Critical weaknesses

The main question is how much the distinction between dropout and noise depends on differences in the strength of perturbations. The parameter ranges were chosen to detect perturbations and maintain accuracy on simple questions, but the actual activation changes are not aligned. Therefore, accuracy above 60% with large values ​​of both parameters does not, by itself, preclude a comparison of perturbation strengths, as stated in Section 6.1. A test is needed with comparable perturbation magnitudes and with perturbation strengths changing between context examples and the test.

Questions for authors
  1. Is the distinction between dropout and Gaussian noise preserved with comparable activation changes? Have you checked the transfer when the perturbation strength in the test examples differs from the strength in the training examples?

  2. Are the main results of the zero-shot and few-shot classifications preserved after removing pre-inserted "introspection" cues? In A.4, such a test is shown for localization, but I would like to understand its impact on perturbation type recognition.

4 comments
AuthorsOpenReview ↗

Response part 1

We thank the reviewer for the thoughtful review!

The parameter ranges were chosen to detect perturbations and maintain accuracy on simple questions, but the actual activation changes are not aligned.

This is an important point. Indeed, we asked ourselves what it means to align activation changes, for the common practice is to add a fixed amount of dropout or noise. Should we match the variance (or entropy, or some quantile?) at every layer and token position, on average? And should the normalization be done in aggregate, or conditional on prefix? Stepping back, we argue there are principled reasons to instead take an outside behavior perspective in this alignment, like we do in the paper:

  1. To the extent that two perturbations yield a similar behavior (e.g., they can be located with the same accuracy) we would like to consider them comparable, relative to what models can verbalize, regardless of the internal differences. Accordingly, our ranges comprise values of dropout rate and noise SD that are high enough to be “perceived” (i.e., such that models can identify a perturbed sentence) and low enough to not compromise capabilities.
  2. Now, provided we are on the same page about the minimum and maximum values of the perturbation strengths worth testing, and we agree to adopt standard practice in applying noise / dropout with constant strength over tokens and layers, what remains is disagreement about the spacing of the intermediate points. Fortunately, alternative alignments in this sense can be viewed as monotonic rescalings of axes. In localization (Fig. 5) and zero-shot classification (Fig. 6) this would not change our results. In experiments that involve both noise and dropout, (e.g., Figs. 7(a) and 9), such alternate alignments amount only to different monotonic paths from the bottom left to the top right of the heatmap, which seldom changes the interpretation of our results.
  3. Finally, as external evidence corroborating the validity of our calibration, we offer an additional experiment (see A.11, Fig. 28). We provide models with a single perturbed sentence and ask them if a perturbation was applied. (This question is harder than localization, because no clean sentence is available for comparison.) Of the five models that succeed at this task, their detection accuracy for both dropout and noise is well-described by a logistic function from p_\min to p_\max (resp. from \sigma_\min to \sigma_\max)---recall the definition of these extremes from localization (§4). In other words, each step on the x-axis yields a ~constant increase in log-odds that is similar both in dropout and noise. We interpret this as evidence of calibration, as far as verbalization is concerned.

That said, the reviewer has a legitimate point: arguably our headline result requires qualification—perhaps it would be more accurate to say our finding is that LMs can distinguish dropout from noise when both are applied at a fixed strength.

If the reviewer remains interested in a more complex way of aligning the perturbation magnitudes (e.g., as outlined above), we are open to suggestions; let us know what you have in mind!

AuthorsOpenReview ↗

Response part 2

A test is needed with comparable perturbation magnitudes and with perturbation strengths changing between context examples and the test. Is the distinction between dropout and Gaussian noise preserved with comparable activation changes? Have you checked the transfer when the perturbation strength in the test examples differs from the strength in the training examples?

First, let us remark that success at our setup does require a form of generalization: at test time both the sentences and the perturbation vectors will differ from the training examples by randomness. Nonetheless, the reviewer suggests an interesting experiment. Here are two variants of our in-context learning task where the perturbation strength changes at test time.

  1. Among our evenly spaced bins, sample perturbation strengths for training pairs at random; at test time, select among unseen perturbation strengths.
  2. Provide an in-context curriculum of increasing perturbations, and test on a larger perturbation strength.

We ran both these experiments, and reported them in Appendix A.18. We observe that the overall accuracy in both cases mostly decreases relative to our original setup. Nonetheless, two models out of eight still learn and generalize under setup (1), and five learn and generalize under setup (2). Notably, Mistral-Small-24B generalizes better under setup (2) than in the original setup, and Gemma4-31B performs comparably well. Overall, we interpret this as: (i) a corroboration that our calibration allows for a fair comparison of the perturbation strengths (in fact, many models struggle to learn when taught with pairs of comparable magnitude); (ii) nonetheless, some models can exploit higher order features of dropout and Gaussian noise to distinguish them.

Are the main results of the zero-shot and few-shot classifications preserved after removing pre-inserted "introspection" cues? In A.4, such a test is shown for localization, but I would like to understand its impact on perturbation type recognition.

This is a good question—we ran additional experiments (reported in Appendix A.16 and A.19) with two new prompts mirroring those of Appendix A.4, in both the zero-shot and the in-context learning settings. The first (i) removes the announcement of the perturbations (as the “no opening” prompt, Fig. 15); the second (ii) additionally removes “I am carefully introspecting…” from the assistant turn (Fig. 16). We did this for the two models with the clearest zero-shot signal, Qwen3-32B and Qwen3.8-27B. In the zero-shot setting, prompt (i) yields a signal similar to the one obtained with the original prompts; prompt (ii) yields a similar signal for Qwen3.8-27B, but less so for Qwen3-32B (in particular, the detection rate of dropout does not increase monotonically in p). In the in-context learning setting we do not observe substantial difference from the original setup, i.e., accuracy overall increases with the number of teaching pairs k, with both correct and inverted labels. With both new prompts and both models, accuracy with the correct labels is above that with the inverted labels at every k.

Thank you for the detailed response and the additional experiments. The tests with different perturbation strengths in the examples and at test time, together with the prompt ablations, provide useful new evidence. I also appreciate that the authors report where generalization becomes weaker.

My remaining concern is about the interpretation of perturbation strength. The behavioral calibration gives a reasonable range of perturbation strengths, but it does not directly match the size of the changes to the activations. The new experiments therefore do not fully rule out an explanation based on perturbation strength. Please soften the statement in Section 6.1 that this explanation is excluded and explicitly acknowledge this remaining limitation.

Overall, I find the observed behavior interesting and the experimental study valuable. The additional experiments strengthen the paper, and a complete explanation of the mechanism is not necessary for its main empirical contribution. I consider the remaining concern a limitation of the interpretation that should be reflected in the text, rather than a reason to reject the paper. I am increasing my rating from 3 to 4 (clear acceptance).

Thank you for your continued engagement and score increase!

Please soften the statement in Section 6.1 that this explanation is excluded and explicitly acknowledge this remaining limitation.

We propose rephrasing the sentence in §6.1\S 6.1 as follows: "This excludes that Qwen3-32B learns only by virtue of comparing a stronger perturbation against a weaker one, at least as measured by our behavioral calibration across fixed perturbation strengths (§4.1,§4.2\S 4.1, \S 4.2). Further experimentation is warranted to test what happens with non-uniform ones."

Reviewer_i5JtOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper identifies and analyses a phenomenon in LLMs where the LLMs can detect/localize and verbalize non-semantic perturbations, specifically dropout and Gaussian noise, to their activations. The authors do this by feeding an LLM (chosen from a variety of open-weight models) a template with an instruction including the type of perturbation applied, two example sentences, one of which has token activations perturbed throughout the model via forward hooks, and a simple answer (A or B). Most models can correctly identify the perturbed sentence, with strong correlation between perturbation strength and detection rate for some models. Models tend to be unable to detect perturbations at very small or very large strengths (which the authors denote as pminp_{min} and pmaxp_{max}). The authors also test whether models can zero-shot classify which perturbation was applied, which most models cannot do well (close to chance-level) while only two Qwen models are good at the task. However, most models benefit strongly from few-shot demonstrations on this task. The authors conducted control experiments for each of these tasks/settings, and present a broad range of controls and ablations in the appendix.

Critical strengths
  1. The paper identifies a previously unobserved and still largely unexplained (elusive, as the authors themselves conclude) phenomenon that will likely open new directions in MechInterp and Safety research as it raises questions about how the model is able to verbalize its internal computation.
  2. The experimental setup is very strong with clear control (null hypothesis) experiments and ablations. The provided supplementary materials further help reproduce these experiments (I am assuming I could reproduce them, but I haven't tried it) through well-organized code and experimental configurations for all settings presented here.
Critical weaknesses
  1. It remains unclear why only two Qwen models are good at zero-shot classification – it would be good to know if it is something in the models' architecture, or the data mix at a particular stage of training that enables this.
  2. I wished to have gained clearer insight into how this phenomenon "emerges" as a function of model size / pretraining compute. There are some hints at this (evaluating Olmo progressively throughout its pretraining checkpoints, and evaluating models as small as Llama 8B and as large as Seed-OSS 36B), which I appreciated, but perhaps the authors could try comparable models (trained on the same data) at various (ideally sub-Billion parameters) sizes, or trained with the same compute budget but different architectures.
  3. As the authors admit themselves, the phenomenon remains largely unexplained despite extensive experimentation. There's no mechanistic answer to why it occurs. (this is not meant to diminish the contribution of identifying the phenomenon in the first place)
Questions for authors
  1. Is it possible to elicit the phenomenon in a non-forced-choice setting, e.g., by probing it with a more ambiguous/open-ended question?
2 comments

We thank the reviewer for the encouraging review and insightful comments! We agree that many important questions remain unanswered, such as why the Qwen family performs better in general and how to get a more precise mechanistic cause for the phenomena that we investigate.

It remains unclear why only two Qwen models are good at zero-shot classification – it would be good to know if it is something in the models' architecture, or the data mix at a particular stage of training that enables this.

We too would like to know this. The Qwen family has been observed to stand out among open weight models in many respects; in particular, they seem to succeed at these “introspective tasks” at a higher rate than others (see, e.g., Latent Introspection: Models Can Detect Prior Concept Injections, Mechanisms of Introspective Awareness). We hope to dig deeper and report back on this in the future.

I wished to have gained clearer insight into how this phenomenon "emerges" as a function of model size / pretraining compute. There are some hints at this (evaluating Olmo progressively throughout its pretraining checkpoints, and evaluating models as small as Llama 8B and as large as Seed-OSS 36B), which I appreciated, but perhaps the authors could try comparable models (trained on the same data) at various (ideally sub-Billion parameters) sizes, or trained with the same compute budget but different architectures.

Unfortunately, we only have full visibility into both the architecture, training procedure, and data for the Olmo models. With considerably more compute to run such experiments ourselves, it would be very interesting to run them.

Is it possible to elicit the phenomenon in a non-forced-choice setting, e.g., by probing it with a more ambiguous/open-ended question?

This is an interesting direction, and one that we have recently started to explore (e.g., analysis of CoT in thinking models). Dropping the forced choice setting, however, brings significant complexity to the evaluation that we will not be able to handle to our satisfaction during the review process.

Within the forced-choice regime, one way of making the prompt less leading is to no longer advertise the perturbation before applying it. To this end, we added more experiments trying prompts with more ambiguous prefixes for both the zero-shot and the in-context learning case (reported in Appendix A.16 and A.19). Results in the zero-shot are similar to the original prompt, except in the case where we use a very minimal prompt, where Qwen3.8-27B still yields the same performances, while Qwen3-32B does not. In the in-context learning setting instead, we do not observe substantial difference from the original setup, i.e., accuracy overall increases with the number of teaching pairs kk, with both correct and inverted labels.

Dear authors,

Thank you for your thoughtful rebuttal and for adding new experiments with more ambiguous prefixes that I believe further strengthen the robustness of claims. I also want to make clear that the points of criticism I raised in my initial review were fairly minor in the first place (not "critical" as the form heading suggests), as indicated by my initial score. I also followed the discussions with the two perhaps more critical reviewers, appreciate the additional clarifications and results on unseen perturbations, and I concur with reviewer TdaN that the paper's contribution stands without a comprehensive explanation of the empirically discovered phenomena. Accordingly, I'm keeping my initial high score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised manuscript in light of reviewers' suggestions

We have uploaded a revised version of the paper, based on the reviewers’ suggestions. The changes are highlighted in blue. Beyond adjustments to formatting, we:

  1. Added two new in-context learning experiments (§A.18), thanks to the suggestions of reviewers TdAN and wUyW. These new tasks test for a new form of generalization in which the perturbation at test time is of different strength than those seen in context. In the first task, the strengths are sampled uniformly; in the second, the strengths increase over the prompt. Both tasks prove harder than the original on average. Nevertheless, two (resp. five) models out of eight still show accuracy increasing in the number of teaching pairs in the first (resp. second) task. In fact, one model (Mistral-24B) performs even better on task two than the original.
  2. Ran additional prompt-ablation experiments with two new prompts mirroring those of §A.4, in both the zero-shot (§A.16) and the in-context learning (§A.19) settings, per reviewer TdaN’s suggestion. In the zero-shot setting, one prompt yields a signal similar to the one obtained with the original prompts, while the other reduces the performance of one model (Qwen3-32B). In the in-context learning setting, the alternate prompts do not make a significant difference.
  3. Added a new task where models are shown only one perturbed sentence, and asked if it is perturbed, alongside associated controls (§4.4, with details in §A.11 and §A.12). This detection task is harder than localization because no clean sentence is available for comparison. We find that the detection rate is sigmoidal within the perturbation ranges defined by our original calibration methodology (based on localization), and we interpret this as supporting evidence of its validity, addressing concerns raised by reviewer TdaN.
  4. Updated the discussion of training awareness (see footnote 2), in light of reviewer wUyW suggestion.
  5. Promoted the description of the ablations from footnotes to the main text (§4.3), and extended Fig. 8 to comprise all eight models, for ease of readability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.