- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
This paper investigates whether large language models can detect, localize, and distinguish between dropout and Gaussian noise applied to their internal activations. Experiments across eight models show that all tested models can detect and localize sufficiently strong perturbations, while most can learn to distinguish the two perturbation types through in context examples. The authors discuss potential implications for model introspection and training awareness, but the mechanisms underlying these capabilities remain unclear.
Critical strengths
- The paper evaluates eight models across multiple perturbation strengths and conducts control experiments to rule out several simple alternative explanations.
- The study explores whether models can verbalize information about non semantic changes to their internal activations, raising interesting questions about the relationship between internal representations and learned semantic concepts.
Critical weaknesses
- The paper demonstrates perturbation recognition but provides limited mechanistic insight into why it occurs. Simple statistical differences between dropout and Gaussian noise may explain the results, but this possibility is not systematically investigated.
- The broader significance of the findings remains unclear. In particular, the connection between recognizing specific perturbations and training awareness is speculative, with insufficient evidence that the observed ability generalizes to realistic training and inference settings.
Questions for authors
- The authors should compare model performance against statistical feature-based classifiers and linear probes, and investigate which activation features and layers enable perturbation recognition. This would help determine whether the results arise from simple statistical cues or more complex internal representations.
- The authors should evaluate additional perturbation types and test generalization to unseen perturbations. They should also clarify the distinction between perturbation recognition and training awareness to better substantiate the broader implications of their findings.
3 comments
Thank you for the detailed response and additional experiments.
I appreciate the clarification that the main contribution lies in connecting internal perturbation signals to semantic concepts, rather than merely detecting statistical differences. I also find the new generalization experiments in Appendix A.18 helpful.
I agree that my original soundness assessment was overly harsh, as my concerns primarily relate to the interpretation and generality of the findings rather than fundamental methodological flaws.
However, the underlying mechanisms remain insufficiently understood, and the generalization experiments are still limited to unseen perturbation magnitudes. The connection to training awareness also remains speculative without direct empirical evidence.
Overall, the response has addressed some of my concerns, but not sufficiently to change my overall recommendation. I therefore maintain my rating, while revising my soundness assessment from 1 to 2.
New experiments: generalizing to unseen perturbations
We thank the reviewer for answering our questions, explaining their rationale, and updating the soundness score. We ultimately agree that the mechanisms behind models’ success at our zero-shot and few-shot experiments remain elusive.
At the reviewer’s suggestion, we ran a new experiment further testing generalization to unseen perturbations. We provide in-context examples of two perturbations among three: dropout, noise, or steering (by “steering” we mean sampling a random unit vector for each sentence and, at every layer, adding to the activation of every sentence’s token, with varying ). At test time, we might apply any of the three perturbations.
We ran two variants of this experiment. After observing in-context examples of perturbations and , models are asked if a test perturbation is (i) “, , or an unseen perturbation”, or (ii) “, , or ”. Our results follow (every cell reports variant (i) / variant (ii)):
Accuracy (%, chance , SE 0.4)
| Model | ||||||
|---|---|---|---|---|---|---|
| Llama3.1-8B | 34 / 34 | 34 / 34 | 34 / 34 | 34 / 34 | 34 / 34 | 34 / 34 |
| Qwen3-14B | 34 / 36 | 35 / 38 | 37 / 36 | 37 / 37 | 37 / 37 | 38 / 37 |
| Qwen3-32B | 35 / 38 | 33 / 38 | 36 / 40 | 36 / 40 | 38 / 41 | 38 / 41 |
| Olmo3.1-32B | 32 / 31 | 33 / 32 | 34 / 33 | 34 / 33 | 35 / 34 | 35 / 34 |
| Qwen3.8-27B | 36 / 43 | 34 / 41 | 38 / 46 | 40 / 47 | 42 / 47 | 43 / 48 |
| Seed-OSS-36B | 36 / 47 | 34 / 45 | 35 / 50 | 36 / 52 | 38 / 51 | 39 / 50 |
| Mistral-Small-24B | 34 / 39 | 34 / 40 | 36 / 41 | 37 / 42 | 38 / 42 | 39 / 42 |
| Gemma4-31B | 32 / 33 | 34 / 38 | 52 / 52 | 59 / 57 | 62 / 58 | 63 / 59 |
AUROC (%, vs , chance , SE 0.5)
| Model | ||||||
|---|---|---|---|---|---|---|
| Llama3.1-8B | 51 / 51 | 51 / 51 | 52 / 51 | 51 / 51 | 51 / 51 | 52 / 51 |
| Qwen3-14B | 52 / 53 | 54 / 55 | 55 / 55 | 55 / 54 | 55 / 54 | 55 / 55 |
| Qwen3-32B | 51 / 56 | 55 / 58 | 56 / 57 | 55 / 57 | 57 / 57 | 56 / 57 |
| Olmo3.1-32B | 49 / 47 | 50 / 50 | 52 / 50 | 52 / 51 | 52 / 51 | 53 / 52 |
| Qwen3.8-27B | 57 / 61 | 57 / 62 | 59 / 63 | 60 / 64 | 60 / 64 | 60 / 64 |
| Seed-OSS-36B | 56 / 66 | 57 / 70 | 65 / 71 | 66 / 70 | 66 / 68 | 65 / 66 |
| Mistral-Small-24B | 52 / 58 | 56 / 58 | 57 / 59 | 58 / 58 | 59 / 58 | 57 / 58 |
| Gemma4-31B | 50 / 49 | 59 / 57 | 68 / 65 | 70 / 67 | 71 / 68 | 71 / 68 |
This task proves harder than both our original setup (§6) and the additional one the reviewer originally suggested (§A.18). Still, all models except Llama improve in the number of examples in variant (i), and all except Llama and Qwen3-14B in variant (ii), with Gemma notably reaching ~ accuracy (AUROC of seen vs unseen 71). This tests generalization to just one unseen perturbation for now, in part because of compute costs (the range of every new perturbation must be calibrated via our localization experiment and its controls, §4.1, §4.2).
Summarizing, our in-context learning experiments now test generalization to unseen sentences, unseen samples of any given perturbation, unseen magnitudes, and unseen perturbations. We hope this resolves the reviewer’s concern about the generality of our findings. Finally, it was not our intention to directly test training awareness, and we are happy to soften any wording that might suggest otherwise.
Thank you again for your engagement with our paper, and the useful feedback!
Thank you for your review of our paper!
Most importantly, we would like to better understand why the reviewer reports significant concerns about the soundness of our paper, and do our best to assuage those concerns. The weaknesses highlighted in the review (limited mechanistic insight and lack of probe baselines, both discussed below) do not, to our minds, bear on the soundness of our methodology. We would appreciate any further information about these concerns.
Now, on to the concrete points raised. The reviewer correctly points out that there are simple statistical tests to differentiate dropout and noise, and no doubt this is what makes the task possible. Even though such patterns exist, it does not follow that LMs can connect these statistical patterns to the tokens “dropout” and “noise”, especially zero-shot. To quote our conclusions (lines 466-470):
We are confident that probes can do this task, but we are happy to double-check this if it is important to the reviewer.
This is an insightful question (shared with reviewer TdaN). First, we remark that success at our setup does require a form of generalization: at test time both the sentences and the perturbation vectors will differ from the ICL examples by randomness. Still, the reviewer suggests an interesting experiment. Here are two variants of our ICL task where the perturbation strength changes at test time.
We ran both these experiments (see A.18). We observe that the overall accuracy in both cases mostly decreases relative to our original setup. Nonetheless, two models out of eight still learn and generalize under setup (1), and five learn and generalize under setup (2). Notably, Mistral-Small-24B generalizes better under setup (2) than in the original, and Gemma4-31B performs comparably well. Overall, we interpret this as: (i) a corroboration that our calibration allows for a fair comparison of the perturbation strengths (in fact, many models struggle to learn when taught with pairs of comparable magnitude); (ii) nonetheless, some models can exploit higher order features of dropout and Gaussian noise to distinguish them.
We thank the reviewer for pushing us on this front. Training awareness and its consequences have recently become a prominent concern for AI safety, as the Hugging Face incident and the system cards of many frontier and open-weight models corroborate (for example, the system card of Claude Mythos 5.2 dedicates 5 pages to this). With models becoming increasingly prone to reason about being under training, and changing their behavior accordingly (a prediction we advanced in our Conclusions, §7), any hint, such as the presence of dropout, could directly leak this information to the model. We remark that dropout is still sometimes used at scale in modern LMs, e.g., in Mai-thinking-1 and MAI-Base-2. At the reviewer’s suggestion, we expand on this in the newly added footnote 2.
That said, training awareness is just one motivation; our main point is to provide evidence about a new (and, to us, surprising) capability of even small, open-weight models: that of locating, detecting, and recognizing perturbations with no semantic content, even at comparable magnitudes, and without having seen them at training. In doing so, our approach has the benefit of acting as a control for related claims about “model’s introspection”, where a semantic confounder is present. Let us take this occasion to highlight another direct implication of this capacity for AI safety: LMs may be able to detect “fuzzing” (noise added to weights or activations), a technique used to mitigate phenomena such as exploration hacking and sandbagging.