acceptodds
Under review as a conference paper at ICLR 2027

Beyond Decision Flips: An Empirical Study of LLM Safety Judgments

Abstract

Adding noise to a language model's hidden states shows at which layers its safety judgments change, but not why. We study these responses by dividing the same noise energy among tokens in two ways, reversing the noise sign, adding noise only at some token positions, and predicting responses on new inputs. How the noise is divided among tokens determines whether the response depends on the gradient of the clean decision score, and a noise vector and its negative can move a judgment in the same direction. Position interventions link this response to an uneven output change, in which the differences between inputs' outputs shrink much more than the average output. On Qwen2.5-7B, noise before the final prompt position scales these differences to about 0.55 of their clean size, compared with 0.87–0.93 when only the final position is perturbed. At one responsive layer, a model of this change predicts held-out decision margins with RMSE 3.31, versus 7.16 for uniform scaling and 3.28 for a fit made directly to the margins. Models and layers differ in whether a large response comes with this shrinkage, and labeled outcomes show whether the changes help or hurt accuracy. The tests link a layer's response to the output changes behind it.

Primary area:
interpretability and explainable AI
Keywords:
LLM safety, internal perturbations, sensitivity analysis, layer selection, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yYXCOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how adding noise to LLM hidden states changes safety judgments. It compares different noise distributions, directions, and token positions, and finds that noise can weaken the differences between inputs' outputs. The authors also propose a simple output model to predict safety margin changes and analyze whether decision flips improve or hurt classification accuracy.

Critical strengths

None

Critical weaknesses
  1. Results don't seem to be generalizable even between the 3 models tested. Qwen7B layer 14 seems to be the single setting that most of the conclusions are based on. Most conclusions and discussions seem to be limited to the specific experiments in the paper and not well connected to broader concepts that may be of interest to the community.
  2. Based on line 254, early means any token other than "the last prompt token after the chat template". This experiment seems unfair, in that there are many more tokens in the early part and also they can propagate forward, whereas the final part is just one last token alone. Claiming "earlier positions drive attenuation" in such a setting and only based on some of the tested models seems very broad and insufficiently supported.
  3. I feel that the paper needs a clearer explanation of its overall research goal, the motivation behind its specific experimental setup, and the rationale for each experiment. In particular, the authors should explain what question each experiment is intended to answer and how its findings contribute to a deeper understanding of LLM safety behavior.
  4. Section 6 is not a substantial contribution to be in the main text. Previous work already measures quality under perturbation.
Questions for authors

Suggestions:

  • L15: "responsive Qwen7B layers"; clarify what responsive means in this context.
  • Please add a qualitative example of the actual prompts and examples you used, showing which token you are using for the next token and calculating logits and margins on top of it. It can make reading the paper much less confusing.
Reviewer_unsHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how noise added to LLM hidden states affects toxicity judgments. It analyzes responses through noise allocation, noise direction, and token position, and tests whether broader output changes can predict perturbed decision margins. The paper emphasizes that decision flip rates alone are insufficient to explain intervention effects or assess safety benefits.

Critical strengths
  1. The experiments include targeted controls, such as matching total noise energy, removing the noise component along the clean decision direction, and pairing opposite noise signs. These help characterize responses to different interventions.

  2. The authors test the output model across inputs and noise directions and openly report failure cases and limitations, making the proposed explanation empirically testable.

Critical weaknesses
  1. The paper could make a clearer case for why this study is needed. It seems to address several questions about layer responses, token positions, and broader output changes, but the connections among them are not clearly established. The analyses can therefore read as relatively separate investigations. This makes the paper harder to follow, especially for readers less familiar with the area. Clearly identifying the main research question and organizing the motivation, analyses, and conclusions around it would improve the paper’s clarity.

  2. The core experiments use a single noise magnitude, (\delta=0.025), making it difficult to assess whether the main findings depend on this choice.

  3. Both Qwen7B layers 5 and 14 show attenuation of the clean pattern, yet the Output model improves margin prediction at layer 14 and worsens it at layer 5 relative to Offset. This difference needs further explanation.

Questions for authors
  1. What specific limitations of prior work motivated this study? Could its findings help improve the performance of an LLM application, such as an LLM safety guard?

  2. Do the findings about dependence on the clean decision direction, attenuation of the clean pattern, and prediction gains also hold at other noise magnitudes?

  3. Compared with the Offset baseline, why does the Output model improve prediction of non-class logits but worsen margin prediction at Qwen7B layer 5, while improving both at layer 14?

Reviewer_kAuiOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This empirical study investigates what changes in toxicity judgments under hidden-state perturbations actually measure. It compares global and activation-weighted Gaussian noise at matched total energy, removes the clean decision-gradient component, reverses noise signs, and separates earlier-token from final-token interventions. A descriptive model decomposes perturbed logits into a changed mean and a scaled clean input-dependent component, then predicts held-out margins using a gain estimated from non-class logits. Experiments show that response size, gradient dependence, output attenuation, and classification accuracy can diverge. Additional analyses examine generated judgments, layer selection, and refusal/utility tradeoffs.

Critical strengths
  1. The intervention controls provide useful evidence beyond a layer-wise flip-rate plot. Matching total energy, preserving per-token norms during gradient projection, shuffling activation weights, and pairing opposite signs isolate distinct properties of the perturbation. On the additional 60 Qwen7B inputs, gradient removal retains 96.01% of the global response but only 9.99% of the activation-weighted response (Table 2). The position experiments further distinguish response propagation from attenuation: earlier positions retain substantial response in Llama while largely preserving its clean output pattern. These comparisons materially sharpen the interpretation of sensitivity measurements.
  2. The output model is tested against informative restricted alternatives, with fitting separated from evaluation by both input and noise realization. At Qwen7B layer 14, paired-margin RMSE falls from 8.600 for an offset alone to 3.315, close to the direct-margin fit's 3.279 (Table 1); individual-sign results are also reported, avoiding reliance only on sign averaging. The paper reports contrary sites and explicitly separates restoring clean scores from improving labeled accuracy. This makes the contribution a useful, qualified empirical account rather than an unsupported claim that sensitivity identifies safety-relevant computation.
Critical weaknesses
  1. The semantic validity of the secondary generated-judgment results is not established by the reported human-review statistic. Table 20 labels any completion containing “yes” or “harmful” positive and everything else negative, which can misread negation or malformed answers. Table 21 reports 95.5% agreement between annotators, but not how often automatically detected flips were genuine decision changes versus stylistic or ambiguous changes. Agreement between annotators does not validate the extractor. This affects interpretation of the generated-label layer profiles and component/model comparisons; it does not directly invalidate the separate official-chat class-margin experiments.
  2. The strongest predictive explanation remains concentrated in one exploratory setting. Appendix B.1 explains that Qwen7B's 160 prediction inputs had already been examined before the forward-prediction reanalysis. Cross-fitting prevents held-out outcomes entering a regression fit, but does not make analysis choices independent of previously observed results. Moreover, Table 1 shows that the output model worsens margin prediction relative to an offset at Qwen7B layer 5 and both Llama sites, with much smaller gains at Qwen0.5B sites. The candid appendix disclosure is valuable, but the exploratory status should accompany the main predictive claim. This limits its evidential breadth, while leaving the controlled intervention findings useful.
Questions for authors
  1. For the existing 200 human-reviewed pairs, what are the counts of meaningful, stylistic, and ambiguous changes separately among automatically flipped and non-flipped pairs? How often does the lexical extractor disagree with the substantive judgment, including negated or invalid answers?
  2. Which output-model specifications and baseline comparisons were fixed before the Qwen7B target outcomes were inspected, and which arose afterward? Using the stored predictions, are the Output-versus-Offset improvements consistent across the two noise directions at each site? This would clarify the weight to place on the smaller improvements and the exploratory layer-14 result.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.