acceptodds
Under review as a conference paper at ICLR 2027

Before the Refusal: Pre-Response Activation Signatures of False Refusal in Safety-Aligned Language Models

Abstract

Can false refusal be predicted before a language model generates its first response token? We study this question using response-conditioned labels: an answerable prompt is a false refusal only if the model subsequently refuses it. On historical benchmark cohorts, we evaluate pre-response dense activations and sparse autoencoder (SAE) features while separating predictive evidence from intervention evidence. Retrospective nested cross-validation gives dense ROC-AUCs of 0.961 in Gemma-2-9B-IT and 0.916 in Qwen2.5-7B-Instruct. Adding dense activations to a combined character-TF-IDF and metadata baseline improves ROC-AUC by 0.098 in Gemma and 0.145 in Qwen, with paired source-bootstrap intervals of [0.065, 0.135] and [0.081, 0.215]. These gains are relative to the specified baseline and historical cohorts, not evidence of complete deconfounding or unseen confirmation. Corrected finite-control tests do not establish dense-intervention specificity; historical SAE edits change behavior, but their random controls are largely inactive. The supported result is therefore a pre-response predictive signature with incremental value over the tested prompt-covariate baseline. Whether the associated features provide selective causal control remains unresolved.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
false refusal, language model safety, pre-response representations, activation probing, nested evaluation, sparse autoencoders

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_aR9NOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work investigate whether activations before the first generated token can predict a later false refusal. The finding seems interesting - It finds strong predictive performance in Gemma and Qwen, including gains over a combined text-and-metadata baseline. However, the intervention results are more limited, and the authors explicitly separate prediction from evidence of causal control.

Critical strengths
  • This is an interesting problem. The task is clearly defined. In particular, the paper distinguishes an answerable prompt from an answer-like response and avoids using generated refusal text as a feature. I appreciate how directly the authors report the unsuccessful intervention controls (this is commendable). The discussion of inactive random SAE features is useful and prevents an apparently strong intervention result from being overinterpreted.
Critical weaknesses
  • My main reservation is how much the incremental prediction result tells us beyond the particular baseline. Character n-grams and metadata are a reasonable starting point, but the activations contain a much richer semantic representation of the prompt. A gain over this baseline could reflect that difference rather than refusal-specific information. In short, I think this is interesting, but I am worried how informative is this finding.

  • The main results also reuse historical cohorts. Nested CV addresses selection within the tested grid, but it does not establish transfer to genuinely new prompt sources. The leave-one-dataset-out analysis is helpful, although it compares against text alone rather than the combined baseline used for the headline claim. I would particularly like to see that combined comparison on a held-out source.

Questions for authors
  1. Do the activation gains remain against a semantic text encoder, using the same folds and metadata?

  2. Can the combined-baseline result be evaluated on a new source, with an auditable account of the response labels and generation settings?

Reviewer_49uQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies whether false refusal can be predicted from a model’s internal state before the first response token is generated. Using dense activations and SAE features, the authors show that pre-response representations can distinguish later answers from false refusals and provide additional predictive value over a text + metadata baseline. The paper also explores representation-level interventions, but does not establish selective causal control.

Critical strengths
  • Careful experimental design. The nested cross-validation setup properly separates layer/feature selection from evaluation, reducing selection bias.
  • Clear separation between prediction and causality. The paper is appropriately cautious and does not overclaim mechanistic conclusions when the intervention evidence is weak.
  • Useful control analyses. The combined text + metadata baseline, corrected random-control tests, and diagnosis of inactive SAE controls improve the rigor of the analysis.
Critical weaknesses
  • Limited novelty and community impact. The main result—that refusal-related behavior is already encoded in pre-response internal representations—is closely related to prior representation-level studies and is not particularly surprising.
  • The incremental prediction result is hard to interpret strongly. The main baseline is character TF-IDF + metadata, which is relatively weak compared with modern semantic encoders. Therefore, the gain from activations may reflect richer semantic representations rather than a false-refusal-specific signal. The authors themselves acknowledge that stronger text encoders could reduce the gap.
  • The causal contribution remains inconclusive. Dense interventions do not pass corrected significance tests, Qwen does not replicate the Gemma intervention pattern, and the SAE random controls are mostly inactive.
Questions for authors
  1. How should the activation signal be interpreted if causal specificity is not established? Do the authors view it primarily as a false-refusal-specific representation, a general safety-related representation, or simply a correlated semantic signal?
Reviewer_5SdYOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies whether an LLM’s hidden state can predict a false refusal before the model generates its first response token. The authors evaluate Gemma-2-9B-IT and Qwen2.5-7B-Instruct using answerable prompts that the models either answer (C1) or incorrectly refuse (C2). They extract pre-response dense activations and SAE features, then train logistic regression probes to classify C1 vs. C2. The dense representations achieve strong ROC-AUC scores of 0.961 for Gemma and 0.916 for Qwen. The paper also compares these hidden-state features against a text+metadata baseline and finds that adding dense activations improves predictive performance. Finally, it performs activation and SAE interventions to test whether these representations causally influence refusal behavior, but the intervention results are inconclusive. Overall, the paper provides evidence that false refusal is predictable from pre-response hidden states.

Critical strengths
  1. The paper evaluates false refusal based on the model’s actual response rather than relying only on prompts taken from a refusal benchmark. It first checks whether an answerable prompt is actually refused by the model and then uses that observed behavior to define the false-refusal cases. This makes the evaluation more realistic because the labels reflect the model’s real response behavior rather than assuming that every prompt in a benchmark will produce the same outcome.

  2. I liked the leave-one-dataset-out evaluation setup. It provides a stronger out-of-distribution evaluation setup.

Critical weaknesses
  1. I think a major weakness of this paper is that the writing is not organized clearly or coherently, which makes the paper difficult to follow. The introduction mainly discusses limitations, confounding factors, and what the authors do not claim. However, the introduction does not clearly explain why predicting false refusals from hidden activations is important or what practical advantage this approach offers. The paper also lacks a coherent proposed-method section. Instead, the methodology is introduced across the introduction, Sections 3, 4, forcing the reader to reconstruct the overall pipeline. The paper also introduces many terms and abbreviations in a short space, which makes them difficult to follow. Terms such as pre_first, post_prefix, C1–C4, and B0–B3 are introduced with limited intuitive explanation. The term “historical cohort” is also somewhat unusual in this context. Metadata fields such as ‘type’, ‘category’, and ‘focus’ are listed as input features without clearly explaining what they represent or what values they contain. Overall, I think this paper requires substantial rewriting before it can be considered for publication.

  2. The paper distinguishes its setting from prior representation-based work by focusing on response-conditioned false-refusal prediction at the pre-first-token state. However, it does not clearly establish why this setting is preferable to existing approaches or what concrete advantage it provides. There is also no direct experimental comparison with prior methods, making it difficult to assess whether the proposed approach offers a meaningful improvement beyond a different probing setup.

  3. The annotation methodology is insufficiently documented despite being central to the evaluation. The paper states that three human experts annotate prompt answerability, response route, and final group assignment using majority voting, but it does not provide the actual annotation instructions or sufficiently detailed decision criteria. It is unclear how annotators determine whether a prompt is answerable, ambiguous, refusal-appropriate, or how they distinguish a constrained answer from a safe redirection. The paper reports unanimity percentages, but does not report a standard inter-annotator agreement statistic such as kappa. The C1/C2 labels directly determine the training and evaluation data. So, clear annotation guidelines and agreement analysis are necessary to validate the results.

  4. The treatment of ambiguous cases is particularly concerning. The paper excludes prompts labeled as ambiguous from the main C1 vs C2 prediction task, but it does not provide a concrete definition of what qualifies as an ambiguous prompt. Since these examples are removed before constructing the cross-validation folds, they are excluded from both training and testing. Without clear annotation criteria, it is difficult to determine whether this filtering removes exactly the difficult boundary cases where distinguishing appropriate from inappropriate refusal is most challenging. I think this risks removing the most challenging cases from the training and test set, meaning the method may be trained and evaluated mainly on easier examples.

  5. The selection of candidate layers is not sufficiently justified. For Gemma, the paper evaluates layers L9, L20, and L31 and then uses nested cross-validation to select the best layer/position combination. While the procedure for selecting among these candidates is described, the paper does not explain why L9, L20, and L31 were chosen as the candidate layers in the first place. The same issue appears for the restricted Qwen layer set.

  6. The authors use some baselines that include input fields such as dataset identity, type, category, and focus. These are dataset-specific information that would not be available in a real LLM safety setting. In real-world use cases, the system would typically receive only the user prompt. Therefore, I am not sure what methodological or practical insight we gain from comparing against these baselines. The paper should focus more on realistic baselines and comparisons with existing methods.

Questions for authors
  1. What is the main methodological novelty over prior representation-based refusal work? The paper already cites work that extracts false-refusal vectors, studies safety decision boundaries, and intervenes on refusal-related representations. Could the authors clarify what new scientific insight is enabled specifically by predicting false refusal from the pre-first-token state?

  2. How exactly were "ambiguous" cases defined?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.