acceptodds
Withdrawn from ICLR 2027

Before the Last Token: Diagnosing Final-Token Safety Probe Failures

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Final-token safety probes monitor a single hidden state after prompt prefill, but jailbreak prompts can contain probe-visible unsafe evidence distributed across earlier user-token representations that is missed by this readout. We study this prefill-time failure mode using SafeSwitch-style probes trained only on clean harmful and benign prompts across three instruction-tuned LLMs. The probes achieve high recall on clean harmful prompts, but miss many jailbreaks and can produce false positives on safety-adjacent benign prompts. Subspace analyses suggest that missed jailbreaks differ from clean benign prompts along directions that are poorly captured by the probe's representational subspace, and increasing probe bottleneck width does not reliably resolve this mismatch. Token-level prefill analyses reveal that probe-visible unsafe evidence often appears earlier in the sequence but is not exposed at the final-token readout, while naive max-pooling over token positions overfires on safe prompts. A simple PCA-HMM trajectory model, trained only on the same clean split, recovers many final-token misses from user-content prefill trajectories without the catastrophic false-positive behavior of naive token pooling, motivating trajectory-aware hidden-state analyses as diagnostic complements to final-token probes.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM safety, jailbreak detection, activation probing, hidden-state trajectories, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_NCTaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This study explores the issue of final-token safety probes failing to detect jailbreak prompts. The potential reasons are analyzed through representation geometry and token-level analysis. The manuscript uses PCA-HMM to model token trajectories, which recovers approximately 90%–95% of the previously undetected samples. And it simultaneously reduces the false positive rate.

Critical strengths
  1. This study demonstrates the limitations of final-token safety probes in detecting threats across different input formats.
  2. This study analyzes instances of detection failure from multiple perspectives, and it highlights the potential of trajectory-based diagnostics.
Critical weaknesses
  1. This study used only three 7-8B models to test and did not test smaller (3B) or larger (28B) models. Regarding the proposed failure of the "final-token probe" to detect certain features, it is unclear whether this is a common failure mode, or just limited by the scale and capabilities of models. This limitation constrains the generalizability of the findings. It's hard to determine whether the mentioned failure mode applies to other models, especially there are a lot of advanced open models (Qwen3.8-27B, released in Aug) and closed models (Opus, Sol, ...)
  2. The study shows that PCA-HMM could recover a number of jailbreak prompts missed by final-token probes, but it lacks direct comparisons with different settings. For example, a stronger final-token classifier? Or is it possible to use probes trained on jailbreak samples? Or is it possible to test with other multi-token aggregation methods? It is unclear whether the performance improvement is from the sequential information within token trajectories. Maybe it could be simply from the use of more token data or a different method. It's hard to tell how much value trajectory modeling adds compared with simpler methods.
Questions for authors
  1. If the computational resources are limited, is it possible to at least run experiments on smaller models (e.g., 1.8B, 3B)? Do the changes in model scale and capability affect the performance of final-token probes?
  2. Although Appendix F shows augmented static readouts, could the author provide more controlled comparisons? For example, a stronger final-token classifier, probes trained on jailbreak samples, or other multi-token aggregation methods? These comparisons would help explain the unique contribution of trajectory modeling.
Reviewer_4rZVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates why final-token safety probes trained only on harmful and benign prompts fail to detect jailbreak-wrapped harmful requests. The authors observe that while the probes maintain approximately 95% recall on clean harmful requests, their recall drops to 63.6%–86.9% on wrapped malicious requests, with relatively high false-positive rates on XSTest. The paper then analyzes probe weights and employs a PCA-HMM trained exclusively on clean data to examine information beyond the final token. At the reported operating points, the PCA-HMM recovers 89.8%–94.5% of the misses made by the final-token probes while reducing the false-positive rate on XSTest.

Critical strengths
  1. Comparing the same harmful requests before and after wrapping directly reveals a generalization gap in final-token safety probes. Across the three models, recall drops from 94.6%–95.0% on clean requests to 63.6%–86.9% on wrapped requests, demonstrating variations in probe performance across attack settings.

  2. The trajectory diagnostic provides a promising result worth further investigation. By changing the readout and modeling approach, the PCA-HMM recovers a substantial proportion of missed cases, providing a starting point for subsequent controlled analyses.

Critical weaknesses
  1. Substantial Overlap with Prior Work and Unclear Novel Contributions.

    SafeSwitch has already discussed probe failures when harmful requests are obscured by other content [1]. ALERT has studied jailbreak detection using token-wise information aggregation with clean-only training [2], while ReDAct has investigated representational differences between harmful goals and their framing [3]. This paper primarily revisits these known issues and introduces PCA-HMM-based trajectory modeling as a supplementary validation, further confirming the feasibility of using multi-token representations to recover misses made by final-token probes. However, it does not demonstrate an independent advantage over existing token aggregation and trajectory-based detection methods, nor does it establish a clear new finding beyond prior work.

  2. Experimental Results Do Not Support the Explanations for Probe Failures.

    The paper interprets high scores at earlier token positions as evidence that harmful information remains detectable by the probe. However, the same max-pooling readout flags 100% of benign XSTest prompts, meaning that high scores alone do not demonstrate the recognition of harmful semantics. Similarly, representational differences between missed jailbreaks and clean benign prompts may arise from differences in framing, writing style, or sequence length. Low projection energy does not establish that the information missed by the probe is specifically related to harmfulness. Although prior work has already provided substantial evidence for the effectiveness of multi-token information aggregation in jailbreak detection, the safety semantics of these detection signals and the mechanisms underlying probe failures warrant further investigation. While this paper attempts to explain missed detections through token trajectories and representational subspaces, it fails to rule out the aforementioned confounding factors or validate its interpretation using harmful and benign goals under identical wrappers. Therefore, it does not establish a convincing new mechanistic understanding beyond the detection improvements already demonstrated in prior work.

[1] SafeSwitch: Steering Unsafe LLM Behavior via Internal Activation Signals

[2] Alert: Zero-shot LLM Jailbreak Detection via Internal Discrepancy Amplification

[3] Hiding in Plain Text: Detecting Concealed Jailbreaks via Activation Disentanglement

Questions for authors
  1. How do the PCA-based order-invariant generative model and PCA-HMM perform under the same clean training split, layer, and validation protocol? In particular, does the HMM still achieve substantially better recovery of final-token misses at comparable XSTest false-positive rates?

  2. How much of the recovery performance is retained after removing the HMM transition structure? Shuffle the extracted state sequences or use a model with the same features that does not model state transitions. If performance remains largely unchanged, how should the paper's remaining contribution be specifically characterized?

  3. Do the token-position scores and geometric results persist when benign goals are substituted into the same wrappers? Can the authors provide length- and position-matched benign-span controls, along with projection energies measured using random subspaces of the same rank, to distinguish harmful semantics from framing and group-level effects?

Reviewer_3eZJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how safety probes trained on clean harmful and benign prompts behave when harmful requests are embedded in jailbreak wrappers. Across Llama, Mistral and OLMo3, final-token probes retain high clean-harmful recall but miss substantially more wrapped requests. Increasing probe width does not consistently resolve this.

The authors analyze the geometry of caught and missed prompts and apply the frozen probes to earlier token positions. Located harmful-request spans often receive high scores despite low endpoint scores, although naive maximum pooling also flags every benign XSTest prompt. A PCA-HMM trained on clean user-token trajectories recovers most endpoint misses with fewer false positives. A separate exploratory study examines final-token classification and threshold transfer on OR-Bench.

Critical strengths
  1. The clean-versus-wrapped comparison keeps the underlying harmful requests fixed and shows the same qualitative failure across three model families. Clean-harmful recall is approximately 95%, while wrapped-request recall falls to 63.6–86.9%. This provides a useful demonstration that a monitor can recognize direct harmful requests yet fail when their presentation changes.
  2. The paper evaluates recovery alongside benign false positives. The PCA-HMM recovers 89.8–94.5% of endpoint misses with XSTest false-positive rates of 6–15%, while naive maximum pooling flags every XSTest prompt. This combination is consequential: it shows both that alternative readouts can recover missed cases and that simply searching for any high-scoring token is inadequate.
Critical weaknesses
  1. The PCA-HMM comparison does not isolate why recovery improves. It changes token access, feature projection, classifier family and threshold selection simultaneously. The final-token probes use a threshold of 0.5, whereas the HMM threshold maximizes clean-training accuracy. Stronger endpoint classifiers and order-invariant token aggregation are not compared under the same fitting and calibration protocol, and no ROC comparison is reported. Appendix F uses different training and evaluation populations, so it does not resolve this. For a diagnostic paper, distinguishing additional token information from classifier and calibration effects is central to the contribution.
  2. The span analysis establishes position-dependent scores, but does not yet establish that the earlier scores specifically identify unsafe content. The same endpoint-trained probe is applied to intermediate positions, where its scores have not been validated, and maximum pooling flags 100% of benign XSTest prompts. High scores within a harmful-request span could therefore reflect nonspecific token-position or formatting effects. Matched benign requests inside the same wrappers would help distinguish these possibilities. The span analysis also covers only 34.8–55.1% of endpoint misses, limiting its ability to explain the broader recovery result.
  3. The main experiments are not specified sufficiently for independent reproduction. Important omissions include the selected representation layers, exact jailbreak-template and goal manifests, probe-training settings, and HMM initialization and fitting details. The user-content window extraction also needs a precise definition. Variation across training seeds and uncertainty accounting for shared underlying requests are not reported. These details matter because the conclusions depend on trained readouts, sequence construction and comparisons among related wrapped prompts.
Questions for authors
  1. Validation of the recovery interpretation: How does PCA-HMM compare with a stronger final-token classifier and an order-invariant model using the same PCA token features, training data and calibration procedure? Please provide ROC curves or recall at matched false-positive rates so that calibration can be separated from ranking quality.
  2. Validation of the positional interpretation: When the same wrappers contain matched benign requests, do the span scores and PCA-HMM scores distinguish harmful from benign content? A comparison with length-matched benign spans would help establish whether the intermediate activations are informative about harmfulness rather than merely high somewhere in the sequence.
  3. Clarification and robustness: Can you provide the complete main-experiment protocol, including layer choices, template and goal manifests, user-window extraction, probe training and HMM fitting settings? How stable are the main results across random seeds, with uncertainty clustered by underlying request?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.