- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies whether jailbreak success can be predicted from prompt representations before generation, and whether the predictive direction coincides with directions that mediate refusal. It estimates a success–failure direction and removes its component within a subspace formed by eleven published refusal directions. Experiments across ten models, two datasets, and four attack families show that the remaining direction retains strong predictive performance but has weaker effects under direct intervention.
Critical strengths
-
The paper asks a meaningful question: directions that control refusal may differ from those that predict which jailbreak succeeds. Studying these properties together is a useful contribution.
-
The evaluation is somewhat comprehensive, which covers multiple model families and attack types.
-
The figures and tables seem to be nicely designed.
Critical weaknesses
-
I found the paper very hard to read. The write-up is unnecessarily complicated, and many sentences require considerable effort to understand. I had to use an AI tool to help interpret many of them. The main ideas are interesting, but the write-up needs substantial improvement before publication (if eventually aiming for human readers).
-
The interpretation of "outside the refusal subspace" depends heavily on how that subspace is constructed. The eleven selected directions do not necessarily capture all refusal-related features, and Section 3.2 explicitly excludes additional SOM directions. Although the authors acknowledge this limitation, it remains unclear whether the main result would hold with a broader refusal subspace. The evidence currently establishes separation from the selected directions rather than from refusal mechanisms more generally.
-
I am not fully convinced by the mechanistic interpretation in Section 4.2. The intervention result reaches significance on only two of the four models tested statistically, and the addition experiments are weaker. Moreover, a direction estimated at one late layer is applied across all layers and positions. A weak effect under this particular operation may reflect how the intervention is implemented, rather than a limited role for the predictive feature. The distinction between predictive performance and intervention effects is supported more clearly than a strong separation between the underlying mechanisms.
Questions for authors
-
Does the predictive advantage remain when the refusal subspace includes the additional SOM directions or other broader sets of refusal features?
-
Would the intervention conclusions change if each direction were applied only at its estimation layer, or at a separately selected intervention layer?
6 comments
Response to Reviewer tt35 (2/3)
[W2/Q1] The interpretation of “outside the refusal subspace” depends on how that subspace is constructed (continued)
[Q1] Does the predictive advantage remain when the refusal subspace includes the additional SOM directions or other broader sets of refusal features?
The answer to Q1 is that the advantage remains. With the 27 dimensions that include the SOM directions, the discrimination gain that adds is still significant at +0.0069, and when is widened with PCA it stays significant up to 50 dimensions. The gain is smaller than with 11 dimensions because, as grows, the share of the squared norm of inside rises from 0.075 to 0.225, and the share left outside shrinks. In the revision we will present this discrimination result as a supporting control. The main conclusion is that intervention, which edits the representation directly, and input search, which changes the input and searches, use different components to change success.
[W3/Q2] The mechanistic interpretation in Section 4.2 is not fully convincing
Compared with the magnitude-matched random control, removing by intervention gives p = 4.7e-15 on set A and p = 1.8e-06 on set C, the failed augmentations, across all ten models. It is correct that two of the four models we tested were significant; a single model has at most 8 combinations of dataset and augmentation family, so the smallest p value a sign test can give is 0.0078, and this limit explains the result. When we extend the test of adding to ten models and test each model separately, the sign test is significant for 5 models and the Wilcoxon test for 7 on set A, and in all ten models the mean success rate is higher than under the random control. The evidence for the effect therefore comes from the consistency across all ten models rather than from individual model tests.
We also examined how the way the intervention is applied affects the effect size. In the submission, we ablated , estimated at relative depth 0.8, at every layer and every position, and under this setting success rose in only 43 of 77 combinations. Changing only the operation to patching on the same 77 combinations raises this to 65 and the effect from +0.117 to +0.199. Adding with coefficient 1, from the last prompt token onward, at the single layer where it was estimated, the fitted layer, gives an effect of +0.126 with success rising in 25 of 31 combinations, and coefficient 3 gives +0.204 with 27 combinations rising. These are in Table R2. The effect was therefore unclear because of the way it was applied, and with a different setting the rise in success becomes clear.
Table R2. Effect of on set A (original harmful request) by application convention. The predicted sign for set A is a rise in success rate. Rows with the same number of combinations are comparable; the 77-combination rows (base success rate 0.148) and the 31-combination rows (base success rate 0.205) are not compared with each other.
| Convention | Operation | Combinations | Effect | Combinations with predicted sign |
|---|---|---|---|---|
| Relative depth 0.8, all layers and positions (in submission) | ablation | 77 | +0.117 | 43 / 77 |
| Relative depth 0.8, all layers and positions | patch | 77 | +0.199 | 65 / 77 |
| Fitted layer only, from the last prompt position | addition (coefficient 1) | 31 | +0.126 | 25 / 31 |
| Fitted layer only, from the last prompt position | addition (coefficient 3) | 31 | +0.204 | 27 / 31 |
Response to Reviewer tt35 (3/3)
[W3/Q2] The mechanistic interpretation in Section 4.2 is not fully convincing (continued)
In the same setting we also added . Under addition with coefficient 1, both and raise success on sets A and C and lower it on set B, the successful augmentations; in every case they exceed the random control, and no combination produced broken output. The results are in Table R3.
Table R3. Addition at coefficient 1, at the fitted layer, from the last prompt position (four models, 31 combinations). Set A: original harmful request; set B: successful augmentation (predicted sign: success drops); set C: failed augmentation. The effect is the change in success rate, with the number of combinations that move in the predicted sign in parentheses. The random direction has the same norm as the added direction, and the sign test is paired across combinations against it.
| Set | Direction | Effect (combinations with predicted sign) | Magnitude-matched random direction | Sign test |
|---|---|---|---|---|
| A | +0.126 (25/31) | +0.002 | 4.9e-05 | |
| A | +0.067 (25/31) | −0.005 | 3.2e-04 | |
| B | −0.223 (27/31) | −0.043 | 5.2e-03 | |
| B | −0.154 (28/31) | −0.015 | 8.4e-06 | |
| C | +0.075 (22/31) | −0.010 | 1.0e-04 | |
| C | +0.056 (24/31) | −0.009 | 1.5e-05 |
We agree that prediction and intervention must be distinguished, and our conclusion rests on that distinction. Discrimination alone cannot tell whether intervention and input search use different components; the intervention and input search experiments can. In the intervention experiment, removing raises success. In the input search experiment, a search whose objective is the component outside , , has a higher success rate than a search using the component inside in all four models, and the difference is significant in three.
[Q2] Would the intervention conclusions change if each direction were applied only at its estimation layer, or at a separately selected intervention layer?
Adding and with coefficient 1 at the single layer where each was estimated raises success on the original harmful requests by +0.126 and +0.067, with p = 4.9e-05 and 3.2e-04. Removing at one separately chosen layer, relative depth 0.8, applied alike across models, gives +0.145 over the magnitude-matched random control (p = 4.7e-15). In the revision we will use the single-layer setting as the default in the main text. The intervention conclusion therefore holds both at the estimation layer and at a separately chosen layer.
Conclusion
After running the checks you asked for, our original claim stands: the success coordinate lies outside the refusal subspace and can be reached from the input, and intervention and input search use different components to change success. Discrimination and intervention hold when is widened to 27 dimensions, and whether we apply the direction at the layer where it was estimated or at a separately chosen layer, adding and moves success in the direction our hypothesis predicts. In the revised manuscript that we upload during the discussion period, we will restructure Sections 4.1, 4.2 and 5 of the main text around this conclusion and make them easier to read, so that you can check it.
We hope this reply addresses your concerns, and please let us know if you need any further explanation.
Response to Reviewer tt35 (1/3)
Overview
Thank you for your detailed review. The Critical Weaknesses consist of three unnumbered paragraphs, so we refer to them in order as W1, W2 and W3, and to the two questions under Questions For Authors as Q1 and Q2. Q1 concerns the same issue as W2 and Q2 the same issue as W3, so we answer them together as [W2/Q1] and [W3/Q2].
We understood your main points to be three. The paper is hard to read, the interpretation of “outside the refusal subspace” depends on how the subspace is constructed, and the mechanistic interpretation in Section 4.2 is not fully convincing. For the first, we will reflect it by improving the writing in the revised manuscript that we will upload during the discussion period. For the second concern, we measured again after widening R, which is spanned by the 11 published refusal directions, to 27 dimensions by adding the 16 SOM directions, and the effect of removing R (+0.117, p = 4.2e-04) was maintained. The discrimination information added by the component outside R shrinks as R is widened but stays positive even at 27 dimensions. For the third concern, we showed that success moves in the hypothesized direction when the success direction is added at the single layer where the direction was estimated, and that in input search the search that uses the component outside R as its objective function has a higher success rate than the search that uses the component inside it in all four models. With these results, we have addressed all three points.
[W1] The paper is hard to read
We accept that the paper was hard to read and will improve the writing in the revised manuscript that we will upload during the discussion period. We thank you for finding the main ideas interesting, and we will polish the text of the revision so that these ideas reach readers clearly.
[W2/Q1] The interpretation of “outside the refusal subspace” depends on how that subspace is constructed
We left the SOM directions out of the submission because adding them to R lowers the share that is specific to refusal, as explained in Section 3.2. This time we re-measured discrimination and intervention with a 27-dimensional subspace made of R plus the 16 SOM directions; the results are in Table R1. The success direction u^sf is the direction fitted to the difference between the representations of successful and failed augmentations of the same request, and u^sf⊥ is that direction with its component in R removed. We measured discrimination by the gain, which is the increase in AUC when u^sf⊥ is added to a classifier fitted on the coordinates of R. We measured intervention by how much success rises when R is removed on set A, the original harmful requests.
Table R1. Discrimination and intervention under two definitions of R. Rows are claims; columns are definitions of R. Discrimination is the mean gain in paired AUC from adding u^sf⊥ to the label-fitted readout on R's coordinates. Intervention is the ablation effect on set A (the original harmful request) against its magnitude-matched random control, at one layer at relative depth 0.8 from the last prompt position, paired across combinations with a sign test.
The control in the intervention experiment is a magnitude-matched random control, which moves the representation along a random direction by the same amount as the intervention. Removing R on the original harmful requests raises success by +0.145 more than this control, while removing only the 16 SOM directions gives a smaller effect, +0.038 (p = 0.033). The effect remains at +0.117 when we widen R to 27 dimensions. The result therefore does not stop at the 11 selected directions and holds for a broader set of refusal directions.