acceptodds
Under review as a conference paper at ICLR 2027

Comparing Internal Signals for Harmful Prompt Detection

Abstract

Recent methods detect harmful prompts using the gradients or hidden representations of Large Language Models (LLMs). These signals capture different parts of how the model processes an input, and it is not clear how much each of them helps detection. We compare seven groups of statistical features computed from losses, gradients, logits, attention weights, hidden states, neuron activations, and normalization outputs. On ToxicChat and SafetyPromptCollections, we examine the feature distributions, the effect of removing each group, feature selection, and the choice of LLM. Removing the hidden state statistics and neuron activation rates lowers the average F1 the most, followed by removing the input embedding gradients. The effect of the other groups varies across datasets. We provide HELM, an implementation that combines these features with existing classifiers. The appendix gives the full feature definitions and further results.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
harmful prompt detection, large language models, jailbreak detection, internal model features, feature selection

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_L4mEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In their work, the authors present a study of different internal representation signals from LLMs to identify a substitute or probe for predicting the harmfulness of a prompt. The authors first extract the internal representation features from the LLM (e.g., 1,912 features from Llama2-7B) and perform feature selection based on Shapley values. Furthermore, the authors propose a machine learning approach called HELM to classify prompts based on internal representations, using a weighted ensemble of 13 individual classifiers, and present various ablations to assess the contributions of individual features within the ensemble. The authors also compare HELM with various harmfulness detection methods and show promising results on the ToxicChat and SafetyPromptCollections datasets.

In general, the paper is well written and easy to follow. The idea is straightforward and interesting. However, it would be more beneficial for the paper's soundness if it were more focused on the HELM method and various ensemble ablations.

Critical strengths

S1: Proposal of the HELM method. The authors propose HELM, which is a method for an ensemble of different machine learning models to accurately predict the harmfulness of a prompt based on the internal representations of an LLM.

S2: Extensive ablation study. The ablations address various questions regarding the HELM method and confirm the importance of different internal representation features.

S3: Readability of the paper. The paper is really easy to read and understand. The structure and chain of thought make sense, and I think they need to be highlighted in the current era.

Critical weaknesses

W1: Inequality of selected features. We can see that all different internal representation features correspond mostly positively to the final results of HELM. However, some internal representations contain significantly more features than others. Maybe an approach beyond feature selection to normalize the features would help the HELM performance.

W2: Cross-entropy computation. On line 104, the authors mention that the cross-entropy loss feature is computed against a fixed reply, “Sure”. This selection is quite questionable, and it's not properly explained in the paper.

W3: More emphasis on the HELM method needed. After reading the introduction, I was unsure whether the main contribution is the HELM or the feature-selection analysis. I would definitely recommend restructuring the contributions and putting HELM in first place of the paper.

Questions for authors

Q1: In Table 5, the authors compare HELM to 5 different methods, and in Table 4 only HELM with its variation. Is there any specific reason to do so?

Q2: High false negative ratios may be dangerous in this domain. Do the authors plan to address the high false negative rates?

2 comments

Thank you for asking how HELM should be presented and how its design choices affect detection.

W1: The feature groups generally appear to help HELM, but some contain many more features than others. Could feature normalization address these differences and improve performance beyond feature selection alone?

We tested each of the seven groups on its own using one feature and then up to three features. For each task, we rank the features in each group on development data. Each of the eleven classifiers gives a feature one vote if its importance exceeds the task’s cutoff. We rank by vote count and break ties by the sum of positive importance scores. Each classifier uses the top feature or the top three, using all available features if the group has fewer than three. Table 1a compares the groups when each classifier uses only the highest ranked feature from its group.

Table 1a. Harmful class F1 using the highest ranked feature in each group

Feature groupToxicChatInformation LeakageMisleading InformationIllegal ActivitiesPolitical LobbyingSexual ContentInsultHarmful SpeechMean F1
Hidden states and neuron activations0.0370.3580.5330.8850.5000.6590.4290.6670.508
Normalization layer statistics0.0270.3000.1430.7530.4850.4270.1880.0710.299
Attention statistics0.0000.6990.1450.6000.1450.2070.1400.1090.256
Loss metrics0.0000.7910.5210.4380.6670.6920.6670.7660.568
Total parameter gradient norm0.0000.6670.2540.8600.6800.7500.6020.7470.570
Output entropy0.1240.0380.1430.1130.0000.2810.1760.0000.109
Input embedding gradients0.0000.3000.0000.6670.1310.2810.4170.1400.242

With one feature per group, total parameter gradient norm gives the highest mean F1 (0.570), closely followed by loss (0.568). The highest ranked hidden state or activation feature gives 0.508. Output entropy gives the lowest mean F1 (0.109). Since every classifier uses one feature, these differences reflect which feature it uses. Table 1b shows how the groups compare when each classifier can use up to three features.

Table 1b. Harmful class F1 using up to three features per group

Feature group (number of features)ToxicChatInformation LeakageMisleading InformationIllegal ActivitiesPolitical LobbyingSexual ContentInsultHarmful SpeechMean F1
Hidden states and neuron activations (3)0.1820.8070.7010.9590.7910.8310.6520.8480.721
Normalization layer statistics (3)0.2680.7000.6070.6590.6390.6750.4650.3480.545
Attention statistics (3)0.0160.5540.4000.6240.2370.4230.3010.1670.340
Loss metrics (2)0.2040.8350.5600.7580.6520.7790.7200.8250.667
Total parameter gradient norm (1)0.0000.6670.2540.8600.6800.7500.6020.7470.570
Output entropy (1)0.1240.0380.1430.1130.0000.2810.1760.0000.109
Input embedding gradients (3)0.1480.6600.4270.6730.4170.4230.3430.3190.426

With up to three features per group, hidden states and neuron activations give the highest mean F1 (0.721) and lead on five tasks. Loss leads on Information Leakage and Insult, and normalization statistics lead on ToxicChat. Output entropy again gives the lowest mean F1 (0.109). Hidden states and activations therefore rank first on average even with just three features, although total gradient norm is the strongest single feature.

W2: Line 104 computes cross-entropy against the fixed reply “Sure”. Why is this choice appropriate? The paper gives little justification.

We chose “Sure” following GradSafe (Xie et al., 2024), which pairs each prompt with the same affirmative reply and examines the gradients produced when the model predicts that reply. Holding the reply constant lets us compare loss and gradients across prompts. To test whether HELM depends on that particular word, we repeated the ToxicChat evaluation with another affirmative reply, “Okay”. We kept the data splits and feature definitions the same. We first changed the reply only when computing test features, then used “Okay” to compute both training and test features. Table 2 shows how F1 changes when “Sure” is replaced with “Okay” at test time or during both training and testing.

Table 2. ToxicChat harmful class F1 when training and testing use “Sure” or “Okay”

FeaturesTraining replyTest replyF1
320 selectedSureSure0.788
320 selectedSureOkay0.784
320 selectedOkayOkay0.794
AllSureSure0.799
AllOkayOkay0.797

With the same 320 selected features and the classifier still trained on “Sure”, changing only the test reply to “Okay” changed F1 from 0.788 to 0.784. Retraining on “Okay” gave 0.794. With all features, training and testing on the same reply gave 0.799 for “Sure” and 0.797 for “Okay”. On ToxicChat, another affirmative reply therefore gives similar F1 even when the classifier was trained on “Sure”. A ROC AUC above 0.5 means harmful prompts tend to have higher values than benign ones.

W3: The introduction leaves it unclear whether HELM or the feature selection analysis is the main contribution. Please list HELM first and give the method and ensemble ablations more emphasis to strengthen the paper.

We agree that the introduction should explain what HELM contributes. We now introduce HELM where we first describe the signal comparisons and state separately how we evaluate it as a detector. After considering the suggested order, we still find the comparison of internal signals the more interesting research question, so it remains the main contribution.

The submitted introduction said, “We provide HELM, an implementation that combines the extracted features with existing classifiers, and compare the resulting detector with established methods.”

After listing the seven signal groups, the revised introduction says, “We introduce HELM, a harmful prompt detector that combines signals from an LLM’s forward and backward passes. HELM lets us vary which signal groups the classifier uses while keeping the data splits and training procedure the same.”

The third contribution now says, “We evaluate HELM against seven established detectors and release the implementation in the supplementary material.” To address your suggestion about ensemble ablations, we compared the ensemble with the best single classifier on the same features and data across three training seeds. Mean F1 across the eight tasks was 0.927 for the ensemble and 0.924 for the single classifier. These scores are close. The ensemble lets us learn how much weight to give each classifier for each task.

Q1: Why does Table 5 compare HELM with five methods while Table 4 compares it only with its variant?

No. We removed the baseline rows from Table 5.

Q2: High false negative rates can be dangerous in this setting. How do you plan to reduce them?

Yes. We plan to make two changes to HELM’s training to reduce false negatives. In our error analysis, the most common misses were harmful requests presented as benign scenarios. First, we will add pairs of training examples that express the same harmful request directly and in a benign scenario. Both will carry the harmful label. We will also include genuinely benign requests on the same topics. For example, asking how to recognize a scam and asking for help carrying one out can use similar words. These examples should help HELM distinguish the user’s goal even when the topic and wording are similar.

Second, we will give more weight during training to harmful requests that HELM tends to miss. When such requests are rare, the classifier can achieve high overall accuracy while still missing many of them. Giving these errors a larger penalty during training could improve recall on similar requests.

Reviewer_gugFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies which internal signals of LLMs help detect harmful prompts. The proposed method HELM collects 1,912 statistical features from seven groups: losses, total parameter-gradient norm, output entropy, attention, hidden states and activation rates, input-embedding gradients, and normalization outputs. It uses an AutoGluon ensemble to train the detector and selects important features using SHAP-based voting. The method is evaluated on ToxicChat and seven categories from SafetyPromptCollections. The experiments study how each feature group affects performance, compare HELM with existing detectors, test whether selected features work across different datasets, and evaluate the method using different LLMs.

Critical strengths
  1. The paper studies how different internal LLM signals work together to detect harmful prompts. Its main contribution is not a new algorithm, but an analysis of how useful these signals are when combined. To measure the importance of each feature group, the paper removes one group at a time, retrains the model, and compares the results. This is a reasonable way to test how much each group contributes to detection performance. The average F1 score drops from 0.943 to 0.837 when hidden-state and activation features are removed, and to 0.884 when input-embedding gradients are removed. These results suggest that both feature groups provide useful information that improves detection. These findings could help researchers build better detectors for harmful prompts.

  2. The paper recognizes that the feature groups have different numbers of features. It also explains that differences in average values between classes do not necessarily mean the model can accurately distinguish individual harmful and harmless prompts. In addition, the evaluation focuses on predicting dataset labels rather than preventing jailbreak attacks or handling attacks designed to bypass the detector. These points help clarify the scope of the study and make its contribution more convincing.

Critical weaknesses
  1. Several inconsistencies in the paper make it difficult to trust the experimental results. Some of these problems affect the main findings and model selection.
  • Table 3: The eight F1 scores for "Total Grad Norm only" have an average of 0.704, not the reported 0.821.
  • Table 6: Some results do not match the reported test-set sizes. For Information Leakage, 100 benign and 50 harmful examples, with FPR=0.020 and FNR=0.020, mean there should be 2 false positives and 1 false negative. This gives a harmful-class F1 score of about 0.970, not 0.980 as reported in other tables. Other results may also need an explanation, such as whether they were averaged across multiple runs.
  • Table A9: The selected settings do not always give the highest development-set F1 scores shown in the search results. For Political Lobbying, the selected settings (T=2, tau=1e-5) give 0.935 in Table A13, although 0.975 is available. For Sexual Content, the selected settings (T=8, tau=1e-4) give 0.975 in Table A14, although another setting reaches 1.000. The stated rule for breaking ties does not explain these choices.
  • Appendix B: The paper states that 0.937 is the highest average F1 score, but Table A2 reports 0.940 and 0.941 for two individual submodels. Tables A7 and A8 also show exactly the same sets of three metric values for two different comparisons. Since these tables use weighted metrics, their scores should not be directly compared with the harmful-class F1 scores in the main text. However, the evaluation settings and repeated values still need to be explained.
  1. The main comparison does not control for important differences between feature groups, making it difficult to draw strong conclusions about which signals are most useful. The experiments show how much each feature group contributes when combined with the others in the current method. However, the groups have very different numbers of features. For example, the hidden-state and activation group contains 1,441 features, while the total parameter-gradient norm is only one value. Although the paper acknowledges this issue, it does not test each group on its own, compare groups with similar numbers of features, or provide enough experiments comparing forward-only and backward-only features. Without these tests, it is difficult to tell whether the performance differences come from the types of signals or simply from the number of features.

  2. There are also other possible explanations for why the features can distinguish harmful from harmless prompts. Under the paper's definition, the global attention mean is always 1/N, meaning it depends on sequence length. In addition, Appendix I shows that the loss is calculated using both the user's text and the affirmative response. As a result, the loss and gradient features may reflect how predictable or long the input text is, rather than how strongly the model resists giving an affirmative response. These possibilities need to be tested before the paper can make stronger claims about what the internal signals actually measure.

  3. The paper seems to miss several closely related works that use internal LLM representations to detect harmful prompts [1][2][3]. The paper also identifies harmful prompts disguised as harmless scenarios as a major cause of false negatives, but does not discuss a related study [4]. These studies use different features, training methods, and evaluation settings from HELM, so they do not necessarily reduce the value of this paper. However, they address closely related problems, especially using internal LLM signals to detect harmful prompts and understanding how different signals can improve detection. The paper should compare HELM with these methods and clearly explain what new findings or improvements it provides. This is especially important because the paper's main contribution is comparing different internal signals, rather than introducing a new detection algorithm.

  4. The paper does not report how much the results vary across different training runs or test samples, making it difficult to judge how reliable the comparisons are. In addition, the experiments that remove individual feature groups are only conducted using Llama 2, so it is unclear whether the findings also apply to other LLMs. The improvements from feature selection (average F1 increasing from 0.937 to 0.943) and ensembling (from 0.942 to 0.943) are very small. Without reporting how much the results vary, it is difficult to know whether these improvements are meaningful or simply due to random variation. The paper also does not clearly explain how the baseline methods are adapted or how their decision thresholds are selected. As a result, it is unclear whether all detectors have access to the same amount of training data from the target datasets, making it difficult to judge whether the comparisons are fair.

[1] Jiao, Difan, et al. "LLM Safety From Within: Detecting Harmful Content with Internal Representations." Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.
[2] Liu, Xiuming, et al. "Quantifying Large Language Model Attacks Through the Lens of Model Cognition." 33rd USENIX Security Symposium (USENIX Security 2026). 2026.
[3] Hu, Xulin, et al. "Tracing the Dynamics of Refusal: Exploiting Latent Refusal Trajectories for Robust Jailbreak Detection." Proceedings of the 43rd International Conference on Machine Learning. 2026.
[4] Farzam, Amirhossein, et al. "Hiding in Plain Text: Detecting Concealed Jailbreaks via Activation Disentanglement." arXiv preprint arXiv:2602.19396 (2026).

Questions for authors
  1. Could the authors recheck the average F1 score in Table 3, the selected settings in Table A9 against the results in Tables A13/A14, and the error rates in Table 6 against the main results?

  2. For Tables A7 and A8, please clarify which datasets, data splits, feature sets, and averaging methods were used for each column. Why do two different comparisons report exactly the same three metric values?

  3. Could the authors clarify which token positions are included in the loss calculation and how the loss is combined across tokens before computing gradients?

  4. Could the authors report how much the main feature group ablation results vary across runs or test samples? Please also clarify whether the baseline methods use thresholds selected on the same development data and which baselines were trained using the same target dataset examples as HELM.

8 comments

Thank you for identifying these inconsistencies.

Several inconsistencies undermine confidence in the main findings and model selection. In Table 3, the eight F1 scores for “Total Grad Norm only” average 0.704 rather than the reported 0.821. Please recheck that average, the Table A9 settings against Tables A13 and A14, and the Table 6 error rates against the main results.

We recalculated the mean from the eight scores in the submission. They sum to 5.633, giving 0.704 when divided by eight. The reported mean of 0.821 was an arithmetic error. Table 1 gives the corrected mean F1 alongside the results for each task. Changed values are shown in bold.

Table 1. Updated submission Table 3: detection results and equally weighted mean harmful class F1 across eight tasks

MethodToxicChat PrecisionToxicChat RecallToxicChat F1Information LeakageMisleading InformationIllegal ActivitiesPolitical LobbyingSexual ContentInsultHarmful SpeechMean F1
OpenAI Moderation API0.8150.1450.2460.1250.2700.6520.6670.8950.8090.7010.546
Perspective API0.6140.1480.2380.0380.3030.5480.6490.7960.6950.6670.492
InjecGuard0.3400.3220.3310.3990.4130.6050.6010.6670.6080.6180.530
Llama Guard0.7530.6670.7070.3810.5110.7300.7540.8830.8180.7770.695
WildGuard0.4800.9190.6310.5450.5620.7870.7720.8570.8300.8480.729
GradSafe-Adapt0.6200.8720.7250.7020.7550.6920.8100.7400.6680.7840.735
ToxicDetector0.9240.3180.4730.9620.9200.9580.9450.9500.9090.9700.886
HELM0.8680.7430.8010.9800.9610.9900.9470.9580.9500.9620.944
HELM (No feature selection)0.8870.7180.7940.9700.9500.9900.9250.9580.9390.9700.937
HELM (Best submodel only)0.8360.7760.7990.9800.9571.0000.9420.9520.9440.9590.942
HELM (Llama 3-8B)0.9220.9440.9330.9870.9730.9900.9470.9670.9620.9870.968
HELM (Vicuna-v1.5-7B)0.9580.9030.9300.9840.9670.9900.9400.9620.9550.9800.964

The Table 6 error rates appear inconsistent with the test counts. For Information Leakage, 100 benign and 50 harmful prompts with FPR and FNR both 0.020 imply two false positives and one false negative. Why does this give harmful class F1 of 0.970 when other tables report 0.980? Could averaging across runs explain other discrepancies?

We checked the number of benign and harmful prompts in each test set. Table 2 gives the corrected counts for the training and test splits. Changed values are shown in bold.

Table 2. Updated submission Table 1: benign and harmful counts in each split

DatasetTrain BenignTrain HarmfulTest BenignTest Harmful
ToxicChat4,6983844,721362
Information Leakage60020010050
Misleading Information60020010050
Illegal Activities60020010050
Political Lobbying60020010046
Sexual Content60020010047
Insult60020010050
Harmful Speech6002009750

We confirmed that several test-set counts in submitted Table 1 were incorrect. To check the two F1 scores, we identified the model behind each and compared error rates with F1 for the same model. Table 3 gives the false positive and false negative rates for the model trained with all features.

Table 3. Additional check of submission Table 6 using all features for all eight tasks

RateToxicChatInformation LeakageMisleading InformationIllegal ActivitiesPolitical LobbyingSexual ContentInsultHarmful SpeechMean rate
False positive rate0.0070.0200.0300.0000.0400.0300.0200.0210.021
False negative rate0.2820.0200.0400.0200.0650.0210.0800.0200.069

Your calculation is correct. For Information Leakage, 49 true positives, two false positives and one false negative give a harmful class F1 of 0.970. This is the result for HELM without feature selection in submitted Table 3. The 0.980 in the same table is for HELM with feature selection. Each model was evaluated once.

In submitted Table 6, all seven SafetyPromptCollections columns used all features. The ToxicChat error rates (0.009 and 0.257) match the model using selected features. For ToxicChat with all features, only 260 true positives, 33 false positives and 102 misses fit the published precision, recall, F1 and test counts after rounding. This gives a false positive rate of 0.007 and a false negative rate of 0.282. With these counts, the error rates on all eight tasks agree with the all-feature F1 scores in Table 1 to three decimals.

Why did Table A9 select settings below the highest development F1? For Political Lobbying, T=2 and τ=10⁻⁵ give 0.935 in Table A13, which reaches 0.975. For Sexual Content, T=8 and τ=10⁻⁴ give 0.975 in Table A14, which reaches 1.000. The tie-breaking rule does not explain either choice.

When preparing Table A9, we mistakenly chose the second highest for Political Lobbying and Sexual Content. We corrected both entries by checking all settings against the full development results and choosing the highest F1 for each category. When settings tied at the highest F1, we chose the one that retained fewer features. Table 4 lists the vote requirement and importance cutoff for each task. Changed values are shown in bold.

Table 4. Updated submission Table A9: corrected vote counts and importance cutoffs

DatasetVotes requiredImportance cutoff
ToxicChat910⁻⁷
Information Leakage910⁻⁸
Misleading Information810⁻⁷
Illegal Activities510⁻⁴
Political Lobbying510⁻⁴
Sexual Content210⁻⁵
Insult410⁻⁷
Harmful Speech510⁻⁶

For Political Lobbying, two votes at 10⁻⁵ gave 0.935. Three and five votes at 10⁻⁴ both gave 0.975. We chose five because it kept fewer features. For Sexual Content, eight votes at 10⁻⁴ gave 0.975, and two votes at 10⁻⁵ gave 1.000.

To test whether these corrections changed the ablation, we repeated all seven removals for the two categories, recalculating importance and using the corrected settings before retraining. Table 5 shows F1 with all groups and after each group removal, including the revised results for Political Lobbying and Sexual Content. Changed values are shown in bold.

Table 5. Updated submission Table 2: harmful class F1 for Llama 2 ablations after correcting the settings for two categories

Features usedToxicChatInformation LeakageMisleading InformationIllegal ActivitiesPolitical LobbyingSexual ContentInsultHarmful SpeechMean F1
All groups0.8010.9800.9610.9900.9470.9580.9500.9620.944
Without loss metrics0.7840.9330.9130.9470.9460.9460.8730.9470.911
Without total parameter gradient norm0.7770.9600.9400.9900.8640.9790.9330.9130.920
Without output entropy0.7850.9600.9130.9900.9570.9680.9200.9270.928
Without attention statistics0.7860.9270.9400.9730.9010.9680.9330.9400.921
Without hidden states and neuron activations0.7020.8530.8730.8870.8170.9470.8330.8530.846
Without input embedding gradients0.7180.9200.9070.9400.9460.9890.8800.8670.896
Without normalization layer statistics0.7700.9600.9330.9900.9570.9780.9130.9130.927

With all groups, F1 is 0.947 on Political Lobbying and 0.958 on Sexual Content. Removing hidden states and activations lowers F1 to 0.817 and 0.947, respectively. The drop is larger on Political Lobbying. Across all eight Llama 2 tasks, removing this group still causes the largest drop in mean F1.

Why does Appendix B call 0.937 the highest mean F1 when two individual submodels in Table A2 score 0.940 and 0.941?

We recalculated the eight-task mean for every classifier to check which scored highest. Table 6 compares the classifiers on each task and gives their mean F1 across all eight tasks. Changed values are shown in bold.

Table 6. Updated submission Table A2: mean harmful class F1 across eight tasks for classifiers trained without feature selection

ClassifierToxicChat PrecisionToxicChat RecallToxicChat F1Information LeakageMisleading InformationIllegal ActivitiesPolitical LobbyingSexual ContentInsultHarmful SpeechMean F1
HELM (No feature selection)0.8870.7180.7940.9700.9500.9900.9250.9580.9390.9700.937
LightGBMXT0.8480.7100.7730.9900.9480.9800.9210.9790.9030.9590.932
XGBoost0.8140.7270.7680.9800.9380.9700.8570.9890.9050.9280.917
NeuralNetFastAI0.8640.7210.7860.9800.9500.9900.9470.9580.9390.9700.940
CatBoost0.8350.7430.7870.9800.9031.0000.8860.9470.9150.9390.920
NeuralNetTorch0.8230.7600.7900.9800.9700.9900.9250.9580.9360.9800.941
LightGBM0.8300.6600.7350.9700.9490.9700.8990.9890.8440.9260.910
RandomForestEntr0.9520.4360.5980.9700.8540.9800.9070.9570.9260.9130.888
RandomForestGini0.9740.4200.5870.9590.8760.9800.8710.9110.9010.9250.876
ExtraTreesEntr0.9910.3070.4680.9500.8910.9690.8430.9450.8540.9130.854
ExtraTreesGini0.9900.2870.4450.9180.8760.9690.8290.9210.8760.9010.842
LightGBMLarge0.8390.6930.7590.8940.8720.9130.8390.9070.8700.9070.870
KNeighborsUnif0.6580.4670.5460.8430.6950.9160.7420.8410.8160.8890.786
KNeighborsDist0.6730.4890.5660.8430.6950.9160.7270.8410.8160.8980.788

NeuralNetTorch has the highest individual mean F1 (0.941), followed by NeuralNetFastAI (0.940). The ensemble without feature selection averaged 0.937. We incorrectly described it as the highest mean across all models. We corrected the Appendix B text and recalculated the means.

Why do the two comparisons in Tables A7 and A8 report the same three metric values? Please specify the dataset, split, feature set and averaging method for each column. The weighted metrics in these tables are not directly comparable to harmful class F1 in the main text.

Both submitted tables contained errors. A7 asked whether combining classifiers improves detection over using the best single classifier, NeuralNetTorch. We compared the ensemble and NeuralNetTorch on the same ToxicChat test prompts and the same features. Table 7 compares their precision, recall and F1 at each feature count.

Table 7. Harmful class precision, recall and F1 on ToxicChat for the classifiers in submitted Table A7

FeaturesClassifierPrecisionRecallF1
1,039Voting ensemble0.8470.7930.819
1,039NeuralNetTorch0.8520.7290.786
320Voting ensemble0.8360.7460.788
320NeuralNetTorch0.8360.7460.788

Submitted A7 reported the ensemble’s F1 weighted by the counts of benign and harmful test prompts (0.975), but the NeuralNetTorch value (0.792) was for the harmful class alone. This difference in averaging made the apparent advantage much larger.

On ToxicChat, the ensemble achieved higher harmful class F1 than NeuralNetTorch with 1,039 features and tied it with 320. We removed A7 because revised Table A3 already compares the ensemble with all 13 submodels.

We used A7 as the template for A8 and mistakenly. Table 8 compares voting with selection by summed importance on ToxicChat.

Table 8. Harmful class precision, recall and F1 on ToxicChat for the feature selection methods in submitted Table A8

SelectionPrecisionRecallF1
Voting0.8470.7930.819
Summed importance0.8260.6550.730

With 1,039 features per method, voting gave higher harmful class F1 on ToxicChat than summing importance scores (0.819 versus 0.730).

Reviewer_vTLMOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper compares seven groups of internal LLM signals for harmful prompt detection. The authors implement HELM to combine these features using an AutoGluon ensemble and SHAP-based feature selection. The main contribution is a systematic comparison of signal contributions within a shared detection pipeline. Evaluation covers ToxicChat and seven SafetyPromptCollections datasets through group ablations and comparisons with existing detectors and different LLMs. The authors also examine feature distributions and the transfer of selected features across datasets. Removing hidden state statistics and neuron activation rates causes the largest performance drop, followed by removing input embedding gradients.

Critical strengths
  1. Useful insight into which internal signals matter. The paper compares seven signal groups using the same extraction and classification procedure. Its group ablations show what each group adds when the other signals remain available. The largest effects come from hidden state statistics and neuron activation rates, followed by input embedding gradients. I see this as a useful contribution because it gives concrete guidance for designing detectors around internal signals.

  2. A straightforward method with supporting evidence. HELM combines statistical features with established classifiers in a straightforward way. It improves F1 over the tested baselines on ToxicChat and most SafetyPromptCollections categories. Experiments with different LLMs and feature selection give further support for the approach. These results make the method convincing within the evaluated settings and support my positive assessment.

Critical weaknesses
  1. The evaluation has a limited scope. In fact this is my main consideration in terms of where the authors should clearly improve. Training and testing use the same datasets and the main group ablations only use Llama 2-7B. The paper also does not evaluate adaptive attacks against the detector. Experiments with other LLMs evaluate the complete detector without repeating the main signal comparisons. This leaves uncertainty about whether the central findings hold across different models and attack distributions.

  2. Unequal feature-group sizes complicate the main comparison. The hidden state and neuron activation group contains 1,441 features. The entire feature set contains 1,912 features across all seven groups. Its removal produces the largest performance drop but also removes most of the available features. This makes it difficult to separate the value of the signal type from the number of features it provides. Comparisons using matched feature counts would strengthen the paper’s main conclusion about which signals matter most.

Questions for authors
  1. Clarification on the ablation results. The paper acknowledges that the feature groups have very different sizes. Do you have existing comparisons with matched feature counts or classifiers trained on individual groups? Such results would help separate signal type from feature count when interpreting the ablations.

  2. Clarification on Appendix F. Tables A7 and A8 report identical values for two different comparisons. Could you confirm whether both tables are correct and explain how the experiments differ? Please also clarify the datasets and metric averaging used in Appendix F. How does Table A7’s large ensemble advantage relate to the near tie reported in Table 3?

5 comments

Thank you for pointing out the limits of the evaluation.

The main concern is the limited evaluation. Training and testing use the same datasets, and the main group ablations cover only Llama 2-7B.

We test whether a trained classifier can detect harmful prompts on a different dataset. The submission used separate training and test splits from each dataset. In its feature transfer experiment, we selected features on one dataset but trained a new classifier on the other. Here we transferred the trained classifier itself. We trained it on all 5,082 ToxicChat training prompts and tested it on the seven SafetyPromptCollections test sets. In the other direction, we trained on all 5,600 SafetyPromptCollections training prompts and tested on all 5,083 ToxicChat test prompts. In both directions, we tested the classifier on the other dataset without retraining it. Table 1 shows precision, recall and F1 for classifier transfer in both directions.

Table 1. Harmful class precision, recall and F1 when the trained classifier is tested on a different dataset

Training datasetTest datasetPrecisionRecallF1
SafetyPromptCollectionsToxicChat0.2550.2680.261
ToxicChatInformation Leakage1.0000.0800.148
ToxicChatMisleading Information0.0000.0000.000
ToxicChatIllegal Activities0.9760.8000.879
ToxicChatPolitical Lobbying0.6670.1740.276
ToxicChatSexual Content0.9430.7020.805
ToxicChatInsult1.0000.1000.182
ToxicChatHarmful Speech0.9710.6800.800
ToxicChatMean over seven categories0.7940.3620.441
Both transfer directionsMean over eight test tasks0.7260.3500.419

When we trained on ToxicChat and tested on SafetyPromptCollections, mean precision was 0.794 and mean recall was 0.362. Recall was higher on two topics covered by both datasets, Illegal Activities (0.800) and Sexual Content (0.702). The classifier missed all harmful prompts in Misleading Information. When we trained on SafetyPromptCollections and tested on ToxicChat, precision was 0.255 and recall was 0.268. A classifier trained on one dataset therefore missed many harmful prompt topics in the other. Differences in the types of requests in the training sets may explain why recall varied by topic. Training on a wider range of harmful requests could improve recall on a new dataset.

Other LLMs were tested only with the full detector. Do the main signal comparisons hold on those models?

To test whether the Llama 2 findings hold on Vicuna and Llama 3, we repeated all seven group removals on the same eight tasks. For each LLM, we retrained on the remaining six groups and compared F1 with its all-group result. Table 2 shows how removing each signal group affects F1 on Vicuna.

Table 2. Harmful class F1 after each group removal on Vicuna-v1.5-7B

DatasetAll groupsWithout lossWithout total gradient normWithout entropyWithout attentionWithout hidden states and activationsWithout input embedding gradientsWithout normalization
ToxicChat0.7580.7300.7620.7240.7770.5670.7410.737
Information Leakage0.9800.9800.9800.9900.9700.9050.9900.961
Misleading Information0.9490.9500.9710.9700.9280.8170.9710.960
Illegal Activities0.9800.9690.9700.9690.9690.9590.9580.969
Political Lobbying0.9680.9570.9460.9570.7030.8150.9450.899
Sexual Content0.9390.9570.9480.9480.9290.8910.9680.939
Insult0.9170.9170.9050.9150.9280.8460.9180.903
Harmful Speech0.9260.9500.9050.9130.9260.9490.9470.950
Mean over eight tasks0.9270.9260.9230.9230.8910.8440.9300.915

On Vicuna, removing hidden states and activations lowers mean F1 from 0.927 to 0.844, with drops on seven of eight tasks. This is the largest mean drop on both Vicuna and Llama 2. Removing input gradients lowers mean F1 on Llama 2 but changes little on Vicuna (0.927 to 0.930). Table 3 shows the same comparison on Llama 3.

Table 3. Harmful class F1 after each group removal on Llama 3-8B Instruct

DatasetAll groupsWithout lossWithout total gradient normWithout entropyWithout attentionWithout hidden states and activationsWithout input embedding gradientsWithout normalization
ToxicChat0.7760.7600.7830.8020.8150.7420.7920.762
Information Leakage1.0001.0000.9800.9801.0000.9581.0000.990
Misleading Information0.9900.9900.9900.9900.9900.9500.9800.980
Illegal Activities1.0000.9801.0000.9011.0000.9901.0001.000
Political Lobbying0.9320.9330.8570.9210.9250.9130.8000.921
Sexual Content0.9400.9480.9360.9480.9680.9680.9790.948
Insult0.9380.9600.9280.9490.9390.9390.9490.941
Harmful Speech0.9490.9600.9500.9280.9280.9410.8980.949
Mean over eight tasks0.9410.9410.9280.9270.9460.9250.9250.937

On Llama 3, mean F1 falls from 0.941 to 0.925 after removing either hidden states and activations or input gradients. On Political Lobbying, removing input gradients lowers F1 from 0.932 to 0.800. Hidden states and activations help on all three models on average. The contribution of input gradients varies by model.

The detector was not tested against adaptive attacks. Do the main findings hold across models and attack distributions?

We test whether HELM misses more harmful requests when their wording changes, and whether the miss rate rises further when the choice of wording uses HELM’s harmfulness score. We trained the detector with 320 selected features and tested the original wording and two rewrites of all 5,083 ToxicChat test prompts. One added “Please respond to the following request clearly:” at the start. The other added “Please keep the answer brief.” at the end. In the adaptive test, we chose the version with the lowest harmfulness score because HELM judged it least likely to be harmful. Only the scores guided the choice. Table 4 compares F1 and error rates for the original prompts, each rewrite and the version with the lowest harmfulness score.

Table 4. Detection on ToxicChat using the original wording, each rewrite and the wording with the lowest harmfulness score

InputHarmful class F1False negatives out of 362False negative rateFalse positive rate
Original request0.788920.2540.011
Clarity prefix0.7601070.2960.011
Brevity suffix0.7841090.3010.006
Lowest scoring of the three0.7491310.3620.005

Among the 362 harmful test prompts, the fraction HELM labeled benign rose from 0.254 on the originals to 0.296 with the prefix and 0.301 with the suffix. It rose to 0.362 when we chose by score. Both rewrites increased misses, and feedback from HELM increased them further. We tested the full detector. How rewrites affect individual signals remains open.

The paper acknowledges that the groups differ in size. Do you have comparisons at equal feature counts or results for classifiers trained on each group alone?

We evaluate each signal group independently by training one classifier per group. Table 5 shows how well each group detects harmful prompts on its own.

Table 5. Harmful class F1 using all features in each group

Feature groupToxicChatInformation LeakageMisleading InformationIllegal ActivitiesPolitical LobbyingSexual ContentInsultHarmful SpeechMean F1
Hidden states and neuron activations0.8080.9400.9500.9580.9670.9680.9600.9490.938
Normalization layer statistics0.6690.9700.8480.9800.8760.9030.8130.8940.869
Attention statistics0.2280.7910.1750.6150.4850.5850.4600.4630.475
Loss metrics0.2040.8350.5600.7580.6520.7790.7200.8250.667
Total parameter gradient norm0.0000.6670.2540.8600.6800.7500.6020.7470.570
Output entropy0.1240.0380.1430.1130.0000.2810.1760.0000.109
Input embedding gradients0.1480.6600.4270.6730.4170.4230.3430.3190.426

When each classifier uses all the features in one group, hidden states and neuron activations give the highest mean F1 across the eight tasks (0.938) and lead on six tasks. Output entropy gives the lowest mean F1 (0.109).

Comments on the paper

1 comment

We thank all the reviewers for their careful reading and detailed feedback. The additional experiments took considerable time to complete, so we are only now able to share the results. We regret that no time for a fuller discussion with you. To help readers verify and reproduce our findings, we have uploaded updated supplementary material with the experiment logs, results, and code to OpenReview.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.