- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In their work, the authors present a study of different internal representation signals from LLMs to identify a substitute or probe for predicting the harmfulness of a prompt. The authors first extract the internal representation features from the LLM (e.g., 1,912 features from Llama2-7B) and perform feature selection based on Shapley values. Furthermore, the authors propose a machine learning approach called HELM to classify prompts based on internal representations, using a weighted ensemble of 13 individual classifiers, and present various ablations to assess the contributions of individual features within the ensemble. The authors also compare HELM with various harmfulness detection methods and show promising results on the ToxicChat and SafetyPromptCollections datasets.
In general, the paper is well written and easy to follow. The idea is straightforward and interesting. However, it would be more beneficial for the paper's soundness if it were more focused on the HELM method and various ensemble ablations.
Critical strengths
S1: Proposal of the HELM method. The authors propose HELM, which is a method for an ensemble of different machine learning models to accurately predict the harmfulness of a prompt based on the internal representations of an LLM.
S2: Extensive ablation study. The ablations address various questions regarding the HELM method and confirm the importance of different internal representation features.
S3: Readability of the paper. The paper is really easy to read and understand. The structure and chain of thought make sense, and I think they need to be highlighted in the current era.
Critical weaknesses
W1: Inequality of selected features. We can see that all different internal representation features correspond mostly positively to the final results of HELM. However, some internal representations contain significantly more features than others. Maybe an approach beyond feature selection to normalize the features would help the HELM performance.
W2: Cross-entropy computation. On line 104, the authors mention that the cross-entropy loss feature is computed against a fixed reply, “Sure”. This selection is quite questionable, and it's not properly explained in the paper.
W3: More emphasis on the HELM method needed. After reading the introduction, I was unsure whether the main contribution is the HELM or the feature-selection analysis. I would definitely recommend restructuring the contributions and putting HELM in first place of the paper.
Questions for authors
Q1: In Table 5, the authors compare HELM to 5 different methods, and in Table 4 only HELM with its variation. Is there any specific reason to do so?
Q2: High false negative ratios may be dangerous in this domain. Do the authors plan to address the high false negative rates?
2 comments
W2: Line 104 computes cross-entropy against the fixed reply “Sure”. Why is this choice appropriate? The paper gives little justification.
We chose “Sure” following GradSafe (Xie et al., 2024), which pairs each prompt with the same affirmative reply and examines the gradients produced when the model predicts that reply. Holding the reply constant lets us compare loss and gradients across prompts. To test whether HELM depends on that particular word, we repeated the ToxicChat evaluation with another affirmative reply, “Okay”. We kept the data splits and feature definitions the same. We first changed the reply only when computing test features, then used “Okay” to compute both training and test features. Table 2 shows how F1 changes when “Sure” is replaced with “Okay” at test time or during both training and testing.
Table 2. ToxicChat harmful class F1 when training and testing use “Sure” or “Okay”
| Features | Training reply | Test reply | F1 |
|---|---|---|---|
| 320 selected | Sure | Sure | 0.788 |
| 320 selected | Sure | Okay | 0.784 |
| 320 selected | Okay | Okay | 0.794 |
| All | Sure | Sure | 0.799 |
| All | Okay | Okay | 0.797 |
With the same 320 selected features and the classifier still trained on “Sure”, changing only the test reply to “Okay” changed F1 from 0.788 to 0.784. Retraining on “Okay” gave 0.794. With all features, training and testing on the same reply gave 0.799 for “Sure” and 0.797 for “Okay”. On ToxicChat, another affirmative reply therefore gives similar F1 even when the classifier was trained on “Sure”. A ROC AUC above 0.5 means harmful prompts tend to have higher values than benign ones.
W3: The introduction leaves it unclear whether HELM or the feature selection analysis is the main contribution. Please list HELM first and give the method and ensemble ablations more emphasis to strengthen the paper.
We agree that the introduction should explain what HELM contributes. We now introduce HELM where we first describe the signal comparisons and state separately how we evaluate it as a detector. After considering the suggested order, we still find the comparison of internal signals the more interesting research question, so it remains the main contribution.
The submitted introduction said, “We provide HELM, an implementation that combines the extracted features with existing classifiers, and compare the resulting detector with established methods.”
After listing the seven signal groups, the revised introduction says, “We introduce HELM, a harmful prompt detector that combines signals from an LLM’s forward and backward passes. HELM lets us vary which signal groups the classifier uses while keeping the data splits and training procedure the same.”
The third contribution now says, “We evaluate HELM against seven established detectors and release the implementation in the supplementary material.” To address your suggestion about ensemble ablations, we compared the ensemble with the best single classifier on the same features and data across three training seeds. Mean F1 across the eight tasks was 0.927 for the ensemble and 0.924 for the single classifier. These scores are close. The ensemble lets us learn how much weight to give each classifier for each task.
Q1: Why does Table 5 compare HELM with five methods while Table 4 compares it only with its variant?
No. We removed the baseline rows from Table 5.
Q2: High false negative rates can be dangerous in this setting. How do you plan to reduce them?
Yes. We plan to make two changes to HELM’s training to reduce false negatives. In our error analysis, the most common misses were harmful requests presented as benign scenarios. First, we will add pairs of training examples that express the same harmful request directly and in a benign scenario. Both will carry the harmful label. We will also include genuinely benign requests on the same topics. For example, asking how to recognize a scam and asking for help carrying one out can use similar words. These examples should help HELM distinguish the user’s goal even when the topic and wording are similar.
Second, we will give more weight during training to harmful requests that HELM tends to miss. When such requests are rare, the classifier can achieve high overall accuracy while still missing many of them. Giving these errors a larger penalty during training could improve recall on similar requests.
Thank you for asking how HELM should be presented and how its design choices affect detection.
W1: The feature groups generally appear to help HELM, but some contain many more features than others. Could feature normalization address these differences and improve performance beyond feature selection alone?
We tested each of the seven groups on its own using one feature and then up to three features. For each task, we rank the features in each group on development data. Each of the eleven classifiers gives a feature one vote if its importance exceeds the task’s cutoff. We rank by vote count and break ties by the sum of positive importance scores. Each classifier uses the top feature or the top three, using all available features if the group has fewer than three. Table 1a compares the groups when each classifier uses only the highest ranked feature from its group.
Table 1a. Harmful class F1 using the highest ranked feature in each group
With one feature per group, total parameter gradient norm gives the highest mean F1 (0.570), closely followed by loss (0.568). The highest ranked hidden state or activation feature gives 0.508. Output entropy gives the lowest mean F1 (0.109). Since every classifier uses one feature, these differences reflect which feature it uses. Table 1b shows how the groups compare when each classifier can use up to three features.
Table 1b. Harmful class F1 using up to three features per group
With up to three features per group, hidden states and neuron activations give the highest mean F1 (0.721) and lead on five tasks. Loss leads on Information Leakage and Insult, and normalization statistics lead on ToxicChat. Output entropy again gives the lowest mean F1 (0.109). Hidden states and activations therefore rank first on average even with just three features, although total gradient norm is the strongest single feature.