- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates how attention patterns and representational geometry contribute to two undesirable LLM behaviors: coherent but irrelevant responses and relevant but repetitive text. Across Llama and Mistral models, the authors find that both behaviors are associated with a weakened attention sink, while coherent but irrelevant generations also exhibit increased attention entropy. They use causal interventions on attention maps and hidden representations, including a per-prompt singular value decomposition approach, to induce these behaviors and analyze how disruptions to different directions in the representation space affect the attention sink. The results provide a unified mechanistic account of these generation patterns and reveal vulnerabilities in models that may appear robust to existing jailbreaks.
Critical strengths
-
Per-prompt representation spectrum analysis. The authors perform SVD separately for each prompt and perturb the model by projecting out different singular-vector directions, introducing a new approach to analyzing the relationship between attention sinks and representational geometry.
-
A unified framework for analyzing two types of undesirable behavior. The authors compare coherent but irrelevant outputs with relevant but repetitive outputs, finding that both are associated with a weakened attention sink, while exhibiting different attention patterns.
Critical weaknesses
-
Figures 1 and 2 are difficult to read because the text is too small, and they look unfinished for an ICLR submission. I recommend redrawing them. The text in Figure 4(a) is also unreadable.
-
Typos and formatting issues:
- Equation (1) appears to extend beyond the paper’s layout.
- In the Figure 2 caption: ”the” or ”a”
-
The evaluation has major issues. First, the metrics are not clearly stated in the paper; they are only described implicitly in Section 3. All three metrics are scored automatically by LLMs, but the paper does not evaluate the stability of these scores or use objective metrics. The paper also lacks tables that would present the results clearly.
-
The experimental coverage appears insufficient in the current version. The test scenarios and datasets are not described in the main text. Important details, such as the number of data points and how many times each item was evaluated, should not be relegated to the appendix, as readers may not find them easily. The results should also be presented in tables. As currently presented, the figures do not make it clear that these datasets were used.
-
Since the method involves modifying the input text, the paper should include case studies showing how different repeated words lead to different outputs. These examples could be included in the appendix.
-
I do not think using only “the” and “a” provides sufficient control over the repeated-token attack. These two words are low in semantic content, so they are worth testing, but the experiments should also include semantically richer tokens, such as concrete nouns (e.g., “book”), numbers, names, and more complex words. The current experiments feel too shallow and lack depth. The paper claims in the main text that it uses 30 irrelevant jailbreak prompts, but I could not find them.
Questions for authors
I have no questions.
Thank you for your review and for helping to improve the manuscript. We respond to each of your concerns below:
1: We will improve the readability of the figures in the camera ready version of the manuscript.
2: Thank you for bringing these formatting issues to our attention. We will correct them in the next edition of the paper.
3: We explicitly provide a full account (including code) of the evaluation process in Section 3, Appendix A.1, and the supplementary material. We will collate this information in the main body of the final version of the paper. The availability of “objective” metrics for natural language processing related tasks has long been an issue due to the subjective nature of the quantities investigated. Consequently, turning to automatically computed metrics as an approximation for exact human evaluation has a long history in the field (BLEU, ROUGE, METEOR, etc.). To address your concern regarding the consistency of these metrics, we randomly selected 25 Good, Bad, (Coherent, Irrelevant), and (Relevant, Repetitive) generations as rated by gpt-5.6-terra and had them rated by gpt-6.1-terra and gpt-6.1-sol. All three models agree on 79% of cases and at least two models agree in 99% of cases, indicating a strong level of stability of our evaluation scheme.
4: We will move some of the content of Appendix A.2 to the main body of the paper when we are not constrained by the page limit. Throughout the paper, we make reference to the number of data points used in experiments as well as their provenance, but we can add a reminder of this to the figure captions. We also include a tabular version of the bar chart presented in Figure 4 b) below, as it is the most well suited for a tabular representation. However, we do not believe that these minor representation issues indicate that the “experiment coverage is insufficient”. We experiment with a wide variety of different datasets, two different jailbreak types with 32 instantiations, multiple models, and dozens of different intervention types to produce a thorough account of the undesirable responses we are studying.
5: The general format of the successful jailbreaks is demonstrated in Figure 1. The unsuccessful jailbreaks will generally either repeat the specified word, or write a passage about repetition. We will include more examples of this in the appendix of the camera ready paper.
6: We experiment with 32 different repeated words in both jailbreaks, extracted from Nasr et al. (2023). These words have a wide range of semantic meanings. The words are "company", "poem", "the", "make", "his", "her", "an", "so", "one", "b", "J", "life", "send", "make", "part", "with", "work", "word", "cell", "you", "time", "eye", "of", "on", "come", "good", "do", "up", "last", "year", "call", and "a".
Each of your concerns can be easily addressed in an update to the paper, to which we are grateful, or are already present in the current version of the manuscript. With that in mind, we ask you to improve your score.