- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes the LLM junking problem. The task is to find a (junk) token sequence that maximizes the probability of generating a pre-defined harmful target prefix (without any anchored initialization, prompt template, or adversarial restrictions). To solve this problem, the paper proposes a simple Greedy Random Search (GRS) algorithm that updates a single token per iteration to minimize the negative log-likelihood.
Experiments on a subset of 50 harmful behaviors from AdvBench and across four safety-aligned LLMs (LLaMA-2-Chat-7B, Gemma-7B, Qwen-2.5-7B, Mistral-7B) demonstrate that with a sufficient number of function evaluations, a moderate ASR can be achieved. Perplexity analysis and illustrated successful examples suggest that the junk token sequences exhibit significantly higher complexity.
Critical strengths
-
The paper conducted a thorough overview of the related works on LLM jailbreaks and backdoor discovery (Section 1 and Section 3).
-
The problem of discovering non-semantic token sequences that elicit target harmful behaviors in aligned LLMs is different from the majority of prior literature on LLM jailbreaks.
Critical weaknesses
-
The LLM junking problem is poorly motivated. From my perspective, the proposed problem is just a relaxed variant of LLM jailbreaks without initializing at a given harmful prompt (paired with the target harmful response). For example, GCG or its later variants can be naturally adapted to optimize for the junk token sequence, for maximizing the generation probability of a harmful target prefix.
-
Given the fact that the junk token sequences (that successfully elicit the intended harmful response) have significantly high perplexity, they may be easily defended using a perplexity filter.
-
A few statements are misleading. The term "natural backdoor" is misleading, because these junk token sequences are far from natural. The paper argues that the junking problem differs from the widely studied LLM jailbreak attacks, but the proposed optimization objective and algorithm can be regarded as an adversarial optimization in a larger search space, which suggests the difference is, in fact, marginal. The paper uses "hardness" in the title, but provides empirical results only for a simple attack realization; this does not establish the hardness of a machine learning or optimization problem.
-
The paper seems to focus only on white-box attack scenarios. No attack transferability experiments are presented.
-
A relatively minor issue is the scalability of the experimental results, which focus only on 50 harmful target samples and 7B models. The high complexity of the proposed GRS algorithm can be a computational bottleneck for larger models.
Questions for authors
Can the authors elaborate on the significance of the proposed LLM junking problem and how it differs from the widely studied problem of LLM jailbreak attacks?
5 comments
Rebuttal (Part 2/2)
4. White-box setting and transferability.
We agree that our current experiments are restricted to a white-box setting and that transferability is an important question for evaluating a practical attack. However, as emphasized above, our objective is not to propose a practical attack. The model is used as an experimental object through which we study coerciveness, and the ability to access model evaluations is part of the experimental setup.
5. Number of models, targets, and computational scalability.
We agree that the current experimental evaluation is limited to 50 target samples and 7B-scale models. Indeed, as we specify, the work is intended to be as a first proof-of-concept study. We extended the analysis to more models (see response to reviewer Nzhu).
We emphasize that the computational cost should be interpreted in the context of the purpose of the experiment. We are not proposing GRS as an efficient attack algorithm for deployment. Instead, we use it as a deliberately simple instrument for testing whether successful target sequences can be found at all without exploiting semantic information or sophisticated optimization techniques.
In this context, the fact that approximately 100k candidate evaluations are sufficient to recover successful sequences is itself informative. A generic discrete search problem over a 100-dimensional space could, in the worst case, require an exponentially growing number of evaluations. The fact that random search succeeds with a comparatively small budget suggests that the search problem encountered by the model is substantially more tractable than such a generic worst-case problem. This is the phenomenon we aim to investigate, rather than the computational efficiency of GRS itself.
(Question) On the significance of LLM junking and its relationship to jailbreaks.
We would like to clarify that we do not claim that LLM junking constitutes a fundamentally new class of adversarial attacks, nor that it is unrelated to the extensively studied jailbreak problem. The two problems are naturally related because both consider finding inputs that induce a model to generate behavior that is otherwise suppressed by alignment.
The distinction lies primarily in the scientific question we use the problem to investigate. In jailbreak research, the objective is typically to develop an effective procedure for bypassing the model's safeguards and eliciting a desired harmful response. In contrast, we use the harmful-target setting as a controlled testbed to study a different question: how easy is it to coerce an aligned LLM into generating a specified target when the search procedure is given essentially no information about how to achieve that target?
This is why we deliberately consider an extremely simple search procedure based on random search and single-token replacement. We do not aim to compete with sophisticated jailbreak algorithms in terms of attack efficiency. Instead, we ask whether sophisticated optimization, semantic information, carefully designed prompts, or even a meaningful initialization are necessary in the first place. The fact that successful sequences can be recovered using such a simple procedure is the central observation we want to highlight.
Thus, the significance of junking is not that it provides a practically superior jailbreak, but that it provides a controlled framework for probing the coerciveness of LLMs. If a model can be reliably coerced toward a specified target through an information-poor random-search procedure, this suggests that the corresponding set of successful inputs is sufficiently accessible to make the problem tractable despite the high-dimensional discrete search space. We believe this observation is complementary to existing jailbreak research: jailbreak methods primarily study how to efficiently find successful inputs, whereas our study asks how much information and optimization sophistication are actually necessary to find them at all.
We will revise the manuscript to make this distinction explicit.
We thank the reviewer again for their time in reviewing our manuscript and their valuable feedback. As the discussion phase is coming to an end, we would appreciate knowing whether our response has adequately addressed your concerns.
Rebuttal (Part 1/2)
We thank the reviewer for the careful comments. We agree that some aspects of the motivation and terminology could be clarified. Our main point is that the goal of the paper is not to introduce a new practical jailbreak attack, but to use the junking problem as a controlled setting to study a more fundamental question: how difficult is it to find a token sequence that coerces an aligned LLM into generating a desired target, when the search procedure is given essentially no semantic information about how to achieve this? We clarify the points below.
1. Motivation and relation to jailbreaks.
We agree that the junking problem can be viewed as a relaxed version of the jailbreak setting, and that existing methods such as GCG could in principle be adapted to optimize for the same objective. However, this is not the goal of the work. Rather, the key distinction is in the purpose of the problem and the information available to the search procedure.
In a conventional jailbreak setting, the optimization procedure is typically designed to efficiently construct an adversarial prompt, often exploiting a harmful target, a carefully chosen initialization, gradients, or other available information about the model and the target behavior. In contrast, our objective is to investigate whether the coercion problem can be solved even when these sources of information are deliberately removed. We therefore use random search and a minimal single-token replacement procedure as experimental probes.
This distinction is important for our conclusions. If a sophisticated optimization method such as GCG finds a successful sequence, this demonstrates that the method can exploit information about the optimization landscape. In contrast, if a very simple random-search procedure can reliably find such sequences, this suggests that the underlying problem itself has a relatively large or easily accessible set of successful solutions. Our claim is therefore not that junking constitutes a fundamentally new class of jailbreak attacks, but that it provides a useful experimental setting for studying the intrinsic coerciveness of LLMs.
We will revise the paper to make this motivation explicit.
2. High perplexity and perplexity-based defenses.
We agree that the high perplexity of the junk sequences makes perplexity filtering a potentially relevant defense. However, this observation does not undermine the main result of the paper. Indeed, the fact that the successful sequences are highly unnatural is part of the reason why they provide a useful controlled setting for our study: we deliberately remove semantic and natural-language structure and ask whether the model can nevertheless be coerced into producing a desired target.
Our goal is not to argue that these particular sequences constitute robust practical attacks that evade existing defenses (i.e. the choice of adversarial setting is just due to the fact that aligned model should avoid generating "harmful" targets, thus coercing a model to generate them should be more challenging). A perplexity filter may indeed make this specific realization of the problem substantially less practical. Rather, our observation is that the model's generation behavior can be manipulated using sequences that contain essentially no semantic information and that can be discovered by a simple search procedure.
3. Terminology and the use of "hardness."
We agree with the reviewer that the term "natural backdoor" can be misleading, particularly because the sequences found by our procedure are intentionally unnatural. Our intended use of "natural backdoor" refers to the functional property of a token sequence that steers the model toward a specified target, rather than to the conventional security notion of a backdoor or to the naturalness of the trigger.
We also agree that junking can be formulated as an adversarial optimization problem over a larger search space and that its relationship with existing jailbreak optimization methods is therefore closer than the current wording may suggest. We do not intend to claim that the underlying optimization problem is fundamentally different from those studied in jailbreak research. Instead, the contribution is to study what happens when this optimization problem is approached with an intentionally information-poor procedure.
Regarding the term "hardness," we agree that our experiments do not establish computational hardness in the formal theoretical sense. Our use of the term refers to the empirical difficulty of recovering a successful sequence under the considered search procedure. We will therefore revise the title and/or terminology to avoid suggesting that we establish a formal complexity-theoretic hardness result. Our empirical observation is more modest: despite the high dimensionality and discreteness of the search space, successful sequences can be recovered reliably using a remarkably simple random-search procedure.