acceptodds
Under review as a conference paper at ICLR 2027

On the Hardness of Junking LLMs

Abstract

Large language models (LLMs) are known to be vulnerable to jailbreak attacks, which typically rely on carefully designed prompts containing explicit semantic structure. These attacks generally operate by fixing an adversarial instruction and optimizing small adversarial components (e.g., suffixes or prefixes). In this setting, prompt structure is fundamental for performance, and recent results show that even simple random search can achieve strong performance when combined with sophisticated prompt design. More recently, it has been observed that harmful behaviors can be elicited even without the adversarial prompt, relying solely on optimized token sequences. This suggests the existence of natural backdoors, i.e., token sequences naturally emerged during LLMs training that trigger unsafe outputs without any meaningful instruction. However, despite these observations, this setting remains largely unexplored, and in particular the hardness of finding natural backdoors has not been assessed yet. In this work, we provide a first proof-of-concept study investigating the hardness of this task, which we refer to as the junking problem. We formalize it as the problem of finding token sequences that maximize the probability of generating a target prefix of harmful responses, propose a simple greedy random-search procedure to assess whether such sequences can be discovered easily, and test it across several widely used 7B aligned models. Our results show that this problem is harder than standard jailbreak attacks, confirming the importance of semantic information in prompt design. At the same time, we find that our simple strategy is sufficient to solve the problem with a high success rate, suggesting that natural backdoors are present and easily recoverable, highlighting the need for improved alignment techniques to mitigate such vulnerabilities. Finally, through perplexity analysis, we observe that the discovered token sequences lie in low-probability regions of the model distribution, supporting the hypothesis that they emerged implicitly from the training process.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language models, adversarial machine learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pmB7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes the LLM junking problem. The task is to find a (junk) token sequence that maximizes the probability of generating a pre-defined harmful target prefix (without any anchored initialization, prompt template, or adversarial restrictions). To solve this problem, the paper proposes a simple Greedy Random Search (GRS) algorithm that updates a single token per iteration to minimize the negative log-likelihood.

Experiments on a subset of 50 harmful behaviors from AdvBench and across four safety-aligned LLMs (LLaMA-2-Chat-7B, Gemma-7B, Qwen-2.5-7B, Mistral-7B) demonstrate that with a sufficient number of function evaluations, a moderate ASR can be achieved. Perplexity analysis and illustrated successful examples suggest that the junk token sequences exhibit significantly higher complexity.

Critical strengths
  • The paper conducted a thorough overview of the related works on LLM jailbreaks and backdoor discovery (Section 1 and Section 3).

  • The problem of discovering non-semantic token sequences that elicit target harmful behaviors in aligned LLMs is different from the majority of prior literature on LLM jailbreaks.

Critical weaknesses
  • The LLM junking problem is poorly motivated. From my perspective, the proposed problem is just a relaxed variant of LLM jailbreaks without initializing at a given harmful prompt (paired with the target harmful response). For example, GCG or its later variants can be naturally adapted to optimize for the junk token sequence, for maximizing the generation probability of a harmful target prefix.

  • Given the fact that the junk token sequences (that successfully elicit the intended harmful response) have significantly high perplexity, they may be easily defended using a perplexity filter.

  • A few statements are misleading. The term "natural backdoor" is misleading, because these junk token sequences are far from natural. The paper argues that the junking problem differs from the widely studied LLM jailbreak attacks, but the proposed optimization objective and algorithm can be regarded as an adversarial optimization in a larger search space, which suggests the difference is, in fact, marginal. The paper uses "hardness" in the title, but provides empirical results only for a simple attack realization; this does not establish the hardness of a machine learning or optimization problem.

  • The paper seems to focus only on white-box attack scenarios. No attack transferability experiments are presented.

  • A relatively minor issue is the scalability of the experimental results, which focus only on 50 harmful target samples and 7B models. The high complexity of the proposed GRS algorithm can be a computational bottleneck for larger models.

Questions for authors

Can the authors elaborate on the significance of the proposed LLM junking problem and how it differs from the widely studied problem of LLM jailbreak attacks?

5 comments
AuthorsOpenReview ↗

Rebuttal (Part 1/2)

We thank the reviewer for the careful comments. We agree that some aspects of the motivation and terminology could be clarified. Our main point is that the goal of the paper is not to introduce a new practical jailbreak attack, but to use the junking problem as a controlled setting to study a more fundamental question: how difficult is it to find a token sequence that coerces an aligned LLM into generating a desired target, when the search procedure is given essentially no semantic information about how to achieve this? We clarify the points below.

1. Motivation and relation to jailbreaks.
We agree that the junking problem can be viewed as a relaxed version of the jailbreak setting, and that existing methods such as GCG could in principle be adapted to optimize for the same objective. However, this is not the goal of the work. Rather, the key distinction is in the purpose of the problem and the information available to the search procedure.

In a conventional jailbreak setting, the optimization procedure is typically designed to efficiently construct an adversarial prompt, often exploiting a harmful target, a carefully chosen initialization, gradients, or other available information about the model and the target behavior. In contrast, our objective is to investigate whether the coercion problem can be solved even when these sources of information are deliberately removed. We therefore use random search and a minimal single-token replacement procedure as experimental probes.

This distinction is important for our conclusions. If a sophisticated optimization method such as GCG finds a successful sequence, this demonstrates that the method can exploit information about the optimization landscape. In contrast, if a very simple random-search procedure can reliably find such sequences, this suggests that the underlying problem itself has a relatively large or easily accessible set of successful solutions. Our claim is therefore not that junking constitutes a fundamentally new class of jailbreak attacks, but that it provides a useful experimental setting for studying the intrinsic coerciveness of LLMs.

We will revise the paper to make this motivation explicit.

2. High perplexity and perplexity-based defenses.
We agree that the high perplexity of the junk sequences makes perplexity filtering a potentially relevant defense. However, this observation does not undermine the main result of the paper. Indeed, the fact that the successful sequences are highly unnatural is part of the reason why they provide a useful controlled setting for our study: we deliberately remove semantic and natural-language structure and ask whether the model can nevertheless be coerced into producing a desired target.

Our goal is not to argue that these particular sequences constitute robust practical attacks that evade existing defenses (i.e. the choice of adversarial setting is just due to the fact that aligned model should avoid generating "harmful" targets, thus coercing a model to generate them should be more challenging). A perplexity filter may indeed make this specific realization of the problem substantially less practical. Rather, our observation is that the model's generation behavior can be manipulated using sequences that contain essentially no semantic information and that can be discovered by a simple search procedure.

3. Terminology and the use of "hardness."
We agree with the reviewer that the term "natural backdoor" can be misleading, particularly because the sequences found by our procedure are intentionally unnatural. Our intended use of "natural backdoor" refers to the functional property of a token sequence that steers the model toward a specified target, rather than to the conventional security notion of a backdoor or to the naturalness of the trigger.

We also agree that junking can be formulated as an adversarial optimization problem over a larger search space and that its relationship with existing jailbreak optimization methods is therefore closer than the current wording may suggest. We do not intend to claim that the underlying optimization problem is fundamentally different from those studied in jailbreak research. Instead, the contribution is to study what happens when this optimization problem is approached with an intentionally information-poor procedure.

Regarding the term "hardness," we agree that our experiments do not establish computational hardness in the formal theoretical sense. Our use of the term refers to the empirical difficulty of recovering a successful sequence under the considered search procedure. We will therefore revise the title and/or terminology to avoid suggesting that we establish a formal complexity-theoretic hardness result. Our empirical observation is more modest: despite the high dimensionality and discreteness of the search space, successful sequences can be recovered reliably using a remarkably simple random-search procedure.

AuthorsOpenReview ↗

Rebuttal (Part 2/2)

4. White-box setting and transferability.
We agree that our current experiments are restricted to a white-box setting and that transferability is an important question for evaluating a practical attack. However, as emphasized above, our objective is not to propose a practical attack. The model is used as an experimental object through which we study coerciveness, and the ability to access model evaluations is part of the experimental setup.

5. Number of models, targets, and computational scalability.
We agree that the current experimental evaluation is limited to 50 target samples and 7B-scale models. Indeed, as we specify, the work is intended to be as a first proof-of-concept study. We extended the analysis to more models (see response to reviewer Nzhu).

We emphasize that the computational cost should be interpreted in the context of the purpose of the experiment. We are not proposing GRS as an efficient attack algorithm for deployment. Instead, we use it as a deliberately simple instrument for testing whether successful target sequences can be found at all without exploiting semantic information or sophisticated optimization techniques.

In this context, the fact that approximately 100k candidate evaluations are sufficient to recover successful sequences is itself informative. A generic discrete search problem over a 100-dimensional space could, in the worst case, require an exponentially growing number of evaluations. The fact that random search succeeds with a comparatively small budget suggests that the search problem encountered by the model is substantially more tractable than such a generic worst-case problem. This is the phenomenon we aim to investigate, rather than the computational efficiency of GRS itself.

(Question) On the significance of LLM junking and its relationship to jailbreaks.
We would like to clarify that we do not claim that LLM junking constitutes a fundamentally new class of adversarial attacks, nor that it is unrelated to the extensively studied jailbreak problem. The two problems are naturally related because both consider finding inputs that induce a model to generate behavior that is otherwise suppressed by alignment.

The distinction lies primarily in the scientific question we use the problem to investigate. In jailbreak research, the objective is typically to develop an effective procedure for bypassing the model's safeguards and eliciting a desired harmful response. In contrast, we use the harmful-target setting as a controlled testbed to study a different question: how easy is it to coerce an aligned LLM into generating a specified target when the search procedure is given essentially no information about how to achieve that target?

This is why we deliberately consider an extremely simple search procedure based on random search and single-token replacement. We do not aim to compete with sophisticated jailbreak algorithms in terms of attack efficiency. Instead, we ask whether sophisticated optimization, semantic information, carefully designed prompts, or even a meaningful initialization are necessary in the first place. The fact that successful sequences can be recovered using such a simple procedure is the central observation we want to highlight.

Thus, the significance of junking is not that it provides a practically superior jailbreak, but that it provides a controlled framework for probing the coerciveness of LLMs. If a model can be reliably coerced toward a specified target through an information-poor random-search procedure, this suggests that the corresponding set of successful inputs is sufficiently accessible to make the problem tractable despite the high-dimensional discrete search space. We believe this observation is complementary to existing jailbreak research: jailbreak methods primarily study how to efficiently find successful inputs, whereas our study asks how much information and optimization sophistication are actually necessary to find them at all.

We will revise the manuscript to make this distinction explicit.

We thank the reviewer again for their time in reviewing our manuscript and their valuable feedback. As the discussion phase is coming to an end, we would appreciate knowing whether our response has adequately addressed your concerns.

Reviewer_ztPvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether large language models can be induced to generate harmful outputs solely through meaningless token sequences, without containing explicit malicious instructions or semantic cues, and defines this phenomenon as Junking. The authors propose a greedy random search algorithm (GRS) based on single-token replacement, which optimizes the generation probability of target harmful response prefixes. Experiments are conducted on four 7B-scale language models.

Critical strengths
  1. The research question has some novelty, focusing on jailbreak attacks that do not rely on explicit malicious instructions or semantic cues.

  2. The effectiveness of a simple random search method is validated across multiple language models. The characteristics of non-semantic adversarial sequences are further analyzed in terms of attack success rate, target prefix matching, and perplexity.

Critical weaknesses
  1. There are some overclaims. For any neural network, a targeted search over input XX can identify an anomalous YY (referring to PPL in this paper). This only indicates that these inputs are uncommon under the model's distribution. The insights presented in this paper are not fundamentally different from adversarial prefix and suffix methods, as they still rely on meaningless token sequences. Moreover, Junking is not entirely independent of semantic information. It requires specifying a particular output target (harmful content) and then performing inverse optimization to find an input that can induce that output. During this search and optimization process, the method actually relies implicitly on the target prefix. If a different sequence needs to be optimized for each specific harmful target, it may not necessarily constitute a backdoor with a stable triggering effect in the conventional sense.

  2. The practical feasibility is limited, as the method introduces a substantial number of irrelevant tokens. There is no comparison with defense methods, and analyses such as transfer attacks are lacking.

  3. The paper does not provide experimental analysis of why single-token replacement is chosen over other replacement algorithms. Additionally, each attack request requires the LLM to perform forward passes on 100,000 candidates, resulting in a relatively high attack cost.

Questions for authors

See weaknesses.

2 comments

We thank the reviewer for the thoughtful comments. We agree that some aspects of the paper could be more clearly framed, and we will revise the manuscript to avoid overclaiming the practical implications of our findings. In particular, our goal is not to propose a practical attack or a conventional backdoor mechanism, but rather to provide a proof-of-concept study of LLM coerciveness: how easily an aligned model can be steered toward a desired target output.

1. On the interpretation of the results and the relation to existing methods.

We agree that targeted search can identify inputs with anomalous perplexity under a model. However, the main objective of our work is not to propose a novel adversarial attack, but to investigate how easily a model can be coerced into generating a desired target. We focus on adversarial targets because aligned models are expected to avoid generating them, making this a particularly challenging setting. Our results show that such target sequences can nevertheless be recovered through an extremely simple search procedure that exploits neither semantic information nor sophisticated optimization techniques. The combination of the perplexity analysis and the success of random search suggests that the problem is more tractable than one might expect given the dimensionality of the input space.

We also acknowledge that our terminology may cause confusion with conventional adversarial-prefix/suffix attacks and backdoors. The term natural backdoor is used in the specific sense defined in the paper: a sequence of tokens that steers the model toward a desired target prefix. We do not claim that it constitutes a conventional backdoor in the security sense.

Regarding Andriushchenko et al., their approach relies on sophisticated templates to construct the attack, whereas we investigate whether such mechanisms are necessary at all. Our results suggest that, once a target is specified, sophisticated templates and semantic information are not necessary to induce the desired behavior in our setting.

2. On practical feasibility, defenses, and transfer attacks.

We agree that the resulting sequences contain many irrelevant tokens, limiting their practical applicability as attacks. However, practical attack performance is not the objective of this work. As clarified in Section 2, junking problem and GRS should not be intended as novel or more efficient attacks. The goal infact is to study the coerciveness of aligned LLMs (i.e., the junking problem).

For the same reason, we do not include comparisons with defenses or transfer attacks. These evaluations would be relevant for a work proposing a practical attack, but fall outside the scope of our proof-of-concept study. We will clarify this scope.

3. On the choice of single-token replacement and computational cost.

We deliberately chose single-token replacement as a minimal baseline to determine whether the problem can be solved using an algorithm that exploits essentially no information beyond model evaluations. More sophisticated optimization strategies could improve the search, but would make it harder to assess the intrinsic tractability of the problem. The fact that such a simple procedure can reliably find successful sequences is therefore an important observation, rather than a claim that it is the optimal attack strategy.

We acknowledge that 100k model evaluations per request represent a substantial computational cost. However, these evaluations should not be interpreted as the cost of a proposed practical attack. Rather, our observation is that a solution can be recovered with only 100k evaluations using a random-search-like procedure, despite the fact that a generic search over a discrete space of this dimensionality could require exponentially many evaluations in the worst case. This suggests that the problem is considerably more tractable than a generic discrete optimization problem.

We will revise the manuscript to make the distinction clearer.

We thank the reviewer again for taking the time to review our work and provide valuable feedback. Since the discussion period is nearing its end, we would be grateful if you could let us know whether our rebuttal has sufficiently addressed your concerns.

Reviewer_NzhuOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to improve our understanding of how difficult it is to discover natural backdoors (i.e., input sequences that trigger targeted harmful responses). It investigates whether a simple greedy random search can optimize input sequences to elicit a targeted harmful response, without relying on semantic information or a more elaborate search mechanism used in existing attacks. The results show that even a basic search procedure can achieve substantial attack success, albeit with a significantly larger compute budget relative to standard (suffix-based) attacks. The paper also provides a perplexity analysis showing that the inputs generated by the attack lie in low-probability regions, yet can still elicit coherent, targeted harmful responses.

Critical strengths

The paper’s findings challenge a possible expectation that jailbreaking safety-tuned models requires a sophisticated attack. While the method itself has limited novelty, I find this an informative contribution because it helps challenge the natural assumption that basic search is unlikely to elicit harmful behavior from safety-aligned models. Moreover, the evaluation checks whether the model produces meaningful, harmful continuations beyond testing for matches with target phrases, thus convincingly evidencing that the reported attacks actually succeed.

Critical weaknesses

While I find that the results add useful information, they don’t establish a substantial advance over what prior work would lead us to expect. In many settings, simpler methods may still succeed given a sufficiently large budget, so this result is not surprising. Prior work also makes this result less surprising, as cited in the results section: Andriushchenko et al. use similar search to optimize adversarial suffixes. Their method and setting differ from this paper’s in the algorithm used and in retaining a semantic prompt while optimizing a suffix, but their success with simple search still makes the current results less surprising.

Although the paper extends its analysis to a distinct setting (specifically, the junking problem), I would have liked additional experiments to substantiate its contribution, including comparisons with methods such as GCG to compare compute budgets or an analysis of how removing semantic context affects search difficulty.

Additionally, the paper uses relatively old 7B models, which might have weaker safety alignment or fewer defenses than newer models. It is unclear whether these results would hold for newer models, limiting how broadly the findings can be generalized.

All in all, the main contributions do not seem substantial enough to justify acceptance to ICLR.

Questions for authors
  • How does the search compare with GCG in the same junking setting, in attack success and the compute required?

  • Have you tested whether these findings extend to newer models?

2 comments

We thank the reviewer for the useful feedback and comments. In the following we address their concerns.

Weakness 1

We thank the reviewer for the comment. We agree that, in several settings, simple methods can succeed given a sufficiently large budget. However, in our case, the situation is somewhat different. In fact, we are trying to solve a discrete optimization problem through a random-search-like procedure. In principle, random search may require a number of function evaluations that grows exponentially with the dimension of the input space. The fact that our procedure can reliably find solutions with a relatively small number of function evaluations, considering the input dimension (which is 100), suggests that the problem is relatively simple despite the lack of semantic information.

The result of Andriushchenko et al. is different from ours, as they use a sophisticated template to perform the attack, and their finding regarding the simplicity of the problem is related to the important role of semantic information. Our result, instead, suggests that such a template is not necessary, and that target token sequences can be recovered even by simply running a random-search procedure.

Weakness 2 - 3 (Additional experiments and Comparison with GCG). We extended our experiments to more modern models (see answer to Question 2) and compared GRS with GCG (see answer to Question 1)

Question 1

The reason we did not provide a comparison with other methods is that the main purpose of the work was to introduce the junking problem and a minimal baseline that can be used as a diagnostic tool for evaluating hardness and demonstrating the hardness of junking for known aligned LLMs. However, comparing GRS with other methods for the junking problem would allow us to address the question: "which method finds junk sequences as fast as possible?"

This is a different, and very interesting, problem, and it is exactly one of the research directions opened by our work, for which GRS is intended as the minimal baseline. Due to the limited time available for discussion, and since we work on a shared machine that is heavily loaded during this period, we were able to compare GRS with GCG on junking Llama 3.1 8B and Mistral 7B on the same dataset used in our work. Below, we report the normalized function value progress at the best substitution after several numbers of function evaluations (#FE).

As expected, we observe that GCG is faster than GRS, since it exploits gradient information. Importantly, this result further supports the claim that GRS is the right tool for modeling simplicity in this problem: the fact that GRS cannot match GCG's performance indicates that additional information is needed to find such effective token sequences. In this sense, GCG "masks" the hardness of finding junk sequences through its use of the gradient, hiding the true difficulty of finding such tokens in the absence of that information.

Mistral 7B

Algorithm#FE = 5k#FE = 10k#FE = 25k#FE = 50k#FE = 75k#FE = 100k
GRS0.5825±0.21630.5825 \pm 0.21630.4045±0.17840.4045 \pm 0.17840.2463±0.12290.2463 \pm 0.12290.1695±0.08900.1695 \pm 0.08900.1382±0.07670.1382 \pm 0.07670.1262±0.07070.1262 \pm 0.0707
GCG0.4317±0.24900.4317 \pm 0.24900.2536±0.16160.2536 \pm 0.16160.1381±0.10040.1381 \pm 0.10040.0953±0.07910.0953 \pm 0.07910.0768±0.06700.0768 \pm 0.06700.0686±0.06450.0686 \pm 0.0645

Llama 3.1 8B

Algorithm#FE = 5k#FE = 10k#FE = 25k#FE = 50k#FE = 75k#FE = 100k
GRS0.4967±0.09840.4967 \pm 0.09840.3928±0.09970.3928 \pm 0.09970.2612±0.10510.2612 \pm 0.10510.1783±0.09390.1783 \pm 0.09390.1379±0.08410.1379 \pm 0.08410.0823±0.02750.0823 \pm 0.0275
GCG0.4291±0.13290.4291 \pm 0.13290.2884±0.13680.2884 \pm 0.13680.1677±0.12050.1677 \pm 0.12050.1088±0.10910.1088 \pm 0.10910.0859±0.10230.0859 \pm 0.10230.0764±0.09870.0764 \pm 0.0987

Question 2

We re-performed the experiments on more modern models and we observed that similar observations are obtained i.e. it is possible to obtain with simple method and limited budget relatively high success rate in such a setting. Below we report the attack success score.

ModelASR
Llama3.1-8B0.5217
Mistral3-8B0.8518
Llama2-13B0.4513

Since these models are bigger we also considered a larger budget of function evaluations (5×1055 \times 10^5). Below we report the results with larger budget

ModelASR
Llama3.1-8B0.8043
Mistral3-8B0.9245
Llama2-13B0.7234

We would like to thank the reviewer again for their time and effort in reviewing our manuscript and for their useful feedback. As the discussion phase is coming to an end, we would appreciate knowing whether our response has adequately addressed your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.