acceptodds
Under review as a conference paper at ICLR 2027

Spurious Prompts: Can Irrelevant Prompts Steer Large Language Models?

Abstract

Large language models are highly sensitive to prompts, but this sensitivity is usually studied through task-relevant instructions, demonstrations, or reasoning cues. In this paper, we study a different form of prompt sensitivity: whether prompts that are semantically unrelated to the task can nevertheless steer model behavior. We call them spurious prompts and show their surprising efficacy. We also propose a simple black-box search procedure for discovering them. Across reasoning and question-answering benchmarks, using models ranging from 0.8B to 27B parameters and spanning three model families, we show that spurious prompts can improve performance, often matching or outperforming standard prompting baselines and task-aware prompt optimization. We further show that they can steer models toward unintended behaviors, such as repeatedly selecting the first answer option, producing incorrect answers, returning an even, prime, or small number without explicitly instructing the model to do so. These findings reveal a new kind of prompt sensitivity: LLMs can be systematically steered by prompts that are unrelated to the task they are asked to solve. We will make our code available upon acceptance

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Prompt Engineering, Automatic Prompt Generation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_Y72SOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether prompts with task-unrelated surface content can systematically steer the behavior of frozen LLMs. Through black-box prompt search and evaluations across seven benchmarks and multiple model families, it finds that such prompts can improve accuracy and induce targeted output biases, although their effectiveness and transferability vary across models and tasks.

Critical strengths
  1. The observations are interesting. The paper reports an interesting finding that prompts with task-unrelated surface content can improve performance and induce targeted behaviors. These observations offer a perspective on the relationship between prompt semantics and their effects on LLM behavior.

  2. The empirical evaluation is comprehensive. The experiments cover seven benchmarks and several model families, examining both accuracy and behavioral steering. Transferability tests and ablations on generator size, semantic coherence, and prompt components further broaden the empirical evaluation.

Critical weaknesses
  1. Task relevance should be isolated through controlled prompt optimization. The method generates prompts with an LLM, enforces task-irrelevance constraints, and selects candidates according to target-model performance. Their effectiveness may therefore partly reflect the benefits of LLM-driven prompt optimization. Although PromptWizard provides a task-aware baseline, its different optimization procedure does not isolate these factors. A task-relevant control using the same generator, search pipeline, and budget would clarify the contribution of spurious framing.

  2. Internal model effects require further discussion. The paper characterizes spuriousness primarily through linguistic content and semantic similarity to task descriptions. These comparisons do not directly characterize how the prompts influence the target model’s internal computation. A more detailed discussion of these internal effects would clarify whether spurious prompts induce distinct computational behavior or operate through mechanisms shared with conventional task instructions.

Questions for authors

Please find the weakness above.

2 comments
AuthorsOpenReview ↗

We would like to thank the reviewer for the positive assessment of our paper. We address the reviewer's concerns below:

W1. Task-relevant control with the same generator, pipeline and budget

We re-ran our unchanged search with only two changes: the generator and mutator are instructed to write task-relevant prompts (naming the domain, giving a solution strategy, checklist, common pitfalls or expert persona) and the task-irrelevance filter is disabled. Under identical optimization, task-relevant prompts are not systematically better than spurious ones:

BenchModelSpurious (Table 1)Task-relevant control
GSM8KLlama-3.2-1B38.7 ± 0.342.5
MATH500Llama-3.2-1B20.7 ± 0.617.0
MedQALlama-3.2-1B40.4 ± 0.539.9
MuSRLlama-3.2-1B60.7 ± 1.256.0
GSM8KOLMo-3-7B89.7 ± 0.490.8
MATH500OLMo-3-7B38.7 ± 2.354.0
MedQAOLMo-3-7B49.0 ± 0.248.8
MuSROLMo-3-7B66.0 ± 3.562.0

In the revised paper, this control is described in Section 4.2 (paragraph "Task-relevant control"), with the results in Table 2.

W2. Internal model effects

Thank you. We agree that this is a very interesting mechanistic interpretability question. However, establishing mechanisms would be outside the scope of the paper and would necessitate another work. We also agree that linguistic content and embedding similarity characterize the prompts, not their effect on the model, so we add a first representation-level analysis. For the Llama-3.2-1B-Instruct GSM8K search we take the last-layer hidden state at the final prompt token (the state from which the model starts answering), averaged over 8 fixed questions, for 111 system prompts: 10 standard task prompts (zero-shot CoT, Plan-and-Solve, PromptWizard, etc.,), 5 random unrelated texts and all 96 spurious candidates of the search, and fit a 2-D PCA:

ObservationValue
Centroid distance: spurious vs. task prompts62.8
Centroid distance: spurious vs. random text67.4
Centroid distance: task prompts vs. random text76.3
Correlation of candidate accuracy with distance to the task-prompt centroid−0.87
Correlation of candidate accuracy with PC1+0.91
Correlation of candidate accuracy with cosine similarity to the task description−0.41

Spurious prompts occupy a region of their own in the model's representation space, separated from both task instructions and random text. Within that region, the better a spurious candidate performs, the closer its pre-answer state lies to the states induced by task instructions, although it never enters their cluster. In the sentence-transformer embedding space the better spurious candidates are, if anything, less similar to the task description than the worse ones (correlation −0.41), so this representational proximity is not accompanied by, and cannot be explained by, the prompt text drifting toward the task. These observations suggest that better-performing spurious prompts induce representations that are closer to those associated with conventional task instructions, despite remaining semantically unrelated to the task. This provides initial evidence of representational convergence, but does not establish whether the underlying computational mechanisms are shared.

In the revised paper, this analysis is in Appendix V (Table 18) and is summarized in Section 4.3 (paragraph "Are spurious prompts really spurious?").

Thanks for the explaination. I will maintain the score.

Reviewer_mgJLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper investigates whether system prompts unrelated to a downstream task (called "spurious prompts" by the authors) can systematically steer LLMs toward different behaviors, ranging from improved or reduced task performance to specific answer patterns, such as selecting a particular option label. To create these prompts, the authors designed a multi-stage LLM-based pipeline. They conducted experiments on seven benchmarks, including multiple-choice QA and mathematical reasoning tasks. Their findings reveal several behavioral patterns, although the discovered prompts generally transfer poorly across models and benchmarks.

Critical strengths

1) An interesting research question with implications for evaluation robustness.

The paper investigates whether apparently unrelated system prompts can produce behavioral changes, considering both accuracy improvements and undesirable steering. This is useful for understanding how prompt construction can affect conclusions about model capabilities and their reliability.

Critical weaknesses

1) Limited generalizability and dependence on task-specific labeled data.

The discovered prompts generally transfer poorly across models and benchmarks, although the universal spurious prompts provide some within-model exceptions. This limits the extent to which the search identifies reusable prompting strategies. Applying the approach to a new model or task may require repeating prompt generation, evaluation, mutation, and selection. Although the resulting prompts are intended to be unrelated to the task, their construction requires knowledge of that task to define what must be excluded, together with labeled examples to identify effective candidates. The approach therefore depends on task-specific information and supervision that may not be available in practical settings.

This dependency also raises a concern about the claimed fairness of the comparison with task-agnostic prompting baselines, which receive no comparable adaptation. These baselines provide useful reference points, but the comparison does not isolate the benefit of spurious prompting from that of task-specific optimization. The task-aware PromptWizard comparison partly addresses this concern, although its search budget and access to labeled data should be clearly specified to establish comparable optimization conditions.

The paper does not sufficiently characterize the amount of supervision required. In particular, it is unclear how many training and validation examples are needed to discover effective prompts, how sensitive prompt selection is to the size and composition of these sets, and whether improvements persist when only a small labeled sample is available. Reporting the exact split sizes would clarify the experimental setting, while an analysis of performance under smaller search and validation sets would establish the approach’s data requirements. This matters because a prompt that works after extensive task-specific selection offers different practical value from one that can be identified with little supervision.

The computational requirements are also insufficiently characterized. Since discovery involves multiple rounds of LLM-based generation and target-model evaluation, reporting the total number of evaluations and wall-clock search time would help readers assess the cost of repeating the procedure for each new setting.

These limitations do not invalidate the observed prompt sensitivity, but leave unclear how practically and broadly the approach can be applied beyond the model–benchmark combinations used during optimization.

2) The presentation of the results needs to be revised for clarity.

The paper is generally well written in its opening sections, but the results are harder to follow. Numerous experiments and analyses are presented in succession without a sufficiently clear narrative explaining what readers should learn from each and how they collectively answer the research question. Since the empirical findings constitute the main contribution, this substantially affects how clearly their significance comes across.

The benchmarks are listed with citations but receive little explanation. Brief descriptions of their tasks, input and answer formats, and evaluated subsets would help readers understand the experimental coverage and interpret differences in performance. The organization and layout also interrupt the discussion: the full-page table on page 6 separates the surrounding narrative, making it harder to connect the numerical results with their interpretation.

More importantly, the discussion should distinguish the strongest findings from mixed outcomes more explicitly. Readers should be able to identify when spurious prompts improve performance, when they are competitive with existing methods, and when they fail to induce the intended behavior. The benchmark-specific, universal, and transfer experiments should then be connected to explain what the results establish about the scope and limitations of the phenomenon. Similarly, the paper should clarify whether the principal improvements are statistically reliable, rather than leaving readers to infer their robustness from the reported means and standard deviations.

A clearer synthesis at the end of the results section would help consolidate these observations into a small number of supported takeaways. At present, the breadth of the experiments is apparent, but the final message is less clear: what has been consistently demonstrated, under which conditions, and with what limitations? This is a fixable presentation issue, but an important one because it affects readers’ ability to assess the paper’s central contribution.

Questions for authors

Clarification questions and suggestions

  1. Which specific template does the "manually specified seed template" at line 178 refer to? Furthermore, I'd recommend adding a brief overview of the discovery procedure in the introduction, around lines 67–68, to help readers understand how these prompts are obtained.

  2. Regarding the replay buffer, could you explain more intuitively why examples from previous rounds are retained and how they influence candidate selection? Specifically, what retention fraction is used, and how was it chosen? Please clarify these aspects in the paper. I would also suggest using distinct notation for the replay buffer and the number of mutation rounds, as both are currently denoted by "R", which can cause confusion.

  3. Regarding prompt mutation, what is it intended to achieve beyond generating additional candidates? Does it preserve particular properties of successful prompts while changing others? I would also suggest annotating the differences between Mutation 1 and Mutation 2 in Figure 2 to make the process easier to understand.

  4. The paper mentions manual inspection alongside the embedding analysis to assess whether the discovered prompts are genuinely spurious. What criteria were used, how many prompts were inspected, and what were the outcomes?

  5. Consider discussing two related works on prompt sensitivity and behavioral reliability in medical multiple-choice QA. ReMedQA (ReMedQA: Are We Done With Medical Multiple-Choice Benchmarks?, EACL 2026) investigates whether perturbations to answer options in the prompt change model predictions, assessing the robustness of medical QA performance. MedQAbstain (LLMs (Almost) Never Abstain Under Medical Uncertainty, ACL 2026) examines whether adversarial attacks and prompt changes affect models’ answer choices and willingness to abstain under medical uncertainty in scenarios where an incorrect answer could have life-threatening consequences for the patient described in the input. These works could inform the prompt-sensitivity and adversarial-prompting discussions, respectively. As a future direction, evaluating spurious prompting in these settings could reveal whether it improves robustness and appropriate abstention or instead induces potentially dangerous answer changes. This would extend the investigation to high-stakes domains where the consequences of prompt sensitivity are particularly important.

  6. The paper states that its description is self-contained and supports reproducibility, but the code will only be released upon acceptance. Why not provide an anonymized implementation during peer review to help reviewers verify the search procedure and evaluation protocol?

Additional presentation suggestions

  • Consider moving Figure 2 to the following page and making the Webson and Pavlick citation in Table 9 link to its bibliography entry.

  • The discussion around lines 54–57 could briefly consider whether the expectation that useful instructions should be task-relevant partly reflects intuitions drawn from human communication. This would be a possible interpretation of the prevailing assumption.

5 comments
AuthorsOpenReview ↗

We would like to thank the reviewer for their constructive feedback. We address reviewer's concerns below:

W1: Limited transferability and task-specific supervision

Poor cross-model transfer is itself an important empirical finding, highlighting the model-dependent nature of spurious prompting. Although limited transferability reduces the practical reusability of individual prompts, it does not diminish the observed sensitivity of target models to semantically unrelated instructions, and within a model one universal prompt is competitive on all seven benchmarks and is the best method overall for Qwen3.5-27B (Table 1). Our spurious prompt search is universal: the task enters the search only as something to avoid (a benchmark name and a short forbidden-word list), far less than task-aware optimizers such as PromptWizard receive, and the labelled examples are used only to rank candidates, as in every prompt optimizer we compare against. Re-running the search for a new model or task needs only black-box access, 0.3–7.3 GPU-hours (see W4) and a few dozen labelled validation examples for the final selection (see W3), which from the robustness perspective that motivates the paper is precisely the concern.

In the revised paper, this is addressed at the end of the paragraph "Transferability of spurious prompts" in Section 4.3 and in the paragraph on universal prompts in Section 4.2. The search cost is in Table 15 (Appendix T), and the small-validation results are in Table 3 (Section 4.2).

W2: Fairness of the comparison and the PromptWizard setup

Thank you for this comment. PromptWizard was run under the same optimization budget and on exactly the same training, validation and test splits as the spurious-prompt search, so Table 1 already contains a task-aware optimizer under comparable conditions; we will state this explicitly in the appendix.

To directly isolate the effect of task content from that of optimization, we additionally re-ran our unchanged search with the generator instructed to write task-relevant prompts and the irrelevance filter disabled. Under identical optimization, task-relevant prompts are not systematically better than spurious ones, which supports our claim that task-relevant content is not necessary for the gains of prompt optimization.

BenchModelSpurious (Table 1)Task-relevant control (new)
GSM8KLlama-3.2-1B38.7 ± 0.342.5
MATH500Llama-3.2-1B20.7 ± 0.617.0
MedQALlama-3.2-1B40.4 ± 0.539.9
MuSRLlama-3.2-1B60.7 ± 1.256.0
GSM8KOLMo-3-7B89.7 ± 0.490.8
MATH500OLMo-3-7B38.7 ± 2.354.0
MedQAOLMo-3-7B49.0 ± 0.248.8
MuSROLMo-3-7B66.0 ± 3.562.0

In the revised paper, the PromptWizard setup is stated in Section 4.1 and Appendix B, and the task-relevant control is described in Section 4.2 (paragraph "Task-relevant control"), with the results in Table 2.

W3: Amount of supervision

Thank you, we agree that the split sizes should be stated explicitly and will add the table below to the appendix. Two benchmarks already run with small splits (MATH500: 320 search / 80 validation examples, MuSR: 160 / 40). For the selection step, we re-selected the final prompt among the top-5 candidates of the paper's search runs using only n randomly drawn validation questions (1,000 resamples). Our results on MATH500 and MuSR additionally provide evidence that effective prompts can be discovered with smaller training sets, but a systematic search-set scaling study remains future work.

BenchmarkSearch (train)ValidationTest
GSM8K5,9761,4971,319
MATH50032080100
MedQA8,1402,0381,273
MuSR1604050
GPQA2847490
OpenBookQA4,957500500
MMLU-Pro800400800

Test accuracy (%) of the prompt selected with n validation questions:

BenchModeln = 25n = 50full validation (n)
GSM8KQwen3.5-0.8B51.651.953.2 ± 0.4 (1,497)
GSM8KLlama-3.2-1B37.537.638.7 ± 0.3 (1,497)
GSM8KOLMo-3-7B88.888.889.7 ± 0.4 (1,497)
MATH500Qwen3.5-0.8B19.520.124.7 ± 2.5 (80)
MATH500Llama-3.2-1B18.919.320.7 ± 0.6 (80)
MATH500OLMo-3-7B32.532.738.7 ± 2.3 (80)
MedQAQwen3.5-0.8B39.840.040.2 ± 0.2 (2,038)
MedQALlama-3.2-1B39.539.640.4 ± 0.5 (2,038)
MedQAOLMo-3-7B47.747.849.0 ± 0.2 (2,038)

In the revised paper, the split sizes are part of the benchmark table (Table 14 in Appendix T), and the small-validation re-selection is in Section 4.2 (paragraph "Amount of supervision"), with the results in Table 3.

W4: Computational requirements

Thank you, we will add the table below to the appendix. One search scores 96 prompts (24 initial + 3 mutation rounds of 24) on disjoint training chunks (with 10% replay) and the top‑5 on the validation split. Runtime depends mainly on dataset size; generator calls vary because prompts violating our format rules are discarded and regenerated.

W5: Lack of a clear narrative in the results section

Thank you, we agree. In the revised paper we keep the order of the experiments in Section 4, but the main result paragraphs now end with a one-sentence takeaway that states what the experiment establishes. The Conclusions (Section 5) now open with a synthesis that connects these takeaways (see W8).

W6: Insufficient benchmark descriptions and disruptive table placement

Thank you. Section 4.2 of the revised paper now describes every benchmark in a few words, for example MuSR as murder mysteries and MMLU-Pro as questions from 14 subject areas. A new benchmark table in Appendix T (Table 14) lists for every benchmark the task, the answer format, the evaluated subset and the number of questions in the search, validation and test splits, for example MuSR with the murder-mysteries subset, two answer options and 160 / 40 / 50 questions. The main results table (Table 1) is now placed directly after the paragraph that discusses it.

W7: Insufficient distinction between strong and mixed results and lack of statistical significance analysis

Thank you, we agree. Appendix U (Table 17) now reports conservative worst-case McNemar tests using the least favourable of three runs, with Holm correction over 28 pairs. Against the strongest of seven task-agnostic baselines, 26/28 comparisons are not significant; against PromptWizard, all 28 are not significant. We therefore describe spurious prompts as competitive, not superior. Significant gains over all seven baselines occur only for OLMo-3-7B on GSM8K (+12.63 points) and Qwen3.5-27B on MMLU-Pro (+18.37). OpenBookQA is a failure case (−8.27 and −4.80 points for Qwen3.5-0.8B and OLMo-3-7B, respectively; neither significant). Section 4.2 now distinguishes strong, comparable and failed steering: option-A exceeds direct instructions in 10/12 cases, incorrect-answer in 7/12 (by at most 5.78 points), and numeric only for Qwen3.5-0.8B and Llama-3.2-1B. Universal prompts are competitive and achieve the highest average accuracy on Qwen3.5-27B. Finally, prompts transfer between GSM8K and MATH500 within models but not across models (Section 4.3), requiring a new search for each model or task.

W8: Lack of clear synthesis of key findings and limitations

Thank you. We have revised the Conclusions (Section 5) to better summarize our main findings. Most importantly, spurious prompts generally achieve performance comparable to established prompting methods, including zero-shot CoT and the task-aware PromptWizard, despite containing no task-specific instructions. This is particularly surprising given that these methods explicitly encourage task solving, whereas our prompts are semantically unrelated to the task. Furthermore, spurious prompts significantly outperform the strongest task-agnostic baseline in two settings after Holm correction. We also clarify the limitations of the approach, including limited transferability across models and the need for labelled examples during search. The statistical analysis is provided in Appendix U (Table 17).

Q1: Seed template and overview of the discovery procedure

Thank you for pointing this out. The "manually specified seed template" is the generator instruction in Appendix G: a fixed instruction to the generator LLM that requests a batch of system prompts and states only the constraints (no task-relevant vocabulary or solution strategies, spurious framing, a final "Final answer:" line); it is not optimized against target-model performance. We revised the introduction accordingly.

In the revised paper, the overview of the discovery procedure is at the end of the second paragraph of the introduction (Section 1), and Section 3 (paragraph "Candidate Prompt Generation") points to the generator instruction, which is Appendix G of the revised paper.

Q2: Replay buffer

Each mutation round evaluates the new candidates on a fresh subset of the training data, so without replay the earlier subsets would no longer contribute any signal to the selection. The replay buffer keeps that signal by adding a random 10% of every earlier subset to the current evaluation set, so candidates are selected for working on both new and previously seen examples. We will state α = 0.1 in Section 3 and rename the buffer to B\mathcal{B} so that RR denotes only the number of mutation rounds.

In the revised paper, this is in Section 3. The paragraph "Replay Buffer" states α = 0.1, which was fixed beforehand and not tuned, the purpose of the replay and the renamed buffer B\mathcal{B}, and the paragraph "Dataset Split" states K=R+1K = R + 1.

Q3: Purpose of mutation

Mutation is the exploitation step of the search: the mutator receives the top-5 prompts and is instructed to keep what makes them work (direct answering, the spurious framing and its control structure) while varying persona, tone, structure, strictness or reasoning posture, mixing near and far variants and recombining elements of several seeds (Appendix H). It operates under the same constraints as generation (no task vocabulary, no competence cues, lexical validator).

In the revised paper, this is in Section 3 (paragraph "Prompt Mutation"), and the mutator instruction is Appendix H of the revised paper.

Q4: Manual inspection of spuriousness

Thank you. We manually inspected every final prompt reported in the paper (three per model–benchmark pair, one per run) with a single criterion: does the prompt name or describe the task or its domain? None did, which is consistent with the embedding analysis in Figure 4 (spurious prompts are as far from the task descriptions as random unrelated text). To make this assessment less subjective we add an LLM-judge test: gpt-oss-120b receives a spurious prompt (all candidates from several search runs, not only the selected ones) together with the list of our seven benchmarks and is asked which benchmark the prompt was produced for.

Prompts made forJudge identifies the benchmark
GSM8K40%
MedQA0%
GPQA6%
MMLU-Pro72%*
MATH5000%
MuSR0%

*MMLU-Pro is the judge's default guess for any prompt that asks for a letter answer.

In the revised paper, the manual inspection and the LLM-judge results are in Section 4.3 (paragraph "Are spurious prompts really spurious?"), together with the embedding analysis in Figure 4.

Q5 Related work on medical QA (ReMedQA, MedQAbstain)

We will cite ReMedQA in the prompt-sensitivity discussion and MedQAbstain in the adversarial-prompting discussion. We also ran both protocols on the MedQA test set (1,273 questions) for our three target models, comparing the paper's MedQA spurious prompt with zero-shot CoT:

(i) ReMedQA-style option perturbation, i.e. four cyclic shifts of the answer options, reporting accuracy over shifts and the share of questions on which the same option text is chosen under all shifts,

(ii) MedQAbstain-style abstention, adding an explicit "I cannot determine the answer" option either with the gold option present (abstaining = over-abstention) or with the gold option removed (abstaining = appropriate).

ModelPromptAccuracy over 4 option shifts (min..max)Consistent across shiftsOver-abstentionAppropriate abstention
Qwen3.5-0.8BZero-shot CoT39.4 (37.5..41.3)18.99.914.1
Qwen3.5-0.8BSpurious39.9 (39.6..40.5)21.913.517.5
Llama-3.2-1BZero-shot CoT40.2 (39.0..41.2)18.87.516.3
Llama-3.2-1BSpurious38.6 (36.8..40.2)27.20.30.8
OLMo-3-7BZero-shot CoT37.2 (36.1..38.1)20.117.821.4
OLMo-3-7BSpurious48.1 (46.3..49.4)37.115.229.1

In the revised paper, ReMedQA is cited in Section 2 (paragraph "Prompt sensitivity"), MedQAbstain in Section 2 (paragraph "Adversarial prompts"), and the experiment is in Section 4.3 (paragraph "Option perturbation and abstention on MedQA"), with the table as Table 16 in Appendix T. Both protocols are also listed as rows of the benchmark table (Table 14 in Appendix T).

Q6 Code availability

We have added an anonymized implementation as a .zip in the supplementary material of this submission; it contains the search procedure (generator, validator, mutation and selection), the evaluation protocol and the configurations used for the reported runs.

In the revised paper, this is stated in the reproducibility statement.

S1 Consider moving Figure 2 to the following page and making the Webson and Pavlick citation in Table 9 link to its bibliography entry.

Thank you, both are done. Figure 2 now appears on page 4, next to the description of the search in Section 3, and Webson and Pavlick (2022) in the comparison table of Appendix E (Table 9) is now a linked citation.

S2 The discussion around lines 54–57 could briefly consider whether the expectation that useful instructions should be task-relevant partly reflects intuitions drawn from human communication. This would be a possible interpretation of the prevailing assumption.

Thank you, we agree and added this interpretation at the end of the first paragraph of the introduction (Section 1). The expectation that useful instructions must be task-relevant mirrors human communication, where a listener assumes that an instruction is relevant to the task (Grice, 1975). Our results show that LLMs do not always follow it, since text that a human would ignore can still change their answers.

Grice, H. Paul. "Logic and Conversation." In Syntax and Semantics 3: Speech Acts, pp. 41–58. Academic Press, 1975.

Reviewer_kKkPOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper introduces the notion of “spurious prompts”, which are fluent prompts that are not semantically related to LLM benchmark tasks, but can influence LLM performance. They use an evolutionary learning procedure that identifies spurious prompts for a given task and show that it can be used for steering various LLM behaviors such as question-answering, mathematical reasoning, constraining answers, or even refusal. They test various aspects of their framework to determine what makes spurious prompts effective across several LLM families and sizes.

Critical strengths
  • The paper presented a method that is simple and easy to understand, and the evaluation is solid. They test across several benchmarks, model families, and model sizes, and the results are consistent across both. They acknowledge limitations and appropriately frame their contribution as a demonstration of spurious prompting, and do not overclaim.
Critical weaknesses

The main weakness is related to the depth of the contribution. While the specific form of spurious prompts introduced in this paper may be new, the idea that LLMs can be steered by prompts unrelated to the task they are solving is not, which limits the extent of contributions of this paper. Specifically, several prior works show that unrelated (often gibberish) prompts can jailbreak models, and while this paper notes that most focus on using these prompt optimization techniques for studying jailbreaking, it could also just as well be adopted/applied for steering behavior on a benchmark. And while Section 4.3 (and appendix O) attempts to address this via the gibberish analysis, it wasn't a sufficiently deep experiment. For example, this experiment could be strengthened by adapting adversarial prompt optimization methods (such as GCG (Zou et al.) - I don't think this was cited) to compare to as a baseline for steering benchmark tasks.

A second weakness is that the discussion of related work could be strengthened. A lot of past research has studied techniques for prompt optimization, and while this is not the specific focus of your work, several threads seem related but are not cited and might provide more context for your specific contribution. For example, Guo et al. also propose an evolutionary approach to prompt generation, with legible prompts that can steer behavior. Your evolutionary approach is also fairly structured, and may benefit from discussing other structured prompt optimization methods more (eg. Khattab et al. + others), as well as persona-based prompting (e.g., Shanahan et al.), as many of the prompts you show in the appendix resemble this kind of prompting, and is explicitly listed in your system prompts.


Zou et al. Universal and Transferable Adversarial Attacks on Aligned Language Models.

Guo et al. Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers.

Khattab et al. DSPy: Compiling Declarative Language Model Calls into State-of-the-Art Pipelines.

Shanahan et al. Role-Play with Large Language Models.

Questions for authors

Upon reading the examples of spurious prompts you provide in the appendix, they appear to be converging to a similar template/style. And while it is still readable English, it's still cryptic language that's not particularly meaningful. Is there a reason we should prefer this kind of prompting to just gibberish prompting that comes from other prompt optimization methods? Or the point is just that there is another "axis" of attack which may appear to be fluent language.

4 comments
AuthorsOpenReview ↗

We thank the reviewer for the careful reading and the positive assessment of our paper. We address the concerns below.

W1: Comparison against GCG

We agree that jailbreaking with unrelated or gibberish text is well established, e.g. GCG [1], and will add this line of work to the related work. Our work, however, concerns spurious prompts: text that reads as ordinary language yet is completely unrelated to the task. This gives them natural advantages over GCG. For instance, under the evaluated perplexity-based filtering setup [2], all GCG strings are rejected while none of our spurious prompts are filtered out. The table below summarises the comparison.

Spurious promptsGCG [1]
Model accessBlack-box: only model outputs are needed.White-box: needs gradients w.r.t. the input at every step.
DetectabilityFluent English with median perplexity 38, below the zero-shot CoT prompts themselves (67.5).Median perplexity 3.4 × 10^5; a perplexity filter [2] set at the perplexity of our worst prompt rejects 100% of GCG strings.
Search cost96 candidate prompts scored from outputs (0.3–7.3 GPU-hours per model and benchmark).Up to 51,200 candidate strings scored on 8 questions each (0.4 M forward passes), plus a chain of thought generated at every step, per objective.

Following the suggestion, we ran GCG as a baseline on Qwen3.5-0.8B, Llama-3.2-1B and OLMo-3-7B with a compute budget at least as large as that of our search.

Accuracy (test, %). Bold: best per cell.

ModelPromptGSM8KMedQAGPQAMMLU-ProAvgRejected by PPL filter
OLMo-3-7BSpurious (ours)89.7 ± 0.449.0 ± 0.233.3 ± 5.646.3 ± 3.854.60%
OLMo-3-7BGCG80.2 ± 5.942.1 ± 1.125.6 ± 5.144.4 ± 0.348.1100%
Llama-3.2-1BSpurious (ours)38.7 ± 0.340.4 ± 0.531.9 ± 2.820.0 ± 0.532.70%
Llama-3.2-1BGCG44.5 ± 1.135.8 ± 1.623.3 ± 5.117.9 ± 3.330.4100%
Qwen3.5-0.8BSpurious (ours)53.2 ± 0.440.2 ± 0.232.2 ± 1.130.5 ± 2.339.00%
Qwen3.5-0.8BGCG52.3 ± 2.439.4 ± 0.527.8 ± 6.929.3 ± 0.937.2100%

Spurious prompts are better than GCG on average for all three models, and none of them is rejected by the perplexity filter, whereas every GCG string is.

Steering (test, %). Bold: best per row.

ObjectiveBenchmarkModelSpurious (ours)GCG
Option AGPQAQwen3.5-0.8B68.2 ± 3.764.8 ± 5.1
Option AOpenBookQAQwen3.5-0.8B82.8 ± 2.888.9 ± 1.2
Option AMMLU-ProQwen3.5-0.8B59.0 ± 16.057.4 ± 2.0
Option AGPQAOLMo-3-7B69.7 ± 8.219.6 ± 3.2
Option AOpenBookQAOLMo-3-7B73.7 ± 1.229.8 ± 5.4
Option AMMLU-ProOLMo-3-7B58.0 ± 9.913.8 ± 0.3
Option AGPQALlama-3.2-1B63.9 ± 0.881.9 ± 4.2
Option AOpenBookQALlama-3.2-1B81.5 ± 1.080.5 ± 7.2
Option AMMLU-ProLlama-3.2-1B70.6 ± 10.178.3 ± 8.4
IncorrectGPQAQwen3.5-0.8B81.5 ± 1.371.9 ± 4.6
IncorrectOpenBookQAQwen3.5-0.8B63.9 ± 2.267.5 ± 11.1
IncorrectMMLU-ProQwen3.5-0.8B78.2 ± 2.681.9 ± 7.1
IncorrectGPQAOLMo-3-7B79.3 ± 1.267.4 ± 7.9
IncorrectOpenBookQAOLMo-3-7B46.4 ± 3.519.4 ± 2.3
IncorrectMMLU-ProOLMo-3-7B85.0 ± 2.551.6 ± 0.9
IncorrectGPQALlama-3.2-1B79.3 ± 2.671.5 ± 6.4
IncorrectOpenBookQALlama-3.2-1B54.9 ± 2.060.9 ± 2.9
IncorrectMMLU-ProLlama-3.2-1B85.7 ± 3.082.0 ± 1.1

Spurious prompts also steer better than GCG on average (12 of 18 cells, including all six on OLMo-3-7B).

In the revised paper, this comparison is in Section 4.3 (paragraph "Comparison with GCG" and Table 6), and the related work on GCG and perplexity filtering is in Section 2 (paragraph "Adversarial prompts").

W2: Discussion of related work

We thank the reviewer for this suggestion. We have added the papers mentioned by the reviewer [3–5] to the related work, together with recent work on automated prompt optimization [6–9], to strengthen the discussion of APE.

In the revised paper, these works are discussed in the new paragraph "Automated prompt optimization" at the start of Section 2.

Q1: Why prefer this kind of prompting to gibberish prompting?

We thank the reviewer for this question. The point is indeed that there is a second axis of attack, and we think it matters for two reasons.

  1. Realism. A fluent persona such as the "lighthouse keeper" can plausibly end up in a system prompt or a shared prompt library; a gibberish string cannot. The perplexity filters that catch gibberish [2] reject none of our prompts and all GCG strings (see W1).
  2. Effectiveness. In our gibberish ablation (Appendix S, Table 13), gibberish prompts match coherent ones on accuracy and incorrect-answer steering but are substantially weaker for option-A steering, and GCG is substantially less effective than spurious prompts on OLMo-3-7B (W1).

In the revised paper, both points are made in Section 4.3, in the paragraphs "Gibberish prompts" and "Comparison with GCG" (Table 6). The gibberish ablation can be find in Appendix S.

References

[1] Zou, Andy, et al. "Universal and transferable adversarial attacks on aligned language models." arXiv'2023.

[2] Alon, Gabriel, and Michael Kamfonas. "Detecting language model attacks with perplexity." arXiv'2023.

[3] Guo, Qingyan, et al. "Connecting large language models with evolutionary algorithms yields powerful prompt optimizers." ICLR'2024.

[4] Khattab, Omar, et al. "DSPy: Compiling declarative language model calls into self-improving pipelines." ICLR'2024.

[5] Shanahan, Murray, Kyle McDonell, and Laria Reynolds. "Role play with large language models." Nature 623.7987 (2023): 493–498.

[6] Wang, Xinyuan, et al. "PromptAgent: Strategic planning with language models enables expert-level prompt optimization." ICLR'2024.

[7] Shi, Wenhang, et al. "No loss, no gain: Gated refinement and adaptive compression for prompt optimization." NeurIPS'2025.

[8] Batorski, Paweł, and Paul Swoboda. "PIAST: Rapid prompting with in-context augmentation for scarce training data." ACL'2026.

[9] Agrawal, Lakshya A., et al. "GEPA: Reflective prompt evolution can outperform reinforcement learning." ICLR'2026.

If the authors have addressed your concerns or offered clarification, please join the discussion and let them know whether everything has been resolved or if you still have any questions.

I thank the authors for the thorough response, I appreciate the extra experiments related to GCG and the comparisons to other methods added to the text. I think my concerns about comparing to related work have been addressed. I have updated my score to reflect this.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.