1) Limited generalizability and dependence on task-specific labeled data.
The discovered prompts generally transfer poorly across models and benchmarks, although the universal spurious prompts provide some within-model exceptions. This limits the extent to which the search identifies reusable prompting strategies. Applying the approach to a new model or task may require repeating prompt generation, evaluation, mutation, and selection. Although the resulting prompts are intended to be unrelated to the task, their construction requires knowledge of that task to define what must be excluded, together with labeled examples to identify effective candidates. The approach therefore depends on task-specific information and supervision that may not be available in practical settings.
This dependency also raises a concern about the claimed fairness of the comparison with task-agnostic prompting baselines, which receive no comparable adaptation. These baselines provide useful reference points, but the comparison does not isolate the benefit of spurious prompting from that of task-specific optimization. The task-aware PromptWizard comparison partly addresses this concern, although its search budget and access to labeled data should be clearly specified to establish comparable optimization conditions.
The paper does not sufficiently characterize the amount of supervision required. In particular, it is unclear how many training and validation examples are needed to discover effective prompts, how sensitive prompt selection is to the size and composition of these sets, and whether improvements persist when only a small labeled sample is available. Reporting the exact split sizes would clarify the experimental setting, while an analysis of performance under smaller search and validation sets would establish the approach’s data requirements. This matters because a prompt that works after extensive task-specific selection offers different practical value from one that can be identified with little supervision.
The computational requirements are also insufficiently characterized. Since discovery involves multiple rounds of LLM-based generation and target-model evaluation, reporting the total number of evaluations and wall-clock search time would help readers assess the cost of repeating the procedure for each new setting.
These limitations do not invalidate the observed prompt sensitivity, but leave unclear how practically and broadly the approach can be applied beyond the model–benchmark combinations used during optimization.
2) The presentation of the results needs to be revised for clarity.
The paper is generally well written in its opening sections, but the results are harder to follow. Numerous experiments and analyses are presented in succession without a sufficiently clear narrative explaining what readers should learn from each and how they collectively answer the research question. Since the empirical findings constitute the main contribution, this substantially affects how clearly their significance comes across.
The benchmarks are listed with citations but receive little explanation. Brief descriptions of their tasks, input and answer formats, and evaluated subsets would help readers understand the experimental coverage and interpret differences in performance. The organization and layout also interrupt the discussion: the full-page table on page 6 separates the surrounding narrative, making it harder to connect the numerical results with their interpretation.
More importantly, the discussion should distinguish the strongest findings from mixed outcomes more explicitly. Readers should be able to identify when spurious prompts improve performance, when they are competitive with existing methods, and when they fail to induce the intended behavior. The benchmark-specific, universal, and transfer experiments should then be connected to explain what the results establish about the scope and limitations of the phenomenon. Similarly, the paper should clarify whether the principal improvements are statistically reliable, rather than leaving readers to infer their robustness from the reported means and standard deviations.
A clearer synthesis at the end of the results section would help consolidate these observations into a small number of supported takeaways. At present, the breadth of the experiments is apparent, but the final message is less clear: what has been consistently demonstrated, under which conditions, and with what limitations? This is a fixable presentation issue, but an important one because it affects readers’ ability to assess the paper’s central contribution.
We would like to thank the reviewer for the positive assessment of our paper. We address the reviewer's concerns below:
We re-ran our unchanged search with only two changes: the generator and mutator are instructed to write task-relevant prompts (naming the domain, giving a solution strategy, checklist, common pitfalls or expert persona) and the task-irrelevance filter is disabled. Under identical optimization, task-relevant prompts are not systematically better than spurious ones:
In the revised paper, this control is described in Section 4.2 (paragraph "Task-relevant control"), with the results in Table 2.
Thank you. We agree that this is a very interesting mechanistic interpretability question. However, establishing mechanisms would be outside the scope of the paper and would necessitate another work. We also agree that linguistic content and embedding similarity characterize the prompts, not their effect on the model, so we add a first representation-level analysis. For the Llama-3.2-1B-Instruct GSM8K search we take the last-layer hidden state at the final prompt token (the state from which the model starts answering), averaged over 8 fixed questions, for 111 system prompts: 10 standard task prompts (zero-shot CoT, Plan-and-Solve, PromptWizard, etc.,), 5 random unrelated texts and all 96 spurious candidates of the search, and fit a 2-D PCA:
Spurious prompts occupy a region of their own in the model's representation space, separated from both task instructions and random text. Within that region, the better a spurious candidate performs, the closer its pre-answer state lies to the states induced by task instructions, although it never enters their cluster. In the sentence-transformer embedding space the better spurious candidates are, if anything, less similar to the task description than the worse ones (correlation −0.41), so this representational proximity is not accompanied by, and cannot be explained by, the prompt text drifting toward the task. These observations suggest that better-performing spurious prompts induce representations that are closer to those associated with conventional task instructions, despite remaining semantically unrelated to the task. This provides initial evidence of representational convergence, but does not establish whether the underlying computational mechanisms are shared.
In the revised paper, this analysis is in Appendix V (Table 18) and is summarized in Section 4.3 (paragraph "Are spurious prompts really spurious?").