- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SLAE, which divides instructions into semantic scaffold and slot-wise semantic actions, with one gene responsible for selecting the scaffold and other genes responsible for iteratively generating semantic actions, and then generates the final prompt text through a deterministic mapping. The paper also proposes two spatial expansion methods, DSE and A2R; the former adds new grammatical scaffolds, and the latter supplements new semantic actions when the search stagnates. Experiments are conducted on two types of tasks, NLU and NLG, and compared with other baselines, SLAE achieves the best results.
Critical strengths
The method in this paper is easy to understand, and judging from the experiments in the paper, it achieves good improvements.
Critical weaknesses
-
Experimentally, this paper only tests on an outdated small-scale model (Qwen2.5-7B) and outdated classification and summarization generation tasks, which completely fails to demonstrate the effectiveness of the proposed method and is entirely disconnected from the current development of LLMs.
-
The extensive gaps in the experiments make the effectiveness of the method unconvincing at present, as it lacks testing on advanced closed-source models (such as Claude-5.5 and GPT-6) and lacks evaluation on high-difficulty math, coding, and long-horizon Agent tasks.
Questions for authors
The SLAE method itself decomposes a prompt into two parts, a scaffolds and slots. Is such a structure suitable for the forms of tasks that people currently focus on, such as Agent tasks? I think the task of prompt evolution seems to have become outdated. At present, more attention is focused on harness evolution, and the task of harness evolution almost completely subsumes prompt evolution.
3 comments
I do not agree with the unqualified argument that models are updated every few months and that a paper’s contribution lies in the method itself. Although methodological contributions are important, LLM research should also be evaluated under settings representative of the field at the time the research is conducted. A method that works on Qwen2.5 may be entirely ineffective on Qwen4.
For example, a method might generate large quantities of synthetic mathematical training data and demonstrate substantial gains on an outdated model. However, if the generated queries are too simple to benefit more capable open-source models, and the method neither supports nor addresses the generation of more challenging problems, its practical relevance under current conditions remains unclear. Strong results on an older model alone would not establish the method’s continued value.
A paper’s contribution may indeed lie in its method, but this does not remove the need to establish its relevance to contemporary models and tasks. The method should demonstrate benefits under reasonably current and representative settings, rather than relying solely on outdated ones. I therefore maintain my original score.
Run unchanged on a later, larger model, SLAE keeps its generation-side saving on every task and its accuracy lead on most
Thank you for the follow-up. We accept the criterion as stated: the method "should demonstrate benefits under reasonably current and representative settings". The risk your comment raises, a method that works on Qwen2.5 but not on a later model, can be tested directly by replacing the model and changing nothing else. The comparison below does this with Qwen3.8-27B-FP8. With every setting unchanged, SLAE keeps its generation-side saving on every task and its accuracy lead on AG News and SST-5, and everything it adds to its search space is written by Qwen3.8-27B-FP8 itself.
Setting. Your example measures relevance by the benefit to "more capable open-source models". Relative to the paper's Qwen2.5-7B, Qwen3.8-27B-FP8 is such a model: it belongs to Qwen's latest publicly released open-weight generation and has nearly four times as many parameters. Every method uses it, in non-thinking mode, both to evaluate and to generate. The baselines are the DE variant of EvoPrompt, the paper's closest baseline, and OPRO; in both, the LLM writes every candidate instruction, whereas in SLAE it writes only the content that builds and expands the search space. The protocol is that of the paper's same-evaluator comparison (revised Section 4.2 and Appendix C.5), with the same three tasks and three seeds; the three methods share the initial prompt pool, the fitness subset, and the candidate-evaluation budget. SLAE keeps every setting of that comparison, and nothing is tuned on Qwen3.8-27B-FP8.
Result. SLAE uses about 3.7% of EvoPrompt's generation-side tokens and about 2.8% of OPRO's, and under 5% of either on every task. At this cost, it has the highest mean held-out accuracy of the three methods on AG News and SST-5, two of the three tasks; on both, it also varies least across seeds.
Replacing the model replaces the writer of everything SLAE adds to its search space. Your example describes a method whose generated queries stay "too simple" as the models they are meant to train become more capable, because the method cannot generate harder ones. In SLAE, as in the paper's setup (Section 4.1), the model the instructions are optimized for also builds and expands the search space. In these runs, Qwen3.8-27B-FP8 extracted the slot phrases from the pool prompts, filled the action sets, wrote new actions when the search stagnated (A²R), wrote new scaffolds (DSE), and scored every candidate; apart from the initial prompt pool, which all three methods share, no generated content in the search comes from an earlier model. The instructions SLAE returns draw on this content: none of the nine is a pool instruction, eight contain a phrase that Qwen3.8-27B-FP8 wrote, and four contain one it wrote during the search.
The case for SLAE therefore no longer rests on results from "an older model alone".
The evaluation model is held fixed across methods; the contributions are properties of the search procedure
Thank you for the review, and for noting that the method "is easy to understand" and "achieves good improvements" in the paper's experiments. The remaining points concern the scope of the evaluation, and we answer them in turn.
Two points on the summary. To clarify how slot genes work: they select among existing actions, and no LLM call is made per candidate; new actions are generated only when search stagnates. This is what reduces generation-side calls from O(Np⋅T) to O(T). The paper also makes a contribution at the level of the search operator: differential-evolution arithmetic on nominal genotype coordinates depends on how categories are indexed, and the label-equivariant landing rule removes this dependence (Proposition 1). This result does not depend on the evaluation model.
What the experiments establish. Within the setting the paper studies, controlled comparisons demonstrate the effectiveness of the method. Across nine tasks, the methods in each table are re-run under the same evaluator, task splits, and budget (Section 4.1). Section 4.5 then tests alternative explanations of the gains: it switches expansion off, changes the landing rule, and replaces the prompt pool with random initialization. Switching expansion off lowers accuracy on AG News by 8.73 points (Table 6).
Models used in the paper. The controlled study of Section 4.5 and the comparison of generation-side tokens with EvoPrompt in Section 4.2 also use Llama-3.1-8B-Instruct, on CoLA. There the expansion effect is +5.25: it replicates on a held-out task with a different model family.
Model scale and task set. The nine tasks are the nine datasets evaluated by EvoPrompt (ICLR 2024), chosen for a like-for-like comparison with the closest baseline (Section 4.1), whose classification results are themselves reported on a 7B open model (Alpaca-7b). Work of this kind continues to be published at this model scale: GEPA (ICLR 2026) uses Qwen3-8B as its open-weight model, and PromptQuine (ICML 2025) uses Meta-Llama-3-8B-Instruct for all methods in its classification comparison, with SST-2, Subj, and AG News among its datasets. Models are replaced every few months; the paper's contributions are properties of the method and do not expire with a model: its cost structure, equivariance, and the attributability of improvements to explicit search events.
Closed-source frontier models. The paper studies the feasibility of a method: instruction search in a structured genotype space, with no LLM call per candidate, that still improves on text-space search under the same evaluator. The evaluation model is a controlled variable shared by every method, not the object of study. The more expensive the model an optimizer calls, the more the reduction from O(Np⋅T) to O(T) generation-side calls saves.
Math, coding, and long-horizon agent tasks. These lie outside the scope of this paper, as its conclusion states.
The scaffold-and-slot structure and agent tasks. At the level of representation, the decomposition acts on instruction text and requires only that the text contain components with fixed roles that can be exchanged. The prompts in an agent harness are such text: a system prompt states a role, a task, tool-use rules, and an output format.
Harness evolution and prompt evolution. We agree that harness evolution subsumes prompt evolution, and recent harness papers say so themselves: HarnessEvolve (arXiv:2609.00829) describes the harness that self-evolving agents optimize as "prompts, skills, tools, and execution logic", and The Last Harness You'll Ever Build (arXiv:2604.21003) describes harness engineering as "designing the prompts, tools, orchestration logic, and evaluation criteria that make a foundation model effective". Prompts are thus among the things a harness optimizer rewrites with an LLM. Prompt evolution also remains current at this venue: GEPA (ICLR 2026) describes itself as "a reflective prompt optimizer for compound AI systems" that "evolves only the set of prompts", and its title is "Reflective Prompt Evolution Can Outperform Reinforcement Learning". Because the larger problem contains this step, it inherits the step's generation cost and the question of whether its search operator is sound. In a new comparison under the same evaluator, SLAE uses about 0.8% of GEPA's generation-side tokens, with comparable accuracy on AG News, TREC, and SST-5 (revised Section 4.2).