acceptodds
Under review as a conference paper at ICLR 2027

MODEPrompt: Taxonomy-Guided Pareto Evolution for Prompt Optimization

Abstract

Large language models (LLMs) are highly sensitive to prompts specifying task instructions, making automatic prompt optimization (APO) important for reliable performance. Existing APO methods typically guide search using scalar validation scores, which can obscure performance imbalances across heterogeneous reasoning patterns and produce prompts with competitive overall accuracy but low worst-group accuracy. We propose MODEPrompt, a reflective multi-objective evolutionary framework that induces a task-specific taxonomy of mutually exclusive reasoning subgroups from development questions without using answer labels, and treats subgroup accuracies as distinct Pareto objectives. MODEPrompt introduces semantic differential evolution, which contrasts Pareto-front and non-front prompts to extract directional semantic patches that guide mutation and crossover. It further induces outcome-grounded meta-rules from trial acceptance outcomes and subgroup-level score changes, accumulating useful search experience across iterations. Extensive experiments across diverse benchmarks show that MODEPrompt achieves strong overall performance, improves worst-group accuracy, and reduces performance disparities across reasoning subgroups. Further analyses support the stability, semantic interpretability, and performance relevance of the induced taxonomy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
automatic prompt optimization, large language models, multi-objective optimization, evolutionary optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KxqbOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that automatic prompt optimization guided by a single validation score can hide weak reasoning subgroups. MODEPrompt has an LLM induce two mutually exclusive reasoning subgroups from unlabeled development questions and treats the two subgroup accuracies as Pareto objectives. A semantic differential evolution step contrasts front and non-front prompts to extract textual patches for mutation and crossover, parent-trial dominance decides replacement, and meta-rules are induced across iterations (§3.4). The deployed prompt is the front member with the highest development accuracy. Experiments use DeepSeek-V3.2 on six BBH and six MMLU tasks against nine baselines, with ablations, subgroup metrics, human evaluation, four backbones and cost analysis.

Critical strengths

S1. The problem is stated concretely and the taxonomy is examined from several angles. Size-matched random partitions give gaps about half as large (Table 8), five independent inductions are compared by ARI (Table 9), and a blinded human evaluation is reported (Table 4). Why this matters to my assessment: it shows the induced split is related to performance, which is the premise of the method.

S2. Unfavourable results are reported. Performance drops for K>2K > 2 (Table 12), the unrestricted ACE is stronger on AppWorld (Table 13), and §4.3 notes that w/o Patch also removes the meta-rule. Why this matters to my assessment: it makes the remaining claims easier to trust and to check.

Critical weaknesses

W1. The reported gains are not reliable enough to carry the conclusions.

  • The same quantity differs between tables. Table 1 and Table 3 both report the macro-averaged held-out accuracy over the same 12 tasks with DeepSeek-V3.2 and 2-shot prompting, yet CoT(FS) rises from 74.36 to 82.30 while MODEPrompt falls from 86.00 to 84.48. The lead over GEPA shrinks from 3.96 to 1.55, and over CoT(FS) from 11.64 to 2.18. The paper does not explain the difference.
  • No run counts or variances are reported for Tables 1, 2, 5 and 12, and the MODEPrompt row of Table 2 is identical to the BBH columns of Table 1, so the ablation differences have no independent replication. The only repeated runs (Appendix C.3) give 91.50±3.9791.50 \pm 3.97 on Boolean Expressions, against 96.00 in Table 1; this standard deviation exceeds most ablation differences in Table 2.
  • The baselines' optimization budgets are not given, and the few-shot examples used by each method are not specified, while Table 11 shows that changing only the examples moves Ruin Names from 51.50 to 85.00.

Why this matters to my assessment: the method's advantage over the strongest baselines is a few points, and the paper does not establish that it exceeds the run-to-run and protocol variation.

W2. Lack of novelty, and the parts that would be new are not isolated. The paper states that it "does not claim the first use of LLM-based multi-objective optimization" and that its distinction is the loop connecting objective construction to language editing (§2, l.137-142). Its own related work already covers differential evolution for prompts (EvoPrompt, l.103), Pareto-style selection (GEPA, l.107) and reflection guiding evolutionary updates (l.144-151). The contribution therefore rests on the loop, but:

  • The deployed prompt is the front member with the highest aggregate development accuracy (Eq. 11). This member cannot be dominated by any other population member, so the choice equals the argmax over the whole population, and the Pareto structure does not act on the delivered prompt.
  • The abstract says meta-rules are induced from "subgroup-level score changes", but the meta-rule template (Fig. 8) receives only prior reflections, new reflections and the replacement count; its caption calls the count the "grounding signal".
  • w/o Tax removes the subgroups and the two-objective comparison together, and w/o Pareto also changes the final selection rule (§4.3, l.370-377), so neither isolates the taxonomy or the Pareto search. No run uses a random or difficulty-based split inside the optimization loop, although the induction template requires the categories to "differ in reasoning difficulty or cognitive demand" (Fig. 4).

Why this matters to my assessment: if the delivered prompt is selected by aggregate accuracy and the subgroups may be a difficulty split, the gains cannot yet be attributed to what the paper claims is new.

Questions for authors
  1. What differs between the protocols of Table 1 and Table 3? Please give all methods under one protocol, with means and standard deviations over full end-to-end runs.
  2. What are the optimization budgets of the seven APO baselines, and which few-shot examples does each method use?
  3. What exactly is passed to the meta-rule operator? Please show one real record of Eq. (12) and the full string sent to the template in Fig. 8.
  4. Can you run the full loop with a size-matched random split and with a simple difficulty split (e.g., question length) as the two objectives, and a w/o Pareto variant that keeps Eq. (11) for final selection?

One more thing

Please add the crossover template, which Appendix E describes as complete but does not include.

Reviewer_jv4DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes MODEPrompt, a multi-objective evolutionary automated prompt optimization algorithm. Before optimization, an LLM splits the development questions into two categories without using the answer labels and the two category accuracies become the Pareto objectives. To mutate the prompt, the method contrasts a Pareto front prompt with a non-front one to get a short semantic patch. It also builds a meta-rule from which patches led to accepted replacements.

Experiment wise, on 6 BBH and 6 MMLU tasks with DeepSeek-v3.2, MODEPrompt outperforms APO baselines such as GEPA, ACE, and MARS. Most of that gain comes from BBH. It also slightly improves worst-group accuracy and ablations support each component.

Critical strengths
  • The motivation is sensible. Ranking prompts by overall dev accuracy can hide weak performance on particular question types. Building groups from unlabeled questions is a reasonable (and simple) way to expose that.
  • Comparisons are broad. MODEPrompt is compared against strong recent baselines such as GEPA, ACE, and MARS.
  • Human study. The paper includes a blinded human study, stability across repeated runs (ARI 0.73), a comparison with random partitions, and an ablation over the number of groups K. This is more thorough check on the taxonomy than previous papers, especially the human study.
Critical weaknesses
  • Novelty. While the paper is honest by saying that MODEPrompt "does not claim the first use of LLM-based multi-objective optimization," much of the algorithm is stitched from previous papers. The meta-rule is ReEvo's long-term reflection, which ReflectivePrompt already applied to prompts. Keeping a Pareto front and then picking the best overall prompt is GEPA's pattern at a coarser granularity.
  • Final selection doesn't target worst-group accuracy. The beginning frames the problem as "low worst-group accuracy." But selection uses aggregate dev accuracy.
  • Table 3 isn't referenced or talked about anywhere. Moreover, the performance discrepancy between this table and table 1 should be explained.
  • The benchmarked comparisons are too narrow scoped. Outside of GEPA, the other methods are not really multi-objective prompt optimization algorithms. APE, ProTeGi, OPRO, EvoPrompt, MARS and ACE all optimize one score.
  • Missing cited works. CAPO [1], Survival of the Safest [2], ReEvo [3] are just a few examples of core works not cited that revolve around evolutionary prompt optimization.

[1] https://proceedings.mlr.press/v293/zehle25a.html [2] https://aclanthology.org/2024.emnlp-industry.76/ [3] https://papers.nips.cc/paper_files/paper/2024/hash/4ced59d480e07d290b6f29fc8798f195-Abstract-Conference.html

Questions for authors
  • How are few-shot demos chosen? Couldn't the few shot demo selection be a con founding factor that should be accounted for?
  • Why not use PUPA benchmark (which is also used by GEPA)? it scores both response quality and privacy preservation which are two objectives that inherently have a tradeoff.
  • How is " MODEPrompt’s token consumption is of the same order of magnitude as MARS"? The token cost for MARS is less than half of MODEPrompt according to table 6.
Reviewer_M1kZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Existing automatic prompt optimization methods primarily rely on aggregate accuracy, which may overlook performance disparities across different reasoning subgroups. To address this issue, the paper proposes MODEPrompt which automatically partitions tasks into reasoning subgroups, uses Pareto-guided semantic differential evolution to refine prompts, and incorporates outcome-based reflection to guide subsequent optimization. The goal is to improve overall accuracy while reducing subgroup performance disparities.

Critical strengths
  1. The paper identifies a meaningful limitation of existing prompt optimization methods: aggregate accuracy can mask poor performance on specific reasoning subgroups. The motivation is clear and well justified.

  2. The proposed framework effectively integrates taxonomy-guided objective decomposition, Pareto-based semantic evolution, and outcome-grounded reflection. The overall design is coherent and provides a structured approach to prompt optimization.

  3. The experiments cover multiple reasoning benchmarks and LLM backbones. MODEPrompt consistently outperforms several strong baselines, and the ablation studies generally support the effectiveness of its main components.

Critical weaknesses
  1. The binary taxonomy may oversimplify task heterogeneity. Moreover, low ARI scores on Formal Fallacies (0.478) and Geometric Shapes (0.587) raise concerns about classification stability.

  2. Strict Pareto-dominance replacement may reject beneficial candidates, e.g. a change from (70, 90) to (80, 85) improves both average accuracy (for balanced groups) and worst-group accuracy but would still be rejected, potentially limiting exploration.

  3. Final prompt selection relies on aggregate accuracy despite the emphasis on worst-group performance, potentially undermining the multi-objective optimization goal.

  4. MODEPrompt consumes 2.37×more tokens than MARS per method-specific optimization cycle. Comparisons under equal token or LLM-call budgets are needed to establish computational efficiency.

  5. Removing semantic patches also disables meta-rule induction, making their individual contributions unclear. More independent ablations are needed.

Questions for authors
  1. Given the relatively low taxonomy stability on certain tasks, how sensitive is the optimization performance to taxonomy quality?

  2. Why is strict Pareto dominance required for parent–trial replacement? Have the authors considered retaining non-dominated candidates that improve overall and worst-group accuracy?

  3. Why is aggregate accuracy used for final prompt selection rather than a criterion that explicitly considers worst-group performance?

  4. Can the authors provide performance comparisons under matched token budgets or LLM-call limits?

  5. Can the contributions of semantic patches and meta-rule induction be evaluated independently?

Comments on the paper

1 comment

Dear reviewers,

The author-reviewer discussion period is approaching the deadline. Please read through the author's rebuttals and update your reviews. Thank you for your time and work.

Best Regards, AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.