- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that automatic prompt optimization guided by a single validation score can hide weak reasoning subgroups. MODEPrompt has an LLM induce two mutually exclusive reasoning subgroups from unlabeled development questions and treats the two subgroup accuracies as Pareto objectives. A semantic differential evolution step contrasts front and non-front prompts to extract textual patches for mutation and crossover, parent-trial dominance decides replacement, and meta-rules are induced across iterations (§3.4). The deployed prompt is the front member with the highest development accuracy. Experiments use DeepSeek-V3.2 on six BBH and six MMLU tasks against nine baselines, with ablations, subgroup metrics, human evaluation, four backbones and cost analysis.
Critical strengths
S1. The problem is stated concretely and the taxonomy is examined from several angles. Size-matched random partitions give gaps about half as large (Table 8), five independent inductions are compared by ARI (Table 9), and a blinded human evaluation is reported (Table 4). Why this matters to my assessment: it shows the induced split is related to performance, which is the premise of the method.
S2. Unfavourable results are reported. Performance drops for (Table 12), the unrestricted ACE is stronger on AppWorld (Table 13), and §4.3 notes that w/o Patch also removes the meta-rule. Why this matters to my assessment: it makes the remaining claims easier to trust and to check.
Critical weaknesses
W1. The reported gains are not reliable enough to carry the conclusions.
- The same quantity differs between tables. Table 1 and Table 3 both report the macro-averaged held-out accuracy over the same 12 tasks with DeepSeek-V3.2 and 2-shot prompting, yet CoT(FS) rises from 74.36 to 82.30 while MODEPrompt falls from 86.00 to 84.48. The lead over GEPA shrinks from 3.96 to 1.55, and over CoT(FS) from 11.64 to 2.18. The paper does not explain the difference.
- No run counts or variances are reported for Tables 1, 2, 5 and 12, and the MODEPrompt row of Table 2 is identical to the BBH columns of Table 1, so the ablation differences have no independent replication. The only repeated runs (Appendix C.3) give on Boolean Expressions, against 96.00 in Table 1; this standard deviation exceeds most ablation differences in Table 2.
- The baselines' optimization budgets are not given, and the few-shot examples used by each method are not specified, while Table 11 shows that changing only the examples moves Ruin Names from 51.50 to 85.00.
Why this matters to my assessment: the method's advantage over the strongest baselines is a few points, and the paper does not establish that it exceeds the run-to-run and protocol variation.
W2. Lack of novelty, and the parts that would be new are not isolated. The paper states that it "does not claim the first use of LLM-based multi-objective optimization" and that its distinction is the loop connecting objective construction to language editing (§2, l.137-142). Its own related work already covers differential evolution for prompts (EvoPrompt, l.103), Pareto-style selection (GEPA, l.107) and reflection guiding evolutionary updates (l.144-151). The contribution therefore rests on the loop, but:
- The deployed prompt is the front member with the highest aggregate development accuracy (Eq. 11). This member cannot be dominated by any other population member, so the choice equals the argmax over the whole population, and the Pareto structure does not act on the delivered prompt.
- The abstract says meta-rules are induced from "subgroup-level score changes", but the meta-rule template (Fig. 8) receives only prior reflections, new reflections and the replacement count; its caption calls the count the "grounding signal".
- w/o Tax removes the subgroups and the two-objective comparison together, and w/o Pareto also changes the final selection rule (§4.3, l.370-377), so neither isolates the taxonomy or the Pareto search. No run uses a random or difficulty-based split inside the optimization loop, although the induction template requires the categories to "differ in reasoning difficulty or cognitive demand" (Fig. 4).
Why this matters to my assessment: if the delivered prompt is selected by aggregate accuracy and the subgroups may be a difficulty split, the gains cannot yet be attributed to what the paper claims is new.
Questions for authors
- What differs between the protocols of Table 1 and Table 3? Please give all methods under one protocol, with means and standard deviations over full end-to-end runs.
- What are the optimization budgets of the seven APO baselines, and which few-shot examples does each method use?
- What exactly is passed to the meta-rule operator? Please show one real record of Eq. (12) and the full string sent to the template in Fig. 8.
- Can you run the full loop with a size-matched random split and with a simple difficulty split (e.g., question length) as the two objectives, and a w/o Pareto variant that keeps Eq. (11) for final selection?
One more thing
Please add the crossover template, which Appendix E describes as complete but does not include.
Dear reviewers,
The author-reviewer discussion period is approaching the deadline. Please read through the author's rebuttals and update your reviews. Thank you for your time and work.
Best Regards, AC.