- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the problem of underspecified user prompts and proposes AGOPS, a framework for automatically learning task-specific prompt guidelines. Given an underspecified query, a reference answer, and a candidate guideline, a prompt-writer language model generates a more complete prompt. A solver model then answers the generated prompt, and its performance is used to evolve the guideline through GEPA. The framework uses character n-gram overlap constraints and rejection sampling to limit answer leakage. Experiments on mathematical reasoning, medical question answering, and coding benchmarks show substantial improvements over the selected baselines.
Critical strengths
-
The method is clear and well structured. AGOPS automatically learns task-specific guidelines that help users identify what information should be included in their prompts. It uses reference answers to simulate more complete prompts and optimizes the guidelines through solver feedback and GEPA. The framework is relatively simple, and the learned guidelines can support both prompt writing and interactive clarification.
-
The experimental results are strong across multiple benchmarks and models. AGOPS is evaluated on mathematical reasoning, medical question answering, and coding tasks using GPT-4.1-mini and Qwen3-32B. In the simulated setting, it consistently outperforms the selected baselines, with improvements of up to 81.7 percentage points over No Guideline. The experiments also show that the learned guidelines improve information seeking in multi-turn interactions (Figure 8). The additional model-scale experiments and user study provide further supporting evidence.
Critical weaknesses
-
The main prompt-engineering evaluation gives the prompt writer access to reference answers for test instances (Appendix B.3), making it unclear whether the improvements would hold in real-world settings without such information. Furthermore, the assumption that reference answers contain the missing knowledge of users (Section 3.2) is not directly tested. The leakage constraint cannot reliably detect rewritten solutions, and Appendix F.2 explicitly instructs the writer to describe the exact algorithm in the reference answer. The No Guideline comparison also raises concerns. Figure 9 allows the writer to use reference answers only when explicitly instructed, but the initial guideline (Figure 10) does not include such instructions, while the optimized guidelines directly ask the writer to extract information from the reference answers. This makes it difficult to determine how much improvement comes from learning better guidelines rather than simply using more information from the reference answers. The small user study (20 participants, only 8 in the AGOPS group) also provides limited evidence of real-world effectiveness. More realistic evaluations and better-controlled baselines are needed.
-
AGOPS combines existing ideas from instruction induction, guideline learning, and evolutionary prompt optimization. Related work, including "Guidelines to Prompt Large Language Models for Code Generation: An Empirical Characterization" (arXiv:2601.13118) and "Large Language Models Are Human-Level Prompt Engineers" (arXiv:2211.01910), explores similar ideas. However, the paper does not compare AGOPS with simpler methods that directly generate guidelines from reference examples without iterative optimization. Furthermore, the guidelines are optimized for specific tasks, but no experiments examine whether they can be used in different domains or adapted with only a few examples. It therefore remains unclear how well AGOPS works on new domains.
Questions for authors
-
Some benchmarks remove nearly all information needed to answer the question, making refusing to answer a reasonable response rather than a failure. Although the paper reports abstention rates separately, the large accuracy drops do not distinguish between incorrect answers and reasonable refusals. An evaluation that rewards appropriate refusals or clarifying questions, together with more naturally underspecified prompts, would provide stronger evidence.
-
Table 6 reports No Guideline performance at K=8 on the validation set (Appendix D.1), including 74.0 on GSM8K-Abstain, while Table 1 reports 8.9 for GPT-4.1-mini and 15.2 for Qwen3-32B on the test set. The Table 6 results are higher than both Table 1 results across all four datasets. The authors should clarify which solver, metric, and evaluation settings were used in Table 6 and explain these large differences. Furthermore, Table 7 reports an average improvement of +19.22, but the displayed values (91.36 and 72.41) give a difference of 18.95 percentage points. The use of reference answers and reasoning traces in Sections 3.1 and 3.2 could also be explained more clearly.
I would be willing to change my rating according to the authors' response.