- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors introduce SEPO, a system prompt ensemble method for improving a model's diversity, quality, and novelty on different tasks.
Critical strengths
-
The method is light-weight and does not require additional compute
-
The authors tested SEPO against four test-time baselines on four datasets, which is pretty comprehensive
-
The three dimensions (i.e. quality, diversity, and novelty) are well explained and defined in the problem definition section.
-
Evaluation for novelty is pretty comprehensive and includes methods from LLM judge to embedding and n-gram metrics.
Critical weaknesses
-
SEPO is conceptually similar to Verbalized Sampling in the sense that it elicits different response variations by prompting the model
-
There’s no explanation for how the base system prompts are constructed or selected? Since system prompt ensemble is the core of the SEPO algorithm, this missing part seems essential. The paper would also benefit from conducting some causal analysis showing how different prompt additions or combinations lead to greater or lower diversity, as well as providing some qualitative examples of what the prompts look like in the main text.
-
[minor]: “Evaluation Tasks” appeared twice in section 5.
Questions for authors
-
Have you compared SEPO with other methods for improving diversity (e.g. spectrum tuning: https://proceedings.iclr.cc/paper_files/paper/2026/hash/7b6d77bf723ab4fed4f88baf544683fb-Abstract-Conference.html or diverse preference optimization: https://arxiv.org/abs/2501.18101)?
-
Please double check your citation style.
-
How are the pool of base system prompts selected?
-
Have you considered using a reward model score to evaluate for quality? (besides an LLM judge)