- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes to discover optimization strategies for a frozen LLM through executable practice. An agent first writes and refines optimizer programs on simple synthetic functions, and the practice is then distilled into a short text of principles. The text is given to the LLM at deployment without running any program, and the paper shows that it improves black-box optimization on new instances and several transfer settings.
Critical strengths
- The paper is well-written and easy to follow.
- The idea of learning strategies through executable practice and then turning them into plain text is interesting.
Critical weaknesses
- Unclear Motivation. It is not shown that executable practice is really necessary. A text written directly by the same LLM without any practice may work just as well, but this comparison is missing.
- Limited Baselines. The comparison with other prompt optimization methods is too thin, and the original programs are never tested on the transfer tasks, so it is unclear what is gained by turning them into text.
Questions for authors
- How would a strategy text written directly by the LLM without practice perform?
- Why is text a better way to carry the strategy than the program itself?
- Missing references. https://arxiv.org/abs/2405.20132, https://arxiv.org/abs/2401.02051 and https://arxiv.org/abs/2402.01145 should be discussed.