- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies a meaningful question: whether a language model can be trained to distill prompt-update rules from interaction experience in order to improve performance on future tasks. Freezing the actor, using verifiable environment rewards, and evaluating across different reflectors and actors provide encouraging empirical signals.
However, the current evaluation selects prompts on a fixed validation set and reports post-selection scores on that same set, without an independent final test. In addition, checkpoint validation is exposed to task types from the meta-test split, which weakens the interpretation of the results as generalization to strictly unseen task types.
Together with the other issues raised in the weaknesses below, in summary, these concerns lead me to conclude that the paper is not yet ready for publication in its current version.
Critical strengths
S1: The paper distinguishes solving the current instance from distilling experience for future instances. This is a more meaningful setting than retrying the same task, since many real-world interactions cannot be rolled back.
S2: Keeping the actor frozen in the main experiments isolates the effect of prompt updates from changes in actor capability.
S3: Task success in practical environments, (i.e., ALFWorld and MiniHack) provides a reliable training signal and reduces evaluation bias.
Critical weaknesses
W1: Training a reflector to improve an actor has precedents in Retroformer [1] and SAMULE [4]; cross-task experience consolidation is explored in ExpeL [2] and MetaReflection [3]; and RL-based prompt generation is established in StablePrompt [5]. The potentially distinctive contribution is the no-retry, cross-instance adaptation setting, rather than these components themselves.
However, comparisons primarily against untrained reflectors do not sufficiently establish the adadamic value over these related approaches. Clearer differentiation and targeted comparisons are needed.
-
[1] Weiran Yao, Shelby Heinecke, Juan Carlos Niebles, Zhiwei Liu, Yihao Feng, Le Xue, Rithesh Murthy, Zeyuan Chen, Jianguo Zhang, Devansh Arpit, Ran Xu, Phil Mui, Huan Wang, Caiming Xiong, and Silvio Savarese. Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization. ICLR, 2024.
-
[2] Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. ExpeL: LLM Agents Are Experiential Learners. Proceedings of the AAAI Conference on Artificial Intelligence, 38(17):19632–19642, 2024.
-
[3] Priyanshu Gupta, Shashank Kirtania, Ananya Singha, Sumit Gulwani, Arjun Radhakrishna, Gustavo Soares, and Sherry Shi. MetaReflection: Learning Instructions for Language Agents using Past Reflections. EMNLP, 2024.
-
[4] Yubin Ge, Salvatore Romeo, Jason Cai, Monica Sunkara, and Yi Zhang. SAMULE: Self-Learning Agents Enhanced by Multi-level Reflection. EMNLP,
-
[5] Minchan Kwon, Gaeun Kim, Jongsuk Kim, Haeil Lee, and Junmo Kim. StablePrompt: Automatic Prompt Tuning using Reinforcement Learning for Large Language Model. EMNLP, 2024.
-
[6] Qizheng Zhang, Changran Hu, Shubhangi Upasani, Boyuan Ma, Fenglu Hong, Vamsidhar Kamanuru, Jay Rainton, Chen Wu, Mengmeng Ji, Hanchen Li, Urmish Thakker, James Zou, and Kunle Olukotun. Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models. ICLR, 2026.
-
[7] Lakshya A. Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J. Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. ICLR, 2026, Oral.
-
[8] Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric Xing, and Zhiting Hu. RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning. EMNLP, pp. 3369–3391, 2022.
-
[9] Yumin Choi, Jinheon Baek, and Sung Ju Hwang. System Prompt Optimization with Meta-Learning. NeurIPS, 2025.
-
[10] Yulun Jiang, Liangze Jiang, Damien Teney, Michael Moor, and Maria Brbić. Meta-RL Induces Exploration in Language Agents. ICLR, 2026.
W2: The training reward optimizes same-instance retries, not generalization to new instances (Eq. 4, p. 5).
- The reward is computed on the same batch whose trajectories the reflector has just seen,
- The actual goal is on new instances from the same task family.
Because is conditioned on , a higher reward on does not imply a higher reward on future instances. The model may learn instance-specific fixes rather than transferable rules. Training with a replayable simulator is reasonable, but asking for "future-oriented" prompts in natural language does not constrain generalization.
Please compare observed-batch reward with fresh-batch reward (observe , evaluate on an independent ) and a mixed variant, and report how retry gains correlate with gains on new instances.
W3: Training in this paper is a one-step GRPO update on the prompt. the four collection rounds provide states from different stages but do not amount to multi-round credit assignment over a long-horizon meta-return.
W4: Best-so-far curves do not show sustained improvement (Fig. 4, p. 8; p. 9). Figure 4 plots , which is non-decreasing by construction. It can keep rising even if the current prompt stops improving or degrades. Likewise, the fraction of best prompts found after round 4 does not show that a long-horizon adaptation strategy has been learned, since sampling more candidates of similar quality would also tend to produce a later maximum.
Please report the held-out performance of the current prompt at each round, the final prompt's performance without validation-based selection, a random-candidate/restart baseline with the same budget, and the average round-to-round gain and degradation rate.
W5: Baselines are too weak to establish the need for a trained reflector. The main table compares the initial prompt, an untrained reflector, and the trained reflector. This shows that fine-tuning helps, but not whether the training pipeline is worth adopting over existing experience-summarization or context-update methods.
Please add
- ExpeL (insights-only)
- ACE-style online context updates
- Well-prompted but untrained reflector
- Multi-candidate prompt search under the same budget.
- Methods that allow repeated evaluation, such as GEPA
could be reported separately as a more permissive setting, with their extra budget stated explicitly.
W6: The batch-size ablation is confounded. Moving between settings changes several factors at once: the experience batch size at training and at test time, the MiniHack training episode length (reduced to 18 steps for ), some context/response lengths, and the checkpoint used. As a result, the failure of on MiniHack cannot be cleanly attributed to too few successful samples and a weak GRPO signal.
Please vary one factor at a time, e.g., change only the training batch size while holding episode length, context limits, test-time batch size, and checkpoint selection fixed.
W7: The initial prompt is excluded from best-so-far tracking (Alg. 1). Algorithm 1 does not initialize the tracked score with the actual performance of , and takes the maximum over only. If every update performs worse than the initial prompt, the curve would still look like steady progress.
Please explain why is excluded, and report how often updated prompts underperform , as well as results with included in the maximum.
W8: It is also unclear whether Tables 1–5 report scores on the same validation set used for prompt selection.
If so, results on a fully independent test set are needed.
Questions for authors
See Weakness