- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper learns a cross-episode self-improvement policy by offline optimizing the natural-language prompt of a Meta-Agent, which rewrites the Actor prompt based on past trajectories. Experiments cover three sequential decision-making benchmarks and show positive gains on some held-out domains.
Critical strengths
- The experiments fix the Actor and largely share the same adaptation LLM across methods, and the Naive Meta-Agent uses the same architecture and models. This rules out part of the explanation that gains come only from a stronger model.
- The experiments cover text games (Jericho), web navigation (WebArena-Lite), and customer-service dialogue with tool use (-bench), with held-out domains relative to meta-training to test transfer.
- Appendix A provides the complete meta-prompts from the main experiments. Several rules, such as distinguishing bad advice from unexecuted advice, preserving verified behaviors, and controlling exploration risk, are reasonable.
Critical weaknesses
W1. The full TTL session can be viewed as a compound system whose evaluation metric is W-AUC. GEPA's original system definition would then in principle cover this optimization problem directly.
Please clarify
- What framing the optimization target as a self-improvement policy adds beyond generic module optimization.
- Whether the two-stage reflection and BIS are necessary design choices.
A key experiment would be to let standard GEPA optimize the same Meta-Agent prompt with the same trajectories, feedback, and budget, and show whether the gap persists.
W2. . BIS is misaligned with the stated objective and may accept candidates that lower average performance (pp. 4–5, Eqs. (4), (6), (7)). The paper states that it optimizes mean W-AUC, .
However, because the new and current candidates are normalized with the same historical mean and standard deviation, BIS reduces to The historical mean cancels out, and only the inverse historical standard deviation remains. BIS therefore does not weight tasks equally. It gives more weight to tasks with low historical variance. A candidate can then obtain a positive BIS while lowering mean W-AUC, e.g., by gaining slightly on stable tasks while losing more on volatile ones.
Please justify this weighting relative to the stated objective:
- Report how often accepted candidates decrease mean W-AUC on .
- Compare against an acceptance rule based directly on the unnormalized mean difference.
W3. Data splits are insufficient to support generalization claims (Tables 1–3). Jericho ID reuses the three meta-training games. This is disclosed, but it is seen-task evaluation, not generalization. The split sizes and task IDs for WebArena and -bench are missing, and the relation between Jericho validation tasks and the training games is unclear.
- There are no repeats over independent splits or meta-training seeds.
- Rerunning the Actor with a fixed learned prompt does not show that the learning algorithm is stable.
- Jericho also has only three OOD games, and Zork 1 and Zork 3 are closely related, which limits claims of broad cross-environment generalization.
W4. The ablation study is mainly a comparison of alternative training methods (Table 5). The comparisons do not isolate the paper's core design choices. Please
- Ablate the two-stage fact extraction
- Replace BIS with mean W-AUC or fixed normalization weights
- Remove the distinction between Actor non-compliance and incorrect advice
- Drop verified retention or bounded exploration
- Compare against a carefully designed prompt of similar length without meta-training.
Without these, it is unclear which components drive the gains. The appendix cases illustrate plausible mechanisms, but they are selected post hoc and lack controls, so they cannot replace causal ablations.
W5. BIS normalizes each task's gain by its historical standard deviation , but the paper does not explain how this is handled when the history is empty in the first round, or when , e.g., when a task has always scored the same or has only one recorded value. In the latter case the score is undefined or dominated by an arbitrary small constant.
Please specify how BIS is computed in these cases and how often they occur in practice.
W6. The novelty claim is overstated. The paper frames its contribution as learning how to improve an agent from experience, rather than using a fixed, hand-designed strategy. This is not new on its own:
-
Promptbreeder [1] evolves the mutation-prompts that control prompt modification.
-
MR-Search [2] learns cross-episode reflection strategies.
-
LSE [3] learns a context-editing policy rewarded by execution improvement.
-
The narrower distinction, a frozen-weight, prompt-space Meta-Agent that transfers to unseen tasks, also overlaps with GEPA [4] and MetaSPO [5]. The limited GEPA comparison in Table 5 does not show that the proposed designs yield gains beyond existing frameworks. GEPA also returns the single best candidate on the validation set, so its Pareto pool alone does not make it unsuited to learning general policies. Please narrow the novelty claim.
[1]Promptbreeder: Self-Referential Self-Improvement via Prompt Evolution. In Proceedings of the 41st International Conference on Machine Learning, 2024.
[2] Meta-Reinforcement Learning with Self-Reflection for Agentic Search. arXiv preprint arXiv:2603.11327, 2026.
[3] Learning to Self-Evolve. arXiv preprint arXiv:2603.18620, 2026.
[4] GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. In International Conference on Learning Representations (ICLR), 2026.
[5] System Prompt Optimization with Meta-Learning.** In Advances in Neural Information Processing Systems (NeurIPS), 2025.
Questions for authors
See Weakness.
5 comments
Response (Part 2/2)
W4: Ablations of design choices.
Two-stage reflection and BIS. see W1.
Hand-written meta-prompt of similar length.
Thanks for raising this. We wrote this meta-prompt by hand. It describes the environment and guides diagnosis, learning from feedback, and exploration (Appendix A.5).
Jericho W-AUC (mean ± std over 5 runs):
| Method | Detective | Zork 1 | Temple | ID Avg. | Balances | Library | Zork 3 | OOD Avg. |
|---|---|---|---|---|---|---|---|---|
| Naive Meta-Agent | 0.327 ± 0.047 | 0.129 ± 0.017 | 0.189 ± 0.056 | 0.215 ± 0.022 | 0.200 ± 0.021 | 0.409 ± 0.077 | 0.248 ± 0.046 | 0.286 ± 0.023 |
| Hand-Written Meta-Prompt | 0.670 ± 0.069 | 0.143 ± 0.013 | 0.208 ± 0.037 | 0.340 ± 0.028 | 0.275 ± 0.034 | 0.451 ± 0.114 | 0.378 ± 0.062 | 0.368 ± 0.046 |
| Meta-TTL | 0.790 ± 0.026 | 0.150 ± 0.007 | 0.248 ± 0.040 | 0.396 ± 0.019 | 0.292 ± 0.046 | 0.522 ± 0.107 | 0.427 ± 0.018 | 0.414 ± 0.023 |
τ²-bench W-AUC (mean ± std over 3 runs):
| Method | Airline | Retail | Telecom (OOD) |
|---|---|---|---|
| Naive Meta-Agent | 0.628 ± 0.046 | 0.547 ± 0.043 | 0.478 ± 0.020 |
| Hand-Written Meta-Prompt | 0.636 ± 0.038 | 0.526 ± 0.038 | 0.510 ± 0.024 |
| Meta-TTL | 0.678 ± 0.038 | 0.686 ± 0.032 | 0.554 ± 0.029 |
Italics mark the stronger baseline in each column. This suggests that the gains of Meta-TTL are hard to replicate with manual guidance. Results on both metrics are in Appendix G.
Rules in the learned meta-prompt. We would like to clarify that §4.4 is an analysis of the strategies that emerge from meta-training, rather than a causal attribution of the gains. Nevertheless, to better understand where the gains come from, we removed the instructions that clearly correspond to verified retention and bounded exploration (W-AUC, mean ± std over 5 runs):
| Method | Detective | Zork 1 | Temple | ID Avg. | Balances | Library | Zork 3 | OOD Avg. |
|---|---|---|---|---|---|---|---|---|
| Meta-TTL | 0.790 ± 0.026 | 0.150 ± 0.007 | 0.248 ± 0.040 | 0.396 ± 0.019 | 0.292 ± 0.046 | 0.522 ± 0.107 | 0.427 ± 0.018 | 0.414 ± 0.023 |
| w/o retention & exploration | 0.628 ± 0.083 | 0.162 ± 0.029 | 0.216 ± 0.033 | 0.335 ± 0.038 | 0.276 ± 0.029 | 0.443 ± 0.123 | 0.371 ± 0.048 | 0.364 ± 0.051 |
These instructions make up about half of the meta-prompt (~800 of ~1,600 words). Removing them lowers both ID and OOD performance, suggesting that these strategies contribute to the gains.
W5: BIS edge cases.
Thanks for the careful check. To clarify, in the normalization step (Algorithm 2, line 17, Appendix C), each task's pool contains the score history plus both candidates' current scores. Since the score being normalized is always in the pool, the pool is never empty, and if σ = 0 the z-score is simply 0. We will make this explicit in Eq. (6) and Appendix C.
W6: Novelty.
- Promptbreeder [1]. Promptbreeder's goal is to evolve a task-prompt for a single task. Its mutation-prompts only assist this search and are not required to transfer to other tasks. In Meta-TTL, the learned update rule (the meta-prompt) is the final output, and it is meant to remain useful on unseen tasks.
- MR-Search [2] and LSE [3]. Both are discussed in our related work (L141–145), and both learn via policy gradients on model weights. Their goals also differ from ours: MR-Search trains a search agent for QA, while LSE reduces self-improvement to a single-step bandit objective and delegates exploration to a fixed tree search at test time.
- MetaSPO [5]. MetaSPO meta-learns a system prompt for the actor, while its meta-agent is a fixed, hand-designed ProTeGi-style optimizer. Meta-TTL's objective is different: it learns a system prompt for the meta-agent (our meta-prompt), i.e., how to improve the actor.
- GEPA [4]. see W1
Following the reviewer's suggestion, we will revise our claim as follows:
Revised positioning. Our work shows that (1) test-time learning in multi-turn agentic tasks benefits from being meta-learned, and (2) this meta-learning can be done entirely in prompt space, which makes the learned meta-policy inspectable and allows explicit separation of task-specific knowledge from meta-strategies. We hope this provides some insight into how meta-learning can be done when gradient-based methods are costly or infeasible (e.g., with long agentic rollouts or closed models).
We hope these results and clarifications address the reviewer's concerns, and we are happy to discuss any questions that remain.
[1] Promptbreeder: Self-Referential Self-Improvement via Prompt Evolution. ICML 2024.
[2] Meta-Reinforcement Learning with Self-Reflection for Agentic Search. COLM 2026.
[3] Learning to Self-Evolve. arXiv:2603.18620, 2026.
[4] GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. ICLR 2026.
[5] System Prompt Optimization with Meta-Learning. NeurIPS 2025.
Gentle Reminder
Dear Reviewer F5Kw,
As the discussion period ends soon, we would appreciate it if you could let us know whether our response addresses your concerns. We are happy to clarify anything further. Thank you for your time and consideration.
Response (Part 1/2)
We thank the reviewer for the detailed and technical review.
W1: Relation to GEPA and necessity of our designs.
We agree that GEPA can in principle be applied (the "GEPA outer loop" in Table 5 does exactly this). However, GEPA is built for per-task optimization and has two failure modes for meta-learning. Our two designs aim to address these:
The issue is not the final selection but the selection pressure during search. GEPA keeps a candidate if it is best on any single task, so the final best-average choice only picks the most general one among specialized experts. Meta-TTL instead selects for generality at every step.
To test both designs, we ran GEPA with the same proposer instruction and budget, and ablated each design on Jericho (W-AUC, mean ± std over 5 runs):
Full per-game results with both metrics are reported in Table 15 of the revised appendix. Removing BIS barely changes ID performance but substantially lowers OOD performance. Removing two-stage reflection lowers both, and GEPA is lower on both.
W2: BIS and the objective in Eq. (4).
Thanks for raising this. BIS addresses two gaps. (1) The plain validation average is a noisy estimate of Eq. (4), since high-variance tasks can dominate acceptance. (2) Eq. (4) is itself a proxy for performance on unseen tasks, and gains concentrated on a few easy-to-improve tasks may not transfer. By normalizing each task against its historical spread, BIS favors candidates that improve broadly. We will make this explicit in §3.3.
Across meta-training runs, 6 of 23 (26%) candidates accepted by BIS lowered the raw mean validation W-AUC, and removing BIS lowers OOD performance (W1).
W3: Data splits and generalization.
Jericho ID games.
This is inherent to text games. Each game is a single task and cannot be split, so the validation tasks are also the training games. Since each game has a vast branching space and the meta-prompt contains no game-specific facts (Appendix A), ID results still reflect in-session learning. Generalization is evaluated on the OOD games.
More OOD games. We added three OOD games unrelated to the Zork series (Deephome, Pentari, and Reverb) (W-AUC, mean ± std over 5 runs):
Split sizes and task IDs. We have added the task splits to Appendix J. The ID evaluation sets are inherently constrained by the size of each domain, since tasks must also be reserved for meta-training. For τ²-bench, we doubled the evaluation sets and added two more seeds. Meta-TTL still outperforms the best baseline in all three domains (Table 11).
Variance across meta-training runs. Meta-TTL has been meta-trained independently in three different settings (main, Table 5, Appendix D) and yields consistent gains across all of them.