acceptodds
Under review as a conference paper at ICLR 2027

Meta-TTL: Meta-Learning Self-Improvement Policies for Language Agents

12 views

Abstract

Test-Time Learning (TTL) enables language agents to iteratively refine their performance through repeated interactions with the environment at inference time. At the core of TTL is a self-improvement policy that updates the actor policy based on experience from previous episodes, thereby improving future behavior. Existing methods rely on hand-crafted self-improvement rather than optimizing them for downstream improvement. We argue that optimal self-improvement policies should be learned from task environments, not hand-engineered based on human intuition. To achieve this, we introduce Meta-TTL, a framework that formulates the discovery of effective self-improvement policies as a bi-level optimization problem. Within this framework, the inner loop executes the standard TTL process, measuring how effectively a candidate self-improvement policy helps an agent correct errors across sequential episodes. Guided by the agent's performance, the outer loop performs reflective meta-training across diverse training tasks, using a balanced improvement score (BIS) to balance task contributions during candidate selection. We evaluate Meta-TTL on Jericho, WebArena-Lite, and -bench across both in-distribution (ID) and out-of-distribution (OOD) settings. Meta-TTL consistently outperforms existing baselines, improving TTL over the strongest baseline by up to 23% on ID tasks and 27% on OOD tasks. These results suggest that the optimized self-improvement policy encodes transferable meta-strategies that generalize beyond the training task distribution. Code is available at https://anonymous.4open.science/r/metattl-DD55.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
test-time learning, language agents, meta-learning, prompt optimization, self-improving AI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_F5KwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper learns a cross-episode self-improvement policy by offline optimizing the natural-language prompt of a Meta-Agent, which rewrites the Actor prompt based on past trajectories. Experiments cover three sequential decision-making benchmarks and show positive gains on some held-out domains.

Critical strengths
  1. The experiments fix the Actor and largely share the same adaptation LLM across methods, and the Naive Meta-Agent uses the same architecture and models. This rules out part of the explanation that gains come only from a stronger model.
  2. The experiments cover text games (Jericho), web navigation (WebArena-Lite), and customer-service dialogue with tool use (τ2\tau^2-bench), with held-out domains relative to meta-training to test transfer.
  3. Appendix A provides the complete meta-prompts from the main experiments. Several rules, such as distinguishing bad advice from unexecuted advice, preserving verified behaviors, and controlling exploration risk, are reasonable.
Critical weaknesses

W1. The full TTL session can be viewed as a compound system Φϕ(g)=RUNTTL(ϕ,g)\Phi_\phi(g) = \mathrm{RUNTTL}(\phi, g) whose evaluation metric is W-AUC. GEPA's original system definition would then in principle cover this optimization problem directly.

Please clarify

  • What framing the optimization target as a self-improvement policy adds beyond generic module optimization.
  • Whether the two-stage reflection and BIS are necessary design choices.

A key experiment would be to let standard GEPA optimize the same Meta-Agent prompt with the same trajectories, feedback, and budget, and show whether the gap persists.

W2. . BIS is misaligned with the stated objective and may accept candidates that lower average performance (pp. 4–5, Eqs. (4), (6), (7)). The paper states that it optimizes mean W-AUC, ϕ∗=maxϕEg∼Dtrain[W-AUC(ξgϕ)]\phi^* = max_\phi E_{g \sim D_{\text{train}}}[\text{W-AUC}(\xi_g^\phi)].

However, because the new and current candidates are normalized with the same historical mean and standard deviation, BIS reduces to BISt=1∣Dval∣∑hshϕ~−shϕ∗σ(Hh,t).BIS_t = \frac{1}{|D_{\text{val}}|} \sum_h \frac{s_h^{\tilde{\phi}} - s_h^{\phi^*}}{\sigma(H_{h,t})}. The historical mean cancels out, and only the inverse historical standard deviation remains. BIS therefore does not weight tasks equally. It gives more weight to tasks with low historical variance. A candidate can then obtain a positive BIS while lowering mean W-AUC, e.g., by gaining slightly on stable tasks while losing more on volatile ones.

Please justify this weighting relative to the stated objective:

  • Report how often accepted candidates decrease mean W-AUC on DvalD_{\text{val}}.
  • Compare against an acceptance rule based directly on the unnormalized mean difference.

W3. Data splits are insufficient to support generalization claims (Tables 1–3). Jericho ID reuses the three meta-training games. This is disclosed, but it is seen-task evaluation, not generalization. The split sizes and task IDs for WebArena and τ2\tau^2-bench are missing, and the relation between Jericho validation tasks and the training games is unclear.

  • There are no repeats over independent splits or meta-training seeds.
  • Rerunning the Actor with a fixed learned prompt does not show that the learning algorithm is stable.
  • Jericho also has only three OOD games, and Zork 1 and Zork 3 are closely related, which limits claims of broad cross-environment generalization.

W4. The ablation study is mainly a comparison of alternative training methods (Table 5). The comparisons do not isolate the paper's core design choices. Please

  • Ablate the two-stage fact extraction
  • Replace BIS with mean W-AUC or fixed normalization weights
  • Remove the distinction between Actor non-compliance and incorrect advice
  • Drop verified retention or bounded exploration
  • Compare against a carefully designed prompt of similar length without meta-training.

Without these, it is unclear which components drive the gains. The appendix cases illustrate plausible mechanisms, but they are selected post hoc and lack controls, so they cannot replace causal ablations.

W5. BIS normalizes each task's gain by its historical standard deviation σ(Hh,t)\sigma(H_{h,t}), but the paper does not explain how this is handled when the history is empty in the first round, or when σ(Hh,t)=0\sigma(H_{h,t}) = 0, e.g., when a task has always scored the same or has only one recorded value. In the latter case the score is undefined or dominated by an arbitrary small constant.

Please specify how BIS is computed in these cases and how often they occur in practice.

W6. The novelty claim is overstated. The paper frames its contribution as learning how to improve an agent from experience, rather than using a fixed, hand-designed strategy. This is not new on its own:

  • Promptbreeder [1] evolves the mutation-prompts that control prompt modification.

  • MR-Search [2] learns cross-episode reflection strategies.

  • LSE [3] learns a context-editing policy rewarded by execution improvement.

  • The narrower distinction, a frozen-weight, prompt-space Meta-Agent that transfers to unseen tasks, also overlaps with GEPA [4] and MetaSPO [5]. The limited GEPA comparison in Table 5 does not show that the proposed designs yield gains beyond existing frameworks. GEPA also returns the single best candidate on the validation set, so its Pareto pool alone does not make it unsuited to learning general policies. Please narrow the novelty claim.

[1]Promptbreeder: Self-Referential Self-Improvement via Prompt Evolution. In Proceedings of the 41st International Conference on Machine Learning, 2024.

[2] Meta-Reinforcement Learning with Self-Reflection for Agentic Search. arXiv preprint arXiv:2603.11327, 2026.

[3] Learning to Self-Evolve. arXiv preprint arXiv:2603.18620, 2026.

[4] GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. In International Conference on Learning Representations (ICLR), 2026.

[5] System Prompt Optimization with Meta-Learning.** In Advances in Neural Information Processing Systems (NeurIPS), 2025.

Questions for authors

See Weakness.

5 comments
AuthorsOpenReview ↗

Response (Part 1/2)

We thank the reviewer for the detailed and technical review.

W1: Relation to GEPA and necessity of our designs.

We agree that GEPA can in principle be applied (the "GEPA outer loop" in Table 5 does exactly this). However, GEPA is built for per-task optimization and has two failure modes for meta-learning. Our two designs aim to address these:

  1. Task-specific facts leak into the learned policy. Two-stage reflection first extracts task-specific facts, so the proposed meta-prompt keeps only transferable strategy.
  2. Selection is driven by individual tasks. Meta-TTL instead compares candidates on all validation tasks at every step. BIS further prevents high-variance tasks from dominating the comparison.

"Pareto pool alone does not make it unsuited to learning general policies"

The issue is not the final selection but the selection pressure during search. GEPA keeps a candidate if it is best on any single task, so the final best-average choice only picks the most general one among specialized experts. Meta-TTL instead selects for generality at every step.

To test both designs, we ran GEPA with the same proposer instruction and budget, and ablated each design on Jericho (W-AUC, mean ± std over 5 runs):

Outer loopID Avg.OOD Avg.
Meta-TTL0.396 ± 0.0190.414 ± 0.023
w/o BIS (plain mean difference)0.384 ± 0.0260.330 ± 0.040
w/o two-stage reflection0.334 ± 0.0560.381 ± 0.042
GEPA (same proposer)0.344 ± 0.0260.380 ± 0.052

Full per-game results with both metrics are reported in Table 15 of the revised appendix. Removing BIS barely changes ID performance but substantially lowers OOD performance. Removing two-stage reflection lowers both, and GEPA is lower on both.

W2: BIS and the objective in Eq. (4).

Thanks for raising this. BIS addresses two gaps. (1) The plain validation average is a noisy estimate of Eq. (4), since high-variance tasks can dominate acceptance. (2) Eq. (4) is itself a proxy for performance on unseen tasks, and gains concentrated on a few easy-to-improve tasks may not transfer. By normalizing each task against its historical spread, BIS favors candidates that improve broadly. We will make this explicit in §3.3.

Across meta-training runs, 6 of 23 (26%) candidates accepted by BIS lowered the raw mean validation W-AUC, and removing BIS lowers OOD performance (W1).

W3: Data splits and generalization.

Jericho ID games.

"Jericho ID reuses the three meta-training games."

This is inherent to text games. Each game is a single task and cannot be split, so the validation tasks are also the training games. Since each game has a vast branching space and the meta-prompt contains no game-specific facts (Appendix A), ID results still reflect in-session learning. Generalization is evaluated on the OOD games.

More OOD games. We added three OOD games unrelated to the Zork series (Deephome, Pentari, and Reverb) (W-AUC, mean ± std over 5 runs):

MethodDeephomePentariReverb
Static0.026 ± 0.0010.096 ± 0.0340.006 ± 0.013
Reflexion0.040 ± 0.0110.118 ± 0.0570.238 ± 0.045
AWM0.066 ± 0.0430.235 ± 0.0720.000 ± 0.000
EvoTest0.054 ± 0.0200.160 ± 0.0790.254 ± 0.020
Naive Meta-Agent0.039 ± 0.0060.146 ± 0.0570.209 ± 0.033
Meta-TTL0.085 ± 0.0270.296 ± 0.0360.286 ± 0.029
Δ vs. best baseline+29%+26%+13%

Split sizes and task IDs. We have added the task splits to Appendix J. The ID evaluation sets are inherently constrained by the size of each domain, since tasks must also be reserved for meta-training. For τ²-bench, we doubled the evaluation sets and added two more seeds. Meta-TTL still outperforms the best baseline in all three domains (Table 11).

Variance across meta-training runs. Meta-TTL has been meta-trained independently in three different settings (main, Table 5, Appendix D) and yields consistent gains across all of them.

AuthorsOpenReview ↗

Response (Part 2/2)

W4: Ablations of design choices.

Two-stage reflection and BIS. see W1.

Hand-written meta-prompt of similar length.

Thanks for raising this. We wrote this meta-prompt by hand. It describes the environment and guides diagnosis, learning from feedback, and exploration (Appendix A.5).

Jericho W-AUC (mean ± std over 5 runs):

MethodDetectiveZork 1TempleID Avg.BalancesLibraryZork 3OOD Avg.
Naive Meta-Agent0.327 ± 0.0470.129 ± 0.0170.189 ± 0.0560.215 ± 0.0220.200 ± 0.0210.409 ± 0.0770.248 ± 0.0460.286 ± 0.023
Hand-Written Meta-Prompt0.670 ± 0.0690.143 ± 0.0130.208 ± 0.0370.340 ± 0.0280.275 ± 0.0340.451 ± 0.1140.378 ± 0.0620.368 ± 0.046
Meta-TTL0.790 ± 0.0260.150 ± 0.0070.248 ± 0.0400.396 ± 0.0190.292 ± 0.0460.522 ± 0.1070.427 ± 0.0180.414 ± 0.023

τ²-bench W-AUC (mean ± std over 3 runs):

MethodAirlineRetailTelecom (OOD)
Naive Meta-Agent0.628 ± 0.0460.547 ± 0.0430.478 ± 0.020
Hand-Written Meta-Prompt0.636 ± 0.0380.526 ± 0.0380.510 ± 0.024
Meta-TTL0.678 ± 0.0380.686 ± 0.0320.554 ± 0.029

Italics mark the stronger baseline in each column. This suggests that the gains of Meta-TTL are hard to replicate with manual guidance. Results on both metrics are in Appendix G.

Rules in the learned meta-prompt. We would like to clarify that §4.4 is an analysis of the strategies that emerge from meta-training, rather than a causal attribution of the gains. Nevertheless, to better understand where the gains come from, we removed the instructions that clearly correspond to verified retention and bounded exploration (W-AUC, mean ± std over 5 runs):

MethodDetectiveZork 1TempleID Avg.BalancesLibraryZork 3OOD Avg.
Meta-TTL0.790 ± 0.0260.150 ± 0.0070.248 ± 0.0400.396 ± 0.0190.292 ± 0.0460.522 ± 0.1070.427 ± 0.0180.414 ± 0.023
w/o retention & exploration0.628 ± 0.0830.162 ± 0.0290.216 ± 0.0330.335 ± 0.0380.276 ± 0.0290.443 ± 0.1230.371 ± 0.0480.364 ± 0.051

These instructions make up about half of the meta-prompt (~800 of ~1,600 words). Removing them lowers both ID and OOD performance, suggesting that these strategies contribute to the gains.

W5: BIS edge cases.

Thanks for the careful check. To clarify, in the normalization step (Algorithm 2, line 17, Appendix C), each task's pool contains the score history plus both candidates' current scores. Since the score being normalized is always in the pool, the pool is never empty, and if σ = 0 the z-score is simply 0. We will make this explicit in Eq. (6) and Appendix C.

W6: Novelty.

  • Promptbreeder [1]. Promptbreeder's goal is to evolve a task-prompt for a single task. Its mutation-prompts only assist this search and are not required to transfer to other tasks. In Meta-TTL, the learned update rule (the meta-prompt) is the final output, and it is meant to remain useful on unseen tasks.
  • MR-Search [2] and LSE [3]. Both are discussed in our related work (L141–145), and both learn via policy gradients on model weights. Their goals also differ from ours: MR-Search trains a search agent for QA, while LSE reduces self-improvement to a single-step bandit objective and delegates exploration to a fixed tree search at test time.
  • MetaSPO [5]. MetaSPO meta-learns a system prompt for the actor, while its meta-agent is a fixed, hand-designed ProTeGi-style optimizer. Meta-TTL's objective is different: it learns a system prompt for the meta-agent (our meta-prompt), i.e., how to improve the actor.
  • GEPA [4]. see W1

Following the reviewer's suggestion, we will revise our claim as follows:

Revised positioning. Our work shows that (1) test-time learning in multi-turn agentic tasks benefits from being meta-learned, and (2) this meta-learning can be done entirely in prompt space, which makes the learned meta-policy inspectable and allows explicit separation of task-specific knowledge from meta-strategies. We hope this provides some insight into how meta-learning can be done when gradient-based methods are costly or infeasible (e.g., with long agentic rollouts or closed models).

We hope these results and clarifications address the reviewer's concerns, and we are happy to discuss any questions that remain.


[1] Promptbreeder: Self-Referential Self-Improvement via Prompt Evolution. ICML 2024.

[2] Meta-Reinforcement Learning with Self-Reflection for Agentic Search. COLM 2026.

[3] Learning to Self-Evolve. arXiv:2603.18620, 2026.

[4] GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. ICLR 2026.

[5] System Prompt Optimization with Meta-Learning. NeurIPS 2025.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer F5Kw,

As the discussion period ends soon, we would appreciate it if you could let us know whether our response addresses your concerns. We are happy to clarify anything further. Thank you for your time and consideration.

Reviewer_VFXiOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how to improve model performance at test time and proposes a method called Meta-TTL. Specifically, the authors first use a policy model to sample from the training data, then employ another model to refine the policy model’s system prompt based on the sampling results and environmental feedback. This process is repeated until the prompt is optimized, after which the resulting prompt is used for downstream evaluation. Experiments on three datasets demonstrate the performance gains achieved by the proposed approach.

Critical strengths
  1. The paper proposes a test-time optimization approach that improves model performance on specific tasks at relatively low cost by iteratively refining the model’s system prompt.

  2. The paper evaluates the proposed approach on three datasets, distinguishing between in-domain and out-of-domain settings, and clearly illustrates the performance differences among various methods.

  3. The paper provides several examples of how the system prompt evolves throughout the optimization process.

Critical weaknesses
  1. The paper appears to overstate its contribution by emphasizing “self-improvement,” whereas the primary experimental setup relies on two different LLMs, such as GPT and Gemini or Gemini and Qwen. This setup does not fully support the claim of self-improvement, as the optimization process depends on interactions between distinct models rather than a single model improving itself. The authors should clarify the scope of their claims to better align them with the experimental design.

  2. Although the paper presents multiple sets of experiments, the experimental configurations vary substantially, particularly in the models used, making the results across experiments difficult to compare directly. If these differences are intentional, the authors should clearly explain the rationale and discuss their implications for the interpretation of the results. Even if smaller models are necessary for comparisons involving SFT or GRPO, the configurations in Tables 4 and 5 should remain consistent to ensure a fair comparison. Furthermore, since the meta-agent is not trained and the authors do not explicitly aim to maintain consistency between the meta-agent and the policy model, the experimental configurations should remain consistent across all experiments to facilitate meaningful comparisons.

  3. A notable observation from Tables 1–3 is that the “Static” approach does not consistently perform worse than the other methods. This finding is somewhat counterintuitive, given that Static serves as the baseline without iterative prompt optimization and might be expected to represent a lower bound for other approaches. The authors should investigate this result and provide further analysis to explain why the optimization methods do not consistently outperform the Static baseline.

Questions for authors
  1. The paper proposes W-AUC as an optimization objective, which appears reasonable. However, it remains unclear whether this metric is appropriate for evaluating downstream inference performance. For example, in approaches such as Static, episodes are independent of one another, making cross-episode comparisons less meaningful. Given the proposed setup, metrics such as Best-of-N, Worst-of-N, and Avg-of-N may better reflect practical usage scenarios and provide a more comprehensive evaluation of the method’s effectiveness.

  2. The reference list consists almost entirely of arXiv preprints rather than the published versions of the corresponding papers. Where peer-reviewed versions are available, the authors should cite those versions in accordance with standard academic practice.

  3. The section title “LLM USAGE STATEMENT” does not conform to the specified guidelines.

2 comments

We thank the reviewer for the feedback. We first clarify our setup, as several concerns stem from it.

Clarification of our setup.

The review describes our method as:

"a test-time optimization approach that improves model performance on specific tasks" (Strengths)

"the authors first use a policy model to sample from the training data, then employ another model to refine the policy model's system prompt … the resulting prompt is used for downstream evaluation" (Summary)

"the meta-agent is not trained" (W2)

We would like to clarify that this misses one level of our method. In our framework, a policy model (actor agent) interacts with the environment, and a meta-agent refines the actor's system prompt. A third model (the proposer) iteratively improves the meta-agent's prompt, validating each candidate by running the process described above on validation tasks.

The review's description should thus be applied one level higher: the meta-agent's prompt is what we train and use for downstream evaluation on unseen tasks, not the actor's. Our core contribution is meta-learning this meta-policy in a gradient-free framework.

W1 (self-improvement with two LLMs).

By "self-improvement", we mean that the agent system improves on its own without human or external intervention. "Self" refers to the agent system as a whole rather than a single LLM. This follows standard usage in the literature: EvoTest [2], a self-improving agentic system, uses an o3 evolver with a Gemini actor, and Retroformer [1] uses a separate retrospective model to improve a GPT actor.

Moreover, Meta-TTL is not bound to a specific backbone (Appendix D) and works when the actor and meta-agent share the same backbone (Table 4). We will clarify this scope in the revision.

W2 (configurations across tables).

Each table answers a different question with all variables controlled within it, so unifying model sizes would not make Tables 4 and 5 meaningfully comparable.

  • Table 4: Does learning the update rule beat updating the actor's weights? Weight-updating methods need an open-weight actor, so all methods use Qwen3-8B, and Meta-TTL's meta-agent also uses Qwen3-8B for a fair comparison.

  • Table 5: Which outer loop best trains the meta-agent? Only the outer loop varies, so we keep the main-experiment Gemini-3-Flash actor. The GRPO baseline requires an open-weight meta-agent, and we use Qwen3-4B because each of its inputs is a full episode log (up to 50 actor turns), making larger models prohibitively expensive to train.

W3 (Static is not a lower bound).

Static is meant as a no-learning reference and is not guaranteed to be a lower bound, since a poorly designed update rule can make the actor worse over episodes. For example, on hard games such as Zork 3, most baselines fall below Static because their update rules (meta-policies) give vague and overly cautious advice to the actor agent, making the actor performing worse after each episode. This also motivates meta-learning the update rule.

Q1 (W-AUC vs. Best/Worst/Avg-of-N).

Avg-of-N is already reported as "Avg. Score" in Tables 1–3, with trends consistent with W-AUC. Following the reviewer's suggestion, we additionally report Best-of-N on Jericho, where dense rewards are available (N = 6, normalized by each game's maximum score, averaged over 5 runs):

MethodDetectiveZork 1TempleID Avg.BalancesLibraryZork 3OOD Avg.
Static0.3670.1310.1430.2140.1960.3330.4290.319
Reflexion0.3780.1370.1600.2250.1960.4270.4000.341
Memory Agent0.4220.1420.1430.2360.1760.4200.3710.323
AWM0.5110.1370.1770.2750.2160.3670.4000.327
TextGrad0.3670.1450.1940.2350.2160.5670.4290.404
EvoPrompt0.3610.1450.2400.2490.1960.5270.4290.384
EvoTest0.8280.1560.2510.4120.2160.3930.4570.355
Naive Meta-Agent0.4000.1540.2230.2590.2940.5800.3430.406
Meta-TTL0.8940.1620.2690.4420.3330.5470.4570.446

Bold marks the best and italics the second best. Meta-TTL achieves the best ID and OOD averages, with smaller margins since Best-of-N ignores how quickly the best episode is reached.

Q2 & Q3 (references and section title).

Thank you for pointing these out. We have updated the references to their peer-reviewed versions where available, and renamed the LLM usage section to follow the ICLR 2027 template in the revision.

We hope this clarifies our setup and addresses the reviewer's concerns.


[1] Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization. ICLR 2024.

[2] EvoTest: Evolutionary Test-Time Learning for Self-Improving Agentic Systems. ICLR 2026.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer VFXi,

As the discussion period ends soon, we would appreciate it if you could let us know whether our response addresses your concerns. We are happy to clarify anything further. Thank you for your time and consideration.

Reviewer_siWGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies test-time learning (TTL) across repeated episodes of the same task. A frozen actor interacts with the environment, while a meta-agent rewrites the actor’s system prompt between episodes. Meta-TTL optimizes the meta-agent’s own prompt offline: an inner loop evaluates complete TTL sessions using weighted area under the learning curve (W-AUC), and an outer loop proposes revised meta-prompts through reflection and selects them using a task-normalized balanced improvement score (BIS). Experiments on Jericho, WebArena-Lite, and τ²-bench report the highest aggregate W-AUC among the evaluated methods in both ID and OOD settings, with relative improvements of up to 23% and 27%, respectively.

Critical strengths
  • Optimizing the rule that updates an actor is a useful framing. The resulting natural-language meta-prompts are readable and inspectable.
  • The Naive Meta-Agent provides a matched control with the same architecture and models. Improvements over this baseline provide evidence that optimizing the meta-prompt is beneficial.
  • The full learned meta-prompts and qualitative examples help explain how the method changes subsequent behavior.
Critical weaknesses
  1. The contribution needs clearer positioning relative to prior work on learning reflection and optimizing adaptation procedures. Retroformer (ICLR 2024) already trains a retrospective model using policy gradients to improve a frozen actor’s prompts from environmental feedback. MetaSPO (NeurIPS 2025) studies bilevel system-prompt optimization across tasks, although its adaptation setting differs from the cross-episode setting here. Neither is discussed. Meta-TTL’s prompt-space optimization may still be a meaningful distinction, but the paper should identify precisely what new capability or methodological insight it contributes beyond these approaches and existing reflective prompt optimization.

  2. The paper does not show that it learns transferable meta-strategies. Each benchmark has its own meta-prompt and OOD tasks are inside the same benchmark. Zork 3 is a Zork 1 sequel, Shopping Admin runs the same store as Shopping. Jericho ID uses the meta-training games (l. 245), so it is training-set performance. The Naive Meta-Agent has an empty strategy section in the released code, so we cannot tell whether the learned prompt beats a hand-written reflection prompt of similar length. Outside Jericho, the margins are small (+0.04 to +0.07 W-AUC) on small eval sets (10 WebArena-Lite ID tasks per site, 20 τ² Airline tasks), and variance is not reported.

Questions for authors
  1. How does Meta-TTL differ from Retroformer and MetaSPO? Can you compare with GEPA using the same proposer instruction, and ablate BIS against a plain mean difference?
  2. How does a meta-prompt trained on one benchmark do on another? And how does it compare with a hand-written meta-prompt of similar length?
3 comments
AuthorsOpenReview ↗

Response (Part 1/2)

We thank the reviewer for the careful reading and the concrete suggestions.

W1: Relation to existing work.

MetaSPO [1]. MetaSPO meta-learns a system prompt for the actor, while its meta-agent is a fixed, hand-designed ProTeGi-style optimizer. Meta-TTL's objective is different: it learns a system prompt for the meta-agent (our meta-prompt), i.e., how to improve the actor.

Retroformer [2]. Retroformer trains the meta-agent's weights with policy gradients, and rewards each reflection only by the next episode's return. Meta-TTL instead optimizes the meta-agent's prompt without gradients (it works with closed models) and rewards the whole learning curve (W-AUC), so it can learn strategies that pay off later. Moreover, Table 5 includes a similar policy-gradient baseline (GRPO), over which Meta-TTL shows better rollout efficiency.

We will add both discussions to the related work.

Revised positioning. Our work shows that (1) test-time learning in multi-turn agentic tasks benefits from being meta-learned, and (2) this meta-learning can be done entirely in prompt space, which makes the learned meta-policy inspectable and allows for explicit separation of task-specific knowledge from meta-strategies. We hope this provides some insight into how meta-learning can be done when gradient-based methods are costly or infeasible (e.g., with long agentic rollouts or closed models).

W2: Evaluation size, variance, and generalization.

Evaluation set size and variance. The ID evaluation sets are inherently constrained by the size of each task domain, since tasks must be split into training and validation sets for meta-training. For WebArena-Lite, we mitigate this by averaging over three runs, and variance is now provided in the revised appendix E.

For τ²-bench, to address this concern, we doubled the evaluation sets to 20 / 40 / 40 tasks for Airline / Retail / Telecom and ran two additional seeds for all methods. W-AUC (mean ± std over 3 seeds):

MethodAirlineRetailTelecom (OOD)
Static0.539 ± 0.0380.476 ± 0.0490.239 ± 0.021
Reflexion0.583 ± 0.0300.536 ± 0.0300.364 ± 0.032
Memory Agent0.592 ± 0.0930.569 ± 0.0260.396 ± 0.014
AWM0.503 ± 0.0320.517 ± 0.0250.399 ± 0.032
TextGrad0.575 ± 0.0380.551 ± 0.0230.353 ± 0.052
EvoPrompt0.528 ± 0.0130.483 ± 0.0550.285 ± 0.013
EvoTest0.564 ± 0.0500.526 ± 0.0440.401 ± 0.009
Naive Meta-Agent0.628 ± 0.0460.547 ± 0.0430.478 ± 0.020
Meta-TTL0.678 ± 0.0380.686 ± 0.0320.554 ± 0.029
Δ vs. best baseline+8%+21%+16%

Italics mark the strongest baseline in each column. Meta-TTL achieves the highest W-AUC on all three domains. Full results with both metrics are reported in Table 11 of the revised appendix.

OOD domains and Jericho ID games.

"Zork 3 is a Zork 1 sequel, Shopping Admin runs the same store as Shopping."

Although Shopping Admin and Shopping are built on the same e-commerce platform, Shopping Admin is a different site with different interfaces and task types. For Jericho, we added three OOD games (Deephome, Pentari, and Reverb) unrelated to the existing ones, and compare against the strongest baseline from each category (W-AUC, mean ± std over 5 runs):

MethodDeephomePentariReverb
Static0.026 ± 0.0010.096 ± 0.0340.006 ± 0.013
Reflexion0.040 ± 0.0110.118 ± 0.0570.238 ± 0.045
AWM0.066 ± 0.0430.235 ± 0.0720.000 ± 0.000
EvoTest0.054 ± 0.0200.160 ± 0.0790.254 ± 0.020
Naive Meta-Agent0.039 ± 0.0060.146 ± 0.0570.209 ± 0.033
Meta-TTL0.085 ± 0.0270.296 ± 0.0360.286 ± 0.029
Δ vs. best baseline+29%+26%+13%

Meta-TTL's gain remains consistent on these three games.

"Jericho ID uses the meta-training games"

This is inherent to text games. Each game is a single task and cannot be split, so the validation tasks are also the training games. Since each game has a vast branching space and the meta-prompt contains no game-specific facts (Appendix A), ID results still reflect in-session learning. Generalization is evaluated on the OOD games.


[1] System Prompt Optimization with Meta-Learning. NeurIPS 2025.

[2] Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization. ICLR 2024.

AuthorsOpenReview ↗

Response (Part 2/2)

Q1: Comparison with GEPA and BIS ablation.

Thank you for suggesting these comparisons. We ran both on Jericho (average W-AUC over the ID and OOD games, mean ± std over 5 runs). All variants use a Gemini-3-Flash actor and a GPT-5 meta-agent.

MethodID Avg. W-AUCOOD Avg. W-AUC
Meta-TTL0.396 ± 0.0190.414 ± 0.023
w/o BIS (plain mean difference)0.384 ± 0.0260.330 ± 0.040
GEPA0.344 ± 0.0260.380 ± 0.052

Full per-game results with both metrics are reported in Table 15 of the revised appendix.

BIS ablation. Without BIS, ID performance barely changes but OOD performance drops substantially, as selection can be dominated by high-variance tasks.

GEPA with the same proposer. With the same proposer prompt and budget, results from GEPA show lower average performance on both ID and OOD games. This suggests that GEPA, as a generic optimizer, is not well-suited to the meta-learning goal.

Q2: Cross-benchmark transfer and hand-written meta-prompt.

Cross-benchmark transfer. We chose three benchmarks to show that Meta-TTL works across quite different settings (text games, web navigation, and conversational tool use), and did not intend to claim transfer across them. We thank the reviewer for pointing this out, and will narrow our claim to transfer to held-out domains where the task structure is similar but the content is different.

Hand-written meta-prompt of similar length.

We wrote a meta-prompt of similar length by hand. It describes the environment in detail and guides diagnosis, learning from feedback, and exploration (Appendix A.5).

Jericho W-AUC (mean ± std over 5 runs):

MethodDetectiveZork 1TempleID Avg.BalancesLibraryZork 3OOD Avg.
Naive Meta-Agent0.327 ± 0.0470.129 ± 0.0170.189 ± 0.0560.215 ± 0.0220.200 ± 0.0210.409 ± 0.0770.248 ± 0.0460.286 ± 0.023
Hand-Written Meta-Prompt0.670 ± 0.0690.143 ± 0.0130.208 ± 0.0370.340 ± 0.0280.275 ± 0.0340.451 ± 0.1140.378 ± 0.0620.368 ± 0.046
Meta-TTL0.790 ± 0.0260.150 ± 0.0070.248 ± 0.0400.396 ± 0.0190.292 ± 0.0460.522 ± 0.1070.427 ± 0.0180.414 ± 0.023

τ²-bench W-AUC (mean ± std over 3 runs):

MethodAirlineRetailTelecom (OOD)
Naive Meta-Agent0.628 ± 0.0460.547 ± 0.0430.478 ± 0.020
Hand-Written Meta-Prompt0.636 ± 0.0380.526 ± 0.0380.510 ± 0.024
Meta-TTL0.678 ± 0.0380.686 ± 0.0320.554 ± 0.029

Italics mark the stronger baseline in each column.

This suggests that the gains of Meta-TTL are hard to replicate with manual guidance. Results on both metrics are in Appendix G.

We hope these additions address the reviewer's concerns, and we are happy to discuss any remaining questions.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer siWG,

As the discussion period ends soon, we would appreciate it if you could let us know whether our response addresses your concerns. We are happy to clarify anything further. Thank you for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their time and constructive feedback.

We are glad that the reviewers found the framing of optimizing the meta-policy useful (siWG), the learned meta-prompts readable and inspectable (siWG, F5Kw), the Naive Meta-Agent a well-matched control (siWG, F5Kw), and the benchmark coverage broad (F5Kw, VFXi). Below we summarize the main concerns and how we addressed them. All new results are included in the revised PDF.

1. What Meta-TTL trains (VFXi). In our response to Reviewer VFXi, we clarified what Meta-TTL trains. In our setting, an actor interacts with the environment, and the meta-agent rewrites the actor's prompt between episodes. Meta-TTL trains the meta-agent's policy (the meta-prompt), not the actor's prompt. A proposer improves the meta-prompt offline, and the learned meta-prompt is then kept frozen and evaluated on unseen tasks.

2. Positioning and novelty (siWG, F5Kw). We clarified the differences from closely related methods. Promptbreeder and MetaSPO optimize the actor's prompt, while Meta-TTL learns the meta-agent's prompt for unseen tasks. Retroformer trains the meta-agent's weights over a single meta-step, while Meta-TTL is gradient-free and optimizes over the entire meta-trajectory. GEPA selects per-task experts, while Meta-TTL selects for generality at every step. Following the reviewers' suggestions, we revised our positioning:

Our work shows that (1) test-time learning in multi-turn agentic tasks benefits from being meta-learned, and (2) this meta-learning can be done entirely in prompt space. This keeps the learned meta-policy inspectable and practical with long agentic rollouts or closed models.

We will narrow the claims in the abstract, introduction and contributions accordingly. We also do not claim transfer across benchmarks, only to held-out domains with similar task structure but with different content.

3. Comparison with GEPA and ablations (siWG, F5Kw). Jericho W-AUC (mean ± std over 5 runs, Table 15 and Appendix G):

MethodID Avg.OOD Avg.
Meta-TTL0.396 ± 0.0190.414 ± 0.023
w/o BIS (plain mean difference)0.384 ± 0.0260.330 ± 0.040
w/o two-stage reflection0.334 ± 0.0560.381 ± 0.042
GEPA (same proposer and budget)0.344 ± 0.0260.380 ± 0.052
Hand-written meta-prompt (similar length)0.340 ± 0.0280.368 ± 0.046
Naive Meta-Agent0.215 ± 0.0220.286 ± 0.023

Meta-TTL outperforms GEPA with the same proposer and budget on both ID and OOD. Removing either BIS or two-stage reflection lowers OOD performance to GEPA's level or below, suggesting that each design contributes to the gain. The hand-written meta-prompt also falls short of Meta-TTL on Jericho and on all three τ²-bench domains, indicating that the learned meta-policy is hard to replicate with human priors.

4. Generalization and evaluation size (siWG, F5Kw).

  • New Jericho OOD games. We added three games unrelated to the Zork series (Deephome, Pentari, Reverb). Meta-TTL improves over the strongest baseline by 13% to 29%.
  • Larger τ²-bench evaluation. We doubled the evaluation sets and added two seeds for all methods. Meta-TTL achieves the highest W-AUC in all three domains, 8% to 21% above the strongest baseline (Table 11).
  • Variance and splits. Variance is now reported in Appendix E, and task splits in Appendix J.

5. Other clarifications.

  • We clarified how BIS handles edge cases (Eq. (6), Appendix C).
  • We updated references to their peer-reviewed versions and renamed the LLM usage section.

Discussion status. Reviewer F5Kw noted that the additional experiments address several concerns, with novelty as the remaining reservation. We further clarified our contribution in our follow-up response. Reviewers siWG and VFXi have not responded to our rebuttal or reminder. Their main concerns and our responses are summarized below.

  • siWG: positioning relative to Retroformer, MetaSPO and GEPA, and evidence of transfer. We addressed these with a matched GEPA comparison, three new OOD games, a larger τ²-bench evaluation and a hand-written baseline (Sections 2–4).
  • VFXi: the "self-improvement" claim and differing model configurations across tables. We clarified that "Self" refers to the agent system as a whole, and Meta-TTL also works with a shared backbone (Table 4). Each table answers different questions and controls variables within it, and W2 partly stems from the reading that the meta-agent is not trained (Section 1). We also added Best-of-N results.

We would appreciate it if the AC could take these responses into account.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.