acceptodds
Under review as a conference paper at ICLR 2027

Learning, Fast and Slow: Towards LLMs That Adapt Continually

23 views

Abstract

Large language models (LLMs) are trained for downstream tasks by updating their parameters (e.g., via RL). However, updating model parameters forces them to absorb task-specific information, which can result in catastrophic forgetting and loss of plasticity. In contrast, in-context learning with fixed LLM parameters can cheaply and rapidly adapt to task-specific requirements (e.g., prompt optimization), but cannot by itself typically match the performance gains available through updating LLM parameters. Inspired by how human learning works at multiple time scales (e.g., System 1 vs. System 2), we propose a fast-slow learning framework for LLMs, instantiated with model parameters as “slow” weights and optimized context as “fast” weights. These fast “weights” can learn from textual feedback to absorb task-specific information, while allowing slow weights to stay closer to the base model and persist general reasoning behaviors. Fast-Slow Training (FST) is up to 3× more sample-efficient than only slow learning (RL) across reasoning tasks, while consistently reaching a higher performance asymptote. Moreover, FST-trained models remain closer to the base LLM (up to 70% less KL divergence), resulting in less catastrophic forgetting than RL training. This reduced drift also preserves plasticity: after training on one task, FST-trained models adapt more effectively to a subsequent task. In continual learning scenarios, where task domains change on the fly, FST continues to acquire each new task while RL stalls.

Primary area:
foundation or frontier models, including LLMs
Keywords:
in-context learning, reinforcement learning, NLP

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ATf5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether jointly adapting textual context and model parameters improves LLM post-training while preserving plasticity for future tasks. The Fast-Slow Training (FST) framework alternates between optimizing a prompt population via GEPA and updating model parameters via reinforcement learning (RL). During RL, responses to the same problem across different prompts share group-relative advantage statistics. The authors report faster learning, higher accuracy (in code, math, and multi-hop retrieval), and lower divergence from the base model. Further experiments explore task transfer, sequential learning, and rubric-based rewards. The core hypothesis is that delegating task-specific adaptation to the textual context reduces the burden on persistent model parameters.

Critical strengths
  1. The interaction between contextual and parametric learning is highly relevant to continual LLM adaptation. The methodology (Section 3, Algorithm 1) is concrete and clearly detailed.

  2. Comparisons against sequential optimization (Appendix B) effectively demonstrate that continuously adapting prompts during training outperforms merely providing a strong initial prompt, even when search budgets are matched.

Critical weaknesses
  1. E-SPL already establishes the core concepts of jointly evolving prompts and parameters, maintaining prompt populations, sampling multiple prompts during RL, and using reflection. A direct empirical comparison with E-SPL on a representative sequential-learning task is necessary.

  2. The results strongly demonstrate faster task acquisition, but not reduced catastrophic forgetting. The evaluation lacks standard continual learning metrics like Forward Transfer (FWT), Backward Transfer (BWT) and average forgetting to substantiate its central claims. While Figures 7 and 22 offer useful data, they leave transfer and retention insufficiently quantified.

Questions for authors
  1. What specific component of FST distinct from E-SPL drives the performance gains, and what evidence isolates this effect?
  2. During continual learning evaluation, how are prompts chosen for previous tasks (e.g., historical task-specific prompts vs. current populations)?
  3. What explains the Math→HoVer-hard collapse? What are the format-validity rates, response-length distributions, and reward trajectories for this run?
5 comments
AuthorsOpenReview ↗

Response to Reviewer ATf5 (1/3)

We thank the reviewer for the careful and constructive review. We are glad the reviewer finds the interaction between contextual and parametric learning useful for continual LLM adaptation, and the method concrete and clearly detailed. Following the review, we ran E-SPL on HoVer-hard and on our continual learning sequence, measured forgetting directly on unrelated benchmarks, computed the standard continual learning metrics, and analyzed the Math → HoVer-hard collapse. We address each concern in detail below.

R3-1. Comparison with E-SPL

We consider E-SPL [4] to be concurrent work. In principle the two methods are very similar, since both train a policy with RL while evolving a population of prompts with LLM reflection. Our work differs in the scope of the experiments, in the design of the prompt optimization, and in the ablations that test each design choice. We cite E-SPL in our related work, and in the revised paper we will discuss it in detail as concurrent work and include the comparison below. The main differences are as follows:

  • Question studied. E-SPL measures final accuracy and generalization from easy to hard problems. We also study forgetting, by measuring how far the weights move from the base model (Figure 5). We study plasticity for later tasks (Figure 6), and continual adaptation, by learning over a sequence of tasks (Figure 7 and Appendix F).

  • Breadth of experiments. E-SPL studies two mixture-of-experts models trained with LoRA, on four tasks in math and agentic search, in three settings: limited training data, generalization from easy to hard problems, and tool use, training for up to 30 steps. Our work studies four dense models trained with full-parameter RL, from 4B to 32B parameters. It covers six tasks, five with verifiable rewards and one with rubric-graded rewards, training up to 2000 steps. Beyond training efficiency, we study forgetting, plasticity, continual learning over two task orders and a six-task stream, scaling to 32B parameters, and tasks without verifiable rewards.

  • How prompts are optimized. E-SPL evolves prompts by mutation and crossover and scores them only by the rewards of the current RL rollouts. FST uses GEPA [1], which scores candidate prompts on a pool of problems and reflects on the text feedback of the reward. E-SPL’s mutation self-reflection uses only problems with both successful and failed rollouts under the selected parent prompt (Appendix A.2), excluding groups in which every sampled rollout fails. GEPA can instead use execution traces and textual evaluation feedback from failed attempts without requiring a successful rollout for the same problem.

  • Prompt population. FST keeps a Pareto set of K prompts, each the best on different problems, so RL rollouts are sampled under several distinct prompts and stay diverse during training (Figure 2). Our ablations show that the population improves accuracy, with K=8 reaching 42.84% against 41.10% for K=1 on CodeIO (Appendix A.1).

  • Advantage baseline. E-SPL computes a baseline for each prompt, while FST computes one baseline per problem across all prompts, so the policy gradient also compares prompts with each other. Our ablations show that the per-problem baseline works better, with 40.65% against 39.30% for the per-prompt baseline on CodeIO (Appendix A.1).

For concreteness, we also compared the two methods empirically, on a single task setting (HoVer-hard) and continual learning setting (HoVer → CodeIO → Physics) as suggested by the reviewer. We used authors’ public code for E-SPL and matched FST wherever possible, using the same Qwen3-8B model, data, 256 rollouts per step, CISPO loss, per-problem baseline and gpt-5.2 as the reflection model. FST reaches the highest accuracy in both settings (Table F). At the end of the continual learning sequence, FST has a mean accuracy of 51.4%, against 46.7% for E-SPL and 44.0% for RL. On HoVer-hard, FST reaches 22.9%, against 20.8% for E-SPL and 19.7% for RL.

Table F. Comparison with E-SPL. Continual learning is the mean validation accuracy over HoVer, CodeIO and Physics at step 600, the end of the sequence. HoVer-hard is the best validation accuracy (mean@4).

\def\arraystretch{1.0}\A{lcc}\hline \T{Method} & \T{Continual learning (mean)} & \T{HoVer-hard} \\\\\hline \T{RL} & 44.0 & 19.7 \\\\ \T{E-SPL} & 46.7 & 20.8 \\\\ \T{FST} & \mathbf{51.4} & \mathbf{22.9} \\\\\hline\Z$$ The gap in the continual learning sequence comes mostly from CodeIO, which E-SPL barely learns (17.9%, against 40.6% for FST). Please refer to Section J (Continual learning comparison with E-SPL), Figure 24, in the Rebuttal section of the revised paper draft for the full learning curves. Also refer to R3-3 where we compare E-SPL on further continual learning metrics. We will add Table F and the extended discussion of E-SPL in the final version of the paper.
AuthorsOpenReview ↗

Response to Reviewer ATf5 (2/3)

R3-2. Which results demonstrate reduced catastrophic forgetting

We agree with the reviewer that our current results mainly show that FST learns new tasks better than RL, which is a gain in plasticity. Our academic-scale RL runs may not capture the full extent of forgetting that can arise during prolonged post-training. For example, an independent evaluation of Composer 2.5 reports substantial losses on math and science benchmarks relative to its Kimi K2.5 base [2]. Shenfeld et al. [3] show that the KL divergence between the fine-tuned and base policy, measured on the new task, reliably predicts the degree of forgetting. We therefore used this KL as a proxy for forgetting in the paper. FST reaches the same reward as RL at a much lower KL from the base model (Figures 5, 16 and 20).

To measure forgetting more directly, we evaluated RL and FST checkpoints on WikiText-2 perplexity and on MMLU, a general knowledge benchmark unrelated to the training task. We took the checkpoints from HoVer-hard training at matched accuracy for RL and FST. We evaluated both checkpoints and the base model without any GEPA prompt (Table E). For MMLU, we used five-shot generative evaluation with greedy decoding and an instruction to return only the answer letter in parentheses; the last parenthesized answer letter was scored.

Table E. Retention of prior abilities after HoVer-hard training (Qwen3-8B, no GEPA prompt at evaluation).

\def\arraystretch{1.0}\A{lccc}\hline \T{Measure} & \T{Base} & \T{RL} & \T{FST} \\\\\hline\A{l}\T{MMLU accuracy, 5-shot generative,}\\\\\T{14,042 questions }(\uparrow)\Z & 66.6 & 65.7 & 66.1 \\\\[3pt] \T{WikiText-2 perplexity }(\downarrow) & 12.735 & 12.854 & 12.769 \\\\[3pt]\A{l}\T{KL from the base model on held-out text,}\\\\\T{10 domains, nats per token }(\downarrow)\Z & 0 & 0.0010\T{ to }0.0122 & 0.0006\T{ to }0.0043 \\\\\hline\Z$$ On five-shot generative MMLU, RL loses 0.88 percentage points relative to the base model and FST loses 0.54 percentage points. FST exceeds RL by 0.34 percentage points (paired 95% CI: [0.09, 0.60], computed from per-question correctness differences over 14,042 questions). This interval measures evaluation uncertainty for these checkpoints, rather than variation across training seeds. On WikiText-2, RL raises perplexity by 0.12 and FST raises it by 0.03. FST has lower KL on all ten measured text domains. We will add these results in the final version. ## R3-3. Standard continual learning metrics Following the reviewer’s suggestion, we computed the standard continual learning metrics for the HoVer → CodeIO → Physics sequence, for RL, E-SPL and FST. Let a(i, j) be the validation accuracy on task j at the end of training stage i, and let b(j) be the base model’s accuracy on task j. We report the following metrics: - **Average accuracy (ACC), backward transfer (BWT) and forward transfer (FWT)**, as defined by Lopez-Paz & Ranzato [5]. BWT is the change in accuracy on earlier tasks after training on later tasks. FWT is the accuracy of a task just before its own stage, relative to the base model. - **Average forgetting**, as defined by Chaudhry et al. [6]. It is the drop from the best accuracy a task reached earlier to its final accuracy. - **Learning accuracy**, as defined by Riemer et al. [7]. It is the mean of a(j, j), the accuracy of each task at the end of its own stage. - **Stage gain**, which is the mean of a(j, j) − a(j−1, j), the accuracy a method adds to a task during that task’s own stage. **Table G.** Continual learning metrics, HoVer → CodeIO → Physics (200 steps per task, values in percentage points). $$\def\A{\begin{array}}\def\Z{\end{array}}\def\T{\text} \def\arraystretch{1.0}\A{lcccccc}\hline \T{Method}&\T{ACC }(\uparrow)&\A{c}\T{Learning}\\\\\T{accuracy }(\uparrow)\Z&\A{c}\T{Stage}\\\\\T{gain }(\uparrow)\Z&\T{BWT }(\uparrow)&\A{c}\T{Average}\\\\\T{forgetting }(\downarrow)\Z&\T{FWT }(\uparrow)\\\\\hline \T{RL}&44.0&44.0&14.8&+0.1&1.0&+2.2\\\\ \T{E-SPL}&46.7&47.2&15.6&-0.8&1.4&\mathbf{+3.6}\\\\ \T{FST}&\mathbf{51.4}&\mathbf{51.1}&\mathbf{23.0}&\mathbf{+0.5}&\mathbf{-0.3}&+0.9\\\\\hline\Z$$ FST has the highest final accuracy, learning accuracy and stage gain, so it learns the most over the sequence. FST also has the best backward transfer and the lowest forgetting, although no method forgets much at this scale of experiments. This is in line with our response in R3-2, where the difference in forgetting shows on benchmarks unrelated to the training tasks (Table E). We will add these metrics and evaluations to the final version of the paper.
AuthorsOpenReview ↗

Response to Reviewer ATf5 (3/3)

R3-4. Prompts used for earlier tasks during continual learning evaluation

We were careful to design FST so that its training is agnostic to the identity of the task and to the task boundary, which represents a more realistic continual learning setting. The prompt population is shared across the task sequence, and GEPA adapts it to whatever task the RL batches come from. At evaluation, we assume access to the task identity and use it to select a prompt produced by FST. Earlier tasks are scored with the prompt selected from the population for that task at the end of the task’s own stage. The current task is scored with the prompt currently selected from the population, and tasks not yet trained are scored without a specialized prompt. We will clarify this protocol in the Appendix of the final version of the paper.

R3-5. The Math → HoVer-hard collapse

Following the reviewer’s request, we reran the plasticity probe from the RL-trained Math checkpoint and recorded the reward, response length and format statistics. The collapse reproduces. Table H compares this run with RL on HoVer-hard starting from Qwen3-8B.

Table H. Training statistics for RL on HoVer-hard, starting from Qwen3-8B and from the RL-trained Math checkpoint.

\small \def\arraystretch{1.0}\A{lcc}\hline \T{Measure} & \T{From Qwen3-8B} & \T{From RL Math checkpoint} \\\\\hline\A{l}\T{Validation accuracy at}\\\\\T{step 0 / 20 / 40 / 60}\Z & 2.6\\,/\\,3.4\\,/\\,8.0\\,/\\,12.1 & 3.2\\,/\\,\mathbf{0.1}\\,/\\,2.8\\,/\\,\mathbf{0.1} \\\\[2pt]\A{l}\T{Median response length at}\\\\\T{step 1 (tokens)}\Z & 562 & 1{,}298 \\\\[2pt]\A{l}\T{Rollouts at the 8,192-token limit,}\\\\\T{step 1 / step 10}\Z & 0\\%\\,/\\,0\\% & 1.6\\%\\,/\\,\mathbf{50\\%} \\\\[2pt]\A{l}\T{Rollouts that finish and contain a query,}\\\\\T{steps 9 to 25}\Z & 100\\% & 43\\%\T{ to }58\\% \\\\\hline\Z$$ Both runs start at about 3% accuracy, but the run from the RL Math checkpoint behaves differently from the start. Its responses are 2.3 times longer, 16% of its rollouts never close the thinking block (0% for Qwen3-8B), and its entropy is 1.1 nats per token against 0.2 to 0.3 for Qwen3-8B. The collapse happens around step 8, where the entropy jumps from 1.1 to 5.6 nats per token. From then on, about half of the rollouts run to the 8,192-token limit and consist of repeated meaningless tokens, and many of the short ones only restate the prompt. With a stricter format check, only 27% to 38% of rollouts in steps 9 to 25 end with a real query, against 97% to 100% for Qwen3-8B. When all rollouts for a problem receive zero reward, they provide no reward-based policy gradient to help the policy recover. The run recovers once, between steps 26 and 42, when a batch happens to contain a few correct rollouts, and it collapses again soon thereafter. Each collapse is accompanied by very large gradient norms before clipping (32.7, 27.9 and 11.3), while the largest norm in the Qwen3-8B run is 0.6. These observations suggest that RL on Math leaves the policy vulnerable to a few large updates when learning a new task with sparse rewards. The FST-trained Math checkpoint does not collapse in the same probe (Figure 6). We will add Table H and the reward and entropy curves to the paper. We thank the reviewer again for the detailed review. We hope these additions address the reviewer's concerns, and we are happy to answer any further questions. If our responses and revisions have resolved your concerns, we would be grateful if you could consider updating your assessment and rating accordingly. ## References [1] Agrawal et al. 2025, GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning [2] Zanichelli & Grietzer. 2026, Coding vs thinking. Paradigm 3. https://www.paradigm3.org/research/composer/ [3] Shenfeld et al. 2026, RL's Razor: Why Online Reinforcement Learning Forgets Less [4] Zhang et al. 2026, Evolutionary System Prompt Learning for Reinforcement Learning in LLMs [5] Lopez-Paz & Ranzato 2017, Gradient Episodic Memory for Continual Learning [6] Chaudhry et al. 2018, Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence [7] Riemer et al. 2019, Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference
Reviewer_yAvUOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes an interesting idea to interleave RL steps and prompt optimization steps. When learning a new task, typically people consider either doing RL or prompt optimization, however this work shows that its efficient to to rapid adaptation in the prompt space and deeper slower changes in the weight space. Overall they show a reduction of 1.4–3× fewer RL steps, and consequently the model needs less weight change which means they can retain more prior knowledge and thus improve catastrophic forgetting.

Critical strengths

I think the idea is creative, currently when we think of optimization methods we focus on running a single method. However, different methods have their strengths and weakness. In particular for FST, RL updates can result in slower but deeper changes to the model via weight updates, but as a consequence the weight updates will forget prior information. On the other hand prompt optimization is fast and requires no changes to the model's weights but its bounded by the model's current capabilities. Interleaving both methods is a clever idea that can get the best of both worlds. The experiments clearly isolate the gain of interleaved optimization.

In addition the investigation into continued adaptation was interesting.

Critical weaknesses

One thing I would make clear is FST doesn't seem to improve retention over RL for catastrophic forgetting. Figure 7 demonstrates that FST can acquire successive tasks while retaining earlier gains. However, because RL also retains the capabilities it successfully acquires, this experiment provides limited evidence that FST reduces forgetting relative to RL.

Questions for authors

Maybe I missed it but could you point out where you demonstrate FST has better catastrophic forgetting compared to RL?

2 comments
AuthorsOpenReview ↗

Response to Reviewer yAvU

We thank the reviewer for the encouraging review. We are glad the reviewer finds the idea creative and the experiments clear, and that the reviewer found the continual adaptation results interesting. We answer the reviewer’s question below.

R2-1. Explaining forgetting results

We agree with the reviewer that our current results mainly show that FST learns new tasks better than RL, which is a gain in plasticity. Our academic-scale RL runs may not capture the full extent of forgetting that can arise during prolonged post-training. For example, an independent evaluation of Composer 2.5 reports substantial losses on math and science benchmarks relative to its Kimi K2.5 base [2]. Shenfeld et al. [3] show that the KL divergence between the fine-tuned and base policy, measured on the new task, reliably predicts the degree of forgetting. We therefore used this KL as a proxy for forgetting in the paper. FST reaches the same reward as RL at a much lower KL from the base model (Figures 5, 16 and 20). We will clarify this properly in the revised version of the paper.

In order to measure forgetting more directly, we evaluated RL and FST checkpoints on WikiText-2 perplexity and on MMLU, a general knowledge benchmark unrelated to the training task. We took the checkpoints from HoVer-hard training at matched accuracy for RL and FST. We evaluated both checkpoints and the base model without any GEPA prompt (Table E). For MMLU, we used five-shot generative evaluation with greedy decoding and an instruction to return only the answer letter in parentheses; the last parenthesized answer letter was scored.

Table E. Retention of prior abilities after HoVer-hard training (Qwen3-8B, no GEPA prompt at evaluation).

MeasureBaseRLFSTMMLU accuracy, 5-shot generative,14,042 questions (↑)66.665.766.1WikiText-2 perplexity (↓)12.73512.85412.769KL from the base model on held-out text,10 domains, nats per token (↓)00.0010 to 0.01220.0006 to 0.0043\def\A{\begin{array}}\def\Z{\end{array}}\def\arraystretch{1.2}\A{lccc}\hline\text{Measure}&\text{Base}&\text{RL}&\text{FST} \\\\\hline\A{l}\text{MMLU accuracy, 5-shot generative,}\\\\\text{14,042 questions }(\uparrow)\Z&66.6&65.7&66.1 \\\\[3pt]\text{WikiText-2 perplexity }(\downarrow)&12.735&12.854&12.769 \\\\[3pt]\A{l}\text{KL from the base model on held-out text,}\\\\\text{10 domains, nats per token }(\downarrow)\Z&0&0.0010\text{ to }0.0122&0.0006\text{ to }0.0043 \\\\\hline\Z

On a five-shot generative MMLU, RL loses 0.88 percentage points relative to the base model and FST loses 0.54 percentage points. FST exceeds RL by 0.34 percentage points (paired 95% CI: [0.09, 0.60], computed from per-question correctness differences over 14,042 questions). On WikiText-2, RL raises perplexity by 0.12 and FST raises it by 0.03. FST has lower KL on all ten measured text domains. We will add these results in the revised version.

We thank the reviewer again for the encouraging review. We hope these additions address the reviewer’s question.

[2] Zanichelli & Grietzer. 2026, Coding vs thinking. Paradigm 3. https://www.paradigm3.org/research/composer/

[3] Shenfeld et al. 2026, RL's Razor: Why Online Reinforcement Learning Forgets Less

Thanks! I look forward to the revised version.

Reviewer_CUsROpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Fast-Slow Training (FST), which interleaves RL updates of LLM parameters (slow weights) with GEPA-based evolution of a population of textual prompts (fast weights). At each cycle, the current prompts are optimized using a frozen reflection model and then mixed within each RL rollout group. Experiments on CodeIO, Polaris math, and HoVer-hard compare FST with RL and several prompt-optimization baselines. The paper reports faster attainment of RL’s peak validation score, fitted higher asymptotes, lower KL drift from the base policy, better transfer to a subsequent task, and stronger performance in continual task streams.

Critical strengths
  1. The prompt population, short update cycle, and shared reward normalization are all well specified. The ablations make a credible case that the gain is not simply from finding one good static prompt.

  2. The evaluation is broad. Besides in-domain curves, the paper looks at OOD behavior without a GEPA prompt, separates prompt and weight contributions, and includes several continual-learning baselines. That is much stronger than reporting one training curve.

Critical weaknesses
  1. FST uses a separate proprietary reflection model, rich textual feedback, and 18-48% extra rollouts per RL step, in addition to reflection calls. Equal optimizer steps or equal training examples therefore do not mean equal tokens, compute, wall-clock time, or external-model cost. The appendix’s matched-peak rollout accounting is useful, but it is not a complete end-to-end comparison for all headline experiments. I would describe the result as fewer RL steps unless total costs are reported consistently.

  2. Several important plots appear to be one training run, without uncertainty across independent runs. This matters for the 1.4x improvement, plasticity and continual-learning claims, and especially the fitted asymptotes. A four-parameter sigmoid extrapolated beyond the observed part of a noisy learning curve is weak evidence that the true ceiling is higher. I would like to see seed-level results and longer observed runs.

Questions for authors
  1. Could you report complete end-to-end costs for each headline experiment: generated tokens, GEPA rollouts, reflection-model tokens/cost, GPU wall-clock, and total compute? How do RL, static prompt optimization, and FST compare at the same total budget?

  2. Could you add independent seeds and checkpoint-level confidence intervals for matched-performance steps, final scores, plasticity, and continual streams?

2 comments
AuthorsOpenReview ↗

Response to Reviewer CUsR (1/2)

We thank the reviewer for the detailed review. We are encouraged that the reviewer finds the method and design choices well specified, our evaluation including OOD analysis broad and ablations supporting a credible case that the gain does not come from one good static prompt. In response to the concerns raised, we added an end-to-end cost accounting, a run of FST with the policy as its own reflection model and a result across three seeds. We answer each question below.

R1-1. Accounting for end-to-end cost

In the paper, we focus more on data-efficiency. We agree with the reviewer that total compute, measured in rollouts, and wall-clock time are equally important for judging the method. Appendix C.2 of the paper compares the methods by total rollouts, following the reviewer’s suggestion, we extend this comparison in Table A with a full breakdown of rollout efficiency and wall-clock time. It reports the rollouts per step, including the rollouts GEPA generates to score candidate prompts, the reflection calls and their cost, and the wall-clock time per step. It also reports the total rollouts and wall-clock time each method needs to reach RL’s peak accuracy.

Table A. Rollout, reflection and wall-clock cost of FST vs RL. Rollouts, reflection calls and reflection cost are averaged per RL step over a run; a GEPA cycle runs once every 6 steps. We use the same throughput for GEPA rollouts as that of RL rollouts.

$ \def\A{\begin{array}} \def\Z{\end{array}} \normalsize \A{lccccc} \hline \A{l}\text{Task}\\\phantom{\text{(overhead)}}\\\phantom{\text{step}}\\\hline\text{CodeIO}\\\text{HoVer-hard}\\\text{Math (Polaris)}\Z &\A{c}\text{Rollouts / step}\\\A{cc}\hline\text{RL}&\A{c}\text{GEPA}\\\text{(overhead)}\Z\\\hline256&116\ (45\%)\\256&123\ (48\%)\\512&92\ (18\%)\Z\Z &\A{c}\text{Reflection}\\\A{cc}\hline\A{c}\text{calls /}\\\text{step}\Z&\A{c}\text{cost /}\\\text{step}\Z\\\hline0.96&\0.04\\\\2.72&\\0.07\\1.70&\0.09\\Z\\Z &\\A{c}\\text{Wall-clock / step}\\\\\\A{cc}\\hline\\A{c}\\text{RL}\\\\\\phantom{\\text{step}}\\Z&\\A{c}\\text{FST}\\\\\\phantom{\\text{step}}\\Z\\\\\\hline163\\text{ s}&205\\text{ s}\\\\44\\text{ s}&111\\text{ s}\\\\209\\text{ s}&272\\text{ s}\\Z\\Z &\\A{c}\\text{Step}\\\\\\text{reduction}\\\\\\phantom{\\text{step}}\\\\\\hline3.0\\times\\\\3.0\\times\\\\1.4\\times\\Z &\\A{c}\\text{At matched accuracy}\\\\\\A{cc}\\hline\\A{c}\\text{rollouts}\\\\\\phantom{\\text{step}}\\Z&\\A{c}\\text{wall-clock}\\\\\\phantom{\\text{step}}\\Z\\\\\\hline-51\\%&-58\\%\\\\-51\\%&-16\\%\\\\-16\\%&-7\\%\\Z\\Z\\\\ \\hline \\Z \

Although FST costs more per step than RL because of the GEPA overhead, it needs 1.4 to 3.0 times fewer steps, so it reaches RL’s peak accuracy with 16% to 51% fewer rollouts and 7% to 58% less wall-clock time. Our current implementation is not optimized for wall-clock time. In Appendix C.1 and Appendix C.2, we describe optimizations that reduce wall-clock time without changing the method: reusing GEPA’s rollouts as RL rollouts, and running GEPA in the background while RL trains. Table B estimates the wall-clock time with these optimizations.

Table B. Estimated wall-clock time for FST to reach RL's peak validation accuracy with an optimized implementation, relative to RL.

\begin{array}{lccc} \hline \text{Task} & \text{As in Table A} & \begin{array}{c}\text{+ GEPA in}\\\\\text{background}\end{array} & \text{+ rollout reuse} \\\\ \hline \text{CodeIO} & -58\\% & -60\\% & -64\\% \\\\ \text{HoVer-hard} & -16\\% & -36\\% & -51\\% \\\\ \text{Math (Polaris)} & -7\\% & -16\\% & -24\\% \\\\ \hline \end{array}$$ With both optimizations, FST reaches RL’s peak accuracy with 24% to 64% less wall-clock time than RL. We will add Table A, Table B and the description of these optimizations to the revised Appendix C.2. ## R1-2. Use of a proprietary reflection model For the main results, we use GPT-5.2 as the reflection model. However, the GEPA paper [1] showed reflective prompt optimization works well both with frontier models and with Qwen3-8B. For completeness, we ran FST with the policy as its own reflection model: **Table C.** Best HoVer-hard validation accuracy (mean@4) of FST with gpt-5.2 and with the policy itself as the reflection model. $$\def\arraystretch{1.2} \begin{array}{llc} \hline \text{Method} & \text{Reflection model} & \text{HoVer-hard accuracy} \\\\ \hline \text{RL} & \text{none} & 19.7 \\\\ \text{FST} & \text{gpt-5.2} & 22.9 \\\\ \text{FST} & \text{Qwen3-8B (the policy itself)} & 22.5 \\\\ \hline \end{array}$$ FST with self-reflection reaches 22.5%, close to FST with gpt-5.2 (22.9%) and 2.8 points above RL (19.7%). The gain therefore does not depend on a proprietary model. We will add Table C to the paper. [1] Agrawal et al. 2025, GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning
AuthorsOpenReview ↗

Response to Reviewer CUsR (2/2)

R1-3. Independent seeds and confidence intervals

We agree with the reviewer that the bands in the paper are rollout-level (mean@k) and not over independent training runs. As per the reviewer’s suggestion, we report results across seeds for the paper’s HoVer-hard configuration, with three seeds per method (Table D).

Table D. Best HoVer-hard validation accuracy (mean@4) within 300 steps across seeds.

MethodSeedsAccuracy per seedMean±stdRL319.9,19.7,19.519.7±0.2FST322.9,21.5,22.322.2±0.7\def\arraystretch{1.2}\begin{array}{lccc}\hline\text{Method}&\text{Seeds}&\text{Accuracy per seed}&\text{Mean}\pm\text{std} \\\\\hline\text{RL}&3&19.9,\\ 19.7,\\ 19.5&19.7\pm0.2 \\\\\text{FST}&3&22.9,\\ 21.5,\\ 22.3&22.2\pm0.7 \\\\\hline\end{array}

FST is 2.5 points above RL, and the 95% confidence interval of this difference (0.9 to 4.1 points) excludes zero. The lowest FST seed (21.5%) is also above the highest RL seed (19.9%).

We would also like to note while many individual runs are single seed, we cover experimental variance through the diversity of settings and thorough ablations. This includes:

  • four verifiable training tasks (CodeIO, Math, HoVer-hard and Physics), a rubric-scored medical task (Appendix E) and a synthetic star-graph search task (Section 5);
  • three additional models (Qwen3-32B, OLMo-3-7B and Qwen3-4B, Appendix D.2 and E);
  • two continual learning orderings (Section 4 and Appendix F.2) and a six-task continual learning stream (Appendix F.3).
  • ablations on hyperparameters (GEPA frequency, prompt pool size, advantage baseline, reflection budget) on HoVer-hard and CodeIO (Appendix A)

While these runs do not replace seeds, together they give many independent observations of the same effect. We will report seed-level confidence intervals for the additional results in the revised version of the paper as in Table D.

We thank the reviewer again for the constructive review. We hope these additions address the reviewer's concerns, and we are happy to answer any further questions or add any further comparison the reviewer would find decisive.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.