- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether jointly adapting textual context and model parameters improves LLM post-training while preserving plasticity for future tasks. The Fast-Slow Training (FST) framework alternates between optimizing a prompt population via GEPA and updating model parameters via reinforcement learning (RL). During RL, responses to the same problem across different prompts share group-relative advantage statistics. The authors report faster learning, higher accuracy (in code, math, and multi-hop retrieval), and lower divergence from the base model. Further experiments explore task transfer, sequential learning, and rubric-based rewards. The core hypothesis is that delegating task-specific adaptation to the textual context reduces the burden on persistent model parameters.
Critical strengths
-
The interaction between contextual and parametric learning is highly relevant to continual LLM adaptation. The methodology (Section 3, Algorithm 1) is concrete and clearly detailed.
-
Comparisons against sequential optimization (Appendix B) effectively demonstrate that continuously adapting prompts during training outperforms merely providing a strong initial prompt, even when search budgets are matched.
Critical weaknesses
-
E-SPL already establishes the core concepts of jointly evolving prompts and parameters, maintaining prompt populations, sampling multiple prompts during RL, and using reflection. A direct empirical comparison with E-SPL on a representative sequential-learning task is necessary.
-
The results strongly demonstrate faster task acquisition, but not reduced catastrophic forgetting. The evaluation lacks standard continual learning metrics like Forward Transfer (FWT), Backward Transfer (BWT) and average forgetting to substantiate its central claims. While Figures 7 and 22 offer useful data, they leave transfer and retention insufficiently quantified.
Questions for authors
- What specific component of FST distinct from E-SPL drives the performance gains, and what evidence isolates this effect?
- During continual learning evaluation, how are prompts chosen for previous tasks (e.g., historical task-specific prompts vs. current populations)?
- What explains the Math→HoVer-hard collapse? What are the format-validity rates, response-length distributions, and reward trajectories for this run?
5 comments
Response to Reviewer ATf5 (2/3)
R3-2. Which results demonstrate reduced catastrophic forgetting
We agree with the reviewer that our current results mainly show that FST learns new tasks better than RL, which is a gain in plasticity. Our academic-scale RL runs may not capture the full extent of forgetting that can arise during prolonged post-training. For example, an independent evaluation of Composer 2.5 reports substantial losses on math and science benchmarks relative to its Kimi K2.5 base [2]. Shenfeld et al. [3] show that the KL divergence between the fine-tuned and base policy, measured on the new task, reliably predicts the degree of forgetting. We therefore used this KL as a proxy for forgetting in the paper. FST reaches the same reward as RL at a much lower KL from the base model (Figures 5, 16 and 20).
To measure forgetting more directly, we evaluated RL and FST checkpoints on WikiText-2 perplexity and on MMLU, a general knowledge benchmark unrelated to the training task. We took the checkpoints from HoVer-hard training at matched accuracy for RL and FST. We evaluated both checkpoints and the base model without any GEPA prompt (Table E). For MMLU, we used five-shot generative evaluation with greedy decoding and an instruction to return only the answer letter in parentheses; the last parenthesized answer letter was scored.
Table E. Retention of prior abilities after HoVer-hard training (Qwen3-8B, no GEPA prompt at evaluation).
\def\arraystretch{1.0}\A{lccc}\hline \T{Measure} & \T{Base} & \T{RL} & \T{FST} \\\\\hline\A{l}\T{MMLU accuracy, 5-shot generative,}\\\\\T{14,042 questions }(\uparrow)\Z & 66.6 & 65.7 & 66.1 \\\\[3pt] \T{WikiText-2 perplexity }(\downarrow) & 12.735 & 12.854 & 12.769 \\\\[3pt]\A{l}\T{KL from the base model on held-out text,}\\\\\T{10 domains, nats per token }(\downarrow)\Z & 0 & 0.0010\T{ to }0.0122 & 0.0006\T{ to }0.0043 \\\\\hline\Z$$ On five-shot generative MMLU, RL loses 0.88 percentage points relative to the base model and FST loses 0.54 percentage points. FST exceeds RL by 0.34 percentage points (paired 95% CI: [0.09, 0.60], computed from per-question correctness differences over 14,042 questions). This interval measures evaluation uncertainty for these checkpoints, rather than variation across training seeds. On WikiText-2, RL raises perplexity by 0.12 and FST raises it by 0.03. FST has lower KL on all ten measured text domains. We will add these results in the final version. ## R3-3. Standard continual learning metrics Following the reviewer’s suggestion, we computed the standard continual learning metrics for the HoVer → CodeIO → Physics sequence, for RL, E-SPL and FST. Let a(i, j) be the validation accuracy on task j at the end of training stage i, and let b(j) be the base model’s accuracy on task j. We report the following metrics: - **Average accuracy (ACC), backward transfer (BWT) and forward transfer (FWT)**, as defined by Lopez-Paz & Ranzato [5]. BWT is the change in accuracy on earlier tasks after training on later tasks. FWT is the accuracy of a task just before its own stage, relative to the base model. - **Average forgetting**, as defined by Chaudhry et al. [6]. It is the drop from the best accuracy a task reached earlier to its final accuracy. - **Learning accuracy**, as defined by Riemer et al. [7]. It is the mean of a(j, j), the accuracy of each task at the end of its own stage. - **Stage gain**, which is the mean of a(j, j) − a(j−1, j), the accuracy a method adds to a task during that task’s own stage. **Table G.** Continual learning metrics, HoVer → CodeIO → Physics (200 steps per task, values in percentage points). $$\def\A{\begin{array}}\def\Z{\end{array}}\def\T{\text} \def\arraystretch{1.0}\A{lcccccc}\hline \T{Method}&\T{ACC }(\uparrow)&\A{c}\T{Learning}\\\\\T{accuracy }(\uparrow)\Z&\A{c}\T{Stage}\\\\\T{gain }(\uparrow)\Z&\T{BWT }(\uparrow)&\A{c}\T{Average}\\\\\T{forgetting }(\downarrow)\Z&\T{FWT }(\uparrow)\\\\\hline \T{RL}&44.0&44.0&14.8&+0.1&1.0&+2.2\\\\ \T{E-SPL}&46.7&47.2&15.6&-0.8&1.4&\mathbf{+3.6}\\\\ \T{FST}&\mathbf{51.4}&\mathbf{51.1}&\mathbf{23.0}&\mathbf{+0.5}&\mathbf{-0.3}&+0.9\\\\\hline\Z$$ FST has the highest final accuracy, learning accuracy and stage gain, so it learns the most over the sequence. FST also has the best backward transfer and the lowest forgetting, although no method forgets much at this scale of experiments. This is in line with our response in R3-2, where the difference in forgetting shows on benchmarks unrelated to the training tasks (Table E). We will add these metrics and evaluations to the final version of the paper.Response to Reviewer ATf5 (3/3)
R3-4. Prompts used for earlier tasks during continual learning evaluation
We were careful to design FST so that its training is agnostic to the identity of the task and to the task boundary, which represents a more realistic continual learning setting. The prompt population is shared across the task sequence, and GEPA adapts it to whatever task the RL batches come from. At evaluation, we assume access to the task identity and use it to select a prompt produced by FST. Earlier tasks are scored with the prompt selected from the population for that task at the end of the task’s own stage. The current task is scored with the prompt currently selected from the population, and tasks not yet trained are scored without a specialized prompt. We will clarify this protocol in the Appendix of the final version of the paper.
R3-5. The Math → HoVer-hard collapse
Following the reviewer’s request, we reran the plasticity probe from the RL-trained Math checkpoint and recorded the reward, response length and format statistics. The collapse reproduces. Table H compares this run with RL on HoVer-hard starting from Qwen3-8B.
Table H. Training statistics for RL on HoVer-hard, starting from Qwen3-8B and from the RL-trained Math checkpoint.
\small \def\arraystretch{1.0}\A{lcc}\hline \T{Measure} & \T{From Qwen3-8B} & \T{From RL Math checkpoint} \\\\\hline\A{l}\T{Validation accuracy at}\\\\\T{step 0 / 20 / 40 / 60}\Z & 2.6\\,/\\,3.4\\,/\\,8.0\\,/\\,12.1 & 3.2\\,/\\,\mathbf{0.1}\\,/\\,2.8\\,/\\,\mathbf{0.1} \\\\[2pt]\A{l}\T{Median response length at}\\\\\T{step 1 (tokens)}\Z & 562 & 1{,}298 \\\\[2pt]\A{l}\T{Rollouts at the 8,192-token limit,}\\\\\T{step 1 / step 10}\Z & 0\\%\\,/\\,0\\% & 1.6\\%\\,/\\,\mathbf{50\\%} \\\\[2pt]\A{l}\T{Rollouts that finish and contain a query,}\\\\\T{steps 9 to 25}\Z & 100\\% & 43\\%\T{ to }58\\% \\\\\hline\Z$$ Both runs start at about 3% accuracy, but the run from the RL Math checkpoint behaves differently from the start. Its responses are 2.3 times longer, 16% of its rollouts never close the thinking block (0% for Qwen3-8B), and its entropy is 1.1 nats per token against 0.2 to 0.3 for Qwen3-8B. The collapse happens around step 8, where the entropy jumps from 1.1 to 5.6 nats per token. From then on, about half of the rollouts run to the 8,192-token limit and consist of repeated meaningless tokens, and many of the short ones only restate the prompt. With a stricter format check, only 27% to 38% of rollouts in steps 9 to 25 end with a real query, against 97% to 100% for Qwen3-8B. When all rollouts for a problem receive zero reward, they provide no reward-based policy gradient to help the policy recover. The run recovers once, between steps 26 and 42, when a batch happens to contain a few correct rollouts, and it collapses again soon thereafter. Each collapse is accompanied by very large gradient norms before clipping (32.7, 27.9 and 11.3), while the largest norm in the Qwen3-8B run is 0.6. These observations suggest that RL on Math leaves the policy vulnerable to a few large updates when learning a new task with sparse rewards. The FST-trained Math checkpoint does not collapse in the same probe (Figure 6). We will add Table H and the reward and entropy curves to the paper. We thank the reviewer again for the detailed review. We hope these additions address the reviewer's concerns, and we are happy to answer any further questions. If our responses and revisions have resolved your concerns, we would be grateful if you could consider updating your assessment and rating accordingly. ## References [1] Agrawal et al. 2025, GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning [2] Zanichelli & Grietzer. 2026, Coding vs thinking. Paradigm 3. https://www.paradigm3.org/research/composer/ [3] Shenfeld et al. 2026, RL's Razor: Why Online Reinforcement Learning Forgets Less [4] Zhang et al. 2026, Evolutionary System Prompt Learning for Reinforcement Learning in LLMs [5] Lopez-Paz & Ranzato 2017, Gradient Episodic Memory for Continual Learning [6] Chaudhry et al. 2018, Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence [7] Riemer et al. 2019, Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference
Response to Reviewer ATf5 (1/3)
We thank the reviewer for the careful and constructive review. We are glad the reviewer finds the interaction between contextual and parametric learning useful for continual LLM adaptation, and the method concrete and clearly detailed. Following the review, we ran E-SPL on HoVer-hard and on our continual learning sequence, measured forgetting directly on unrelated benchmarks, computed the standard continual learning metrics, and analyzed the Math → HoVer-hard collapse. We address each concern in detail below.
R3-1. Comparison with E-SPL
We consider E-SPL [4] to be concurrent work. In principle the two methods are very similar, since both train a policy with RL while evolving a population of prompts with LLM reflection. Our work differs in the scope of the experiments, in the design of the prompt optimization, and in the ablations that test each design choice. We cite E-SPL in our related work, and in the revised paper we will discuss it in detail as concurrent work and include the comparison below. The main differences are as follows:
Question studied. E-SPL measures final accuracy and generalization from easy to hard problems. We also study forgetting, by measuring how far the weights move from the base model (Figure 5). We study plasticity for later tasks (Figure 6), and continual adaptation, by learning over a sequence of tasks (Figure 7 and Appendix F).
Breadth of experiments. E-SPL studies two mixture-of-experts models trained with LoRA, on four tasks in math and agentic search, in three settings: limited training data, generalization from easy to hard problems, and tool use, training for up to 30 steps. Our work studies four dense models trained with full-parameter RL, from 4B to 32B parameters. It covers six tasks, five with verifiable rewards and one with rubric-graded rewards, training up to 2000 steps. Beyond training efficiency, we study forgetting, plasticity, continual learning over two task orders and a six-task stream, scaling to 32B parameters, and tasks without verifiable rewards.
How prompts are optimized. E-SPL evolves prompts by mutation and crossover and scores them only by the rewards of the current RL rollouts. FST uses GEPA [1], which scores candidate prompts on a pool of problems and reflects on the text feedback of the reward. E-SPL’s mutation self-reflection uses only problems with both successful and failed rollouts under the selected parent prompt (Appendix A.2), excluding groups in which every sampled rollout fails. GEPA can instead use execution traces and textual evaluation feedback from failed attempts without requiring a successful rollout for the same problem.
Prompt population. FST keeps a Pareto set of K prompts, each the best on different problems, so RL rollouts are sampled under several distinct prompts and stay diverse during training (Figure 2). Our ablations show that the population improves accuracy, with K=8 reaching 42.84% against 41.10% for K=1 on CodeIO (Appendix A.1).
Advantage baseline. E-SPL computes a baseline for each prompt, while FST computes one baseline per problem across all prompts, so the policy gradient also compares prompts with each other. Our ablations show that the per-problem baseline works better, with 40.65% against 39.30% for the per-prompt baseline on CodeIO (Appendix A.1).
For concreteness, we also compared the two methods empirically, on a single task setting (HoVer-hard) and continual learning setting (HoVer → CodeIO → Physics) as suggested by the reviewer. We used authors’ public code for E-SPL and matched FST wherever possible, using the same Qwen3-8B model, data, 256 rollouts per step, CISPO loss, per-problem baseline and gpt-5.2 as the reflection model. FST reaches the highest accuracy in both settings (Table F). At the end of the continual learning sequence, FST has a mean accuracy of 51.4%, against 46.7% for E-SPL and 44.0% for RL. On HoVer-hard, FST reaches 22.9%, against 20.8% for E-SPL and 19.7% for RL.
Table F. Comparison with E-SPL. Continual learning is the mean validation accuracy over HoVer, CodeIO and Physics at step 600, the end of the sequence. HoVer-hard is the best validation accuracy (mean@4).
\def\arraystretch{1.0}\A{lcc}\hline \T{Method} & \T{Continual learning (mean)} & \T{HoVer-hard} \\\\\hline \T{RL} & 44.0 & 19.7 \\\\ \T{E-SPL} & 46.7 & 20.8 \\\\ \T{FST} & \mathbf{51.4} & \mathbf{22.9} \\\\\hline\Z$$ The gap in the continual learning sequence comes mostly from CodeIO, which E-SPL barely learns (17.9%, against 40.6% for FST). Please refer to Section J (Continual learning comparison with E-SPL), Figure 24, in the Rebuttal section of the revised paper draft for the full learning curves. Also refer to R3-3 where we compare E-SPL on further continual learning metrics. We will add Table F and the extended discussion of E-SPL in the final version of the paper.