acceptodds
Under review as a conference paper at ICLR 2027

RRU: Improving Agent Skill Generation with Reflect-Replay-Update

Abstract

Agent skills extend language agents with workflows and knowledge beyond their pre-trained capabilities. Existing skill generation frameworks adopt a Reflect- then-Update approach: the agent solves a batch of tasks, reflects on the rollouts, and updates the skill to capture recurring patterns and resolve conflicts. This ap- proach can let rollouts from a single batch remove an instruction supported by earlier batches (overwriting) or become an instruction applied to all tasks (over- generalization). These failures arise because each update conditions only on the current batch, as earlier rollouts are often discarded after reflection. We introduce Reflect–Replay–Update (RRU), which improves grounding by maintaining com- pact records of the rollouts that provide evidence supporting each instruction and replaying these rollouts during skill updates. With replay, an update can revise an earlier instruction rather than remove it, and generalize a new instruction only when earlier batches support it. On AppWorld, SpreadsheetBench, and Legal Agent Bench with two open-weight models, RRU outperforms strong baselines (Trace2Skill, SkillOpt, and Combee) by an average of 8.5% on AppWorld and 5.6% on SpreadsheetBench across the two models, and by 3.7% and 1.2% on Le- gal Agent Bench with Qwen and Gemma. Ablations show that replay improves accuracy by 5.0% on average and reduces variation across seeds in three of four settings, and that compact records reach higher accuracy than complete rollouts in three of four settings and use fewer tokens in three of four settings (up to 30%).

Primary area:
foundation or frontier models, including LLMs
Keywords:
Skill Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ThAGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies automatic generation of agent skills (natural-language SKILL.md instruction files) for frozen LLM agents. It identifies two failure modes of the prevailing Reflect-then-Update loop: overwriting (a new batch's update deletes an instruction supported by earlier batches) and over-generalization (a pattern from one batch becomes a global instruction that breaks previously solved tasks), both arising because each update conditions only on the current batch while earlier rollouts are discarded. The proposed Reflect–Replay–Update (RRU) maintains compact rollout records (task, actions, outcome, reflection; ~12% of full rollout size) and has each skill instruction cite a support set of records. Each update: Reflect (structured per-rollout reflections + batch-level recurring patterns), Replay (for each proposed edit, retrieve records of tasks exhibiting the pattern plus the support sets of the instructions it touches, without re-execution), Update (apply edits with records in context, maintain citations). The design is framed as a truth-maintenance system for skills. Evaluation covers AppWorld, SpreadsheetBench, and Legal Agent Bench (LAB) with two open-weight agents (Qwen3.6-35B-A3B, Gemma-4-26B-A4B) against Trace2Skill, SkillOpt, and Combee(ACE), with 3 seeds. RRU reports gains over the best baseline per benchmark; the replay ablation shows +5.0% average accuracy and lower seed variance in 3/4 settings; RRU matches 2 epochs of GRPO with ~8× fewer rollouts, and its skill adds +11.1–14.0% on top of GRPO checkpoints.

Critical strengths

A clearly presented, well-motivated mechanism. The overwriting/over-generalization diagnosis is crisply articulated, the evidence-citation design directly targets it, and the paper is unusually readable, with defined notation, a sensible TMS framing, and 3-seed reporting with standard deviations — better statistical hygiene than most skill-optimization papers. The replay ablation does show a consistent effect direction across the four model×benchmark settings with markedly reduced variance (e.g., AppWorld-Qwen SGC ±10.2 → ±2.5), and the worst-case analysis (the effect is mostly worst-case protection) is honest. This keeps the paper above a clear rejection.

Critical weaknesses
  1. The main comparative claims are not budget-controlled, and much of the gain is not from the proposed mechanism. RRU performs an extra rollout pass over all training tasks with the empty skill before its first update (≈180 rollouts on AppWorld vs ≈90 for the baselines' single epoch; Figure 5 itself labels "72.3% (180 rollouts)") and consumes 1.7–2.5× the tokens of even its own no-replay variant (Table 3). Baselines run "default configurations" (e.g., SkillOpt batches of 40 vs RRU's 16, so update frequency differs), and baseline rollout/token costs are never reported. Decisively, the no-replay variant — RRU's prompts with a plain Reflect-then-Update loop — already matches or beats the best baseline in most settings (66.1 vs 67.4 TGC; 67.2 vs 66.0 SSB on Qwen): roughly half the headline margin over baselines comes from RRU's prompt/harness design rather than the replay mechanism the paper claims as its contribution. Without budget-matched baselines, the main table does not establish that replay, rather than extra compute and prompt engineering, drives the improvement. The motivating failure-mode evidence is also anecdotal — Figures 2/3/6 are single-seed illustrations with no quantification of how often overwriting or over-generalization actually occur in the baselines.
  2. The most novel-looking evidence is compromised by circularity and selection issues. On Legal Agent Bench, the judge is Qwen3.6-35B-A3B — the same model as the updater M_U for all methods and the same as the agent in the Qwen block — and §5.1 states RRU's LAB skills "mainly list what the criteria check," i.e., the pipeline may be learning to mirror the judge's own rubric; no independent judge or human evaluation rules this out. The LAB split is author-constructed (the 96 lowest-cost tasks from a no-skill run — selection toward easy/cheap items), and Gemma's LAB delta (+1.2, stds ~2) is within noise. The RL comparison is similarly fragile: GRPO is a single run and likely undertrained (2 epochs, lr 1e-6, no KL), and the "GRPO + skill" result uses a skill from "a separate RRU run" scoring 78.3% on its own — ~2.8σ above the Table 1 mean of 72.3±2.1 — with no disclosure of whether it is a cherry-picked best seed. There are no significance tests anywhere (n=3 seeds), so the smaller margins (LAB +3.7 Qwen, ≈5 graded outputs out of 144) are not established.
Questions for authors
  1. Were Trace2Skill, SkillOpt, and Combee given matched rollout/token budgets and matched update frequency? Please report per-method training costs (rollouts, tokens, USD) and the main table under equal budgets. This question most affects my assessment.
  2. Given that the no-replay variant already matches the best baselines, how do you attribute the Table 1 gains between replay and prompt/harness design? What would RRU's prompts with a plain update loop score under exactly matched conditions?
  3. On LAB, given judge = M_U = the Qwen agent and criteria-mirroring skills, can you re-grade with an independent judge family or human experts? Please disclose the exact selection procedure for the 96-task subset.
  4. Is the 78.3% skill used in the GRPO+skill experiment a cherry-picked best seed? Please report GRPO+skill with the mean-performing RRU skill, and add significance tests for the main comparisons.
  5. (Suggestion) Quantify the frequency of overwriting/over-generalization in the baselines; add GEPA as a baseline.
5 comments
AuthorsOpenReview ↗

Reply to Reviewer ThAG 1/2

We sincerely thank the reviewer for the careful review. The budget question comes first, as you asked. All numbers are test accuracy in %, Qwen agent unless stated, mean over seeds; differences are computed from unrounded means, except the Table 1 margins, which are differences of the table entries as in the paper. The new runs in this response use a different serving setup from the submission; on it, the Table 1 RRU skills score 71.2 on AppWorld and 72.5 on SpreadsheetBench (72.3 and 73.1 in Table 1), and the model without a skill 56.8 and 59.6 (58.8 and 59.4). Rows marked Table 1, the baseline rows with more epochs and the row without the initial pass are from the submission's runs.

Q1. Equal budgets, training cost and update frequency

In Table 1, RRU uses 180 / 400 rollouts (AppWorld / SpreadsheetBench) and the baselines one epoch. Below, the baselines are given more epochs (n = 3; training rollouts in brackets; SkillOpt's count includes its validation gate):

AppWorld TGCSpreadsheetBench
Trace2Skill, 1 / 2 / 3 epochs62.7 [90] / 55.8 [180] / 68.2 [270]62.6 [200] (1 epoch)
SkillOpt, 1 / 2 / 3 epochs61.4 [120] / 66.0 [268] / 70.3 [416]66.0 [347] / 64.8 [707] / 74.3 [1053]
Combee, 1 epoch67.4 [124]56.5 [275]
RRU without the initial empty-skill rollouts (records and replay kept)70.1 [90]70.3 [200]
RRU (Table 1)72.3 [180]73.1 [400]
  • Baselines with more rollouts. RRU's mean is above every baseline row, including the rows with more rollouts, with one exception: SkillOpt at three epochs on SpreadsheetBench (74.3 with 1,053 rollouts, against 73.1 with 400). In tokens, SkillOpt at three epochs on AppWorld uses 100.3M prompt tokens against RRU's 71.4M, and scores 70.3 against 72.3.
  • RRU with one pass over the training tasks. With the initial pass removed, at 90 / 200 rollouts (no more than one epoch of any baseline), RRU scores 70.1 and 70.3, above every baseline at one epoch.

Training cost and update frequency. The table above matches rollouts. The methods spend a rollout differently (RRU reflects on every rollout and replays records at the update; Trace2Skill updates once per epoch; SkillOpt spends rollouts on its validation gate), so we also report tokens and USD for every row (mean of 3 seeds, agent plus M_U, cumulative over the epochs; at the list prices used in the paper, USD 0.05 / USD 0.70 per million prompt / completion tokens):

AppWorldRolloutsPrompt tokensCompletion tokensUSDTGC
Trace2Skill, 1 epoch9022.7M0.45M1.4562.7
SkillOpt, 1 epoch12030.4M0.62M1.9561.4
Combee, 1 epoch12438.3M0.77M2.4567.4
Trace2Skill, 2 epochs18046.0M0.88M2.9255.8
SkillOpt, 2 epochs26865.5M1.32M4.2066.0
Trace2Skill, 3 epochs27069.4M1.32M4.3968.2
RRU18071.4M1.3M4.4772.3
SkillOpt, 3 epochs416100.3M1.95M6.3870.3
SpreadsheetBenchRolloutsPrompt tokensCompletion tokensUSDAccuracy
Trace2Skill, 1 epoch20025.4M1.07M2.0262.6
Combee, 1 epoch27554.9M2.02M4.1656.5
SkillOpt, 1 epoch34761.3M1.94M4.4366.0
RRU400122.3M3.0M8.2373.1
SkillOpt, 2 epochs707131.0M4.03M9.3764.8
SkillOpt, 3 epochs1053190.5M5.76M13.5674.3

On AppWorld, the baselines at RRU's cost or above (Trace2Skill at three epochs, USD 4.39; SkillOpt at three epochs, USD 6.38) are below RRU. On SpreadsheetBench, SkillOpt at two epochs (USD 9.37) is below RRU (USD 8.23), and at three epochs (USD 13.56) it is above. One epoch of RRU's prompts in the Reflect-then-Update loop (the w/o replay stage ablation of Q2) costs 40.0M / 54.4M prompt tokens and USD 2.51 / USD 3.90.

Combee's rollout count includes one re-attempt of each failed task after reflection; its row is from three further runs of the Table 1 configuration. RRU uses the most tokens per run, mostly in rollouts (61.2M of the 71.4M prompt tokens on AppWorld).

Update frequency. RRU updates after every 16 rollouts of a learning epoch (the initial pass makes no update), SkillOpt after every 40 of the 72 / 160 training tasks left by its 20% validation split (its released default), Combee's curator after every 8, and Trace2Skill once per epoch. We also ran SkillOpt with RRU's batch of 16, from the empty skill (n = 3, tested after its first epoch, 180 / 600 rollouts, on the new serving setup): 63.5 on AppWorld and 66.5 on SpreadsheetBench, against 61.4 and 66.0 at its default batch of 40.

AuthorsOpenReview ↗

Reply to Reviewer ThAG 2/2

Q2. Replay and RRU's prompts, under exactly matched conditions

We first clarify what the "w/o replay stage" row of Table 2 is: RRU's prompts in the Reflect-then-Update loop, which is the setting you ask about. It stores no records and has no initial pass with the empty skill; the written reflections of earlier batches stay readable to the update.

We developed RRU in two steps. We first asked how far the Reflect-then-Update loop could go with prompts alone, and worked on the prompts (Appendix F); the w/o replay stage ablation is the result of this step. We then added records and replay, for the one thing the prompts could not do: check an edit against the rollouts behind an instruction. This is why we report both comparisons: against the best baselines (Table 1), and against our own loop without the replay stage (Table 2).

Under exactly matched conditions, we reran this comparison with 10 seeds per arm instead of 3 and at equal rollouts. The w/o replay stage ablation has no initial pass, so it runs one more epoch (AppWorld, n = 10 per arm, tested after each epoch):

Training rollouts90180270360
RRU74.076.876.9
w/o replay stage: RRU's prompts in the Reflect-then-Update loop72.273.074.673.0

At 360 rollouts RRU scores 76.9 and the w/o replay stage ablation 73.0: with the same prompts and the same rollouts, records and replay add 3.9 points (n = 10 per arm). The difference grows with training (+1.0, +2.2 and +3.9 at 180, 270 and 360 rollouts).

Q3. Legal Agent Bench: an independent judge, and the subset

We regraded the saved Table 1 outputs with a judge from another model family, gpt-oss-120b, with the same rubric, prompt and code, and without re-solving. Both columns are over the same episodes (Qwen agent: two of the three attempts per task, so the Qwen-judge column differs slightly from Table 1; Gemma agent: all episodes). Mean over 3 seeds.

Legal Agent BenchQwen judgegpt-oss-120b
Qwen agent (all-pass rate)
RRU19.412.2
SkillOpt17.011.1
Combee11.28.4
Trace2Skill12.59.7
Gemma agent (criteria met)
RRU60.256.6
SkillOpt59.055.2
Combee55.551.7
Trace2Skill57.353.7

Under the independent judge, RRU remains above the three baselines for both agents (over SkillOpt, the best baseline: +1.0 for the Qwen agent and +1.4 for the Gemma agent).

Selection of the 96 tasks. Legal Agent Bench is expensive to run: a single episode holds about 150k to 170k tokens of context, and in a pilot with the benchmark's own harness 7 of 32 tasks ran past the 262k-token context window of the model. Running all 498 tasks for every method, seed and agent was beyond our budget, so we selected a subset using only runs without a skill. We keep the two memo task kinds (playbook escalation and term negotiation); the other kinds mostly stalled on a tracked-changes script within the turn cap. We rank their tasks by cost in those runs (whether the run stalled, then wall time and output tokens), keep the first 96, and split them 48/48 so that no combination of contract type and task kind appears in both splits.

Q4. RRU and GRPO

The 78.3 skill in Sec. 5.3 is the highest of the three seeds (74.4 / 75.0 / 78.3) of an earlier development version of RRU. As you suggested, we repeat the experiment with the three Table 1 skills, including the mean-performing one (Seed 2). The GRPO checkpoints of the submission were lost, so for this we retrained GRPO on AppWorld with LoRA on Tinker (rank 32, learning rate 1e-5, 8 × 8 rollouts per step for 22 steps, i.e. two epochs and 1,408 rollouts). All cells are evaluated on the same serving setup.

AppWorld TGCNo skill+ Seed 1 skill+ Seed 2 skill+ Seed 3 skillMean with skill
Base model55.167.071.175.071.0
GRPO checkpoint61.967.671.475.371.4

With the mean-performing skill, the GRPO checkpoint goes from 61.9 to 71.4; over the three skills, it also reaches 71.4 on average.

Significance of the main comparisons. For the main comparison, RRU against the w/o replay stage ablation, we now report 10 seeds per arm instead of 3, at equal rollouts (Q2).

Q5. GEPA

GEPA (v0.1.4, default settings, an empty starting skill, n = 3): 55.0 and 63.5 at RRU's budget of 180 / 400 rollouts, and 57.5 and 63.6 at three times that budget. RRU is above GEPA at both budgets on both benchmarks. Under RRU's rollout budget, GEPA returned the empty starting skill in three of the six runs, so we also allowed it three times the budget.

Dear Reviewer ThAG,

We wanted to express our gratitude again to your thoughtful and careful feedback. We hope our new results have strengthened the empirical evidence, and we are eager to learn your further comments.

Best Regards,

The authors

Reviewer_k7iuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper studies how language agents can update natural-language skills without overwriting useful instructions or over-generalizing from recent experience. It proposes Reflect–Replay–Update (RRU), which associates instructions with compact records of their supporting rollouts and revisits those records when editing the skill.

Critical strengths

The paper identifies a practical failure mode in skill maintenance and provides an understandable example of losing a useful conditional instruction.

Critical weaknesses
  1. The closest prior mechanism is not discussed. AutoManual (Chen et al., NeurIPS 2024) already links each rule to the episodes it was induced from via validation records. Its Consolidator also retrieves those trajectories before updating, merging, or deleting rules, and its prompts explicitly guard against discarding rule details and against unscoped rules. This is the paper's central idea. The paper's description is accurate for the frameworks it cites, but its general framing presents retaining and consulting source rollouts as missing from the field. AutoManual does both and is never cited. RRU's remaining contributions appear to be replaying before every update rather than only during consolidation, and the compact record format. These differences are neither stated nor tested.
  2. Key analyses are missing, and the work is hard to reproduce. Only RRU's own token cost is reported. Rollouts, tokens, update counts, and time are not given for the baselines, even though RRU adds a full initial collection pass and uses a different batch size from SkillOpt. Test-time context cost is also not measured. The paper also does not show that iterative learning is needed. The no-replay variant already recovers most of RRU's gain over No skill, and the learned skills mostly encode interface and grader conventions. A short system prompt might recover much of this, but no such reference is evaluated: no generic or hand-written prompt, no one-pass distilled skill, and no prompt-optimization baseline. For reproducibility, no code is released, the prompts are given only as abridged excerpts, and the No skill prompt is not specified.
Questions for authors
  1. How does RRU differ from AutoManual's validation-log-based trajectory retrieval? Can you compare consolidation-only replay with per-operation replay under matched budgets?
  2. What are the rollouts, tokens, update counts, and operation time for each baseline? Does RRU's advantage hold when the initial collection pass and total budget are matched?
  3. What prompt does the No skill agent receive? Does it include the submission and output conventions provided by the official benchmark harnesses?
  4. How much of RRU's gain remains over (i) the top three learned instructions used as a system prompt, (ii) a skill distilled in one pass from the initial rollouts, and (iii) GEPA under the same rollout budget?
  5. The no-replay variant also removes the earlier-attempt content marked † from the reflection prompts (App. D). Can you ablate replay at the update stage alone, and compare against replaying the same number of random or similarity-retrieved records?
  6. How many test-time context tokens does each method's skill add per task?
  7. Can you release the full prompt for reproducibility?
5 comments
AuthorsOpenReview ↗

Reply to reviewer k7iu 1/2

We thank the reviewer for the review. All numbers are test accuracy in %, Qwen agent unless stated, mean over seeds; differences are computed from unrounded means. The new runs in this response use a different serving setup from the submission, so rows are compared within a table. The cost tables of Q2 and the result without the initial pass are from the submission's runs.

Q1. RRU and AutoManual

Thank you for pointing us to AutoManual. We read it carefully. Linking each rule to the episodes it came from is a good idea, and we now cite and discuss it in Sec. 3.

RRU differs from AutoManual in the two ways you name: the compact record, and when replay happens.

The compact record. AutoManual learns online, one episode at a time: after every episode, the Builder revises the rules from that episode's trajectory. RRU follows the batch-level setting of the skill-generation frameworks we compare with (Trace2Skill, SkillOpt, Combee). An update is made from a batch of 16 rollouts and edits several instructions, so one replay brings the support sets of several instructions into context, together with the rollouts of the batch. Rollouts of long-running agents are long, mostly because of the environment observations, so RRU keeps a compact record of each rollout (task, actions, outcome, reflection; observations dropped). A record is 12.6% and 11.4% of the rollout size on AppWorld and SpreadsheetBench (Sec. 4.2; Table 3 compares records with complete rollouts).

When replay happens. In AutoManual, earlier episodes are retrieved by the Consolidator, when the manual grows beyond N_max rules; the Builder's edits after each episode see only that episode. In RRU, replay is part of every update: before an instruction is edited, the records it cites are read.

We also ran the comparison you asked for. On AppWorld, with the same prompts, the same records and the same 180 rollouts (three seeds), RRU scores 72.5, and the variant that replays only at one final consolidation step scores 71.0.

Q2. Cost of every method, and equal budgets

Rollouts, tokens and cost of every method, with the baselines also given more epochs (mean of 3 seeds, agent plus M_U, cumulative over the epochs; at the list prices used in the paper, USD 0.05 / USD 0.70 per million prompt / completion tokens):

AppWorldRolloutsPrompt tokensCompletion tokensUSDTGC
Trace2Skill, 1 epoch9022.7M0.45M1.4562.7
SkillOpt, 1 epoch12030.4M0.62M1.9561.4
Combee, 1 epoch12438.3M0.77M2.4567.4
Trace2Skill, 2 epochs18046.0M0.88M2.9255.8
SkillOpt, 2 epochs26865.5M1.32M4.2066.0
Trace2Skill, 3 epochs27069.4M1.32M4.3968.2
RRU18071.4M1.3M4.4772.3
SkillOpt, 3 epochs416100.3M1.95M6.3870.3
SpreadsheetBenchRolloutsPrompt tokensCompletion tokensUSDAccuracy
Trace2Skill, 1 epoch20025.4M1.07M2.0262.6
Combee, 1 epoch27554.9M2.02M4.1656.5
SkillOpt, 1 epoch34761.3M1.94M4.4366.0
RRU400122.3M3.0M8.2373.1
SkillOpt, 2 epochs707131.0M4.03M9.3764.8
SkillOpt, 3 epochs1053190.5M5.76M13.5674.3

Updates in one epoch (AppWorld / SpreadsheetBench): RRU 6 / 13, SkillOpt 2 / 4, Trace2Skill 1 / 1, Combee 12 / 25 curator rounds. SkillOpt's rollout count includes its validation gate. Combee's rollout count includes one re-attempt of each failed task after reflection; its token row is from three further runs of the Table 1 configuration.

  • Total budget matched. On AppWorld, every baseline row is below RRU, including SkillOpt at three epochs, which uses more rollouts, tokens and USD (416 rollouts, USD 6.38; 70.3 against 72.3). On SpreadsheetBench, every baseline row is below RRU except SkillOpt at three epochs (74.3 with 1,053 rollouts and USD 13.56, against 73.1 with 400 rollouts and USD 8.23).
  • Initial pass matched. With the initial pass removed, RRU uses 90 / 200 rollouts, no more than one epoch of any baseline, and scores 70.1 / 70.3, above every baseline at one epoch.
AuthorsOpenReview ↗

Reply to Reviewer k7iu 2/2

Q3. The No skill prompt

The no-skill agent receives the same task prompt as every method, and this prompt states how to submit on each benchmark; the three prompts are now printed in full in Appendix G. On SpreadsheetBench, the task gives the two fields of the official prompt, instruction_type and answer_position, with the official descriptions verbatim (ending "You only need to modify or fill in values within the cell range specified by answer_position"), as the environments of Trace2Skill and SkillOpt also do, and grading uses the official evaluator after recalculating the workbook, as the official protocol prescribes. On AppWorld, the prompt gives the benchmark's completion call, apis.supervisor.complete_task(), with answer= when the task asks for one, and how to look up the API documentation. On Legal Agent Bench, the agent reads the benchmark's task file, which lists the deliverable file names, and the prompt states that each specific in the task is graded pass/fail.

Q4. Top-three instructions, a one-pass skill, and GEPA

With these conventions in its prompt, the agent without a skill scores 56.8 / 59.6. All rows are on the same serving setup; the skill rows use the same three seeds:

AppWorld TGCSpreadsheetBench
No skill (one run)56.859.6
(iii) GEPA, RRU's rollout budget (180 / 400)55.063.5
(iii) GEPA, three times that budget (540 / 1200)57.563.6
(ii) One-pass skill from the initial rollouts67.767.5
RRU72.571.2

RRU is above the one-pass skill and above GEPA on both benchmarks, also when GEPA is given three times the rollouts. The one-pass skill is written in a single turn from the 90 / 200 rollouts of RRU's initial pass; with more epochs RRU reaches 76.9 on AppWorld (Q5). GEPA is v0.1.4 with its default settings, started from the empty skill, and its budget counts its own validation rollouts. Under RRU's rollout budget, GEPA returned the empty starting skill in three of the six runs, so we also allowed it three times the budget.

(i) We also tested the three sections of the learned skill that cite the most training tasks in place of the full skill: after three epochs on AppWorld they score 69.5, against 75.8 for the full skill (n = 5, the first five of the ten seeds of Q5).

Q5. Replay at the update alone, and random records

RRU replays records in two places: at the update, before each operation is applied (Sec. 4.2.2), and during reflection, where the record of a task's earlier attempt is compared with the new one (marked † in Appendix D). We compare RRU with two ablations:

  • w/o replay stage (the row of Table 2). No records are stored, so nothing is replayed in either place, and there is no initial pass with the empty skill. It is the Reflect-then-Update loop with RRU's prompts.
  • w/o replay at the update (the ablation you asked for). The records, the citations, the initial pass and the replay during reflection (†) are kept. Only the replay at the update is removed: the update and the batch analysis see the records of the current batch only.

We ran all three with 10 seeds and for three epochs (AppWorld, tested after each epoch):

Training rollouts90180270360
RRU74.076.876.9
w/o replay at the update72.775.475.6
w/o replay stage72.273.074.673.0
  • Replay at the update alone. RRU is above the w/o replay at the update ablation by +1.3, +1.5 and +1.4 at 180, 270 and 360 rollouts.
  • Random records. With each update shown randomly drawn records, as many as the skill cites at that point, in place of the cited ones, the score is 70.1 on AppWorld at 180 rollouts (three seeds; RRU 72.5 on the same seeds).
  • Records and replay, at equal rollouts. At 360 rollouts RRU scores 76.9 and the w/o replay stage ablation 73.0: with the same prompts and the same number of rollouts, records and replay add 3.9 points.

Q6. Test-time context tokens

The agent reads the whole SKILL.md before every task. Tokens of SKILL.md, mean of 3 seeds, Qwen agent:

SKILL.md tokensAppWorldSpreadsheetBenchLegal
RRU2,6994,3133,939
SkillOpt1,0606,5861,735
Combee1,1433,604982
Trace2Skill (with its reference files)1,300 (4,930)1,298 (7,094)1,060 (5,272)

RRU's record files are not accessible to the agent at test time (Appendix B).

Q7. Full prompts

Appendix D gives abridged excerpts; the full prompts of the reflection on a rollout, the reflection on a batch, and the update with replay are now in Appendix F, and the task prompt of each benchmark in Appendix G.

Dear Reviewer k7iu,

We wanted to express our gratitude again to your thoughtful and careful feedback. We hope our new results have strengthened the empirical evidence, and we are eager to learn your further comments.

Best Regards,

The authors

Reviewer_WevJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at automatic generation of agent skills (SKILL.md files) from the agent's own rollouts. It argues that current "Reflect-then-Update" frameworks (Trace2Skill, SkillOpt, ACE/Combee) throw rollouts away after reflection, so each update sees only the current batch and the current skill text. The authors name two failure modes that follow from this: overwriting (one batch removes an instruction that earlier batches supported) and over-generalization (one batch's pattern becomes a rule for every task). The proposed method, Reflect-Replay-Update (RRU), stores a compact record of each rollout (task, actions, outcome, reflection, with environment observations dropped), has each instruction cite the records that support it (its "support set"), and puts those support sets back into context when an update operation touches the instruction. The framing borrows from truth maintenance systems.

Experiments cover AppWorld (90 train / 168 test), SpreadsheetBench (200/144) and a 48/48 split of Legal Agent Bench memo tasks. The agents are Qwen3.6-35B-A3B and Gemma-4-26B-A4B-it, and Qwen does reflection and updates in every setting. RRU beats all baselines in Table 1 (e.g., Qwen AppWorld TGC 72.3 vs 67.4 for Combee; SpreadsheetBench 73.1 vs 66.0 for SkillOpt). Removing replay costs 1.5 to 6.4 points and makes seed variance larger in three of four settings (Table 2). Compact records do about as well as full rollouts with fewer tokens in three of four settings (Table 3). One GRPO run per benchmark is outperformed with about 8x fewer rollouts, and adding an RRU skill on top of the GRPO checkpoints helps further (Figure 5).

Critical strengths
  1. The paper is well written. The motivating example (Figure 3) makes the overwriting failure concrete, and the method section is short and precise. Two smaller presentation issues. First, the terminology for the second failure mode is confusing. What the paper calls "over-generalization" (a pattern seen in one batch becoming a rule for all tasks) is, from a learning point of view, overfitting the skill to a single batch. Calling it over-generalization reads as the opposite. I'd suggest explaining how the two terms relate, or renaming it. Second, the notation in Sec. 4.2 is overloaded. E(.) is applied both to an instruction t (E(t), its support set) and to a whole skill (E(S_{k-1}), the union), and the elements of these sets are records rho(tau_x), but E is never applied to a record. Defining E once as a map from instructions to sets of records, and writing the union explicitly, would make Eq. (2) easier to read. Some protocol details that matter for interpreting the comparisons (whether the no-replay ablation also does the initial record-collection pass, how many rollouts and updates each baseline uses) are missing or only implied.

  2. The paper tries to solve a problem which will be encountered much often in the context of using multiple, self-evolving skills.

  3. The replay ablation (Table 2, Table 4) is a convincing evidence in the paper. It keeps the prompts and batches fixed and removes only the record-related parts (marked with a dagger in Appendix D). It shows consistent mean gains (+6.2/+5.9 with Qwen, +6.4/+1.5 with Gemma) and clearly lower seed variance with Qwen (TGC std 7.8 -> 2.1; SpreadsheetBench 5.7 -> 1.9). The per-seed table helps. The observation that the best seeds barely move (75.0 vs 73.5, 75.4 vs 74.7) while the worst seeds improve a lot matches the stated mechanism: replay protects against bad updates rather than raising the ceiling.

Critical weaknesses
  1. The comparisons are not compute- or data-matched, so it is unclear how much of the gain comes from replay rather than extra rollouts and the authors' own pipeline. This matters because the main claim is that replay fixes a general flaw of Reflect-then-Update.

    • Extra rollout pass. Before training, RRU "first collects one rollout per training task with the empty skill" (Sec. 5, App. B). Figure 5 counts 180 rollouts for 90 AppWorld tasks and 400 for 200 SpreadsheetBench tasks, i.e. two per training task.
    • The ablation probably lacks it. Sec. 5.2 says RRU uses 1.7 to 2.5x more tokens than the no-replay variant, "most of them in rollouts, including the initial record collection". This suggests the no-replay variant skips the extra pass, but the paper never says so explicitly.
    • Extra signal, not just records. The reflection prompt asks for a "Verdict against earlier attempts" (pass -> fail, fail -> pass, ...), which the ablation removes (Appendix D). So part of the Table 2 gap may come from seeing each task twice with a before/after comparison, not from conditioning updates on support sets.
    • Baselines at different budgets. Baselines run "with the default configurations of their released implementations" (one epoch; e.g. SkillOpt with batches of 40 vs. b=16 for RRU), and their rollout and token counts are not reported.
    • Pipeline vs. replay. RRU's own no-replay variant already matches or beats the best baseline (66.1 vs 67.4 TGC; 67.2 vs 66.0 SSB), so much of the Table 1 margin comes from prompts, batch size and harness integration.
    • What I'd want: (a) the no-replay variant given the same initial pass and "earlier attempt" information, (b) baselines at matched rollout budgets, and (c) a rollout/token accounting table for every method in Table 1.
  2. Statistical support is thin for several claims, and some evidence is single-run or possibly selected. The paper's central diagnosis, the two failure modes, is also never measured directly.

    • Margins within noise. Everything uses three seeds (population std), and some no-skill cells are a single evaluation (Table 5). Gemma Legal is 60.2 +/- 2.0 vs SkillOpt 59.0 +/- 1.7; Qwen Legal is 19.7 +/- 2.1 vs 16.0 +/- 2.6 on 48 test tasks; Gemma SpreadsheetBench replay gives +1.5 (69.4 +/- 1.0 vs 67.9 +/- 0.3). In Table 4, Gemma AppWorld seed 2 is lower with replay (56.8) than without (58.0). The compact-vs-complete claim (Table 3) is "three of four", averages 1.8 points, and has no variance.
    • Failure modes shown only by anecdote. The motivating curves (Figures 2 and 6) are single-seed runs. Nothing measures how often overwriting or over-generalization happen, e.g. the fraction of updates that delete or broaden a supported instruction, with and without replay.
    • The RL section is the weakest. GRPO is one run per benchmark with a 100k-token context (App. B), while the agent elsewhere uses a 1M YaRN context. The skill added to the GRPO checkpoints comes from "a separate RRU training run" scoring 78.3% on AppWorld alone, higher than any of the three seeds in Table 4 (69.9, 72.0, 75.0). The combination is only 1.8 points above that skill alone on AppWorld (78.3 -> 80.1) and 2.4 on SpreadsheetBench (74.1 -> 76.5). The headline "+11.1-14.0%" is measured against GRPO, which says more about the skill's strength than about complementarity.
    • Legal Agent Bench. The judge is the same model as M_U, the subset is "the 96 tasks with the lowest cost in a run without a skill", and the RRU skills "mainly list what the criteria check". Are the gains better legal drafting, or better fitting to this particular judge?
Questions for authors
  1. Does the "w/o replay stage" variant in Tables 2 and 3 also run the initial empty-skill rollout over all training tasks? If not, can you report a variant that does (with the records available to reflection but not to updates), so that replay is isolated from the extra pass and from the "verdict against earlier attempts" signal?

  2. How many agent rollouts and total tokens (agent plus M_U) does each baseline in Table 1 use, and how many skill updates does each one perform? Can you run at least one strong baseline (SkillOpt or Combee) at a matched rollout budget, e.g. two epochs, or with b=16?

  3. Can you quantify the two failure modes directly? For example, over all update operations, how often is an instruction with a non-empty support set deleted or broadened, and how often does this happen with vs. without replay? Also, how often do replayed records actually change the verdict (the "refines boundary" vs. "breaks rule" verdicts in the Appendix D prompt)?

  4. How was the "separate RRU training run" used in Sec. 5.3 chosen, and why does it score 78.3% when the three Table 4 seeds score 69.9 to 75.0? What does adding each of the three Table 1 skills to the GRPO checkpoints give? Was the GRPO checkpoint evaluated with the same 1M context and turn cap as the skill runs?

  5. Which skill is evaluated at the end of training: the final one, or one selected by test or validation accuracy? Figure 6 shows RRU peaking at 69.4% and ending at 67.4%, so the selection rule matters.

  6. On Legal Agent Bench, does the training feedback f(tau) include the judge's per-criterion verdicts? If so, how much of the improvement survives with a different judge model at test time?

  7. Table 3 shows compact records using 17% more tokens than complete rollouts with Gemma on AppWorld (80.3M vs 68.7M). What explains this? Is the support-set size bounded as the skill grows, and how does replay context scale beyond one epoch of 90 to 200 tasks?

Suggestions (not required for my assessment):

  1. ExpeL (Zhao et al., 2024), which you cite only as memory, keeps a pool of experiences and edits insights with explicit add/edit/upvote/downvote operations based on compared trajectories. It is a close conceptual relative and deserves a direct comparison in the related work. The analogy to rehearsal/experience replay in continual learning (to avoid forgetting) is also natural and worth discussing.

  2. Report gains as percentage points rather than "%". The abstract's "8.5%" is an average absolute difference, not a relative one.

5 comments
AuthorsOpenReview ↗

Reply to Reviewer WeVJ 1/3

We sincerely thank the reviewer for the careful and concrete feedback. The central comparison now uses 10 seeds per arm instead of 3. All numbers are test accuracy in %, Qwen agent unless stated, mean over seeds; differences are computed from unrounded means. The new runs in this response use a different serving setup from the submission, so rows are compared within a table. The cost tables, the baseline rows with more epochs and the result without the initial pass are from the submission's runs.

Q1. The w/o replay stage ablation, and replay alone at equal rollouts

We first clarify the setting. RRU stores a record of every rollout that ends with a submission as a file in the skill folder (Sec. 4.2, Appendix B), and replays records in two places: at the update, before each operation is applied (Sec. 4.2.2), and during reflection, where the record of a task's earlier attempt is compared with the new one (marked † in Appendix D). We compare RRU with two ablations:

  • w/o replay stage (the row of Table 2). No records are stored, so nothing is replayed in either place, and there is no initial pass with the empty skill. It is the Reflect-then-Update loop with RRU's prompts.
  • w/o replay at the update (the ablation you asked for). The records, the citations, the initial pass and the replay during reflection (†) are kept. Only the replay at the update is removed: the update and the batch analysis see the records of the current batch only.

The written reflections of earlier batches stay readable to the update in all three. We ran all three with 10 seeds instead of 3 and at equal rollouts, by letting each method continue for further epochs (AppWorld):

Training rollouts90180270360
RRU (n = 10)74.076.876.9
w/o replay at the update (n = 10)72.775.475.6
w/o replay stage (n = 10)72.273.074.673.0
Trace2Skill (n = 3)62.755.868.2
SkillOpt (n = 3), at 120 / 268 / 416 rollouts61.466.070.3
  • Records and replay, at equal rollouts. The w/o replay stage ablation has no initial pass, so it runs one more epoch. At 360 rollouts RRU scores 76.9 and this ablation 73.0, so with the same prompts and the same number of rollouts, records and replay add 3.9 points. The difference grows with training (+1.0, +2.2 and +3.9 at 180, 270 and 360 rollouts).
  • Replay at the update alone. RRU is above the w/o replay at the update ablation by +1.3, +1.5 and +1.4 at 180, 270 and 360 rollouts.
  • Baselines with more rollouts. RRU after 180 rollouts is above SkillOpt after 416 (its count includes its validation gate) and Trace2Skill after 270.

Pipeline and replay. We developed RRU in two steps. We first asked how far the Reflect-then-Update loop could go with prompts alone, and worked on the prompts (Appendix F); the w/o replay stage ablation is the result of this step. We then added records and replay, for the one thing the prompts could not do: check an edit against the rollouts behind an instruction. This is why we report both comparisons: against the best baselines (Table 1), and against our own loop without the replay stage (Table 2).

AuthorsOpenReview ↗

Reply to Reviewer WeVJ 2/3

Q2. Rollouts, tokens and updates of every method, and matched budgets

Rollouts, tokens and cost of every method, with the baselines also given more epochs (mean of 3 seeds, agent plus M_U, cumulative over the epochs; at the list prices used in the paper, USD 0.05 / USD 0.70 per million prompt / completion tokens):

AppWorldRolloutsPrompt tokensCompletion tokensUSDTGC
Trace2Skill, 1 epoch9022.7M0.45M1.4562.7
SkillOpt, 1 epoch12030.4M0.62M1.9561.4
Combee, 1 epoch12438.3M0.77M2.4567.4
Trace2Skill, 2 epochs18046.0M0.88M2.9255.8
SkillOpt, 2 epochs26865.5M1.32M4.2066.0
Trace2Skill, 3 epochs27069.4M1.32M4.3968.2
RRU18071.4M1.3M4.4772.3
SkillOpt, 3 epochs416100.3M1.95M6.3870.3
SpreadsheetBenchRolloutsPrompt tokensCompletion tokensUSDAccuracy
Trace2Skill, 1 epoch20025.4M1.07M2.0262.6
Combee, 1 epoch27554.9M2.02M4.1656.5
SkillOpt, 1 epoch34761.3M1.94M4.4366.0
RRU400122.3M3.0M8.2373.1
SkillOpt, 2 epochs707131.0M4.03M9.3764.8
SkillOpt, 3 epochs1053190.5M5.76M13.5674.3

Updates in one epoch (AppWorld / SpreadsheetBench): RRU 6 / 13, SkillOpt 2 / 4, Trace2Skill 1 / 1, Combee 12 / 25 curator rounds. SkillOpt's rollout count includes its validation gate. Combee's rollout count includes one re-attempt of each failed task after reflection; its token row is from three further runs of the Table 1 configuration. One epoch of the w/o replay stage ablation uses 90 / 200 rollouts, 40.0M / 54.4M prompt tokens and USD 2.51 / USD 3.90. On Legal Agent Bench RRU uses 96 rollouts and 3 updates, and the baselines one epoch over the 48 training tasks.

  • Baselines at matched or larger budgets. On AppWorld, every baseline row is below RRU, including SkillOpt at three epochs, which uses more rollouts, tokens and USD (416 rollouts, USD 6.38; 70.3 against 72.3). On SpreadsheetBench, every baseline row is below RRU except SkillOpt at three epochs (74.3 with 1,053 rollouts and USD 13.56, against 73.1 with 400 rollouts and USD 8.23).
  • RRU with one pass over the training tasks. With the initial pass removed, RRU uses 90 / 200 rollouts, no more than one epoch of any baseline, and scores 70.1 / 70.3, above every baseline at one epoch.
  • SkillOpt with RRU's batch of 16, as you suggested (from the empty skill, n = 3, tested after its first epoch): 63.5 on AppWorld and 66.5 on SpreadsheetBench, against 61.4 and 66.0 at its default batch of 40.

Q3. Quantifying the two failure modes

We agree that this would be the most direct evidence. We tried to count it and could not do so reliably: whether an edit removed, merged, reworded or broadened an instruction depends on reading its meaning, and without replay there are no records to check an instruction against. We therefore do not report a frequency, and compare with and without replay by test accuracy (Q1).

The part of your question that we can count is how often replayed records change the verdict: in 47% of the cases. We counted every verdict on an existing instruction in the Table 1 runs (Qwen agent, three seeds per benchmark); each verdict is given after that instruction's records are replayed:

Benchmark (3 seeds each)Verdicts on an existing instruction"refines boundary""breaks rule"Share
AppWorld3519054%
SpreadsheetBench8437145%
Legal Agent Bench2812043%
All14768147%

Q4. RRU and GRPO

The 78.3 skill in Sec. 5.3 is the highest of the three seeds (74.4 / 75.0 / 78.3) of an earlier development version of RRU. As you asked, we repeat the experiment with each of the three Table 1 skills. The GRPO checkpoints of the submission were lost, so for this we retrained GRPO on AppWorld with LoRA on Tinker (rank 32, learning rate 1e-5, 8 × 8 rollouts per step for 22 steps, i.e. two epochs and 1,408 rollouts). All cells are evaluated on the same serving setup, with the same 1M context window, 40-turn cap and sampling settings as the skill runs.

AppWorld TGCNo skill+ Seed 1 skill+ Seed 2 skill+ Seed 3 skillMean with skill
Base model55.167.071.175.071.0
GRPO checkpoint61.967.671.475.371.4

Adding a Table 1 skill raises the GRPO checkpoint from 61.9 to 71.4 on average.

Q5. Which skill is evaluated

The final one. Every number is from the skill at the end of training, and nothing is selected by test or validation accuracy. Figure 6 is a separate single-seed run, as its caption states, so its curve is not a Table 1 number.

AuthorsOpenReview ↗

Reply to Reviewer WeVJ 3/3

Q6. Legal Agent Bench and the judge

During training, every method receives the number and titles of the failed criteria; RRU's records also keep the judge's verdict text for each criterion. We regraded the saved Table 1 outputs with a judge from another model family, gpt-oss-120b, with the same rubric, prompt and code, and without re-solving. Both columns are over the same episodes (Qwen agent: two of the three attempts per task, so the Qwen-judge column differs slightly from Table 1; Gemma agent: all episodes). Mean over 3 seeds.

Legal Agent BenchQwen judgegpt-oss-120b
Qwen agent (all-pass rate)
RRU19.412.2
SkillOpt17.011.1
Combee11.28.4
Trace2Skill12.59.7
Gemma agent (criteria met)
RRU60.256.6
SkillOpt59.055.2
Combee55.551.7
Trace2Skill57.353.7

Under the independent judge, RRU remains above the three baselines for both agents (over SkillOpt, the best baseline: +1.0 for the Qwen agent and +1.4 for the Gemma agent).

Selection of the 96 tasks. Running all 498 tasks for every method, seed and agent was beyond our budget (a single episode holds about 150k to 170k tokens of context), so we selected a subset using only runs without a skill. We keep the two memo task kinds (playbook escalation and term negotiation); the other kinds mostly stalled on a tracked-changes script within the turn cap. We rank their tasks by cost in those runs (whether the run stalled, then wall time and output tokens), keep the first 96, and split them 48/48 so that no combination of contract type and task kind appears in both splits.

Q7. Table 3 tokens, and the size of support sets

We have observed that different settings can produce skills that change how many turns the agent spends; for example, some skills ask the model to check more criteria before returning (one of these skills, for Gemma on AppWorld, says: "Self-check: if a task requires aggregation across a library (e.g., 'all artists,' 'all songs,' 'find oldest'), verify the count of items retrieved is plausibly complete"). In the Gemma + AppWorld setting, we see that the skills learned with compact records lead the solver agent to spend 3.3 more turns per rollout than the alternative (average over three seeds). Although we do not see a direct attribution of this to the compact record itself, compact records change the skill, and potentially the agent's behavior; we therefore chose to report two models × two benchmarks, so that we could provide a more rigorous view of the mechanism.

In the current version, we do not impose an upper bound on the support set, so it is bounded by the size of the training set times the number of epochs. Empirically, beyond one epoch, in the three-epoch AppWorld runs (10 seeds), the skill cites on average 38, 52 and 60 of the 90 training tasks after 6, 12 and all 17 updates. We have also observed that some instructions can have quite a large support set (e.g., after three epochs, one instruction on when to pass an answer to complete_task() cites 43 of the 90 training tasks). On the one hand, we could ask the model to note which records are the most important under an instruction, so that updating it takes less context. On the other hand, an instruction with many supporting records intuitively has more evidence than one with few, and a simple heuristic such as a constant upper bound might suffice. We think this could lead to a more principled way of managing the support set in the future.

Other points

Terminology. "Over-generalization" describes the instruction: a pattern seen in one batch is written as a rule for all tasks. From the learning point of view this is, as you say, the skill overfitting to one batch; the two terms name the same event.

Notation. E maps an instruction t to its set of records, E(t). For a skill, E(S) is the union of E(t) over the instructions t in S, which is how Eq. (2) is to be read.

ExpeL and experience replay. Agreed. ExpeL keeps a pool of experiences and edits its insights while comparing trajectories; as far as we can tell, an insight does not record which experiences support it, whereas in RRU each instruction cites its records and an edit replays them. Rehearsal in continual learning replays stored examples while training on new data; RRU replays the stored records of an instruction when that instruction is edited, and is indeed conceptually close. We will add this to the manuscript.

Percentage points. Agreed. All gains here are percentage points, and the abstract's 8.5 is an average absolute difference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.