- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies automatic generation of agent skills (natural-language SKILL.md instruction files) for frozen LLM agents. It identifies two failure modes of the prevailing Reflect-then-Update loop: overwriting (a new batch's update deletes an instruction supported by earlier batches) and over-generalization (a pattern from one batch becomes a global instruction that breaks previously solved tasks), both arising because each update conditions only on the current batch while earlier rollouts are discarded. The proposed Reflect–Replay–Update (RRU) maintains compact rollout records (task, actions, outcome, reflection; ~12% of full rollout size) and has each skill instruction cite a support set of records. Each update: Reflect (structured per-rollout reflections + batch-level recurring patterns), Replay (for each proposed edit, retrieve records of tasks exhibiting the pattern plus the support sets of the instructions it touches, without re-execution), Update (apply edits with records in context, maintain citations). The design is framed as a truth-maintenance system for skills. Evaluation covers AppWorld, SpreadsheetBench, and Legal Agent Bench (LAB) with two open-weight agents (Qwen3.6-35B-A3B, Gemma-4-26B-A4B) against Trace2Skill, SkillOpt, and Combee(ACE), with 3 seeds. RRU reports gains over the best baseline per benchmark; the replay ablation shows +5.0% average accuracy and lower seed variance in 3/4 settings; RRU matches 2 epochs of GRPO with ~8× fewer rollouts, and its skill adds +11.1–14.0% on top of GRPO checkpoints.
Critical strengths
A clearly presented, well-motivated mechanism. The overwriting/over-generalization diagnosis is crisply articulated, the evidence-citation design directly targets it, and the paper is unusually readable, with defined notation, a sensible TMS framing, and 3-seed reporting with standard deviations — better statistical hygiene than most skill-optimization papers. The replay ablation does show a consistent effect direction across the four model×benchmark settings with markedly reduced variance (e.g., AppWorld-Qwen SGC ±10.2 → ±2.5), and the worst-case analysis (the effect is mostly worst-case protection) is honest. This keeps the paper above a clear rejection.
Critical weaknesses
- The main comparative claims are not budget-controlled, and much of the gain is not from the proposed mechanism. RRU performs an extra rollout pass over all training tasks with the empty skill before its first update (≈180 rollouts on AppWorld vs ≈90 for the baselines' single epoch; Figure 5 itself labels "72.3% (180 rollouts)") and consumes 1.7–2.5× the tokens of even its own no-replay variant (Table 3). Baselines run "default configurations" (e.g., SkillOpt batches of 40 vs RRU's 16, so update frequency differs), and baseline rollout/token costs are never reported. Decisively, the no-replay variant — RRU's prompts with a plain Reflect-then-Update loop — already matches or beats the best baseline in most settings (66.1 vs 67.4 TGC; 67.2 vs 66.0 SSB on Qwen): roughly half the headline margin over baselines comes from RRU's prompt/harness design rather than the replay mechanism the paper claims as its contribution. Without budget-matched baselines, the main table does not establish that replay, rather than extra compute and prompt engineering, drives the improvement. The motivating failure-mode evidence is also anecdotal — Figures 2/3/6 are single-seed illustrations with no quantification of how often overwriting or over-generalization actually occur in the baselines.
- The most novel-looking evidence is compromised by circularity and selection issues. On Legal Agent Bench, the judge is Qwen3.6-35B-A3B — the same model as the updater M_U for all methods and the same as the agent in the Qwen block — and §5.1 states RRU's LAB skills "mainly list what the criteria check," i.e., the pipeline may be learning to mirror the judge's own rubric; no independent judge or human evaluation rules this out. The LAB split is author-constructed (the 96 lowest-cost tasks from a no-skill run — selection toward easy/cheap items), and Gemma's LAB delta (+1.2, stds ~2) is within noise. The RL comparison is similarly fragile: GRPO is a single run and likely undertrained (2 epochs, lr 1e-6, no KL), and the "GRPO + skill" result uses a skill from "a separate RRU run" scoring 78.3% on its own — ~2.8σ above the Table 1 mean of 72.3±2.1 — with no disclosure of whether it is a cherry-picked best seed. There are no significance tests anywhere (n=3 seeds), so the smaller margins (LAB +3.7 Qwen, ≈5 graded outputs out of 144) are not established.
Questions for authors
- Were Trace2Skill, SkillOpt, and Combee given matched rollout/token budgets and matched update frequency? Please report per-method training costs (rollouts, tokens, USD) and the main table under equal budgets. This question most affects my assessment.
- Given that the no-replay variant already matches the best baselines, how do you attribute the Table 1 gains between replay and prompt/harness design? What would RRU's prompts with a plain update loop score under exactly matched conditions?
- On LAB, given judge = M_U = the Qwen agent and criteria-mirroring skills, can you re-grade with an independent judge family or human experts? Please disclose the exact selection procedure for the 96-task subset.
- Is the 78.3% skill used in the GRPO+skill experiment a cherry-picked best seed? Please report GRPO+skill with the mean-performing RRU skill, and add significance tests for the main comparisons.
- (Suggestion) Quantify the frequency of overwriting/over-generalization in the baselines; add GEPA as a baseline.
5 comments
Reply to Reviewer ThAG 2/2
Q2. Replay and RRU's prompts, under exactly matched conditions
We first clarify what the "w/o replay stage" row of Table 2 is: RRU's prompts in the Reflect-then-Update loop, which is the setting you ask about. It stores no records and has no initial pass with the empty skill; the written reflections of earlier batches stay readable to the update.
We developed RRU in two steps. We first asked how far the Reflect-then-Update loop could go with prompts alone, and worked on the prompts (Appendix F); the w/o replay stage ablation is the result of this step. We then added records and replay, for the one thing the prompts could not do: check an edit against the rollouts behind an instruction. This is why we report both comparisons: against the best baselines (Table 1), and against our own loop without the replay stage (Table 2).
Under exactly matched conditions, we reran this comparison with 10 seeds per arm instead of 3 and at equal rollouts. The w/o replay stage ablation has no initial pass, so it runs one more epoch (AppWorld, n = 10 per arm, tested after each epoch):
| Training rollouts | 90 | 180 | 270 | 360 |
|---|---|---|---|---|
| RRU | 74.0 | 76.8 | 76.9 | |
| w/o replay stage: RRU's prompts in the Reflect-then-Update loop | 72.2 | 73.0 | 74.6 | 73.0 |
At 360 rollouts RRU scores 76.9 and the w/o replay stage ablation 73.0: with the same prompts and the same rollouts, records and replay add 3.9 points (n = 10 per arm). The difference grows with training (+1.0, +2.2 and +3.9 at 180, 270 and 360 rollouts).
Q3. Legal Agent Bench: an independent judge, and the subset
We regraded the saved Table 1 outputs with a judge from another model family, gpt-oss-120b, with the same rubric, prompt and code, and without re-solving. Both columns are over the same episodes (Qwen agent: two of the three attempts per task, so the Qwen-judge column differs slightly from Table 1; Gemma agent: all episodes). Mean over 3 seeds.
| Legal Agent Bench | Qwen judge | gpt-oss-120b |
|---|---|---|
| Qwen agent (all-pass rate) | ||
| RRU | 19.4 | 12.2 |
| SkillOpt | 17.0 | 11.1 |
| Combee | 11.2 | 8.4 |
| Trace2Skill | 12.5 | 9.7 |
| Gemma agent (criteria met) | ||
| RRU | 60.2 | 56.6 |
| SkillOpt | 59.0 | 55.2 |
| Combee | 55.5 | 51.7 |
| Trace2Skill | 57.3 | 53.7 |
Under the independent judge, RRU remains above the three baselines for both agents (over SkillOpt, the best baseline: +1.0 for the Qwen agent and +1.4 for the Gemma agent).
Selection of the 96 tasks. Legal Agent Bench is expensive to run: a single episode holds about 150k to 170k tokens of context, and in a pilot with the benchmark's own harness 7 of 32 tasks ran past the 262k-token context window of the model. Running all 498 tasks for every method, seed and agent was beyond our budget, so we selected a subset using only runs without a skill. We keep the two memo task kinds (playbook escalation and term negotiation); the other kinds mostly stalled on a tracked-changes script within the turn cap. We rank their tasks by cost in those runs (whether the run stalled, then wall time and output tokens), keep the first 96, and split them 48/48 so that no combination of contract type and task kind appears in both splits.
Q4. RRU and GRPO
The 78.3 skill in Sec. 5.3 is the highest of the three seeds (74.4 / 75.0 / 78.3) of an earlier development version of RRU. As you suggested, we repeat the experiment with the three Table 1 skills, including the mean-performing one (Seed 2). The GRPO checkpoints of the submission were lost, so for this we retrained GRPO on AppWorld with LoRA on Tinker (rank 32, learning rate 1e-5, 8 × 8 rollouts per step for 22 steps, i.e. two epochs and 1,408 rollouts). All cells are evaluated on the same serving setup.
| AppWorld TGC | No skill | + Seed 1 skill | + Seed 2 skill | + Seed 3 skill | Mean with skill |
|---|---|---|---|---|---|
| Base model | 55.1 | 67.0 | 71.1 | 75.0 | 71.0 |
| GRPO checkpoint | 61.9 | 67.6 | 71.4 | 75.3 | 71.4 |
With the mean-performing skill, the GRPO checkpoint goes from 61.9 to 71.4; over the three skills, it also reaches 71.4 on average.
Significance of the main comparisons. For the main comparison, RRU against the w/o replay stage ablation, we now report 10 seeds per arm instead of 3, at equal rollouts (Q2).
Q5. GEPA
GEPA (v0.1.4, default settings, an empty starting skill, n = 3): 55.0 and 63.5 at RRU's budget of 180 / 400 rollouts, and 57.5 and 63.6 at three times that budget. RRU is above GEPA at both budgets on both benchmarks. Under RRU's rollout budget, GEPA returned the empty starting skill in three of the six runs, so we also allowed it three times the budget.
Dear Reviewer ThAG,
We wanted to express our gratitude again to your thoughtful and careful feedback. We hope our new results have strengthened the empirical evidence, and we are eager to learn your further comments.
Best Regards,
The authors
Reply to Reviewer ThAG 1/2
We sincerely thank the reviewer for the careful review. The budget question comes first, as you asked. All numbers are test accuracy in %, Qwen agent unless stated, mean over seeds; differences are computed from unrounded means, except the Table 1 margins, which are differences of the table entries as in the paper. The new runs in this response use a different serving setup from the submission; on it, the Table 1 RRU skills score 71.2 on AppWorld and 72.5 on SpreadsheetBench (72.3 and 73.1 in Table 1), and the model without a skill 56.8 and 59.6 (58.8 and 59.4). Rows marked Table 1, the baseline rows with more epochs and the row without the initial pass are from the submission's runs.
Q1. Equal budgets, training cost and update frequency
In Table 1, RRU uses 180 / 400 rollouts (AppWorld / SpreadsheetBench) and the baselines one epoch. Below, the baselines are given more epochs (n = 3; training rollouts in brackets; SkillOpt's count includes its validation gate):
Training cost and update frequency. The table above matches rollouts. The methods spend a rollout differently (RRU reflects on every rollout and replays records at the update; Trace2Skill updates once per epoch; SkillOpt spends rollouts on its validation gate), so we also report tokens and USD for every row (mean of 3 seeds, agent plus M_U, cumulative over the epochs; at the list prices used in the paper, USD 0.05 / USD 0.70 per million prompt / completion tokens):
On AppWorld, the baselines at RRU's cost or above (Trace2Skill at three epochs, USD 4.39; SkillOpt at three epochs, USD 6.38) are below RRU. On SpreadsheetBench, SkillOpt at two epochs (USD 9.37) is below RRU (USD 8.23), and at three epochs (USD 13.56) it is above. One epoch of RRU's prompts in the Reflect-then-Update loop (the w/o replay stage ablation of Q2) costs 40.0M / 54.4M prompt tokens and USD 2.51 / USD 3.90.
Combee's rollout count includes one re-attempt of each failed task after reflection; its row is from three further runs of the Table 1 configuration. RRU uses the most tokens per run, mostly in rollouts (61.2M of the 71.4M prompt tokens on AppWorld).
Update frequency. RRU updates after every 16 rollouts of a learning epoch (the initial pass makes no update), SkillOpt after every 40 of the 72 / 160 training tasks left by its 20% validation split (its released default), Combee's curator after every 8, and Trace2Skill once per epoch. We also ran SkillOpt with RRU's batch of 16, from the empty skill (n = 3, tested after its first epoch, 180 / 600 rollouts, on the new serving setup): 63.5 on AppWorld and 66.5 on SpreadsheetBench, against 61.4 and 66.0 at its default batch of 40.