- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Skill-R1 keeps the task model frozen and instead trains a small 4B editor to improve the natural-language skills that model uses. For each task instance, the editor revises a population of skills over generations, conditioned on earlier (skill, rollout, reward) triples. The main technical contribution is a "bi-level" GRPO advantage. It combines the usual within-generation term with times the change in mean population reward from one generation to the next (Eq. 9–10). Evaluation covers 50 GAIA tasks and 65 tasks each from WebWalker and Geometry3k. Grok-4.20 and GPT-4o-mini serve as task models, and GPT-4o-mini also tests whether edited skills transfer. On GAIA, accuracy rises from 36% to 62% with Grok and from 8% to 40% with GPT-4o-mini.
Critical strengths
S1. The setting is practical. Only a small editor is trained, so the approach works with closed task models, and editing costs under 1% of training tokens (Table 6). Many people running skill-based agents cannot fine-tune the executor, so this matters.
S2. The GAIA gains are large and transfer across task models: 20/50 vs. 4/50 with GPT-4o-mini and 31/50 vs. 18/50 with Grok. These differences are unlikely to be noise. The case study in App. A.2 also helps. The untrained editor pastes unrelated task content into a skill, and the trained one does not.
Critical weaknesses
W1. The main methodological contribution is never isolated. The claimed novelty is the inter-generation term. Yet no experiment compares with at matched . The Table 4 ablation ("Intra-gen, ") changes the objective and the number of generations at the same time, and possibly the number of evaluation attempts too. Its 18-point gap could therefore come from multi-round revision or from extra attempts. In Table 5, all three values give exactly 50.8% on Geometry3k. I read that as "no measurable effect", not as evidence that the method is insensitive to λ. Fig. 2 also shows generations 1–3 converging to the warm start late in training, with Gen 2 below it. A vs. run on GAIA, with and the same protocol, is what I need to see.
W2. Methods are not evaluated under the same protocol. App. A.1 says Skill-R1 gets a writable copy of the skill directory while baselines share a read-only one. If the bank keeps changing during evaluation, Skill-R1 learns across the test set and the baselines cannot. Skill-R1 is also scored as any-correct over 3 attempts, while Best-of-N appears to return a single majority answer. Its per-level GAIA counts match ReAct+skills and Direct exactly, which looks odd. Outside GAIA the margins are 2–3 problems out of 65, and "best variant" is chosen on the test set from two editors. Please report results with a frozen bank, the same attempt budget and scoring for all methods, and confirm that the 20 distillation tasks do not overlap the GAIA test set.
Questions for authors
- [Clarification] At test time, does Skill-R1 write edited skills back to the bank? What is the accuracy with the bank frozen?
- [Clarification] What is GAIA accuracy for , ? Without this I cannot credit any gain to the inter-generation term.
- [Clarification] How many attempts does "Intra-gen ()" get in Table 4, and how is Best-of-N scored?
- [Clarification] Are the 20 distillation tasks and the 115 training tasks disjoint from the 50 GAIA test tasks?
- [Suggestion] A feedback-driven prompt optimizer such as GEPA or TextGrad would be a stronger baseline than EvoPrompt, which falls below the no-skill baseline in several settings.