- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
SkillRein trains a Qwen3-4B skill proposer with RL while the executor stays frozen. The executor first solves a batch of 50 related tasks without skills, the proposer reads selected successes and failures and writes one shared skill set, and the executor solves the batch again with it. The reward is , where averages the outcome change over tasks that cite each used skill and is skill coverage. On search QA, success rises from 30.9% to 36.5%, while the untrained proposer and five large models lower it. The paper also reports transfer to Qwen3-8B and a WebShop gain from 3.3% to 17.7%.
Critical strengths
- The formulation and reward are simple and well targeted, and the reward correlates with downstream success change (Spearman 0.78).
- The learned behavior is stable across seeds (35.8% to 36.5%) and easy to interpret. Learned skills lead to 1.5 searches and 12% unanswered questions, versus 2.4 to 2.6 searches and 60% to 64% with large-model skills.
Critical weaknesses
- The main evaluation batches are sampled from the training question pool, and the shown examples are also scored. Generalization is therefore not measured.
- The transfer claim is not supported by the released checkpoint. The main text reports 30.1% on Qwen3-8B, above the large models, but Appendix E reports 29.2% with the released checkpoint versus 31.4% to 34.5% for the large models.
- Baselines are weak. There is no hand-written skill baseline, no prompt-optimization baseline, and no related RL method. The reward relies on self-reported skill use, and the reward terms are not ablated.
Questions for authors
- Do the Table 1 batches come from the training pool, and what are the results on official test splits, including unseen datasets?
- What are the results if the shown examples are excluded from scoring?
- Which Qwen3-8B result is correct, and how does it change the transfer claim?
- How does a simple hand-written "answer within budget" skill perform under the same wrapper?
- How often does the executor report skills it does not follow?
Suggestions for future work:
- Ablate the coverage term and per-skill weighting, and strengthen the WebShop baselines.