- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work proposes a new pipeline for learning skills from training samples, as well as a skill distillation method based on lessons and skills. The skill learning pipeline involves retrying failed samples while providing teacher-generated hints until the policy succeeds. Each trajectory is then consolidated into a lesson, and lessons are consolidated into a reflection skill. This is motivated by the fact that lessons generated from more iterations are higher quality, and that learning skills from a few samples is less at risk of overfitting than standard SFT. The authors then propose a simple distillation method, which consists in (1) training a small policy with SFT on trajectories and lessons generated from a stronger model, (2) deriving a reflection skill from the fine-tuned small policy via iterative revision, (3) further training with GRPO and additional SFT on samples that first fail but succeed after retries with the reflection skill. Experiments show the proposed skill pipeline improves performance compared to baselines on document-QA tasks, ALFWorld, and multimodal benchmarks. The distillation method is validated on ALFWorld and WebShop.
Critical strengths
- The proposed iterative refinement before aggregating into lessons is sound.
- The methods show good empirical performance compared to baselines.
Critical weaknesses
- I have concerns about the novelty of the work. The main addition compared to prior works appears to be the iterative revision before extracting lessons, which seems rather incremental. The discussion and results around the analogy with gradient descent do not add much to the argument as most skill learning pipelines similarly derive lessons and skills from a few samples. The distillation baseline proposed is a standard SFT+RL finetuning pipeline, with some additional fine-tuning on samples that succeed after retrying with the reflection skill in context.
- Experimental results do not establish whether the comparison with baselines is fair. The manuscript just states that all baselines and the proposed method share the same tool sets and tool descriptions (l.308), but provides very little details about the implementations of each baseline, how the compute/search budget is set to ensure a fair comparison between baselines and the proposed method, how the train/test split is done for each benchmark, etc. The Appendix only provides the prompts used for ALFWorld but no additional details on the experimental setup. Since the code is not provided either, this raises concerns about the reproducibility of the work.
Questions for authors
- Regarding the iterative refinement phase, the model solving a sample is not deterministic: different rollouts can succeed or fail despite having the same context. How do you take this into account to distinguish a real improvement from a lucky rollout when iteratively refining the lesson?
- To what tokens is the CE loss applied in Eq.1? Only to the generated tokens or also to the inserted reflection skill?