- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies which textual skills an agent may write into its weights, since such writes cannot simply be deleted later. It argues that score-based selection can fail in four ways and proposes AGWC, which tests each candidate with paired short training runs, re-tests candidates as the selected set grows, stops early, and commits or rolls back the fully trained set. On ALFWorld it reports 44.48 percent scenario-OOD success against 36.52 for a Direct-Validation baseline, at 10.8 times the compute of training without skills.
Critical strengths
- The concern is real. A skill trained into the weights cannot be removed like prompt text, so asking what evidence is needed to authorize such a write is a fair question. The matched-write control at 18 skills and the fresh-retraining check are sensible design ideas.
Critical weaknesses
- The results cannot be traced to experiments. The main body calls its evidence "the supplied results" and states that "the experimental package contains aggregate tables rather than run-level records". Appendix D adds that "No new experiments were run in producing the manuscript", and the base model, skill generator, training settings, thresholds and uncertainty intervals are not reported.
- The judge is costly and depends on an unspecified training step. The short paired training runs cost 10.8 times the reference compute against 8.6 for Direct-Validation, with no equal-budget comparison in the main body. Their reliability cannot be assessed: the training step is unspecified, and probes already differ from full writes by 3.10 points on average.
- The evaluation is too narrow for a selection method. All behavioral results use ALFWorld with one skill generator and model family, against baselines the paper describes as "task-specific controls, not separately imported published algorithms". Effects outside ALFWorld, such as overfitting or lost general ability, are not measured.
Questions for authors
- Appendices A and D state that the numbers were not recomputed from logs and that no new experiments were run for the manuscript. Which experiments were run, and how were the tables derived?
- Which base model, fine-tuning method, skill generator and training lengths produced the results?
- How does AGWC compare with other selectors at equal compute, and does writing ALFWorld skills change performance outside ALFWorld?