- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper asks whether sufficiently capable LLM agents still need humans to specify how they should self-improve. The authors introduce a controlled teacher–player setting in which a teacher autonomously generates games and updates a player's prompts or skill files based on gameplay feedback. Experiments cover six model versions from three families. The results show that newer models can improve their performance in this setting, with some gains transferring across game horizons, game engines, 3D navigation environments, and other models.
Critical strengths
- The paper studies whether capable LLM agents still need humans to specify how they should self-improve. Its focus on autonomously organizing learning from self-generated experience offers a novel and timely perspective.
- The experiments are well controlled: the teacher and player use the same frozen model, the teacher cannot access external test tasks or validation scores, and players are evaluated in independent sessions. This helps distinguish autonomous learning from external supervision. The broad evaluation, including transfer tests and tool ablations, provides evidence that frontier models can achieve transferable improvements without a prescribed learning procedure.
Critical weaknesses
- The authors argue that models that improve more also receive higher SRL scores. However, this comparison is mainly between older and newer models within the same family, and newer models have stronger overall capabilities. Their higher SRL scores may simply reflect these stronger capabilities, rather than explain their self-improvement. Therefore, this analysis provides limited information beyond “newer models are stronger” and does not explain which strategies enable models to self-improve.
- The inverted-U pattern in Figure 5 does not by itself establish that teachers actively select games of intermediate difficulty. Empirical pass rates and gameplay counts are calculated from the same attempts and are therefore statistically coupled. For example, a game played only once must have a pass rate of either 0 or 1, while intermediate pass rates require multiple attempts. This sampling effect could contribute to the observed pattern even without difficulty selection.
Questions for authors
- Could you address the concerns raised in the Weaknesses section?
2 comments
Thank you for the detailed responses and additional analyses.
I consider both weaknesses addressed by the additional analyses and methodological revisions. While some of the observed trends are inconsistent, e.g., the trends in Figure 16, my methodological concerns are addressed.
Overall, I appreciate these improvements and maintain my original rating.
We thank the reviewer for their insightful feedback. We are glad that they found the paper's perspective to be novel and timely, and the experiments well controlled. We address their questions below.
We agree with the reviewer that more broadly capable models may achieve higher SRL scores. We have updated the text in Section 5.2 to note that we do not fully disentangle a causal relationship between SRL scores and self-improvement.
Ablating specific SRL behaviors is difficult, because a single teacher action may fall under multiple categories. We instead add three complementary analyses on self-improvement strategies.
In Figure 5, we examine four teacher behaviors that can be algorithmically measured:
Self-improving models A/B test more often and produce fewer trivial games. We further corroborate this with histograms over solution lengths of generated games (Figure 19), showing that models that self-improve produce games with longer solutions. Note that these behaviors can be associated with SRL categories (e.g., A/B testing is a form of self-experimentation.)
In Appendix D.7 we investigate teacher causal attribution; e.g., the ability to perform targeted updates. We identify updates made by
Claude Opus-5that improved teacher-observed performance on at least one self-generated game. We then post-hoc evaluate the actual player win-rate on those self-generated games using artifacts before and after the identified update. Averaged over games, teacher updates improve actual performance from 51.9% to 72.1%.In Appendix D.9 we add timeline plots showing gameplay patterns across teacher sessions. Common patterns among self-improving models are visually apparent, such as A/B testing, revisiting failed games, and re-evaluation across successful games after updates.
We thank the reviewer for pointing this out. To control for this potential effect, we have added Figure 16 (Appendix D.4), and added a selection of panels from Figure 16 to Figure 7 (previously Figure 5). For each generated game, we plot the observed win rate over the first k plays, against the number of future plays of that game, for different values of k. This analysis thus answers the question: if the teacher observed an x win-rate on the first k times it played a game, how many times did it replay it later? We observe that newer models spend more gameplay effort on games that they measure to be low-to-medium difficulty. This preference is also visually apparent in the added timeline plots.