- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This submission introduces S3Gym, a benchmark for testing LLM agent self-improvement. The authors separate experience-driven learning into three components: self-testing (where the agent explores different strategies and gathers evidence), self-judging (where the agent evaluates their actions and the resulting outcomes), and self-improvement (where the agent uses its experience to affect future decisions). The benchmark contains seven text-based games: Chess, Minesweeper, Nullify, Tetris, Snake, PvZ, and Trust Evolution. These games are chosen to test abilities like latent-rule induction, constraint satisfaction, spatial planning, resource allocation, and multi-agent strategy.
The main experiments compare two ways of retaining experience across seven different models (GPT-4o, GPT-4.1, o3-mini, Gemini-2.5-Flash, Gemini-2.5-Pro, GPT-5.5, and Gemini-3.5-Flash): History ICL and Summary Memory. In History ICL, the agent stores its history of play in-context. When the context limit is reached, the tail is truncated while preserving the current episode/prompt. In Summary Memory, the model compresses its previous exploration history into practical recommendations. The summary contains things like strategies to retain and mistakes to avoid. The results show significant variations across models and games: in some settings, Summary Memory helps and in others it hurts when compared to History ICL. Under both History ICL and Summary Memory, GPT-5.5 has the highest score across most games.
The authors also study parameter training using Qwen3-8B. In this setting, Qwen is fine-tuned on data from previous examples, and is evaluated without History ICL or Summary Memory. The authors find that fine-tuning produces significant gains on Trust Evolution, intermittent successes on Snake, and significant degredation on PvZ. The rest of the games yield scores near or at zero before and after fine-tuning.
Critical strengths
A strength of this work is that its setup is largely sound: the authors use game engines to assess the actual outcomes, use separate random seeds, and evaluates model learning ability separately from their ability to make good decisions from experience in-context. These factors are important because it helps us to draw reliable conclusions from the authors' experiments.
Critical weaknesses
A significant weakness of this work is that it is incremental in novelty when compared to the boatload of other benchmarking papers in the literature. For example, most of the games in this benchmark already appear in KORGym: A Dynamic Game Platform for LLM Reasoning Evaluation. There are also other benchmarks which measure very similar things, such as LifelongAgentBench and ContinualSkillBench. This submission feels more like a solid (but incremental) piece of engineering, as opposed to a novel scientific contribution that passes the bar for a conference like ICLR.
Another weakness of this submission is that it does not make an apples-to-apples comparison between History ICL/Summary Memory and fine-tuning, as Qwen3-8B is the only model used for fine-tuning and it is not used in the other settings this paper considers. As a result, it is hard to conclude much about the effectiveness of one method versus another. A stronger study would run all three approaches on multiple models, with the complete training procedure disclosed.
Questions for authors
Why did you choose not to use the same models for fine-tuning that you used for History ICL/Summary Memory?