acceptodds
Under review as a conference paper at ICLR 2027

S³Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement?

Abstract

Large language models (LLMs) accumulate interaction experience, but this experience does not necessarily improve their subsequent behavior. We introduce S3Gym, a benchmark connecting Self-Testing, Self-Judging, and Self-Improvement across seven text-based games. The protocol separates permissive exploration from strict held-out evaluation and uses executable environment verifiers to audit self-judgments and performance. We compare History ICL and score-conditioned Summary Memory across seven proprietary models, and separately study parameter training with Qwen3-8B. The results reveal model- and task-dependent benefits rather than a universally effective experience representation. Self-Judging is only partially reliable: agreement about positive rewards can coexist with large normalized magnitude discrepancies, and local judgment accuracy does not reliably predict subsequent gains. Summaries do not consistently outperform direct history. Selected cases suggest that explicit, state-dependent action rules grounded in observed outcomes provide more useful iteration directions than generic reminders. Parameter training improves some tasks but also produces transient gains and persistent degradation. Late-stage declines may reflect overfitting to exploration patterns and reduced generalization to held-out conditions, although the experiment does not establish this mechanism. These findings motivate evaluating judgment quality, actionable experience reuse, and retained performance separately. SGym supports this diagnosis by distinguishing accumulated experience from measurable improvement in held-out behavior.

Primary area:
foundation or frontier models, including LLMs
Keywords:
self-improvement, self-testing, self-judging, LLM agents, benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_WY1JOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This submission introduces S3Gym, a benchmark for testing LLM agent self-improvement. The authors separate experience-driven learning into three components: self-testing (where the agent explores different strategies and gathers evidence), self-judging (where the agent evaluates their actions and the resulting outcomes), and self-improvement (where the agent uses its experience to affect future decisions). The benchmark contains seven text-based games: Chess, Minesweeper, Nullify, Tetris, Snake, PvZ, and Trust Evolution. These games are chosen to test abilities like latent-rule induction, constraint satisfaction, spatial planning, resource allocation, and multi-agent strategy.

The main experiments compare two ways of retaining experience across seven different models (GPT-4o, GPT-4.1, o3-mini, Gemini-2.5-Flash, Gemini-2.5-Pro, GPT-5.5, and Gemini-3.5-Flash): History ICL and Summary Memory. In History ICL, the agent stores its history of play in-context. When the context limit is reached, the tail is truncated while preserving the current episode/prompt. In Summary Memory, the model compresses its previous exploration history into practical recommendations. The summary contains things like strategies to retain and mistakes to avoid. The results show significant variations across models and games: in some settings, Summary Memory helps and in others it hurts when compared to History ICL. Under both History ICL and Summary Memory, GPT-5.5 has the highest score across most games.

The authors also study parameter training using Qwen3-8B. In this setting, Qwen is fine-tuned on data from previous examples, and is evaluated without History ICL or Summary Memory. The authors find that fine-tuning produces significant gains on Trust Evolution, intermittent successes on Snake, and significant degredation on PvZ. The rest of the games yield scores near or at zero before and after fine-tuning.

Critical strengths

A strength of this work is that its setup is largely sound: the authors use game engines to assess the actual outcomes, use separate random seeds, and evaluates model learning ability separately from their ability to make good decisions from experience in-context. These factors are important because it helps us to draw reliable conclusions from the authors' experiments.

Critical weaknesses

A significant weakness of this work is that it is incremental in novelty when compared to the boatload of other benchmarking papers in the literature. For example, most of the games in this benchmark already appear in KORGym: A Dynamic Game Platform for LLM Reasoning Evaluation. There are also other benchmarks which measure very similar things, such as LifelongAgentBench and ContinualSkillBench. This submission feels more like a solid (but incremental) piece of engineering, as opposed to a novel scientific contribution that passes the bar for a conference like ICLR.

Another weakness of this submission is that it does not make an apples-to-apples comparison between History ICL/Summary Memory and fine-tuning, as Qwen3-8B is the only model used for fine-tuning and it is not used in the other settings this paper considers. As a result, it is hard to conclude much about the effectiveness of one method versus another. A stronger study would run all three approaches on multiple models, with the complete training procedure disclosed.

Questions for authors

Why did you choose not to use the same models for fine-tuning that you used for History ICL/Summary Memory?

Reviewer_9mSdOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a benchmark of seven text-based games with easier exploration configurations and harder evaluation configurations. Agents learn from experience collected in the easier settings and are evaluated on held-out instances in the harder settings to assess whether their performance improves. The benchmark frames this process around three capabilities: self-testing, self-judging, and self-improvement. Experiments compare raw-history ICL with summary memory, analyze the relationship between self-judgment accuracy and performance gains, and separately examine parameter training.

Critical strengths
  1. The game environments provide an interesting setting for studying experience reuse. The tasks cover different demands, including spatial planning, resource allocation, and strategic interaction. Environments such as Plants-vs-Zombies offer opportunities to learn reusable strategies across episodes. Separating exploration from stricter evaluation, while excluding evaluation trajectories from subsequent learning, is a useful design for testing transfer beyond the interactions used to collect experience.

  2. The history-versus-summary experiments provide informative mixed results. In particular, the finding that summaries can substantially help some model–game combinations while hurting others challenges the assumption that compressing experience into recommendations is consistently beneficial. Reporting these failures alongside successes makes the comparison useful for researchers designing memory mechanisms.

Critical weaknesses
  1. The operational definitions and experimental controls limit what can be concluded about the three capabilities. Self-testing is implemented primarily as interaction under permissive exploration conditions, without a dedicated measure of whether the agent designs informative tests or collects useful learning experience. Although the paper frames the capabilities as coupled, its experiments do not isolate their contributions to subsequent improvement. The self-judging metric also has a limited relationship to the broader capability of evaluating useful experience. It primarily measures agreement with immediate environment rewards. Under hidden information or stochastic outcomes, even an accurate expected-reward prediction can disagree with the realized reward. More fundamentally, immediate reward need not reflect an action’s contribution to eventual success: in these games, expected cumulative return under a continuation policy may be more relevant to evaluating strategic decisions. The acknowledged ambiguity between immediate and cumulative scores in some prompts further complicates interpretation. Numerical immediate-reward prediction is also a particular interface choice rather than a requirement of learning from experience. Agents may learn through qualitative reflections, state transitions, or reusable skills without making such predictions. The current results do not establish whether self-scores contribute to learning or distinguish failures of reward prediction from failures of the harness to use experience effectively. Controls that remove scores, shuffle them, or replace them with environment rewards would help establish their contribution. Consequently, the near-zero judgment–gain correlations are informative descriptive results, but provide limited evidence about which component drives successful or unsuccessful learning.

  2. The principal improvement metric can reward fluctuations without sustained improvement. AUC+ counts only performance above the initial baseline and ignores regressions below it. An agent that fluctuates around its initial performance can therefore receive a positive score despite having no sustained or net gain. The normalized version additionally becomes sensitive to small baselines. Average and maximum scores provide complementary information about capability, but do not resolve this issue. Given that the benchmark’s central purpose is measuring improvement, signed baseline-relative gains, final gains, and uncertainty across repeated runs would provide a more direct basis for interpreting its results.

Questions for authors
  1. Distinguishing judgment from experience reuse: Do you have controls that remove self-scores, shuffle them, or replace them with environment rewards while keeping the underlying trajectories fixed? These would help determine whether self-scores contribute to improvement and distinguish inaccurate judgments from a harness that cannot effectively use the judgments.

  2. Meaning of judgment accuracy: Are self-scores intended to predict expected immediate rewards or realized rewards? How do you account for hidden information, stochastic outcomes, and the acknowledged ambiguity between immediate and cumulative scores when interpreting disagreement? What evidence supports immediate-reward prediction as a diagnostic of the agent’s ability to identify experience useful for future improvement?

  3. Self-testing: Is self-testing intended to mean ordinary interaction during exploration, or actively selecting informative experiments? If the latter, what evidence distinguishes the quality of self-testing from the effectiveness of the subsequent judgment and learning mechanisms?

  4. Measuring improvement: Do the main conclusions hold under signed baseline-relative area and final performance gain, which account for regressions? Are repeated-run results available to determine whether positive AUC+ reflects sustained learning rather than fluctuations around the initial baseline?

  5. Training dynamics: Are there curves showing SFT training loss and game performance on the exploration configurations over successive updates? These would help distinguish failure to learn the training examples, learning unhelpful targets, and improvements on the exploration distribution that fail to transfer to harder evaluation.

Reviewer_hXGzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Motivated by the fact the most proposed benchmarks in the literature don't take into account that these days in reality models run on tasks and improve on them either by a test-time scaling method or by user simply interacting with the model on the task a few times, they offer a benchamrk which mixes 3 different test-time improvement methods with a few benchmarks and show the performance and generality of these methods on the various benchmarks.

Critical strengths

None

Critical weaknesses

The paper says that current benchmarks don't evaluate the capability of improving from past interactions. This suggests a misunderstanding of the landscape. There are different components in this landscape: a model, a benchmark, a method for test-time scaling (with or without changing weights). If someone is introducing a fine-tuning based test-time scaling method, then they compare it on some common benchmarks with other methods that also do fine-tuning based test-time scaling, it's natural to fit a test-time scaling method into a benchmark.

The paper mixes a few specific test-time improvement methods with a few specific benchmarks and claims it to be a new benchmark. Ok. But I don't see anything novel about this benchmark. None of the test-time scaling methods are novel, none of the base benchmarks are novel, and the combination looks arbitrary.

Questions for authors

What would we miss if we take a few test-time scaling methods we want to compare and run them on the benchmarks we're interested in?

What do we gain by bringing 3 common test-time scaling methods (ICL, reflection, and fine-tuning on successful traces) together under this benchmark?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.