acceptodds
Under review as a conference paper at ICLR 2027

Do LLM Agents Still Need to be Told How to Self-Improve?

Abstract

LLM agents increasingly drive their own learning. Self-improvement methods delegate various components of model optimization to the models themselves. As models become more capable, this motivates a basic question: do these increasingly capable models still need us to specify how they should self-improve? In this paper we study an autonomous paradigm in which, given basic primitives, resources, and a general goal for self-improvement, agents construct their own self-learning algorithm. Using the setting of gameplay as a testbed, a frozen LLM agent is given a compute budget and tools for collecting experience and updating text documents, and allowed to freely experiment without an externally imposed loop or access to real-world data. We find that sufficiently capable models do self-improve under these conditions, demonstrating strategies that align with classic cognitive frameworks for self-regulated learning and constructs from machine learning literature on self-generated curricula. Furthermore, we study the extent to which the learned text artifacts transfer to external benchmarks, observing improvement across different game horizons, mechanics, and 3D navigation tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLMs, agents, self-play, self-improvement, reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_inJROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether sufficiently capable LLM agents still need humans to specify how they should self-improve. The authors introduce a controlled teacher–player setting in which a teacher autonomously generates games and updates a player's prompts or skill files based on gameplay feedback. Experiments cover six model versions from three families. The results show that newer models can improve their performance in this setting, with some gains transferring across game horizons, game engines, 3D navigation environments, and other models.

Critical strengths
  • The paper studies whether capable LLM agents still need humans to specify how they should self-improve. Its focus on autonomously organizing learning from self-generated experience offers a novel and timely perspective.
  • The experiments are well controlled: the teacher and player use the same frozen model, the teacher cannot access external test tasks or validation scores, and players are evaluated in independent sessions. This helps distinguish autonomous learning from external supervision. The broad evaluation, including transfer tests and tool ablations, provides evidence that frontier models can achieve transferable improvements without a prescribed learning procedure.
Critical weaknesses
  • The authors argue that models that improve more also receive higher SRL scores. However, this comparison is mainly between older and newer models within the same family, and newer models have stronger overall capabilities. Their higher SRL scores may simply reflect these stronger capabilities, rather than explain their self-improvement. Therefore, this analysis provides limited information beyond “newer models are stronger” and does not explain which strategies enable models to self-improve.
  • The inverted-U pattern in Figure 5 does not by itself establish that teachers actively select games of intermediate difficulty. Empirical pass rates and gameplay counts are calculated from the same attempts and are therefore statistically coupled. For example, a game played only once must have a pass rate of either 0 or 1, while intermediate pass rates require multiple attempts. This sampling effect could contribute to the observed pattern even without difficulty selection.
Questions for authors
  • Could you address the concerns raised in the Weaknesses section?
2 comments

We thank the reviewer for their insightful feedback. We are glad that they found the paper's perspective to be novel and timely, and the experiments well controlled. We address their questions below.

The authors argue that models that improve more also receive higher SRL scores...Their higher SRL scores may simply reflect these stronger capabilities

We agree with the reviewer that more broadly capable models may achieve higher SRL scores. We have updated the text in Section 5.2 to note that we do not fully disentangle a causal relationship between SRL scores and self-improvement.

Ablating specific SRL behaviors is difficult, because a single teacher action may fall under multiple categories. We instead add three complementary analyses on self-improvement strategies.

  1. In Figure 5, we examine four teacher behaviors that can be algorithmically measured:

    • A/B testing (testing the same game with and without text artifacts).
    • Generating trivial games (both short solution length and high player win-rate).
    • Updates that improve performance on at least one generated game.
    • Update size.

    Self-improving models A/B test more often and produce fewer trivial games. We further corroborate this with histograms over solution lengths of generated games (Figure 19), showing that models that self-improve produce games with longer solutions. Note that these behaviors can be associated with SRL categories (e.g., A/B testing is a form of self-experimentation.)

  2. In Appendix D.7 we investigate teacher causal attribution; e.g., the ability to perform targeted updates. We identify updates made by Claude Opus-5 that improved teacher-observed performance on at least one self-generated game. We then post-hoc evaluate the actual player win-rate on those self-generated games using artifacts before and after the identified update. Averaged over games, teacher updates improve actual performance from 51.9% to 72.1%.

  3. In Appendix D.9 we add timeline plots showing gameplay patterns across teacher sessions. Common patterns among self-improving models are visually apparent, such as A/B testing, revisiting failed games, and re-evaluation across successful games after updates.

Empirical pass rates and gameplay counts are calculated from the same attempts and are therefore statistically coupled. For example, a game played only once must have a pass rate of either 0 or 1...

We thank the reviewer for pointing this out. To control for this potential effect, we have added Figure 16 (Appendix D.4), and added a selection of panels from Figure 16 to Figure 7 (previously Figure 5). For each generated game, we plot the observed win rate over the first k plays, against the number of future plays of that game, for different values of k. This analysis thus answers the question: if the teacher observed an x win-rate on the first k times it played a game, how many times did it replay it later? We observe that newer models spend more gameplay effort on games that they measure to be low-to-medium difficulty. This preference is also visually apparent in the added timeline plots.

Thank you for the detailed responses and additional analyses.

I consider both weaknesses addressed by the additional analyses and methodological revisions. While some of the observed trends are inconsistent, e.g., the trends in Figure 16, my methodological concerns are addressed.

Overall, I appreciate these improvements and maintain my original rating.

Reviewer_mVvTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether a frozen LLM can organize useful context optimization when given a goal, tools, and a budget, without a prescribed optimization loop. A teacher creates games, observes a separate player, and edits player prompts or skill files. Experiments across six models, tool ablations, and transfer evaluations show that some capable models produce useful artifacts. Improvement here concerns the induced player context, not model weights.

Critical strengths
  1. S1: The isolated teacher/player setup, disabled external access and default memory, and explicit editable context make the question interpretable.
  2. S2: Transfer is examined across PuzzleScript horizons, ARC Level 1, MazeBench, and models, rather than only on generated training games.
  3. S3: Read/Write, Bash, Gameplay, Python-game, and budget ablations help separate prior knowledge, execution experience, and environmental cues.
Critical weaknesses
  1. W1: There is no resource-matched prescribed context optimizer or strong manually written/one-shot strategy baseline. Read/Write alone already helps. The incremental value of letting the model organize its own procedure therefore remains unclear relative to conventional optimization (§6.1).
  2. W2: The evaluation panels are small: 20 games per PuzzleScript split, 17 ARC games, and 16 MazeBench rooms. PuzzleScript and ARC are also filtered using Opus-5 performance. Standard deviations over three teacher seeds do not capture uncertainty from task selection or establish a broad model-generation trend (Appendix B).
  3. W3: Edit and gameplay counts are incomplete cost proxies. Actual teacher tokens, Bash computation, artifact length, and player reading overhead are not jointly reported, limiting resource comparisons between tool configurations (§4.2; Appendix C.2).
  4. W4: Figure 5 relates replay counts to success rates estimated from the same adaptive sampling process. Stopping after success, retrying failures, and estimation noise could produce similar patterns. This is descriptive evidence consistent with learnability, rather than identification of an active learnability objective (§5.3)..
Questions for authors
  1. Q1 → W1: Can the authors compare with a prescribed reflection loop, a GEPA/ACE-style optimizer, and a strong one-shot strategy prompt under matched total tokens, execution costs, and artifact lengths?
  2. Q2 → W2: Can a larger panel without Opus-5 performance filtering be evaluated, with paired uncertainty over games and teacher seeds reported separately? Please also reconcile the GLM version labels.
  3. Q3 → W4: Can success probabilities be estimated from independent pilot runs and used to predict subsequent selection, alongside a simple retry-failure/stop-on-success baseline?
2 comments

We thank the reviewer for their positive remarks on our controlled setup and generalization experiments. We address questions and comments below.

Main Response:

Q1/W1: Can the authors compare with a prescribed reflection loop, a GEPA/ACE-style optimizer?

GEPA is not a baseline for our setting because it requires real data, whereas in our setup agents must self-generate games. However, we agree that comparing to a prescribed optimization loop provides valuable calibration for the scale of expected performance gains.

We have added Figure 3, showing GEPA optimization with 20 real in-distribution games (10 train, 10 test) and Claude Opus-5. Under matched game-rollout and token budgets, self-organized learning with self-generated games achieves comparable final performance to GEPA that hill-climbs on in-distribution real games (45% v. 48%). Figure 13 shows trajectories over individual seeds.

Q2/W2: Can a larger panel without Opus-5 performance filtering be evaluated?

We filter out saturated tasks to ensure that our benchmarks are of sufficient difficulty. To further address this question we have added Table 4 in Appendix D.6. We evaluate Claude Opus-5 with optimized artifacts from three teacher sessions on 10 randomly sampled saturated games. Performance remains at 100% for 8/10 games, and above 90% for the remaining two.

W4: Figure 5 relates replay counts to success rates estimated from the same adaptive sampling process...This is descriptive evidence consistent with learnability, rather than identification of an active learnability objective.

To address this potential effect we have added Figure 16 (Appendix D.4). For each generated game, we plot the observed win rate over the first kk plays, against the number of future plays, for different values of kk. Newer models spend more gameplay effort on medium- and lower-scoring games. We have replaced parts of Figure 7 (previously Figure 5) with sample panels from Figure 16.

We agree that our analyses are descriptive evidence consistent with learnability, and do not claim to identify an active objective; Section 5.3 states that the inverted-U shape resembles learnability task rewards.

Further Clarification:

W1: Can the authors compare to...a strong one-shot strategy prompt under matched total tokens, execution costs, and artifact lengths?

Our Read-Write setting in Figure 10 is a stronger version of the suggested baseline, in which the teacher iterates on text artifacts without access to gameplay. Read-Write artifacts after the first update is equivalent to a one-shot prompt. Enforcing a fixed artifact length is incompatible with our autonomous setting in which the agent decides how to structure text artifacts.

W3: Actual teacher tokens, Bash computation, artifact length, and player reading overhead are not jointly reported, limiting resource comparisons between tool configurations*

For resource accounting, teacher and player token budgets are reported in Tables 2-3. Artifact length varies between teacher runs but is inherently capped based on the teacher token budget; Figure 17 also summarizes update size (token edit distance). Bash commands run on the CPU.

W4: Can success probabilities be estimated from independent pilot runs and used to predict subsequent selection, alongside a simple retry-failure/stop-on-success baseline?

We respectfully request that the reviewer clarify this question. We are unsure what "success probabilities estimated from pilot runs" refers to, as generated games differ between independent teacher runs. Moreover, replaying failed games more and successful games less is consistent with behavior that seeks low/intermediate-difficulty tasks.

I appreciate the authors' efforts. The author’s response has indeed given me a better understanding of the paper; however, these revisions are not sufficient to warrant a higher score, so I am maintaining my original rating.

Thank you for the response, but the additional results still do not resolve my core concern regarding.

Reviewer_gEwjOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors present a study on the ability on the fact that strong LLMs can devise their own self-improvement routines using basic tools and a compute budget, without a human-prescribed learning loop. With model weights frozen, they improve on unseen games by creating practice tasks, testing strategies, and refining prompts or skill files.

Critical strengths

A key strength of the paper is its detailed analysis of LLM behavior in autonomous self-improvement settings, supported by extensive experiments across multiple environments, model families, and model sizes. The ablation studies in the main paper are compelling, and the additional ablations in the appendix provide useful insights for researchers seeking to reproduce the results.

Critical weaknesses

None

Questions for authors

Have the authors investigated how LLM agents behave and self-improve in stochastic environments, such as NetHack [1]? To what extent do the observed self-improvement gains generalize to multi-agent settings?

[1] The NetHack Learning Environment, Küttler et al., 2020

2 comments

We thank the reviewer for the positive assessment of our work. We appreciate that they found our experiments and analysis to be compelling and extensive. We address their remaining questions below.

Have the authors investigated how LLM agents behave and self-improve in stochastic environments, such as NetHack?

Our evaluation benchmarks (PuzzleScript, ARC-AGI-3, MazeBench) are deterministic environments, though our setting is in principle extensible to stochastic environments. We do find, however, that in settings where the teacher is not constrained to PuzzleScript games, the teacher may implement stochastic games. One example is an excerpt in Appendix E.2 from a Claude Opus-5 Read-Write-Bash session. The teacher writes stochastic market simulations and includes a section in the player artifacts on separating "Signal vs noise", with guidance about collecting multiple measurements in noisy settings. Whether such observations lead to actual improvement in held-out stochastic environments is a rich avenue for further study.

To what extent do the observed self-improvement gains generalize to multi-agent settings?

In principle, the single-agent self-play setting can be generalized and parallelized in a multi-agent setting by allowing the teacher to spawn several agents. A key distinction however is whether the multi-agent setup should be cooperative (allowing for agents to communicate and collaborate) or competitive (in which agents play against each other in isolation). Another design choice, which could be left to the teacher, is whether the agents share a single set of text artifacts, or if the text artifacts are curated for particular roles in a multi-agent game. It is thus non-trivial to design a full multi-agent experiment, however we agree it is an exciting future direction.

I would like to than the authors for their answers, I would have appreciated to see an additional experiment on a simple stochastic environment though. I won't change my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.