acceptodds
Under review as a conference paper at ICLR 2027

Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?

Abstract

Learning from experience is essential for LLM agents to adapt to unfamiliar and dynmaic environments. Evaluating this ability is therefore important for understanding how effectively agents acquire and use new knowledge. Existing benchmarks have sought to evaluate this ability, but they primarily evaluate tasks whose rules are provided in the instructions or already familiar to pretrained models, making it difficult to distinguish learning from interactions from reasoning with existing knowledge. To address this, we introduce Learn2Play Bench, a benchmark of newly designed text-based games, whose rules are novel or counterintuitive, requiring agents to acquire knowledge through interaction rather than rely solely on pretrained knowledge. These games provide reproducible feedback and automatic scoring, enabling controlled evaluation of learning across repeated attempts. We also vary game instances to test whether agents can apply what they have learned to new situations. Therefore, we evaluate how backbone models, self-evolving methods, and agent harnesses affect agents’ learning ability, revealing three findings: (1) Experience retention: Retaining complete records of actions and feedback can support more effective learning than summarizing these experiences into rules or strategies. (2) Human–agent gap: Top-performing human players achieve higher peak scores than the evaluated agents. Human explore more varied strategies, and repeat actions less. (3) Harness matters: With the backbone fixed, changing the harness can improve performance while reducing estimated inference cost. Together, these findings provide insights into how LLM agents learn from experience and suggest directions for future work to improve their learning ability. Code is available at https://anonymous.4open.science/r/Learn2Play-Bench-F1D8/.

Primary area:
datasets and benchmarks
Keywords:
Continual Learning, LLM, Agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_C4m5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tackles the problem of "distinguishing learning from interactions from reasoning vs. existing knowledge" in LLM agents. To address this, the author proposed Learn2Play Bench, a benchmark whose rules are novel or counterintuitive, requiring agents to get knowledge through interaction rather than rely on pretrained knowledge. The empirical results show that (1) preserving evidence can matter more than compressing it, (2) top-performing human players outperform the best agent, and (3) better harnesses can improve agent performance and learning while reducing inference cost.

Critical strengths

The paper tackles the clear and important problem of "distinguishing learning from interactions from reasoning vs. existing knowledge" in LLM agents. Specifically, Learn2Play Bench consists of 20 text games where each game has novel or counterintuitive rules.

Extensive experiments with a human baseline are included.

Interesting results such as (1) simple MEMORY wins over other self-evolving methods in some models, or (2) harness selection is as important as model selection

Critical weaknesses

First of all, "unfamiliarity" of 20 games in Learn2Play Bench is not verified. This is important because the purpose of this benchmark is to separate pretrained knowledge vs. learning from interaction. I know that it's hard to verify, but there is a good proxy: use the first episode score of the "Naive" method to classify familiar/unfamiliar games. For instance, the 'last and found' game seems easy for LLM agents, so it can be classified as a "familiar" game perhaps.

Next, I'm not sure if the comparison btw human vs. agent is fair enough. While abstract says "Top-performing human players achieve higher peak scores than the evaluated agents", this seems to be toned down or modified. This is because (1) the gap between Human Top-1 (Max) vs. Claude Code (Max) is within the margin of error, (2) Human Top-1 is scored from N>20 people, while the agent has only N=3 trials, and (3) the mean score of humans is much lower than that of the agent.

Questions for authors

I'm curious about the statistics for each game. For instance, (1) how many turns each game requires to complete (to see if it is a long-horizon task), (2) similarly, how many tokens do LLM agents spend to complete each game, (3) the amount of time for human participants to complete, or any statistics that help us understand the context behind the games and the benchmark.

Another tone down: Again, I'm not certain if it is ok to write a headline like "Experience retention: Retaining complete records of actions and feedback can support more effective learning than summarizing these experiences into rules or strategies". This currently applies only to the Kimi K3 case in Table 1, and I would be happy to see whether this tendency holds for other LLMs.

Minor typo: Humansss, requiresss

Reviewer_imwWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Learn2Play Bench evaluates whether LLM agents improve through experience in unfamiliar text-based games with hidden rules. It comprises 20 newly designed games, including fixed games and games whose visible instances change across episodes while the underlying rules remain stable. The paper compares backbone models, experience-retention methods, agent harnesses, and human players using measures of achieved performance and improvement over repeated episodes. It finds substantial differences across models and harnesses, and examines when retaining interaction histories helps subsequent play.

Critical strengths
  1. The benchmark targets an important evaluation gap. Hidden and sometimes counterintuitive rules make prior familiarity a less straightforward explanation for success. Deterministic feedback and automatic scoring permit controlled comparisons over repeated attempts.
  2. This comparisons are informative. In particular, the same-backbone harness comparisons show that measured agent performance depends substantially on the surrounding agent system, not only on the model.
Critical weaknesses
  1. Score improvement is not sufficient evidence of rule discovery. Learning Gain and Learning Slope show that performance changes across episodes, but do not distinguish learning a hidden rule from remembering a successful action or improving execution. Although the appendix presents game-specific rule-discovery evidence, it does not provide a consistent aggregate measure tied to the scored learning trajectories. This leaves the paper’s central claim about learning from unfamiliar environments less directly established than the score results suggest.
  2. The robustness of the aggregate conclusions is uncertain. The main evaluation uses one pinned instance per game template, gives challenge games three times the weight of other games, and relies on estimated or partially established reference scores for some games. Without sensitivity analyses for these choices, it is difficult to tell whether the principal system rankings reflect broadly stable differences or particulars of the evaluation panel.
Questions for authors
  1. Can the authors evaluate discovery of predefined hidden rules under consistent criteria within the scoring window, and report how that measure relates to Learning Gain and Learning Slope?
  2. Do the main rankings persist with additional game-instance seeds, equal game weights, and only games with exact reference scores?
  3. The advantage of retaining full histories varies across backbones. Under matched backbone, harness, context allowance, and inference budget, does raw-history retention still outperform processed memories? This would help separate the proposed mechanism from differences in resources or implementation.
3 comments

Thank you for the additional analyses. Target coverage is only weakly correlated with Learning Gain and Learning Slope. Score gains alone therefore remain insufficient evidence of rule discovery. The weight and normalization checks are useful, but robustness across game seeds and a resource-matched memory comparison remain open. I maintain my rating.

AuthorsOpenReview ↗

Response by Authors (1/2)

Thanks for your response. There seems to be a misunderstanding about how we interpret these metrics.

As indicated in Section 4.1, LG and LS only quantify the magnitude and rate of improvement across episodes. However, agents also learn from feedback within an episode, including the first one. An agent that discovers useful rules early may therefore achieve a high first-episode score, leaving less room for further improvement across episodes. This early learning is not separately captured by LG or LS. We therefore interpret them alongside Max and Mean, which summarize attained performance, to assess both the outcomes and progression of learning from experience.

To address your concern about whether the benchmark's scores align with its intended learning targets, we extended our earlier analysis to Max and Mean using all 11 configurations across 20 games. We used the same Spearman correlation analysis as before, accounting for differences in average coverage and performance across games.

Table 1. Associations between target coverage and attained performance. 95% CIs and Holm-adjusted, two-sided p-values are based on bootstrap resampling by game.

Association with target coverageSpearman ρ95% bootstrap CIHolm-adjusted p
Max0.628[0.532, 0.710]0.00008
Mean0.605[0.500, 0.691]0.00008

As shown in the table, target coverage is substantially associated with both Max and Mean, supporting the intended alignment between learning-target attainment and task performance. Both associations remain significant after multiple-testing correction (adjusted p < 0.001).

We will revise our paper according to the new discussion.

AuthorsOpenReview ↗

Response by Authors (2/2)

As for the experiments across game seeds, our benchmark already includes games covering a wide range of tasks and rules. And we evaluated 29 agent configurations, running each configuration independently three times on every game. Running this full evaluation with one seed per game costs about $9,500. Each additional seed per game would require approximately another $9,500, so extending the full evaluation to multiple seeds would involve a substantial additional cost.

For the resource-matched memory comparison, we further compared Memory and a controlled ReasoningBank variant using Kimi K3 on four games, with the same interaction loop, context allowance, and inference settings. Each method ran three times per game, with up to ten episodes and a $4 budget covering both gameplay and memory processing.

Table 1 shows that Memory achieved higher mean scores on PrimordialSoup and Butterfly, while ReasoningBank scored higher on GemForge and Roommates. Their overall scores were close: 73.6 for Memory and 74.4 for ReasoningBank.

Table 1. Kimi K3 results under common cost and episode limits. Scores average the best normalized score from each of three runs.

GameMemory scoreReasoningBank score
PrimordialSoup69.768.2
GemForge47.248.3
Butterfly87.286.0
Roommates90.595.2

We emphasize this finding because, when designing memory systems, one might naturally expect more elaborate methods to outperform simply retaining raw histories. However, we found that memory sometimes can outperform more elaborate methods. As clarified in our earlier rebuttal, our point is that this simple baseline can perform better in some settings, not that it always outperforms more elaborate methods.

We hope these details and additional results help address your concerns, and we would be happy to answer any further questions.

Reviewer_EEpLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • What it is: 20 new text-based games with hidden, often counterintuitive rules (e.g., stealing a colleague’s lunch gets you promoted). Agents must discover the rules through play across repeated episodes, with no weight updates.
  • The goal is to separate real learning from recalling pretrained knowledge.
  • Findings: (1) Raw history beats summaries. (2) Opus 5 led. (3) Harness matters. (4) Humans still win at the top.
Critical strengths
  • Well-motivated problem and design (separating learning from interaction from recalling pretrained knowledge).
  • Broad and controlled comparisons.
  • Human baseline and qualitative evidence.
Critical weaknesses
  • Since your main sales pitch is a benchmark, the main question here is benchmark validity. The main concern here is that 20 hand-designed games is small.

  • Relatedly: How confident are we about your Statistical power? The evals use one seed per game template and three trials per game, over only few episodes. At the same time, I don't see any CIs or error bars.

  • Are the comparisons of agents vs humans apples-to-apples? Seems like human Top-k are selected per game by max out of the players. While agents get three trials. (So max-of-many vs max-of-few).

  • A few related works do very similar work and similar motivation. Given these existing works, what is the value your work adding to the existing literature?

Questions for authors

Included in the previous box.

1 comment

I appreciate the response by the authors. I will maintain my current score.

Comments on the paper

2 comments

Thank you for your hard work, and my original concerns have mostly been resolved.

However, I found that the response to Reviewer imwW ("Does score improvement reflect rule discovery?") provides an unexpectedly low correlation (<0.3) between {task coverage} and {learning gain/slope}, which raises another concern about whether this benchmark is well constructed as intended (in terms of "how well do agents learn from experience, or discover the rule"). Therefore, I remain with the current score.

AuthorsOpenReview ↗

Response by Authors

Thank you for noting that your original concerns have largely been resolved. There seems to be a misunderstanding about how we interpret these metrics.

As indicated in Section 4.1, LG and LS only quantify the magnitude and rate of improvement across episodes. However, agents also learn from feedback within an episode, including the first one. An agent that discovers useful rules early may therefore achieve a high first-episode score, leaving less room for further improvement across episodes. This early learning is not separately captured by LG or LS. We therefore interpret them alongside Max and Mean, which summarize attained performance, to assess both the outcomes and progression of learning from experience.

To address your concern about whether the benchmark's scores align with its intended learning targets, we extended our earlier analysis to Max and Mean using all 11 configurations across 20 games. We used the same Spearman correlation analysis as before, accounting for differences in average coverage and performance across games.

Table 1. Associations between target coverage and attained performance. 95% CIs and Holm-adjusted, two-sided p-values are based on bootstrap resampling by game.

Association with target coverageSpearman ρ95% bootstrap CIHolm-adjusted p
Max0.628[0.532, 0.710]0.00008
Mean0.605[0.500, 0.691]0.00008

As shown in the table, target coverage is substantially associated with both Max and Mean, supporting the intended alignment between learning-target attainment and task performance. Both associations remain significant after multiple-testing correction (adjusted p < 0.001).

We hope this clarification and the additional results address your concern. We would be happy to address any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.