- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper tackles the problem of "distinguishing learning from interactions from reasoning vs. existing knowledge" in LLM agents. To address this, the author proposed Learn2Play Bench, a benchmark whose rules are novel or counterintuitive, requiring agents to get knowledge through interaction rather than rely on pretrained knowledge. The empirical results show that (1) preserving evidence can matter more than compressing it, (2) top-performing human players outperform the best agent, and (3) better harnesses can improve agent performance and learning while reducing inference cost.
Critical strengths
The paper tackles the clear and important problem of "distinguishing learning from interactions from reasoning vs. existing knowledge" in LLM agents. Specifically, Learn2Play Bench consists of 20 text games where each game has novel or counterintuitive rules.
Extensive experiments with a human baseline are included.
Interesting results such as (1) simple MEMORY wins over other self-evolving methods in some models, or (2) harness selection is as important as model selection
Critical weaknesses
First of all, "unfamiliarity" of 20 games in Learn2Play Bench is not verified. This is important because the purpose of this benchmark is to separate pretrained knowledge vs. learning from interaction. I know that it's hard to verify, but there is a good proxy: use the first episode score of the "Naive" method to classify familiar/unfamiliar games. For instance, the 'last and found' game seems easy for LLM agents, so it can be classified as a "familiar" game perhaps.
Next, I'm not sure if the comparison btw human vs. agent is fair enough. While abstract says "Top-performing human players achieve higher peak scores than the evaluated agents", this seems to be toned down or modified. This is because (1) the gap between Human Top-1 (Max) vs. Claude Code (Max) is within the margin of error, (2) Human Top-1 is scored from N>20 people, while the agent has only N=3 trials, and (3) the mean score of humans is much lower than that of the agent.
Questions for authors
I'm curious about the statistics for each game. For instance, (1) how many turns each game requires to complete (to see if it is a long-horizon task), (2) similarly, how many tokens do LLM agents spend to complete each game, (3) the amount of time for human participants to complete, or any statistics that help us understand the context behind the games and the benchmark.
Another tone down: Again, I'm not certain if it is ok to write a headline like "Experience retention: Retaining complete records of actions and feedback can support more effective learning than summarizing these experiences into rules or strategies". This currently applies only to the Kimi K3 case in Table 1, and I would be happy to see whether this tendency holds for other LLMs.
Minor typo: Humansss, requiresss
Thank you for the additional analyses. Target coverage is only weakly correlated with Learning Gain and Learning Slope. Score gains alone therefore remain insufficient evidence of rule discovery. The weight and normalization checks are useful, but robustness across game seeds and a resource-matched memory comparison remain open. I maintain my rating.