- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
To store knowledge implicitly in model parameters rather than inspectable rules or strategies, the authors propose External Language-State Agent (ELSA), a framework to infuse the external state into LLM agents, and also update such external state in the framework as well. The co-evolving structure enable Qwen3-4B-Instruct to perform better in four symbolic, text-observable environments, also more effective than baselines like GRPO.
Critical strengths
1.The entire framework is simple, straightforward yet effective. The setup and the control make the problem well defined, and the results in Figure 3 shows the that ELSA beats popular baselines like GRPO. Table 1 shows that adding the game name alone raises zero-shot success, and randomizing the board symbols barely changes ESLA. I believe those evidence combined potentially shows that ELSA is effective to enable agents learn from interaction rather than recalled from pretraining. 2. I like the idea of changing the rules and the training those books into the agent through policy updates.
Critical weaknesses
-
The novelty is somewhat limited, at least the idea of learning environment and dynamic rules and strategies from interaction. For instance, WALL-E [1]. As the angle is not entirely novel, given that the entire framework is simple, the novelty contribution is somewhat limited.
-
I think the related work should be shorter and the Figure 3 should be substantially larger.
[1] Zhou, Siyu, et al. "Wall-e: World alignment by rule learning improves world model-based llm agents." arXiv preprint arXiv:2410.07484 (2024).
Questions for authors
Q1. How would you clarify the difference of angle and methodology between this work and WALL-E [1] (this type of work, there are some papers out there)?
[1] Zhou, Siyu, et al. "Wall-e: World alignment by rule learning improves world model-based llm agents." arXiv preprint arXiv:2410.07484 (2024).
3 comments
Thank you for the comments. I maintain my rating.
Thank you for considering our response. We particularly appreciate your positive assessment of the core idea of jointly refining external knowledge and training the agent to use it effectively through policy updates. Your suggestion to discuss WALL-E helped us clarify ELSA’s contributions and its relationship to prior work in the revised manuscript. We sincerely appreciate your time and thoughtful review.
Thank you for recognizing ELSA's effectiveness and highlighting WALL-E. We agree that learning explicit rules from interaction is valuable prior work. ELSA studies how an agent can autonomously construct external knowledge and learn to use that evolving knowledge through policy training. The Rulebook and Playbook remain explicit inputs during both training and inference. GRPO trains the agent to make effective decisions conditioned on this evolving knowledge. We clarify the relationship in terms of research setting, decision mechanism, learning mechanism, and transfer capability.
Thank you very much for bringing this valuable work to our attention. We have added WALL-E to Related Work and explicitly discussed its relationship to ELSA.
Thank you for these helpful suggestions. We have streamlined the related work section and included an enlarged version of Figure 3 to improve readability.
[1] Zhou, Siyu, et al. "Wall-e: World alignment by rule learning improves world model-based llm agents." arXiv preprint arXiv:2410.07484 (2024).