acceptodds
Under review as a conference paper at ICLR 2027

Cogito, Ergo Ludo: Learning External Language States for Interactive Agents

Abstract

Reinforcement learning can produce effective behavior from interaction, but the knowledge it acquires is often stored implicitly in model parameters rather than as inspectable rules or strategies. We introduce the External Language-State Agent (ELSA), an LLM-agent framework for prompt-level unknown-rule interaction, where task names, hand-written rules, and demonstrations are not provided in the prompt. ELSA represents learned rules and strategies as an external language state initialized without task-specific content. During each episode, this state is fixed and conditions a language-based value function, a language-based world model, and action selection. Across episodes, post-episode reflection revises the external state, while GRPO trains the LLM to better use the evolving state from sparse outcome feedback. Across four symbolic, text-observable environments, ELSA improves over name-free prompting and static rule-given prompting controls while producing inspectable external-state artifacts. Experiments and controls show that these gains depend on iterative external-state refinement and learned use of that state, rather than static prompting or policy optimization alone. These results suggest a route toward interactive LLM agents whose task competence is accompanied by explicit, inspectable, and reusable language-level knowledge.

Primary area:
applications to robotics, autonomy, planning
Keywords:
LLM Agent, Agent memory, Language-based world models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dpXrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

To store knowledge implicitly in model parameters rather than inspectable rules or strategies, the authors propose External Language-State Agent (ELSA), a framework to infuse the external state into LLM agents, and also update such external state in the framework as well. The co-evolving structure enable Qwen3-4B-Instruct to perform better in four symbolic, text-observable environments, also more effective than baselines like GRPO.

Critical strengths

1.The entire framework is simple, straightforward yet effective. The setup and the control make the problem well defined, and the results in Figure 3 shows the that ELSA beats popular baselines like GRPO. Table 1 shows that adding the game name alone raises zero-shot success, and randomizing the board symbols barely changes ESLA. I believe those evidence combined potentially shows that ELSA is effective to enable agents learn from interaction rather than recalled from pretraining. 2. I like the idea of changing the rules and the training those books into the agent through policy updates.

Critical weaknesses
  1. The novelty is somewhat limited, at least the idea of learning environment and dynamic rules and strategies from interaction. For instance, WALL-E [1]. As the angle is not entirely novel, given that the entire framework is simple, the novelty contribution is somewhat limited.

  2. I think the related work should be shorter and the Figure 3 should be substantially larger.

[1] Zhou, Siyu, et al. "Wall-e: World alignment by rule learning improves world model-based llm agents." arXiv preprint arXiv:2410.07484 (2024).

Questions for authors

Q1. How would you clarify the difference of angle and methodology between this work and WALL-E [1] (this type of work, there are some papers out there)?

[1] Zhou, Siyu, et al. "Wall-e: World alignment by rule learning improves world model-based llm agents." arXiv preprint arXiv:2410.07484 (2024).

3 comments

Q1: The novelty is somewhat limited, at least the idea of learning environment and dynamic rules and strategies from interaction. For instance, WALL-E [1]. As the angle is not entirely novel, given that the entire framework is simple, the novelty contribution is somewhat limited.

How would you clarify the difference of angle and methodology between this work and WALL-E [1] (this type of work, there are some papers out there)?

Thank you for recognizing ELSA's effectiveness and highlighting WALL-E. We agree that learning explicit rules from interaction is valuable prior work. ELSA studies how an agent can autonomously construct external knowledge and learn to use that evolving knowledge through policy training. The Rulebook and Playbook remain explicit inputs during both training and inference. GRPO trains the agent to make effective decisions conditioned on this evolving knowledge. We clarify the relationship in terms of research setting, decision mechanism, learning mechanism, and transfer capability.

DimensionWALL-EELSASupporting Evidence
1. Research settingUses environment-specific knowledge in rule-induction prompts to align world-model predictions with environment dynamics.Studies a prompt-level unknown-rule setting without task names, hand-written rules, or demonstrations. The Rulebook and Playbook are initially empty, and knowledge is acquired through interaction.WALL-E: Section 3.2 and Appendix B.1; ELSA: Section 4.1, Table 1.
2. Decision mechanismUses learned rules to correct predictions from a frozen world model, which guide MPC planning.Integrates explicit world-model reasoning into a trainable LLM's decision process, guided by the evolving Rulebook and Playbook. Specifically, the LWM predicts candidate actions' immediate state changes and outcomes, while the LVF assesses the current state's long-term potential. Both assessments jointly inform final action prediction. The shared LLM generates these assessments and the final action in one call, while GRPO optimizes the entire decision trace using outcome rewards, enabling world-model reasoning and decision-making capabilities to improve alongside evolving external knowledge and model parameters.WALL-E: Algorithm 1; ELSA: Section 3.2, Table 16 (LWM–LVF integration and ablations).
3. Learning mechanismPrimarily learns environmental rules for world-model alignment without policy-network training.Learns both environmental mechanics (Rulebook) and reusable decision strategies (Playbook) from interaction, coupling reflection-based knowledge revision with GRPO updates to the knowledge-conditioned policy. This enables the LLM agent to continuously refine its understanding of the environment while learning to make better decisions using the evolving knowledge. The same Qwen3-4B model performs decision-making, reflection, and merging without a stronger teacher.WALL-E: Figure 1, Algorithm 1; ELSA: Figure 2, Figure 4(Knowledge–policy co-evolution).
4. Transfer capabilityDemonstrates rule-based world-model alignment in Minecraft and ALFWorld.Additionally demonstrates cross-environment adaptation with frozen model parameters. We freeze the source-trained model, initialize the target Rulebook and Playbook as empty, and update only external knowledge through target interaction and reflection.Minesweeper → FrozenLake reaches 97.3%, and FrozenLake → Minesweeper reaches 46.9% (Table 2; Figure 5). The source-game books are not transferred, and no target-environment parameter updates are performed. These post-interaction results demonstrate that the trained agent can adapt through newly acquired task knowledge.WALL-E: Table 1 and 2; ELSA: Table 2 and Figure 5.

Thank you very much for bringing this valuable work to our attention. We have added WALL-E to Related Work and explicitly discussed its relationship to ELSA.

Q2: I think the related work should be shorter and the Figure 3 should be substantially larger.

Thank you for these helpful suggestions. We have streamlined the related work section and included an enlarged version of Figure 3 to improve readability.

[1] Zhou, Siyu, et al. "Wall-e: World alignment by rule learning improves world model-based llm agents." arXiv preprint arXiv:2410.07484 (2024).

Thank you for the comments. I maintain my rating.

Thank you for considering our response. We particularly appreciate your positive assessment of the core idea of jointly refining external knowledge and training the agent to use it effectively through policy updates. Your suggestion to discuss WALL-E helped us clarify ELSA’s contributions and its relationship to prior work in the revised manuscript. We sincerely appreciate your time and thoughtful review.

Reviewer_WVJZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ELSA for prompt-level unknown-rule interaction. ELSA maintains a Rulebook and a Playbook, revises them through reflection, and uses GRPO to train the policy with the evolving external language state. The framework is evaluated in four symbolic, text-observable environments.

Critical strengths

The alternating updates to the external language state and policy improve performance across four environments. Ablations show that both continued external-state refinement and GRPO contribute to these gains.

Critical weaknesses

A more systematic comparison with reflection memory combined with GRPO would help clarify the additional benefit of ELSA’s specific design. Appendix N compares frozen-policy reflection methods and GRPO with thinking without external memory, but does not test this combination. The paper also does not discuss how ELSA differs from related methods that update both external memory and the policy, such as UI-Mem and JERP.

Questions for authors

How does ELSA differ from UI-Mem and JERP in its joint refinement of external knowledge and the policy? How does it compare with a standard reflection-memory agent trained with GRPO under matched interaction and context budgets?

4 comments

Thank you for highlighting these related works. ELSA addresses autonomous exploration and self-evolution under unknown environment rules, with four key characteristics: (1) Autonomous exploration: The agent starts with empty Rulebook and Playbook, without task names, rules, or demonstrations; (2) Knowledge acquisition: Through interaction, the agent autonomously discovers environmental mechanics and reusable decision strategies; (3) Knowledge–policy co-evolution: Reflection continuously refines external knowledge, while GRPO trains the policy to make better decisions using the evolving knowledge; (4) Model usage: One shared model performs both decision-making and knowledge refinement, without a stronger teacher supplying knowledge. Together, this setting and learning process distinguish our contribution beyond jointly updating memory and policy.

Q1: ELSA vs. UI-Mem.

Thank you for highlighting UI-Mem as a relevant work. ELSA differs from UI-Mem in the following aspects.

DimensionUI-MemELSA
Knowledge Acquisition SettingUses annotated trajectories to define subtasks and extracts knowledge from existing interaction experiences.Requires no task names, predefined environment rules, or demonstrations. The Rulebook and Playbook start empty, and the agent autonomously acquires knowledge through interaction.
Knowledge ConstructionUses Seed1.8 for experience extraction and DeepSeek-V3 for experience abstraction, with annotated trajectories providing subtask definitions.Uses the same Qwen3-4B model to construct and revise knowledge while undergoing GRPO policy updates, without teacher-generated knowledge.
Knowledge Quality VerificationPrimarily evaluates task performance without directly checking whether learned rules accurately reflect environment mechanics.Explicitly evaluates both task performance and rule correctness. A two-stage proposal–merge procedure consolidates clear and consistent revisions while filtering ambiguous or conflicting changes. Manual inspection of all 25 entries in the final Minesweeper Rulebook shows 92.0% correctness and 88.9% coverage of reference environment facts (Table 14).

These results provide direct evidence that ELSA can autonomously discover largely correct environment rules without prior rule provision.

Q2: ELSA vs. JERP.

JERP combines experiential rule learning with policy optimization. ELSA differs in its unknown-rule setting, knowledge organization and revision, and knowledge-conditioned decision-making, as detailed below.

DimensionJERPELSA
Knowledge DiscoveryLearns rules from interaction, but does not specifically examine rule discovery without task names, rules, or demonstrations.Starts with empty Rulebook and Playbook under unknown-rule settings, autonomously discovering environmental mechanics and decision strategies through interaction.
Knowledge RepresentationMaintains a unified Experiential Rule Pool to store rules.Explicitly separates environment mechanics (Rulebook) from action strategies (Playbook), forming two distinct learning objects.
Knowledge RevisionUses reflection to generate rule-editing and merging operations.Separates revision proposals from conservative merging, consolidating consistent knowledge while filtering ambiguous or conflicting modifications.
Knowledge-conditioned Decision-makingRetrieves Top-K experiential rules as policy context to guide action generation, with GRPO optimizing the policy.Explicitly integrates learned knowledge into world-model reasoning and value-guided decision-making. The Rulebook and Playbook condition LWM consequence prediction and LVF long-term value assessment, which jointly guide action prediction. GRPO optimizes the entire structured decision trace using outcome rewards.
Model UseUses a frozen LLM for rule editing while updating the policy. The specific policy and rule-editing models are not disclosed.Uses the same Qwen3-4B model for decisions, reflection, and merging, without a stronger teacher. GRPO updates the shared parameters, which are also used for subsequent knowledge refinement.
Cross-environment GeneralizationCross-environment adaptation with newly constructed target rules is not reported.Demonstrates cross-environment adaptation with frozen policy parameters and initially empty target books. Through target-environment interaction and reflection alone, achieves 97.3% on Minesweeper → FrozenLake and 46.9% in the reverse (Table 2, Figure 5), without transferring source-game books or updating model parameters.

Q3: A more systematic comparison with reflection memory combined with GRPO would help clarify the additional benefit of ELSA’s specific design. Appendix N compares frozen-policy reflection methods and GRPO with thinking without external memory, but does not test this combination.

How does it compare with a standard reflection-memory agent trained with GRPO under matched interaction and context budgets?

Neither UI-Mem nor JERP has publicly released its official code. To directly answer your question, we have extended Table 15 with Reflexion + GRPO and ExpeL + GRPO on Minesweeper and Sokoban. These baselines retain their respective experience-maintenance and use mechanisms while adding GRPO, without adopting ELSA’s two-stage proposal–merge procedure, Rulebook/Playbook separation, or prescribed LWM/LVF reasoning.

ELSA, Reflexion + GRPO, and ExpeL + GRPO use the same environment-interaction budget and identical token limits for decision inputs and external memory.

To stabilize policy training under evolving memory, ELSA introduces an interval-based update schedule that keeps external knowledge fixed within each five-episode training interval. Reflexion + GRPO and ExpeL + GRPO also use this update schedule, while retaining their respective memory-update mechanisms between intervals.

The table below retains the existing Table 15 results and adds the two GRPO baselines.

MethodMinesweeper success (%)Sokoban success (%)
Static rule-given prompting (reference)25.873.0
Reflection / memory with a frozen policy
Reflexion31.347.7
ExpeL34.860.5
Voyager-style29.756.3
ELSA without GRPO36.376.2
External-state refinement and policy training
Reflexion + GRPO42.458.3
ExpeL + GRPO46.372.2
Full ELSA53.584.0

Both baselines benefit from GRPO, yet ELSA consistently outperforms them under the same interval-based memory-update schedule. The key difference lies in how knowledge is organized, refined, and used for decision-making. Unlike standard reflection memory, ELSA separates environmental mechanics from decision strategies, maintains them through proposal and conservative merging, and uses them to condition LWM consequence prediction and LVF value assessment. GRPO then trains the policy to use this evolving knowledge through explicit reasoning. These results demonstrate the additional benefit of ELSA's overall design beyond combining reflection memory with GRPO.

Thank the authors for the additional experiments. These results address my main concern regarding the lack of comparisons with reflection-based memory combined with DRPO. Elsa outperforms baselines that simply combine these two mechanisms. I will raise my score accordingly.

Thank you for your constructive feedback and updated assessment. We are glad that the additional comparisons with reflection-based memory trained with GRPO address your main concern. Your suggestions have helped us strengthen the revised manuscript through additional experiments and clearer discussions of the differences between ELSA, UI-Mem, and JERP, making our contributions more explicit. We sincerely appreciate your time and thoughtful review.

Reviewer_jSaAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ELSA, which is an LLM agent that has an external language state consisting of a "rulebook" containing inferred mechanics and a "playbook" comprising various strategies. This state begins off-empty and is updated every five episodes using a reflect-then-merge procedure. The same model is trained in parallel with GRPO using sparse outcome rewards, with the training conditioned on the current state. At each step, the model produces a brief value assessment, one-step predictions for two possible actions, and selects an action. The experiments include Minesweeper, FrozenLake, Sokoban, and 9x9 Go played against a random opponent, with the names of the games and their rules being kept hidden from the prompt. ELSA performs better than versions that do not use game names and those that are given the rules.

Critical strengths

It is a worthwhile question to ask whether or not interaction can lead to the acquisition of task knowledge that is explicit and editable, alongside the policy being trained to make use of it.

The state-intervention controls are convincing; when the policy is fixed and the learned state is removed or replaced by text of the same length, success drops by 25 to 35 points (see Table 12).

Critical weaknesses

Positioning. A reflective playbook with incremental merging is close to ACE, Training-Free GRPO, and AutoManual. Multi-turn GRPO on Sokoban/FrozenLake with explicit state and transition reasoning is close to RAGEN and VAGEN. None of these are discussed or compared. The only new contribution is running context refinement and GRPO together. However, the reflection and merge calls receive no learning signal, so “co-evolution” amounts to two procedures running side by side. LVF and LWM are prompt fields rather than trained value or world models, and removing them costs only 1 to 4 points (Table 16). The evaluation doesn't currently support the figures in the headline. The key findings seem to have been obtained using the 32 training layouts, while the results from the held-out data (in Table 2) are lower and given for only two of the games. The policy-only baseline appears to be flawed. While the base model achieves 12.9% zero-shot accuracy on Minesweeper, the action only GRPO method shows 0/32 mixed-outcome groups even when using 64 rollouts and having first-action diversity between 81 and 97%. It is very unlikely that this situation occurs unless there is a failure of the rollouts due to the format or parsing. Furthermore, Figure 11 displays 3 all success groups, which goes against the claim of 0%. The SFT result at 0% is lower than that of the untrained base. The tasks involved are small and since the Go opponent makes completely random legal moves, a win rate of 65.6% doesn't indicate much about long-term strategy.

Questions for authors

Has the main results been calculated using the training layouts? You should provide the results from the held-out, final-checkpoint tests for all four environments, based on at least three different training seeds.

With respect to action-only GRPO, what is the distribution of the reasons for episode termination (i.e. format error, invalid action, mine, step limit)? How does it come about that the three 'All' groups in Figure 11 have zero percent success?

Are the decision prompts the same in different environments? The '(row, col)' action format in Figure 14 seems to be specific to Minesweeper. Was the randomized-symbol run trained from scratch and on which environments exactly? Table 1 and Appendix F contradict each other.

Does the reflection and merge generations receive any gradients? And if not, what is the mechanical meaning of 'co-evolution'?

4 comments

Thank you for your detailed feedback and for recognizing the value of learning explicit, editable task knowledge alongside the policy, as well as the evidence from our state-intervention controls. Below, we clarify the evaluation protocol, baseline behavior, prompt design, and co-evolution mechanism.

Q1: Differences from prior work

These works emphasize either external-context adaptation with fixed model parameters (ACE, Training-Free GRPO, and AutoManual) or multi-turn policy learning and state/transition reasoning (RAGEN and VAGEN). ELSA studies the joint discovery of unknown task knowledge and learning of its use in decisions. Starting with empty books and no supplied task names, rules, or demonstrations, the same Qwen3-4B model discovers and revises environment mechanics and action strategies, while GRPO trains the policy to apply them in value assessment, consequence prediction, and action selection.

(1) Learning both knowledge and how to use it. ACE and AutoManual improve playbooks or environment manuals while keeping model parameters fixed. ELSA also trains the shared model to apply its self-discovered knowledge: mechanics and strategies are revised in separate books through proposal and conservative merging, and GRPO optimizes their use in LVF/LWM reasoning and action selection. AutoManual’s main setup starts with initial rules manually derived from an example; ELSA begins with empty books and acquires task knowledge without supplied rules or demonstrations.

Training-Free GRPO performs inference-only experience editing without parameter updates, and generates semantic advantages only for groups with clear winners and losers. ELSA’s knowledge refinement can proceed even when all trajectories fail and outcome rewards are identical: observed state transitions and failure experience still provide evidence for revising rules and strategies. This allows knowledge acquisition to continue when outcome comparisons provide no relative-advantage signal.

(2) Learning through unknown-rule interaction over longer episodes. RAGEN aims to optimize policies from multi-turn trajectories. ELSA additionally accumulates experience in explicit, editable rules and strategies across episodes, allowing interaction to improve both the available knowledge and the policy that uses it. Our evaluation also covers longer episodes: RAGEN’s reported main setup uses at most 5 interaction turns and 10 actions per episode, while ELSA learns in a Go-like environment with a horizon of 100 moves and sparse terminal feedback.

VAGEN supplies task names and rules in its game-training prompts, and VAGEN-Full uses ground-truth states to reward the correctness of state descriptions and predictions at each turn. ELSA learns without either supplied task rules or an additional prediction-correctness reward: the agent discovers environment mechanics and strategies from its own interactions, while sparse outcome rewards train their use in decisions. ELSA therefore demonstrates autonomous knowledge acquisition and learned use without these additional sources of task knowledge and reasoning supervision.

Q2: LVF/LWM: implementation and empirical contribution

(1) Implementation. LVF and LWM implement qualitative state-value assessments and one-step consequence predictions within the shared LLM. Their structure is specified by the prompt. GRPO uses outcome rewards to optimize the complete decision trace, including LVF assessments, LWM predictions, and action selection (see Sections 3.2–3.4). These functions are thus trained as part of the knowledge-conditioned policy, without separate value-regression or transition-prediction losses.

(2) Prediction validity. The LWM’s modeling function is evaluated by whether its action-conditioned predictions match real environment transitions. Table 8 directly tests this on 100 Sokoban decisions from the final checkpoint, sampled equally from successful and failed episodes: predictions for chosen actions are 79% correct, 12% partially correct, and 9% incorrect. This checks the correctness of the generated predictions against environment dynamics and provides evidence that the LWM performs its intended prediction function.

(3) Empirical contribution. With external knowledge refinement and GRPO retained, removing either LVF or LWM reduces performance across all four environments (Table 16). These results support their incremental contributions to knowledge-conditioned decision-making.

SettingFrozenLakeMinesweeperSokobanGo-like
Full ELSA97.353.584.065.6
Without LVF95.751.281.661.7
Without LWM94.549.280.163.3
State constructed once, then frozen82.815.260.945.3
Without GRPO (frozen policy)87.936.376.254.7

Q3: Evaluation on training and unseen layouts

The main learning curves evaluate task learning on 32 fixed training layouts, starting from empty books and jointly refining external knowledge and the policy. Table 2 separately evaluates generalization on unseen layouts using the final training checkpoint. We include the additional Sokoban result below.

EnvironmentTraining-layout success (%)Unseen-layout success (%)Drop (percentage points)
Minesweeper53.550.43.1
FrozenLake97.393.83.5
Sokoban84.082.41.6

The small performance decreases show that the learned policy and knowledge remain effective beyond the training layouts. Go always starts from an empty board and is evaluated through games against a random legal-move opponent, so a train/test split by initial board layout is unnecessary.

Q4: Episode termination and mixed-outcome groups

(1) Episode termination and valid-action sampling. We observe that action-only GRPO predominantly fails by repeating actions until the step limit. We additionally sample directly from the valid-action set according to model-assigned probabilities (Appendix E.1). This increases first-action diversity from 81.3% to 96.9%, yet neither sampler produces a mixed-outcome group among the 32 inspected groups. Thus, valid-action sampling and greater initial-action diversity do not resolve the lack of outcome variation in this diagnostic. Without supplied rules or accumulated knowledge, diverse valid actions alone do not ensure effective exploration. ELSA can extract useful knowledge even from failed interactions: in Sokoban, pushing a box into a non-goal corner and observing that it cannot be moved out provides evidence for avoiding such deadlocks, guiding subsequent exploration before any successful trajectory is obtained.

(2) The “All” groups in Figure 11. Thank you for pointing out this ambiguity. The three all-success groups in Figure 11 do not conflict with the absence of mixed-outcome groups. A mixed group contains both successes and failures; an all-success group contains only successes. All-success and all-failure groups both have uniform rewards, leaving no within-group reward differences for GRPO’s relative-advantage computation. Table 6 records 0 mixed groups out of 32: the mixed-group proportion is therefore 0%, which does not imply zero episode success. We will remove the “Success (%)” column and correct the accompanying text to distinguish these quantities.

(3) SFT generalization. In this setting, the SFT-trained model relies on memorized solutions to the training layouts and fails to generalize to unseen boards, resulting in 0% episode success on held-out layouts. We varied the training data size, number of training steps, and fine-tuning method, including LoRA, but this behavior persisted.

Q5: Decision prompts and randomized-symbol experiments

(1) Shared prompt template. All environments use the same decision-prompt template for LVF assessment, LWM consequence prediction, and action selection. Only the action-format instructions and examples differ: FrozenLake and Sokoban use directions such as Left, while Minesweeper and Go use coordinates such as (row, col). Figure 14 shows the coordinate-action version. We will clarify its scope in the caption and add a directional-action example. These instructions specify the interaction interface without supplying game rules or strategies.

(2) Randomized-symbol training. Group B reports ELSA trained with randomized symbols in FrozenLake, Minesweeper, and Sokoban. Each randomized-symbol run restarts task training from the same pretrained backbone with empty Rulebook and Playbook, and uses randomized symbols throughout training and evaluation. Appendix F explains the experiment using Minesweeper as an example, while Table 1 reports all three environments. We will make this scope explicit and extend Appendix F to describe FrozenLake and Sokoban.

Q6: Gradients and the mechanism of co-evolution

Reflection and merge outputs receive no direct GRPO loss. ELSA updates policy parameters through GRPO on decision traces and outcome rewards, and revises external books through reflection and merging of interaction experience. Both operations use the same model, so policy training also updates the parameters used for subsequent reflection and merging.

“Co-evolution” describes how these updates depend on each other. The current books guide action selection and thus shape the trajectories used for GRPO. The updated policy generates new successes and failures that inform book revisions. The revised books then guide subsequent rollouts and policy updates. Knowledge shapes policy learning, while policy-generated experience shapes knowledge refinement.

Table 17 demonstrates the benefit of this continued refinement: with the initial policy, initial books, and GRPO procedure held fixed, success reaches 54.3% when knowledge is continually revised, compared with 22.7% when the books are frozen. Revising knowledge during policy learning therefore improves on training with a fixed knowledge state.

Dear Reviewer jSaA,

Thank you again for your thoughtful review. We have provided responses to your questions and updated the manuscript accordingly. When you have a chance, we would appreciate your feedback and would be happy to address any remaining concerns. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.