- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces dual-loop agents, a self-improvement framework coupled with an upper RL loop and a lower RL loop, corresponding to high-level strategy and low-level control, and instantiates the framework as PREinforce. The upper RL loop is implemented using a frozen foundation model that constructs the program for goal generation. Based on human-supplied game knowledge, online returns, and trajectories, the foundation model constructs and improves a strategic program. The program provides a goal-outcome interface that performs goal selection and outcome labeling in the decision-making process. The lower RL loop follows the interface and uses TD InfoNCE to train a goal-conditioned neural policy. The experiment on Atari 100k demonstrates that PREinforce outperforms the human reference score in all 26 evaluated games.
Critical strengths
Instead of using a foundation model itself to select the goal, constructing an editable program for goal selection could efficiently reduce the computation cost, which could be a promising way to integrate a foundation model into hierarchical RL approaches.
Critical weaknesses
The main claim of introducing the dual-loop agent architecture is not justified well. Several works already use the hierarchical architecture with LLMs to provide high-level plans/goals and RL policies to execute them, such as the Plan-Seq-Learn introduced in the related work. The main comparison is also insufficient. I agree with the novelty of using an editable program to reduce the cost of calling an LLM. However, the comparison does not include comparable architectures, e.g., direct LLM goal generation vs program-based goal generation.
The proposed method seems heavily dependent on human-supplied game knowledge, which limits the generality. Most recent works on Atari, e.g., the compared EfficientZero V2, learn to play without human knowledge; further adding knowledge sounds like a step back. There is also no baseline to demonstrate the performance of PREinforce without supplying game knowledge at first. Furthermore, now that an LLM is used, using it to discover game knowledge would be a more convincing approach, but it is not discussed.
Questions for authors
- How specific or how general can the editable program be? From my understanding, finding subgoals for Atari games is a nontrivial task that sometimes requires learning-based methods with NNs for extracting image features and constructing the subgoals. It is not really clear to me how this complicated process can be distilled into a Python program.
- Please justify why some prior-guided RL works cannot be applied to the Atari domain, as their proposed frameworks seem similar. For example, although not applied to Atari games, the MaestroMotif work also uses a game environment called NetHack.
- Why cannot leverage LLM world knowledge and reasoning abilities to observe the game knowledge? If solving a task requires a lot of human-supplied domain knowledge for constructing the program, its success can be considered highly reliant on human knowledge.
Response to Reviewer FBxZ
[W1] Contribution beyond reducing model calls
Thank you for recognizing the value of editable programs. Our shared goal–outcome interface makes the program responsible for both goal generation and achieved-outcome labeling, allowing strategy revision and primitive-action learning to use the same experience. The program represents strategic knowledge and supplies supervision for the lower-level learner, contributing directly to the acquisition and continual improvement of control capabilities.
[W1, Q2] Comparison with Plan-Seq-Learn
Plan-Seq-Learn also learns low-level control online, but relies on visual pose estimation and motion planning to bring the robot to the initial region for each stage. Its Appendix B.3 identifies limitations in dynamic tasks and when high-level plans are incorrect, and discusses online updates to the Plan and Seq modules as future work. Atari's image observations and discrete-action interface do not directly provide this motion-planning capability. PREINFORCE jointly addresses goal grounding and execution learning through local goals, achieved-outcome supervision, and feedback-driven program revision. Plan-Seq-Learn
[Q1] How programs construct subgoals
The shared elements are the interfaces for perception, motion prediction, planning, and outcome labeling. The model generates game-specific implementations using knowledge and online evidence. In Ms. Pac-Man, for example, the program uses colors, connected components, and geometric relationships to locate objects. It combines successive observations to update motion predictions, plan routes, and generate local goals, while the executor learns from images how to reach them. Debug visualizations enable inspection of estimated positions and routes. The current experiments use game-specific programs. More complex visual settings could incorporate perceptual models, as discussed in Section 6.
[Q2] Comparison with MaestroMotif
MaestroMotif requires the designer to specify a skill set in advance and provide natural-language descriptions of each skill, together with high-level specifications of initiation and termination conditions and how skills should be interleaved during training (Sections 3.1 and 3.3). It then learns skill rewards from LLM preferences, trains skills with RL, and generates code to compose them. Its Section 6 explicitly acknowledges that the behaviors it can express are constrained by the selected skill set. Its NetHack implementation also uses game messages, character statistics, character encodings, and previously collected trajectories. In our Atari setting, the corresponding semantic information must be derived from images, and the relevant exploration and learning must fit within the 100,000-interaction budget per game. Adapting MaestroMotif therefore still requires addressing skill-set design, semantic extraction, and control supervision. PREINFORCE uses a revisable program to generate local goals and label outcomes, trains a shared goal-conditioned executor to learn primitive actions, and adjusts the strategy using execution feedback, providing a concrete approach for this setting. MaestroMotif
[W2, Q3] Amount and role of human-provided knowledge
In the current v8 implementation, the human-provided instructions are brief, averaging approximately 700 characters per game in the original Chinese text. This counts the body text, including rules, strategic advice, and implementation suggestions. The system must still turn these descriptions into executable programs and use interaction to calibrate perception, revise strategies, and learn control. With the same human-provided knowledge, code policies exceed the human reference on 14/26 games and the full method on 26/26, demonstrating the importance of how knowledge is translated into behavior. Our study investigates the integration of priors with online experience. The 100,000-step budget measures online interactions, and the difference in prior information relative to the RL baselines is explicitly disclosed.
[W1, W2, Q3] Model knowledge and ongoing comparisons
The model's own knowledge already contributes to development. Appendix B.2 instructs it to use both supplied rules and its own priors, calibrate action effects through observation, and test and revise the program. We are conducting the requested direct-LLM-goal-generation comparison and an ablation removing the human-provided prompts. These experiments remain in progress owing to token-budget and time constraints. Their results will further clarify the respective benefits and the influence of the source of prior knowledge.