acceptodds
Under review as a conference paper at ICLR 2027

Dual-Loop Agents: Human-Level Atari via Prior-Guided Self-Improvement

Abstract

General self-improving agents must turn prior knowledge and limited online experience into effective action. However, existing approaches expose a tension: reward-driven reinforcement learning (RL) may fail to discover long-horizon strategies, whereas foundation-model priors often leave primitive actions unspecified. We introduce _dual-loop agents_, a self-improvement framework unifying priors and online experience through two coupled RL loops for learning high-level strategy and primitive control, linked by a shared goal–outcome interface. We instantiate this framework as PREinforce: a frozen foundation model constructs and revises an editable strategic program from online feedback, while a goal-conditioned neural policy learns fine-grained execution through temporal-difference contrastive learning. With 100,000 online interactions per game, PREinforce is, to our knowledge, the first agent to exceed the human reference in per-game mean score on all 26 Atari 100k games.

Primary area:
reinforcement learning
Keywords:
Agentic self-improvement, Sample-efficient reinforcement learning, Prior-guided learning, Hierarchical reinforcement learning, Contrastive reinforcement learning, Programmatic policies, Atari 100k

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FBxZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces dual-loop agents, a self-improvement framework coupled with an upper RL loop and a lower RL loop, corresponding to high-level strategy and low-level control, and instantiates the framework as PREinforce. The upper RL loop is implemented using a frozen foundation model that constructs the program for goal generation. Based on human-supplied game knowledge, online returns, and trajectories, the foundation model constructs and improves a strategic program. The program provides a goal-outcome interface that performs goal selection and outcome labeling in the decision-making process. The lower RL loop follows the interface and uses TD InfoNCE to train a goal-conditioned neural policy. The experiment on Atari 100k demonstrates that PREinforce outperforms the human reference score in all 26 evaluated games.

Critical strengths

Instead of using a foundation model itself to select the goal, constructing an editable program for goal selection could efficiently reduce the computation cost, which could be a promising way to integrate a foundation model into hierarchical RL approaches.

Critical weaknesses

The main claim of introducing the dual-loop agent architecture is not justified well. Several works already use the hierarchical architecture with LLMs to provide high-level plans/goals and RL policies to execute them, such as the Plan-Seq-Learn introduced in the related work. The main comparison is also insufficient. I agree with the novelty of using an editable program to reduce the cost of calling an LLM. However, the comparison does not include comparable architectures, e.g., direct LLM goal generation vs program-based goal generation.

The proposed method seems heavily dependent on human-supplied game knowledge, which limits the generality. Most recent works on Atari, e.g., the compared EfficientZero V2, learn to play without human knowledge; further adding knowledge sounds like a step back. There is also no baseline to demonstrate the performance of PREinforce without supplying game knowledge at first. Furthermore, now that an LLM is used, using it to discover game knowledge would be a more convincing approach, but it is not discussed.

Questions for authors
  1. How specific or how general can the editable program be? From my understanding, finding subgoals for Atari games is a nontrivial task that sometimes requires learning-based methods with NNs for extracting image features and constructing the subgoals. It is not really clear to me how this complicated process can be distilled into a Python program.
  2. Please justify why some prior-guided RL works cannot be applied to the Atari domain, as their proposed frameworks seem similar. For example, although not applied to Atari games, the MaestroMotif work also uses a game environment called NetHack.
  3. Why cannot leverage LLM world knowledge and reasoning abilities to observe the game knowledge? If solving a task requires a lot of human-supplied domain knowledge for constructing the program, its success can be considered highly reliant on human knowledge.
1 comment
AuthorsOpenReview ↗

Response to Reviewer FBxZ

[W1] Contribution beyond reducing model calls

Thank you for recognizing the value of editable programs. Our shared goal–outcome interface makes the program responsible for both goal generation and achieved-outcome labeling, allowing strategy revision and primitive-action learning to use the same experience. The program represents strategic knowledge and supplies supervision for the lower-level learner, contributing directly to the acquisition and continual improvement of control capabilities.

[W1, Q2] Comparison with Plan-Seq-Learn

Plan-Seq-Learn also learns low-level control online, but relies on visual pose estimation and motion planning to bring the robot to the initial region for each stage. Its Appendix B.3 identifies limitations in dynamic tasks and when high-level plans are incorrect, and discusses online updates to the Plan and Seq modules as future work. Atari's image observations and discrete-action interface do not directly provide this motion-planning capability. PREINFORCE jointly addresses goal grounding and execution learning through local goals, achieved-outcome supervision, and feedback-driven program revision. Plan-Seq-Learn

[Q1] How programs construct subgoals

The shared elements are the interfaces for perception, motion prediction, planning, and outcome labeling. The model generates game-specific implementations using knowledge and online evidence. In Ms. Pac-Man, for example, the program uses colors, connected components, and geometric relationships to locate objects. It combines successive observations to update motion predictions, plan routes, and generate local goals, while the executor learns from images how to reach them. Debug visualizations enable inspection of estimated positions and routes. The current experiments use game-specific programs. More complex visual settings could incorporate perceptual models, as discussed in Section 6.

[Q2] Comparison with MaestroMotif

MaestroMotif requires the designer to specify a skill set in advance and provide natural-language descriptions of each skill, together with high-level specifications of initiation and termination conditions and how skills should be interleaved during training (Sections 3.1 and 3.3). It then learns skill rewards from LLM preferences, trains skills with RL, and generates code to compose them. Its Section 6 explicitly acknowledges that the behaviors it can express are constrained by the selected skill set. Its NetHack implementation also uses game messages, character statistics, character encodings, and previously collected trajectories. In our Atari setting, the corresponding semantic information must be derived from images, and the relevant exploration and learning must fit within the 100,000-interaction budget per game. Adapting MaestroMotif therefore still requires addressing skill-set design, semantic extraction, and control supervision. PREINFORCE uses a revisable program to generate local goals and label outcomes, trains a shared goal-conditioned executor to learn primitive actions, and adjusts the strategy using execution feedback, providing a concrete approach for this setting. MaestroMotif

[W2, Q3] Amount and role of human-provided knowledge

In the current v8 implementation, the human-provided instructions are brief, averaging approximately 700 characters per game in the original Chinese text. This counts the body text, including rules, strategic advice, and implementation suggestions. The system must still turn these descriptions into executable programs and use interaction to calibrate perception, revise strategies, and learn control. With the same human-provided knowledge, code policies exceed the human reference on 14/26 games and the full method on 26/26, demonstrating the importance of how knowledge is translated into behavior. Our study investigates the integration of priors with online experience. The 100,000-step budget measures online interactions, and the difference in prior information relative to the RL baselines is explicitly disclosed.

[W1, W2, Q3] Model knowledge and ongoing comparisons

The model's own knowledge already contributes to development. Appendix B.2 instructs it to use both supplied rules and its own priors, calibrate action effects through observation, and test and revise the program. We are conducting the requested direct-LLM-goal-generation comparison and an ablation removing the human-provided prompts. These experiments remain in progress owing to token-budget and time constraints. Their results will further clarify the respective benefits and the influence of the source of prior knowledge.

Reviewer_m7QoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces dual-loop agents, a hierarchical framework geared to self-improving agents. It combines high-level program revision with low-level control learning, and is implemented as PREinforce, which uses a foundation model to revise a program that will select goals/label outcomes, and a low-level neural controller that learns the primitive actions to complete the program (achieve the selected goals). Evaluation on Atari 100k reports greater than human reference per-game mean scores.

Critical strengths
  • Strong empirical performance. Practical value of the framework is demonstrated by its strong Atari performance and improvements over both RL/code-only baselines. The results support combining programmatic guidance with some learned control.
  • Evidence of online improvement. Frozen-program ablation serves to highlight the importance of revision the program as experience accumulates. Supports value of using interaction feedback in an online manner to improve the agent beyond it's prior-guided program.
Critical weaknesses
  • Limited architectural novelty and isolation of new contributions. Hierarchical frameworks like the propose method are not new and is well established in the RL field, including components like LM-guidance/execution feedback. The contribution therefore rests on the editable goal–outcome interface and contrastive executor, whose benefit over a conventional goal-conditioned learner using the same interface is not established.
  • Reliability of headline Atari results. Each game is evaluated using a single trained agent, and it is unclear whether repeating program generation/controller training produces consistently strong performance. This leaves room for uncertainty in the reliability of the framework.
Questions for authors
  • How does a conventional goal-conditioned RL controller perform using the same program interface, supplied knowledge, and interaction budget?
  • Do the reported results hold when the full development process, including program generation, revision, and controller training, is repeated independently for each game?
1 comment
AuthorsOpenReview ↗

Response to Reviewer m7Qo

[W1] Interface contribution and empirical evidence

Thank you for recognizing the empirical performance and the role of online improvement. Our central contribution is the revisable goal–outcome interface. The upper level converts priors into local goals and labels achieved outcomes from trajectories. The lower level learns primitive actions from these labels, while the upper level revises the program using execution feedback. The same experience therefore improves both strategy and control. Under the same model, prompts, and interaction budget, code policies exceed the human reference on 14/26 games, the full method on 26/26, and the frozen-initial-program variant on 2/26, supporting the practical value of this mechanism.

[W1, Q1] Comparison with a conventional goal-conditioned controller

We are conducting an additional comparison that replaces only the lower-level learning algorithm while holding the interface, priors, and interaction budget fixed. This experiment is ongoing and has not yet been completed owing to token-budget and time constraints. Our current executor predicts program-defined future outcomes, allowing it to learn from trajectories that miss the requested goal or receive no task reward (Section 4.3). The existing BBF-with-auxiliary-reward experiment examines another way to incorporate priors. The relative performance of controllers using the same interface will be assessed through the ongoing comparison.

[W2, Q2] Reproducibility of the complete development process

The current results use one development run per game, followed by evaluation of the frozen program and executor on five seeds unseen during development. These evaluation results are used solely for reporting (Appendix D.1). The 26/26 result refers to the per-game mean scores in this evaluation. We are conducting additional independent repetitions of the full process, including program generation, online revision, and executor training, to assess variability across development runs. Complete development requires substantial model usage and training time. These repetitions remain in progress owing to token-budget and time constraints.

Reviewer_bQYVOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • The paper bridges primitive actions learned by RL policies with high-level strategies from an LLM to improve sample efficiency. The LLM emits the intermediate goals and a goal-conditioned neural policy learns to reach them.
  • A frozen GPT model writes and keeps revising a Python program for each game. A “Developer” LLM writes the code and an “Auditor” LLM reviews gameplay and decides whether to keep or roll back each change.
  • The executor predicts outcomes labeled by the program. No low-level reward function is hand-designed.
  • Results: on Atari 100k, the agent beats the human reference on all 26 games.
Critical strengths
  • No LLM calls at inference time.
  • Experiment with initial program frozen shows online program revision matters.
  • Paper includes abridged prompt templates, token and wall-clock costs and held-out evaluation seeds.
Critical weaknesses
  • The authors claim that the main results are based on one development run per game. But a second run with the LLM could write different code and have different updates. The length could also be varied dramatically. Without more run, the current result is unconvincing. Also, how much of the performance gain is from human hint, or GPT's pretrained knowledge of Atari, or the interaction itself? Also what about other models? It is unclear how exactly does using LLM contribute to the performance gain. There need to be more elaborate experiment and careful analysis.

  • The sample-efficiency comparison neglected budget on human-written game knowledge and time needed for agentic development when compared with RL baselines with no prior knowledge.

  • The example is toy. Section 3 reduces Private Eye to a 1D random walk. The 2D movement, jumping, scene changes and hazards are all abstracted away.

  • Not novel. The framework is very similiar to SayCan, ELLM and similiar code as policies and language as abstraction HRL methods. Although the paper cites all of these, but also the lower loop is only a small modification/extension of existing contrastive goal-conditioned RL.

  • Moreover, the paper provides no significant new knowledge. The main takeaway regarding sample efficiency is expected while the theory is a stardard hoedffding/martingale bound on a biased random walk. The theory provided little explanation on why this particular system works but rather only shows a random walk rarely reaches a distant target.

Questions for authors
  • How much of the gain came from LLM-written program and how much from the learned control?
  • Can you report the compute costs for all of the 26 games rather than reporting a subset?
2 comments
AuthorsOpenReview ↗

Response to Reviewer bQYV

[W4, W5] Core contribution

We disagree with the assessment that our paper lacks novelty and provides no significant new knowledge. Characterizing its similarity to SayCan and ELLM as “a language model provides high-level guidance and RL handles low-level execution” overlooks the central problem addressed by our work. Strategic priors typically describe task relationships and behavioral objectives, whereas primitive actions require precise positioning, timing, and feedback control. Our dual-loop framework connects prior-guided strategy revision with primitive-action learning through a shared goal–outcome interface, allowing the agent to improve its strategy from experience while its execution capabilities are still developing.

[W4] Role of the goal–outcome interface

The interface has a concrete methodological role. The upper-level program converts task knowledge into intermediate-granularity goals that retain a connection to strategic intent while limiting the execution horizon faced by the lower-level controller. The program also labels achieved outcomes from observations and executed actions, using the same semantics and encoding for goals and outcomes. It therefore determines what to pursue next and supplies outcome supervision for control learning. The lower level learns actions that realize these goals, while the upper level revises goal selection and program logic using returns and execution trajectories. The same interaction data improve both strategy and execution, and the program provides a persistent representation in which prior knowledge can be tested and corrected through experience. Section 4 and Figure 2 describe this mechanism.

[W4] Comparison with SayCan and Code as Policies

SayCan combines the task relevance assigned by a language model with feasibility estimates from skill value functions to select and compose pretrained skills. It assumes that candidate skills already have executable policies and corresponding value functions, so planning is constrained by the available skills and their execution capabilities. Our Atari 100k setting provides game knowledge, image observations, environment rewards, and primitive discrete actions. Execution policies and value functions must be learned within the same interaction budget. For example, crossing an obstacle to obtain an item is itself a behavior that must be learned. Without these policies and value functions, SayCan's skill-selection procedure lacks necessary inputs, limiting its direct applicability. PREINFORCE addresses this gap by using program-generated local goals and outcome labels to learn primitive control online, while continuously revising the strategy from execution feedback. Code as Policies generates programs that organize perception and control APIs, including reactive and waypoint-based control. Our program additionally provides outcome labels and learning supervision, undergoes revision using online evidence, and supports a goal-conditioned executor trained alongside it. These differences concern how execution capabilities are acquired, how priors become training signals, and how experience continually changes the strategy.

[W4] Comparison with ELLM

ELLM also studies goal-conditioned policies. It uses a language model to propose exploratory goals and constructs intrinsic rewards from the semantic similarity between goal text and transition descriptions, encouraging useful behaviors during pretraining. This process depends on existing observation and transition captioners, which supply the current context to the language model and describe achieved behavior for reward computation. Its main experiments construct descriptions using simulator ground truth or semantic sensors, and its learned transition-captioner experiment uses additional human and synthetic annotations (Sections 3.5, 4.1, and 4.2, and Appendices I–J of ELLM). Our setting provides no such ready-made semantic observations, simulator internal states, or task-specific perception labels. Directly applying ELLM would therefore require constructing both semantic interfaces needed for goal generation and reward computation. A stronger language model would still require a mechanism for recognizing states and behavioral outcomes from game images, together with validation of that mechanism. Moreover, ELLM's original algorithm proposes exploratory goals from the current description without an upper-level learning process that jointly revises a strategic program and outcome labeler using accumulated returns and execution trajectories. PREINFORCE generates goals and labels in a shared control-outcome space through a revisable program, using achieved outcomes to train the executor. In Private Eye, for example, collecting a gun and returning it to the gun shop can be translated into local positional goals along a route. Even a jump that fails to collect the item and receives no task reward provides an achieved position for learning. Explicit goals and outcomes identifiable from trajectories connect strategic knowledge to primitive actions, providing concrete learning targets for fine-grained control. Adapting ELLM to our setting thus requires substantive methodological work that is obscured by grouping both methods under “language-guided RL.”

[W1, W5, Q1] Contributions of the program and learned control

The experiments further establish the practical value of PREINFORCE. With the same foundation model, game-knowledge prompts, and 100,000-interaction budget per game, code policies that directly output actions exceed the human reference on 14/26 games, while the full method does so on 26/26. Retaining executor learning while freezing the initial program reduces this number to 2/26 (Figure 4). These comparisons show that, even with the same prior knowledge, the organization of control learning and the use of feedback to revise the program substantially affect performance. The general expectation that priors can improve sample efficiency does not by itself establish this design or its observed performance across all 26 games.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.