- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CoPA, a framework where an LLM writes an executable policy, runs it for several steps, and pauses to inspect the environment and revise the policy. The agent decides when to intervene using an adaptive step budget and event triggers, while a persistent REPL allows it to inspect the current state and test actions. Experiments on Atari and Crafter show improvements in environment-step efficiency and task performance over LLM agent baselines.
Critical strengths
- Using executable policies with occasional LLM intervention seems like a practical approach for long-horizon tasks. The persistent REPL allows the agent to inspect and revise its policy without waiting for an episode to end.
- The results on Montezuma's Revenge are promising, with CoPA achieving a mean score of 4,900 while the reproduced LLM baselines score zero. The example in Figure 4 also helps illustrate how the agent improves its policy through interaction.
Critical weaknesses
- The technical novelty seems limited. CoPA mainly combines existing ideas such as code-as-policy, REPL-based debugging, and reflection-based policy revision. Even the adaptive intervention mechanism seems to rely on LLM-selected step budgets and event triggers rather than a new algorithm. It is unclear whether this integration offers a sufficiently substantial contribution beyond existing agent frameworks.
- The empirical results do not seem strong enough to fully justify the contribution. Pong and Breakout mainly show improvements in environment-step efficiency, while the improvement on Crafter is relatively small. Although Montezuma shows promising results, the ablations suggest that much of the benefit can already be achieved with simpler intervention strategies. The additional gains from adaptive scheduling are less clear, especially when considering LLM computation costs and the lack of variance estimates.
Questions for authors
-
CoPA seems to combine several existing ideas, including code-as-policy, interactive debugging, and policy revision. Compared to prior methods such as Heuristic Learning and other code-based agents, what is the key new contribution beyond integrating these components?
-
The adaptive intervention mechanism seems to rely on the LLM choosing the step budget (k) and event triggers based on general instructions. Is there a more systematic mechanism behind these decisions? It is unclear how this differs from a standard coding agent deciding when to execute or inspect its code.
-
In Table 5, fixed (k=1000) with event triggers already achieves 4,200 points, compared to 4,900 for the full method, which uses more LLM interventions (344 vs. 280). Could the authors provide token costs and variance estimates for these ablations, and clarify whether the advantage remains under matched computation budgets?
-
In the "w/o REPL (live probing)" ablation in Table 4, is execution-state persistence also removed, or are only additional action probes disabled? Which capabilities remain available for policy revision in this setting?