acceptodds
Withdrawn from ICLR 2027

Code-Probe-Adapt: Interactive Heuristic Learning for Self-Evolving Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Existing LLM-based autonomous agents generally follow two paradigms: ReAct- style methods, which interleave dense reasoning with continuous API calls; and code-as-policy methods, which synthesize executable code to bypass LLM infer- ence at runtime. However, in long-horizon interactive tasks, ReAct agents incur prohibitive inference costs and context overload, whereas offline policies lack test- time adaptability to unforeseen environmental dynamics. To bridge this gap, we introduce Code-Probe-Adapt (COPA), an interactive heuristic learning framework that transitions LLM agents from post-hoc log analyzers to active, online policy designers. Equipped with a persistent REPL (Read-Eval-Print Loop) harness, COPA enables live, on-demand introspection. To enable adaptive code-as-policy, COPA modulates policy execution using adaptive step budgets and event-triggered interruptions, returning control to the agent for immediate diagnostic probing and policy refinement. Evaluated on five games, COPA significantly outperforms re- cent LLM-based policy synthesizers, and demonstrates superior sample efficiency compared to extensively trained reinforcement learning (RL) algorithms. Notably, on the hard-exploration task Montezuma’s Revenge, where offline LLM heuristic learning fails entirely (scoring 0 even after 6.0M steps), COPA’s interactive prob- ing achieves an average score of 4,900 in 2.4M steps. This result also compares favorably against the RL baseline DreamerV3 (1,853.5 at 200M steps, digitized from its learning curve), scoring 2.6× as high with fewer environment interactions. Attained entirely without gradient updates or task-specific engineering, COPA successfully unites the execution efficiency of code with the adaptive power of live debugging.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, interactive heuristic learning, code-as-policy, test-time adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Hk47OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoPA, a framework where an LLM writes an executable policy, runs it for several steps, and pauses to inspect the environment and revise the policy. The agent decides when to intervene using an adaptive step budget and event triggers, while a persistent REPL allows it to inspect the current state and test actions. Experiments on Atari and Crafter show improvements in environment-step efficiency and task performance over LLM agent baselines.

Critical strengths
  • Using executable policies with occasional LLM intervention seems like a practical approach for long-horizon tasks. The persistent REPL allows the agent to inspect and revise its policy without waiting for an episode to end.
  • The results on Montezuma's Revenge are promising, with CoPA achieving a mean score of 4,900 while the reproduced LLM baselines score zero. The example in Figure 4 also helps illustrate how the agent improves its policy through interaction.
Critical weaknesses
  • The technical novelty seems limited. CoPA mainly combines existing ideas such as code-as-policy, REPL-based debugging, and reflection-based policy revision. Even the adaptive intervention mechanism seems to rely on LLM-selected step budgets and event triggers rather than a new algorithm. It is unclear whether this integration offers a sufficiently substantial contribution beyond existing agent frameworks.
  • The empirical results do not seem strong enough to fully justify the contribution. Pong and Breakout mainly show improvements in environment-step efficiency, while the improvement on Crafter is relatively small. Although Montezuma shows promising results, the ablations suggest that much of the benefit can already be achieved with simpler intervention strategies. The additional gains from adaptive scheduling are less clear, especially when considering LLM computation costs and the lack of variance estimates.
Questions for authors
  1. CoPA seems to combine several existing ideas, including code-as-policy, interactive debugging, and policy revision. Compared to prior methods such as Heuristic Learning and other code-based agents, what is the key new contribution beyond integrating these components?

  2. The adaptive intervention mechanism seems to rely on the LLM choosing the step budget (k) and event triggers based on general instructions. Is there a more systematic mechanism behind these decisions? It is unclear how this differs from a standard coding agent deciding when to execute or inspect its code.

  3. In Table 5, fixed (k=1000) with event triggers already achieves 4,200 points, compared to 4,900 for the full method, which uses more LLM interventions (344 vs. 280). Could the authors provide token costs and variance estimates for these ablations, and clarify whether the advantage remains under matched computation budgets?

  4. In the "w/o REPL (live probing)" ablation in Table 4, is execution-state persistence also removed, or are only additional action probes disabled? Which capabilities remain available for policy revision in this setting?

Reviewer_eWzkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper propose a new paradigm of self-evolving agent. Instead of training and update the weight, the paper propose an interactive heuristic learning framework that combines efficient code execution with timely inspection and revision. The experiment shows that the method can bring improvement with less environment steps.

Critical strengths

The research question is well-motivated and the method is intuitive.

Critical weaknesses
  • The main evaluation only use GPT-5.5 as backbone. It's unclear if the improvement is coming from the strong foundation model's self-reasoning capabilities or the proposed harness. We need more experiments results with other backbone models to prove the generalizability and performance of the proposed harness.
  • Does the extra cost really worth it? I didn't find the comprehensive cost-efficient analysis for the proposed method. The paper only mention there is a reduce for the environment steps, but it might comes with more cost in time and budget since the token usage is much higher. A more comprehensive cost-efficient analysis is needed otherwise the comparison is unclear.
Questions for authors

Can the framework generalize beyond the game application?

Reviewer_YTSzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Code-Probe-Adapt (CoPA), a code-as-policy framework in which an LLM writes a policy, tracking logic, and event triggers that execute locally between sparse interventions. A persistent REPL keeps the reached environment state available, allowing the model to inspect observations and traces, run short continuations, and revise its policy within an episode. The agent also adjusts each segment's step budget and maintains cross-intervention memory. On three Atari games, CoPA reaches the maximum reported Pong and Breakout scores with many fewer environment steps than two coding-agent baselines, and obtains 4,900 points on Montezuma's Revenge where the reported baseline reproductions score zero. Ablations attribute much of the Montezuma result to live probing and automatic interruption, while Crafter and VizDoom provide additional evidence beyond Atari. The paper presents CoPA as a gradient-free form of interactive heuristic learning rather than dense LLM control.

Critical strengths
  1. The persistent-session design is a practical and well-motivated middle ground between per-action LLM control and fully offline policy synthesis. The component and horizon ablations are unusually relevant to the mechanism: on Montezuma, removing live probing or automatic interruption lowers the reported score from 4,900 to 160 or 400, and the paper accounts for discarded branches as environment interactions. This makes the central engineering idea credible and potentially useful.

  2. The Montezuma result is striking for a coding agent, and the paper does more than present a single trajectory: it reports five independent CoPA discovery runs and evaluates each retained policy under three seeds with memory fixed and clone/restore disabled. Together with the Crafter and VizDoom results, this is promising evidence that executable policies can be repaired from sparse interventions rather than dense action-level reasoning.

Critical weaknesses
  1. The comparison does not yet isolate algorithmic improvement from interface advantage. CoPA is given a persistent live-state harness and, in one variant, explicit clone/restore, while the descriptions of free-form Codex and the Heuristic Learning reproduction do not establish matched access to held states, short continuation tests, or equivalent debugging tools; the authors themselves relate restore-enabled behavior to Go-Explore. Baseline run counts and uncertainty are also not reported, and the reproduced Heuristic Learning score of zero differs from the cited original mean of 133.3. Since the main evidence is comparative, matched interfaces and repeated baseline runs could materially change the conclusion.

  2. The broad efficiency and task-agnostic claims are not adequately supported. CoPA uses more LLM tokens than both baselines on Pong, no wall-clock or monetary cost is measured, and cached input is excluded, so environment steps alone do not establish overall efficiency. Moreover, Montezuma uniquely exposes public RAM, the games are likely familiar to the underlying model, and evaluation covers only a small set of environments. A pixel-only or genuinely held-out-game study with end-to-end resource accounting is needed to distinguish general interactive learning from exploitation of game knowledge and favorable observability.

Questions for authors
  1. [Clarification] How many independent discovery runs were conducted for each baseline and each ablation, and what are the score, step, and token variances for all main comparisons? Were the baseline entries single runs, and why does the Heuristic Learning reproduction differ from its cited original result?

  2. [Clarification] Please specify, method by method, access to the persistent reached state, raw frames, RAM, traces, short continuation tests, and clone/restore. In standard CoPA, are probes irreversible continuations of the current state, or can any probe reset to the pause point? A matched-access baseline would substantially affect my assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.