acceptodds
Under review as a conference paper at ICLR 2027

Continual Policy Programming for LLM Agents: Learning Executable Policies from Online Experience

Abstract

LLM agents can solve increasingly complex long-horizon tasks, while continually improving from ongoing interaction remains an important challenge. We introduce Continual Policy Programming, a framework that enables a frozen LLM agent to learn from fine-grained online experience by updating a persistent executable policy. Policy updates take effect during the current task and persist across future tasks, allowing newly acquired knowledge to accumulate over time. Across ALFWorld, Webshop, MiniGrid, and Appworld, Continual Policy Programming enables agents to recover from failures online, reuse newly acquired behaviors across subsequent tasks, and accumulate capabilities over time. Our results show that persistent executable policies provide an effective way to turn streaming interaction experience into cumulative behavioral improvement.

Primary area:
applications to robotics, autonomy, planning
Keywords:
LLM Agents, Continual Learning, Test-Time Adaptation, Executable Policies, Online Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5C6HOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduced continual policy programming, which is used to enable a frozen LLM to learn from fine-grained online experience. CCP's learning is interleaved with agent interaction and accumulated into a persistent policy as code. The policy directly controls the agent's actions and manages a task-specific working memory that records information needed to the current task.

Critical strengths
  1. Experimentally, CCP is compared against multiple baselines in multiple different environments with text-based household tasks and outperforms all other baselines in terms of success rate and reward.

  2. Trained CCP policies show good adaptability across tasks, as supported by the experimental results in Table 3.

  3. The CCP framework aims to learn policies that are both timely and persistent.

Critical weaknesses
  1. The critic in CCP generates (cek,δek)(c_e^k, \delta_e^k), wherein δek\delta_e^k contains a list of disconnected information including the quality of the taken action, whether the observation matches the expectation, and whether subgoals and state variables still represent g_e correctly. First, they are all coupled quantities; for example, the accuracy of the state variable affects the effectiveness of the taken action and may not be considered parallel. Second, there is a lack of a quantitative metric for these evaluations.

  2. Because of the last point, the update scheme of CCP lacks an improved performance guarantee.

  3. In the beginning of the paper, the authors claim that CCP can improve the frozen LLM's policy by training through a few fine-grained interactions in the task environment. However, there is a lack of explanation of how these interactions/training data are prepared. Are they interaction trajectories including observations, actions, etc.? Did the authors make any modifications to the training trajectories?

Questions for authors
  1. Since working memory is associated with the specific task, would every trained policy for every fundamental task carry its own memory? Is there going to be a scalability issue when the number of tasks increases?

  2. What modifications need to be made to prepare "fine-grained" interactions from the ongoing task in order to improve the frozen policy? Are they realistic?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 5C6H (1/4)

We thank the reviewer for the thoughtful comments. We clarify these aspects and provide additional experimental evidence below.

[W1] The Unclear formulation and effectiveness of critic-generated revision feedback δek\delta_e^k

Clarification on Critic-Generated Revision Feedback. We would like to clarify that δek\delta_e^k is not a collection of independent evaluation scores. Instead, it represents actionable revision instructions generated by the critic through a joint assessment of recent interactions, the task objective, and the current executable policy. Action quality, observation consistency, and subgoal correctness are considered together to identify the underlying causes of policy failures. The critic then translates the diagnosis into concrete modifications to the working-memory operations or action-selection logic, providing targeted guidance for the patcher.

Effectiveness of Revision Feedback. The effectiveness of critic-generated feedback is reflected in the resulting policy improvements. As shown in Table 2, removing the critic substantially degrades task performance. For example, with GPT-4.1, success rates decrease from 98.5% to 30.8% on ALFWorld and from 94.3% to 56.0% on MiniGrid. These results demonstrate the importance of critic-guided policy refinement. We will further clarify the formulation and role of δek\delta_e^k in the revised paper.

To quantitatively evaluate the contribution of diagnostic revision points, we conducted an additional w/o Revision Points ablation that removes δek\delta_e^k while retaining the critic's binary revision decision cekc_e^k. In this variant, the critic still determines whether a policy update is needed, but the patcher must identify what to modify without receiving explicit diagnostic guidance. We evaluate this variant across three benchmarks and three backbone models.

BenchmarkBackboneCPP (Full) SR (%) ↑w/o Revision Points SR (%) ↑SR Drop (pp)
ALFWorldGPT-4.198.532.865.7
ALFWorldDeepSeek-V4-Flash99.090.38.7
ALFWorldGLM-5.3-Flash99.098.50.5
MiniGridGPT-4.194.344.050.3
MiniGridDeepSeek-V4-Flash96.776.020.7
MiniGridGLM-5.3-Flash95.090.05.0
AppWorldGPT-4.176.741.035.7
AppWorldDeepSeek-V4-Flash77.369.08.3
AppWorldGLM-5.3-Flash77.073.04.0

Effectiveness of Diagnostic Revision Points. Removing δek\delta_e^k consistently reduces success rates across all nine settings, with an average degradation of 22.1 percentage points. The impact is particularly pronounced for GPT-4.1, with performance drops of 65.7, 50.3, and 35.7 percentage points on ALFWorld, MiniGrid, and AppWorld, respectively. Similar trends are observed across DeepSeek-V4-Flash and GLM-5.3-Flash, although the magnitude varies with backbone capability.

These results highlight that identifying when to revise a policy is insufficient without actionable guidance on what to revise. By translating interaction-level observations into targeted revision points, δek\delta_e^k provides explicit guidance for modifying working-memory operations and action-selection logic. The ablation demonstrates that diagnostic revision feedback plays a critical role in effective policy refinement, beyond the binary decision of whether an update is needed.

AuthorsOpenReview ↗

Response to Reviewer 5C6H (2/4)

[W2]Because of the last point, the update scheme of CCP lacks an improved performance guarantee.

We would like to clarify that CPP does not assume or claim a formal monotonic performance improvement guarantee for every individual policy update. Since policy revisions are derived from limited online interaction experience, guaranteeing improvement after every update would require additional assumptions about the environment and feedback quality. Instead, CPP incorporates several mechanisms to support reliable policy refinement.

First, CPP employs targeted revision and validation mechanisms to mitigate performance regression. The critic identifies both the policy components requiring correction and the behaviors that should be preserved. The patcher then incorporates targeted revisions guided by δek\delta_e^k. These mechanisms help prevent invalid policy updates, although functional correctness alone does not guarantee improved task performance.

Second, we provide empirical evidence for the effectiveness and retention of policy updates. We extended the retention evaluation from the original 10-10-10 setting in ALFWorld to a 20-60-20 task stream on both ALFWorld and AppWorld. On ALFWorld, the test suite consists of 20 initial Pick tasks, 60 intervening tasks, and 20 final Pick tasks. The final 20 Pick tasks are identical to the initial 20. On AppWorld, the test suite consists of 20 initial "Phone + Venmo" tasks, 60 intervening tasks, and 20 final "Phone + Venmo" tasks identical to the initial 20. Policy updates are disabled during the final 20 tasks to directly evaluate knowledge retention without additional adaptation. This enables a direct comparison of performance on the same task instances before and after learning intervening tasks.

BenchmarkBackboneInitial 20 SR (%)Middle 60 SR (%)Final 20 SR (%)
ALFWorldGPT-4.195.096.7100.0
ALFWorldDeepSeek-V4-Flash100.095.0100.0
ALFWorldGLM-5.3-Flash100.0100.0100.0
AppWorldGPT-4.175.073.380.0
AppWorldDeepSeek-V4-Flash75.076.780.0
AppWorldGLM-5.3-Flash80.076.785.0

Across the extended 100-task streams, CPP demonstrates consistent retention of previously acquired capabilities without observed performance degradation. On ALFWorld, final Pick success rates reach 100% across all three backbones, matching or exceeding initial performance (95%, 100%, and 100%). On AppWorld, all three backbones achieve a 5-percentage-point improvement on the repeated Phone + Venmo tasks, despite learning from 60 intervening tasks. Importantly, the initial and final phases use identical task instances, enabling a direct assessment of forgetting and eliminating task-instance differences as a confounding factor. These results provide stronger evidence that CPP can retain previously learned behaviors while acquiring new capabilities over extended task streams, with no observed forgetting in the evaluated settings.

Together, these results provide empirical support for effective policy refinement with limited observed regression, while we acknowledge that formal monotonic improvement guarantees remain an open direction for future research. We will clarify this distinction in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 5C6H (3/4)

[W3] In the beginning of the paper, the authors claim that CCP can improve the frozen LLM's policy by training through a few fine-grained interactions in the task environment. However, there is a lack of explanation of how these interactions/training data are prepared. Are they interaction trajectories including observations, actions, etc.? Did the authors make any modifications to the training trajectories?

We thank the reviewer for raising this concern. We would like to provide a detailed explanation of how the interaction data are collected and used for policy learning, as described in Algorithm 1 and Equations (11)–(12).

First, the interaction data are collected directly from the agent's ongoing execution in the task environment. At each step, the current executable policy selects an action, executes it in the environment, and receives a new observation. After every short interaction window, CPP collects the naturally occurring sequence of observations and actions, denoted as the interaction segment ρek\rho_e^k. Importantly, we do not manually construct, modify, or preprocess trajectories into training examples. All interaction data are generated during normal task execution, without requiring additional training data or complete task trajectories.

Second, CPP directly uses the collected interactions to refine its executable policy. As described in Algorithm 1 and Equations (11)–(12), CPP updates its policy through the following mechanism:

  1. The current executable policy θek\theta_e^k directly selects actions and updates working memory during environment interaction, producing the interaction segment ρek\rho_e^k.

  2. After a short interaction window, the critic examines the task goal geg_e, recent observations and actions in the interaction segment ρek\rho_e^k, and current executable policy θek\theta_e^k.

  3. The critic outputs a binary revision decision cekc_e^k and diagnostic revision points δek\delta_e^k.

  4. If a revision is requested, the patcher modifies the relevant policy code, including the memory-update operator UU, action-selection operator PP, or working-memory schema WW. A memory-migration operator μek\mu_e^k preserves or transforms relevant task state when the schema changes.

  5. The revised policy immediately controls subsequent interactions from the current environment state, and its updates are retained for future tasks.

Importantly, CPP does not perform gradient-based training or update the frozen LLM's parameters. The interaction data are simply the observations and actions generated during normal task execution. The critic examines recent interaction experience together with the task specification and current policy, then determines whether a revision is necessary. If so, the patcher updates the executable code, and the agent continues interacting with the environment using the revised policy. No environment reset, additional trajectory annotation, or model-parameter update is required.

Therefore, the term "learning" refers to incorporating naturally collected online interaction experience into the executable policy, enabling immediate behavioral improvements and accumulating reusable capabilities across tasks.

Reviewer_bCbxOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Continual Policy Programming (CPP), a framework that allows LLM agents to learn from their interactions and improve their behavior over time without updating the model parameters. Instead of relying solely on prompting or storing past experiences, CPP maintains an executable policy that guides the agent's actions and manages its working memory. As the agent interacts with the environment, an LLM-based critic reviews recent interactions to identify potential problems, and a patcher updates the policy to address them. These changes can be applied during the current task and carried over to future tasks, allowing the agent to gradually improve its behavior through experience. Experimental results show that CPP consistently outperforms several baselines based on reasoning, memory, and skill learning. The paper also includes ablation studies on the roles of task decomposition and the critic, along with analyses of failure recovery and the ability to transfer and retain improvements across tasks. Overall, the paper explores a practical way for LLM agents to continually adapt their behavior by refining executable policies rather than modifying the underlying model.

Critical strengths
  1. Important and practical research problem: This paper addresses an important challenge for LLM agents: how to continuously improve their behavior through online interactions. This is particularly useful in dynamic environments, where agents need to learn from failures and apply what they have learned to future tasks.

  2. Clear and intuitive framework: The paper introduces a simple and well-structured framework that represents agent behavior as an executable policy. By separating reusable knowledge from task-specific memory, the framework allows agents to update their behavior based on new experiences without modifying the underlying LLM.

  3. Strong experimental results: The experiments show consistent improvements over several reasoning-, memory-, and skill-based baselines across different benchmarks and LLM backbones.

Critical weaknesses
  1. Unclear contribution of continual learning: Although CPP shows strong performance improvements, it is unclear how much of the gain comes from learning across tasks rather than simply correcting mistakes within each task. More experiments would help clarify this distinction.

  2. Limited evaluation of long-term retention: The experiments provide limited evidence that CPP can retain knowledge over time without forgetting. The retention study only includes 30 ALFWorld tasks across three phases. Also, the first and last Pick phases use different task instances, making it difficult to directly measure forgetting.

  3. Limited methodological novelty: The main idea of using an LLM-based critic to identify problems and a patcher to update executable code is similar to existing self-improving agent frameworks. While CPP combines these components into a continual learning framework, the paper could better clarify what is fundamentally new compared to prior work.

Questions for authors

Please see Weakness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer bCbx (1/2)

We thank the reviewer for the thoughtful comments and acknowledgment of our work. We address each concern below.

[W1] Unclear contribution of continual learning

We want to provide two complementary pieces of evidence.

First, Figure 2 shows that patching rates generally decrease as the agent processes more tasks. Since CPP retains policy modifications across task boundaries, this trend is consistent with previously learned behaviors being reused in subsequent tasks, reducing the need for additional corrections. However, we recognize that patching dynamics alone do not conclusively isolate the effect of cross-task accumulation.

Second, to directly isolate the contribution of policy persistence, we conducted an additional w/o Persistence ablation, in which the executable policy is reset at the beginning of each new task rather than inherited from the preceding task. This ablation preserves the ability to correct mistakes within each task while removing cross-task knowledge accumulation. We present two metrics, success rate and token cost comparison, across three benchmarks: ALFWorld, MiniGrid, and Appworld.

BenchmarkBackboneCPP (Full) SR (%) ↑w/o Persistence SR (%) ↑LLM CallsToken Cost
ALFWorldGPT-4.198.563.42.28×2.76×
ALFWorldDeepSeek-V4-Flash99.082.82.30×2.82×
ALFWorldGLM-5.3-Flash99.098.52.14×2.95×
MiniGridGPT-4.194.359.03.11×3.40×
MiniGridDeepSeek-V4-Flash96.778.02.79×3.36×
MiniGridGLM-5.3-Flash95.089.03.52×3.91×
AppWorldGPT-4.176.745.92.77×3.04×
AppWorldDeepSeek-V4-Flash77.361.02.61×2.83×
AppWorldGLM-5.3-Flash77.070.02.42×2.98×

(SR is reported in %. LLM Calls and Token Cost indicate the multiplicative overhead of w/o Persistence relative to full CPP.)

We want to highlight the following takeaways:

Success Rate. Removing persistence consistently reduces success rates across all nine settings, with an average drop of 18.4 percentage points. The degradation is particularly pronounced for GPT-4.1, reaching 35.1, 35.3, and 30.8 points on ALFWorld, MiniGrid, and AppWorld, respectively. Since within-task correction remains enabled, these results directly highlight the contribution of cross-task knowledge accumulation, beyond online error recovery alone.

Efficiency. Without persistence, the agent requires 2.14–3.52× more LLM calls and 2.76–3.91× higher token costs, averaging 2.66× and 3.12×, respectively. This suggests that resetting the policy forces the agent to repeatedly rediscover useful behaviors, whereas CPP reuses accumulated executable knowledge across tasks. Together, these findings provide direct evidence that policy persistence improves both task success and adaptation efficiency, supporting the core continual-learning claim beyond the decreasing patch rates in Figure 2.

[W2] Limited evaluation of long-term retention:

To address this concern, we extended the retention evaluation from the original 10-10-10 setting in ALFWorld to a 20-60-20 task stream on both ALFWorld and AppWorld. On ALFWorld, the test suite consists of 20 initial Pick tasks, 60 intervening tasks, and 20 final Pick tasks. The final 20 Pick tasks are identical to the initial 20. On AppWorld, the test suite consists of 20 initial "Phone + Venmo" tasks, 60 intervening tasks, and 20 final "Phone + Venmo" tasks identical to the initial 20. Policy updates are disabled during the final 20 tasks to directly evaluate knowledge retention without additional adaptation. This enables a direct comparison of performance on the same task instances before and after learning intervening tasks.

BenchmarkBackboneInitial 20 SR (%)Middle 60 SR (%)Final 20 SR (%)
ALFWorldGPT-4.195.096.7100.0
ALFWorldDeepSeek-V4-Flash100.095.0100.0
ALFWorldGLM-5.3-Flash100.0100.0100.0
AppWorldGPT-4.175.073.380.0
AppWorldDeepSeek-V4-Flash75.076.780.0
AppWorldGLM-5.3-Flash80.076.785.0

(Analysis continued on next response)

AuthorsOpenReview ↗

Response to Reviewer bCbx (2/2)

Across the extended 100-task streams, CPP demonstrates consistent retention of previously acquired capabilities without observed performance degradation. On ALFWorld, final Pick success rates reach 100% across all three backbones, matching or exceeding initial performance (95%, 100%, and 100%). On AppWorld, all three backbones achieve a 5-percentage-point improvement on the repeated Phone + Venmo tasks, despite learning from 60 intervening tasks. Importantly, the initial and final phases use identical task instances, enabling a direct assessment of forgetting and eliminating task-instance differences as a confounding factor. These results provide stronger evidence that CPP can retain previously learned behaviors while acquiring new capabilities over extended task streams, with no observed forgetting in the evaluated settings.

.

[W3] Novelty compared to prior works We highlight three distinctions.

(1) The object of learning is the executable decision policy itself. Unlike approaches that optimize prompts, auxiliary skills, or the broader agent harness, CPP maintains a single executable policy, represented as code, that directly determines the agent's actions and how task-relevant information is stored and used. The policy jointly specifies the working-memory schema, memory-update logic, and action-selection procedure. This unified representation enables a more lightweight, targeted, and effective learning process, allowing CPP to directly refine decision-making behaviors without introducing additional layers of agent orchestration.

(2) Fine-grained policy learning from ongoing interactions. CPP enables agents to acquire new behavioral knowledge from only a few recent interactions, without requiring complete trajectories, task-level feedback, or extensive evaluation rollouts. After each short interaction window, CPP identifies deficiencies and directly incorporates targeted corrections into the executable policy. Crucially, newly acquired knowledge immediately improves subsequent decisions within the same task, with policy revisions taking effect without resetting the environment or interrupting ongoing execution. This enables continuous learning and adaptation at the granularity of individual interaction windows.

(3) Continuous accumulation of reusable capabilities across tasks. CPP transforms fine-grained online corrections into persistent behavioral knowledge that is progressively refined and reused across subsequent tasks. By retaining executable policy updates while resetting task-specific working memory, CPP accumulates generalizable decision-making procedures without carrying over stale task-specific information. This design connects immediate online adaptation with long-term capability accumulation. Our additional w/o Persistence ablation provides direct empirical evidence: removing cross-task persistence decreases success rates by 18.4 percentage points on average, while increasing LLM calls by 2.66× and token costs by 3.12×.

Dear Reviewer bCbx,

Thank you again for your thoughtful and constructive feedback. As the discussion period is approaching its end, we would like to kindly follow up on our rebuttal. We hope our clarifications and additional experimental results have addressed your concerns.

We would greatly appreciate any further feedback and would be happy to address any remaining questions.

Thank you again for your time and consideration!

Reviewer_nxxvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Continual Policy Programming (CPP): freezing LLM weights and writing agent behavior as a persistent executable policy code, so that actions are generated by running the code rather than LLM calls. Periodically during interaction, the LLM acts as a critic to review recent experiences, after which a "patcher" component selectively modifies the policy code. These updates take effect immediately and persist across tasks, enabling both real-time recovery from failures and the long-term accumulation of capabilities. CPP consistently outperforms baselines such as ReAct and Reflexion across benchmarks including ALFWorld, WebShop, MiniGrid, and AppWorld.

Critical strengths

Interesting problem setting. Code-as-policy is not new, but placing continual learning in the decision procedure code itself and updating it at test time, within an episode and without environment resets, is a well-separated setting from offline rollout-based repair and from harness optimization. Two well-motivated separations. The U/P split makes a bad action attributable to a named function, enabling local repair. The θ/working-memory split lets procedures accumulate across tasks while episode-specific facts are discarded. Broad and consistent evaluation. Four environments, three backbones, three seeds, best in all twelve combinations with large margins. The low variance across backbones supports the claim that gains come from the framework rather than the base model.

Critical weaknesses

The core continual-learning claim is not directly isolated. The paper does not provide performance-over-time curves or a ‘w/o persistence’ ablation that resets the policy between tasks. Since Figure 2 only shows decreasing patch rates, it is unclear how much of the gain actually comes from cross-task accumulation. The ablation results are highly inconsistent across backbones. For example, removing the critic severely hurts GPT-4.1 on ALFWorld but has almost no effect on GLM-5.3-Flash. The paper also does not report policy exceptions or fallback rates, making it difficult to tell whether some failures arise from weaker learning or broken generated code. The method relies heavily on environment-specific structure and may have limited applicability to more open-ended tasks. CPP depends on admissible action lists, structured goal parsing, and environment-specific policy code. This makes it unclear how well the approach scales to settings where decisions rely more on open-ended semantic reasoning rather than structured low-level actions. There is no systematic efficiency evaluation. CPP introduces additional decomposition, critic, and policy-generation calls, including rewriting the policy at the beginning of each task. However, the paper does not report token cost, latency, or end-to-end computational overhead compared with the baselines. The paper lacks reproducibility statements and ethics statements.

Questions for authors

n/a

5 comments
AuthorsOpenReview ↗

Response to Reviewer nxxv (1/3)

We thank the reviewer for constructive comments. We address each concern below.

[W1] The core continual-learning claim is not directly isolated. The paper does not provide performance-over-time curves or a ‘w/o persistence’ ablation that resets the policy between tasks.

To directly isolate the contribution of policy persistence, we conducted an additional w/o Persistence ablation, in which the executable policy is reset at the beginning of each new task rather than inherited from the preceding task. This ablation preserves the ability to correct mistakes within each task while removing cross-task knowledge accumulation. We present two metrics, success rate and token cost comparison, across three benchmarks: ALFWorld, MiniGrid, and Appworld.

We want to highlight the following takeaways:

BenchmarkBackboneCPP (Full) SR (%) ↑w/o Persistence SR (%) ↑LLM CallsToken Cost
ALFWorldGPT-4.198.563.42.28×2.76×
ALFWorldDeepSeek-V4-Flash99.082.82.30×2.82×
ALFWorldGLM-5.3-Flash99.098.52.14×2.95×
MiniGridGPT-4.194.359.03.11×3.40×
MiniGridDeepSeek-V4-Flash96.778.02.79×3.36×
MiniGridGLM-5.3-Flash95.089.03.52×3.91×
AppWorldGPT-4.176.745.92.77×3.04×
AppWorldDeepSeek-V4-Flash77.361.02.61×2.83×
AppWorldGLM-5.3-Flash77.070.02.42×2.98×

(Note: SR is reported in %. LLM Calls and Token Cost indicate the multiplicative overhead of w/o Persistence relative to full CPP.)

We present the following takeaways:

Success Rate. Removing persistence consistently reduces success rates across all nine settings, with an average drop of 18.4 percentage points. The degradation is particularly pronounced for GPT-4.1, reaching 35.1, 35.3, and 30.8 points on ALFWorld, MiniGrid, and AppWorld, respectively. Since within-task correction remains enabled, these results directly highlight the contribution of cross-task knowledge accumulation, beyond online error recovery alone.

Efficiency. Without persistence, the agent requires 2.14–3.52× more LLM calls and 2.76–3.91× higher token costs, averaging 2.66× and 3.12×, respectively. This suggests that resetting the policy forces the agent to repeatedly rediscover useful behaviors, whereas CPP reuses accumulated executable knowledge across tasks. Together, these findings provide direct evidence that policy persistence improves both task success and adaptation efficiency, supporting the core continual-learning claim beyond the decreasing patch rates in Figure 2.

Since within-task patching remains enabled in this ablation, the performance differences directly demonstrate the additional benefit of retaining and reusing learned policy updates across tasks, beyond correcting mistakes within individual tasks.

We will incorporate this ablation into the revised paper to more explicitly substantiate the continual-learning contribution.

[W2] Since Figure 2 only shows decreasing patch rates, it is unclear how much of the gain actually comes from cross-task accumulation. The ablation results are highly inconsistent across backbones. For example, removing the critic severely hurts GPT-4.1 on ALFWorld but has almost no effect on GLM-5.3-Flash.

First, we intentionally evaluate CPP with various backbone models, including models with different reasoning capabilities and inference behaviors. For example, GLM-5.3-Flash employs an explicit reasoning mechanism, while GPT-4.1 does not expose the same reasoning mechanism. Such differences affect how much each backbone relies on external critic feedback. This explains why removing the critic has a much larger effect on GPT-4.1 than on GLM-5.3-Flash.

Second, the newly conducted w/o Persistence ablation provides direct evidence that cross-task accumulation contributes to CPP’s performance, independently of within-task correction.

Third, the decreasing patching rates in Figure 2 provide complementary evidence. As the persistent policy incorporates behaviors learned from earlier interactions, it can increasingly handle subsequent situations without requiring further revisions. In other words, a lower patching rate is consistent with the policy entering new tasks with more reusable knowledge already encoded.

AuthorsOpenReview ↗

Response to Reviewer nxxv (2/3)

[W3] The paper also does not report policy exceptions or fallback rates, making it difficult to tell whether some failures arise from weaker learning or broken generated code.

We examined execution logs to distinguish failures caused by policy learning from those caused by generated-code execution errors across two benchmarks, ALFWorld and AppWorld.

BenchmarkBackboneEpisodesStepsPolicy ExceptionsHarness Fallback occurence
ALFWorldGPT-4.11003,52000
ALFWorldGLM-5.3-Flash1001,72000
ALFWorldDeepSeek-V4-Flash1002,17100
AppWorldGPT-4.11002,12700
AppWorldGLM-5.3-Flash1001,77400
AppWorldDeepSeek-V4-Flash1001,92600

The harness fallback rate is zero for all the runs, and no hardcoded action replacement occurs. These results indicate that while CPP implements a fallback strategy for robustness of execution, the generated-policy execution failures are rare in the examined runs. Hence, observed performance differences are unlikely to be primarily explained by policy crashes or harness-level fallbacks. We will include these execution diagnostics in the revised paper for greater transparency.

[W4] The method relies heavily on environment-specific structure and may have limited applicability to more open-ended tasks. CPP depends on admissible action lists, structured goal parsing, and environment-specific policy code. This makes it unclear how well the approach scales to settings where decisions rely more on open-ended semantic reasoning rather than structured low-level actions.

We would like to clarify three aspects of CPP.

First, CPP does not rely on a pre-existing, handcrafted environment-specific policy. As shown in Algorithm 1, each independent run starts from an initially empty policy. The policy's decision logic and working-memory operations are generated and continually revised by the LLM based on task specifications and online interaction experience. The resulting behavioral procedures are not manually predefined.

Second, CPP does not require ground-truth structured goal decomposition. The decomposition module generates subgoals and relevant state variables directly from natural-language instructions. This decomposition serves as an initial scaffold rather than an assumed-correct plan. As described in Section 3.3, decomposition errors can be identified and corrected through subsequent interactions and policy revisions.

Third, CPP's core learning mechanism does not inherently depend on admissible action lists. The central mechanism is to continually revise an executable policy based on online experience, rather than learning from a predefined set of candidate actions. Although our implementation uses admissible action lists in certain environments, they are not a fundamental requirement of the proposed framework.

To further investigate this concern, we conducted additional experiments on ALFWorld and AppWorld without providing admissible action lists. The results are summarized below.

BackboneALFWorld (Original)ALFWorld (w/o Action Lists)AppWorld (Original)AppWorld (w/o Action Lists)
GPT-4.198.5 ± 1.394.876.7 ± 2.569.0
DeepSeek-V4-Flash99.0 ± 1.197.877.3 ± 3.573.0
GLM-5.3-Flash99.0 ± 1.198.577.0 ± 3.075.0

All values are success rates (%). Original results are reported as mean ± standard deviation.

Results. CPP maintains strong performance even without admissible action lists, achieving 94.8–98.5% success on ALFWorld and 69.0–75.0% on AppWorld across three backbone models. The relatively small performance degradation suggests that CPP's effectiveness is not primarily driven by access to predefined candidate actions. Instead, the learned executable policy can continue to support effective decision-making when such action lists are unavailable.

Evaluation Clarification. We emphasize that, in the original evaluation, all compared methods receive the same environment information, ensuring a fair comparison. The additional experiments modify the available action information and are intended to evaluate CPP's robustness. The results without admissible action lists should not be directly compared with baseline results obtained under the original evaluation protocol.

AuthorsOpenReview ↗

Response to Reviewer nxxv (3/3)

[W5] There is no systematic efficiency evaluation. CPP introduces additional decomposition, critic, and policy-generation calls, including rewriting the policy at the beginning of each task. However, the paper does not report token cost, latency, or end-to-end computational overhead compared with the baselines.

We thank the reviewer for highlighting the importance of computational efficiency. To address this concern, we conducted an additional efficiency evaluation comparing CPP against five baselines, measuring token consumption per environment interaction step, average LLM calls per task, and average execution time (minutes).

MethodTokens / Env. Step ↓LLM Calls / Task ↓Avg. Time (min) ↓
CPP (Ours)4,0096.56.6
ReAct7,00635.512.2
Reflexion8,31453.98.0
SkillAdaptor7,40148.211.2
ReCode4925.70.7
ReMem10,25357.415.4

Computational Efficiency. Despite introducing additional decomposition, critic, and policy-generation calls, CPP achieves 43–61% lower token consumption per environment interaction step than ReAct, Reflexion, SkillAdaptor, and ReMem. CPP also requires substantially fewer LLM calls per task (6.5 vs. 35.5–57.4) and achieves shorter average execution time. The learned executable policy directly controls subsequent actions without requiring an LLM call at every step. This amortizes the cost of policy construction and refinement across interactions, resulting in substantially lower computational overhead.

Efficiency–Performance Trade-off. ReCode achieves the lowest computational cost, but at the expense of substantially weaker task performance. Its relatively limited task decomposition can leave complex task requirements insufficiently addressed. For example, on MiniGrid, ReCode achieves only 47.3%, 50.7%, and 55.3% success rates across the three backbones, compared with 94.3%, 96.7%, and 95.0% for CPP. Similar performance gaps are observed on ALFWorld and AppWorld. These results suggest that CPP achieves a more favorable balance between computational efficiency and task success, instead of simply minimizing token usage at the cost of task completion.

[W6] The paper lacks reproducibility statements and ethics statements.

Thank you for your suggestion. We’ve updated the manuscript to include these two sections.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Message to all reviewers

We sincerely thank all reviewers for their thoughtful and constructive feedback. We appreciate the recognition of CPP's motivation, methodological design, and experimental results. In response to the reviewers' concerns, we conducted several additional experiments and provided further methodological clarifications.

We summarize the key additions below:

  1. Isolating the contribution of continual learning. We introduced a w/o Persistence ablation across three benchmarks and three backbones. Removing cross-task policy persistence reduces success rates by 18.4 percentage points on average, while increasing LLM calls by 2.66× and token costs by 3.12×, directly demonstrating the benefits of cross-task knowledge accumulation.

  2. Evaluating long-term knowledge retention. We extended the retention evaluation to 100-task streams on both ALFWorld and AppWorld, using identical tasks in the initial and final phases and disabling policy updates during final evaluation. CPP exhibits no observed forgetting across all six evaluated settings.

  3. Validating critic-guided policy refinement. We introduced a w/o Revision Points ablation to isolate the contribution of diagnostic feedback. Removing the revision points δek\delta_e^k reduces success rates by 22.1 percentage points on average, demonstrating the importance of actionable guidance for policy updates.

  4. Evaluating robustness, execution reliability, and efficiency. We conducted additional experiments without admissible action lists, examined policy execution exceptions and fallback behavior, and compared computational costs against five baselines. CPP maintains strong performance without admissible action lists, exhibits zero policy exceptions and action replacements in the inspected runs, and achieves 43–61% lower token consumption per environment interaction step than four of the five baselines.

We also clarified CPP's methodological novelty, particularly its ability to learn executable decision policies from short online interaction windows, jointly refine policy and working-memory operations, and accumulate reusable capabilities across tasks. Additional clarifications address the source of interaction data, policy validation, scalability, and reproducibility.

We believe these additional experiments and clarifications substantially strengthen the empirical evidence for CPP's core continual-learning contributions and directly address the reviewers' main concerns. We sincerely appreciate the reviewers' feedback and consideration and welcome any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.