- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduced continual policy programming, which is used to enable a frozen LLM to learn from fine-grained online experience. CCP's learning is interleaved with agent interaction and accumulated into a persistent policy as code. The policy directly controls the agent's actions and manages a task-specific working memory that records information needed to the current task.
Critical strengths
-
Experimentally, CCP is compared against multiple baselines in multiple different environments with text-based household tasks and outperforms all other baselines in terms of success rate and reward.
-
Trained CCP policies show good adaptability across tasks, as supported by the experimental results in Table 3.
-
The CCP framework aims to learn policies that are both timely and persistent.
Critical weaknesses
-
The critic in CCP generates , wherein contains a list of disconnected information including the quality of the taken action, whether the observation matches the expectation, and whether subgoals and state variables still represent g_e correctly. First, they are all coupled quantities; for example, the accuracy of the state variable affects the effectiveness of the taken action and may not be considered parallel. Second, there is a lack of a quantitative metric for these evaluations.
-
Because of the last point, the update scheme of CCP lacks an improved performance guarantee.
-
In the beginning of the paper, the authors claim that CCP can improve the frozen LLM's policy by training through a few fine-grained interactions in the task environment. However, there is a lack of explanation of how these interactions/training data are prepared. Are they interaction trajectories including observations, actions, etc.? Did the authors make any modifications to the training trajectories?
Questions for authors
-
Since working memory is associated with the specific task, would every trained policy for every fundamental task carry its own memory? Is there going to be a scalability issue when the number of tasks increases?
-
What modifications need to be made to prepare "fine-grained" interactions from the ongoing task in order to improve the frozen policy? Are they realistic?
7 comments
Response to Reviewer 5C6H (2/4)
[W2]Because of the last point, the update scheme of CCP lacks an improved performance guarantee.
We would like to clarify that CPP does not assume or claim a formal monotonic performance improvement guarantee for every individual policy update. Since policy revisions are derived from limited online interaction experience, guaranteeing improvement after every update would require additional assumptions about the environment and feedback quality. Instead, CPP incorporates several mechanisms to support reliable policy refinement.
First, CPP employs targeted revision and validation mechanisms to mitigate performance regression. The critic identifies both the policy components requiring correction and the behaviors that should be preserved. The patcher then incorporates targeted revisions guided by . These mechanisms help prevent invalid policy updates, although functional correctness alone does not guarantee improved task performance.
Second, we provide empirical evidence for the effectiveness and retention of policy updates. We extended the retention evaluation from the original 10-10-10 setting in ALFWorld to a 20-60-20 task stream on both ALFWorld and AppWorld. On ALFWorld, the test suite consists of 20 initial Pick tasks, 60 intervening tasks, and 20 final Pick tasks. The final 20 Pick tasks are identical to the initial 20. On AppWorld, the test suite consists of 20 initial "Phone + Venmo" tasks, 60 intervening tasks, and 20 final "Phone + Venmo" tasks identical to the initial 20. Policy updates are disabled during the final 20 tasks to directly evaluate knowledge retention without additional adaptation. This enables a direct comparison of performance on the same task instances before and after learning intervening tasks.
| Benchmark | Backbone | Initial 20 SR (%) | Middle 60 SR (%) | Final 20 SR (%) |
|---|---|---|---|---|
| ALFWorld | GPT-4.1 | 95.0 | 96.7 | 100.0 |
| ALFWorld | DeepSeek-V4-Flash | 100.0 | 95.0 | 100.0 |
| ALFWorld | GLM-5.3-Flash | 100.0 | 100.0 | 100.0 |
| AppWorld | GPT-4.1 | 75.0 | 73.3 | 80.0 |
| AppWorld | DeepSeek-V4-Flash | 75.0 | 76.7 | 80.0 |
| AppWorld | GLM-5.3-Flash | 80.0 | 76.7 | 85.0 |
Across the extended 100-task streams, CPP demonstrates consistent retention of previously acquired capabilities without observed performance degradation. On ALFWorld, final Pick success rates reach 100% across all three backbones, matching or exceeding initial performance (95%, 100%, and 100%). On AppWorld, all three backbones achieve a 5-percentage-point improvement on the repeated Phone + Venmo tasks, despite learning from 60 intervening tasks. Importantly, the initial and final phases use identical task instances, enabling a direct assessment of forgetting and eliminating task-instance differences as a confounding factor. These results provide stronger evidence that CPP can retain previously learned behaviors while acquiring new capabilities over extended task streams, with no observed forgetting in the evaluated settings.
Together, these results provide empirical support for effective policy refinement with limited observed regression, while we acknowledge that formal monotonic improvement guarantees remain an open direction for future research. We will clarify this distinction in the revised manuscript.
Response to Reviewer 5C6H (3/4)
[W3] In the beginning of the paper, the authors claim that CCP can improve the frozen LLM's policy by training through a few fine-grained interactions in the task environment. However, there is a lack of explanation of how these interactions/training data are prepared. Are they interaction trajectories including observations, actions, etc.? Did the authors make any modifications to the training trajectories?
We thank the reviewer for raising this concern. We would like to provide a detailed explanation of how the interaction data are collected and used for policy learning, as described in Algorithm 1 and Equations (11)–(12).
First, the interaction data are collected directly from the agent's ongoing execution in the task environment. At each step, the current executable policy selects an action, executes it in the environment, and receives a new observation. After every short interaction window, CPP collects the naturally occurring sequence of observations and actions, denoted as the interaction segment . Importantly, we do not manually construct, modify, or preprocess trajectories into training examples. All interaction data are generated during normal task execution, without requiring additional training data or complete task trajectories.
Second, CPP directly uses the collected interactions to refine its executable policy. As described in Algorithm 1 and Equations (11)–(12), CPP updates its policy through the following mechanism:
-
The current executable policy directly selects actions and updates working memory during environment interaction, producing the interaction segment .
-
After a short interaction window, the critic examines the task goal , recent observations and actions in the interaction segment , and current executable policy .
-
The critic outputs a binary revision decision and diagnostic revision points .
-
If a revision is requested, the patcher modifies the relevant policy code, including the memory-update operator , action-selection operator , or working-memory schema . A memory-migration operator preserves or transforms relevant task state when the schema changes.
-
The revised policy immediately controls subsequent interactions from the current environment state, and its updates are retained for future tasks.
Importantly, CPP does not perform gradient-based training or update the frozen LLM's parameters. The interaction data are simply the observations and actions generated during normal task execution. The critic examines recent interaction experience together with the task specification and current policy, then determines whether a revision is necessary. If so, the patcher updates the executable code, and the agent continues interacting with the environment using the revised policy. No environment reset, additional trajectory annotation, or model-parameter update is required.
Therefore, the term "learning" refers to incorporating naturally collected online interaction experience into the executable policy, enabling immediate behavioral improvements and accumulating reusable capabilities across tasks.
Response to Reviewer 5C6H (1/4)
We thank the reviewer for the thoughtful comments. We clarify these aspects and provide additional experimental evidence below.
Clarification on Critic-Generated Revision Feedback. We would like to clarify that δek is not a collection of independent evaluation scores. Instead, it represents actionable revision instructions generated by the critic through a joint assessment of recent interactions, the task objective, and the current executable policy. Action quality, observation consistency, and subgoal correctness are considered together to identify the underlying causes of policy failures. The critic then translates the diagnosis into concrete modifications to the working-memory operations or action-selection logic, providing targeted guidance for the patcher.
Effectiveness of Revision Feedback. The effectiveness of critic-generated feedback is reflected in the resulting policy improvements. As shown in Table 2, removing the critic substantially degrades task performance. For example, with GPT-4.1, success rates decrease from 98.5% to 30.8% on ALFWorld and from 94.3% to 56.0% on MiniGrid. These results demonstrate the importance of critic-guided policy refinement. We will further clarify the formulation and role of δek in the revised paper.
To quantitatively evaluate the contribution of diagnostic revision points, we conducted an additional w/o Revision Points ablation that removes δek while retaining the critic's binary revision decision cek. In this variant, the critic still determines whether a policy update is needed, but the patcher must identify what to modify without receiving explicit diagnostic guidance. We evaluate this variant across three benchmarks and three backbone models.
Effectiveness of Diagnostic Revision Points. Removing δek consistently reduces success rates across all nine settings, with an average degradation of 22.1 percentage points. The impact is particularly pronounced for GPT-4.1, with performance drops of 65.7, 50.3, and 35.7 percentage points on ALFWorld, MiniGrid, and AppWorld, respectively. Similar trends are observed across DeepSeek-V4-Flash and GLM-5.3-Flash, although the magnitude varies with backbone capability.
These results highlight that identifying when to revise a policy is insufficient without actionable guidance on what to revise. By translating interaction-level observations into targeted revision points, δek provides explicit guidance for modifying working-memory operations and action-selection logic. The ablation demonstrates that diagnostic revision feedback plays a critical role in effective policy refinement, beyond the binary decision of whether an update is needed.