- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework named CAPER, which achieves improvement by alternately using reinforcement learning (RL) policy learning and failure-based LLM action prior optimization. This method identifies the weak points in the current policy-prior pair through low-return trajectories, retrieves relevant task knowledge, and generates revised prompt candidates for subsequent RL training. Two prompt selection strategies are considered in the paper: the reward-based selection (RBS) evaluates candidate solutions through environmental interaction, while the trajectory consistency selection (TAS) selects based on the consistency of prompts with actions in high-return trajectories. During training, the LLM remains in a frozen state.
Critical strengths
- A reasonable framework for adaptively incorporating LLM action priors during reinforcement learning training.
- Extensive experimental evaluation, including useful component analysis.
Critical weaknesses
- The connection between policy failure and adjustment of action priorities has not been fully established.
- The contribution of closed-loop optimization to additional task knowledge and computational costs needs to be more clearly defined.
Questions for authors
Q1. Failure Attribution and Action Prior Quality.
Can the author analyze whether the low-return trajectories are mainly caused by the missing candidate actions in the LLM prior, or by the incorrect selection of the reinforcement learning strategy? In particular, measuring the coverage of the optimal action in the candidate action set and the accuracy of the strategy selection separately can help clarify the contribution of prior optimization.
Q2. RBS Refinement Trigger.
In Section 3.3.1, when the current return exceeds the previously stored reference return, RBS will trigger a new round of optimization. Why this condition was chosen instead of triggering optimization when performance stagnates or declines? Since the strategies change between each round of optimization, how does this method distinguish between strategy improvements and improvements in action priors?
Q3. Task Knowledge and Computational Efficiency.
The optimized prompt statement incorporates detailed task-related knowledge extracted from the environment description and the source code. Can the author further quantify the extent to which the performance improvement brought about by the guided optimization through the optimization process is derived from the optimization process itself, compared to providing highly task-oriented prompts from the very beginning? Furthermore, Table 10 shows that the training time cost of CAPER is significantly higher than that of DQN-Prior. Can the author provide a more detailed breakdown of the costs for LLM reasoning, retrieval, candidate selection, and reinforcement learning optimization, especially for TAS, which although avoids additional environment runs, still requires repeated generation of LLM?
Q4. Fixed-Dataset Policy Learning.
The CAPER-CQL experiments use a fixed dataset during policy updates, but collect additional online trajectories for action prior optimization. Can the author further explain the envisioned offline reinforcement learning setup and evaluate whether TAS can achieve prior optimization solely through the fixed dataset, without the need for additional interaction with the environment? Such comparisons can help verify the applicability of CAPER when access to the online environment is unavailable.
Q5. Interpretation of the Finite-Sample Guarantees.
Proposition 3.1 provides a selection guarantee under the corresponding overall criterion. However, for TAS, this criterion measures consistency with high-reward trajectories, rather than the expected environmental rewards. Can the author explain under what conditions maximizing trajectory consistency can lead to an improvement in downstream benefits? This distinction is of great significance for understanding the theoretical contribution of TAS.
4 comments
Q6: Interpretation of the selection guarantee
A6: Thank you for distinguishing agreement from return. For TAS, Proposition 3.1 places the selected prompt's population agreement score within 2ε of the best candidate when estimation errors are uniformly bounded by ε. Agreement can guide learning when reference trajectories cover relevant states, their actions remain useful, and the policy can exploit those actions. We tested the score–return relationship with one saved DQN policy fixed per task. We sample eight prompts with different returns using equally spaced return targets, then compute the TAS score for each prompt. Spearman correlation measures agreement between score and return ranks, computed as their Pearson correlation with average ranks for ties. Ten TAS scoring repetitions against the same returns yield positive correlations throughout. The table gives their mean and standard deviation (Appendix B.15.4, Table 13; Section 4.5). Figure 17 also shows higher returns with TAS than random prompt selection under identical components and interaction budgets. Together with Q1's coverage analysis, these results support trajectory agreement as a refinement criterion.
| Task | Spearman correlation |
|---|---|
| Overcooked Salad | 0.67 ± 0.09 |
| VirtualHome Food Preparation | 0.62 ± 0.14 |
| TextWorld-Commonsense | 0.71 ± 0.13 |
We appreciate the reviewer's comments, which helped us improve the clarity and completeness of the paper.
The rebuttal addresses most of my concerns. Some limitations remain regarding the RBS trigger (Q2) and the scope of the theoretical guarantee (Q5). Overall, I am satisfied with the responses and maintain my original rating.
Thank you for your follow-up. We would like to clarify the two remaining points.
Regarding the RBS trigger (Q2), between refinement rounds, the prompt stays fixed while the policy learns. The trigger therefore detects progress under unchanged guidance. Once triggered, the current prompt and new candidates are compared with the same policy held fixed, separating prompt changes from concurrent policy updates. We choose progress-triggered refinement to address remaining failures as learning advances, rather than wait for stagnation. Non-improvement-triggered refinement is also a viable alternative, as supported by our Salad experiment reported in our initial response.
Regarding the scope of the theoretical guarantee (Q5), TAS favors prompts that propose actions found in high-return reference trajectories. This can help when those trajectories cover states relevant to the current policy, their actions remain useful, and the policy can select them from the candidate set. Proposition 3.1 bounds selection error under this agreement criterion. The practical relevance of this agreement criterion is supported by the positive Spearman correlations in Table 13, the fixed-policy coverage analysis in Table 12, and the TAS-versus-random comparison in Figure 17.
Thank you again for your feedback. We hope these clarifications address your remaining concerns about the RBS trigger and the interpretation of the TAS guarantee.
We sincerely thank the reviewer for the constructive feedback on our adaptive prior framework. We respond below with the additional evidence and clarifications.
Q1: Failure attribution
A1: Thank you for suggesting this analysis. We compare prompts before and after the first accepted refinement in three CAPER-DQN (TAS) runs on TWC, using 136 state-history inputs from low-return trajectories in total. Each run holds the inputs and Q network fixed. Correct actions follow the task's object-placement rules. The table reports candidate coverage, accuracy conditional on a correct candidate being available, and overall accuracy under the evaluation policy, as percentages with means and standard deviations across runs. Missing candidates fall from 28.93% to 15.17% of decisions, while incorrect selection despite an available correct candidate changes from 16.68% to 16.50%. Improved coverage thus accounts for most of the 13.94-percentage-point gain in overall accuracy. These results are in Appendix B.15.3, Table 12, and Section 4.3.
Q2: Refinement trigger
A2: Thank you for suggesting refinement when progress stalls. We tested a non-improvement trigger on Salad: after initialization, refinement starts when the current mean return does not exceed the preceding measurement. Under the same RL-training budget as the original method, this variant achieves 0.959 ± 0.239, reported as mean ± standard deviation across runs. Compared with the return of 0.98 for original CAPER-DQN (RBS) (Appendix B.4, Figure 10), this result indicates that both approaches support effective learning, although their motivations differ. Original RBS refines unconditionally at initialization, then triggers when the current policy–prior return exceeds the return stored when the preceding prompt was selected. It uses failures remaining after policy progress to guide further refinement. The alternative seeks new guidance when progress stalls. The joint return determines when to refine, while comparisons with the policy held fixed assess the contribution of changes to the prior, as illustrated by the coverage and selection-accuracy analysis in Q1.
Q3: Continued refinement beyond initialization
A3: Thank you for raising the role of initialization. Static Full-KB Prior + DQN distills all task knowledge into a fixed guide before training, yet CAPER achieves higher returns (Figure 8(b)). Single Refinement freezes the prompt after its first refinement. Continuing refinement on TWC raises return from 4.08 to 4.67 for RBS at 290 total environment episodes, and from 2.64 to 5.35 for TAS at 260 episodes, with equal budgets within each selector (Appendix B.7, Figure 15). CAPER also outperforms w/o Failure Feedback, which retains the knowledge base and refinement components but replaces failures with the task description (Figure 8(a)). These controls support continued, experience-guided revision beyond knowledge-based initialization.
Q4: Computational costs
A4: Thank you for asking about these costs. Appendix B.15.6, Table 16 provides a detailed time breakdown for CAPER-GFlan (TAS) in Table 10. The logged total is 41.16 minutes, corresponding to 0.69 hours in Table 10. TAS thus replaces candidate-specific environment validation with repeated LLM computation on stored trajectories. Separately, the fixed-input benchmark on one RTX 4090D reports average five-action generation latencies of 0.948 seconds for RBS and 0.947 seconds for TAS on inputs from TWC (Appendix B.15.6, Table 15).
Q5: Fully offline refinement
A5: Thank you for suggesting this variant. We tested fully offline TAS on Salad, drawing both low-return refinement feedback and high-return selection references from 51 fixed trajectories with 1,030 transitions. This variant follows the original TAS training configuration, but uses only trajectories from the fixed dataset for prompt refinement and candidate scoring. Policy training and prompt adaptation use zero additional environment episodes. As shown below, fully offline TAS achieves a higher mean return than CQL-Prior using fixed experience alone. We also include original TAS with online prompt adaptation for comparison. Values are means and standard deviations. The comparison is in Appendix B.15.5, Table 14, and Section 4.2.