CAPER: Closing the Loop between LLM Action Priors and Reinforcement Learning

Reinforcement learning (RL) often requires extensive environment interaction, while large language models (LLMs) can provide action priors that guide exploration using pretrained knowledge. However, fixed LLM priors may be mismatched with the environment and cannot adapt when grounded failures reveal unsuitable guidance. We propose $\underline{C}$losed-loop $\underline{A}$ction $\underline{P}$rior $\underline{E}$nhancement for $\underline{R}$einforcement Learning (CAPER), which alternates between RL policy learning and failure-guided action-prior refinement. Low-return trajectories identify weaknesses in the current policy–prior pair, which guide the retrieval of relevant task knowledge and the generation of revised prompt candidates. Candidate prompts are selected either through direct return evaluation or through agreement with actions in high-return trajectories, trading evaluation fidelity for environment-interaction cost. Across five tasks from three environments and multiple RL settings, CAPER consistently improves over baselines under total environment-interaction budgets. The implementation is available at https://anonymous.4open.science/r/CAPER-FD27.