Think Before Retrieval: Latent Policy Refinement for Efficient Agentic RAG

Agentic retrieval-augmented generation (RAG) improves knowledge-intensive question answering through iterative reasoning and search, but repeatedly decides whether retrieval is warranted. Wrong decisions cause over-search when available knowledge is sufficient and under-search when necessary evidence is missing, harming search efficiency and answer quality respectively. Existing methods typically assess retrieval decisions after execution, leaving no explicit mechanism to revise a pending action beforehand. We introduce LPR-RAG, which formulates retrieval control as pre-action Latent Policy Refinement, selectively updating latent decision states before committing to either SEARCH or NONSEARCH decisions. Training proceeds in two stages. In Stage I, ordinary on-policy trajectories optimize the execution adapter with answer and format rewards, while paired counterfactual preferences supervise the decision pathway. To construct this supervision, selected states are forked into matched SEARCH and NONSEARCH continuations under the same frozen policy snapshot and budget; terminal success determines the preferred action, retrieval and processed-token costs break ties, and unresolved pairs are excluded from decision supervision. The resulting preferences train an initial gate and a shared two-step updater to correct initially wrong decisions while preserving correct ones. Stage II freezes all preceding modules and trains a selector to choose the minimum effective refinement depth of zero, one, or two updates. At inference, LPR-RAG applies the selected depth before acting, without counterfactual rollouts, gold answers, or external judges. Across seven single-hop and multi-hop QA benchmarks, LPR-RAG consistently improves answer quality and retrieval decision quality by reducing both redundant and insufficient search. Code is available HERE.