acceptodds
Under review as a conference paper at ICLR 2027

Decoupling Policy Extraction for Offline Reinforcement Learning

Abstract

Offline RL methods commonly jointly train the actor and critic, where the critic is used to guide the actor toward higher-value actions. This coupled learning process is well motivated in online RL, where an improved actor collects new data that can further update the actor and the critic. However, training data remains fixed in offline RL, making actor-side policy improvement unable to generate new data to validate or correct the critic. Moreover, retaining this coupled paradigm leads to two related challenges. Firstly, actor updates can drift toward high-valued but potentially out-of-distribution (OOD) actions and amplify critic overestimation. Secondly, conservative value estimation or behavior-cloning regularization creates a difficult trade-off between suppressing OOD actions and selecting high-value actions within the data-supported region. Motivated by this observation, we revisit the conventional offline RL paradigm and propose decoupling policy improvement from actor training. Specifically, we train the actor solely to model the behavior distribution and perform policy improvement at inference time by reranking multiple actor-generated proposals with a separately learned critic. We refer to this paradigm as the decoupled policy extraction paradigm. Under such paradigm, the actor provides behavior-supported action candidates, while the critic performs value-based selection within this candidate set. Extensive experiments show that the decoupled policy extraction paradigm outperforms both behavior cloning and jointly learned offline RL methods, while remaining effective even with a naive Q-learning critic.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, offline reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_x7C5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a “decoupled policy extraction” paradigm for offline RL, which separates actor learning from critic-guided policy improvement. Specifically, a behavior cloning (BC) actor is trained to model the offline data distribution, while a separately learned critic selects the highest-valued action from multiple actor-generated candidates at inference time. The framework aims to address critic-error amplification and the support–value trade-off associated with coupled policy improvement. The experiments report performance improvements with several critic backbones, including a basic Q-learning critic, and the authors discuss potential benefits for large-scale policy models.

Critical strengths

The paper provides a clear motivation for decoupling actor learning from policy improvement in offline RL. The proposed framework is simple and flexible, supporting different critic backbones. Extensive experiments across 30 OGBench tasks demonstrate its effectiveness.

Critical weaknesses

1.The paper primarily offers a unified perspective on behavior-based proposal generation and critic-based selection, a design already explored in prior work. Its contribution is mainly empirical and conceptual, with a limited methodological advance over existing approaches. 2.The main results use task-specific budgets, they do not establish how well the framework performs when the budget cannot be tuned for each task. 3.The support–value trade-off is examined only for CQL on Cube-Double, which is insufficient to establish it as a general limitation of coupled policy improvement. Moreover, Section 5.4 attributes the proposed method’s performance decline at larger N to the selection of overestimated OOD actions, leaving unclear whether decoupling resolves this trade-off or merely shifts it to the choice of N.

Questions for authors

Comparison with related methods. The proposed framework closely resembles existing approaches such as SfBC and EMaQ, which also combine behavior-based action generation with critic-based action selection. Although these methods are discussed in Section 6.2, they are not included in the experimental comparisons. The authors should further clarify the key differences and provide stronger evidence of the advantages of the proposed framework over these closely related methods. Candidate budget selection and reproducibility. Table 5 selects N separately for each method and task from {1,2,4,8,16,32} using the five-seed mean. Is this selection based on validation or test performance? Why are the selected N values for Q-Learning not reported? Generality of the support–value trade-off. The support–value trade-off is demonstrated only using CQL on a single task in Section 3.2. Could the authors provide further evidence that this observation generalizes to other offline RL methods and tasks? Writing and presentation issues. Several aspects of the paper could be clarified to improve readability. (1) The Related Work section appears relatively late. Introducing closely related methods earlier would help readers better understand the novelty of the work. (2) Abbreviations such as EDP and CRR should be briefly explained when first introduced, rather than leaving their descriptions to the appendix. (3) Section 5.1 provides limited information about the environments and task objectives, making the results for task1–task5 in Table 1 difficult to interpret. (4) The caption of Table 1 should explain the meaning of the blue-highlighted values, including whether ties are highlighted.

Reviewer_Z71oOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether offline policy improvement should be separated from actor training. It freezes a behavior-cloned proposer and uses an independently learned critic to select the highest-valued action among N candidates at deployment. The authors evaluate this procedure with IQL, transitive, and Q-learning critics on 30 OGBench tasks. They also examine actor drift, vary the candidate budget, and evaluate the method on two real-robot tasks. They report substantial aggregate gains and argue that decoupling avoids critic-driven actor drift while simplifying the trade-off between support and value.

Critical strengths

The decomposition is simple and useful in practice. The experiments show substantial improvements over FM-IQL across six environments, with statistics from five seeds. The authors also compare against a coupled policy with reranking. On Puzzle, decoupled extraction reaches 78.2% at N=16, compared with 68.9% for the reranked coupled actor. Varying the candidate budget and measuring inference costs helps assess the trade-off at deployment.

Critical weaknesses
  1. The paper does not establish enough new understanding of the mechanism. It acknowledges IDQL and V-GPS; IDQL already uses independently learned behavior proposals and greedy critic selection. Much of the paper's added contribution therefore depends on the diagnostics. BC95, however, measures projected deviation from a BC reference; it does not establish whether actions are truly OOD or whether the critic overestimates their values. Standard IQL's critic is also independent of the actor, so the discussion should distinguish its one-way value-guided extraction from reciprocal feedback between actor and critic. Increasing N still involves a trade-off between support and ranking error.
  2. The evaluation does not fully establish whether the gains hold after accounting for selection and inference budget. For BC+IQL/Transitive, the authors select N for each method and task using the five-seed mean, without specifying whether they use independent validation. The matched-reranking comparison covers only Puzzle, where the coupled actor has higher mean success at N=2 and N=4. The broader claim needs further validation and a clearer statement of its limits.
Questions for authors
  1. Can you distinguish one-way critic-guided actor fitting from reciprocal feedback between actor and critic? Please provide a focused calibration check of the actor-drift diagnostic against returns, or narrow the mechanism claims accordingly.
  2. In the candidate-budget sweep, were the seeds and episodes used to select N independent of those used for final evaluation? Did the baselines receive comparable tuning budgets? Please report summary results with N fixed or selected independently, and clarify how the coupled and decoupled policies share the critic in that comparison.
Reviewer_FtNsOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper argues that offline RL methods that use the critic to train the actor ("coupled" policy extraction, e.g., AWR in IQL and BC regularized Q maximization in TD3+BC) suffer from an OOD amplification loop between actor drift and critic overestimation, and from a support–value trade-off induced by conservative or BC regularization. It proposes "decoupled policy extraction": a flow-matching BC actor is trained and frozen, a critic is trained separately (Q-learning, IQL, or TRL), and at test time the action with the highest Q among NN actor samples is executed (Eq. 4–5).

Critical strengths
  • Clean, practically useful evidence for sampling-based extraction. The same frozen flow-matching proposer is paired with three critics on 30 tasks, and best-of-NN selection improves over the single-sample actors at negligible inference cost. The finding that a plain Q-learning critic is competitive is also a useful data point. This is the only reason I see for acceptance.
Critical weaknesses
  1. No new insight: decoupled policy extraction is long established. Separating value learning from policy extraction is the defining property of in-sample offline RL since IQL [1] and IVR [2]. The critic is trained only on dataset actions and never queries the actor, so there is no actor-to-critic feedback, and the policy is extracted in a separate step. The test-time rule in Eq. (4)–(5), sampling NN actions from a behavior model and executing the argmax of QQ, is the policy of BCQ [3] and EMaQ [4]. Pairing an IQL critic with a generative behavior model and critic-based selection is exactly IDQL [5], and SfBC [6] and V-GPS [7] do the same with other critics and with frozen generalist robot policies. Even the "single inference-time knob" NN (Sec. 4.3) is known: best-of-NN sampling satisfies KL(πN∣πβ)≤log⁡N−(N−1)/N\mathrm{KL}(\pi_N | \pi_\beta) \le \log N - (N-1)/N, and Q-chunking [8] already uses this to read NN as the regularization strength.

  2. The analysis does not explain the gains, and the paradigm keeps the trade-off it claims to remove.

    • The headline baseline has no feedback loop. The OOD amplification loop of Sec. 3.1 needs a critic trained on actor actions. FM-IQL's critic never sees actor actions, and App. D.1 itself states that the AWR actor is trained only on dataset actions. Fig. 2 shows drift only for EDP-CRR and EDP-TD3, whose critic or advantage estimates query policy samples, so it does not explain the FM-IQL → decoupled-IQL gain in Fig. 1.
    • The gain comes from reranking, not from decoupling. In Table 2, reranking the coupled FM-IQL actor raises success from 32.0 to 71.6 ± 10.6 (N=4N = 4), and the decoupled actor reaches 78.2 ± 7.7 (N=16N = 16), within one standard deviation. Most of the improvement is best-of-NN selection, which is the known result.
    • The trade-off is moved into NN. Sec. 4.2 argues that the critic need not handle OOD actions, but Sec. 5.4 attributes the decline at large NN to OOD candidates that the critic overvalues. NN therefore plays the role of αCQL\alpha_{\mathrm{CQL}} in Fig. 3. Table 5 tunes NN per task from 1,…,32{1, \dots, 32} "using the five-seed mean", apparently on the same seeds reported in Table 1, while FM-IQL and TRL get no comparable per-task tuning.

[1] Kostrikov et al. Offline Reinforcement Learning with Implicit Q-Learning. ICLR 2022.

[2] Xu et al. Offline RL with No OODing via Implicit ValueRegularization. ICLR 2023.

[3] Fujimoto et al. Off-Policy Deep thout Exploration. ICML 2019.

[4] Ghasemipour et al. EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline Online RL. ICML 2021.

[5] Hansen-Estruch et al. IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies. arXiv 2023.

[6] Chen et al. Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling. ICLR 2023.

[7] Nakamoto et al. Steering Your Generalists: Improving Robotic Foundation Models via Value Guidance. CoRL 2024.

[8] Li et al. Reinforcement Learning with Action Chunking. 2025.

Questions for authors

See the weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.