- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a “decoupled policy extraction” paradigm for offline RL, which separates actor learning from critic-guided policy improvement. Specifically, a behavior cloning (BC) actor is trained to model the offline data distribution, while a separately learned critic selects the highest-valued action from multiple actor-generated candidates at inference time. The framework aims to address critic-error amplification and the support–value trade-off associated with coupled policy improvement. The experiments report performance improvements with several critic backbones, including a basic Q-learning critic, and the authors discuss potential benefits for large-scale policy models.
Critical strengths
The paper provides a clear motivation for decoupling actor learning from policy improvement in offline RL. The proposed framework is simple and flexible, supporting different critic backbones. Extensive experiments across 30 OGBench tasks demonstrate its effectiveness.
Critical weaknesses
1.The paper primarily offers a unified perspective on behavior-based proposal generation and critic-based selection, a design already explored in prior work. Its contribution is mainly empirical and conceptual, with a limited methodological advance over existing approaches. 2.The main results use task-specific budgets, they do not establish how well the framework performs when the budget cannot be tuned for each task. 3.The support–value trade-off is examined only for CQL on Cube-Double, which is insufficient to establish it as a general limitation of coupled policy improvement. Moreover, Section 5.4 attributes the proposed method’s performance decline at larger N to the selection of overestimated OOD actions, leaving unclear whether decoupling resolves this trade-off or merely shifts it to the choice of N.
Questions for authors
Comparison with related methods. The proposed framework closely resembles existing approaches such as SfBC and EMaQ, which also combine behavior-based action generation with critic-based action selection. Although these methods are discussed in Section 6.2, they are not included in the experimental comparisons. The authors should further clarify the key differences and provide stronger evidence of the advantages of the proposed framework over these closely related methods. Candidate budget selection and reproducibility. Table 5 selects N separately for each method and task from {1,2,4,8,16,32} using the five-seed mean. Is this selection based on validation or test performance? Why are the selected N values for Q-Learning not reported? Generality of the support–value trade-off. The support–value trade-off is demonstrated only using CQL on a single task in Section 3.2. Could the authors provide further evidence that this observation generalizes to other offline RL methods and tasks? Writing and presentation issues. Several aspects of the paper could be clarified to improve readability. (1) The Related Work section appears relatively late. Introducing closely related methods earlier would help readers better understand the novelty of the work. (2) Abbreviations such as EDP and CRR should be briefly explained when first introduced, rather than leaving their descriptions to the appendix. (3) Section 5.1 provides limited information about the environments and task objectives, making the results for task1–task5 in Table 1 difficult to interpret. (4) The caption of Table 1 should explain the meaning of the blue-highlighted values, including whether ties are highlighted.