acceptodds
Under review as a conference paper at ICLR 2027

RAPO: Role-Adaptive Policy Optimization in Offline Reinforcement Learning

Abstract

Policy regularization in offline reinforcement learning balances policy improvement against reliance on uncertain value estimates. This balance can differ between selecting actions for execution and supplying actions for critic bootstrapping, yet methods such as TD3+BC couple these roles through a shared policy. We propose Role-Adaptive Policy Optimization (RAPO), which adapts policy-update coefficients according to their roles in value learning and execution. RAPO learns these coefficients by differentiating through candidate policy updates formed using the base algorithm's actor objective. For TD3+BC, RAPO separates bootstrap and execution actors and adapts their coefficients independently: the bootstrap objective penalizes policy-induced changes in target values, while the execution objective evaluates a local policy-improvement surrogate. For IQL, whose value learning is already independent of the execution actor, RAPO preserves the original value updates and adapts only the inverse temperature in advantage-weighted policy extraction. Experiments on D4RL locomotion and AntMaze tasks show improvements over both base algorithms, with larger gains for TD3+BC, whose RAPO instantiation outperforms baselines on average.

Primary area:
reinforcement learning
Keywords:
Offline Reinforcement Learning, Adaptive Regularization, Meta-Gradient Learning, Policy Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9KEwOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that using a single policy for both action execution and bootstrapping is problematic and proposes RAPO, a method that separates the policy's roles and trains them accordingly. In TD3+BC, the method splits the actor into an execution actor and a critic target actor, and instead of fixing the coefficient alpha, which is usually treated as a hyperparameter, it learns and adjusts alpha during training. The method can also be extended to IQL, and the paper reports results on D4RL.

Critical strengths

Given that the regularization parameter is usually fixed, the paper takes an interesting approach by adapting alpha during training.

Critical weaknesses

The paper does not sufficiently justify its design choices. There is neither a theoretical argument nor an experiment showing that using a single policy for two roles is actually problematic, which makes the motivation hard to accept. In addition, the method consists of many components that are presented without specific justification, so it is unclear why each part of the method is necessary.

The experiments are insufficient. Major baselines such as ReBRAC are missing, and most of the reported results do not appear to differ from the baselines once the standard deviations are taken into account.

Questions for authors
  • Is there any theoretical or empirical evidence that using a single policy for both roles in TD3+BC is actually a problem? Since this is the main motivation of the paper, I think it is important to justify it.
  • Compared with the numbers reported in the ReBRAC paper, there appears to be no performance difference. Does this not suggest that fixed coefficients can perform just as well? Given that the authors cite ReBRAC, a direct comparison seems important.
  • Taking the standard deviations in Table 1 into account, the results do not seem to demonstrate the superiority of RAPO. I am curious how the method performs on other benchmarks such as Adroit or OGBench.
  • The trends in Figure 2 look quite monotonic. Could a simple hand-designed schedule achieve the same effect?
  • In Section 4.3, why does the Q variant perform worse? I would like to know the authors' explanation for the ablation results in general.
  • In the ablations of Sections 4.3 and 4.4, if the coefficient learning rate were also tuned for each control condition, would they not reach similar performance? Table 7 suggests that the results vary considerably with the learning rate, so an experiment that rules out this effect seems necessary.
  • Unlike the baselines, the proposed method uses two policies and a meta-learning procedure. How much additional computational cost does this incur?
Reviewer_hfTyOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes RAPO, a role‑adaptive meta‑gradient offline reinforcement learning method that decouples execution and bootstrap actors for TD3‑style algorithms to learn distinct regularization coefficients through role‑tailored surrogate objectives.

Critical strengths

It introduces a novel role-aware perspective that highlights the conflicting demands of the execution and bootstrap policy roles in offline RL, and achieves empirical improvements over strong TD3-style baselines on the D4RL benchmarks.

Critical weaknesses
  1. The convergence and superiority of the proposed method cannot be theoretically proven.
  2. The experiments are insufficient: the hyperparameter sensitivity analysis and ablation studies of the proposed method are not complete enough, the benchmark tasks lack manipulation tasks, the number of random seeds is insufficient, and there is a lack of hardware efficiency analysis of the proposed method.
Questions for authors
  1. Why integrate the proposed method into TD3BC and IQL? Can it be integrated into other SOTA offline DRL methods? If so, experimental validation is recommended.
  2. The ablation is incomplete: the gains brought by the dual-actor structure and the gains from meta-gradient adaptation are not disentangled.
  3. The paper lacks a hardware efficiency comparison between the proposed method and the baselines, such as wall-clock running time, GPU memory usage, and FLOPs, so the runtime efficiency of the proposed method cannot be seen.
  4. The paper lacks a convergence analysis of the proposed method and a theoretical analysis of why it can outperform existing offline DRL methods.
  5. The experiments only include locomotion-type tasks such as MuJoCo and lack tasks such as robotic manipulation. It is recommended to add relevant experiments to demonstrate the generality of the proposed method.
  6. The proposed method introduces multiple hyperparameters; for example, the meta-gradient update interval and other hyperparameters have not been subjected to sensitivity testing.
  7. How does the proposed method perform on low-quality datasets of HalfCheetah, Hopper, and Walker2d? It is recommended to add relevant comparative experiments.
  8. Many offline DRL works use at least five random seeds for experiments. It is recommended that the authors add performance validation of the proposed method under more random seeds.
Reviewer_aSnYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The work proposes a scheme that regularises execution and value estimation policies separately. Adaptive learned regularisation coefficients push both policies towards the behaviour policy with different strength. The authors provide comparisons with baselines on a range of benchmark tasks. Ablation studies are provided on the effect of using different objectives as outer losses and the impact of having regularisation coefficients adapt throughout training as opposed to remaining fixed to the final converged value.

Critical strengths

The idea of separating value-estimation and behaviour policy is interesting and provides a novel perspective on behaviour-regularised offline RL methods. This is a challenging research question that has the potential of greatly improving the applicability of offline RL.

The empirical evaluations provide evidence that such a scheme can be beneficial in practice. The evaluation covers a range of benchmark tasks and comparison with ASPC shows strong performance relative to a strong similarly adaptive-regularisation baseline. The ablation studies motivate empirically the outer loss objectives used for the regularisation coefficients.

Critical weaknesses

The motivation for the losses was difficult to follow. My impression is that the outer losses are chosen to incentivise desirable behaviour (consistent policy improvement and stable bootstrapped values) but that the choice of outer loss is not unique and is instead guided by a combination of mathematically-derived and empirical (eg the "Q" and "Linear" ablations) heuristics.

I found some aspects of the presentation of the paper hard to follow. In particular, how the outer losses come into play in the broader algorithm and the motivation for them was hard to follow. I found it difficult to parse how exactly the different ingredients came into play from the description in Algorithm 1.

Questions for authors
  • What is the relevance of Lemma 1? It seems to me that the rest of the section does not reference it and instead uses another quadratic approximation, with Lemma 1 not really being used anywhere.
  • What other reasonable choices for outer losses could be considered? Are these unique or could a family of such reasonable losses lead to useful adaptive-regularisation algorithms?

Comments on the paper

3 comments

Thanks for the additional experiments. Two of my main concerns remain.

  1. Motivation and design. The paper never shows why using one policy for both execution and bootstrapping is a problem, or why each component is needed. The response gives a final-score comparison, but that doesn't show the problem exists. Is there any direct evidence that the two roles actually need different regularization? The same applies to the other components, which are still justified only by ablation scores rather than by an explanation of what goes wrong without them.

  2. Experiments. The response does not address my question about other benchmarks such as Adroit or OGBench. With only 15 D4RL tasks, where many results are within one std of the strongest baselines, it is hard to tell whether RAPO is actually better. The added CIs don't help here, since they are only against TD3+BC and IQL.

AuthorsOpenReview ↗

Response to Reviewer 9KEw

Thank you for the follow-up. We clarify why the two roles use different adaptation criteria and report comparisons with ReBRAC and ASPC on additional benchmarks.

1. Why separate the actors and adaptation criteria?

In TD3+BC, the same actor selects actions for execution and, through its target copy, supplies actions for Bellman targets. MCEP [1] found that relaxing the constraint on the evaluation policy could improve its return while retaining a stronger constraint on the policy used for Q estimation. This motivates allowing the two roles to adapt their regularization strengths separately.

As an additional diagnostic for this discussion, we evaluated the two RAPO outer losses at the same shared actor and coefficient on AntMaze-large-play. Their stop-gradient hypergradients had opposite signs in 31 of 100 paired evaluations. This shows that a shared coefficient can receive opposing signals from the two criteria. The realized changes from one meta-step were very small in this diagnostic. The end-to-end comparison in revised Table 11 gives complementary evidence. TD3+RAPO scores 83.0 overall versus 81.1 for the adaptive shared-policy control, while the shared control has a slightly higher AntMaze average, 70.7 versus 70.0.

With a shared actor, combining the two losses still produces a single policy update that affects both roles. With separate actors, holding the other training variables fixed at the update where each coefficient takes effect gives

\qquad \frac{\partial y_B^{\mathrm{next}}}{\partial \alpha_E}=0.$$ The execution criterion can therefore favor candidate actions with higher estimated value after a curvature correction. The bootstrap criterion can separately limit how much the candidate moves the targets the critic must fit. Its per-transition RMS keeps positive and negative target changes from canceling. The two actors still interact through the critic over training. **2. Additional comparisons with strong baselines** Revised Table 1 includes ReBRAC and ReBRAC+RAPO on all 15 D4RL tasks, with four training seeds per task. We reused the tuned TD3+RAPO hyperparameters for each task without a separate ReBRAC+RAPO tuning sweep. | D4RL average | ReBRAC | ReBRAC+RAPO | |---|---:|---:| | Locomotion | 85.84 | **88.54** | | AntMaze | 77.08 | **83.79** | | All 15 tasks | 82.34 | **86.64** | On Adroit, TD3+RAPO has a higher mean than ASPC on all three tasks. ReBRAC has the highest mean on Door-expert and Relocate-expert, and TD3+RAPO has the highest mean on Hammer-cloned. We used four training seeds per method and task for these additional experiments within the discussion period. | Adroit dataset | TD3+RAPO | ASPC | ReBRAC | |---|---:|---:|---:| | Door-expert | 105.19 ± 0.69 | 103.75 ± 3.95 | **105.81 ± 0.75** | | Hammer-cloned | **7.90 ± 3.58** | 5.58 ± 5.14 | 2.16 ± 0.79 | | Relocate-expert | 109.22 ± 0.46 | 107.74 ± 1.55 | **109.52 ± 1.11** | On OGBench AntMaze, TD3+RAPO has a higher mean success rate than ASPC on all ten reported Medium and Large tasks. Table 14 gives the per-task results, including the TD3+BC comparison. Table 10 reports nominal 95% Welch intervals against all six baselines. On Locomotion, the TD3+RAPO intervals are above zero for all six. On the full suite, the intervals against ASPC and ReBRAC include zero. **Changes in the revised manuscript.** Sections 3.1 and 5 connect role separation to MCEP, and Sections 3.2–3.3 explain the two criteria. Table 1 includes the ReBRAC comparison. Tables 10–11 report the confidence intervals and the adaptive shared-policy control, and Tables 14 and 16 report OGBench and Adroit. [1] Xu et al., *Mildly Constrained Evaluation Policy for Offline Reinforcement Learning*. https://openreview.net/forum?id=imAROs79Pb

Thank you for considering our paper and following the discussion. We would like to highlight two comparisons in the revised manuscript that address the main questions raised by the reviewers.

Role separation with adaptive regularization. Table 11 compares TD3+RAPO with a shared-policy control that also adapts its coefficient and selects its learning rate from the same grid. TD3+RAPO improves the overall D4RL mean from 81.1 to 83.0, with the largest gain on Hopper. The shared control has a slightly higher AntMaze mean. Section 3.3 explains why execution and bootstrapping use different criteria, and Algorithm 1 clarifies how each criterion enters the update.

Applying RAPO to ReBRAC. Across all 15 D4RL tasks, adding RAPO to ReBRAC raises the overall mean from 82.3 to 86.6, with higher means on both Locomotion and AntMaze (Table 1). We reused the RAPO hyperparameters selected for TD3+RAPO without a separate tuning sweep for ReBRAC. The revision also reports confidence intervals for TD3+RAPO and IQL+RAPO against six baselines (Table 10) and adds navigation, random-policy, and manipulation experiments (Tables 14–16). TD3+RAPO achieves higher mean success rates than ASPC on all ten reported OGBench AntMaze tasks and higher mean returns on all three reported Adroit tasks.

We would appreciate your taking these results into account in the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.