acceptodds
Under review as a conference paper at ICLR 2027

Paired Relative-Value Regularization for Offline Reinforcement Learning

Abstract

Offline reinforcement learning (RL) suffers from unreliable value estimates for actions that are insufficiently covered by the offline dataset. Conservative value learning mitigates this issue by suppressing the values of such actions, but overly strong regularization can also limit useful policy improvement. In this work, we propose Paired Relative-Value Regularization (PaRE), which approaches conservative value regularization from a policy-local perspective. PaRE constructs candidate actions around the current policy and regularizes their values relative to the data action observed at the same state, allowing the regularization strength to adapt to the local relative-value structure. We theoretically analyze the relationship between PaRE and policy-local value overestimation, and characterize how its gradient determines the overall regularization strength and its allocation across candidates. Experiments on the D4RL MuJoCo locomotion and Adroit dexterous manipulation benchmarks show that PaRE achieves competitive performance across datasets with different data qualities.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rEPdOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Paired Relative-Value Regularization (PaRE) for offline reinforcement learning. At each dataset state, PaRE compares the critic values of the current policy action and several nearby perturbed actions with the value of the logged action. It aggregates these differences using log-mean-exp and applies a softplus penalty, yielding a regularization signal whose total strength depends on the aggregated gap and whose allocation favors higher-valued candidates. The method is implemented in a TD3+BC-based framework adapted from ReBRAC, with two LayerNorm critics and a behavior-regularized deterministic actor. The theoretical analysis provides a conditional bound on candidate overestimation, a variational interpretation of the regularizer, and a small-radius comparison with mean aggregation that identifies an additional action-gradient penalty. Experiments cover 24 D4RL locomotion and dexterous manipulation datasets. PaRE obtains the second-highest reported MuJoCo average, and the highest reported Adroit average.

Critical strengths
  • The policy-local candidate construction is well motivated for deterministic actor–critic learning. Because the actor's update depends on the critic's action derivative at its current action, regularizing nearby values addresses a part of the critic that directly affects policy improvement. This gives the design in Section 3.1 a clear practical rationale.

  • PaRE is a small addition to an existing actor–critic implementation. It requires additional critic evaluations but no learned behavior model, distance estimator, or ensemble beyond the two existing critics. This simplicity makes the idea relatively easy to test in established offline RL codebases, and the reported runtime offers preliminary evidence that its computational overhead is manageable.

Critical weaknesses
  • A comparison using the exact same backbone with PaRE disabled would make its contribution easier to assess. The implementation includes three-layer networks, critic LayerNorm, a twin-minimum actor objective, and benchmark-dependent optimization settings, while the baseline scores come from earlier papers. Keeping these choices and the behavior-cloning coefficients fixed while setting λ=0\lambda = 0 could help clarify how much of the observed improvement is associated with the proposed regularizer.

  • The ablations provide an initial view of the components, although I would welcome further evidence on the benefit of local neighbors and selective aggregation. Figure 1 uses one seed on HalfCheetah-Medium with the original TD3+BC backbone, and the curves with and without neighbors appear fairly similar. Repeating this comparison across seeds and tasks, together with mean aggregation over the same candidates, would help assess the practical relevance of the mechanism discussed in Proposition 3.

  • Some additional detail on parameter selection would make the robustness of the shared configuration easier to assess. Table 4 uses behavior-cloning coefficients ranging from 0.001 to 0.10, and Figure 2 suggests that this choice can noticeably affect performance. Clarifying their selection procedure and tuning budget, along with modest sensitivity sweeps over the PaRE-specific parameters, would help readers judge how readily the settings might transfer to new datasets.

Questions for authors

Please see the weaknesses.

1 comment

Thank you for your careful review and for acknowledging the motivation for policy-local regularization and the simplicity of PaRE. We add ablation results below and explain how the hyperparameters were chosen.
Answer to Q1&Q2: To address both points, we used the main implementation and compared Full PaRE, No PaRE, Policy-only and Mean aggregation on HalfCheetah-medium and Hopper-medium across 4 training seeds. Within each task, the network architecture, including critic LayerNorm and the twin-minimum actor objective, the BC coefficient and the optimization settings are identical. No PaRE turns off the regularizer. Policy-only removes the local neighbors. Mean aggregation replaces the log-mean-exp with the arithmetic mean over the same candidates.The results are below.

& \text{HalfCheetah-medium} & & \text{Hopper-medium} & \\\\ \hline \text{Variant} & \text{Last100k} & \text{AUC} & \text{Last100k} & \text{AUC} \\\\ \hline \text{Full PaRE} & 66.59 \pm 1.35 & 63.68 \pm 0.82 & 102.38 \pm 3.54 & 91.18 \pm 2.59 \\\\ \text{No PaRE} & 65.58 \pm 0.98 & 61.84 \pm 0.59 & 97.78 \pm 8.72 & 77.50 \pm 14.52 \\\\ \text{Policy-only} & 65.88 \pm 1.14 & 62.96 \pm 0.83 & 97.20 \pm 10.49 & 85.31 \pm 6.76 \\\\ \text{Mean aggregation} & 65.73 \pm 0.17 & 63.10 \pm 0.53 & 100.10 \pm 1.32 & 88.68 \pm 0.97 \\\\ \end{array}$$ Full PaRE is best on both tasks and both metrics. All four variants share the same backbone, BC coefficient and optimization settings. On Hopper-medium, where training is unstable, PaRE also gives a much smaller spread across seeds, so training is more reliable.\ **Answer to Q3:** In the submission, the BC coefficients for each dataset follow existing work, and we did not run a systematic grid search. The PaRE parameters are shared across tasks and were not tuned per task. In all matched ablations, the BC coefficient is kept the same within a task.During the rebuttal we chose the PaRE parameters by their performance on HalfCheetah-medium, then kept them fixed for every task. The ablations above use λ = 0.03, T = 1 and w = 0.1.
Reviewer_Ro9tOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes PaRE, an offline RL method that adds a value regularizer from a policy-local point. At each state, it takes the current policy action, samples a few candidates around it, and penalizes their Q-values relative to the Q-value of the logged action. The paper gives some theoretical analysis of this regularizer and evaluates it on D4RL locomotion and Adroit.

Critical strengths

Value regularization is an important problem in offline RL, and the paper provides a theoretical characterization of its proposed regularizer.

Critical weaknesses
  • The method is CQL with the action samples restricted to a small ball around the policy action, and the paper never says so. If you plug Eq. 4 into Eq. 5, z(s)z(s) is just a log-sum-exp over the candidate values minus the data-action value (Eq. 35 makes this explicit), which is the CQL penalty with a different sampling range. Eq. 6 then wraps it in a softplus. The gradient in Eq. 13 has the same structure as CQL's, a softmax-weighted push-down on the sampled action values and an equal push-up on the data-action value, with the only difference being that the overall strength is the state-dependent g∗g^* instead of a constant α\alpha. Yet the only CQL numbers in the paper seem to be copied from other work. A comparison of CQL and PaRE on the same backbone seems necessary before anything can be said about what PaRE adds.

  • The main results are not strong, and the ablations do not back up the claims. Looking at Tables 1 and 2, I would not say the method clearly beats the existing algorithms. The one ablation (Fig. 1) is a single seed on a single task, and it shows that dropping the local candidates, which is the only genuinely new part, makes no visible difference. On top of that, the implementation uses the ReBRAC recipe and tunes βBC\beta_{BC} per dataset, and Fig. 2 shows that this coefficient alone moves the score a lot. There is no run with the same backbone and tuning but without PaRE, and no sensitivity study for T,w,ρ,λT, w, \rho, \lambda. So I cannot attribute the reported numbers to the regularizer.

Questions for authors
  • Could you train all baselines with the same backbone and recipe as PaRE (Table 3)? The method is a small change on top of existing ones, so I think a fair comparison matters here. CQL on the same TD3+BC backbone in particular.
  • If you run the Fig. 1 ablation on more seeds and tasks, does removing the local candidates still make no difference? If so, why keep them?
  • With T=w=0.1T = w = 0.1 and Q-values in the tens or hundreds, the log-mean-exp should be close to a hard max and the softplus close to a ReLU. Could you show a sensitivity analysis over T,w,ρ,λT, w, \rho, \lambda, and check whether pkp_k and g∗g^* are actually one-hot/step-like during training?
  • How were the per-dataset βBC\beta_{BC} values chosen? If they were picked by evaluation return, that is tuning on the test environment. What happens with one βBC\beta_{BC} for all datasets?
  • Why no AntMaze? That is where conservative value regularization usually matters most.
  • Fig. 3 compares runtime against DOGE and MCQ, which are not in Tables 1 and 2 and are only mentioned once in related work, while ReBRAC, SSAR, and ASPC are missing from Fig. 3. Why are the two comparisons over different sets of methods?
2 comments

Thank you for your careful review and constructive comments. We answer your questions one by one below.
Answer to Q1: With limited time and compute during the rebuttal, we prioritized reproducing CQL, which you specifically raised. We kept the original CQL objective and replaced only the critic with our three-layer, 256-unit critic with LayerNorm, on all 24 tasks. After 1M updates, the average normalized score over the last 100k updates is 84.51 on MuJoCo and 30.71 on Adroit. With the stronger critic, CQL still scores lower than PaRE on average on both benchmarks, so the critic architecture alone does not explain the difference. ASPC and ReBRAC already use a three-layer critic with LayerNorm, so these baselines also use the stronger critic architecture.
Answer to Q2: We used the main implementation and extended the ablation to HalfCheetah-medium and Hopper-medium across multiple training seeds. We compared Full PaRE, No PaRE, Policy-only and Mean aggregation, keeping the backbone, BC coefficient and optimization settings fixed within each task. Policy-only removes the local neighbors, and Mean aggregation uses the same candidates as Full PaRE. The results are below.

& \text{HalfCheetah-medium} & & \text{Hopper-medium} & \\\\ \hline \text{Variant} & \text{Last100k} & \text{AUC} & \text{Last100k} & \text{AUC} \\\\ \hline \text{Full PaRE} & 66.59 \pm 1.35 & 63.68 \pm 0.82 & 102.38 \pm 3.54 & 91.18 \pm 2.59 \\\\ \text{No PaRE} & 65.58 \pm 0.98 & 61.84 \pm 0.59 & 97.78 \pm 8.72 & 77.50 \pm 14.52 \\\\ \text{Policy-only} & 65.88 \pm 1.14 & 62.96 \pm 0.83 & 97.20 \pm 10.49 & 85.31 \pm 6.76 \\\\ \text{Mean aggregation} & 65.73 \pm 0.17 & 63.10 \pm 0.53 & 100.10 \pm 1.32 & 88.68 \pm 0.97 \\\\ \end{array}$$ Full PaRE is best on both tasks and both metrics. On Hopper-medium, where training is unstable, PaRE also gives a much smaller spread across seeds\ **Answer to Q3:** The absolute scale of Q values alone does not decide whether the aggregation approaches a hard max, or whether the response approaches ReLU. How concentrated the candidate weights are depends on the spread of the value differences relative to T, and the response depends on z/w, the ratio of the aggregated paired gap to w. Adding the same constant to all action values in a state changes neither.\ We checked this on HalfCheetah-medium. With λ = 0.03 and w = 0.1 fixed, raising T from 0.1 to 1 increases the effective number of candidates from 2.86 to 7.31 (out of 8). The average gate moves from 0.934 to 0.884, and the Last100k scores are 66.85 and 67.94. The candidate weights therefore change clearly with temperature and are not necessarily close to one-hot. The high gate values also suggest the response may sit near saturation.\ **Answer to Q4:** The per-dataset BC coefficients in the submission follow existing work such as ReBRAC, and we did not run a systematic grid search. Setting the behavior constraint per dataset is not specific to PaRE. The suitable strength of this constraint can vary with data coverage and quality,weaker constraints allow more policy improvement but rely more on the critic estimate, while stronger constraints limit how far the policy moves from the data actions. A single BC coefficient is not necessarily suitable for different tasks.\ **Answer to Q5:** Thank you for suggesting AntMaze. AntMaze also tests sparse-reward propagation, long-horizon credit assignment and trajectory stitching, so its performance reflects more than the effect of a conservative value regularizer. Our evaluation focuses on locomotion and dexterous manipulation under different data quality. We see AntMaze as a valuable complementary evaluation and will consider extending our experiments to this benchmark.\ **Answer to Q6:** We chose MCQ and DOGE for the runtime comparison to examine PaRE's computational overhead relative to methods that require additional learned components. The main performance table focuses on methods that, like PaRE, modify the learning objective within an actor–critic framework. The two comparisons therefore serve different purposes, and we will make this explicit in the paper.

Thanks for the response. Two of my main concerns remain.

  1. Relation to CQL. The response does not address my main point that PaRE is essentially the CQL penalty with candidates sampled around the policy action. The new CQL experiment keeps the original CQL objective and only swaps the critic, which is not the comparison I asked for. What is needed is the CQL penalty in place of the PaRE term on the same TD3+BC backbone, with everything else fixed. I agree that the Q scale alone does not determine how sharp the aggregation is. However, as you note, the gate may be near saturation. In that case, the state-dependent strength is close to a constant, and that was the only difference from CQL I could see. The paper should clearly discuss this relation and show what PaRE adds beyond it.

  2. Source of the improvement. I am still not convinced that the improvement comes from the PaRE term. In the new ablation, No PaRE is within one std of Full PaRE on both tasks, and so is Policy-only. The gain over ReBRAC, which uses the same BC coefficients and differs mainly in the critic penalty, is also small. The main evidence for this paper is empirical, so it should show that the gain comes from the proposed regularizer rather than from the ReBRAC recipe.

Reviewer_KPpvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Paired Relative-Value Regularization (PaRE) for offline reinforcement learning. It samples candidate actions around the current deterministic policy and compares their critic values with that of the logged action at the same state. Log-mean-exp aggregation followed by softplus determines how the conservative correction is distributed and how strong it is. The paper derives a conditional overestimation bound and analyzes gradient allocation and local sensitivity. PaRE is incorporated into a TD3+BC-based learner, with experiments on 24 D4RL tasks showing competitive performance in locomotion and dexterous manipulation.

Critical strengths

PaRE is simple to implement and requires neither a behavior model nor additional critic networks. The analysis separates the regularizer's state-level response from its allocation across candidates and characterizes its additional local sensitivity compared with mean aggregation. The paper also discusses the limitations due to clipping and smoothness. The evaluation uses shared PaRE hyperparameters across 24 tasks. Results are strong on HalfCheetah and Pen, and PaRE has the highest reported Adroit average among the compared methods.

Critical weaknesses
  1. The experiments do not sufficiently isolate PaRE's contribution in the main implementation. They compare PaRE against published baseline results, while PaRE uses a ReBRAC-derived configuration with critic LayerNorm, three-layer networks, and dataset-specific BC coefficients. The component ablations include TD3+BC and other controls, but use the original TD3+BC backbone and cover only one task with one seed. Removing neighbors produces a similar trajectory. The mean-aggregation counterpart analyzed in the theory is not evaluated, so the evidence for both PaRE's incremental gain and the benefit of local selective aggregation remains preliminary. The substantial sensitivity to BC coefficients also makes it important to report how those coefficients were selected.

  2. The experiments do not validate the connection between adaptive correction and actual estimation error. In the error decomposition, a positive gap can reflect genuine advantage or overestimation. The overestimation bound assumes reference accuracy and includes a distance term between the policy and data; it does not establish that reducing the penalty during learning reduces actual overestimation. The visualization shows the computed weights without checking them against estimation errors. The analysis explains how the penalty behaves, but the experiments do not establish when it corrects erroneous values while preserving useful improvement.

Questions for authors
  1. Can you compare full PaRE, no PaRE, policy-only candidates, and mean aggregation on two representative existing tasks using the main implementation? Matching the BC coefficients and using multiple seeds would help test PaRE's incremental contribution and the benefit of local aggregation.
  2. How did you select the dataset-specific BC coefficients and shared PaRE settings? What validation signal and search budget did you use, and were the ablation controls otherwise matched?
  3. Can you compare estimated gaps and correction weights with rollout-based value-error estimates on a small held-out set of states and candidates from an existing task? This would help distinguish genuine advantage from overestimation.
1 comment

Thank you for your review and for noting PaRE’s simplicity and theoretical analysis. To answer your questions, we ran matched-backbone ablations across multiple seeds and a rollout-based value-error diagnostic on held-out states. We answer your three questions below.
Answer to Q1: We compared all four variants using the main implementation on HalfCheetah-medium and Hopper-medium across 4 seeds. Within each task, the backbone, BC coefficient, and training settings were kept fixed. The results are shown below.\

& \text{HalfCheetah-medium} & & \text{Hopper-medium} & \\\\ \hline \text{Variant} & \text{Last100k} & \text{AUC} & \text{Last100k} & \text{AUC} \\\\ \hline \text{Full PaRE} & 66.59 \pm 1.35 & 63.68 \pm 0.82 & 102.38 \pm 3.54 & 91.18 \pm 2.59 \\\\ \text{No PaRE} & 65.58 \pm 0.98 & 61.84 \pm 0.59 & 97.78 \pm 8.72 & 77.50 \pm 14.52 \\\\ \text{Policy-only} & 65.88 \pm 1.14 & 62.96 \pm 0.83 & 97.20 \pm 10.49 & 85.31 \pm 6.76 \\\\ \text{Mean aggregation} & 65.73 \pm 0.17 & 63.10 \pm 0.53 & 100.10 \pm 1.32 & 88.68 \pm 0.97 \\\\ \end{array}$$ We report mean ± sample standard deviation across training seeds. Last100k is the mean evaluation score over the final 100k updates. AUC is the area under the learning curve divided by the training interval length, measuring average performance over training.\ Full PaRE is best on both tasks and both metrics. The gain is sizeable on Hopper-medium, where training is unstable. Its seed spread there is much smaller, which shows that PaRE improves training stability, not just the final score. \ **Answer to Q2:** For the submitted experiments, BC coefficients for each dataset follow existing work, and we did not run a systematic grid search. The PaRE hyperparameters in the submission are shared across all tasks and were not tuned per task. During the rebuttal, we explored the PaRE hyperparameters using the evaluation return on HalfCheetah-medium, and the ablations above use λ = 0.03, T = 1, w = 0.1. The other PaRE settings are unchanged.\ **Answer to Q3:** Our analysis shows how predicted relative value sets the regularization pressure. We do not claim that PaRE identifies estimation error. We ran the rollout diagnostic on 100 held-out states in Hopper-medium. Within the same state and critic, candidates under larger pressure have on average a larger relative estimation error, but also a larger rollout reference advantage. Among candidates with a positive predicted gap, 43.09% have a non-positive rollout gap, and they receive nearly the same share of the correction pressure (42.97%). Pressure is therefore associated with estimation error, but it does not cleanly separate overestimation from genuine advantage. This is expected, since PaRE is a relative-value regularizer, not an error detector.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.