- Rating
- 3
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Paired Relative-Value Regularization (PaRE) for offline reinforcement learning. At each dataset state, PaRE compares the critic values of the current policy action and several nearby perturbed actions with the value of the logged action. It aggregates these differences using log-mean-exp and applies a softplus penalty, yielding a regularization signal whose total strength depends on the aggregated gap and whose allocation favors higher-valued candidates. The method is implemented in a TD3+BC-based framework adapted from ReBRAC, with two LayerNorm critics and a behavior-regularized deterministic actor. The theoretical analysis provides a conditional bound on candidate overestimation, a variational interpretation of the regularizer, and a small-radius comparison with mean aggregation that identifies an additional action-gradient penalty. Experiments cover 24 D4RL locomotion and dexterous manipulation datasets. PaRE obtains the second-highest reported MuJoCo average, and the highest reported Adroit average.
Critical strengths
-
The policy-local candidate construction is well motivated for deterministic actor–critic learning. Because the actor's update depends on the critic's action derivative at its current action, regularizing nearby values addresses a part of the critic that directly affects policy improvement. This gives the design in Section 3.1 a clear practical rationale.
-
PaRE is a small addition to an existing actor–critic implementation. It requires additional critic evaluations but no learned behavior model, distance estimator, or ensemble beyond the two existing critics. This simplicity makes the idea relatively easy to test in established offline RL codebases, and the reported runtime offers preliminary evidence that its computational overhead is manageable.
Critical weaknesses
-
A comparison using the exact same backbone with PaRE disabled would make its contribution easier to assess. The implementation includes three-layer networks, critic LayerNorm, a twin-minimum actor objective, and benchmark-dependent optimization settings, while the baseline scores come from earlier papers. Keeping these choices and the behavior-cloning coefficients fixed while setting could help clarify how much of the observed improvement is associated with the proposed regularizer.
-
The ablations provide an initial view of the components, although I would welcome further evidence on the benefit of local neighbors and selective aggregation. Figure 1 uses one seed on HalfCheetah-Medium with the original TD3+BC backbone, and the curves with and without neighbors appear fairly similar. Repeating this comparison across seeds and tasks, together with mean aggregation over the same candidates, would help assess the practical relevance of the mechanism discussed in Proposition 3.
-
Some additional detail on parameter selection would make the robustness of the shared configuration easier to assess. Table 4 uses behavior-cloning coefficients ranging from 0.001 to 0.10, and Figure 2 suggests that this choice can noticeably affect performance. Clarifying their selection procedure and tuning budget, along with modest sensitivity sweeps over the PaRE-specific parameters, would help readers judge how readily the settings might transfer to new datasets.
Questions for authors
Please see the weaknesses.
Thank you for your careful review and for acknowledging the motivation for policy-local regularization and the simplicity of PaRE. We add ablation results below and explain how the hyperparameters were chosen.
& \text{HalfCheetah-medium} & & \text{Hopper-medium} & \\\\ \hline \text{Variant} & \text{Last100k} & \text{AUC} & \text{Last100k} & \text{AUC} \\\\ \hline \text{Full PaRE} & 66.59 \pm 1.35 & 63.68 \pm 0.82 & 102.38 \pm 3.54 & 91.18 \pm 2.59 \\\\ \text{No PaRE} & 65.58 \pm 0.98 & 61.84 \pm 0.59 & 97.78 \pm 8.72 & 77.50 \pm 14.52 \\\\ \text{Policy-only} & 65.88 \pm 1.14 & 62.96 \pm 0.83 & 97.20 \pm 10.49 & 85.31 \pm 6.76 \\\\ \text{Mean aggregation} & 65.73 \pm 0.17 & 63.10 \pm 0.53 & 100.10 \pm 1.32 & 88.68 \pm 0.97 \\\\ \end{array}$$ Full PaRE is best on both tasks and both metrics. All four variants share the same backbone, BC coefficient and optimization settings. On Hopper-medium, where training is unstable, PaRE also gives a much smaller spread across seeds, so training is more reliable.\ **Answer to Q3:** In the submission, the BC coefficients for each dataset follow existing work, and we did not run a systematic grid search. The PaRE parameters are shared across tasks and were not tuned per task. In all matched ablations, the BC coefficient is kept the same within a task.During the rebuttal we chose the PaRE parameters by their performance on HalfCheetah-medium, then kept them fixed for every task. The ablations above use λ = 0.03, T = 1 and w = 0.1.Answer to Q1&Q2: To address both points, we used the main implementation and compared Full PaRE, No PaRE, Policy-only and Mean aggregation on HalfCheetah-medium and Hopper-medium across 4 training seeds. Within each task, the network architecture, including critic LayerNorm and the twin-minimum actor objective, the BC coefficient and the optimization settings are identical. No PaRE turns off the regularizer. Policy-only removes the local neighbors. Mean aggregation replaces the log-mean-exp with the arithmetic mean over the same candidates.The results are below.