- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Power Mirror aims to improve a policy while keeping its deviation from the behavior policy under control. The authors develop two complementary variants: PMVI, which always uses the behavior policy as a fixed reference, and PMPI, which updates the reference policy at each iteration. The paper conducts a detailed theoretical analysis of the proposed framework, complemented by experiments that evaluate its performance under different data-coverage conditions.
Critical strengths
The paper presents an interesting way to connect value backups with policy improvement through the gradient of a power-mean operator. The explicit probability-ratio bounds are a useful property, particularly for offline RL where distribution shift is a major concern. The distinction between fixed and moving reference policies is also well motivated. Another strength is that the paper discusses the assumptions and limitations of its theoretical results.
Critical weaknesses
-
The fixed-shift assumption in PMVI does not directly carry over to its practical implementation. The monotonicity and contraction guarantees for PMVI rely on a fixed shift parameter . However, the implementation uses , where the shift is recomputed from the current or learned critic span. In fact, when depends on the Q-values, the resulting backup is not necessarily monotone. Also, the analysis is formulated around a well-defined behavior policy assumption, whereas real offline datasets may contain trajectories collected by different policies. More importantly, PMVI uses a fixed behavior policy as its reference, but the accuracy of policy is never been involved.
-
The experimental comparisons do not adequately isolate the benefits of the proposed Power-Mean operator. The KL-fixed and KL-moving baselines use a fixed temperature of 1, whereas PMVI employs a shift proportional to the learned critic span and evaluates multiple values of . Consequently, these methods are not compared under matched policy-update strengths or likelihood-ratio budgets. Similarly, IQL uses fixed expectile and advantage-temperature settings across different dataset sizes and compositions, without further sensitivity analysis. Moreover, the implementation combines the proposed operator with a bounded critic and a specialized actor, making it difficult to separate their contributions. The evaluation also lacks comparisons with some more recent behavior-regularized offline RL methods.
- Li, Jiachen, Edwin Zhang, Ming Yin, Qinxun Bai, Yu-Xiang Wang, and William Yang Wang. "Offline reinforcement learning with closed-form policy improvement operators." In International Conference on Machine Learning, pp. 20485-20528. PMLR, 2023.
- Ma, Yi, Jianye Hao, Xiaohan Hu, Yan Zheng, and Chenjun Xiao. "Iteratively refined behavior regularization for offline reinforcement learning." Advances in Neural Information Processing Systems 37 (2024): 56215-56243.
Questions for authors
- The Power-Mean backup lies between the behavior-policy expectation and the supported maximum, a structure also encountered in expectile-based value estimation. Could the authors further clarify which theoretical advantages are specific to the Power-Mean construction, particularly compared with expectile-based or KL-regularized operators?
- Although the paper discusses the additional assumptions required to transfer ratio guarantees to the true logger, could the authors provide empirical sensitivity analysis with respect to behavior-model estimation error?
- The paper presents PMVI and PMPI as complementary approaches, but it is unclear how one should choose between them in practice. Is there a principled criterion for selecting PMVI or PMPI based on some factors?
- In Section 4 and Appendix K.2, PMPI is described as first evaluating the current policy to obtain , followed by a Power-Mirror policy improvement step. However, in
Code/src/d4rl_bench/train.py(lines 503–538), thepm_movingbranch uses a Power-Mean Bellman backup as the TD target rather than the expected Q-value under the current policy. What motivated this implementation choice? - Also, the theoretical PMPI update provides a bounded probability ratio between consecutive policies. However, the continuous implementation uses a free
TanhGaussianActor(Code/src/d4rl_bench/train.py, lines 52 and 366–369), trained through weighted likelihood fitting (lines 632–635), not the ratio-preserving actor used for PMVI.
6 comments
Official Comment by Authors (2/4)
W2. How do the new experiments separate the operator from calibration and fitting? (continued)
We also added IBR [2] using the authors' implementation, the common 50,000-update budget, and offline validation (Appendix M.5, Fig. 12). Together with KL, Pearson chi-square, and IQL/AWAC, these comparisons cover regularized updates, policy extraction, and iterative behavior regularization. CFPI [1], discussed as related work in Appendix A, shares the goal of behavior-constrained improvement and constructs actions through a local critic approximation and a Gaussian-mixture behavior model; our operator comparisons study how to reweight actions under fixed or moving references.
Q1. What is specific to Power Mirror compared with expectiles and KL?
Thank you for asking us to clarify this distinction. The mean–maximum sandwich and bounded policy weights are shared with other constructions. KL admits ratio bounds through span/temperature calibration, and an upper expectile has an implicit actor with bounded weights (Appendices A and I.3). Our contribution is a joint characterization of the backup, its induced policy, the improvement decomposition, and the bound on each action's probability ratio within the same centered power-mean family.
The construction starts from the backup: subtract the reference mean of the action values, add a positive shift, and take a power mean. Differentiating this backup gives a normalized policy, including the additive correction caused by centering (§3). The same power and shift then determine both its sharp ratio bounds and its exact improvement. In particular, the improvement splits into two nonnegative terms: the increase of the backup over the reference value, and the further gain of the induced policy over that backup. This ties the value used in a Bellman update directly to the value of the policy extracted from it.
This characterization also gives a quantitative tradeoff within the family. Write , where is the action-value span and the shift. The sharp upper ratio bound is , so a prescribed upper budget gives
The same power appears in the coverage bound through , where is the reference probability of actions within of the supported maximum. At fixed shift, increasing the power tightens that coverage bound while widening the ratio interval. Thus the analysis explicitly connects approximation to the maximum with permitted probability changes; Theorem 13 further connects those changes to critic-error sensitivity (§3 and Appendix G). This is the quantitative tradeoff we establish, rather than a claim of uniformly better bounds than KL or expectiles.
The quadratic case makes the connection to existing regularized improvement precise: at , PM is exactly a Pearson chi-square constrained update at its induced radius (Proposition 1 and Appendix I.1). We present this as an equivalence, not a separate new optimizer. The contribution extends across the power family and into repeated Bellman updates: fixed-reference PMVI has a contraction and coverage-dependent guarantee under its stated assumptions, while moving-reference PMPI accumulates improvements with consecutive-policy ratio bounds (Theorems 2–3). These results explain how the same construction supports two distinct policy-improvement procedures. Section 2 discusses the closest regularized and policy-improvement methods; Appendix A gives the broader comparison, including Li et al. [1].
References
[1] Li et al. (2023). Offline Reinforcement Learning with Closed-Form Policy Improvement Operators.
[2] Ma et al. (2024). Iteratively Refined Behavior Regularization for Offline Reinforcement Learning.
Official Comment by Authors (3/4)
Q2. How does behavior-model estimation affect the method?
The reference used by the update is the policy relative to which its ratios are controlled. In the new Wet Chicken plots this is the fitted categorical behavior policy. In Hopper and MetaDrive it is a fitted conditional Gaussian mixture, frozen and shared across methods. The curve labeled Fitted behavior evaluates that same policy (shared model settings at the start of Appendix L; reference definitions in Appendix M.1).
We added a direct sensitivity experiment on Wet Chicken, where the true logger is known. We vary the data used to fit the behavior model over 100, 500, 2,000, 10,000, and 50,000 observations, while keeping the policy dataset and critic fixed. Figure 11 shows return, mean L1 behavior-model error, and the maximum true-logger ratio computed within each dataset and then averaged over 100 datasets.
The results show lower behavior-model error and improved return with larger fitting budgets. Appendix M.4 gives the results for three policy-data sizes and all three powers.
Lower mean behavior-model error does not, however, imply recovery of the nominal ratio bound relative to the true logger. At the largest fitting budget of 50,000 observations, Figure 11 reports mean per-dataset maximum true-logger ratios of 5.64, 5.37, and 5.91 for the three policy-data sizes at , compared with the fitted-reference upper bound of 1.5 (Appendix M.4). The structural guarantee is relative to the fitted reference; transfer to the true logger requires a uniform approximation condition of the kind stated below.
The paper also states how a model-relative bound transfers to the true logger. On common support, if the uniform log-density error is at most , then
\quad\Longrightarrow\quad Le^{-\varepsilon_\beta}\le\frac{\pi}{w}\le Ue^{\varepsilon_\beta}.$$ Here $\widehat w$ is the fitted reference and $w$ is the true logger. The error condition and transfer are given in Eq. (233). For data from multiple collectors, we fit one reference to the pooled data and use it as an executable policy. The paper distinguishes this pooled state-conditional policy from the trajectory mixture obtained when one collector is chosen per episode (Appendix M.1, Eqs. (234)–(235)). This identifies which reference is used for policy improvement and which logging likelihood is required by the deployment test. ### Q3. How should one choose between PMVI and PMPI? The practical criterion is which reference the application needs to keep controlled. **Choose PMVI** when every policy must remain within a specified ratio interval around the original behavior reference. **Choose PMPI** when successive improvements may move farther from that reference and current-policy evaluation is available. For PMPI, each ratio bound applies to the preceding policy. These bounds multiply across iterations, giving the cumulative departure from the original reference. An application with a prescribed final budget can use that product or the behavior-box safeguard in Appendix F. The population improvement result with exact evaluation is stated in Theorem 3. **KL-adaptive-moving** is the KL comparison that updates both its reference and its span-calibrated temperature. Wet Chicken illustrates these choices: PMVI keeps the original reference constraint, while PMPI achieves higher return through successive improvements (Figs. 4 and 6). Appendix M.2 discusses the results. Within PMVI, fixed and adaptive shifts select how to handle the critic's scale: the fixed variant uses one prescribed scale, and the adaptive variant follows the current span bound as explained in W1.
Official Comment by Authors (1/4)
Thank you for recognizing the value of the probability-ratio bounds and the distinction between fixed and moving references. We have clarified the role of the adaptive shift, added controlled comparisons and behavior-model sensitivity experiments, and corrected the continuous PMPI implementation. We address each concern below, with references to the revised paper and code. Key revisions are highlighted in blue in the revised paper.
W1. Why use an adaptive shift in the experiments?
Theorem 2 states the monotonicity and contraction result for a fixed reference, power, and shift. We now distinguish PMVI-fixed, which keeps the shift fixed during training, from PMVI-adaptive, which recomputes it from the current critic-span bound. Both keep the behavior reference fixed.
The fixed-shift contraction guarantee does not automatically extend to the adaptive operator: making the shift depend on the critic can invalidate monotonicity. We therefore present PMVI-adaptive as a practical variant supported by the reported experiments, rather than by Theorem 2.
The adaptive variant is motivated by changes in the scale of the learned action values. As training progresses, the differences between action values can become smaller. A shift chosen from a broad initial range then produces increasingly small policy changes. For example, the quadratic update in Eq. (179) gives
=\frac{|Q(a)-\mathbb E_wQ|}{\sqrt{b_0^2+\operatorname{Var}_w(Q)}} \le\frac{\operatorname{span}(Q)}{b_0},$$ where $w$ is the reference and $b_0$ is the fixed shift. Setting $b_k(s)=2cD_k(s)$ keeps the shift proportional to the current span bound. This lets the remaining value differences guide improvement at the current scale, while the power $p$ and multiplier $c$ specify the probability-ratio budget. Appendix M.1 explains this motivation and the calibration in Eq. (231). Wet Chicken gives a direct implementation of this idea. We solve policy evaluation in the empirical MDP and compute the span by taking the maximum minus the minimum over all reference-supported actions. Thus the shift uses the span of the actual empirical critic used for improvement (Eq. (236)). In continuous control, the bounded critic supplies the learned span bound $D_{\phi_k}(s)$. The new comparisons show both fixed and adaptive variants at $c=1$ on all three benchmarks (Figs. 5, 13, and 18). Tuning $c$ is shown separately (Figs. 8, 17, and 19). Behavior-model accuracy and pooled data are addressed in Q2 below. ### W2. How do the new experiments separate the operator from calibration and fitting? We now compare PM and KL using the same declared upper probability-ratio bound. For a span bound $D$, the calibration is $$U_p=(3/2)^{p-1},\qquad b=2cD,\qquad T_{\mathrm{KL}}=\frac{cD}{\log U_p}.$$ Here $b$ is the PM shift and $T_{\mathrm{KL}}$ is the KL temperature. At $c=1$, both updates have upper ratio bound $U_p$; their respective lower bounds are given in Appendix M.1. A KL curve's $p$ identifies the matching PM budget. **KL-fixed** keeps its temperature fixed, while **KL-adaptive** recomputes it from the current span bound; both use a fixed behavior reference. The primary comparisons use $p=2,c=1$ for all four PMVI/KL variants. The power sweeps report $p\in\{2,4,6\}$ separately; power is not selected using test return. Separate tuning ablations select $c$ from $\{1,2,4,8\}$ using offline validation (Figs. 8, 17, and 19). IQL also tunes its expectile and advantage temperature, and AWAC tunes its advantage temperature. The grids and selection procedure are specified in Appendix M.1. The new continuous PM/KL comparisons use the same fitted behavior reference, bounded critic architecture, training budget, and 20-action proposal pool. The policy is obtained directly from the computed weights, so there is no actor-fitting step in these comparisons. We also compare a single PM or KL update using **the same frozen critic and reference** (Figs. 7 and 14–15). These comparisons show improvements with PM under a shared critic; the results are discussed in Appendices M.2 and M.5. The Hopper comparison uses one critic per data size and ten rollout seeds, as stated in its caption.