acceptodds
Under review as a conference paper at ICLR 2027

Power Mirror: Offline Policy Improvement with Bounded Reweighting

Abstract

Offline reinforcement learning (RL) must balance policy improvement with reliable use of logged data. We introduce *Power Mirror*, a policy-improvement framework derived from shifted power-mean value backups. With a suitable shift, the induced updates satisfy explicit two-sided bounds on actionwise probability ratios, together with population improvement guarantees and sensitivity bounds for critic error. We develop two complementary algorithms: fixed-anchor value iteration maintains a permanent bound relative to the behavior policy, while moving-anchor policy iteration improves monotonically and converges in value to the support-constrained optimum under exact evaluation and persistent updates. We also construct a ratio-preserving actor for continuous actions and a held-out certification rule that deploys a learned policy only when improvement is certified, otherwise retaining the behavior policy. Together, these results connect value estimation, policy optimization, and deployment through explicit control of policy reweighting. Experiments support our theoretical findings, with Power Mirror achieving strong policy improvement while remaining conservative under limited data coverage

Primary area:
reinforcement learning
Keywords:
offline reinforcement learning, safe policy improvement, behavior regularization, distribution shift, power mean

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5DEgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Power Mirror aims to improve a policy while keeping its deviation from the behavior policy under control. The authors develop two complementary variants: PMVI, which always uses the behavior policy as a fixed reference, and PMPI, which updates the reference policy at each iteration. The paper conducts a detailed theoretical analysis of the proposed framework, complemented by experiments that evaluate its performance under different data-coverage conditions.

Critical strengths

The paper presents an interesting way to connect value backups with policy improvement through the gradient of a power-mean operator. The explicit probability-ratio bounds are a useful property, particularly for offline RL where distribution shift is a major concern. The distinction between fixed and moving reference policies is also well motivated. Another strength is that the paper discusses the assumptions and limitations of its theoretical results.

Critical weaknesses
  1. The fixed-shift assumption in PMVI does not directly carry over to its practical implementation. The monotonicity and contraction guarantees for PMVI rely on a fixed shift parameter bb. However, the implementation uses b(s)=2Dϕ(s)b(s)=2D_\phi(s), where the shift is recomputed from the current or learned critic span. In fact, when bb depends on the Q-values, the resulting backup is not necessarily monotone. Also, the analysis is formulated around a well-defined behavior policy assumption, whereas real offline datasets may contain trajectories collected by different policies. More importantly, PMVI uses a fixed behavior policy as its reference, but the accuracy of policy is never been involved.

  2. The experimental comparisons do not adequately isolate the benefits of the proposed Power-Mean operator. The KL-fixed and KL-moving baselines use a fixed temperature of 1, whereas PMVI employs a shift proportional to the learned critic span and evaluates multiple values of pp. Consequently, these methods are not compared under matched policy-update strengths or likelihood-ratio budgets. Similarly, IQL uses fixed expectile and advantage-temperature settings across different dataset sizes and compositions, without further sensitivity analysis. Moreover, the implementation combines the proposed operator with a bounded critic and a specialized actor, making it difficult to separate their contributions. The evaluation also lacks comparisons with some more recent behavior-regularized offline RL methods.

  • Li, Jiachen, Edwin Zhang, Ming Yin, Qinxun Bai, Yu-Xiang Wang, and William Yang Wang. "Offline reinforcement learning with closed-form policy improvement operators." In International Conference on Machine Learning, pp. 20485-20528. PMLR, 2023.
  • Ma, Yi, Jianye Hao, Xiaohan Hu, Yan Zheng, and Chenjun Xiao. "Iteratively refined behavior regularization for offline reinforcement learning." Advances in Neural Information Processing Systems 37 (2024): 56215-56243.
Questions for authors
  1. The Power-Mean backup lies between the behavior-policy expectation and the supported maximum, a structure also encountered in expectile-based value estimation. Could the authors further clarify which theoretical advantages are specific to the Power-Mean construction, particularly compared with expectile-based or KL-regularized operators?
  2. Although the paper discusses the additional assumptions required to transfer ratio guarantees to the true logger, could the authors provide empirical sensitivity analysis with respect to behavior-model estimation error?
  3. The paper presents PMVI and PMPI as complementary approaches, but it is unclear how one should choose between them in practice. Is there a principled criterion for selecting PMVI or PMPI based on some factors?
  4. In Section 4 and Appendix K.2, PMPI is described as first evaluating the current policy to obtain QπkQ^{\pi_k}, followed by a Power-Mirror policy improvement step. However, in Code/src/d4rl_bench/train.py (lines 503–538), the pm_moving branch uses a Power-Mean Bellman backup as the TD target rather than the expected Q-value under the current policy. What motivated this implementation choice?
  5. Also, the theoretical PMPI update provides a bounded probability ratio between consecutive policies. However, the continuous implementation uses a free TanhGaussianActor (Code/src/d4rl_bench/train.py, lines 52 and 366–369), trained through weighted likelihood fitting (lines 632–635), not the ratio-preserving actor used for PMVI.
6 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/4)

Thank you for recognizing the value of the probability-ratio bounds and the distinction between fixed and moving references. We have clarified the role of the adaptive shift, added controlled comparisons and behavior-model sensitivity experiments, and corrected the continuous PMPI implementation. We address each concern below, with references to the revised paper and code. Key revisions are highlighted in blue in the revised paper.

W1. Why use an adaptive shift in the experiments?

Theorem 2 states the monotonicity and contraction result for a fixed reference, power, and shift. We now distinguish PMVI-fixed, which keeps the shift fixed during training, from PMVI-adaptive, which recomputes it from the current critic-span bound. Both keep the behavior reference fixed.

The fixed-shift contraction guarantee does not automatically extend to the adaptive operator: making the shift depend on the critic can invalidate monotonicity. We therefore present PMVI-adaptive as a practical variant supported by the reported experiments, rather than by Theorem 2.

The adaptive variant is motivated by changes in the scale of the learned action values. As training progresses, the differences between action values can become smaller. A shift chosen from a broad initial range then produces increasingly small policy changes. For example, the quadratic update in Eq. (179) gives

=\frac{|Q(a)-\mathbb E_wQ|}{\sqrt{b_0^2+\operatorname{Var}_w(Q)}} \le\frac{\operatorname{span}(Q)}{b_0},$$ where $w$ is the reference and $b_0$ is the fixed shift. Setting $b_k(s)=2cD_k(s)$ keeps the shift proportional to the current span bound. This lets the remaining value differences guide improvement at the current scale, while the power $p$ and multiplier $c$ specify the probability-ratio budget. Appendix M.1 explains this motivation and the calibration in Eq. (231). Wet Chicken gives a direct implementation of this idea. We solve policy evaluation in the empirical MDP and compute the span by taking the maximum minus the minimum over all reference-supported actions. Thus the shift uses the span of the actual empirical critic used for improvement (Eq. (236)). In continuous control, the bounded critic supplies the learned span bound $D_{\phi_k}(s)$. The new comparisons show both fixed and adaptive variants at $c=1$ on all three benchmarks (Figs. 5, 13, and 18). Tuning $c$ is shown separately (Figs. 8, 17, and 19). Behavior-model accuracy and pooled data are addressed in Q2 below. ### W2. How do the new experiments separate the operator from calibration and fitting? We now compare PM and KL using the same declared upper probability-ratio bound. For a span bound $D$, the calibration is $$U_p=(3/2)^{p-1},\qquad b=2cD,\qquad T_{\mathrm{KL}}=\frac{cD}{\log U_p}.$$ Here $b$ is the PM shift and $T_{\mathrm{KL}}$ is the KL temperature. At $c=1$, both updates have upper ratio bound $U_p$; their respective lower bounds are given in Appendix M.1. A KL curve's $p$ identifies the matching PM budget. **KL-fixed** keeps its temperature fixed, while **KL-adaptive** recomputes it from the current span bound; both use a fixed behavior reference. The primary comparisons use $p=2,c=1$ for all four PMVI/KL variants. The power sweeps report $p\in\{2,4,6\}$ separately; power is not selected using test return. Separate tuning ablations select $c$ from $\{1,2,4,8\}$ using offline validation (Figs. 8, 17, and 19). IQL also tunes its expectile and advantage temperature, and AWAC tunes its advantage temperature. The grids and selection procedure are specified in Appendix M.1. The new continuous PM/KL comparisons use the same fitted behavior reference, bounded critic architecture, training budget, and 20-action proposal pool. The policy is obtained directly from the computed weights, so there is no actor-fitting step in these comparisons. We also compare a single PM or KL update using **the same frozen critic and reference** (Figs. 7 and 14–15). These comparisons show improvements with PM under a shared critic; the results are discussed in Appendices M.2 and M.5. The Hopper comparison uses one critic per data size and ten rollout seeds, as stated in its caption.
AuthorsOpenReview ↗

Official Comment by Authors (2/4)

W2. How do the new experiments separate the operator from calibration and fitting? (continued)

We also added IBR [2] using the authors' implementation, the common 50,000-update budget, and offline validation (Appendix M.5, Fig. 12). Together with KL, Pearson chi-square, and IQL/AWAC, these comparisons cover regularized updates, policy extraction, and iterative behavior regularization. CFPI [1], discussed as related work in Appendix A, shares the goal of behavior-constrained improvement and constructs actions through a local critic approximation and a Gaussian-mixture behavior model; our operator comparisons study how to reweight actions under fixed or moving references.

Q1. What is specific to Power Mirror compared with expectiles and KL?

Thank you for asking us to clarify this distinction. The mean–maximum sandwich and bounded policy weights are shared with other constructions. KL admits ratio bounds through span/temperature calibration, and an upper expectile has an implicit actor with bounded weights (Appendices A and I.3). Our contribution is a joint characterization of the backup, its induced policy, the improvement decomposition, and the bound on each action's probability ratio within the same centered power-mean family.

The construction starts from the backup: subtract the reference mean of the action values, add a positive shift, and take a power mean. Differentiating this backup gives a normalized policy, including the additive correction caused by centering (§3). The same power and shift then determine both its sharp ratio bounds and its exact improvement. In particular, the improvement splits into two nonnegative terms: the increase of the backup over the reference value, and the further gain of the induced policy over that backup. This ties the value used in a Bellman update directly to the value of the policy extracted from it.

This characterization also gives a quantitative tradeoff within the family. Write η=D/b\eta=D/b, where DD is the action-value span and bb the shift. The sharp upper ratio bound is (1+η)p−1(1+\eta)^{p-1}, so a prescribed upper budget ρ>1\rho>1 gives

p≤1+log⁡ρlog⁡(1+η)(η>0).p\le 1+\frac{\log\rho}{\log(1+\eta)}\qquad(\eta>0).

The same power appears in the coverage bound through Wε1/pW_\varepsilon^{1/p}, where WεW_\varepsilon is the reference probability of actions within ε\varepsilon of the supported maximum. At fixed shift, increasing the power tightens that coverage bound while widening the ratio interval. Thus the analysis explicitly connects approximation to the maximum with permitted probability changes; Theorem 13 further connects those changes to critic-error sensitivity (§3 and Appendix G). This is the quantitative tradeoff we establish, rather than a claim of uniformly better bounds than KL or expectiles.

The quadratic case makes the connection to existing regularized improvement precise: at p=2p=2, PM is exactly a Pearson chi-square constrained update at its induced radius (Proposition 1 and Appendix I.1). We present this as an equivalence, not a separate new optimizer. The contribution extends across the power family and into repeated Bellman updates: fixed-reference PMVI has a contraction and coverage-dependent guarantee under its stated assumptions, while moving-reference PMPI accumulates improvements with consecutive-policy ratio bounds (Theorems 2–3). These results explain how the same construction supports two distinct policy-improvement procedures. Section 2 discusses the closest regularized and policy-improvement methods; Appendix A gives the broader comparison, including Li et al. [1].

References

[1] Li et al. (2023). Offline Reinforcement Learning with Closed-Form Policy Improvement Operators.

[2] Ma et al. (2024). Iteratively Refined Behavior Regularization for Offline Reinforcement Learning.

AuthorsOpenReview ↗

Official Comment by Authors (3/4)

Q2. How does behavior-model estimation affect the method?

The reference used by the update is the policy relative to which its ratios are controlled. In the new Wet Chicken plots this is the fitted categorical behavior policy. In Hopper and MetaDrive it is a fitted conditional Gaussian mixture, frozen and shared across methods. The curve labeled Fitted behavior evaluates that same policy (shared model settings at the start of Appendix L; reference definitions in Appendix M.1).

We added a direct sensitivity experiment on Wet Chicken, where the true logger is known. We vary the data used to fit the behavior model over 100, 500, 2,000, 10,000, and 50,000 observations, while keeping the policy dataset and critic fixed. Figure 11 shows return, mean L1 behavior-model error, and the maximum true-logger ratio computed within each dataset and then averaged over 100 datasets.

The results show lower behavior-model error and improved return with larger fitting budgets. Appendix M.4 gives the results for three policy-data sizes and all three powers.

Lower mean behavior-model error does not, however, imply recovery of the nominal ratio bound relative to the true logger. At the largest fitting budget of 50,000 observations, Figure 11 reports mean per-dataset maximum true-logger ratios of 5.64, 5.37, and 5.91 for the three policy-data sizes at p=2p=2, compared with the fitted-reference upper bound of 1.5 (Appendix M.4). The structural guarantee is relative to the fitted reference; transfer to the true logger requires a uniform approximation condition of the kind stated below.

The paper also states how a model-relative bound transfers to the true logger. On common support, if the uniform log-density error is at most εβ\varepsilon_\beta, then

\quad\Longrightarrow\quad Le^{-\varepsilon_\beta}\le\frac{\pi}{w}\le Ue^{\varepsilon_\beta}.$$ Here $\widehat w$ is the fitted reference and $w$ is the true logger. The error condition and transfer are given in Eq. (233). For data from multiple collectors, we fit one reference to the pooled data and use it as an executable policy. The paper distinguishes this pooled state-conditional policy from the trajectory mixture obtained when one collector is chosen per episode (Appendix M.1, Eqs. (234)–(235)). This identifies which reference is used for policy improvement and which logging likelihood is required by the deployment test. ### Q3. How should one choose between PMVI and PMPI? The practical criterion is which reference the application needs to keep controlled. **Choose PMVI** when every policy must remain within a specified ratio interval around the original behavior reference. **Choose PMPI** when successive improvements may move farther from that reference and current-policy evaluation is available. For PMPI, each ratio bound applies to the preceding policy. These bounds multiply across iterations, giving the cumulative departure from the original reference. An application with a prescribed final budget can use that product or the behavior-box safeguard in Appendix F. The population improvement result with exact evaluation is stated in Theorem 3. **KL-adaptive-moving** is the KL comparison that updates both its reference and its span-calibrated temperature. Wet Chicken illustrates these choices: PMVI keeps the original reference constraint, while PMPI achieves higher return through successive improvements (Figs. 4 and 6). Appendix M.2 discusses the results. Within PMVI, fixed and adaptive shifts select how to handle the critic's scale: the fixed variant uses one prescribed scale, and the adaptive variant follows the current span bound as explained in W1.
Reviewer_xb2fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposed an offline reinforcement learning algorithm. The main idea to introduce a shifted power-mean value backup which introduces two parameters: shift and power. With a good value function, they show the policy update with this backup achieves guaranteed improvement while satisfy explicit bounds on the action-wise probability ratios between the learner policy and behavior policy. Based on this updating operator, they proposed two algorithms: value iteration and policy iteration, distinguished by whether the anchor policy is moving. Besides, they also provide high-probability results and continuous action space results. Finally, they conducted experiments to show the effectiveness of their proposed algorithm.

Critical strengths
  1. The main claims in Section 2, 3 and 4 look correct to me. And the proposed algorithm achieves the authors’ stated goal in the theoretical perspective which is to seamlessly incorporate the behavior constrains with policy improvement. The resulting property: guaranteed improvement with bounded ratios of all actions is desirable and useful in many scenarios.
Critical weaknesses
  1. I think the main criticism is the presentation which is below the bar of publication in my opinion. This paper is extremely hard to read and follow despite of paying decent efforts by readers. It is understandable given the fact that the content is very theoretical, however, there are lots of places, that a concept, or a claim is introduced suddenly and only briefly without concrete and intuitive explanation or supporting references. I feel the presentation needs substantial improvement to make the ideas easier to digest.

  2. Some parts of the algorithm is unclear to me under the finite data setting. For example, in line 199, Q function is estimated with one step estimator. The algorithm considers the supported action set. In theory, it is fine since one can always sample an action that the behavior policy assigns nonzero probability, however, in finite data setting, how is it guaranteed that r(s,a) is accessible even if a is a supported action? In other words, if talking about the sample complexity, what will the results be like?

Questions for authors
  1. What will the results of the proposed algorithm look like in the finite data setting, i.e., the sample complexity?
4 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

Thank you for recognizing the value of combining policy improvement with bounded actionwise probability ratios. We have revised the presentation to explain the main constructions before stating their guarantees, and added a finite-data tabular PMVI result with an explicit sample requirement. We address the missing-action issue and summarize that result below. Key revisions are highlighted in blue in the revised paper.

1. Presentation

Thank you for this feedback. We have substantially revised the presentation to better motivate the central idea, provide intuition behind the theoretical results, and clarify their connections to existing literature. We have also revisited the manuscript to identify concepts, terminology, and claims that were introduced abruptly, adding explanations, context, and supporting references where needed.

The Introduction now uses a concrete rare-action example to motivate actionwise ratio control. Section 3 explains the intuition behind each component of Power Mirror: centering captures relative action values, shifting ensures positivity, and the power mean emphasizes high-value actions. We connect these design choices to the induced policy, sharp ratio bounds, and critic-error sensitivity, highlighting the tradeoff between approaching the supported maximum and controlling policy deviation. Section 2 clarifies the relationship to the established frameworks of Belousov–Peters and Geist et al.

Sections 4–5 motivate PMVI and PMPI through their different choices of reference: maintaining a permanent behavior-relative constraint versus allowing successive improvements to accumulate. This distinction provides the intuition behind Theorems 2–3. Sections 6–7 then explain how ratio control supports deployment certification and continuous policy construction.

Throughout the revised manuscript, we provide motivation and intuitive explanations alongside the main theoretical results, while placing extended derivations and technical comparisons in the appendix. These revisions make the theoretical development more accessible and easier to follow.

2. How is the reward available for a supported action?

A positive logging probability does not ensure that an action appears in a finite dataset. The expression identified in the review defines the population Bellman quantity. Its empirical counterpart uses estimated rewards and transition frequencies:

\qquad \widehat Q_V(s,a)=\widehat r(s,a)+\gamma\widehat P(\cdot\mid s,a)^\top V.$$ Thus $r(s,a)$ is not assumed accessible merely because the behavior policy supports the action. Estimating it requires observations of that pair or an additional model assumption. In the reported tabular experiments, improvement at a visited state is restricted to observed actions; an entirely unvisited state uses the known logger as fallback. Zero-count backups are set to zero (Appendix L.1). This convention does not treat an unobserved reward as known. It also means that the actions available to the empirical update can be fewer than those supported by the true logger. The learning result below considers datasets that cover the original reference support. The total-count condition in part 4 below states when independent one-step data provide that coverage with high probability, rather than assuming it follows from positive logging probability alone. ### References [1] Belousov and Peters (2018). [*f-Divergence Constrained Policy Improvement*](https://arxiv.org/abs/1801.00056). [2] Geist, Scherrer, and Pietquin (2019). [*A Theory of Regularized Markov Decision Processes*](https://proceedings.mlr.press/v97/geist19a.html).
AuthorsOpenReview ↗

Official Comment by Authors (2/2)

3. Finite-data improvement and sample complexity

Proposition 9 in Appendix N gives an explicit sample requirement for retaining PMVI's improvement up to a statistical tolerance. Consider a finite MDP with SS states, discount 0≤γ<10\le\gamma<1, rewards in [0,R][0,R], and a known reference ww. Let M=∑s∣supp⁡w(⋅∣s)∣M=\sum_s|\operatorname{supp}w(\cdot\mid s)| be the number of reference-supported state–action pairs. Suppose each pair has at least nn independent reward–next-state observations from its correct conditional law, with counts selected independently of the observed outcomes.

Construct sample-mean rewards and transition frequencies on that full support. Use the same fixed powers 1<ps<∞1<p_s<\infty and shifts bs>R/(1−γ)b_s>R/(1-\gamma) in the empirical and population algorithms. Empirical Bellman updates and policy extraction are exact, and empirical PMVI starts at the empirical value of ww.

For confidence 1−δ1-\delta, the statistical tolerance is

\sqrt{\frac{S\log2+\log(4M/\delta)}{n}}.$$ Consequently, $B_n\le\varepsilon$ is ensured by the sufficient per-pair sample count $$n\ge\left\lceil \frac{2R^2}{\varepsilon^2(1-\gamma)^4} \left(S\log2+\log\frac{4M}{\delta}\right) \right\rceil.$$ A balanced dataset contains $Mn$ transitions. This is a sample requirement for improvement accuracy relative to the reference and the population PMVI target. With probability at least $1-\delta$, **all empirical PMVI iterates** satisfy, componentwise, $$V_{\mathcal M}^{\widehat\pi_k} \ge V_{\mathcal M}^w-B_n\mathbf1.$$ The estimated improvement is also accurate to the same tolerance: $$\left\| (V_{\mathcal M}^{\widehat\pi_k}-V_{\mathcal M}^w) -(V_{\widehat{\mathcal M}}^{\widehat\pi_k} -V_{\widehat{\mathcal M}}^w) \right\|_\infty\le B_n.$$ Here $\mathcal M$ and $\widehat{\mathcal M}$ are the true and empirical MDPs. An empirical improvement larger than $B_n$ therefore transfers to positive true improvement. The result also retains the population backup's improvement margin. Let $V_C$ be the population PMVI backup fixed point, $\nu$ the initial-state distribution, and $g_C=\nu^\top(V_C-V_{\mathcal M}^w)\ge0$. For the policy extracted at the empirical fixed point, $$J_\nu(\widehat\pi_C)-J_\nu(w)\ge g_C-B_n.$$ Thus a positive population margin remains positive once the statistical tolerance is smaller than that margin. This result concerns improvement over the reference; the coverage-dependent distance of the population PMVI target from the optimum is analyzed separately in Section 4. **Why the guarantee covers policies learned from the same data.** Uniform reward and transition concentration bounds the value difference between the two models by $B_n/2$ for every supported policy simultaneously. Transferring both the candidate and reference values then costs $B_n$. Empirical PMVI improves in the empirical model, so this uniform event applies to the resulting data-dependent policies and every iteration. The proof and sample requirement are in Appendix N, Proposition 9, Eq. (251). ### 4. When does a finite dataset cover the supported actions? Corollary 12 makes the coverage requirement quantitative. Suppose $N$ independent one-step observations draw $(s,a)\sim\mu$ and then the reward and next state from the MDP's conditional law. Define $\mu_{\min}=\min_{(s,a):w(a\mid s)>0}\mu(s,a)>0$. A sufficient total count to obtain at least $n$ observations per supported pair and retain the joint $1-\delta$ guarantee is $$N \ge \lceil \max(2n/\mu_{\min},\, (8/\mu_{\min})\log(2M/\delta)) \rceil.$$ This answers the missing-action question through an explicit dependence on sampling coverage: a supported pair with smaller sampling probability requires more data. The condition controls the probability that any supported pair is insufficiently observed; it does not presume access to its reward before it is sampled (Appendix N, Corollary 12, Eq. (253)). The per-pair result and this total-count corollary state their sampling assumptions explicitly. The corollary uses independent one-step observations; extending the count bound to dependent trajectory data requires accounting for that dependence. The fixed-shift PMVI guarantee is also distinguished from the adaptive fitted variants described in Appendix L. The critic-error result in Appendix G and the held-out deployment test in Section 6 provide the complementary guarantees for estimated critics and the final deployment decision.
AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer xb2f,

As the November 18 discussion deadline approaches, we would greatly appreciate your feedback on whether our improvements to the presentation and the additional finite-data analysis have addressed your concerns.

If any questions or concerns remain, we would be happy to clarify them before the discussion closes.

Thank you again for your time and consideration.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer xb2f,

We would appreciate your follow-up on whether our detailed responses and substantial revisions, particularly regarding presentation and finite-data guarantees, have addressed your concerns.

Thank you for your time and consideration.

Reviewer_f4i7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a new policy improvement framework to control policy mismatch in offline RL. Specifically, the authors introduce the shifted power-mean value backup, designed to bound the actionwise probability ratio of reference and learned policies. The proposed backup induces two regularized policy improvement procedures: fixed-anchor VI (PMVI) & moving-anchor PI (PMPI). They also discuss their extension to continuous action as well as the certification rule for policy improvement guarantee. Finally, the empirical performance of PMVI and PMPI is investigated in numerical experiments.

Critical strengths
  • Introducing an interesting family of regularized policy improvement methods
Critical weaknesses

Major weaknesses

  • Weak motivation/contextualization for introducing new algorithms. The key difference from existing offline RL regularization seems to be the uniform boundedness of the policy ratio. However, there is no clear motivation or empirical justification for pursuing this objective. Moreover, the comparison to the previous regularized policy improvement algorithms is insufficient. The authors should clearly discuss the novelty in comparison to, for example, Belousov, B., & Peters, J. (2018) and Geist, M., Scherrer, B., & Pietquin, O. (2019) in a dedicated related-work section in the main text. Finally, the paper lacks the empirical comparison to the chi-divergence-regularized PI, which seems one of the closest relative of the proposed method.
  • Requires substantial improvement around formatting. For example,
    • L54-64: Duplicate paragraphs
    • The presentation is cluttered with excessive caveats (all over the paper, especially around L330-335)
    • notation/terminology without sufficient definition/explanation (e.g., bounded invariant value box, realized damped one-step bounds, p_k, b_k, weighted maximum likelihood, consequential transition)
    • nonstandard technical terms (e.g., squared ratio movement, probability policies)
    • The symmetric ratio interval [1/ρ,ρ][1/\rho,\rho] is introduced abruptly at L118 without sufficient explanation.
    • Eq. 20 is hard to interpret.

Other issues

  • PDIS used for the deployment test is often not practically usable for long horizon RL tasks, and the authors did not evaluate its usefulness in the experiments.
  • The hyperparameter tuning in the experiments (especially those with NeoRL) seems insufficient because the behavior policy beats most of the competitors, suggesting they are not really learning anything.
  • Thus p and b can be selected from an interpretable probability-ratio budget rather than only tuned as opaque regularization coefficients. (L120) ... I do not understand why it is interpretable.
  • L358: the exact calibratoin with the proposed parameterization also seems hard if the action space is continuous.
Questions for authors

Please address the weaknesses above, especially the motivation to introduce the new form of regularization.

6 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/4)

Thank you for recognizing the interest of the policy-improvement family and for identifying the need for stronger motivation, clearer positioning, and closer experimental comparisons. We have added a dedicated related-work section in the main text, expanded the comparisons with calibrated KL and Pearson chi-square methods, and revised the explanations around the main constructions. We address the motivation and novelty first, followed by the experiments and the presentation and implementation concerns. Key revisions are highlighted in blue in the revised paper.

1. Motivation for introducing Power Mirror

The central motivation is the tension between effective policy improvement and reliable policy updates in offline RL. A policy should increasingly favor actions with higher estimated values, but aggressive reweighting can amplify critic errors, particularly on actions rarely observed under the behavior policy. Conventional average-divergence constraints do not directly limit this amplification: for example, increasing an action's probability from 10−610^{-6} to 10−310^{-3} yields a 1,000-fold increase while the KL divergence remains small. This motivates controlling policy improvement through explicit actionwise probability-ratio bounds.

The key question is then how to approach greedy improvement while retaining quantitative control over policy deviation and critic-error sensitivity. Power Mirror addresses this question by starting from the Bellman backup rather than imposing a divergence penalty separately. The centered, shifted power mean provides a smooth interpolation between behavior evaluation and the supported maximum, while its gradient induces a normalized policy with nonnegative predicted improvement. Crucially, the same construction yields sharp actionwise ratio bounds, directly linking the aggressiveness of improvement to its potential amplification of critic error.

This produces an explicit and interpretable tradeoff: increasing the power emphasizes better actions and tightens the approximation to the supported maximum, but also widens the permitted probability-ratio interval. We recognize that KL can likewise be calibrated to enforce actionwise bounds. The motivation for Power Mirror is therefore not ratio control alone, but a unified backup-based construction in which improvement, approximation to greedy behavior, and robustness to critic error are quantitatively connected.

2. Comparison with Belousov–Peters (2018) and Geist et al. (2019)

We have added a dedicated related-work section in the main text (Section 2) to clarify these connections. Belousov–Peters establish policy improvement under general ff-divergence constraints, while Geist et al. develop a general framework for regularized Bellman operators and policy improvement. We position Power Mirror within these established frameworks rather than claiming regularized improvement or power-based weighting as new principles.

Our contribution is the joint characterization of the centered, shifted power-mean backup, its gradient-induced policy, exact improvement decomposition, and uniformly sharp actionwise ratio bounds. In particular, for D=span⁡(Q)<bD=\operatorname{span}(Q)<b, the induced policy satisfies

\le \frac{\pi_{\mathrm{PM}}(a)}{w(a)} \le (1+D/b)^{p-1}.$$ The same power $p$ enters the coverage bound through $W_\varepsilon^{1/p}$, explicitly connecting approximation to the supported maximum with the permitted probability-ratio budget and critic-error sensitivity. ### References [1] Belousov and Peters (2018). [*f-Divergence Constrained Policy Improvement*](https://arxiv.org/abs/1801.00056). [2] Geist, Scherrer, and Pietquin (2019). [*A Theory of Regularized Markov Decision Processes*](https://proceedings.mlr.press/v97/geist19a.html).
AuthorsOpenReview ↗

Official Comment by Authors (2/4)

3. Comparison with chi-square-regularized policy improvement

We have added Pearson chi-square controls on both Wet Chicken and Hopper. Appendix M.3 distinguishes standard chi-square PI/VI, whose penalty coefficient is selected by validation and fixed across iterations, from a separately reported span-calibrated chi-square PI variant (Figs. 9–10). Wet Chicken uses exact empirical-model policy evaluation and direct calculation of the finite-action span, making the stated policy updates explicit.

The Hopper comparison is in Appendix M.5, Fig. 16. Chi-square PI uses the current policy as its reference, ordinary current-policy evaluation, and direct policy extraction on fresh proposal pools. Its penalty coefficient is selected by offline validation and then fixed across states and iterations. The methods share the training data, fitted behavior model, and 50,000-update critic budget, with ten training seeds at each of the three data sizes. PMPI and both PMVI variants have higher mean return than this control across those sizes. The figure and accompanying protocol are now included in the paper.

The quadratic equivalence above remains important for interpreting these experiments: the same critic and reference give identical one-step policies when the chi-square coefficient is chosen as m2m_2. The empirical studies compare full training procedures with their stated calibration rules. Differences can therefore arise from how the coefficient is chosen and how evaluation and reference updates are repeated, while the analytic comparison identifies exactly where the two updates coincide.

4. Why use a strong behavior reference, and how is tuning assessed?

Our aim is to improve a reasonably strong behavior policy while controlling the risk introduced by the update. A strong but nonoptimal reference makes this a meaningful test: the learned policy has room to improve, and preserving the reference's useful behavior matters. This motivates the benchmark and reference choices, rather than treating behavior as a deliberately weak comparator.

Even a strong behavior policy can generate finite data that leave some action values poorly estimated. The risk comes from the candidate's critic: improvement can assign much more probability to an action whose value is overestimated, producing a large loss relative to an otherwise good reference. The benchmark choices let us examine this issue through complementary outcomes. Wet Chicken measures return together with waterfall cost, NeoRL measures whether learning improves on an executable locomotion reference, and MetaDrive measures task completion together with off-road failures and episode cost. These evaluations ask whether the method finds useful improvements while retaining the strengths of the reference (Section 8; Appendices L and M).

AuthorsOpenReview ↗

Official Comment by Authors (3/4)

4. Why use a strong behavior reference, and how is tuning assessed? (continued)

A strong behavior result is therefore relevant to the intended problem. We also address the tuning concern directly through expanded validation studies. The primary PMVI/KL comparisons use the same predeclared c=1c=1 and report the power sweep p∈{2,4,6}p\in\{2,4,6\} separately. The calibration ablations select c∈{1,2,4,8}c\in\{1,2,4,8\} by offline validation for each power and data condition. IQL tunes its expectile and advantage temperature, AWAC tunes its advantage temperature, and Hopper includes validation-tuned IBR. The behavior curve evaluates the same frozen fitted reference used by the continuous methods; powers are not selected using test returns (Appendix M.1).

The resulting comparisons show improvement over this strong reference. Both PMVI variants have higher displayed mean returns than fitted behavior and tuned IQL/AWAC at the smallest and largest Hopper data sizes (Fig. 12). The power and tuning studies also identify gains over calibrated KL: PMVI-adaptive exceeds KL-adaptive across the displayed Hopper sizes at p=4p=4, while validation-tuned PMVI-fixed exceeds KL-fixed across sizes at p=2p=2 (Figs. 13 and 17). Wet Chicken provides the accompanying return--cost comparisons (Figs. 4–5), and MetaDrive reports improved task completion alongside low off-road rates (Fig. 18). The shared-critic comparisons further examine the update separately from critic fitting (Figs. 7 and 14–15).

5. Presentation and readability

We have revised the presentation to make the motivation and the roles of the main results clearer. The Introduction motivates ratio control through a rare-action example; Sections 3–5 explain how the backup induces an improved policy and why fixed and moving references lead to different guarantees. We have consolidated duplicated introductory passages and reduced repeated caveats, keeping the theorem assumptions with the results and grouping the deployment conditions in Section 6 and Appendix H.

The terminology has also been simplified. Section 4 describes the value-function range through reward-based bounds; Section 5 explains mixing as taking a fraction of the proposed update. Appendix F explains the iteration-dependent power pkp_k and shift bkb_k, and Section 7 describes actor fitting using sampled actions and target-ratio weights. Nonstandard labels such as “squared ratio movement” and “probability policies” have been removed.

Section 3 now explains the symmetric interval as a bound on multiplicative probability increases and decreases. Section 4 introduces the meaning of the coverage quantities before Eq. (20). Their interpretations are given below.

6. Interpreting the ratio budget and Eq. (20)

The symmetric interval [1/ρ,ρ][1/\rho,\rho] is now explained in probability units. It permits at most a factor-ρ\rho increase or decrease relative to the reference. For ρ=2\rho=2, an action with reference probability 1% can receive between 0.5% and 2%. With η=D/b∈(0,1)\eta=D/b\in(0,1), the paper gives the sufficient choice

p≤1+log⁡ρ−log⁡(1−η).p\le1+\frac{\log\rho}{-\log(1-\eta)}.

Thus an application can specify the allowed probability change, and the bound translates it into a power once the shift is chosen. A larger shift narrows the interval. The derived PM interval can be tighter than the requested symmetric budget: at p=2,η=1/2p=2,\eta=1/2, it is [1/2,3/2][1/2,3/2]. For PMVI, the interval refers to the fixed behavior reference; for PMPI, it refers to each preceding policy and accumulates across updates (Sections 3–5).

Section 4 now explains the quantities and their roles before presenting Eq. (20):

\le\frac1{1-\gamma}\, \mathbb E_{s\sim d_\nu^{\pi^{\mathrm{cmp}}}} [\overline\Delta^C_{\star,\varepsilon}(s)].$$ The local term bounds how far the backup falls below the optimal value at each state. The comparison policy's state-visitation distribution weights those shortfalls, and $1/(1-\gamma)$ converts normalized discounted occupancy into total discounted return. The comparison policy is constructed in the proof. This explains why the result depends both on behavior probability on good actions and on which states matter for return. The bound's optimal-value quantities belong to the population analysis; Appendix N separately gives a finite-data PMVI improvement result under explicit sampling coverage.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of revisions and rebuttal responses

We thank all reviewers for their constructive feedback and careful consideration. We have substantially revised the manuscript, with particular emphasis on strengthening the motivation, improving the presentation, and providing more comprehensive experimental evidence. Key revisions are highlighted in blue in the revised paper.

  • Stronger motivation and clearer contributions (Sections 1–3). We now motivate Power Mirror through the tradeoff between effective policy improvement and critic-error amplification, illustrated by a concrete rare-action example. We also clarify its novelty and relationship to existing regularized policy-improvement frameworks.
  • Improved presentation and intuition (Sections 1–7). We have reorganized the theoretical development, added intuitive explanations for the main constructions and guarantees, clarified terminology, and improved the connections between results to make the paper easier to follow.
  • Expanded experiments and stronger baselines (Appendix M). We added calibrated KL, Pearson chi-square, IBR, and shared-critic comparisons, together with systematic validation-based hyperparameter tuning and additional ablations. The results provide stronger evidence for Power Mirror's effectiveness, particularly on Wet Chicken, while also demonstrating competitive performance and favorable results in several comparisons on the other benchmarks.

In our individual responses, we have also addressed the reviewers' specific concerns:

  • Reviewer f4i7. We strengthened the motivation, clarified novelty relative to prior work, expanded comparisons with KL and chi-square regularization, and addressed baseline tuning and practical deployment.
  • Reviewer xb2f. We substantially improved readability and intuition, clarified the distinction between population assumptions and finite observations, and established an additional finite-data improvement guarantee with explicit sample requirements (Appendix N).
  • Reviewer 5DEg. We clarified the adaptive-shift assumptions, added behavior-model sensitivity experiments and controlled comparisons, and corrected the continuous PMPI implementation with new experiments and a justification of its consecutive-policy ratio bounds. We appreciate the reviewer's follow-up and updated assessment.

We would greatly appreciate it if you could revisit our detailed responses and revised manuscript. Please let us know if any concerns remain; we would be happy to provide further clarification before the discussion period closes. Thank you again for your time and thoughtful consideration.

Area_Chair_FtyyOpenReview ↗

Reviewer–Author Discussion Period Ending Soon

Dear Reviewers,

With the discussion period coming to a close, I encourage you to revisit the authors’ responses and let us know if you have any further comments.

Thank you for your careful reviews and engagement in the discussion!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.