acceptodds
Withdrawn from ICLR 2027

Is One Step Enough for Offline Policy Improvement?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Behavior regularization in offline reinforcement learning limits the exploitation of critic errors, but strong anchoring can also restrict policy improvement. We study how policy improvement is composed through multi-step proximal policy improvement (MPI), which re-centers each proximal objective on the preceding policy. We parameterize the procedure by a nominal total horizon and stages with local horizon , distinguishing subdivision at a fixed total horizon from additional refinement at a common local horizon. Our analysis shows that sequential re-centering can reach endpoints unavailable to any single proximal step and characterizes how subdivision reduces the leading local discretization error of ideal updates under a fixed critic. We consider TD3+BC and IQL-based policy extraction to examine how improvement composition interacts with actor objectives and policy geometry. TD3+BC experiments on D4RL locomotion suggest that subdivision can broaden the range of useful total horizons, while adding refinement stages at a fixed small local horizon can improve return. The results identify improvement composition as a design choice alongside regularization strength, with distinct effects from horizon subdivision and additional policy extraction.

Primary area:
reinforcement learning
Keywords:
Offline Reinforcement Learning, Multi-step, Policy Improvement, Proximal Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_WGkaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes to split the step size T of the policy update during the policy improvement step in offline RL, and take K repeated gradient steps instead, of size T/K each. The motivation is that repeated gradient steps can reach different points in function space during optimization, compared to a single step with a large step size.

Critical strengths
  1. The idea is rather simple and straightforward.

  2. The presentation is, for the most part, clear.

  3. Results could be promising.

Critical weaknesses

Motivation, novelty and significance:

The motivation for the approach, as I understand it, is that multiple smaller updates can arrive at different points during optimization, compared to a single update. This is a well known property to my knowledge and goes beyond regularization (which is the main point of view taken by this work). The approach proposed in this work, as I understand it, is simply to do multiple smaller updates. This idea is, again, not novel (for example, see [1]), and seems to me to be effectively but another way to increase the replay-ratio (the number of updates per iteration, as opposed to the size of each update). This reduces the novelty of the approach singificantly. Additionally, increased replay ratio is not discussed or compared to I believe, which prevents me from being able to evaluate the significance of this contribution.

Evaluation, its fairness, and its presentation:

The method, unless I misunderstand it (in which case, please correct me), comprises keeping K policy heads / individual DNNs (line 160), and training them with respect to each other. This raises multiple questions which I think are major and are not addressed by the paper (unless I missed them - in which case, again please correct me):

  1. Why keep K actors, and not just train the same actor regularized with respect to itself with stop-gradient?

  2. What is the increased cost, and is it worse it?

    1. First, the cost in parameters from maintaining K - 1 additional actors, and isn't it better to spend the same parameters increasing the critic / standard actor's network sizes?.
    2. Second, the cost in number of updates (The "replay ratio"). K updates have approximate times-K the cost of a single update. Is this tradeoff every worthwhile? From the results, it seems to me the answer is strongly no, but an explicit experiment is critical here in my opinion, either way.
  3. It's not clear to me what exactly is being evaluated in the experiment when T is increased. I see two possibilities, both are problematic:

    1. T is effectively the learning rate of the baseline, and the I-MPI-K does K updates of size T/K. If this is the case: Is it even meaningful to increase T? T is presented as a notion of horizon, but unless I misunderstand (in which case as usual please correct me) its much more prudent to think of it as a learning rate (Figure 1). Decades (more?) of experience in ML tell us that changing the learning rate can cause almost arbitrary decay in performance, and that tuning learning rate is super important. So this makes comparisons to the baseline close to meaningless, in my opinion.
    2. T decides the size of the steps for I-MPI-K. In which case, the comparison to the baseline is unfair, as its strictly getting more updates? And how does this correlate with tuning learning rate?
  4. Tables are the least-preferable way to visualize data. I strongly recommend changing Table 1 to something more visually informative - perhaps a bar plot?

Statistical significance:

Missing from many results (Figures 2, 5, and 6, Table 1).

Inappropriate stat. signif. metric for the other results (Figures 3 and 4). Please replace the single standard-dev. over seeds with a proper confidence bound (for example, 95% Gaussian CI, e.g. 2xStandard error of the mean, when eveluating the significance of the mean performance).

[1] Oren et. al., Value Improved Actor Critic Algorithms. NIPS 2025.

Questions for authors

Please see the weaknesses section.

Reviewer_k8GFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies multi-step proximal policy improvement for offline reinforcement learning, where successive actor stages are regularized toward the preceding policy. It examines how update depth and nominal improvement horizon affect policy extraction, presents theoretical properties of composed updates, and evaluates deterministic and Gaussian variants on D4RL locomotion tasks.

Critical strengths

(a) The figures and tables are generally organized and make the reported trends easy to follow.

(b) The experimental study reflects substantial effort across horizon and depth sweeps, multiple actor formulations, and additional controls and diagnostics.

(c) The paper also reports negative results and acknowledges several limitations.

Critical weaknesses

I recognize that the authors carefully qualify their theoretical results and empirical claims, and my concerns do not depend on attributing stronger claims to the paper than it makes. Instead, they concern the mismatch between what the paper actually establishes within those qualifications and the substantive question motivating its proposed approach. The weaknesses below should be read in this light.


(a) The theory does not provide a substantive contribution beyond elementary properties of composing proximal updates, since the propositions are straightforward applications of standard proximal-optimization and numerical-analysis arguments.

More specifically, Proposition 1 establishes different reachable endpoints without showing whether the additional endpoints can be beneficial. Proposition 2 establishes that subdivision reduces the leading local error relative to a chosen flow, sidestepping the paper’s central question of which update composition is preferable. Evaluating candidate updates by their deviation from that flow already privileges the continuous-time limit as the benchmark, without establishing why proximity to that limit should be preferred. In my view, this shifts the focus away from determining which update composition yields better policy improvement. Finite-step deviations may themselves be beneficial and are part of the trade-off the paper should investigate. Showing that subdivision reduces those deviations therefore does not explain which step size or composition is preferable. Leaving this choice of benchmark unjustified substantially limits the result’s relevance to the paper’s central argument.

Furthermore, Proposition 3 only telescopes assumed per-stage objective inequalities and appends a surrogate-error bound, and this raises the question of what the paper is actually trying to establish. It could isolate update composition under clean assumptions, such as exact value functions, and derive a nontrivial performance comparison that provides new insight into the paper’s central question. Alternatively, it could address the offline setting by analyzing how critic errors and data coverage change that comparison. It does neither. Introducing these quantities without explaining their consequences adds formalism, not insight into the trade-off being studied. The accumulation of standard results leaves the paper’s own scientific objective unclear and provides little substantive support for its proposed approach.


(b) The offline RL framing does not lead to a substantive insight into the difficulty that motivates the method. The relevant question is how successive policy shifts balance useful improvement against exploitation of learned-critic errors outside the data-supported region. The theory assumes a surrogate-error bound without characterizing how it depends on those shifts, depth, or data coverage. The experiments measure action displacement and target-value perturbation without establishing whether critic errors are reduced or increasingly exploited. Thus, neither component explains the defining offline trade-off.

The experiments also do not establish a sufficiently clear empirical conclusion to fill this gap. At a fixed total horizon, increasing depth simultaneously changes the first actor’s regularization, the bootstrap targets it supplies, and the actor-update cost. The matched re-centering control in Appendix C.3 yields a mean difference of −0.73 points, with a task-bootstrap interval of [−10.6, 8.1] and substantial positive and negative effects across tasks. This does not establish an aggregate benefit from re-centering, and the paper does not identify conditions that explain or predict these differing effects. The Gaussian experiments likewise report gains and failures without establishing when a particular composition should be preferred. Consequently, the observations do not develop into a clear account of what makes composition useful or a reliable empirical principle for choosing it.


The paper combines generic facts about update composition with changes in empirical return, but does not establish the connection needed to make this a substantive offline RL contribution. Resolving the aforementioned issues would require new explanatory results and experiments that establish when and why the central mechanism is beneficial. In its current form, I do not find the contribution sufficiently substantial or supported for acceptance.

Questions for authors

See weaknesses.

Reviewer_ZxRsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose MPI, a refinement approach to offline RL, where multiple policies are trained sequentially. Each new policy is trained to stay close to the previously trained policy using a proximal objective. The authors employ a square distance metric as a proximal objective penalty for deterministic policies, and the KL divergence for stochastic policies.

Critical strengths

The authors study the notoriously challenging task of offline RL, which remains highly relevant.

The authors include several theoretical analyses for the proposed method.

Critical weaknesses

The paper suffers from somewhat poor presentation. Some sections and results are provided with little context and guidance to the reader, leaving them to guess what their purpose is.

The experimental results feature only three distinct and relatively simple environments from the D4RL benchmark. These are fairly saturated and are, in my view, insufficient to establish the superiority of the proposed method.

Questions for authors

What is the purpose of Sections 3.1 and 3.2?

Lemma 1 is provided with little context or discussion. What does this lemma establish and how does it help/justify the proposed method?

The paper "Revisiting the Minimalist Approach to Offline Reinforcement Learning" also proposes an improvement to TD3+BC. How does MPI compare against it?

Comments on the paper

1 comment

I thank the authors for their more detailed rebuttal. Unfortunately, it is still phrased in a way that is difficut for me to understand how, if, and to what extent it addresses the concerns I've raised.

I would like to explicitly restate the following:

Soundness:

It remains unclear to me what is the fundemental advantage of this approach, and if there's reason to expect it to be worth the compute cost. This is not to suggest that there isn't any - it is merely that it remains unclear to me from the current presentation of the paper. I encourage the authors to iterate on this in the next version of the document.

Statistical significance:

Every result presented in the paper needs to be paired with the appropriate metric for statistical significance. Table 1 is still not paired with statistical significance. Results are close to meaningless without statistical significance. The reader does not need to go to the appendix to see it.

Presentation and results:

In my understanding, the most important comparisons for a method that increases the number of updates is comparison to other increased-update methods (e.g. increased replay ratio), as well as wallclock runtime. I believe sufficient comparisons of this kind are still missing.

In addition, I find the presentation of the results a bit unclear. I think the main reason is that the author's approach is not entirely standard (at least, not that I've seen). That's not a bad thing - it suggests novelty. However, it makes the results harder to interpret, and thus appreciate the authors' contribution. I think the method and the experiment section will benefit from clearer presentation.

For these reasons, I am unable to change my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.