acceptodds
Under review as a conference paper at ICLR 2027

Robust Actor-Critic Learning under Distributional Uncertainty

21 views

Abstract

We study finite-horizon Markov decision processes in which the transition dynamics are only known to lie within a Wasserstein ball around a reference model. Our goal is to learn policies that remain effective under such distributional uncertainty. A central difficulty is that the worst-case transition kernel depends on the policy itself, so the usual policy-gradient expression does not fully capture the sensitivity of the robust objective. Our main contribution is a characterization of one-sided directional derivatives of the robust value function with respect to the policy parameters, obtained via the Lagrangian dual of the inner robust problem. These derivatives replace the classical policy gradient and determine both the policy update and the auxiliary quantities that must be estimated. Building on this result, we develop a robust dynamic programming principle that applies jointly to the value function and its sensitivity, yielding backward recursions that a critic can learn to approximate. The resulting actor–critic algorithm uses three parametrized components, with separate networks for the policy, the robust value function, and its sensitivity. Under suitable regularity conditions, the latter is vector-valued and provides the policy gradient. On finite-state benchmarks the method recovers the exact robust dynamic programming solution, and on a linear–quadratic control task in continuous state space, it produces policies that remain reliable under shifted dynamics.

Primary area:
reinforcement learning
Keywords:
Robust Markov decision processes, Robust reinforcement learning, Policy gradient, Distributionally robust optimization, Dynamic programming principle, Randomized policies

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Pxv1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

this paper establishes a robust DPP for finite-horizon MDPs under general set-valued transition kernels and randomized policies and proposes a tractable directional actor–critic algorithm.

Critical strengths

the proofs are sound and rigorous, algorithms are clearly motivated and the numerics are extensive and informative. moreover the contribution to the literature is non-trivial

Critical weaknesses

the paper is a bit abstract, and while i have been able to follow the general ideas, i believe the target audience of this paper may be limited at iclr.

Questions for authors

do the authors have a route forward to incorporate function approximation into this work? can they convince me that it wouldn't make the current obsolete?

1 comment
AuthorsOpenReview ↗

response to Reviewer Pxv1

Thank you for the review. We answer both points below. New proofs are in Appendix G of the revised PDF, and new experiments are in our anonymous repository.

1. “Abstract nature and limited ICLR audience.”

Existing robust policy-gradient methods can use the gradient from one selected worst-case kernel. When several worst-case laws are optimal, this gradient can point in the wrong direction. Theorem 3.5 give the exact one-sided derivative by considering all optimal laws. This issue occurs in our experiments. At the exact dual optimum, 13–35% of transport cells have multiple optimal destinations (Appendix E, Table 13). In 1,332 deliberatly constructed tie tests across 12 shared parameter settings, the selected-kernel gradient wrongly predicted ascent 92 times (6.9%), reaching 39.6% in one setting. Our derivative gave the correct sign in every test.

The correct direction can also improve performance. Using the same exact-objective line search, our method achieves higher final robust value than selected-kernel policy gradient in 10 of 12 settings (paired 95% intervals). It also beats an RNAC-style baseline based on Zhou et al. (2023) in 10 of 12 settings and nominal continuation in all 12 settings.We also implement a sampled actor–critic with a sensitivity critic on LQ control, pendulum and lost-sales inventory tasks.

2. “Route to function approximation. Would it make this work obsolete?”

No. We already use function approximation through a neural policy, robust value critic and sensitivity critic. Theorem 3.5 tells what sensitivity these components should estimate. It applies to compact, potentially continuous spaces under the stated regularity assumptions. The exact ascent guarantee, however, requires a finite model.

We also made improvements toward larger problems.First, the exact ascent of Algorithm 1 (Theorem 3.8, Appendix C) is now much cheaper. Proposition G.1 replaces the exponential enumeration of near-active branches with one polynomial-size LP and Proposition G.2 replaces the global constants C1,C2C_1,C_2 of Theorem C.1(iv) with a backtracking step. On the benchmarks of Table 3 (Appendix E.1.2), a run now takes less than 0.1 s instead of up to 8 minutes. The method now runs with up to 30 states where listing branches is infeasible and the objective never decreased in any of our runs (see repository). Second, Proposition G.3 bounds the error from estimating the reference model by O(N−1/2)O(N^{-1/2}), using the dual of Proposition 3.2. An update that improves the estimated objective improves the true objective up to an error of twice this bound. Appendix E.1.3 shows that the error decreases as the sample size grows. Third, if the value critic has error ete_t at stage tt, the total propagated value error is at most ∑tet\sum_t e_t. Apendix E.1.3 also measures the effect of critic-label noise.

The exact method therefore gives us a reliable reference for developing and testing approximate methods

Reviewer_ERm2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a specific formulation of robust MDP, where the uncertainty set is modeled as a Wasserstein ball. It derives a actor critic algorithm to learn the optimal robust policy. Several theorems build up the theoretical foundation of the algorithm. Then a practical version of the algorithm motivated from the framework is implemented in several classic control tasks, which further validates the theory.

Critical strengths

This paper is mathematically rigorous, and experiments validate many theoretical findings. It provides new results, such as one-sided directional derivatives of the value function and pointing out classical policy gradient is incomplete, which in my opinion are non-trivial. Limitations are also being clearly stated in the paper. There are extensive ablation study to examine the proposed methods.

Critical weaknesses

The practical algorithm is not theoretically guaranteed. Not sure how the sensitivity term affect empirical performances, based on my understanding this is hard to get, but the empirical performances without this term are in general on par with dropping it. Not sure if this is because the tasks are too trivial. The computational overhead of the proposed method is too high. existing scalable robust RL algorithms are able to achieve similar or less computation overhead.

Questions for authors
  1. line 68 "We develop a policy gradient framework for robust RL under distributional ambiguity around the reference transition kernels" and citation of the paper Wang and Zou for comparison(btw the first author name Yining Wang or Yue Wang? Just want to confirm it's the same paper I found, as they propose an online RL problem) mislead reader that this is a paper solve RL problem, which involves exploration and optimization. But Iater I found this is an MDP control paper. I think this is minor but worth care.

  2. How the sensitivity term affect performance both theoretically and empirically?

  3. At the beginning, there are four large assumptions in total take up almost a page. Some of them only are claimed to be very useful in subsequent analysis, a thorough discussion is missing. What does these assumptions mean and why they are important? How are these assumptions related to existing works?

  4. There are two ICML papers are related to the topic. Clipped Q-Learning: Your Value Clipping Is Secretly A Robust Operator: include a deep robust Q-learning implementation. Doubly Regularized Markov Decision Processes for Robust Reinforcement Learning: include a deep robust SAC implementation. According to the ICLR policy they are published at least two months before the submission ddl and thus should be included in discussion.

3 comments
AuthorsOpenReview ↗

response to Reviewer ERm2

Thank you for the careful review. We answer each point below. New results are in our anonymous repository.

1. “The practical algorithm is not theoretically guaranteed.”

The gap is now smaller. Algorithm 1 no longer has to check every combination of worst-case kernels, and it no longer needs the constants C1,C2C_1,C_2 (Appendix G). If the reference model is estimated from NN samples for each state, action and time step, the method still improves the true objective, up to an error of order 1/N1/\sqrt N. If the critic has error ete_t at step tt, the error in the value is at most ∑tet\sum_te_t.

2. “How does the sensitivity term affect performance?”

It depends on whether parameters are shared.

With tabular softmax, the exact and nominal directions have the same fixed points. The transported continuation changes the state weights but not the sign of each update (Appendix E.4, Table 5). This explains the on-par results on tabular tasks. The two gradients still differ, and the nominal one is far from the exact derivative (Appendix E.5, Table 6).On the lost-sales inventory task with a neural policy, the exact recursion gives a lower worst-shift cost (Appendix E.8, Table 11).

With shared parameters, the two directions can lead to different policies. In a three-state example with one parameter, the nominal fixed point is not even stationary. In new experiments with a linear policy on a few shared features, the nominal direction loses much more robust value than with independent parameters. In a second study, it ends worse in all 12 shared-parameter settings(see repository). We will add these results to the appendix.

3. “Computational overhead.”

We define the overhead as the time per iteration of the robust method at ε=1\varepsilon=1 divided by that of the non-robust method. In the submitted code, the grid search over λ\lambda took most of a robust iteration (86–88%). We now solve the scalar dual exctly, since it is concave and piecewise linear, so no grid is needed (revised PDF,paragraph "Robust overhead" and Table 15). Measured on an Apple M4 with one thread, the overhead per iteration drops from 8.7–12.6× to 2.2–2.8×. Table 15 also gives the time split of the new iteration. The 9.4× and 13× of the submitted paper were measured on an older machine, where the same code runs about 1.9× slower

4. “Line 68 and Wang and Zou.”

we are sorry for the typo. In the revised PDF, the reference to Wang and Zou (2022), “Policy Gradient Method for Robust Reinforcement Learning” (ICML 2022), now gives the first author as Yue Wang. The contribution sentence (Introduction, “Our contributions”, page 2) now describes a policy-gradient framework for robust MDPs with a sampled actor–critic with exploration out of scope.

5. “Four long assumptions.”

The assumptions follow the conditions used by Neufeld, Sester and Sikic (2023) for robust dynamic programming on Borel spaces. Compactness and continuity of the ambiguity sets give a measurable worst-case kernel. Continuity of policies and reward makes the value functions continuous. Weak continuity of the reference kernel and C1C^1 policies with bounded score give the derivative formula. In common cases they are much simpler. For Wasserstein balls on compact spaces, the first condition holds automatically. In finite spaces, only the C1C^1 bounded-score condition is needed and softmax policies satisfy it. We will add this discussion after the assumptions in the final version

6. “The two ICML 2026 papers.”

The revised PDF now cites both. Liu and Xu (2026) show that value clipping estimates a TV-regularized robust Bellman operator. He Liu and Xu (2026) give regret bounds for policy and ff-divergence regularized MDPs. Both are value-based with penalized ambiguity and closed-form duals. Neither uses Wasserstein balls or the derivative at nonunique worst cases, so they are complementary.

Thanks for the responses. I have read the rebuttal, which answers many of my questions. The scope now is much clearer with additional discussion.

We thank the reviewer for their careful consideration of our rebuttal and positive feedback on our clarifications.

Reviewer_jECKOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
1(−1)
Presentation
2
Soundness
3(+2)
Read review
Paper summary

The paper studies robust finite-horizon MDPs with (s,a) rectangular Wasserstein ball around a reference model. The main contribution characterizes one-sided directional derivatives of this robust MDP's value function with respect to the policy parameters. Using this one sided derivative, the authors develop an actor critic algorithm. Empirical evaluations for this algorithm are also provided.

Critical strengths
  1. The one-sided derivative calculations and the subsequent robust actor-critic algorithm proposed seem novel.
Critical weaknesses
  1. There are already multiple robust actor-critic methods which are absent from discussion in the paper (Zhou et al. 2023, Xu et al. 2026). More specifically, Xu et al. 2026 already studies robust ac with Wasserstein uncertainty and provides sample complexity guarantees to the optimal policy. So I am unclear what this paper in particular is contributing.

  2. As mentioned in the paper, the algorithm can require an exponentially large branch set and uses global constants to determine the step size. This seems too restrictive for practical implementation.

  3. Moreover, the results seem weak (only iteration bounds) given that there are existing finite sample bounds for robust Actor-Critic methods, including the Wasserstein case.

  4. The experiments, while they are better at supporting the calculations in the paper, don't really seem to show much of an improvement over using the typical robust policy gradient update.


Xu et al. Efficient Q-Learning and Actor-Critic Methods for Robust Average-Reward Reinforcement Learning. UAI 2026.

Zhou et al. Natural actor-critic for robust reinforcement learning with function approximation. NeurIPS 2023

Questions for authors

Please clarify the contribution of the paper. It is not clear how the paper improves upon existing robust actor-critic methods, both theoretically and experimentally. In particular, please compare with Xu et al. 2026.

8 comments
AuthorsOpenReview ↗

response to Reviewer jECK

Thank you for the review. New results are in our anonymous repository.

1. Novelty versus Zhou et al. (2023) and Xu et al. (2026).

Our contribution is the exact one-sided directional derivative of finite-horizon Wasserstein-robust values when worst-case laws are nonunique.

Zhou et al. (2023) develop robust natural actor–critic with function approximation and double-sampling/IPM uncertainty. Lemma 2 constructs a Fréchet supergradient using one worst-case kernel. Xu et al.(2026) develop model-free robust Q-learning and actor–critic for infinite-horizon average-reward MDPs, proving semi-norm contraction, critic bounds and finite-sample guarantees, including for Wasserstein ambiguity. We do not question these results. Neither characterizes the directional envelope over all optimal Wasserstein laws.

Theorem 3.5 derives this exact envelope and a coupled backward value–sensitivity recursion for compact, potentially continuous state/action spaces under stated assumptions. It motivates a sensitivity critic. Theorem 3.6 specialize to finite models, supporting exact ascent and directional-stationarity guarantees.For illustration, consider the three states of Remark B.1 with P0=(1/2,0,1/2)P_0=(1/2,0,1/2), c(i,j)=∣i−j∣c(i,j)=|i-j|, radius 1/21/2, and let Hθ=(1+θ,0,1−θ)H_\theta=(1+\theta,0,1-\theta). Then G(θ)=min⁡W1(P,P0)≤1/2EP[Hθ]=12−12∣θ∣.G(\theta)=\min_{W_1(P,P_0)\leq1/2}\mathbb E_P[H_\theta]=\tfrac12-\tfrac12|\theta|. At θ=0\theta=0, multiple worst-case laws are optimal. One yields a valid supergradient +1/2+1/2, which wrongly suggests ascent to the right, while the true right derivative is −1/2-1/2. our envelope recognizes θ=0\theta=0 as a stationary maximizer. Table 2 (R4) reports median derivative error 9.60×10−29.60\times10^{-2} for a selected coupling versus 1.49×10−111.49\times10^{-11} for the envelope. At engineered ties, selected-kernel directions falsely predicted ascent on 92/1,332 probes (6.9%), reaching 39.6% in one setting. Ours made no such errors in these tests (these are sign tests not accepted steps).

The revised PDF now discusses both papers in the related-work paragraph of the Introduction and adds them to Table 1 (blue text). Xu et al. appeared at UAI in August 2026, after the July 17 cutoff. The ICLR guidelines state that missing contemporaneous comparisons cannot alone justify rejection. We nonetheless compare their method in point 4.

2. Exponential branches and global constants.

Both points apply to Algorithm 1 as submitted (the derivative theorem is unaffected).We have since addressed both. By LP duality the near-active direction problem becomes a polynomial-size LP, so we no longer need to list all branches. Its gain Δ~(r)\widetilde\Delta(r) bounds the true gain Δ(r)\Delta(r) from below up to KζK\zeta, Δ(r)≥Δ~(r)−Kζ\Delta(r)\geq\widetilde\Delta(r)-K\zeta, so Δ~(r)>Kζ\widetilde\Delta(r)>K\zeta guarantees ascent. The constant KK comes from finite-model sensitivity bounds and is not used to set the step size. For the step size we use Armijo backtracking on the exact robust objective, which removes C1,C2C_1,C_2 and makes every accepted step an improvement. Proofs in Appendix G.

Runs that took up to about 8 minutes when listing branches now take under 1 s. In about 1,000 test runs the objective never decreased (anonymous repository).

3. Iteration bounds only.

Agreed. O(ζ−2)O(\zeta^{-2}) counts exact finite-model iterations and not samples. We additionally derive a finite-model plug-in O(N−0.5)O(N^{-0.5}) bound for robust-value estimation and empirical-policy performance supported by observed slopes near −1/2-1/2 (proof in Proposition G.3).

4. Gain over the usual robust policy gradient.

We ran 12 additional finite-model settings with 20 paired seeds each, using the same exact-objective line search for all methods. At ζ=10−5\zeta=10^{-5}, our method reaches a higher final robust value than selected-kernel policy gradient in 10/12 settings (paired 95% intervals)by up to 0.0110.011. With a fixed tuned step size instead of line search, 0.08% of our steps lower the objective in one setting and none in the other 11 against 1.6–13.6% of steps for polcy gradient on the random models.

We also ran Xu et al.'s benchmark tasks. Their setting is infinite-horizon average reward, so we trained our method on a 50-step version of each task and scored the learned policy by its exact robust average reward. Our method and our reimplementation of their actor–critic both find the optimal policy. This is expected since these tasks do not involve ties between worst-case laws (where our envelope matters)

Given the soundness score of 1, could the reviewer identify a specific theorem or proof step of concern? we would be glad to address it

I thank the authors for their response. First, the soundness score of 1 was a mistake, I intended to give 1 for informativeness (I have now fixed this). My main concern is the significance of the contribution.

I acknowledge the computational improvements in the revision, but my main concerns remain:

  1. Zhou et al. already allows shared policy parameters and uses natural policy gradient. It seems that their particular uncertainty sets are designed to make robust critic estimation tractable. Since your main convergence result assumes exact robust policy evaluation, this does not explain why a different actor update is needed. What prevents combining their robust NPG actor with the Wasserstein evaluation routine used here, under comparable approximation assumptions? The examples showing problems with selected-kernel policy gradients do not establish the same limitation for robust NPG.

  2. The experiments better support the derivative calculations, but I still do not see a convincing practical improvement over existing robust AC methods. Comparisons against selected-kernel PG do not resolve the comparison with robust NPG, and the reported comparison with Xu et al. shows both methods finding the optimal policy.

  3. The additional theoretical results still do not provide convergence or sample-complexity guarantees for the sampled actor–critic, whereas existing robust AC methods already have finite-sample guarantees.

Moreover, Xu et al. was available on arXiv from June 2025, already covering Wasserstein uncertainty and actor-critic methods. Its later conference publication date therefore does not remove the need for comparison.

Although the derivative characterization seems distinct, I remain unconvinced that it provides a sufficiently significant contribution. My overall rating remains unchanged.

We thank the reviewer for the follow-up and for clarifying the soundness score. We address the three points below.

1. Combining robust NPG with our Wasserstein critic.

We agree that Zhou et al.'s robust NPG actor could be combined with our Wasserstein evaluation routine. Our results identify the correct sensitivity needed for such a combination. The key distinction is between evaluating the robust value and computing its policy sensitivity. At nonunique worst-case laws, different optimal kernels can produce different policy derivatives. Natural-gradient preconditioning changes the actor update but does not, by itself, resolve which worst-case sensitivity should be used. Theorem 3.5 gives the exact one-sided derivative over all optimal Wasserstein laws, while Theorem 3.6 makes it computable in finite models. This characterization can guide standard policy gradients, natural gradients and other actor updates. The contribution therefore goes beyond the choice of actor optimizer.

2. Comparison with robust NPG.

We conducted additional comparisons against an RNAC-style natural-gradient baseline adapted from Zhou et al. (2023), using the same Wasserstein tasks and exact-objective line search. Our method achieves higher final robust value in 10 of 12 shared-parameter settings under paired 95% confidence intervals, with gains up to 0.028. It also improves over selected-kernel policy gradient in 10 of 12 settings and over nominal continuation in all 12 settings. These results directly address the comparison with natural-gradient methods and show that correctly handling worst-case sensitivity can improve the policies obtained under shared parameters.

3. Theoretical guarantees and Xu et al.

Our results address a different theoretical question from the convergence analysis in Zhou et al. and Xu et al. Theorem 3.5 characterizes exact directional sensitivity for finite-horizon Wasserstein-robust MDPs. Theorem 3.8 establishes monotone ascent and directional stationarity for the exact finite-model algorithm, while Corollary C.4 gives an O(ζ−2)O(\zeta^{-2}) iteration bound. We further derive an O(N−1/2)O(N^{-1/2}) finite-model bound for robust-value estimation and empirical-policy performance, together with a bound on how critic errors propagate through the backward recursion.

We appreciate the clarification regarding Xu et al.'s June 2025 preprint. We will clarify the distinction between its robust average-reward learning guarantees and our finite-horizon Wasserstein sensitivity results.

Our central contribution is an exact characterization of policy sensitivity under nonunique Wasserstein worst-case laws, supported by finite-model ascent guarantees and controlled comparisons with robust policy-gradient and natural-gradient methods. This provides a principled foundation for robust actor updates, independently of the optimization method used.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised Manuscript and Code Availability

Dear Reviewers,

The code and additional experiments are available in our anonymous repository. All modifications to the revised paper are highlighted in blue.

We remain happy to address any further questions.

Thank you for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.