acceptodds
Withdrawn from ICLR 2027

AMPO: Adaptive Minimax Policy Optimization for Federated Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In federated reinforcement learning (FRL), we aim to learn a shared policy that performs reliably across all local environments. We therefore focus on the worst-case return, rather than the average return optimized by conventional methods. Achieving this is challenging since the worst-case environment is induced by the learned policy itself and shifts during training. We formulate this goal as an adaptive minimax objective and address the resulting non-stationarity through a saddle-point reformulation: we propose adaptive minimax policy optimization (AMPO), a primal–dual policy gradient method in which the dual variable tracks the shifting worst-case environment. Our analysis both guarantees convergence—to a stationary point with sublinear dual regret—and informs the algorithm design: it prescribes how fast the dual variable may be updated relative to the policy, and quantifies a robustness–scalability trade-off in which emphasizing difficult environments reduces the effective parallelism gained from federation. Experiments on heterogeneous MuJoCo benchmarks show that AMPO improves worst-case performance over average-return baselines while retaining competitive average performance.

Primary area:
reinforcement learning
Keywords:
Federated Reinforcement Learning, Minimax Optimization, Robust Policy Optimization, Environment Heterogeneity, Primal-Dual Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8i7jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose an adaptive minimax policy optimization (AMPO), which is a primal–dual policy gradient method to consider the worst-case environment in adjusting the learning policy.

Critical strengths
  1. The paper is well writen. It is easy to follow the main idea and the results.
  2. The motivation of the minimax policy is reasonable. This could be a good theoritical way to improve the conventional FRL.
Critical weaknesses
  1. In order to learning the minimax objective, it seems that the privacy is compromised. Therefore, this AMPO method may not be useful to FL scenarios.
  2. The AMPO is not compared with the SOTA baselines.
  3. The experimental results are not analyzed sufficiently. For example, why the return drops in fig1(d). What are the impact of those phenomenons to the methods.
Questions for authors

Please address the weaknesses above.

Reviewer_gosROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AMPO, an adaptive minimax policy optimization method for federated reinforcement learning under heterogeneous environments. Instead of optimizing average return, the method focuses on worst-case performance and uses a primal-dual approach to adaptively reweight local environments. The authors provide convergence and dual regret guarantees, analyze the trade-off between robustness and effective parallelism, and evaluate the method on several MuJoCo benchmarks.

Critical strengths

I think the paper addresses an interesting problem, as optimizing worst-case performance is particularly important when local environments are highly heterogeneous. The theoretical analysis provides some interesting insights, especially the connection between adaptive environment weighting and effective parallelism in federated learning. I appreciate that the authors include several MuJoCo environments and comparisons between adaptive and uniform weighting, which help demonstrate the benefits of the proposed approach.

Critical weaknesses
  1. I have some concerns about the novelty of the proposed method. The minimax formulation and adaptive weighting strategy are closely related to Agnostic Federated Learning (Mohri et al., 2019) and Group DRO (Sagawa et al., 2020). While I understand that federated RL introduces additional challenges, the primal-dual updates seem fairly standard. I would like to see a clearer explanation of what is fundamentally new in the algorithm and theoretical analysis.

  2. I am not fully convinced that the theoretical results directly support the minimax objective. Theorem 1 establishes stationarity of the weighted objective and static dual regret, but these guarantees do not necessarily imply convergence to a solution of the original worst-case optimization problem in Eq. (2).

  3. There seems to be a gap between the theoretical analysis and the practical PPO implementation. The practical method uses EMA-smoothed, undiscounted returns and less frequent dual updates, while the theoretical guarantees rely on unbiased return estimates and a different update procedure.

  4. I find the experimental evaluation somewhat limited. Most baselines focus on average-return optimization, and I would have liked to see a more direct comparison with a DRO-based method. Although the authors include some sensitivity experiments, more extensive evaluations with different numbers of clients would help support the scalability claims.

Questions for authors
  1. Could the authors clarify the main novelty compared with Agnostic Federated Learning and Group DRO? In particular, which parts of the theoretical analysis specifically address challenges that do not arise in existing minimax optimization methods?

  2. Could the authors explain how the guarantees in Theorem 1 relate to the original minimax objective in Eq. (2)? Is it possible to provide a more direct guarantee on worst-case performance?

  3. How do the differences between the theoretical algorithm and the practical PPO implementation affect the convergence guarantees?

  4. Could the authors include a DRO-based baseline and further evaluate how the number of clients affects worst-case performance and effective parallelism?

Reviewer_kZoNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores federated reinforcement learning (FRL) in heterogeneous environments with distinct transition dynamics. Unlike most existing FRL studies that target on average return, this paper focuses on the worst-case minimum local return. Specifically, the authors formulate it as an adaptive minimax problem and resolve the non-stationarity through a saddle-point reformulation. A primal-dual algorithm named AMPO is developed, which combines server-side policy-gradient ascent with projected descent on environment weights. The theoretical guarantee of AMPO is presented in term of a sublinear dual regret. The performance of AMPO is further evaluated via experiments on four MuJoCo tasks.

Critical strengths
  • The problem is well motivated in the sense that this paper optimizes a minimum return across heterogeneous environments, which addresses a meaningful limitation of average-return FRL.
  • The performance of proposed AMPO is theoretically guaranteed with a sublinear regret. In addition, Proposition 2 connects weight concentration to gradient-noise reduction through the quantity KeffK_{eff}. Under some sampling assumptions, this provides a clear explanation of how emphasizing fewer environments can reduce the variance-averaging benefit of federatioin.
  • The performance of AMPO is also evaluated through extensive experiments on four MuJoCo tasks.
Critical weaknesses
  • There is a discrepancy between the established theoretical results and the claims made in the paper. As the authors acknowledge that by switching the max-min orders in equations (2) and (5), this leads to an upper-bounding relaxation with a possible duality gap. And in Theorem 1, the authors establish stationarity only in the policy variable of changing weighted objective and the dual regret against the best fixed weight in hindsight. However, this does not establish stationarity of minkJk(πθ)min_k J_k(\pi_\theta), nor the saddle-point stationarity. Therefore, there is a discrepancy, and the claims seem to be much stronger that what has been proved. Note that although we did not expect a guarantee for the global optimality, the proved quantities should match the stated objectives and claims.
  • From a methodological perspective, it is not quite clear what is fundamentally new compared to existing robust reweighting methods. In particular, agnostic federated/reinforcement learning and DPO methods have already combined model updates with adversarial group-weight updates. To that end, their worst-performing group can also change with the model. Hence, it is not clear what is the uniqueness or fundamentally new properties of the policy-dependent switching as advocated in this paper. In addition, the experimental evaluations mainly compare with average-return methods, without considering any related robust-reweight alternatives.
  • The empirical results are promising in general but not uniform. For example, Figure 1 shows substantial late-training worst-case gains in some settings, but the average-return gains and improvements from adaptive weighting are less consistent elsewhere.
Questions for authors
  • Regarding the above weakness 1, what can we tell from Theorem 1 regarding the original objective minkJk(πθ)min_k J_k(\pi_\theta)?
  • Can you clarify the theoretical novelty and new analytical methods? In particular, which specific analytical step addresses an RL-specific or FRL-specific difficulty beyond stochastic optimization of a nonconcave objective with unbiased gradient estimates?
  • Corollary 1 requires a deterministic lower bound that the harmonic mean Kˉeff≥κ\bar{K}_{eff}\geq \kappa holds true almost surely. However, the simplex update permits concentration on a single environment. Under what conditions can you establish such a nontrivial persistent κ>1\kappa>1? How do you select κ\kappa for the prescribed step sizes before observing the weight trajectory?
  • As described in Appendix F.2, the Kˉeff\bar{K}_{eff} is computed from weights that are averaged across evaluation points and seeds. However, Corollary 1 requires the harmonic mean of per-round effective counts. These quantities can differ substantially. Can you report the theorem-consistent quantity separately for each training run and clarify what the current tables/results demonstrates?
  • Another concern is on the claim of linear speedup in Corollary 1. Given the KK-dependent dual step size, what is the joint dependence of primal stationarity and dual regret on K,K, κ,\kappa, and MM? Does the claimed linear speedup concern only stationarity of the changing weighted objective, or also a comparable robust-optimization accuracy criterion?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.