acceptodds
Under review as a conference paper at ICLR 2027

Finite-Time Pareto Convergence of Decentralized Multi-Policy Gradient with Periodic Reuse for Multi-Agent Multi-Objective Reinforcement Learning

Abstract

Decentralized multi-agent policy gradient methods enable collaborative decision optimization across multiple agents without a central coordinator. However, most existing theoretical studies on decentralized policy gradients are restricted to single-objective settings, while real-world complex tasks typically require each agent to simultaneously optimize multiple conflicting objectives. To the best of our knowledge, the theoretical foundations of decentralized multi-objective policy gradient algorithms remain largely underexplored. To tackle this problem, we propose D-PRMPG, a decentralized periodic-reuse multi-policy gradient algorithm. By exploiting the reusability of dynamic multi-objective preference weights, our method substantially reduces the excessive computational overhead caused by per-iteration dynamic weight computation. We further present a rigorous convergence analysis, demonstrating that D-PRMPG attains an convergence rate under both general convex and nonconvex objective functions, where denotes the total number of iterations. Finally, empirical evaluations in multi-agent multi-objective reinforcement learning environments validate the superiority of the proposed algorithm.

Primary area:
reinforcement learning
Keywords:
Policy gradient, Decentralized reinforcement learning, Multi-objective optimization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_L8uAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposed a decentralized multi-policy gradient with periodic reuse for multi-agent multi-objective RL. I am afraid the paper is a direct combination of several existing ingredients "decentralized + multi-objective + policy gradient" and looks close to the cited paper PRMPG in AAAI. Nevertheless, although the authors have tried to argue that the combination makes a difference in this problem, they fail to give more theoretical differences and do not provide sufficient empirical results.

Critical strengths

The paper proposed a decentralized multi-policy gradient with periodic reuse for multi-agent multi-objective RL.

Critical weaknesses

The technical novelty of the paper is limited, with direct combination of several existing ingredients in the literature. The authors fail to give more theoretical differences and do not provide sufficient empirical results.

Questions for authors

Please clarify the uniqueness and novelty of the theorectical results.

Reviewer_sohROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes D-PRMPG, a decentralized multi-objective multi-agent reinforcement learning algorithm. To address the high computational overhead of solving weight subproblems at every iteration in decentralized settings, the authors introduce a periodic reuse mechanism that updates dynamic weights only every $\mathcal{D}$ steps while retaining previous weights otherwise. Theoretically, they prove convergence to the global Pareto-optimal solution at a rate of O(1/T)\mathcal{O}(1/\sqrt{T}) under general convex conditions, and to a Pareto-stationary point under nonconvex conditions. Experiments on the MOMALand benchmark evaluate the hypervolume and computational cost across different reuse periods.

Critical strengths
  1. The periodic reuse mechanism is practical and computationally efficient, reducing the frequency of solving weight optimization subproblems while maintaining the ability to approximate the Pareto front.
  2. The theoretical analysis provides formal convergence guarantees for decentralized multi-objective policy gradients in both convex and nonconvex settings, helping fill a clear gap in the literature.
  3. Experiments on the MOMAL and benchmarks show that a moderate reuse period such as D=3D=3 achieves a favorable trade-off between Pareto front quality and computational cost.
Critical weaknesses
  1. The experimental evaluation lacks comparisons with existing decentralized multi-objective baselines, making it difficult to substantiate the claim that D-PRMPG achieves better performance than prior methods.
  2. The assumption that parameter iterates remain uniformly bounded in distance from the Pareto-optimal solution is strong and may not hold during early training, which weakens the practical relevance of the convergence bound.
  3. The policy parameterization is tabular and scales poorly with the size of the state space, and the paper does not explain how the method extends to high-dimensional or continuous environments.
Questions for authors
  1. Why were existing decentralized multi-objective baselines not included in the experiments? Can you provide a rigorous comparison to justify the claimed superiority of D-PRMPG?
  2. How do you justify Assumption 2? Can the algorithm guarantee that parameter iterates remain within a bounded distance from the Pareto-optimal set during early training when policies are essentially random?
  3. The policy parameterization is tabular and scales poorly with the state space. How does the method extend to high-dimensional or continuous environments, and how are the local gradients estimated in practice?
Reviewer_reaBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies multi-agent multi-objective reinforcement learning with discounted reward. The paper proposed a MGDA based algorithm with a mechanism to reuse dynamic multi-objective preference weights. Theoretically, the paper provide a convergence rate under some assumptions. Finally the paper verified the algorithm performance from the MOMALand benchmark.

Critical strengths

This paper is well-organized and easy to follow. The authors did a good job in explaining all the context and defining the problem.

Critical weaknesses

My major concern is that the paper failed to mention a large number of existing related works. As far as I am concerned, a lot of theoretical work has been done in related area. Listing only small number of them here: [1] studies finite-time analysis of multi-agent RL under discounted reward, [2] studies same topic as [1] under average reward in tabular case, [3] studies convergence of multi-objective RL under both discounted and average reward, the algorithm is also MGDA based. Given this, what additional challenge does it bring to combine multi-agent and multi-objective in RL?

My another concern is that the problem considered in this paper is very simple, i.e. the state space is limited to tabular case which is like a grid. This is not to say that simple is a problem, but I was expecting to see larger contribution that is built upon what we already have. For example, [3] already assumes linear value function approximation and provides convergence analysis, and [4] already proved that policy gradient reaches the global optimum in non-convex case. So in my point of view the contribution in this paper is not ready for publish.

Questions for authors

It is confusing in Line 70 to mention that existing decentralized policy gradient methods do not recover the front, as MGDA also does not explore perato front and can only converge to a random stationary point. Also I feel like the topic can be more interesting if we consider a preference oriented optimization or reconstruction the whole front.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.