acceptodds
Under review as a conference paper at ICLR 2027

SWEEP: Achieving Global Convergence and Enabling Pareto Front Exploration for Deep Neural MORL

Abstract

Multi-objective reinforcement learning (MORL) has gained significant attention in recent years due to its wide range of applications. However, theoretical understanding on MORL algorithmic design remains in its infancy thus far, especially for the widely used deep neural network (DNN)-based actor–critic methods. This motivates us to study MORL from a theoretical perspective and develop a DNN-based actor–critic approach that (i) offers global convergence guarantees to weak Pareto-optimal policies and (ii) enables systematic exploration of the entire Pareto front (PF). Inspired by recent advances in multi-objective optimization (MOO) theory, our basic idea to enable systematic PF exploration is to scalarize the original vector-valued MORL problem with the weighted-Chebyshev (WC) technique and leverage the one-to-one correspondence between the PF and the WC-scalarization. However, the non-smoothness incurred by the WC-scalarization introduces difficulty in multi-objective policy gradient evaluation in MORL. Toward this end, we propose a new parameterized smoothed-WC actor-critic technique to overcome this aforementioned challenge to solve the deep-neural MORL problems with a global convergence rate of , where denotes the total number of iterations. Notably, to establish the global convergence rate guarantee for DNN-based MORL, we propose a new set of theoretical assumptions and analysis. To our knowledge, these assumptions and analysis are new in the literature and could be of independent interest to the field of actor-critic MORL with nonlinear approximation. Finally, extensive numerical experiments further validate the effectiveness of our method.

Primary area:
optimization
Keywords:
Multi-objective optimization, reinforcement learning.

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JshXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes SWEEP, a deep MORL algorithm based on smoothed weighted-Chebyshev scalarization. Under a set of assumptions, the authors establish a finite-time global convergence guarantee for the scalarized objective under a fixed preference. Experiments on one robotic control task and a recommendation dataset show certain advantages over the selected baselines.

Critical strengths

The paper attempts to establish finite-time global convergence guarantees for deep MORL actor-critic methods, accounting for neural function approximation, critic estimation errors, and Markovian sampling errors.

Critical weaknesses

As I understand it, the paper has two main goals: establishing global convergence for deep MORL and enabling Pareto front exploration. In my view, neither goal is convincingly achieved.

Regarding the theory:

  • First, what the paper proves is that, under a set of assumptions and for a given preference vector, the algorithm can approach an approximate global optimum of the corresponding scalarized objective. However, the title and introduction could easily give the impression that the algorithm is proven to converge to the entire Pareto front, which is not what the theorem establishes.
  • The convergence guarantee relies on strong assumptions. For example, Assumption 6 directly imposes a relaxed PL condition. Ignoring the error term, this condition implies that any stationary point must be globally optimal. This already rules out many difficult cases where non-convex optimization gets trapped at suboptimal stationary points. General deep RL problems do not necessarily satisfy these additional conditions. So the broad claim of "establishing global convergence guarantees for DNN-based MORL" is misleading.
  • The authors contrast SWEEP's global optimality guarantee with MGDA-based methods, which generally guarantee convergence to Pareto-stationary solutions. However, these guarantees are established under substantially different assumptions. MGDA is generally analyzed for non-convex multi-objective problems without requiring many of the strong assumptions imposed in this paper. This comparison does not seem entirely fair.
  • Lemma 3 does not hold as stated. A weakly Pareto-optimal solution is not necessarily a global minimizer of the smoothed Chebyshev objective under any preference vector. Therefore, the claimed guarantee of recovering all weakly Pareto-optimal solutions is not valid in general.
  • The claim that "SWEEP enjoys a finite-time convergence rate of O(1/T) to a globally optimal policy" is not entirely accurate. Theorem 1 contains additional approximation and sampling error terms that do not vanish with the number of iterations when the corresponding parameters are fixed. Therefore, the theorem only guarantees an O(1/T) decay for part of the optimization error.
  • The sign of the TD error in Algorithm 1 appears to be incorrect.
  • Prior work [1] has already studied actor-critic methods with nonlinear scalarization and multiple critics, and [2] has investigated similar nonlinear scalarized policy gradients and provided convergence analysis. These closely related studies are missing from the discussion.

Regarding Pareto front exploration:

  • In my view, SWEEP is essentially a single-policy method. It optimizes one policy for a given preference vector at a time, without introducing a particularly effective mechanism for exploring the Pareto front. For example, in the Walker2d experiments, the authors manually add preference vectors to fill sparsely covered regions of the front.
  • The paper overlooks several important and closely related works on deep MORL. For example, [3] studies how to guide Pareto front exploration. [4] combines MGDA-based policy improvement with multi-policy exploration and adaptive selection to address the limitations of local gradient-based optimization and better cover non-convex regions of the front. [5] studies theoretical guarantees for Pareto front exploration. These works are highly relevant but are not discussed.

Regarding the experiments:

  • SWEEP claims to explore the entire Pareto front. However, the evaluation in the recommendation task mainly measures the quality of a single policy. Also, the Hypervolume comparison actually uses single-solution Boxvolume, which does not directly reflect the coverage of the entire front.
  • In the recommendation experiments, two of the baselines use linear function approximation, whereas SWEEP uses DNN. This comparison may not be entirely fair, as some of the performance differences could be attributed to model capacity rather than the proposed method itself.
  • The robotic control experiments are conducted on only one task, with the main comparison being against a linear-scalarization variant, without including existing MORL algorithms as baselines.
  • The experiments do not establish whether the theoretical assumptions hold in the tested environments.

[1] Agarwal et al., Multi-Objective Reinforcement Learning with Non-Linear Scalarization. AAMAS 2022. [2] Bai et al., Joint Optimization of Concave Scalarized Multi-Objective Reinforcement Learning with Policy Gradient Based Algorithm. JAIR. [3] Xu et al., Prediction-Guided Multi-Objective Reinforcement Learning for Continuous Robot Control. ICML 2020. [4] Hu et al., PA2D-MORL: Pareto Ascent Directional Decomposition Based Multi-Objective Reinforcement Learning. AAAI 2024. [5] Röpke et al., Divide and Conquer: Provably Unveiling the Pareto Front with Multi-Objective Reinforcement Learning. AAMAS 2025.

Questions for authors

Please refer to Weaknesses.

Reviewer_QQV1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SWEEP, a DNN-based actor-critic algorithm for multi-objective reinforcement learning (MORL). It aims to achieve global convergence and preference-guided Parato front exploration. SWEEP adopts weighted Chebyshev (WC) scalarization to convert the MOO problem into a scalar optimization problem, and employs smoothing techniques to address the non-differentiability of vanilla WC objective. This paper establishes a finite-time convergence analysis with O(1/T)O(1/T) rate. Additionally it is also capable to explore the entire weak Pareto front by varying preference weights. Experiments on MuJuCo and KuaiRand recommendation datasets demonstrate the effectiveness of the proposed method.

Critical strengths

The strengths are summarized below:

  1. The application of smoothed WC scalarization is well motivated.
  2. This paper establishes a theoretical framework considering finite-step neural critic training, DNN function approximation, errors in scalarization weights, etc. It then provides the global convergence analysis.
Critical weaknesses

The weaknesses are summarized below:

  1. The global convergence guarantee relies heavily on Assumption 5 and 6, particularly 6, which applies a relaxed PL condition directly on the smoothed WC objective. This assumption itself, already ensures that stationary points are approximately globally optimal. Under such condition, I think many existing methods, or even standard gradient-based optimization methods can also achieve global convergence. Therefore, I concern that, the claimed global convergence is due to the assumed favorable optimization landscape.

  2. Assumption 5 requires a compatible advantage approximation using policy score under the optimal policy's stationary distribution. Although it is used in the derivation of global performance gap bound, the proposed SWEEP does not explicitly construct this approximation, and it does not justify why the DNN actor satisfies the assumption.

  3. The paper repeatedly emphasizes the difficulty and novelty of establishing the global convergence guarantee of DNN-based MORL. However, convergence for neural TD learning, neural actor-critic methods, smoothed WC scalarization have already been developed. The novel component appears to be the neural critic estimation errors in the scalarization weights.

Questions for authors

See above weaknesses discussion.

Reviewer_PanCOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The main contribution of the paper is the development of a deep neural network-based actor–critic method for multi-objective reinforcement learning that combines systematic exploration of the Pareto front with theoretical convergence guarantees. The proposed method employs a smoothed weighted-Chebyshev scalarization to address the non-smoothness of the underlying multi-objective optimization problem. The principal theoretical contribution is the derivation of a global convergence rate of O(1/T) towards weak Pareto optimality, based on a new set of assumptions and analytical techniques for nonlinear function approximation.

Critical strengths
  1. Addresses a relevant and challenging theoretical problem: The paper tackles an important gap between the practical use of deep neural actor–critic methods in MORL and the relatively limited theoretical guarantees available for such methods. In particular, establishing convergence to globally weakly Pareto-optimal policies, rather than merely Pareto-stationary points, would represent a meaningful theoretical advance. The combination of nonlinear function approximation, global convergence analysis, and multi-objective optimization makes the problem both relevant and technically challenging.

  2. A coherent approach connecting scalarization and policy optimization: The use of smoothed weighted-Chebyshev scalarization provides a conceptually acceptable way to connect MORL with established multi-objective optimization techniques. The resulting differentiable objective allows the authors to employ policy-gradient methods while retaining, under the stated conditions, the ability to recover weakly Pareto-optimal solutions through different preference weights. This provides a unified framework for policy optimization and preference-guided Pareto front exploration.

Critical weaknesses

The following are potential weaknesses, but I don't consider them to be "critical".

  1. The principal theoretical result (Theorem 1, p. 8) depends on six assumptions, including a new compatibility condition (Assumption 5) and a relaxed Polyak–Łojasiewicz-type gradient domination condition (Assumption 6). In particular, Assumption 6 already imposes a strong relationship between the gradient magnitude and global suboptimality of the scalarized objective. This raises the question of how much of the claimed global convergence result follows from the algorithm itself, rather than from assumptions that effectively exclude problematic local optima. The paper would benefit from a more substantial discussion of when these assumptions can be verified for practical DNN-based actor–critic problems.

  2. The empirical evidence does not fully establish the claimed practical advantages: The experimental results are promising but leave several questions unanswered. The robotic experiments mainly demonstrate improved Pareto front exploration relative to linear scalarization, while the recommendation experiments show mixed performance across objectives. In Table 1 (p. 9), SWEEP outperforms PDPG on several metrics but performs worse on others, including Dislike, Forward, and Watch Time.

Questions for authors

There is a distinction between Pareto optimality and weak Pareto optimality. The paper's formal results concern weak Pareto optimality, whereas its narrative frequently refers to exploration of the entire Pareto front. These are not equivalent notions, as the authors themselves acknowledge in Definition 1 (p. 4). What is the impact on applicability?

Comments on the paper

1 comment

Dear Reviewers,

The authors have provided rebuttals to your review. Could you please check their responses and provide any further feedback or questions for the authors to address at your convenience?

Thank you for your time and contribution.

Regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.