- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes SWEEP, a deep MORL algorithm based on smoothed weighted-Chebyshev scalarization. Under a set of assumptions, the authors establish a finite-time global convergence guarantee for the scalarized objective under a fixed preference. Experiments on one robotic control task and a recommendation dataset show certain advantages over the selected baselines.
Critical strengths
The paper attempts to establish finite-time global convergence guarantees for deep MORL actor-critic methods, accounting for neural function approximation, critic estimation errors, and Markovian sampling errors.
Critical weaknesses
As I understand it, the paper has two main goals: establishing global convergence for deep MORL and enabling Pareto front exploration. In my view, neither goal is convincingly achieved.
Regarding the theory:
- First, what the paper proves is that, under a set of assumptions and for a given preference vector, the algorithm can approach an approximate global optimum of the corresponding scalarized objective. However, the title and introduction could easily give the impression that the algorithm is proven to converge to the entire Pareto front, which is not what the theorem establishes.
- The convergence guarantee relies on strong assumptions. For example, Assumption 6 directly imposes a relaxed PL condition. Ignoring the error term, this condition implies that any stationary point must be globally optimal. This already rules out many difficult cases where non-convex optimization gets trapped at suboptimal stationary points. General deep RL problems do not necessarily satisfy these additional conditions. So the broad claim of "establishing global convergence guarantees for DNN-based MORL" is misleading.
- The authors contrast SWEEP's global optimality guarantee with MGDA-based methods, which generally guarantee convergence to Pareto-stationary solutions. However, these guarantees are established under substantially different assumptions. MGDA is generally analyzed for non-convex multi-objective problems without requiring many of the strong assumptions imposed in this paper. This comparison does not seem entirely fair.
- Lemma 3 does not hold as stated. A weakly Pareto-optimal solution is not necessarily a global minimizer of the smoothed Chebyshev objective under any preference vector. Therefore, the claimed guarantee of recovering all weakly Pareto-optimal solutions is not valid in general.
- The claim that "SWEEP enjoys a finite-time convergence rate of O(1/T) to a globally optimal policy" is not entirely accurate. Theorem 1 contains additional approximation and sampling error terms that do not vanish with the number of iterations when the corresponding parameters are fixed. Therefore, the theorem only guarantees an O(1/T) decay for part of the optimization error.
- The sign of the TD error in Algorithm 1 appears to be incorrect.
- Prior work [1] has already studied actor-critic methods with nonlinear scalarization and multiple critics, and [2] has investigated similar nonlinear scalarized policy gradients and provided convergence analysis. These closely related studies are missing from the discussion.
Regarding Pareto front exploration:
- In my view, SWEEP is essentially a single-policy method. It optimizes one policy for a given preference vector at a time, without introducing a particularly effective mechanism for exploring the Pareto front. For example, in the Walker2d experiments, the authors manually add preference vectors to fill sparsely covered regions of the front.
- The paper overlooks several important and closely related works on deep MORL. For example, [3] studies how to guide Pareto front exploration. [4] combines MGDA-based policy improvement with multi-policy exploration and adaptive selection to address the limitations of local gradient-based optimization and better cover non-convex regions of the front. [5] studies theoretical guarantees for Pareto front exploration. These works are highly relevant but are not discussed.
Regarding the experiments:
- SWEEP claims to explore the entire Pareto front. However, the evaluation in the recommendation task mainly measures the quality of a single policy. Also, the Hypervolume comparison actually uses single-solution Boxvolume, which does not directly reflect the coverage of the entire front.
- In the recommendation experiments, two of the baselines use linear function approximation, whereas SWEEP uses DNN. This comparison may not be entirely fair, as some of the performance differences could be attributed to model capacity rather than the proposed method itself.
- The robotic control experiments are conducted on only one task, with the main comparison being against a linear-scalarization variant, without including existing MORL algorithms as baselines.
- The experiments do not establish whether the theoretical assumptions hold in the tested environments.
[1] Agarwal et al., Multi-Objective Reinforcement Learning with Non-Linear Scalarization. AAMAS 2022. [2] Bai et al., Joint Optimization of Concave Scalarized Multi-Objective Reinforcement Learning with Policy Gradient Based Algorithm. JAIR. [3] Xu et al., Prediction-Guided Multi-Objective Reinforcement Learning for Continuous Robot Control. ICML 2020. [4] Hu et al., PA2D-MORL: Pareto Ascent Directional Decomposition Based Multi-Objective Reinforcement Learning. AAAI 2024. [5] Röpke et al., Divide and Conquer: Provably Unveiling the Pareto Front with Multi-Objective Reinforcement Learning. AAMAS 2025.
Questions for authors
Please refer to Weaknesses.
Dear Reviewers,
The authors have provided rebuttals to your review. Could you please check their responses and provide any further feedback or questions for the authors to address at your convenience?
Thank you for your time and contribution.
Regards,
AC