acceptodds
Under review as a conference paper at ICLR 2027

A Deep Reinforcement Learning-Assisted Operator Selection for Large-Scale Multi-Objective Optimization

Abstract

Large-scale multi-objective optimization problems (LSMOPs) pose significant challenges to conventional multi-objective evolutionary algorithms due to the exponential search space from the high-dimensional decision variables. Recently, adaptive operator selection by deep reinforcement learning (DRL) has become a prominent trend for solving LSMOPs. However, existing methods rely too heavily on conventional evolutionary operators and neglect specialized operators designed for large-scale problems, severely jeopardizing their performance. To address this limitation, this paper proposes a multi-objective evolutionary algorithm with DRL-assisted operator selection for LSMOPs, termed DRLOS-LSMOEA. First, a heterogeneous operator pool is constructed by integrating four conventional evolutionary operators with four large-scale dedicated operators, which substantially enriches the action space of the DRL agent. Second, a compact state representation built upon convergence and diversity is employed, alongside an adaptively normalized reward function. This design captures evolutionary dynamics and alleviates the curse of dimensionality, enabling effective training of the DRL model. Furthermore, a dimension control mechanism is proposed to dynamically regulate the decision variables that conventional evolutionary operators can conduct on in the later stage of evolution, thereby adaptively transitioning the search focus from global exploration to local exploitation. Extensive experiments on the LSMOP and UF benchmark suites demonstrate that DRLOS-LSMOEA significantly outperforms six state-of-the-art baselines across various test instances. The source code is available at https://github.com/XJFCD/DRLOS_LSMOEA.

Primary area:
optimization
Keywords:
Multi-Objective Optimization, Deep Reinforcement Learning, Operator Selection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3wk7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DRLOS-LSMOEA, a deep reinforcement learning-assisted evolutionary algorithm for large scale multiobjective optimization. Its main idea is to expand the operator pool by combining four conventional evolutionary operators with four offspring generation procedures derived from existing optimization algorithms. A DQN selects one operator at each generation using a two-dimensional population state based on convergence and diversity. The reward combines adaptively scaled improvements in these two measures. Additionally, a dimension control mechanism restricts conventional operators to subsets of decision variables during later search stages to promote local exploitation.

Critical strengths
  1. The paper is generally well written and easy to follow. The framework and its main components are clearly explained.
  2. The experiments cover two established benchmark suites and decision dimensions ranging from 500 to 8000, with repeated runs and statistical significance tests. The component ablations and parameter analyses provide additional empirical support.
  3. Treating offspring-generation procedures from existing algorithms as selectable operators provides a practical way to combine complementary search mechanisms within a unified framework.
Critical weaknesses
  1. The paper explains the motivation for selecting the large-scale dedicated operators, but does not sufficiently establish why this four large-scale dedicated operators combination is needed. Removing operators one at a time, testing smaller or larger subsets, or replacing individual operators would help assess sensitivity to the composition and size of the pool. These experiments would also clarify whether most of the gains come from one or two strong operators.
  2. The experiments do not include LMOEA-LGCS, RLDS-LSMOEA, or LSDMC, although these methods are discussed in the related work. Please compare against these closely related methods or explain why such comparisons are not feasible. Since the proposed pool incorporates the offspring-generation procedure of LMOEA-LGCS, a comparison with the full algorithm would be particularly useful for assessing the added benefit of the proposed scheduling framework.
  3. The ablation study removes the large-scale dedicated operators LS and DCM separately, but does not test a variant with both removed. Adding this configuration would help distinguish the individual contributions of the two components and examine their interaction.
  4. The reward function assigns weights of 0.7 and 0.3 to convergence and diversity improvements, respectively, but the basis for this choice is unclear. Please explain how these weights were chosen and provide a sensitivity analysis to determine whether the reported gains depend on this particular setting.
Questions for authors
  1. The paper explains the motivation for selecting the large-scale dedicated operators, but does not sufficiently establish why this four large-scale dedicated operators combination is needed. Removing operators one at a time, testing smaller or larger subsets, or replacing individual operators would help assess sensitivity to the composition and size of the pool. These experiments would also clarify whether most of the gains come from one or two strong operators.
  2. The experiments do not include LMOEA-LGCS, RLDS-LSMOEA, or LSDMC, although these methods are discussed in the related work. Please compare against these closely related methods or explain why such comparisons are not feasible. Since the proposed pool incorporates the offspring-generation procedure of LMOEA-LGCS, a comparison with the full algorithm would be particularly useful for assessing the added benefit of the proposed scheduling framework.
  3. The ablation study removes the large-scale dedicated operators LS and DCM separately, but does not test a variant with both removed. Adding this configuration would help distinguish the individual contributions of the two components and examine their interaction.
  4. The reward function assigns weights of 0.7 and 0.3 to convergence and diversity improvements, respectively, but the basis for this choice is unclear. Please explain how these weights were chosen and provide a sensitivity analysis to determine whether the reported gains depend on this particular setting.
4 comments
AuthorsOpenReview ↗

Response(1/4)

e thank you for the detailed review. We have added ablation experiments on variants with single large-scale operator, variant with the DQN-only, variants with different reward weights, and comparisons with the LMOEA-LGCS.

1. Why these four large-scale operators?

We ran experiments with four variants that keep the DQN, the DCM, and the four conventional operators, and retain exactly one large-scale operator (competitive learning, LMOEA-DS, MOPSO, or two-direction sampling). On the 18 LSMOP instances at D = 500 (M ∈ {2,3}, 20 runs each), every variant is significantly worse than the complete DRLOS-LSMOEA on the great majority of the instances (Wilcoxon rank-sum, p < 0.05), and only two isolated significant wins occur across all four variants on both metrics (competitive learning on one instance by IGD and LMOEA-DS on one instance by HV). No single large-scale operator recovers the performance of the heterogeneous pool. This indicates that the gains come from the synergy of all these operators instead of one super operator. Please refer to Appendix C.8 (Tables S.50–S.53) for more details.

Table S.52: IGD results of variants with single large-scale operator and DRLOS-LSMOEA on 3-objective LSMOPs (D = 500) (For 2-objective LSMOPs, please refer to Response(2/3) to VZ6S)

ProblemMDOnly CLOnly LMOEA-DSOnly MOPSOOnly TDSDRLOS-LSMOEA
LSMOP135003.1521e-1 (5.95e-2) -3.2769e-1 (1.27e-1) -1.1017e+0 (1.11e-1) -5.5316e-1 (1.73e-1) -1.7559e-1 (6.78e-2)
LSMOP235004.2211e-2 (7.08e-4) -3.7841e-2 (1.97e-3) =4.4283e-2 (2.76e-3) -4.5443e-2 (1.91e-3) -3.8294e-2 (1.35e-3)
LSMOP335004.8367e+0 (2.32e+0) -8.3712e-1 (3.89e-2) -1.4870e+1 (4.14e+0) -8.3477e-1 (4.11e-2) -8.1085e-1 (4.41e-2)
LSMOP435001.1836e-1 (7.86e-3) -8.9386e-2 (6.15e-3) -1.4262e-1 (3.31e-3) -9.1955e-2 (7.73e-3) -7.6111e-2 (5.82e-3)
LSMOP535007.6042e-1 (3.44e-1) -5.2080e-1 (1.91e-2) -3.1267e+0 (4.09e-1) -5.2819e-1 (2.41e-2) -4.2864e-1 (4.85e-2)
LSMOP635007.9628e+0 (1.02e+1) -6.7224e-1 (5.67e-2) -2.5887e+2 (1.05e+2) -1.2336e+0 (1.67e-1) -6.2284e-1 (4.45e-2)
LSMOP735001.0807e+3 (1.61e+3) -8.7330e-1 (1.29e-2) =1.9853e+3 (2.58e+3) -8.7034e-1 (1.60e-2) =8.7152e-1 (6.56e-3)
LSMOP835006.1646e-1 (1.27e-1) -2.6965e-1 (3.63e-2) -8.9346e-1 (1.28e-1) -3.2845e-1 (3.77e-2) -8.5860e-2 (4.97e-2)
LSMOP935001.0799e+1 (9.66e-1) -5.8730e-1 (1.11e-3) =8.6331e+0 (1.01e+0) -6.7197e-1 (2.41e-1) -5.8732e-1 (1.58e-3)
+/-/=0/9/00/6/30/9/00/8/1

This is consistent with Appendix C.3 (Table S.9 and Fig. S.2), which shows that each operator is selected at a non-trivial rate. In the late search stage, the selection probability of competitive learning, two-direction sampling, and LMOEA-DS are all significantly above the random baseline, and the conventional SBX and PM fall significantly below it. No operator approaches a dominating share. The reward analysis (Table S.10) also shows distinct utility. Four large-scale operators gain positive average rewards, while most conventional operators gain negative ones, and statistically significant differences are observed within the large-scale operator group. These show that the heterogeneous pool can provide complementary search behaviours, and the DQN can dynamically exploit them at different stages. Moreover, the DCM then confines conventional operators to variable subgroups late in the search, which prevents them from undoing the progress of the directional large-scale operators.

AuthorsOpenReview ↗

Response(2/4)

2. LMOEA-LGCS, RLDS-LSMOEA, and LSDMC.

We added comparisons with the LMOEA-LGCS on the LSMOP and UF benchmarks (LSMOP1–9 and UF1–10, M ∈ {2,3}, 20 runs each). The comparison results are reported in Appendix C.7 (Tables S.35–S.41), including a summary in Table S.41 (Please refer to Response(2/3) to VZ6S). The two methods have complementary strengths, i.e., LMOEA-LGCS wins on a subset of the LSMOP problems, with the largest gaps on LSMOP1, LSMOP8 and LSMOP9, while the proposed method is markedly better on the UF suite. Specifically, in terms of IGD, DRLOS-LSMOEA is significantly better on 68, significantly worse on 51, and comparable on 21 instances, respectively, than LMOEA-LGCS. From the HV perspective, these counts are 64, 43, and 33, respectively. In summary, the proposed DRLOS-LSMOEA performs slightly better than the LMOEA-LGCS on the 140 tested instances. However, the LMOEA-LGCS shows advantages in the LSMOP benchmarks. Notably, the results in Appendix C.8-3) show that none of the versions with single large-scale operator can beat the complete DRLOS-LSMOEA, indicating that the two-direction sampling strategy in LMOEA-LGCS may not act as the key contributor for its good behaviors. However, we only adopt its first stage, i.e., the two-direction sampling, which may explain why our algorithm underperforms it on the LSMOP benchmark suite. Please refer to Appendix C.7 for more details.

For RLDS-LSMOEA and LSDMC, as far as we know, there are no public implementations available, and it could be time-consuming to reproduce these codes. Thus, we have to admit that a faithful comparison is not possible at this time. But we have conducted comparisons against their underlying search components (LMOEA-DS and competitive learning), which are both stand-alone baselines and members of our pool.

AuthorsOpenReview ↗

Response(3/4)

3. Ablation with both large-scale operators and the DCM removed.

We have added experiments on the variant with the DQN retained and both the four large-scale operators and the DCM removed in Appendix C.8 (Tables S.46–S.49). Results show that on 90 LSMOP instances, it is significantly worse than the complete DRLOS-LSMOEA on all 90 instances by IGD and on 79 of 90 by HV (11 ties, no wins). The mean IGD grows from about two orders of magnitude above the complete DRLOS-LSMOEA at D = 500 to more than three orders at D = 8000. Combined with the existing ablations (w/o LS: significantly worse on 90/90 instances; w/o DCM: 51/90 by IGD), the individual contributions are clearly differentiated. Large-scale operators deliver the primary performance gain, DCM stabilizes the search, while removing both is significantly worse than the complete method on all 90 instances, confirming that the DQN cannot compensate for the loss of either component. Please refer to the Appendix C.8 (Tables S.46–S.49) for more details.

Table S.46: IGD results of the DQN-only variant and DRLOS-LSMOEA on 2-objective LSMOPs

ProblemDDQN-onlyDRLOS-LSMOEA
LSMOP15008.7575e-1 (1.57e-1) -1.9131e-1 (3.91e-2)
LSMOP110001.4201e+0 (1.61e-1) -1.5907e-1 (5.22e-2)
LSMOP120001.8739e+0 (1.49e-1) -1.6144e-1 (6.00e-2)
LSMOP150002.4002e+0 (1.55e-1) -1.6605e-1 (5.49e-2)
LSMOP180002.7899e+0 (2.77e-1) -1.5976e-1 (3.61e-2)
LSMOP25004.9364e-2 (9.96e-4) -4.3299e-3 (7.44e-4)
LSMOP210003.0255e-2 (6.82e-4) -2.8453e-3 (3.62e-4)
LSMOP220002.0691e-2 (3.31e-4) -2.0896e-3 (1.34e-4)
LSMOP250009.7271e-3 (1.04e-4) -1.6625e-3 (9.73e-5)
LSMOP280006.7350e-3 (7.32e-5) -1.5761e-3 (6.36e-5)
LSMOP35001.9003e+1 (1.33e+0) -1.5627e+0 (4.20e-3)
LSMOP310002.4359e+1 (1.03e+0) -1.5666e+0 (6.55e-3)
LSMOP320002.7709e+1 (8.20e-1) -1.5688e+0 (8.12e-3)
LSMOP350002.9919e+1 (4.99e-1) -1.5733e+0 (4.67e-3)
LSMOP380003.0334e+1 (4.62e-1) -1.5732e+0 (6.58e-3)
LSMOP45006.7254e-2 (4.05e-3) -2.1180e-2 (4.18e-3)
LSMOP410005.0751e-2 (9.42e-4) -1.3549e-2 (1.60e-3)
LSMOP420003.2322e-2 (2.77e-4) -7.3519e-3 (6.89e-4)
LSMOP450001.6851e-2 (2.45e-4) -3.9211e-3 (2.89e-4)
LSMOP480001.2418e-2 (3.02e-4) -2.8881e-3 (2.54e-4)
LSMOP55001.7815e+0 (1.92e-1) -6.3878e-1 (1.49e-1)
LSMOP510003.4723e+0 (5.34e-1) -6.9313e-1 (1.29e-1)
LSMOP520004.6138e+0 (6.01e-1) -6.8549e-1 (1.48e-1)
LSMOP550005.9489e+0 (6.87e-1) -7.2992e-1 (3.59e-2)
LSMOP580006.5747e+0 (1.30e+0) -6.8775e-1 (1.47e-1)
LSMOP65007.6168e-1 (5.49e-2) -2.5170e-1 (7.36e-2)
LSMOP610007.4983e-1 (6.02e-2) -2.8013e-1 (4.36e-2)
LSMOP620007.5328e-1 (7.35e-4) -2.5967e-1 (5.46e-2)
LSMOP650007.4671e-1 (2.24e-4) -2.2744e-1 (6.50e-2)
LSMOP680007.4496e-1 (2.35e-4) -2.4910e-1 (5.97e-2)
LSMOP75006.5223e+2 (1.53e+2) -1.5026e+0 (8.73e-4)
LSMOP710002.3593e+3 (7.65e+2) -1.5107e+0 (4.03e-4)
LSMOP720004.0273e+3 (8.16e+2) -1.5141e+0 (4.71e-4)
LSMOP750006.8654e+3 (2.49e+3) -1.4879e+0 (1.26e-1)
LSMOP780008.5633e+3 (4.12e+3) -1.5165e+0 (4.00e-4)
LSMOP85001.5881e+0 (2.48e-1) -4.9898e-1 (2.10e-1)
LSMOP810002.6583e+0 (2.32e-1) -5.1317e-1 (2.05e-1)
LSMOP820003.6650e+0 (3.31e-1) -5.3692e-1 (2.11e-1)
LSMOP850004.4150e+0 (6.97e-1) -4.0872e-1 (2.13e-1)
LSMOP880004.9078e+0 (6.41e-1) -5.0793e-1 (1.90e-1)
LSMOP95001.2992e+0 (8.31e-2) -4.5978e-1 (1.92e-1)
LSMOP910003.9692e+0 (1.12e+0) -4.7902e-1 (1.45e-1)
LSMOP920007.9143e+0 (9.05e-1) -4.7058e-1 (1.54e-1)
LSMOP950008.2044e+0 (9.97e-1) -4.2479e-1 (1.71e-1)
LSMOP980008.5450e+0 (1.26e+0) -4.0674e-1 (1.58e-1)
+/-/=0/45/0
AuthorsOpenReview ↗

Response(4/4)

4. Reward weights.

We have added ablation experiments on more reward weights settings 0.9/0.1, 0.3/0.7, and 0.1/0.9 on the same 10 LSMOP instances used in Appendix C.4 (i.e., LSMOP1, 4, 6, 8, 9; M ∈ {2,3}; D = 500; 20 runs per instance) in Appendix C.9 (Tables S.54–S.57). Results show that none of these three alternatives is ever significantly better than the default setting, i.e., 0.7/0.3. The 0.3/0.7, 0.9/0.1, and 0.1/0.9 settings are significantly worse on 1, 1 and 2 of the 10 instances in terms of IGD, and on 1, 2 and 3 in terms of HV, and statistically indistinguishable from the default on the rest. The weights 0.7/0.3 is therefore not a knife-edge choice, and the larger weights on the diversity reward hurt performance on a few instances with no compensating gain elsewhere. Overall, the current weights, i.e., 0.7/0.3, for the convergence and diversity in the reward constitutes a reasonably sound choice. Please refer to Appendix C.9 (Tables S.54–S.57) for more details.

Table S.54: IGD results of DRLOS-LSMOEA with different reward weights on 2-objective LSMOPs

ProblemMD0.9/0.10.9/0.10.3/0.70.3/0.70.1/0.90.1/0.90.7/0.30.7/0.3 (default)
LSMOP125001.6921e-1 (5.30e-2) =1.8879e-1 (5.27e-2) =1.8872e-1 (4.98e-2) =1.9131e-1 (3.91e-2)
LSMOP425002.3913e-2 (3.66e-3) -2.3587e-2 (2.95e-3) =2.4947e-2 (3.14e-3) -2.1180e-2 (4.18e-3)
LSMOP625002.6823e-1 (4.83e-2) =2.8756e-1 (3.39e-2) =2.8429e-1 (3.56e-2) =2.5170e-1 (7.36e-2)
LSMOP825004.7685e-1 (1.76e-1) =5.1755e-1 (1.84e-1) =4.5215e-1 (1.94e-1) =4.9898e-1 (2.10e-1)
LSMOP925004.9761e-1 (1.91e-1) =5.6545e-1 (9.93e-2) -5.9407e-1 (9.83e-2) -4.5978e-1 (1.92e-1)
+/-/=0/1/40/1/40/2/3

Table S.55: IGD results of DRLOS-LSMOEA with different reward weights on 3-objective LSMOPs

ProblemMD0.9/0.10.9/0.10.3/0.70.3/0.70.1/0.90.1/0.90.7/0.30.7/0.3 (default)
LSMOP135001.8308e-1 (6.07e-2) =1.8992e-1 (6.91e-2) =1.8294e-1 (7.40e-2) =1.7559e-1 (6.78e-2)
LSMOP435007.4644e-2 (4.87e-3) =7.7659e-2 (7.91e-3) =7.7918e-2 (4.17e-3) =7.6111e-2 (5.82e-3)
LSMOP635006.4044e-1 (5.80e-2) =6.2729e-1 (2.79e-2) =6.3808e-1 (4.50e-2) =6.2284e-1 (4.45e-2)
LSMOP835007.7035e-2 (4.23e-2) =8.6457e-2 (6.24e-2) =8.7991e-2 (4.44e-2) =8.5860e-2 (4.97e-2)
LSMOP935005.8754e-1 (1.12e-3) =5.8732e-1 (9.97e-4) =5.8756e-1 (1.02e-3) =5.8732e-1 (1.58e-3)
+/-/=0/0/50/0/50/0/5

We hope these experiments and clarifications address your concerns, and would be grateful if you would reconsider your assessment.

Reviewer_fTjEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DRLOS-LSMOEA for large-scale multi-objective optimization. The core idea is to let a DQN choose from eight offspring-generation operators: four conventional evolutionary operators and four operators adapted from existing large-scale methods. The policy uses a two-feature state describing convergence and diversity, with a normalized reward based on changes in those features. A dimension control mechanism limits the variables modified by conventional operators later in the run. The authors evaluate the method on the LSMOP and UF suites with 500 to 8000 decision variables, report IGD and HV against six baselines, and include ablations for the specialized operators and dimension control.

Critical strengths
  1. The heterogeneous operator pool is a practical and well-motivated design choice. Incorporating complete offspring-generation procedures from established large-scale optimizers gives the selector meaningfully different search behaviors. The ablation study shows a consistent loss across the reported test instances, making this contribution especially persuasive.

  2. The experimental coverage is substantial. Results span two benchmark suites, two and three objectives, and a broad range of decision dimensions. Reporting both IGD and HV, together with repeated runs and component ablations, makes the overall performance claim easier to assess. The dimension control ablation also helps explain the gains beyond the expanded operator pool.

Critical weaknesses
  1. The experiments establish that the full system works well, but it seems the authors do not clearly isolate the benefit of learned selection from the benefit of having a strong operator pool. A uniform-random selector and, if space permits, a simple fixed mixture using the same eight operators, environmental selection, dimension control, and evaluation budget would make this point much clearer. A small representative subset of problems would already be informative. This would strengthen the paper without changing its main method.

  2. The DQN action policy needs one clarification. Appendix B.1 defines ε = 0.95 and says an operator is selected at random with probability ε. If ε remains fixed, the learned choice is used in only 5% of post-warm-up decisions. Please confirm whether ε is an exploration or exploitation probability, state any schedule used, and reconcile this description with the nonuniform selection frequencies in Appendix C.3. This looks straightforward to resolve, but it matters for interpreting how much adaptation the DQN provides.

Questions for authors

Please see the Weaknesses.

4 comments
AuthorsOpenReview ↗

Response(1/2)

We appreciate your careful reading and encouraging remarks on our ablation design and experimental coverage. Your comments on isolating the contribution of the learning-based selector and clarifying the ε schedule of DQN are very helpful. We have updated the manuscript and supplemented supporting experiments accordingly, as detailed in the following responses.

1. Isolating the benefit of learned selection from a strong pool.

We have added the new ablation experiment as you suggest in Appendix C.8 (Tables S.42–S.45). A variant that replaces the DQN with a uniform-random selector is implemented and compared with the original DRLOS-LSMOEA. Over the 90 LSMOP instances (LSMOP1–9, M ∈ {2,3}, D ∈ {500, 1000, 2000, 5000, 8000}, 20 runs each), the proposed method is significantly better on 16 instances by IGD and never significantly worse than the ablation variant. The differences are modest, yet they consistently favor the learned policy in terms of IGD. This aligns with the non-uniform, stage-dependent selection frequencies reported in Appendix C.3, i.e., a static selector cannot dynamically reallocate the budget across operators as the search progresses. Please refer to Appendix C.8 (Tables S.42–S.45) for more details.

Table S.44: IGD results of DRLOS-LSMOEA-Random and DRLOS-LSMOEA on 3-objective LSMOPs (For 2-objective LSMOPs please refer to Response (3/3) to reviewer VZ6S)

ProblemDDRLOS-LSMOEA-RandomDRLOS-LSMOEA
LSMOP15001.7006e-1 (4.65e-2) =1.7559e-1 (6.78e-2)
LSMOP110001.6473e-1 (5.22e-2) =1.6448e-1 (6.28e-2)
LSMOP120001.6250e-1 (4.36e-2) =1.5163e-1 (5.15e-2)
LSMOP150001.8116e-1 (9.56e-2) =1.6485e-1 (6.02e-2)
LSMOP180001.6206e-1 (5.04e-2) =1.6634e-1 (5.31e-2)
LSMOP25004.0727e-2 (1.25e-3) -3.8294e-2 (1.35e-3)
LSMOP210003.2270e-2 (7.14e-4) -3.0998e-2 (8.27e-4)
LSMOP220002.7231e-2 (3.17e-4) -2.6694e-2 (4.33e-4)
LSMOP250002.4205e-2 (2.47e-4) -2.3907e-2 (3.08e-4)
LSMOP280002.3654e-2 (2.01e-4) -2.3488e-2 (2.18e-4)
LSMOP35008.2103e-1 (4.11e-2) =8.1085e-1 (4.41e-2)
LSMOP310008.2595e-1 (4.22e-2) =8.1605e-1 (4.56e-2)
LSMOP320008.4074e-1 (2.77e-2) =8.2076e-1 (3.91e-2)
LSMOP350008.2671e-1 (3.64e-2) =8.2071e-1 (3.92e-2)
LSMOP380008.3979e-1 (3.15e-2) =8.3350e-1 (3.17e-2)
LSMOP45007.7373e-2 (2.99e-3) =7.6111e-2 (5.82e-3)
LSMOP410005.3544e-2 (2.17e-3) =5.2567e-2 (2.61e-3)
LSMOP420004.0438e-2 (1.04e-3) -3.8823e-2 (1.48e-3)
LSMOP450003.0806e-2 (5.41e-4) -2.9137e-2 (7.15e-4)
LSMOP480002.7214e-2 (4.10e-4) -2.6228e-2 (5.10e-4)
LSMOP55004.4172e-1 (4.12e-2) =4.2864e-1 (4.85e-2)
LSMOP510004.3989e-1 (4.59e-2) =4.4774e-1 (4.13e-2)
LSMOP520004.6111e-1 (3.36e-2) =4.5666e-1 (4.27e-2)
LSMOP550004.6126e-1 (3.68e-2) =4.3171e-1 (5.73e-2)
LSMOP580004.3851e-1 (3.58e-2) =4.5720e-1 (4.30e-2)
LSMOP65006.4472e-1 (4.32e-2) -6.2284e-1 (4.45e-2)
LSMOP610006.5507e-1 (1.10e-1) =6.2163e-1 (5.25e-2)
LSMOP620006.2898e-1 (4.31e-2) =6.0710e-1 (4.95e-2)
LSMOP650006.3377e-1 (7.85e-2) =6.2214e-1 (1.67e-1)
LSMOP680006.1699e-1 (5.40e-2) -5.8413e-1 (4.87e-2)
LSMOP75008.6936e-1 (1.49e-2) =8.7152e-1 (6.56e-3)
LSMOP710008.4533e-1 (1.93e-2) =8.4660e-1 (1.81e-2)
LSMOP720008.4079e-1 (2.76e-3) =8.3981e-1 (6.51e-3)
LSMOP750008.0285e-1 (1.10e-1) =8.1919e-1 (5.03e-2)
LSMOP780007.9900e-1 (1.08e-1) =8.1795e-1 (4.17e-2)
LSMOP85009.0459e-2 (4.87e-2) =8.5860e-2 (4.97e-2)
LSMOP810006.7947e-2 (3.27e-2) =8.0760e-2 (4.97e-2)
LSMOP820008.6293e-2 (5.44e-2) =7.4458e-2 (4.55e-2)
LSMOP850007.8401e-2 (4.87e-2) =8.0168e-2 (5.29e-2)
LSMOP880007.0499e-2 (4.02e-2) =9.0814e-2 (5.21e-2)
LSMOP95005.8690e-1 (1.45e-3) =5.8732e-1 (1.58e-3)
LSMOP910005.8322e-1 (2.85e-3) =5.8232e-1 (3.85e-3)
LSMOP920005.7769e-1 (5.87e-3) =5.7973e-1 (2.88e-3)
LSMOP950005.7339e-1 (9.08e-3) =5.7332e-1 (8.19e-3)
LSMOP980005.7311e-1 (9.55e-3) =5.6977e-1 (1.10e-2)
+/-/=0/10/35

In addition, we have also added ablation experiments on the DQN-only variant and variants with a single large-scale operator in Appendix C.8. All these results prove the necessity of the DQN, DCM, and the heterogeneous operator pool.

Thank you for the additional experiments on ablations and furthur clarifications. In the response for isolating the benefit of learned selection from a strong pool, it seems the authors make a mistake in the content "reported in Sec.~???". Besides, I suggest the authros present the added tables and figures directly in your response so the reviewer could more easily see what you have done in this period. Generally, I maintain my positive assessment as 3.

AuthorsOpenReview ↗

Response(2/2)

2. The ε convention.

We have fixed the wording error in Appendix B.1. In the implementation (inherited from DRLOS-EMCMO), ε = 0.95 is the probability of exploiting the learned policy: the operator with the maximum predicted Q-value is chosen with probability 0.95, and a random operator is uniformly selected with probability 0.05. In all experiments, ε is fixed and there is no decay schedule. The warm-up phase (first 20% of evaluations) is separate and uses pure random selection to fill the replay buffer. After the warm-up, 95% of decisions are greedy, which is what produces the clear non-uniform, stage-dependent selection frequencies in Appendix C.3 (Table S.9 and Fig. S.2), while the remaining 5% of random exploration ensures that no operator is starved.

We have corrected this description in Appendix B.1 and Section 3.2. No other hyperparameters are affected (γ = 0.9, replay buffer 200, batch size 20, network update every 20 generations).

We hope these experiments and clarifications address your concerns, and would be grateful if you would reconsider your assessment.

We greatly appreciate you for pointing out our typographical errors and for your valuable suggestions. We have included relevant experimental tables in our rebuttal.

Reviewer_VZ6SOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a multiobjective evolutionary algorithm with deep reinforcement learning-assisted operator selection. Experiments are conducted on two benchmark suites, and the proposed algorithm achieves better performance than the compared methods.

Critical strengths
  • The paper combines four conventional evolutionary operators and four large-scale dedicated operators into the operator pool, exploiting their complementary search capabilities.

  • The proposed method is straightforward and easy to understand. The experiments are relatively comprehensive, and the proposed algorithm achieves significantly better performance than the compared methods on the benchmark suites.

Critical weaknesses
  • Using deep reinforcement learning for operator selection is not a new idea in evolutionary computation. Several existing methods have already employed deep reinforcement learning for evolutionary operator selection.

  • The experimental validation is limited to two benchmark suites. No experiments on real-world problems are provided.

  • The baseline algorithms are relatively outdated. Comparisons with more recently proposed methods are lacking.

Questions for authors
  • The ablation study mainly removes all four large-scale dedicated operators together. What is the individual contribution of each operator?

  • Would the proposed algorithm still achieve superior performance if DQN were replaced with random selection, history-based adaptive selection, or other simple operator selection strategies? What advantages does the deep reinforcement learning approach provide over these alternatives?

3 comments
AuthorsOpenReview ↗

Response(1/3)

We thank you for your careful review and constructive remarks. We acknowledge your concerns about the novelty, experimental benchmarks, baselines, and ablation design. Corresponding revisions and new experiments have been added to the paper, and our detailed responses are provided below.

1. Is deep reinforcement learning for operator selection new?

We agree that DRL has been used for operator selection before in DRLOS-EMCMO, whose agent architecture we inherit, and by MOEA/D-DQN, which is included as a baseline. Our contribution is not limited in using the reinforcement learning for operator selection, the heterogeneous operator pool, decision problem, and selector solves distinguish our work from prior ones:

  • Heterogeneous action granularity. In prior DRL-based operator selection (including DRLOS-EMCMO and MOEA/D-DQN) all actions perturb the decision space in the same way; the action's effect is approximately stationary. In our pool, alongside conventional operators, we incorporate four large-scale operators, where large-scale operators update all D variables along estimated search directions, whereas conventional operators act, after 30% of the budget, only on an α = 0.3 fraction of variable groups through the DCM. The same action therefore has a different cost and effect depending on the evolutionary stage, and the agent has to learn when to allocate budget on full-space directional search versus localized refinement. This decision problem does not arise in homogeneous pools.

  • Reward design for heterogeneous operators. Because the two operator families produce improvements of very different magnitude and variance, the reward uses sliding-window normalized convergence and diversity gains rather than raw improvement. The one-way ANOVA reported in Appendix C.3 confirms that the resulting reward separates operator utilities.

  • The learned policy also matters. With the identical pool, DCM, environmental selection, and budget, replacing the DQN by uniform sampling is never significantly better, and removing both the large-scale operators and the DCM while keeping the DQN degrades performance on every instance tested (see Appendix C.8). The learned policy also outperforms the existing DRL-based operator-selection baseline MOEA/D-DQN on the large majority of the 140 instances.

2. Real-world experiments.

We have added experiments on the real-world TREE suite (time-varying ratio-error estimation of voltage transformers; M = 2, with D = 3000 on TREE1–TREE2, D = 6000 on TREE3–TREE5 and D = 12000 on TREE6) in Appendix C.6 (Tables S.32-S.34). Compared with DGEA, FDV, LERD, LMOCSO, LMOEA-DS, and MOEA/D-DQN, the proposed method attains the highest mean HV on TREE1, TREE2 and TREE5, and obtains close HV on TREE3 and TREE4 with the LMOEA-DS, which is slightly better on TREE3 than the DRLOS-LSMOEA. Similarly, the IGD and the statistics of the number of feasible solutions in the final population also prove the superiority of DRLOS-LSMOEA over other competitors. More details can be found in Appendix C.6.

Table S.33: HV

ProblemDGEAFDVLERDLMOCSOLMOEA-DSMOEA/D-DQNDRLOS-LSMOEA
TREE17.5679e-1 (9.09e-2) -7.3971e-1 (6.04e-3) -7.8462e-1 (1.65e-2) -8.0580e-1 (5.34e-3) -8.3695e-1 (7.13e-5) -7.8547e-1 (8.94e-3) -8.4603e-1 (1.91e-3)
TREE28.4115e-1 (2.18e-2) -7.6541e-1 (1.44e-2) -7.9814e-1 (1.68e-2) -8.1653e-1 (2.54e-3) -8.5004e-1 (1.14e-4) -7.9969e-1 (6.40e-3) -8.5589e-1 (3.66e-4)
TREE36.8783e-1 (2.17e-1) -n/a7.8139e-1 (1.94e-2) -n/a8.8742e-1 (1.56e-5) +n/a8.8735e-1 (7.58e-4)
TREE44.3777e-1 (3.71e-1) -n/a1.8838e-1 (1.38e-1) -n/a9.6425e-1 (1.01e-4) =0.0000e+0 (0.00e+0) =9.6415e-1 (3.34e-4)
TREE55.0223e-1 (2.85e-1) -n/a7.7282e-1 (1.48e-2) -7.2593e-1 (0.00e+0) =9.2728e-1 (1.17e-5) -n/a9.3249e-1 (2.47e-3)
+/-/=0/5/00/2/00/5/00/2/11/3/10/2/1

Table S.34: IGD

ProblemDGEAFDVLERDLMOCSOLMOEA-DSMOEA/D-DQNDRLOS-LSMOEA
TREE13.3130e+0 (4.31e+0) -2.1016e+0 (2.66e-2) -2.1319e+0 (1.07e-1) -2.7880e+0 (6.43e-2) -1.0701e+0 (2.32e-1) -2.8089e+0 (5.90e-2) -5.7858e-1 (2.28e-1)
TREE21.0783e+1 (1.60e-1) -1.1258e+1 (3.64e-1) -1.1174e+1 (2.21e-1) -1.0737e+1 (2.49e-2) -9.9276e+0 (2.04e+0) -1.0733e+1 (4.38e-2) -4.2033e-1 (2.15e-1)
TREE37.0177e+1 (1.92e+1) -n/a8.2775e+1 (1.03e-1) -n/a8.5417e+0 (2.53e+1) -n/a3.4931e-1 (5.74e-1)
TREE48.7364e+0 (2.07e+1) -n/a1.2612e+0 (3.05e-1) -n/a3.0059e-3 (8.83e-4) =2.3712e+0 (1.38e-1) =1.0477e-2 (1.81e-2)
TREE56.7669e+1 (4.91e+1) -n/a9.2661e+0 (4.23e-2) -9.2839e+0 (0.00e+0) =8.5369e+0 (1.26e+0) -n/a5.6182e-1 (2.38e-1)
+/-/=0/5/00/2/00/5/00/2/10/4/10/2/1
AuthorsOpenReview ↗

Response(2/3)

3. Baselines.

We added comparisons with the LMOEA-LGCS, whose first stage, i.e., a two neural network-driven direction sampling is used as a candidate in our operator pool. The complete comparison on all 140 instances is reported in Appendix C.7 (Tables S.35–S.41), including a summary in Table S.41. Overall, in terms of the IGD metric, the proposed method is significantly better on 68, significantly worse on 51, and tied on 21 instances, respectively. We acknowledge that the advantage of the proposed DRLOS-LSMOEA is mainly manifested in the UF suite, whereas LMOEA-LGCS is stronger on a number of LSMOP instances, especially on LSMOP1, LSMOP8, and LSMOP9. In summary, the proposed DRLOS-LSMOEA performs slightly better than the LMOEA-LGCS on the 140 tested instances. However, the LMOEA-LGCS shows advantages in the LSMOP benchmarks. Notably, the results in Appendix C.8-3) show that none of the versions with single large-scale operator can beat the complete DRLOS-LSMOEA, indicating that the two-direction sampling strategy in LMOEA-LGCS may not act as the key contributor for its good behaviors. However, we only adopt its first stage, i.e., the two-direction sampling, which may explain why our algorithm underperforms it on the LSMOP benchmark suite. We would be very grateful if you can take a look at the Appendix C.7 for more details of the comparison results and analyses.

Additionally, RLDS-LSMOEA and LSDMC (raised by Reviewer 3wk7) are also recent RL-based LSMOEAs. However, as far as we know, there are no public implementations available, and it could be time-consuming to reproduce these codes. Thus, we have to admit that a faithful comparison is not possible at this time. But we have conducted comparisons against their underlying search components (LMOEA-DS and competitive learning), which are both stand-alone baselines and members of our pool.

Table S.41: Summary of IGD and HV comparisons between DRLOS-LSMOEA and LMOEA-LGCS

ProblemMIGDHV
LSMOP220/16/915/15/15
LSMOP328/10/728/11/6
UF20/33/20/28/7
UF33/9/30/10/5
+/-/=51/68/2143/64/33

4. Individual contribution of each large-scale operator.

Appendix C.3 (Table S.9 and Figure S.2) shows that all eight operators are used throughout the evolution and that no single operator consistently dominates the selection. What we can tell from Figure S.2 is that in the late stages, the four large-scale operators account for about 63% of the selections, with three of them with selection probability significantly above the 12.5% random baseline (competitive learning, two-direction sampling, and LMOEA-DS). In contrast, the selection probability of most conventional operators fall below the random baseline late in the search. Moreover, the average rewards are positive for all four large-scale operators and mostly negative for the conventional ones, and post-hoc tests show significant differences even within the large-scale group (Appendix C.3, Tables S.9–S.11). The operators therefore play distinct, non-redundant roles.

In addition, the new ablation experiment in Appendix C.8 confirms that keeping the DQN, the DCM, and the four conventional operators but retaining only one large-scale operator significantly degrades performance. Thus the benefit comes from the synergy of all operators but not one super operator. Please see Appendix C.8 (Tables S.50–S.53) for more details.

Table S.50: IGD comparisons with DRLOS-LSMOEA.

ProblemMDOnly CLOnly LMOEA-DSOnly MOPSOOnly TDSDRLOS-LSMOEA
LSMOP125003.5816e-1 (6.96e-2) -2.8830e-1 (9.22e-2) -9.4471e-1 (1.19e-1) -4.4914e-1 (1.38e-1) -1.9131e-1 (3.91e-2)
LSMOP225001.4397e-2 (6.82e-3) -6.6021e-3 (1.56e-3) -4.1471e-2 (9.86e-4) -7.3385e-3 (3.16e-3) -4.3299e-3 (7.44e-4)
LSMOP325001.0719e+1 (2.49e+0) -1.5643e+0 (2.00e-3) =2.2097e+1 (1.79e+0) -1.5672e+0 (3.14e-3) -1.5627e+0 (4.20e-3)
LSMOP425004.2638e-2 (3.27e-3) -3.1407e-2 (2.53e-3) -7.4489e-2 (2.37e-3) -3.4123e-2 (6.52e-3) -2.1180e-2 (4.18e-3)
LSMOP525005.6523e-1 (1.86e-1) +7.4209e-1 (2.28e-16) -2.0089e+0 (2.78e-1) -7.4209e-1 (2.28e-16) -6.3878e-1 (1.49e-1)
LSMOP625005.3651e-1 (1.43e-1) -3.0542e-1 (1.05e-2) -7.8113e-1 (3.97e-3) -5.8251e-1 (1.53e-1) -2.5170e-1 (7.36e-2)
LSMOP725003.8344e+1 (3.53e+1) -1.5034e+0 (9.07e-4) -6.8310e+2 (1.61e+2) -1.5037e+0 (2.19e-3) =1.5026e+0 (8.73e-4)
LSMOP825004.4169e-1 (2.47e-1) =6.4191e-1 (1.54e-1) -1.8496e+0 (1.98e-1) -7.2751e-1 (3.56e-2) -4.9898e-1 (2.10e-1)
LSMOP925001.3291e+0 (3.49e-1) -6.4546e-1 (1.38e-1) -2.9172e+0 (5.80e-1) -8.1004e-1 (1.14e-16) -4.5978e-1 (1.92e-1)
+/-/=1/7/10/8/10/9/00/8/1
AuthorsOpenReview ↗

Response (3/3)

5. Random or history-based selection versus DQN.

We have added ablation experiments with a variant that replaces the DQN with a uniform-random selector while keeping all other components untouched in Appendix C.8 (Tables S.42–S.45). In terms of the IGD metric, the DQN variant is significantly better than the variant with the random selector on 16 of the 90 instances and is never significantly worse, and by HV the differences are mixed (13 better, 4 worse, 73 ties). Although the superiority is not that significant, it still reflects that the learned policy is better than the random selection strategy. The complete results and analyses are provided in Appendix C.8 (Tables S.42–S.45).

We did not implement a bandit baseline for the history-based selection. We note that (i) the random selector is the history-free extreme, (ii) MOEA/D-DQN is an adaptive, history-dependent selection scheme and is significantly outperformed, and (iii) the stage-dependent shift in Fig. S.2 (near-uniform early, large-scale favoring late) is driven by the population state and cannot be reproduced by an operator-frequency rule that ignores the state.

Table S.42: IGD results of DRLOS-LSMOEA-Random and DRLOS-LSMOEA on 2-objective LSMOPs

ProblemDDRLOS-LSMOEA-RandomDRLOS-LSMOEA
LSMOP15001.7908e-1 (5.00e-2) =1.9131e-1 (3.91e-2)
LSMOP110001.6495e-1 (4.89e-2) =1.5907e-1 (5.22e-2)
LSMOP120001.5584e-1 (5.62e-2) =1.6144e-1 (6.00e-2)
LSMOP150001.5834e-1 (6.07e-2) =1.6605e-1 (5.49e-2)
LSMOP180001.4008e-1 (5.94e-2) =1.5976e-1 (3.61e-2)
LSMOP25004.4062e-3 (8.13e-4) =4.3299e-3 (7.44e-4)
LSMOP210002.9510e-3 (4.18e-4) =2.8453e-3 (3.62e-4)
LSMOP220002.2257e-3 (2.34e-4) -2.0896e-3 (1.34e-4)
LSMOP250001.7386e-3 (9.35e-5) -1.6625e-3 (9.73e-5)
LSMOP280001.6439e-3 (5.07e-5) -1.5761e-3 (6.36e-5)
LSMOP35001.5621e+0 (3.38e-3) =1.5627e+0 (4.20e-3)
LSMOP310001.5699e+0 (2.71e-3) =1.5666e+0 (6.55e-3)
LSMOP320001.5703e+0 (5.59e-3) =1.5688e+0 (8.12e-3)
LSMOP350001.5723e+0 (5.28e-3) =1.5733e+0 (4.67e-3)
LSMOP380001.5699e+0 (1.15e-2) =1.5732e+0 (6.58e-3)
LSMOP45002.3394e-2 (4.20e-3) =2.1180e-2 (4.18e-3)
LSMOP410001.3373e-2 (1.68e-3) =1.3549e-2 (1.60e-3)
LSMOP420007.5183e-3 (8.71e-4) =7.3519e-3 (6.89e-4)
LSMOP450003.9600e-3 (3.08e-4) =3.9211e-3 (2.89e-4)
LSMOP480002.8935e-3 (1.26e-4) =2.8881e-3 (2.54e-4)
LSMOP55007.1358e-1 (5.47e-2) -6.3878e-1 (1.49e-1)
LSMOP510006.9842e-1 (1.27e-1) =6.9313e-1 (1.29e-1)
LSMOP520006.9304e-1 (1.08e-1) =6.8549e-1 (1.48e-1)
LSMOP550006.9972e-1 (9.22e-2) =7.2992e-1 (3.59e-2)
LSMOP580006.9218e-1 (1.31e-1) =6.8775e-1 (1.47e-1)
LSMOP65002.9378e-1 (2.95e-2) -2.5170e-1 (7.36e-2)
LSMOP610002.8232e-1 (3.72e-2) =2.8013e-1 (4.36e-2)
LSMOP620002.8478e-1 (3.08e-2) =2.5967e-1 (5.46e-2)
LSMOP650002.6588e-1 (5.03e-2) =2.2744e-1 (6.50e-2)
LSMOP680002.3769e-1 (6.55e-2) =2.4910e-1 (5.97e-2)
LSMOP75001.5034e+0 (7.99e-4) -1.5026e+0 (8.73e-4)
LSMOP710001.5105e+0 (4.45e-4) =1.5107e+0 (4.03e-4)
LSMOP720001.5141e+0 (4.29e-4) =1.5141e+0 (4.71e-4)
LSMOP750001.5116e+0 (1.93e-2) =1.4879e+0 (1.26e-1)
LSMOP780001.5165e+0 (5.70e-4) =1.5165e+0 (4.00e-4)
LSMOP85004.3197e-1 (1.80e-1) =4.9898e-1 (2.10e-1)
LSMOP810004.7546e-1 (2.03e-1) =5.1317e-1 (2.05e-1)
LSMOP820004.8936e-1 (1.99e-1) =5.3692e-1 (2.11e-1)
LSMOP850005.4279e-1 (2.09e-1) =4.0872e-1 (2.13e-1)
LSMOP880004.1773e-1 (1.60e-1) =5.0793e-1 (1.90e-1)
LSMOP95005.5049e-1 (1.15e-1) =4.5978e-1 (1.92e-1)
LSMOP910004.8487e-1 (1.26e-1) =4.7902e-1 (1.45e-1)
LSMOP920004.1763e-1 (1.74e-1) =4.7058e-1 (1.54e-1)
LSMOP950003.9660e-1 (1.81e-1) =4.2479e-1 (1.71e-1)
LSMOP980003.7184e-1 (1.76e-1) =4.0674e-1 (1.58e-1)
+/-/=0/6/39

We hope these experiments and clarifications address your concerns, and would be grateful if you would reconsider your assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.