acceptodds
Under review as a conference paper at ICLR 2027

Direct Pareto Optimization: Learning to rank for Multi-Objective Optimization

Abstract

Many real-world sequential design problems, such as molecule optimization and neural architecture search, require one to balance multiple conflicting objectives simultaneously. Prior approaches typically address this by decomposing the problem into a series of single-objective problems or optimizing auxiliary indicators such as hypervolume. While effective, these methods either do not directly optimize the quality of the solution set or are limited by the effectiveness of the indicators. In this work, we propose _Direct Pareto Optimization (DPaO)_, a framework that directly optimizes with respect to Pareto Dominance by casting the multi-objective sequential optimization task as a learning-to-rank problem. Specifically, we learn a scalar valued latent reward function from the _partial ordering_ induced by Pareto dominance, thereby transforming the _whole_ multi-objective problem into a single-objective one. We evaluate the proposed method in both synthetic and real-world tasks, where DPaO consistently ranks among top methods, regardless of the shape of the solution set or number of objectives. Furthermore, we empirically validate that the learned reward function aligns with the Pareto dominance relation.

Primary area:
reinforcement learning
Keywords:
multi-objective optimization, reinforcement learning, preference based reinforcement learning, learning-to-rank

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BeMoOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes DPaO, which turns multi-objective sequential optimization into a learning-to-rank problem. It builds rankings from Pareto dominance and uses a Plackett–Luce model to train a generative policy directly. Experiments compare DPaO with policy-based and GFlowNet baselines on synthetic sequence tasks, neural architecture search, DNA design, and molecule generation, using hypervolume and other quality and diversity metrics. Ablations examine the effects of tie-breaking, the ranking model, and the entropy coefficient.

Critical strengths
  • The method trains a generative policy directly from Pareto-based rankings, without choosing scalarization weights or a hypervolume reference point for training. This reduces task-specific choices and provides a useful alternative to decomposition-based approaches.

  • The experiments cover several domains with 10 seeds per setting, showing clear gains on Bigram-3D/4D and improved hypervolume on some DNA tasks. The repeated runs and ablations strengthen the evidence that the approach is effective on these tasks.

Critical weaknesses
  • Since ranking-based optimization is already established, the contribution depends on how LTR is adapted to Pareto optimization. However, Eq. 1 introduces preferences between some Pareto-incomparable solutions based on their domination counts, and these preferences can change with the batch composition. Respecting dominance does not justify these additional preferences. The paper does not sufficiently explain why learning them should improve Pareto-set quality, leaving its central motivation only partially supported.

  • The experiments do not establish a clear overall performance advantage. On Unigram-2D, DPaO achieves an HV of 0.16 ± 0.20, compared with 0.46 ± 0.03 for OP-GFN (Table 6). It also trails POCCO on the two-objective DARTS tasks. On molecule generation, its higher HV comes with worse IGD+ and R2 than MOGFN (Table 19). These results limit the evidence for a reliable improvement over existing methods.

Questions for authors
  • Could you clarify how DPaO relates to RaM [1] and Global-Order GFlowNets [2]? RaM uses LTR for offline optimization, while Global-Order GFlowNets address conflicts in local Pareto supervision. What additional challenge does DPaO solve, and how does its ranking construction differ from these approaches?

  • Eq. 1 uses domination counts to assign ranks, but Appendix D mentions non-dominated sorting. These are not generally equivalent. Which one was used in the experiments?

  • I am not sure why different domination counts should imply a preference between two Pareto-incomparable solutions. These preferences can also change depending on the other samples in the batch. What justifies learning them, and have you checked their effect on the resulting solution set?

  • When all samples in a batch are mutually non-dominated, Eq. 5 seems to give a likelihood of one and therefore no ranking gradient. Is this correct? If so, what encourages the policy to maintain coverage in this case?

  • Appendix H says that training uses 32,000 trajectories, while Figures 5 and 6 show roughly three million. Could you clarify how trajectories are counted and what budget was used? Were all methods given the same number of objective evaluations?

[1] Tan et al. Offline Model-Based Optimization by Learning to Rank. ICLR 2025.

[2] Pastor-Pérez et al. Global-Order GFlowNets. ICLR 2025 workshop.

5 comments
AuthorsOpenReview ↗

Comparison to existing methods

  1. Comparison to existing methods: Our paper focuses on multi-objective online optimization while RaM focuses on single-objective offline optimization. RaM considers single-objective, offline optimization, with the goal of improving performance of surrogate models with OOD samples. In contrast, we focus on multi-objective online optimization, with the goal of removing environment specific information and scalarization. Notably, the partial ordering, induced by the Pareto dominance relation, makes the learning task far more complicated than in the case of RaM. Global Order GFNs (GR-GFNs) use the same ranking as we do; However, they minimize the divergence between the learned reward and partial ordering, while we frame it as a LTR task. We performed experiments with GR-GFNs (our implementation, code was not available), and found it to underperform compared to DPaO (see table below). We have added mentions of GR-GFNs and RaM in related work. To restate: Our key insight is noting the clear correspondence between the ranking induced by the Pareto dominance, and partitioned preference used in LTR literature, which allows us to convert the whole multi-objective optimization task into a single-objective LTR task. Notably, we do this in an online case, where we have access only to batches of data, which makes the learning task even more challenging.

GR-GFN comparison in representative environments. Better performing method in each metric is bolded. We can provide full evaluation results for GR-GFN upon request.

EnvMethodHV (⇑\Uparrow))IGD+ (⇓\Downarrow)R2 (⇓\Downarrow)ϵ\epsilon-metric (⇓\Downarrow)PC entropy (⇑\Uparrow)
Unigram-2dGR-GFN0.02$\pm$0.010.64$\pm$0.048.47$\pm$0.370.85$\pm$0.040.60$\pm$0.35
DPaO (ours)0.16$\pm$0.200.35$\pm$0.195.64$\pm$1.400.76$\pm$0.300.67$\pm$0.76
Bigram-2dGR-GFN0.00$\pm$0.000.75$\pm$0.029.36$\pm$0.210.93$\pm$0.020.42$\pm$0.36
DPaO (ours)0.38$\pm$0.160.18$\pm$0.134.31$\pm$1.030.48$\pm$0.250.96$\pm$0.41
Bigram-4dGR-GFN0.00$\pm$0.000.58$\pm$0.018.65$\pm$0.100.94$\pm$0.001.47$\pm$0.34
DPaO (ours)0.04$\pm$0.010.10$\pm$0.034.31$\pm$0.200.62$\pm$0.092.70$\pm$0.44
DNA (Energy, length)GR-GFN0.13$\pm$0.145.85$\pm$0.94N/AN/AN/A
DPaO (ours)0.60$\pm$0.092.86$\pm$0.54N/AN/AN/A
DARTS (Err, Params, FLOPs)GR-GFN0.80$\pm$0.021.96$\pm$0.12N/AN/AN/A
DPaO (ours)0.94$\pm$0.011.25$\pm$0.03N/AN/AN/A

2. Relation between non-dominated sorting and dominance count: Our initial draft had a mistake, caused by our goal to specify the learning target in the most compact way. Indeed, we use the Pareto Ranking [1, 2] which can be computed by iteratively finding Pareto fronts from the dataset, always removing the currently found Pareto front from the dataset. We have updated the manuscript (Sec 4) with the correction, and included a pseudocode for computing Pareto Rank.

3. I am not sure why different domination counts should imply a preference between two Pareto-incomparable solutions? As noted above, our initial draft had a mistake; Indeed, we do not assume preference between two Pareto-incomparable solutions, and instead learn a ranking over the partial ordering induced by Pareto dominance.

4.What happens when all solutions in a batch are mutually non-dominated: When all points in a batch are non-dominated, there is no information to perform an effective update. This is an inherent disadvantage for all batch based update rules. While this is theoretically plausible scenario, we empirically demonstrate that this is not an issue in practice. Appendix D performs ablation over the batch size, showing that the number of partitions is always positive. Furthermore, the performance remains consistent with changing batch size.

5. Number of samples: The x-axis label for Figures 5 and 6 contained a typo (10^5 instead of 10^4), which has now been corrected. All methods are given 32 000 samples. We do not perform any caching of values, so each generated solution is removed from the total budget. This is standard practice followed by other baselines (e.g.[3, 4] ).

[1] K. Deb, et al., "A fast and elitist multiobjective genetic algorithm: NSGA-II," in IEEE Transactions on Evolutionary Computation, 2002

[2] Pastor-Pérez et al. Global-Order GFlowNets. ICLR 2025 workshop.

[3] Jain, M. et al. Multi-Objective GFlowNets. ICML 2023

[4] Chen, Y., & Mauch, L. Order-Preserving GFlowNets. ICLR 2024

Weakness 2:

  • We have performed additional experiments in molecular generation tasks, considering synthetic accessibility and molecular weight as additional objectives in addition to the LogP and QED objectives to better match the real-world molecule generation (Table 19). In this updated task, we obtain the best performance in HV, IGD+, R2 and ϵ\epsilon-metric.
  • We also fixed a mistake in DNA task normalization. After rerunning these experiments (Tables 16, 17, 18), we obtain the best performance in most cases (tied with MOGFN in 3D case for HV, but better in R2)
  • In 2D DARTS benchmarks, with near convex solution sets we obtain competitive performance with POCCO, while outperforming it in higher dimensional tasks with non-convex solution sets.

Table 16: DNA (Energy, Inv. Length):

MethodHV (⇑\Uparrow)R2 (⇓\Downarrow)
POCCO (L)0.36$\pm$0.074.44$\pm$0.65
POCCO (C)0.08$\pm$0.066.24$\pm$0.63
MOReinforce (L)0.12$\pm$0.016.45$\pm$0.36
MOReinforce (C)0.13$\pm$0.026.42$\pm$0.37
MOURex (L)0.11$\pm$0.115.98$\pm$0.85
MOUrex (C)0.05$\pm$0.036.47$\pm$0.17
CAPQL0.12$\pm$0.016.73$\pm$0.28
MOGFN0.33$\pm$0.054.40$\pm$0.58
OP-GFN0.41$\pm$0.133.79$\pm$0.89
DPaO (ours)0.60$\pm$0.092.86$\pm$0.54

Table 19: Small molecule optimization (LogP, QED, SA, MW)

MethodHV (⇑\Uparrow)IGD+ (⇓\Downarrow)R2 (⇓\Downarrow)ϵ\epsilon-metric (⇓\Downarrow)PC entropy (⇑\Uparrow)
POCCO (L)0.29$\pm$0.020.44$\pm$0.022.96$\pm$0.170.39$\pm$0.021.37$\pm$0.20
POCCO (C)0.24$\pm$0.020.50$\pm$0.033.27$\pm$0.190.46$\pm$0.031.14$\pm$0.29
MOReinforce (L)0.26$\pm$0.020.46$\pm$0.023.09$\pm$0.120.42$\pm$0.031.31$\pm$0.34
MOReinforce (C)0.27$\pm$0.020.46$\pm$0.023.05$\pm$0.160.42$\pm$0.031.28$\pm$0.35
MOURex (L)0.26$\pm$0.020.47$\pm$0.033.12$\pm$0.170.42$\pm$0.021.32$\pm$0.13
MOUrex (C)0.26$\pm$0.010.47$\pm$0.023.09$\pm$0.150.42$\pm$0.031.36$\pm$0.12
CAPQL0.21$\pm$0.050.52$\pm$0.053.43$\pm$0.460.48$\pm$0.061.15$\pm$0.30
MOGFN0.27$\pm$0.020.46$\pm$0.033.05$\pm$0.170.41$\pm$0.031.38$\pm$0.18
OP-GFN0.22$\pm$0.050.49$\pm$0.073.50$\pm$0.380.46$\pm$0.070.92$\pm$0.34
DPaO (ours)0.30$\pm$0.050.44$\pm$0.022.93$\pm$0.410.39$\pm$0.011.35$\pm$0.18
Reviewer_sf4qOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Direct Pareto Optimization (DPaO), which reformulates multi-objective sequential design as a problem of learning to rank within partitioned sets. The method constructs ordered partitions based on the multi-objective performance of candidate solutions—leaving the relative ordering of items within each partition unspecified—and trains a generative policy by maximizing the Plackett–Luce ranking likelihood. By leveraging the relationship between the maximum-entropy optimal policy and the underlying reward function, the authors parameterize a latent scalar reward in terms of the log-probabilities of the policy, thereby eliminating the need to train a separate reward model. The experiments cover tasks including Regex, neural architecture search, DNA sequence design, and small-molecule generation, utilizing ten random seeds and a variety of evaluation metrics. DPaO demonstrates superior performance in terms of Hypervolume (HV) on certain high-dimensional or irregular tasks, though this advantage does not extend to all tasks and metrics. In the small-molecule experiments—where the focus is solely on optimizing logP and QED—DPaO achieves a higher HV but exhibits significant runtime volatility; furthermore, no consistent improvement is observed across other advanced quality metrics.

Critical strengths
  1. Constructing preferences in the face of conflicting multi-objective goals is driven by clear motivations. This approach does not require the pre-specification of a fixed set of attribute weights; instead, it employs partitioned ranking to preserve certain unresolved trade-offs. Compared to methods that first scalarize objectives and then force a complete ordering, this approach reduces the extent to which subjective human choices influence the training signals.

  2. The experiments are extensive, examining a variety of ranking designs. The use of ten random seeds, multi-domain tasks, multiple metrics, and an ablation study on tie-breaking rules all support the empirical value of the proposed method. The appendix includes instances where DPaO underperforms on certain tasks or metrics, which helps to clarify the scope of its applicability.

Critical weaknesses
  1. The distinct contribution of this work—relative to existing DPO approaches and the practical implications of the mapping between rankings and probabilities—has not yet been sufficiently isolated. Standard DPO already leverages a reward-policy reparameterization to bypass the need for an explicit reward model, and the extension to Plackett–Luce list-based ranking is derived in Appendix A.3; furthermore, established methods already exist for calculating the likelihood of partitioned preferences. Consequently, the novelty of this paper must be specifically rooted in the construction of Pareto preferences, the handling of partitions, and their role in multi-objective generation—rather than simply in the direct optimization of policy probabilities or the adoption of list-based ranking.

  2. The molecular design experiments are not yet sufficient to confirm that a higher Hypervolume (HV) corresponds to better chemical candidates or superior coverage of the Pareto front. For the small-molecule tasks, only logP and QED are used. In Table 19, DPaO achieves an HV of 1.19±0.741.19 \pm 0.74, yet its IGD+ is 0.41±0.140.41 \pm 0.14 and R2 is 11.21±15.5811.21 \pm 15.58, indicating no consistent advantage. It remains to be determined whether this discrepancy stems from effective expansion of the Pareto front, the influence of a few extreme data points, or the inclusion of candidates that fall outside the normalization range. Furthermore, the appendix states that the reward values ​​are constrained to the interval [0,1][0, 1] with a reference point at zero; however, the reported HV values ​​exceed 1, a point that also requires clarification.

Questions for authors

1.The original DPO already supports ranking based on PL (Plackett-Luce) lists. Please compare the following—using the same architecture, preference data, and sampling budget: the pairwise preference loss (based solely on actual dominant pairs), the partitioned list loss proposed in this paper, and versions both with and without the reference policy. Which gains can be explicitly attributed to the new design presented in this paper?

2.Please clarify whether "potential reward" refers to a scalar utility implied by the policy's log-probabilities, rather than a continuous molecular embedding. Does the actual implementation utilize dominance counts or non-dominated layering (fronts)? Beyond positive rank correlation, what evidence supports the maximum-entropy interpretation and the coverage of multiple modes? Furthermore, how does the method derive effective updates when all solutions within a batch are mutually non-dominated?

3.Does xx in πθ(x)\pi_\theta(x) represent a SELFIES string or a canonicalized molecular graph? If multiple strings decode to the same molecule, is the aggregation of their probabilities—and the issue of overcounting—accounted for? Additionally, please examine whether sequence length and the complexity of the representation influence the correlation between policy probabilities and Pareto rankings.

4.Please provide scatter plots for each seed target, the original logP/QED ranges, and examples of extreme candidate structures. Also, include metrics for molecular weight, heavy atom count, number of unique molecules, and scaffold coverage. Do the observed advantages persist when evaluated based on an equal number of unique molecules? Please explain the significance of an HV (Hypervolume) value greater than 1 under normalized reward conditions, and clarify whether the conclusions remain stable when applying reasonable logP ranges or specific property constraints.

5.How do the approximately 32,000 training trajectories in Appendix H relate to the ~3 million trajectories shown in Figures 5 and 6? Please provide a breakdown of the number of generated sequences, distinct molecules, objective function evaluations, and update steps, and specify the caching rules used. Additionally, please clarify the direction, reduction method, and reference point for the Chebyshev scalarization, and explain why this baseline performs near zero across multiple tasks.

5 comments

1. Comparison to DPO: Compared to DPO, we consider the maximum entropy formulation, instead of KL-regularized case and thus we do not have a reference model. Furthermore, as noted in Sec 4, PL model (and consequently, BT model) assume total ordering, which we do not have in the case of MOO. Regardless, we have updated Sec 5.5 (Fig. 7) to include a naive BT variant that finds all the dominated pairwise relations and updates the BT model based on that. We find that in general, our proposed method DPaO, either matches or outperforms it, especially when the number of objectives increases, providing further evidence for our proposed formulation.

2. Relation between non-dominated sorting and dominance count: Our initial draft had a mistake, caused by our goal to specify the learning target in the most compact way. Indeed, we use the Pareto Ranking [1] which can be computed by iteratively finding Pareto fronts from the dataset, always removing the currently found Pareto front from the dataset. We have updated the manuscript (Sec 4) with the correction, and included a pseudocode for computing Pareto Rank.

3. Evidence that we have learned the multi-modal reward: We have added Appendix E, where we study the learned reward function in NB201 task, as its small solution space allows us to enumerate all states. We found that we match the real distribution with L1 distance 9.3×10−5±8.6×10−69.3\times 10^{-5} \pm 8.6\times 10^{-6}, (10 seeds), highlighting that DPaO can learn the true ranking distribution.We further visualize the learned reward functions both in state space (Fig 9) and reward space (Fig 10).

4. Update when all points in a batch are mutually non-dominated: When all points in a batch are non-dominated, there is no information to perform an effective update. This is an inherent disadvantage for all batch based update rules. While this is theoretically plausible scenario, we empirically demonstrate that this is not an issue in practice. Appendix D performs ablation over the batch size, showing that the number of partitions is always positive. Furthermore, the performance remains consistent with changing batch size.

5. SELFIES presentation: xx in Eq. 1 can represent any sequence, such as SELFIES string. Currently, we don’t take into account the fact that the multiple sequences (strings) may represent the same molecule. While this breaks our assumption of tree-like MDP structure (Sec 3.), we suspect this has limited effect on the learned latent reward due to the large scale of the search-space. Regardless, we see the integration of DPaO and Gflownets, which explicitly can take the overcounting into account, as a promising avenue for future work.

6. Normalization and HV values over 1: We noticed a bug in DNA normalization values, which we have now fixed. We have also added the exact normalization values in Appendix H. However, as these normalization values are approximations, it is possible to find solutions with properties exceeding these values, which we now highlight in Appendix H.

7. Number of samples: The x-axis label for Figures 5 and 6 contained a typo (10510^5 instead of 10410^4), which has now been corrected. All methods are given 32 000 samples. We do not perform any caching of values, so each generated solution is removed from the total budget. This is standard practice followed by other baselines (e.g. [2, 3]).

8. Chebyshev variants: We found a small bug in POCCO’s Chebyshev variant and updated all results (incl. Appendix P). Regardless, the Chebyshev variant of POCCO underperforms, which is inline with results presented in original paper [4]. Furthermore, it has been reported that the linear scalarization tends to perform better than Chebyshev in design tasks [2], a conclusion that is further supported by our results.

9. Zero hypervolume of baselines The REGEX tasks result in zero HV for most baselines, as finding solutions with positive rewards is challenging. For example, the chance of finding the optimal solution for a single-objective task of maximizing one letter (unigram task) with random policy is 1428\frac{1}{4}^{28}. With scalarization, one has to solve multiple of these challenging subproblems, making generalization difficult.

10. Scatter plots: we have added the missing scatter plots in Appendix K.

11. Molecule benchmark: We are currently performing additional experiments in the small-molecule benchmark, taking into account more statistics such as synthesizability score and molecular weight. We will update the manuscript as soon as these results are finalized.

[1] K. Deb, et al., "A fast and elitist multiobjective genetic algorithm: NSGA-II," in IEEE Transactions on Evolutionary Computation, 2002

[2] Jain, M. et al. Multi-Objective GFlowNets. ICML 2023

[3] Chen, Y., & Mauch, L. Order-Preserving GFlowNets. ICLR 2024

[4] Fan, M. et al Preference-Driven Multi-Objective Combinatorial Optimization with Conditional Computation. NeurIPS2025

AuthorsOpenReview ↗

Clarification of questions

We would kindly ask the reviewer to provide further details about the following questions to make sure we have understood them correctly:

  1. Please examine whether sequence length and the complexity of the representation influence the correlation between policy probabilities and Pareto rankings.

We would respectfully ask what is meant by “complexity of representation”. In most environments, the sequence length is either fixed (NAS) or has very limited variability (REGEX). Regardless, we normalize the log-probabilities with the sequence length to ensure that longer sequences do not get inadvertent priority. This is shown in Eq. 4 of the updated manuscript.

  1. Please clarify whether "potential reward" refers to a scalar utility implied by the policy's log-probabilities, rather than a continuous molecular embedding.

We would kindly ask for clarification on this misunderstanding. We do not use continuous molecular embeddings in this work as we focus specifically on discrete settings.

AuthorsOpenReview ↗

Additional experiments in molecular task

We have performed an additional experiment in the small molecule generation task. In addition to the existing logP and QED metrics, we also consider the synthetic accessibility (SA) score and Molecular weight (MW), resulting in 4d objective space. We follow [1, 2] and define these metrics as SA=(9−SA)/10SA = (9 - SA) / 10 and MW=clip((300−mw)/700+1,0,1)MW = \mathrm{clip}((300 - mw) / 700 + 1, 0, 1) where the former formulation prioritizes solutions that are easier to synthesize while latter focuses generating molecules with molecular weight under 300. As one can see from below table (also Table 19 in manuscript), DPaO achieves best HV, IGD+, R2 and ϵ\epsilon-metric under these new target objectives. As expected, the performance is slightly lower in PC-entropy, as we focus on finding solutions based on dominance relation, rather than focusing in diversity.

We furthermore display the statistics of the properties computed from the generated molecules in Appendix G and display the new solution sets in Fig. 23 and Fig 24 (Appendix L). Moreover, we have updated all results (Table 1, Fig 2, Fig 3) to match the new evaluations.

MethodHV (⇑\Uparrow)IGD+ (⇓\Downarrow)R2 (⇓\Downarrow)ϵ\epsilon-metric (⇓\Downarrow)PC entropy (⇑\Uparrow)
POCCO (L)0.29$\pm$0.020.44$\pm$0.022.96$\pm$0.170.39$\pm$0.021.37$\pm$0.20
POCCO (C)0.24$\pm$0.020.50$\pm$0.033.27$\pm$0.190.46$\pm$0.031.14$\pm$0.29
MOReinforce (L)0.26$\pm$0.020.46$\pm$0.023.09$\pm$0.120.42$\pm$0.031.31$\pm$0.34
MOReinforce (C)0.27$\pm$0.020.46$\pm$0.023.05$\pm$0.160.42$\pm$0.031.28$\pm$0.35
MOURex (L)0.26$\pm$0.020.47$\pm$0.033.12$\pm$0.170.42$\pm$0.021.32$\pm$0.13
MOUrex (C)0.26$\pm$0.010.47$\pm$0.023.09$\pm$0.150.42$\pm$0.031.36$\pm$0.12
CAPQL0.21$\pm$0.050.52$\pm$0.053.43$\pm$0.460.48$\pm$0.061.15$\pm$0.30
MOGFN0.27$\pm$0.020.46$\pm$0.033.05$\pm$0.170.41$\pm$0.031.38$\pm$0.18
OP-GFN0.22$\pm$0.050.49$\pm$0.073.50$\pm$0.380.46$\pm$0.070.92$\pm$0.34
DPaO (ours)0.30$\pm$0.050.44$\pm$0.022.93$\pm$0.410.39$\pm$0.011.35$\pm$0.18

[1] Jain, M. et al. Multi-Objective GFlowNets. ICML 2023

[2] Chen, Y., & Mauch, L. Order-Preserving GFlowNets. ICLR 2024

Reviewer_B28HOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Direct Pareto Optimization (DPaO), which formulates multi-objective sequential design as a learning-to-rank problem. It learns a latent scalar reward from Pareto-based partial rankings using a Plackett–Luce likelihood. The method is evaluated on synthetic sequence tasks, neural architecture search, molecular generation, and DNA sequence generation.

Critical strengths
  1. The formulation connects Pareto dominance with preference-based policy optimization, avoiding the need to specify scalarization weights.

  2. The evaluation covers several discrete design domains and reports results across ten random seeds.

Critical weaknesses
  1. The molecular experiments show improved hypervolume, but provide limited analysis of the chemical properties and plausibility of the candidates responsible for these gains. Their practical value for molecular design therefore remains unclear (Table 19; Figure 11).

  2. The DNA evaluation focuses on secondary-structure-related objectives, without establishing how improvements in these scores translate into useful aptamer properties, such as target-binding affinity or specificity.

  3. The supplied implementation differs from the paper’s formulation. For example, the training objective uses length-normalized sequence log-probabilities, whereas Eq. 5 uses sequence log-probabilities. This leaves the correspondence between the derivation and the implemented method unclear.

Questions for authors

See above.

2 comments

1. Molecular benchmark: We are currently performing additional experiments in the small-molecule benchmark, taking into account more statistics such as synthesizability score and molecular weight. We will update the manuscript as soon as these results are finalized.

2. DNA benchmark: We agree that the DNA benchmark may not use targets that are directly related to properties useful in the real-world. However, our goal is to describe a new general approach for multi-objective sequential optimization, instead of focusing specifically on DNA sequence optimization. Furthermore, this DNA benchmark is commonly used in related works [1-3], making the results between multiple papers and implementations comparable.

3. Difference between Eq 4 and code: In practice we normalize the log probabilities with the length of the sequence to ensure that longer sequences are not given inadvertent priority. This distinction does not matter in most environments, where the sequence length is fixed. We have updated the equation and thank the reviewer for pointing this inconsistency out.

[1] Jain, M. et al. Multi-Objective GFlowNets. ICML 2023

[2] Chen, Y., & Mauch, L. Order-Preserving GFlowNets. ICLR 2024

[3] Pastor-Pérez et al. Global-Order GFlowNets. ICLR 2025 workshop.

AuthorsOpenReview ↗

Additional experiments in molecular task

We have performed an additional experiment in the small molecule generation task. In addition to the existing logP and QED metrics, we also consider the synthetic accessibility (SA) score and Molecular weight (MW), resulting in 4d objective space. We follow [1, 2] and define these metrics as SA=(9−SA)/10SA = (9 - SA) / 10 and MW=clip((300−mw)/700+1,0,1)MW = \mathrm{clip}((300 - mw) / 700 + 1, 0, 1) where the former formulation prioritizes solutions that are easier to synthesize while latter focuses generating molecules with molecular weight under 300. As one can see from below table (also Table 19 in manuscript), DPaO achieves best HV, IGD+, R2 and ϵ\epsilon-metric under these new target objectives. As expected, the performance is slightly lower in PC-entropy, as we focus on finding solutions based on dominance relation, rather than focusing in diversity.

We furthermore display the statistics of the properties computed from the generated molecules in Appendix G and display the new solution sets in Fig. 23 and Fig 24 (Appendix L). Moreover, we have updated all results (Table 1, Fig 2, Fig 3) to match the new evaluations.

MethodHV (⇑\Uparrow)IGD+ (⇓\Downarrow)R2 (⇓\Downarrow)ϵ\epsilon-metric (⇓\Downarrow)PC entropy (⇑\Uparrow)
POCCO (L)0.29$\pm$0.020.44$\pm$0.022.96$\pm$0.170.39$\pm$0.021.37$\pm$0.20
POCCO (C)0.24$\pm$0.020.50$\pm$0.033.27$\pm$0.190.46$\pm$0.031.14$\pm$0.29
MOReinforce (L)0.26$\pm$0.020.46$\pm$0.023.09$\pm$0.120.42$\pm$0.031.31$\pm$0.34
MOReinforce (C)0.27$\pm$0.020.46$\pm$0.023.05$\pm$0.160.42$\pm$0.031.28$\pm$0.35
MOURex (L)0.26$\pm$0.020.47$\pm$0.033.12$\pm$0.170.42$\pm$0.021.32$\pm$0.13
MOUrex (C)0.26$\pm$0.010.47$\pm$0.023.09$\pm$0.150.42$\pm$0.031.36$\pm$0.12
CAPQL0.21$\pm$0.050.52$\pm$0.053.43$\pm$0.460.48$\pm$0.061.15$\pm$0.30
MOGFN0.27$\pm$0.020.46$\pm$0.033.05$\pm$0.170.41$\pm$0.031.38$\pm$0.18
OP-GFN0.22$\pm$0.050.49$\pm$0.073.50$\pm$0.380.46$\pm$0.070.92$\pm$0.34
DPaO (ours)0.30$\pm$0.050.44$\pm$0.022.93$\pm$0.410.39$\pm$0.011.35$\pm$0.18

[1] Jain, M. et al. Multi-Objective GFlowNets. ICML 2023

[2] Chen, Y., & Mauch, L. Order-Preserving GFlowNets. ICLR 2024

Comments on the paper

1 comment

We thank the reviewers for their valuable comments. We have provided answers to the reviewers questions, and made the following improvements to the manuscript to further strengthen our contributions:

  • Added additional comparison to naive Bradley-Terry model that does not consider mutually non-dominated points (Fig. 6), showcasing that modeling the true partial order is beneficial (Reviewer Sf4q).
  • Added ablation on batch-size (Appendix D), demonstrating that DPaO always attains an effective update, and is robust to changes in batch-size. (Reviewers, sf4q, BeMO).
  • Added ablation (Appendix L) validating that DPaO effectively learns the multi-modal reward function induced by the Pareto dominance relation (Reviewer sf4q).
  • Added additional experiments in Molecular design tasks with more objectives (Table 19), where DPaO outperforms other baselines in 4 / 5 metrics (Reviewers B28H, sf4q).

We have uploaded an updated revision our manuscript, with all the changes highlighted.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.