acceptodds
Under review as a conference paper at ICLR 2027

Preference-Guided Multi-Objective Reinforcement Learning with Explicit Objective Bounds

Abstract

Preference-based multi-objective reinforcement learning (MORL) learns policies that adapt to user-specified preferences, but practical decision problems often involve explicit objective bounds as well. Existing preference-based MORL does not directly handle such bounds, while constrained MORL does not explicitly preserve a given user preference. Motivated by this gap, we formulate preference-guided MORL with explicit objective bounds. In this setting, preference alignment, Pareto-front progress, and bound satisfaction can induce conflicting local updates, while the realized RL update may further drift from the selected direction during optimization. To address these challenges, we propose PBO, a single-projection method for preference-guided MORL with explicit objective bounds. Specifically, PBO constructs a preference-guided gradient-coefficient reference and obtains the final policy direction through a single projection enforcing preference-preservation and bound-handling constraints in the induced Fisher geometry. A directional KL guard further limits the accumulation of drift during repeated policy updates. We also provide first-order bound guarantees and a conditional contraction bound for constraint violations. Across control benchmarks and offline recommendation, PBO consistently achieves strong performance in preference following, bound satisfaction, and feasible Pareto-front quality.

Primary area:
reinforcement learning
Keywords:
Multi-objective Optimization, Multi-objective Reinforcement Learning, Constrained Reinforcement Learning, Preference-based Policy Optimization.

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5o4GOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work proposes PBO for preference-guided multi-objective reinforcement learning with explicit objective bounds. It combines preference guidance and constraint handling through a Fisher-based projection and directional KL control. The authors provide conditional constraint-violation bounds and report improvements in preference alignment and feasibility on simulated control and offline recommendation tasks.

Critical strengths

1. PBO provides a concrete mechanism for jointly handling preference guidance and explicit objective bounds. Its combination of preference protection, slack-based bound repair, and directional KL control is a reasonable attempt to balance objective improvement against feasibility restoration.

2. The experiments show improvements in preference alignment and feasible-solution diversity across the control tasks, with particularly strong feasibility results on Reacher, Circle, and offline recommendation. The component ablations and transfer to a TRPO backbone provide additional support for the proposed design.

Critical weaknesses

1. Computational efficiency remains unverified. Appendix A.2 reports hardware and interaction budgets, but not end-to-end training time or the computational overhead of gradient computation, Fisher operations, and projection. The coefficient-space formulation may reduce curvature-computation costs relative to Fisher-based CoMOGA, so an empirical efficiency comparison could strengthen the paper. CoMOGA already provides both training-time comparisons and a breakdown of computational costs in Appendices C.3 and D.2. CoMOGA

2. The role and empirical consequences of training-time constraint enforcement are insufficiently established. §3.3 and Appendices D.3–D.4 provide conditional violation bounds, but do not establish feasibility throughout the implemented PPO training process. The paper also does not report how constraint violations evolve during training, unlike the training curves provided by CoMOGA and CR-MOPO. Since the experiments use simulated environments and offline recommendation data, it remains unclear whether intermediate feasibility is an application requirement or an optimization device. This distinction matters: restricting intermediate policies may limit exploration and final solution quality, and prior work on relaxed exploration provides empirical evidence for this trade-off. CoMOGA, §6, CR-MOPO, §6, Shperberg et al., 2024

3. The methodological contribution needs a clearer explanation of the resulting optimization behavior. The preference-balancing criterion and objective-wise protection closely follow EPO, while jointly handling objective improvement and feasibility through local constrained optimization has clear precedent in CoMOGA. The proposed combination may improve the trade-off between intermediate feasibility and final solution quality, but the final results and component-removal ablations do not directly establish this mechanism. Figure 3 varies the task bounds themselves, rather than the degree or schedule of training-time enforcement under fixed final bounds. A more direct analysis would help distinguish the contribution of the proposed constraint-handling design. EPO, CoMOGA

Questions for authors

1. Could the authors report end-to-end training time and per-update computational costs under the shared experimental setup, particularly against EPO and CoMOGA? Please also clarify the Fisher and projection implementations, and how the original baselines are adapted to the common PPO backbone.

2.1 Is PBO primarily intended to maintain feasibility during learning, or to obtain a feasible final policy? Given the simulated and offline experimental settings, what motivates restricting intermediate policies, and what training-time behavior should the conditional guarantees in §3.3 and Appendices D.3–D.4 lead us to expect?

2.2 Could the authors report constraint values or violations throughout training, including variability across seeds and preferences? These results would help assess whether the proposed bound repair and KL guard actually improve training-time constraint adherence.

3. Under fixed final objective bounds, how do different degrees or schedules of training-time relaxation affect optimization trajectories, preference alignment, and final feasible-solution quality? Comparisons could vary the margin allowance α\alpha, slack penalty ρB\rho_B, or use initially relaxed or delayed bound enforcement under comparable training budgets. Experimental evidence, a controlled example, or theoretical analysis showing when the proposed design reaches better final feasible solutions would substantially strengthen its contribution.

The proposed method offers an interesting approach to addressing restricted optimization paths in constrained multi-objective optimization. I believe a clearer discussion of this issue would strengthen the research contribution beyond the reported improvements in final performance. If the authors address the concerns raised here and provide experimental analysis and insights on this issue, I would be happy to raise my rating.

9 comments
AuthorsOpenReview ↗

Response to Reviewer 5o4G (Response 1: W1, Q1)—-1.1 Training time and computational costs.

We thank the reviewer for the careful reading and constructive suggestions. Your comments have helped us strengthen the evaluation and clarify the method. We address each concern below and would be happy to provide further clarification or discuss any remaining questions.

Response 1: (W1, Q1) Computational Efficiency and Implementation.

Response 1.1: (W1, Q1) Training time and computational costs.

Thank you for this thoughtful suggestion. To answer your question, we present the following results.

End-to-end training time. We evaluate all methods on a single NVIDIA A100 GPU with the same backbone. Since CoMOGA trains a universal policy, we use the same number of environment steps and the same evaluation schedule as the other methods to compare computational costs under a fixed budget. The table reports mean ± sample standard deviation across three paired seeds, in minutes.

MethodDSTFruitTreeReacherCircle
Linear scalarization4.99 ± 0.035.15 ± 0.1414.70 ± 0.2356.29 ± 1.90
Primal–dual5.07 ± 0.045.05 ± 0.0114.49 ± 0.0354.69 ± 1.29
EPO5.73 ± 0.115.13 ± 0.0314.70 ± 0.1055.01 ± 2.53
CR-MOPO5.20 ± 0.055.24 ± 0.0315.18 ± 0.1955.15 ± 2.06
CoMOGA6.44 ± 0.096.97 ± 0.0615.50 ± 0.8754.05 ± 0.47
PBO4.47 ± 0.155.02 ± 0.1715.67 ± 0.4054.69 ± 3.20

PBO’s total runtime remains comparable to the baselines despite its additional constraint handling. Its mean time is lowest on DST and FruitTree and comparable to the baselines on Circle. On Reacher, PBO takes 15.67 minutes, approximately one additional minute relative to EPO.

The KL guard can reduce training time by stopping PPO updates early. When the projected direction may reduce the margin of the most critical bound, PBO uses a tighter KL threshold. Once that threshold is reached, it stops the remaining PPO updates on the current rollout. This limits deviation from the projected direction and can reduce the number of minibatches and epochs executed. Thus, although PBO adds Fisher computation and projection, it can perform fewer policy updates.

Computational breakdown. We measure the complete policy/value optimization stage and individual operations within each update. The tables report means across three seeds. Times are in milliseconds per update, and memory is peak allocated GPU memory in MiB.

DST

MethodOptimization (ms)Gradients (ms)Fisher (ms)Solver (ms)Memory (MiB)
EPO153.663.62—1.9417.97
CR-MOPO157.954.468—0.23317.96
CoMOGA245.3811.8616.041.5118.68
PBO56.034.503.611.4317.99

FruitTree

MethodOptimization (ms)Gradients (ms)Fisher (ms)Solver (ms)Memory (MiB)
EPO154.536.08—1.6918.03
CR-MOPO160.865.242—1.11818.03
CoMOGA303.4120.6568.372.2318.80
PBO128.188.585.893.2118.07

Reacher

MethodOptimization (ms)Gradients (ms)Fisher (ms)Solver (ms)Memory (MiB)
EPO736.145.19—1.7820.05
CR-MOPO797.756.448—2.03520.05
CoMOGA633.0235.66126.562.5423.10
PBO816.637.805.341.8922.70

Circle

MethodOptimization (ms)Gradients (ms)Fisher (ms)Solver (ms)Memory (MiB)
EPO469.323.07—1.2020.13
CR-MOPO459.721.947—0.09220.13
CoMOGA370.2216.3463.931.7623.40
PBO303.313.613.201.2722.93

PBO’s projection requires only a small fraction of optimization time. Its solver takes 1.27–3.21 ms per update, approximately 0.2–2.6% of the complete optimization time. Fisher computation takes 3.20–5.89 ms, compared with 16.04–126.56 ms for CoMOGA in these implementations. These results support the computational benefit of solving a single projection over gradient coefficients. PBO’s peak allocated GPU memory is close to the other methods.

The breakdown also explains why total runtimes can be similar despite different optimization costs. On Circle, rollout collection and evaluation take approximately 53 minutes, while the complete policy/value optimization stage takes only 1–2 minutes. PBO’s KL guard can also stop repeated PPO updates early, reducing optimization time despite its additional Fisher computation and projection.

AuthorsOpenReview ↗

Response to Reviewer 5o4G (Q1)—-1.2 Fisher, projection, and baseline implementations.

Response 1.2: (Q1) Fisher, projection, and baseline implementations.

Fisher and projection implementation.

  • Fisher: We compute Hessian–vector products of the analytic policy KL using automatic differentiation, without forming the full Fisher matrix. We construct G(F + λ_F I)Gᵀ with λ_F = 10⁻³, symmetrize it, and floor its eigenvalues at a small positive threshold to obtain the positive-definite matrix M\mathbf M (to avoid being singular).
  • Projection: Once per rollout, we solve the convex quadratically constrained quadratic program using SciPy SLSQP. It combines the preference and bound conditions, penalized slack, and the quadratic trust-region constraint. The resulting coefficients combine the advantages used by PPO. During the minibatch update of PPO, When the KL threshold is exceeded, the KL guard stops the remaining minibatches update for the current rollout. Training then continues with a new rollout.

Baseline adaptation. For baselines adapted to PPO, we use the same actor–critic architecture, rollout collection, and objective-wise value and GAE computation, while retaining each method's original mechanism for determining the policy update:

  • EPO: The official EPOSearch LP computes objective weights from the gradient Gram matrix. These weights combine the advantages for clipped PPO.
  • CR-MOPO: A violated bound triggers constraint rectification; otherwise, preference-weighted CAGrad computes the gradient combination used by PPO.
  • Linear scalarization and Primal–dual: Fixed preference weights or adaptive constraint multipliers define the combined PPO objective.
  • CoMOGA: We retain its preference-conditioned universal policy, Fisher–vector products, conjugate-gradient computation, coefficient QP, and target-policy fitting update while sharing the training backbone as all other methods.
AuthorsOpenReview ↗

Response to Reviewer 5o4G (Response 2: W2, Q2.1) and (Response 3: W3)

Response 2: (W2, Q2.1) Purpose of Training-Time Constraint Handling and Theoretical Guarantees

Response 2.1: Why Apply Constraints During Training?

Thanks for raising this concern. We'd like to clarify that PBO aims to obtain a high-quality final feasible policy; it does not require every intermediate policy to satisfy all bounds. The final bounds are used to guide training by specifying which constraint violations need to be reduced and how preference improvement should account for those bounds.

Introducing the bounds only later can require substantial policy adjustments, reversing earlier return gains that depended on violating them. Thus, PBO keeps these requirements active throughout learning while allowing flexibility: α\alpha permits some decrease in a bounded objective that already exceeds its required minimum, and penalized slack permits relaxation of local bound conditions when they conflict with other update conditions. Temporary violations remain possible, but bound repair is considered in every update.

The experiments in Response 5 directly test this design. Moderate flexibility improves FruitTree’s final return while preserving 9/9 feasible policies. On Reacher, default enforcement produces more feasible final policies than delayed or annealed enforcement, with similar final returns.

Response 2.2: What Does the Theory Establish for the Implemented Updates?

Our theory explains how the bound conditions in PBO’s projection can reduce actual constraint violations over successive policy updates. Under the stated assumptions, it connects these conditions to the next policy’s violation and shows how slack and prediction errors limit the reduction. With the additional conditions in Appendix D.3, we expect violations to decrease over training toward a level determined by slack and errors, while temporary increases remain possible. Response 4 examines this behavior through violation and feasibility curves across seeds and preferences.

This analysis explains the roles of the projection and KL guard. The projection determines the coefficients combining objective and bound gradients, while penalizing relaxation of the local bound conditions within the same optimization problem. Appendix D.4 shows how differences between the projected direction and the actual PPO update can weaken bound repair, motivating the KL guard. Once its threshold is exceeded, the remaining PPO updates on the current rollout stop, and training continues with fresh data.

The theory thus clarifies how bound handling, slack, and update errors jointly affect training violations.

Response 3: (W3) Methodological Contribution Beyond EPO and CoMOGA.

PBO differs from EPO and CoMOGA in both the problem formulation and the policy update. We introduce a preference-guided MORL setting in which a specified preference and explicit lower or upper bounds apply to the same objective vector. This setting is not explicitly studied by EPO or CoMOGA. For example, DST’s time return is both a preference objective and subject to a lower bound. The update must account for how preference improvement affects the bounds and how bound repair affects preference alignment.

1. Beyond EPO: preference guidance with objective bounds in RL. EPO was introduced for supervised multi-task optimization without explicit objective-bound handling. PBO extends its preference-balancing criterion and objective-wise protection to RL updates that jointly handle preference improvement and bound repair, with controlled flexibility when these requirements conflict.

2. Beyond CoMOGA: preference guidance remains active during bound repair. CoMOGA focuses on environmental safety constraints; PBO directly imposes lower or upper bounds on objective returns, which may also participate in preference following. CoMOGA combines objective improvement and safety constraints when feasible, but switches to safety-only recovery when constraints are violated. PBO retains the preference reference and preservation conditions during bound repair, using the same projection structure for feasible and infeasible policies.

Therefore, PBO goes beyond combining EPO’s preference criterion with CoMOGA’s constraint handling: it changes how preference improvement and bound repair interact within each update. When a bound is satisfied, α\alpha permits some change in that objective to improve preference following; when requirements conflict, penalized slack permits relaxation and temporary violations. The final bounds continue to guide correction throughout learning.

The following Responses 4 and 5 further provide training trajectories and controlled studies varying α, slack penalties, and enforcement schedules under fixed final bounds. These directly examine the resulting preference alignment, training violations, and final policy quality.

Reviewer_2NhROpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper aims at Pareto-front progress at the same time a preference alignment to the respective goal, and tries to solve the potential conflicts between these goals by a projection method which requires some normalization w.r.t. the objectives. The algorithm works like other PG methods with a Fisher geometry, but it is not clear whether this makes a significant difference or whether it is buried by the sampling noise in all but trivial practical cases.

Critical strengths

The proposed method is reasonable and is based on a clear gap in the MORL literature. If more work is added it can potentially grow into a really useful algorithm.

Critical weaknesses

Although it is quite standard, the problem formulation is not clear at all (what is B\cal B, how are nn, mm, and kk related, what is Δ\Delta etc.?). It does not contain any references, although it uses expressions like the “original Fisher geometry” or the “he local policy-KL Hessian” which are ambiguous with out (reference to) a definition. This lack of clarity continues in the Methods, which claims that rewards can simply be normalized which is questionable in general, and even more so in MORL, also this normalization would be compromised when a “smoothed normalized return” is used (or rather a “normalized smoothed return”?). It is not clear how to use a KL divergence (it is also not stated what “KL” means here nor even that it’s a “divergence”) in such conditions, so the methods cannot be considered as reproducible.

Theorem 1 assumes is not based on proper definitions and even necessary conditions are formulated only in an appendix that does not make them explicit. I understand the theorem in machine learning are often not directly needed to obtain practical results, but the should be notwithstanding a clear formulation. The problem with the theorem remains that the slack variable can be estimated better (in theory), but also that the composed quantities have no robust practical meaning for most problems, and it is characteristic that no data of the actual progress in the Pareto space are shown.

Results are good for the classical Deep-Sea Treasure problem, but not typically not significantly different from the second best in the other tests problems.

Questions for authors

Can you use the unlimited space in the appendix to make sure that all quantities are properly defined?

Can you discuss the relation between the theorem and your implementation and results?

Can you show a few detailed traces illustrating the progress in the Pareto space for some example runs e.g. for the DST but if possible also for a more critical problem?

Can you comment on scaling (w.r.t. to number of objectives and problem size) and on problems with non-connected Pareto fronts (A3 contains a singular mention of "connected components" but it is not clear what it means or what it results to).

6 comments
AuthorsOpenReview ↗

Response to Reviewer 2NhR (W1 & Q1)

We sincerely thank the reviewer for the careful reading and constructive feedback. We appreciate these suggestions and have improved the organization, explanations, and notation in the revised manuscript. We address each concern below and would be happy to provide further clarification or discuss any remaining questions.

W1 & Q1: Clarification of the problem formulation and notation

We thank the reviewer for the careful reading and constructive suggestions. We'd like to mention that we formulate a novel setting of preference-guided MORL with explicit objective bounds, where user-specified preferences determine the desired trade-offs among objectives, while explicit lower or upper bounds specify feasibility requirements. Unlike conventional constrained MORL, our setting additionally requires preserving the specified preference while handling these bounds.

Following suggestions, we have carefully revised Section 2 (Problem Formulation) to introduce the key quantities and their roles more clearly (reorganized Section 2 to introduce the problem setting →\rightarrow policy gradients →\rightarrow local policy geometry and trust region →\rightarrow gradient combinations in a clearer and logical sequence). We have also added Table 9 (Main Notation) in Appendix F, summarizing the mathematical symbols, their dimensions, and meanings. Then we address the specific concerns as follows.

1. Problem formulation and notation (§2).

We now explicitly define the relationships among the quantities mentioned by the reviewer:

  • nn is the total number of return objectives, and P\mathcal P is the subset of m≤nm\leq n preference objectives.
  • A user preference ω\omega is given over a subset P\mathcal{P} of mm objectives, which we call the preference objectives. It lies in the probability simplex

\Delta^{m-1}=\left\\{\omega\in\mathbb{R}^{m}:\omega_i\geq 0,\\ \sum_{i=1}^{m}\omega_i=1\right\\}

and ωi\omega_i is the weight of objective ii.

  • B\mathcal B contains the individual lower- or upper-bound conditions. An objective may be both a preference objective and a bounded objective, and a two-sided bound contributes two separate conditions.
  • K=m+∣B∣\mathcal |B| is the total number of gradients used to construct the policy update.

These definitions are now introduced before the corresponding equations, with Table 9 in Appendix F providing a consolidated notation reference.

2. KL divergence and local Fisher geometry (§2 and §3.3).

We now explicitly define KL as the Kullback–Leibler divergence and clarify its two uses in PBO.

In Local Policy Geometry and Trust Region (§2), we define the expected policy KL divergence as DˉKL(θ)=Es[DKL(πθold(⋅∣s)∥πθ(⋅∣s))]\bar D_{\mathrm{KL}}(\theta)=\mathbb E_s[D_{\mathrm{KL}}(\pi_{\theta_{\mathrm{old}}}(\cdot\mid s)\parallel\pi_\theta(\cdot\mid s))], which measures changes between the current and updated policy distributions.

We define FF as its Hessian at the current policy and explain the local quadratic approximation DˉKL(θold+d)≈12d⊤Fd\bar D_{\mathrm{KL}}(\theta_{\mathrm{old}}+d)\approx\frac12d^\top Fd, following TRPO and CPO. This clarifies what the Fisher geometry measures and why it is used to restrict policy updates.

Separately, in Overall Preference Preservation Constraints (§3.3), DKL(ω∥p)D_{\mathrm{KL}}(\omega\|p) measures the mismatch between the requested preference ω\omega and the current return proportions pp, determining when the overall preference-preservation constraint is activated. We now explicitly define pp from the smoothed normalized returns, distinguishing this use from the policy KL divergence.

3. Return normalization and smoothing (§3.1).

In Normalized Returns (§3.1), we clarify both the order and purpose of these operations. Specifically, each preference-objective return is first normalized to zi∈[0,1]z_i\in[0,1] using its running minimum and maximum, and an exponential moving average is then applied to obtain the smoothed normalized return ζi\zeta_i.

The user preference ω\omega specifies the relative importance of different objectives, while explicit bounds specify fixed limits on objective returns. Since objectives can have different reward scales, we normalize their returns before using them for preference balancing and front seeking (§3.1), so that differences in reward magnitude do not dominate preference following. In contrast, bound handling (§3.2) uses the original return estimates to evaluate feasibility against the specified bounds. The predicted improvements are then normalized within the trust region, allowing preference and bound requirements to be considered together in the projection while preserving the original bounds.

Together with Table 9 in Appendix F.1, these revisions make the notation, KL definitions, and normalization procedures explicit and easier to follow.

AuthorsOpenReview ↗

Response to Reviewer 2NhR (W2 & Q2)

W2 & Q2: Clarification of Theorem 1 and its connection to the implementation and results

We thank the reviewer for raising these concerns. We have reorganized the theoretical presentation to clarify the assumptions, interpretation, and connection of Theorem 1 to the actual PBO updates. In the revised manuscript, §3.2 first establishes the first-order behavior of bound handling (Proposition 1), and §3.4 then extends this analysis to the realized policy update (Theorem 1). The complete proofs and additional conditions are provided in Appendix D.

1. Theorem 1: Interpretation and assumptions.

Theorem 1 characterizes how the violation of an objective bound changes across policy iterations. For each bound bb, we define its true margin cb(θ)c_b(\theta), violation vb,t=[−cb(θt)]+v_{b,t}=[-c_b(\theta_t)]_+, and prediction error ϵb,t\epsilon_{b,t}, which measures how much the realized margin improvement falls below its local prediction. Under the conditions νb,t>0\nu_{b,t}>0 and c^b,t≤cb(θt)\hat c_{b,t}\leq c_b(\theta_t), Theorem 1 establishes:

+\nu_{b,t}s_{b,t}^{\star} +\epsilon_{b,t}$$ The three terms have direct interpretations in PBO: - **Bound handling (§3.2):** $\nu_{b,t}$ is the largest predicted improvement of the bound margin within the trust region (Eq. 6). The first term represents the remaining violation after the improvement requested by the bound-handling constraint. - **Projection slack (Eq. 7):** $s_{b,t}^\star$ is obtained directly by solving the projection problem, rather than requiring separate estimation. It captures how much the projection relaxes the requested bound improvement when different constraints conflict. The penalty $\rho_B$ controls this relaxation. - **Policy-update error (§3.4):** $\epsilon_{b,t}$ measures the shortfall between the predicted and realized bound improvement. Appendix D.5 further relates this error to gradient estimation error, deviation from the selected update direction, and local curvature. **2. Connection to the implementation.** These three terms correspond directly to the steps performed at each PPO policy iteration. PBO first estimates the bound margins from the rollout, subtracting one standard error to account for uncertainty in the estimates. The estimated margins determine the bound-handling thresholds in §3.2. PBO then solves Eq. 7 to obtain the projected coefficients $\beta^\star$ and slack variables $s^\star$, and uses the resulting combined advantage for PPO. Additionally, since repeated PPO updates can deviate from the selected direction, the **KL guard** tightens PPO's early-stopping threshold when the most critical bound is predicted to lose margin. This is intended to limit the policy-update error identified in Theorem 1. **3. Theoretical implications and connection to empirical results.** Theorem 1 characterizes how PBO reduces constraint violations across policy iterations under explicit conditions. In practice, the one-standard-error adjustment provides a more conservative estimate of the bound margin, while the slack penalty and directional KL guard help control the terms that affect the realized improvement. **The distinction from stochastic PPO training is that the projection uses local predictions based on estimated gradients, while PPO realizes the update through stochastic minibatches and repeated optimization steps.** The prediction-error term $\epsilon_{b,t}$ explicitly captures the difference between the predicted and realized margin improvements, rather than assuming they coincide. Appendix D.5 further characterizes this error under local assumptions, connecting the theoretical analysis to the actual training process. Under the additional conditions in **Appendix D.4**, including positive local improvement capacity and bounded violations, slack, and prediction errors, the analysis establishes geometric convergence of the maximum violation to a feasibility neighborhood. The empirical results support the effectiveness of these mechanisms. In Table 2, removing bound handling on Reacher increases constraint violation (CV) from 0.122 to 4.787 and reduces feasibility from 24/30 to 13/30. Full PBO also achieves zero final CV on Circle (Table 1). These results demonstrate the practical benefit of the proposed bound-handling mechanism and complement the theoretical analysis. Overall, the revised presentation makes explicit how **the bound-handling constraints determine the requested local improvement, the projection slack captures conflicts between constraints, and the policy-update error accounts for the difference between the selected direction and the realized policy update**. Theorem 1 connects these three components and characterizes their joint effect on constraint violations during training.
AuthorsOpenReview ↗

Response to Reviewer 2NhR (W3)

W3 Performance Beyond DST Across Evaluation Metrics

Thank you for raising this point. We would like to clarify that PBO’s advantages extend beyond DST. The comparison should consider all metrics for each method together, since the second-best results often come from different baselines. A baseline can score well on one metric while totally failing other requirements. High returns may involve bound violations or repeatedly selecting similar solutions across preferences; low violation may come from conservative solutions with little objective improvement.

For example, on DST, CoMOGA achieves zero violation but returns only 2.33 distinct feasible solutions, compared with 8.00 for PBO, and has higher normalized regret (0.207 vs. 0.123). Together, these results indicate less variation across preferences and weaker preference following. Its feasible utility is also substantially lower (2.109 vs. 17.120). Perfect feasibility alone therefore does not establish success across the requirements of our setting.

Beyond DST, PBO achieves the best mean result on all six metrics on Reacher. On Circle, it achieves 27/27 feasibility, compared with at most 16/27 for any baseline, despite a smaller difference in hypervolume. On FruitTree, it provides more distinct feasible solutions and lower regret than CoMOGA while retaining high feasibility. On KuaiRand, it matches BC’s perfect feasibility while producing solutions that better reflect different preferences. These results show PBO’s consistent strength in satisfying bounds and following preferences across tasks. The radar plots furtehr summarize these consistent benefits across metrics.

Reviewer_Kw2aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a new preference-based multi-objective RL algorithm that considers preference alignment, front seeking, and, more importantly, explicit objective bounds. The authors manually design mechanisms to achieve preference balancing and front seeking. By combining these mechanisms, they propose an adaptive reference to favor both preference balancing and front seeking. Then, they introduce constraints to ensure that the objective bounds are not violated. Finally, to prevent the bound-handling constraints from interfering with the preference alignment requirement, they introduce additional manually designed constraints and formulate the constrained optimization problem in Equation 13 to determine the weights for the objective gradient vectors.

Critical strengths
  1. This paper proposes a series of carefully designed mechanisms that translate the requirements of preference alignment and bound handling into different constraints. By solving the proposed constrained optimization problem, the method determines the weights for the objective gradient vectors. Overall, I think these mechanisms are sensibly designed and intuitively reasonable.

  2. The experiments demonstrate the individual contributions of each component of the proposed algorithm.

Critical weaknesses
  1. This paper is not well written or organized, which makes it difficult to follow and understand the role of each proposed component. In the abstract, the authors state that "PBO constructs a preference-guided gradient-coefficient reference and obtains the final policy direction through a single projection enforcing preference-preservation and bound-handling constraints in the induced Fisher geometry." However, in the method section, the authors mainly introduce a series of constraints or mechanisms designed to achieve different goals. I cannot clearly see the connection between the summary in the abstract and the components introduced in the method section. The authors should at least establish clear connections between these components and provide a better summary of the overall algorithm. Throughout the paper, there are too many terms that are not explicitly explained or defined, such as "progress vector" and "unconstrained Fisher capacity." There are also many unclear descriptions, such as "rescale it to the Fisher boundary, denoting the result by ...". Moreover, many components are introduced without sufficient explanation of the motivation or intuition behind their design. I strongly suggest that the authors reorganize the paper and more clearly explain the ideas behind each equation and notation, as well as the motivation for each mechanism and constraint.

  2. Equations 3 and 4 provide the basic formulation underlying the proposed method. However, both equations lack sufficient explanation and motivation, particularly regarding how they are derived and how their roles should be understood. Although I understand that the authors aim to restrict the weighted gradient update to a region where the first-order approximation remains reliable, it is still important to explicitly explain these equations and their underlying intuition, especially since the proposed method is built upon them.

Questions for authors

Please see Weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Kw2a (W1: 1/2)

We sincerely thank the reviewer for the careful reading and constructive feedback. We appreciate these suggestions and have improved the organization, explanations, and notation in the revised manuscript. We address each concern below and would be happy to provide further clarification or discuss any remaining questions.

W1: Organization and clarity of the method

We formulate a novel setting of preference-guided MORL with explicit objective bounds, where user-specified preferences determine the desired trade-offs among objectives, while explicit lower or upper bounds define acceptable objective returns.

Following suggestions, we have carefully reorganized Sections 2 (Problem Formulation) and 3 (Method) in the updated manuscript. Specifically, we have (1) reorganized Section 2 to introduce the problem setting →\rightarrow policy gradients →\rightarrow local policy geometry and trust region →\rightarrow gradient combinations in a clearer and logical sequence; (2) added a short Overview of PBO before Section 3 (Method) that presents the complete projection before introducing its components; (3) reorganized Section 3 so that each subsection defines a corresponding part of this projection (a convex QP), with clearer explanations of the motivation and intuition behind each mechanism and constraint; and (4) added Algorithm 1 to summarize PBO in one PPO iteration. The main changes are detailed below.

1. We now introduce the overall method first as a single convex projection problem (end of §2).

The core idea of PBO is to determine how the gradients of the preference objectives and bound margins should be combined into a single policy update. At each PPO policy iteration, PBO considers KK gradients, one for each preference objective and bound margin, and determines their coefficients β\beta through a single projection. These coefficients define the update direction d(β)=∑kβkgkd(\beta)=\sum_k\beta_k g_k (Eq. 4). PBO selects the coefficients before eahc PPO update and uses them to construct the advantage given to PPO.

The new Overview of PBO paragraph introduces the central projection (Eq. 7, previously Eq. 13) before presenting its individual components. Mathematically, it is a convex quadratically constrained quadratic program:

The optimal coefficients (β⋆,s⋆)(\beta^\star,\mathbf{s}^\star) are obtained by solving

\frac{1}{2}(\beta-\bar{\beta})^\top\mathbf{M}(\beta-\bar{\beta}) +\sum_{j\in\mathcal{C}}\frac{\rho_j}{2}s_j^2$$ subject to $$\frac{1}{2}\\|\beta\\|_{\mathbf{M}}^2\leq\delta, \qquad \frac{\mathbf{u}_j^\top\beta}{\nu(\mathbf{u}_j)} +s_j\geq h_j,\quad j\in\mathcal{C}.$$ This formulation provides the overall picture of PBO: the objective keeps the selected coefficients close to a preference-guided reference $\bar{\beta}$, while the constraints handle explicit bounds and preserve the intended preference within the trust region. Section 3 is then organized around this formulation: §3.1 constructs the reference coefficients $\bar{\beta}$ to be projected, §3.2–3.3 define the projection constraints, and §3.4 explains how the resulting coefficients are used for policy optimization. **2. Each subsection of §3 now explains a corresponding component of Eq. 7 and its motivation.** Instead of introducing individual mechanisms separately, we reorganized Section 3 around the components of the projection and their roles in the policy update: - **§3.1 — Coefficients to Be Projected: Following the Preference.** This subsection defines $\bar{\beta}$, the reference toward which the objective of Eq. 7 pulls $\beta$. It adaptively combines preference balancing and front seeking. When the preference objectives are imbalanced, the reference emphasizes balancing their weighted gaps; as they become balanced, it increasingly emphasizes progress toward the Pareto front. This transition is controlled by the non-uniformity measure $\mu$ (representing the status of preference balance). If $\bar{\beta}$ already satisfies all projection constraints, it remains unchanged; otherwise, the projection adjusts the coefficients while minimizing their distance from the reference and penalizing constraint violations. - **§3.2 — Projection Constraints: Bound Handling.** This subsection defines the constraints in Eq. 7 for explicit bound satisfaction. Each bound adds a constraint requiring the predicted improvement of its margin to reach a threshold $h_b$ determined by the estimated margin. A violated bound requests improvement, while a satisfied bound may lose part of its margin, controlled by $\alpha$. Proposition 1 establishes the corresponding first-order behavior, including the effect of slack. (continue on next block)
AuthorsOpenReview ↗

Response to Reviewer Kw2a (W1: 2/2)

  • §3.3 — Projection Constraints: Preference Preservation. Since bound handling may move the selected direction away from the intended preference, we introduce two additional types of constraints. Objective-wise preference constraints limit decreases in preference objectives based on those allowed by preference balancing, while an overall preference constraint encourages the predicted improvements to move return proportions toward the specified preference ω\omega.

  • §3.4 — Policy Update and Bound Stability. The projected coefficients β⋆\beta^\star define the combined advantage used by PPO. Since repeated PPO updates may deviate from the selected direction, the KL guard tightens PPO's early-stopping threshold when the most critical bound is predicted to lose margin. Theorem 1 connects these mechanisms by bounding the next constraint violation through three terms: the remaining violation after bound handling, the projection slack, and the error between predicted and realized improvement.

All projection constraints share the form in Eq. 7: a normalized predicted improvement must reach a specified threshold (R.H.S), with slack where defined. The normalization allows objectives and bound margins with different scales to be compared within the same trust region. The revised subsection titles explicitly identify the roles of the components, and each subsection now explains why its component is needed before introducing the corresponding equations. This makes the connections between the mechanisms explicit rather than presenting them as independent constraints.

3. We now introduce and explain the key concepts in §2 before using them in the method.

Section 2 has been reorganized into clearly labeled paragraphs with bold headings: Setting →\rightarrow Policy Gradients for Preferences and Bounds →\rightarrow Local Policy Geometry and Trust Region →\rightarrow Combining Gradients in Coefficient Space →\rightarrow Normalized Improvement within the Trust Region and Overview of PBO. These paragraphs progressively introduce the problem setting, policy gradients, the Fisher information matrix-based local approximation and trust region, the coefficient-space update, and the normalized improvements used by the projection.

We have also simplified the terminology used to refer to previously defined mathematical quantities. We stop introducing additional descriptive terms in later sections, and now refer directly to their mathematical expressions when used:

  • Instead of "progress vector," we use Cβ\mathbf{C}\beta (Eq. 5), whose entries represent the predicted improvements of the preference objectives and bound margins, with C=GG⊤\mathbf{C}=\mathbf{G}\mathbf{G}^\top.
  • Instead of "unconstrained Fisher capacity," we use ν(u)=2δ u⊤M−1u\nu(\mathbf{u})=\sqrt{2\delta\,\mathbf{u}^\top\mathbf{M}^{-1}\mathbf{u}} (Eq. 6), the largest predicted improvement of a given quantity within the trust region.
  • Instead of "rescaling to the Fisher boundary," we use "rescaling to the boundary of the trust region" (§2), expressed by Πδ\Pi_\delta. This makes clear that the purpose is to construct reference directions with the same update size before projection.

We further explain how the KL-based trust region leads to the quadratic constraint on β\beta, and why normalized improvements are used to compare objectives and bound margins with different scales in revised Section 2. Table 9 in Appendix F also presents and explans all the notation used.

4. Algorithm 1 now summarizes the complete PBO procedure in one PPO policy iteration.

To make the implementation easier to follow, Algorithm 1 — One Policy Iteration Using PBO presents the PBO steps in execution order: computing the gradients and matrices C,M\mathbf{C},\mathbf{M} (§2), constructing the reference βˉ\bar{\beta} (§3.1), defining the projection constraints (§3.2–3.3), solving Eq. 7 for β⋆\beta^\star, and performing the PPO update with the directional KL guard (§3.4). Each step points to the corresponding section.

We have also aligned the abstract and introduction with this organization. The high-level description quoted by the reviewer now corresponds directly to the single projection in Eq. 7, its individual components in §3, and the complete procedure in Algorithm 1. Together, these changes clarify how PBO combines preference following, bound handling, and preference preservation within one unified policy-update method, rather than introducing a series of disconnected mechanisms.

AuthorsOpenReview ↗

Response to Reviewer Kw2a (W2)

W2: Motivation and interpretation of the original Eqs. (3) and (4)

We thank the reviewer for this suggestion. We have expanded Section 2 to explain the motivation and interpretation of the original Eqs. (3) and (4), now Eqs. (4) and (5) in the revised manuscript. In particular, we introduce Local Policy Geometry and Trust Region before Combining Gradients in Coefficient Space, so that readers can understand why these equations are needed and how they lead to the final projection.

1. Original Eq. (3) → Revised Eq. (4): Why combine gradients?

PBO needs to determine a single policy update while considering multiple preference objectives and explicit bounds at each PPO iteration. Each objective or bound margin has its own gradient, indicating a direction in which it is expected to improve locally. Since these directions may conflict, PBO must determine how much each gradient contributes to the final update. Following gradient-based multi-objective optimization, we express this as

d(β)=G⊤β=∑k=1Kβkgk.d(\beta)=\mathbf{G}^\top\beta=\sum_{k=1}^{K}\beta_k g_k.

Here, each coefficient βk\beta_k controls the contribution of gradient gkg_k. Instead of directly optimizing a direction over all policy parameters, PBO only needs to determine these KK coefficients. At the current policy, using the combined advantage ∑kβkAt,k\sum_k\beta_k A_{t,k} produces the same policy gradient d(β)d(\beta), allowing the selected direction to be used directly by PPO

2. Original Eq. (4) → Revised Eq. (5): How do we evaluate a combined direction?

Once the gradients are combined, PBO needs to answer two questions: How does this direction affect each objective or bound margin? And how much does it change the policy?

For the first question, the predicted improvement of objective or margin kk is gk⊤d(β)g_k^\top d(\beta). Substituting the combined direction gives

=\sum_{j=1}^{K}\beta_j g_k^\top g_j =(\mathbf{C}\beta)_k, \qquad \mathbf{C}=\mathbf{G}\mathbf{G}^\top.$$ The inner product $g_k^\top g_j$ indicates whether moving along gradient $g_j$ also helps objective $k$ or works against it. Thus, **$\mathbf{C}\beta$ tells us the predicted change of every objective and bound margin under the selected direction**, including the effects of conflicting gradients. For the second question, we use the **local policy geometry and trust region** introduced earlier in revised Eq. (3). Since large policy changes make first-order predictions less reliable, we measure policy change using the expected KL divergence, locally approximated by $\frac12 d^\top Fd$, where $F$ is the Fisher information matrix. Substituting $d(\beta)=\mathbf{G}^\top\beta$ expresses this change in terms of $\beta$: $$\frac12 d(\beta)^\top Fd(\beta) =\frac12\beta^\top\mathbf{G}F\mathbf{G}^\top\beta.$$ With Fisher damping for numerical stability, we define $$\mathbf{M}_0=\mathbf{G}(F+\lambda_F I)\mathbf{G}^\top.$$ We further apply eigenvalue flooring to obtain a positive-definite matrix, giving the conservative quadratic trust-region constraint $$\frac{1}{2}\\|\beta\\|_{\mathbf{M}}^2\leq\delta.$$ **3. How these equations support the projection.** Together, these quantities have distinct roles: **$\mathbf{C}\beta$ predicts how the objectives and bound margins change, while $\mathbf{M}$ measures the size of the selected update in the local policy geometry.** This allows PBO to compare predicted improvements while restricting the size of the policy update. We further explain the normalized improvements in revised Eq. (6), which account for different objective scales within the trust region. These quantities are then used to formulate the **single projection in revised Eq. (7)**. The revised Section 2 now explains these connections step by step.
AuthorsOpenReview ↗

A gentle follow-up on the rebuttal

Dear Reviewer Kw2a,

We sincerely thank you for your time and effort in reviewing our paper, as well as your thoughtful feedback and constructive suggestions throughout the discussion.

In our rebuttal, we have (1) substantially reorganized Sections 2 and 3 around the single-projection formulation, presenting the complete optimization problem before its components and clarifying how preference balancing and front seeking construct the reference (§3.1), how bound handling and preference preservation define the projection constraints (§3.2–3.3), and how the resulting coefficients are used for policy updates (§3.4), with clearer explanations of the motivation and intuition behind each mechanism; (2) expanded the motivation and interpretation of the original Eqs. 3 and 4 (now Eqs. 4 and 5), reorganizing Section 2 to introduce the problem setting →\rightarrow policy gradients →\rightarrow local policy geometry and trust region →\rightarrow gradient combinations in a clearer sequence; and (3) added Algorithm 1 and a notation table (Appendix F) to summarize the complete PBO procedure and clarify the mathematical quantities used throughout the paper.

With less than 13 hours remaining in the discussion period, we would greatly appreciate hearing whether our responses have adequately addressed your concerns. If any questions or concerns remain, we would be happy to provide further clarifications and continue the discussion before the deadline. Thank you again for your valuable feedback, time, and consideration, which have been greatly appreciated and helpful in improving our work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear ACs and Reviewers,

We sincerely thank the reviewers for their constructive feedback. In response to their suggestions, we have revised and uploaded our manuscript to improve its clarity and presentation and conducted extensive additional experiments during the rebuttal. We summarize our main contributions below.

Our work makes three main contributions:

  1. A novel setting of preference-guided MORL with explicit objective bounds. We formulate and study a multi-objective reinforcement learning problem in which policies must follow user-specified preferences while satisfying explicit lower or upper bounds on objectives. Unlike existing preference-based or constrained MORL formulations, our setting requires preference alignment, Pareto-front progress, and bound satisfaction to be addressed jointly.

  2. A unified single-projection method for policy optimization. At each policy iteration, PBO solves a single convex projection problem in gradient-coefficient space, obtaining coefficients that combine preference- and bound-objective gradients into one policy update. Specifically, PBO adaptively combines preference balancing and front seeking to construct a preference-guided coefficient reference, then projects it under bound-handling and preference-preservation constraints in the induced Fisher geometry. Margin allowance and penalized slack provide controlled flexibility, while the KL guard limits drift during repeated PPO updates.

  3. Extensive empirical validation with theoretical support. We evaluate PBO on four control benchmarks spanning discrete and continuous action spaces, 2–6 objectives, overlapping preference and bounded objectives, and one- or two-sided bounds, as well as offline recommendation on KuaiRand. PBO consistently demonstrates strong performance in preference following, bound satisfaction, and feasible Pareto-front quality. We also establish first-order guarantees for the progress induced by our bound-handling constraints and further characterize how projection slack and discrepancies between predicted and actual PPO updates affect realized constraint violations. Under explicit additional conditions, we establish geometric contraction toward a feasibility neighborhood.

We appreciate Reviewer 2NhR's recognition of the clear gap in MORL, Reviewer Kw2a's positive assessment of the proposed mechanisms and component evaluations, and Reviewer 5o4G's recognition of PBO's joint preference-and-bound handling and empirical strengths.

During the rebuttal, we substantially strengthened the manuscript through:

  • Clearer formulation and methodological presentation: We reorganized Sections 2 and 3 around the our single-projection formulation, expanded the motivation and interpretation of the key equations, and added Algorithm 1 and a comprehensive notation table (Appendix F). We also clarified the assumptions and practical interpretation of Theorem 1 and its connection to the implemented PPO updates.

  • Direct evidence of optimization behavior: We added objective-space training trajectories on DST and Reacher (Appendix E.1, Fig 5), constraint violation and feasibility curves during training (Appendix E.2, Fig 6), and curves of the KL guard effect (Appendix E.3, Fig 7). These analyses show how PBO improves preferred objectives while controlling bound violations, and demonstrate the KL guard's benefits for training-time constraint adherence and final feasibility.

  • Controlled experiments supporting PBO's bound-handling design: We systematically studied margin allowance, slack penalties, and different constraint-enforcement schedules (immediate, delayed, and annealed) under fixed final bounds (Appendix E.4, Fig 8). The results show that moderate margin allowance and slack can improve preference following while maintaining strong final feasibility. All findings support our bound-handling constraint design within the projection: maintaining guidance from the final bounds throughout training while allowing controlled flexibility.

  • Computational efficiency and scalability: We provided end-to-end training times, detailed per-update computational breakdowns, and implementation details. PBO maintains competitive end-to-end runtime, with the projection solver accounting for only 0.2–2.6% of the complete optimization time. Additional scaling experiments (Appendix E.5, Fig 9) examine increasing objective counts and network sizes, supporting the practicality of PBO.

Overall, the revised manuscript and additional evidence further support PBO's methodological contribution and effectiveness in jointly handling preference following, Pareto-front progress, and explicit objective bounds under our newly formulated preference-guided MORL with objective bound setting.

We welcome any remaining questions or feedback. Thank you again to the Area Chairs and reviewers for your time and consideration.

Best,
38154 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.