- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work proposes PBO for preference-guided multi-objective reinforcement learning with explicit objective bounds. It combines preference guidance and constraint handling through a Fisher-based projection and directional KL control. The authors provide conditional constraint-violation bounds and report improvements in preference alignment and feasibility on simulated control and offline recommendation tasks.
Critical strengths
1. PBO provides a concrete mechanism for jointly handling preference guidance and explicit objective bounds. Its combination of preference protection, slack-based bound repair, and directional KL control is a reasonable attempt to balance objective improvement against feasibility restoration.
2. The experiments show improvements in preference alignment and feasible-solution diversity across the control tasks, with particularly strong feasibility results on Reacher, Circle, and offline recommendation. The component ablations and transfer to a TRPO backbone provide additional support for the proposed design.
Critical weaknesses
1. Computational efficiency remains unverified. Appendix A.2 reports hardware and interaction budgets, but not end-to-end training time or the computational overhead of gradient computation, Fisher operations, and projection. The coefficient-space formulation may reduce curvature-computation costs relative to Fisher-based CoMOGA, so an empirical efficiency comparison could strengthen the paper. CoMOGA already provides both training-time comparisons and a breakdown of computational costs in Appendices C.3 and D.2. CoMOGA
2. The role and empirical consequences of training-time constraint enforcement are insufficiently established. §3.3 and Appendices D.3–D.4 provide conditional violation bounds, but do not establish feasibility throughout the implemented PPO training process. The paper also does not report how constraint violations evolve during training, unlike the training curves provided by CoMOGA and CR-MOPO. Since the experiments use simulated environments and offline recommendation data, it remains unclear whether intermediate feasibility is an application requirement or an optimization device. This distinction matters: restricting intermediate policies may limit exploration and final solution quality, and prior work on relaxed exploration provides empirical evidence for this trade-off. CoMOGA, §6, CR-MOPO, §6, Shperberg et al., 2024
3. The methodological contribution needs a clearer explanation of the resulting optimization behavior. The preference-balancing criterion and objective-wise protection closely follow EPO, while jointly handling objective improvement and feasibility through local constrained optimization has clear precedent in CoMOGA. The proposed combination may improve the trade-off between intermediate feasibility and final solution quality, but the final results and component-removal ablations do not directly establish this mechanism. Figure 3 varies the task bounds themselves, rather than the degree or schedule of training-time enforcement under fixed final bounds. A more direct analysis would help distinguish the contribution of the proposed constraint-handling design. EPO, CoMOGA
Questions for authors
1. Could the authors report end-to-end training time and per-update computational costs under the shared experimental setup, particularly against EPO and CoMOGA? Please also clarify the Fisher and projection implementations, and how the original baselines are adapted to the common PPO backbone.
2.1 Is PBO primarily intended to maintain feasibility during learning, or to obtain a feasible final policy? Given the simulated and offline experimental settings, what motivates restricting intermediate policies, and what training-time behavior should the conditional guarantees in §3.3 and Appendices D.3–D.4 lead us to expect?
2.2 Could the authors report constraint values or violations throughout training, including variability across seeds and preferences? These results would help assess whether the proposed bound repair and KL guard actually improve training-time constraint adherence.
3. Under fixed final objective bounds, how do different degrees or schedules of training-time relaxation affect optimization trajectories, preference alignment, and final feasible-solution quality? Comparisons could vary the margin allowance , slack penalty , or use initially relaxed or delayed bound enforcement under comparable training budgets. Experimental evidence, a controlled example, or theoretical analysis showing when the proposed design reaches better final feasible solutions would substantially strengthen its contribution.
The proposed method offers an interesting approach to addressing restricted optimization paths in constrained multi-objective optimization. I believe a clearer discussion of this issue would strengthen the research contribution beyond the reported improvements in final performance. If the authors address the concerns raised here and provide experimental analysis and insights on this issue, I would be happy to raise my rating.
9 comments
Response to Reviewer 5o4G (Q1)—-1.2 Fisher, projection, and baseline implementations.
Response 1.2: (Q1) Fisher, projection, and baseline implementations.
Fisher and projection implementation.
- Fisher: We compute Hessian–vector products of the analytic policy KL using automatic differentiation, without forming the full Fisher matrix. We construct G(F + λ_F I)Gᵀ with λ_F = 10⁻³, symmetrize it, and floor its eigenvalues at a small positive threshold to obtain the positive-definite matrix (to avoid being singular).
- Projection: Once per rollout, we solve the convex quadratically constrained quadratic program using SciPy SLSQP. It combines the preference and bound conditions, penalized slack, and the quadratic trust-region constraint. The resulting coefficients combine the advantages used by PPO. During the minibatch update of PPO, When the KL threshold is exceeded, the KL guard stops the remaining minibatches update for the current rollout. Training then continues with a new rollout.
Baseline adaptation. For baselines adapted to PPO, we use the same actor–critic architecture, rollout collection, and objective-wise value and GAE computation, while retaining each method's original mechanism for determining the policy update:
- EPO: The official EPOSearch LP computes objective weights from the gradient Gram matrix. These weights combine the advantages for clipped PPO.
- CR-MOPO: A violated bound triggers constraint rectification; otherwise, preference-weighted CAGrad computes the gradient combination used by PPO.
- Linear scalarization and Primal–dual: Fixed preference weights or adaptive constraint multipliers define the combined PPO objective.
- CoMOGA: We retain its preference-conditioned universal policy, Fisher–vector products, conjugate-gradient computation, coefficient QP, and target-policy fitting update while sharing the training backbone as all other methods.
Response to Reviewer 5o4G (Response 2: W2, Q2.1) and (Response 3: W3)
Response 2: (W2, Q2.1) Purpose of Training-Time Constraint Handling and Theoretical Guarantees
Response 2.1: Why Apply Constraints During Training?
Thanks for raising this concern. We'd like to clarify that PBO aims to obtain a high-quality final feasible policy; it does not require every intermediate policy to satisfy all bounds. The final bounds are used to guide training by specifying which constraint violations need to be reduced and how preference improvement should account for those bounds.
Introducing the bounds only later can require substantial policy adjustments, reversing earlier return gains that depended on violating them. Thus, PBO keeps these requirements active throughout learning while allowing flexibility: permits some decrease in a bounded objective that already exceeds its required minimum, and penalized slack permits relaxation of local bound conditions when they conflict with other update conditions. Temporary violations remain possible, but bound repair is considered in every update.
The experiments in Response 5 directly test this design. Moderate flexibility improves FruitTree’s final return while preserving 9/9 feasible policies. On Reacher, default enforcement produces more feasible final policies than delayed or annealed enforcement, with similar final returns.
Response 2.2: What Does the Theory Establish for the Implemented Updates?
Our theory explains how the bound conditions in PBO’s projection can reduce actual constraint violations over successive policy updates. Under the stated assumptions, it connects these conditions to the next policy’s violation and shows how slack and prediction errors limit the reduction. With the additional conditions in Appendix D.3, we expect violations to decrease over training toward a level determined by slack and errors, while temporary increases remain possible. Response 4 examines this behavior through violation and feasibility curves across seeds and preferences.
This analysis explains the roles of the projection and KL guard. The projection determines the coefficients combining objective and bound gradients, while penalizing relaxation of the local bound conditions within the same optimization problem. Appendix D.4 shows how differences between the projected direction and the actual PPO update can weaken bound repair, motivating the KL guard. Once its threshold is exceeded, the remaining PPO updates on the current rollout stop, and training continues with fresh data.
The theory thus clarifies how bound handling, slack, and update errors jointly affect training violations.
Response 3: (W3) Methodological Contribution Beyond EPO and CoMOGA.
PBO differs from EPO and CoMOGA in both the problem formulation and the policy update. We introduce a preference-guided MORL setting in which a specified preference and explicit lower or upper bounds apply to the same objective vector. This setting is not explicitly studied by EPO or CoMOGA. For example, DST’s time return is both a preference objective and subject to a lower bound. The update must account for how preference improvement affects the bounds and how bound repair affects preference alignment.
1. Beyond EPO: preference guidance with objective bounds in RL. EPO was introduced for supervised multi-task optimization without explicit objective-bound handling. PBO extends its preference-balancing criterion and objective-wise protection to RL updates that jointly handle preference improvement and bound repair, with controlled flexibility when these requirements conflict.
2. Beyond CoMOGA: preference guidance remains active during bound repair. CoMOGA focuses on environmental safety constraints; PBO directly imposes lower or upper bounds on objective returns, which may also participate in preference following. CoMOGA combines objective improvement and safety constraints when feasible, but switches to safety-only recovery when constraints are violated. PBO retains the preference reference and preservation conditions during bound repair, using the same projection structure for feasible and infeasible policies.
Therefore, PBO goes beyond combining EPO’s preference criterion with CoMOGA’s constraint handling: it changes how preference improvement and bound repair interact within each update. When a bound is satisfied, permits some change in that objective to improve preference following; when requirements conflict, penalized slack permits relaxation and temporary violations. The final bounds continue to guide correction throughout learning.
The following Responses 4 and 5 further provide training trajectories and controlled studies varying α, slack penalties, and enforcement schedules under fixed final bounds. These directly examine the resulting preference alignment, training violations, and final policy quality.
Response to Reviewer 5o4G (Response 1: W1, Q1)—-1.1 Training time and computational costs.
We thank the reviewer for the careful reading and constructive suggestions. Your comments have helped us strengthen the evaluation and clarify the method. We address each concern below and would be happy to provide further clarification or discuss any remaining questions.
Response 1: (W1, Q1) Computational Efficiency and Implementation.
Response 1.1: (W1, Q1) Training time and computational costs.
Thank you for this thoughtful suggestion. To answer your question, we present the following results.
End-to-end training time. We evaluate all methods on a single NVIDIA A100 GPU with the same backbone. Since CoMOGA trains a universal policy, we use the same number of environment steps and the same evaluation schedule as the other methods to compare computational costs under a fixed budget. The table reports mean ± sample standard deviation across three paired seeds, in minutes.
PBO’s total runtime remains comparable to the baselines despite its additional constraint handling. Its mean time is lowest on DST and FruitTree and comparable to the baselines on Circle. On Reacher, PBO takes 15.67 minutes, approximately one additional minute relative to EPO.
The KL guard can reduce training time by stopping PPO updates early. When the projected direction may reduce the margin of the most critical bound, PBO uses a tighter KL threshold. Once that threshold is reached, it stops the remaining PPO updates on the current rollout. This limits deviation from the projected direction and can reduce the number of minibatches and epochs executed. Thus, although PBO adds Fisher computation and projection, it can perform fewer policy updates.
Computational breakdown. We measure the complete policy/value optimization stage and individual operations within each update. The tables report means across three seeds. Times are in milliseconds per update, and memory is peak allocated GPU memory in MiB.
DST
FruitTree
Reacher
Circle
PBO’s projection requires only a small fraction of optimization time. Its solver takes 1.27–3.21 ms per update, approximately 0.2–2.6% of the complete optimization time. Fisher computation takes 3.20–5.89 ms, compared with 16.04–126.56 ms for CoMOGA in these implementations. These results support the computational benefit of solving a single projection over gradient coefficients. PBO’s peak allocated GPU memory is close to the other methods.
The breakdown also explains why total runtimes can be similar despite different optimization costs. On Circle, rollout collection and evaluation take approximately 53 minutes, while the complete policy/value optimization stage takes only 1–2 minutes. PBO’s KL guard can also stop repeated PPO updates early, reducing optimization time despite its additional Fisher computation and projection.