acceptodds
Under review as a conference paper at ICLR 2027

DROPS: Robust Pareto Set Learning under Preference Drift for Multi-Objective Robotic Control

Abstract

Multi-objective Robotic Control is increasingly important in real-world applications, with Multi-Objective Reinforcement Learning (MORL) providing a promising approach for learning policies over conflicting objectives. However, existing MORL methods typically assume clean observations and fixed environment dynamics. These assumptions often fail in practice, where uncertainty can cause severe **_preference drift_**, substantially undermining the robustness of learned control policies. In this paper, we propose **DROPS**, a Preference-Drift-Aware approach for learning a Robust Pareto Set of control policies against uncertainty. Specifically, DROPS constructs **_(i) a drift-aware trust region_** for each preference, introduces **_(ii) hierarchical adversarial perturbations_** to establish a degradation bound with a policy-independent coefficient, and leverages **_(iii) contrastive drift alignment_** to recover robust preference-to-policy mappings. Central to our method is strengthening the robustness of the learned policy Pareto Set by tightening the degradation bound and aligning drifted preferences, enabling DROPS to learn a Pareto Set robust to preference drift. Experiments on five continuous robotic-control tasks, together with a discrete benchmark, show its consistently superior performance in obtaining robust control policies.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Preference Drift, Robust Pareto Set Learning, Multi-Objective Robotic Control

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_V6j8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the observation uncertainty and the variant environment in MORL. To this end, the paper proposes a Preference-Drift-Aware approach for learning a Robust Pareto Set (DROPS). DROPS includes several components, such as drift-aware trust region for each preference, hierarchical adversarial perturbations, and contrastive drift alignment. Extensive numerical results on MO-MuJoCo are reported.

Critical strengths

This paper considers an interesting and emerging scenario in MORL. Besides, the numerical results are rich.

Critical weaknesses
  • My main concern is that the paper remains largely heuristic, and no theoretical analysis is provided to ensure the correctness of the approaches and claims.
  • Even though the problem is interesting (variant environment and uncertain observations), the solutions seem insufficient to address it. Specifically: (1) the policy class considered in the paper remains a stationary policy class, which cannot characterize the dynamics of the environment; (2) when the observation is uncertain, the partially observable MDP (POMDP), rather than the conventional MDP, is usually the better model; and (3) although the authors claim that "the formulation does not require the policy to solve the general belief state POMDP", they do not provide clear support.
  • Most experiments only include 2 objectives, and the MO-MuJoCo setting is restrictive, making it inconvincible to verify the effectiveness of the approach in large-scale or practical applications. An LLM-based experiments with more objectives would strengthen the experiments, and validate the scalability of the method.
Questions for authors

Please try to address the Weaknesses.

Moreover, could the authors provide the codes of experiments?

1 comment
AuthorsOpenReview ↗

We thank the reviewer. We have expanded the theory, clarified partial observability, and added higher-objective continuous-control experiments. The experimental code is provided in the ***Supplementary Material*** with detailed instructions.

We thank the reviewer. We have expanded the theory, clarified partial observability, and added higher-objective continuous-control experiments. The experimental code is provided in the Supplementary Material with detailed instructions.

Q1. Theoretical justification. To address your concern, we have substantially strengthened the theoretical analysis by providing rigorous derivations, clarifying the conditions under which the results hold, and establishing additional propositions. Specifically, the revisions include:

  • Section 3.1 and revised Appendix "DROPS Training Details": We establish a conditional guarantee for trust-region membership under local Lipschitz continuity, explicitly relating preference perturbation magnitude to the policy-divergence threshold. This provides a quantitative interpretation of the trust-region construction while distinguishing theoretical guarantees from empirical minibatch-based selection.
  • Section 3.2 and revised Appendix "Theoretical Analysis of Preference Drift": We provide a detailed Bellman-based derivation of the objective-wise return degradation bound, whose coefficient is independent of the policy parameterization. Building on Definition 1, we derive an explicit upper bound on preference drift under local inverse stability. These analyses clarify the theoretical connection between uncertainty-induced return degradation and preference drift, together with its applicability and limitations.
  • Added Appendix "Theoretical Analysis of Contrastive Pairing": We establish quantitative behavioral separation between in-region and out-of-region preferences through KL-based pairing. We further derive the gradients and similarity-margin properties of the InfoNCE objective to characterize how contrastive alignment regulates trajectory representations. Importantly, we clarify that embedding alignment serves as a tractable surrogate for preference correspondence regularization.
  • Added Appendix "Reactive Policy Evaluation under Partial Observability": We establish two propositions addressing the theoretical validity of reactive policies under uncertain observations and varying environments. Proposition 1 proves that, under a memoryless observation kernel and a fixed episode-level configuration, a reactive policy induces an effective Markov transition kernel and admits latent-state Bellman evaluation without explicit belief-state inference. Proposition 2 characterizes policy evaluation across persistent episode-level environment configurations and demonstrates why an averaged transition kernel cannot generally represent their effects.

Q2. Stationary policies and POMDP modeling.

  • Stationarity. DROPS learns a family of stationary, preference-conditioned reactive policies, not an explicitly time-varying controller. However, the observation mechanism and transition kernel vary with the uncertainty realization. Proposition 2 establishes that when an environment configuration is sampled at reset and remains fixed within an episode, the expected return is an average of configuration-conditioned returns, which generally differs from evaluation using an averaged transition kernel.

  • Partial observability. We clarify that observation corruption motivates a POMDP. Sections 2 and 3.2 explicitly describe uncertainty-induced partially observable environments using latent states, observation mechanisms, and environment-dependent transitions. The restriction concerns the policy class: DROPS uses reactive observation-conditioned policies within this model, without assuming that corrupted observations reveal the latent state.

  • Belief states. Proposition 1 proves that under conditionally memoryless observations and a fixed episode configuration, a reactive policy induces an effective latent-state action kernel and Markov transition kernel. The corresponding value function satisfies a Bellman equation. This supports executing and evaluating the specified policy without explicit belief-state inference, not solving the general POMDP.

Q3. Higher-objective evaluation. The existing study includes three-objective MO-Hopper-v3 and six-objective FTN. To strengthen validation in continuous control, we added Appendix "Higher-Objective Continuous-Control Evaluation", covering MO-Ant-v5 (3 objectives) and MO-Lunar-Lander-Continuous-v3 (4 objectives). At uncertainty intensity ϑ=1\vartheta=1, we compare DROPS against Hyper-MORL, PD-MORL, PSL-MORL, and PG-MORL using step-level observation and episode-level environment perturbations over seven independent runs. DROPS obtains the highest mean HV and the lowest mean preference-drift U on both tasks. These results support the broader continuous-control applicability of our proposed DROPS.

Reviewer_LhwPOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DROPS for learning preference-conditioned policies that remain robust to observational and environmental uncertainty. It introduces preference drift as a change in the correspondence between requested preferences and realized returns. DROPS combines a policy-based trust region, hierarchical adversarial perturbations, contrastive trajectory alignment, and diversity regularization within a hypernetwork-based policy generator. Experiments evaluate Pareto quality and preference robustness across continuous-control and discrete benchmarks.

Critical strengths

The paper identifies a meaningful problem in MORL. Preserving the overall quality of a Pareto set does not necessarily preserve the correspondence between individual preferences and their realized outcomes. Explicitly examining this correspondence under uncertainty provides a valuable perspective on robustness beyond aggregate Pareto-set metrics. The empirical evaluation is substantial. The paper considers multiple continuous-control tasks, a six-objective discrete benchmark, several relevant learned baselines, and seven random seeds for the main comparisons. Component ablations, sensitivity analyses, and additional nominal-environment results provide useful supporting evidence. The reported results are promising and consistent across tasks. DROPS achieves the highest mean hypervolume and lowest mean preference-drift metric in the main comparisons, with competitive spacing results. The loss ablations also support the practical contribution of the individual components to the combined method.

Critical weaknesses
  1. The rationale for the contrastive pairing needs clearer explanation. Equation (6) encourages the anchor trajectory embedding to resemble that of an out-of-region preference, while separating it from embeddings of in-region preferences. These samples are selected according to policy KL divergence, which does not directly identify which preference corresponds to the anchor’s uncertainty-induced drift. Since different preferences can legitimately require different behaviors, it is unclear why this particular pairing should restore the intended correspondence. The ablation supports the usefulness of the loss, but further reasoning would help connect its effect to the drift definition.

  2. The conditions supporting the theoretical bound require clarification. The conversion from return degradation to preference drift assumes that distinct preferences produce sufficiently distinguishable nominal returns. This condition may fail in regions where multiple preferences select the same policy and obtain identical returns, as can occur in discrete tasks such as FTN. The paper should clarify the regions to which the assumption and resulting bound apply.

  3. The positioning relative to existing uncertainty-aware methods could be clearer. The related-work discussion briefly summarizes several robust and adversarial RL approaches, but provides limited comparison of their uncertainty assumptions and robustness objectives with those of DROPS. A more explicit discussion of the closest robust MORL methods would help establish what is distinctive about the proposed preference-drift formulation and alignment mechanism.

  4. The title in the PDF contains an unreadable character.

Questions for authors
  1. Could the authors provide further reasoning or evidence explaining why the contrastive pair construction in Eq. (6) corrects uncertainty-induced preference drift while preserving legitimate differences between preferences?

  2. How does the theoretical robustness argument apply to regions where several preferences share the same nominal return?

  3. Could the authors expand the discussion of recent robust and uncertainty-aware RL/MORL literature and more explicitly position DROPS relative to the closest approaches?

1 comment
AuthorsOpenReview ↗

We thank the reviewer for the constructive comments. We have strengthened the theoretical analysis, conducted additional experiments, and clarified the positioning of DROPS.

We thank the reviewer for the constructive comments. We have strengthened the theoretical analysis, conducted additional experiments, and clarified the positioning of DROPS. Our revisions are detailed below.

Q1: We clarify that KL divergence does not directly identify the preference corresponding to uncertainty-induced drift and out-of-region preferences serve as behaviorally distinct alignment candidates.

  • More Reasoning: In Appendix B (Theoretical Analysis of Contrastive Pairing), we provide a detailed theoretical analysis of the proposed pairing strategy. First, we establish the behavioral separation induced by KL-based pairing through the mean-squared action discrepancy of auxiliary Gaussian-smoothed policies, explaining how in-region and out-of-region preferences are distinguished at the policy level. Second, we derive the gradients and similarity-margin properties of InfoNCE to characterize how the contrastive objective encourages alignment with out-of-region candidates while maintaining discrimination against in-region references. Importantly, we clarify that these properties characterize embedding-space optimization rather than guaranteeing a reduction in the preference-drift distance defined in Definition 1.

  • More Evidence: We introduce a new experiment, “How does the contrastive pairing strategy affect preference drift and Pareto Set quality?”, comparing DROPS with Distance-only and Reversed pairing across five MO-MuJoCo tasks. DROPS consistently achieves the highest HV and lowest U, reducing average U by 49.4% and 38.7%, respectively. These results provide additional empirical evidence supporting the effectiveness of the proposed pairing strategy.

Q2: We have substantially expanded Appendix A (Theoretical Analysis of Preference Drift) to clarify the theoretical assumptions, derivation, and applicability of the preference-drift bound.

  • Inverse Stability and Return Identifiability: We introduce a local return-separation modulus to characterize the inverse-stability condition. We demonstrate that this condition fails when distinct preferences produce identical nominal returns and further provide a mathematical counterexample showing that injectivity alone is insufficient to ensure inverse stability.

  • Return-Space Bound without Inverse Stability: We distinguish two levels of theoretical guarantees. The return-space bound does not require inverse stability but remains subject to the assumptions underlying the objective-wise degradation bound. In contrast, converting this result into a preference-space bound additionally requires positive local return separation, with both the anchor preference and its closest nominal inverse preference lying within the corresponding region. This clarification explicitly identifies the conditions under which return degradation can be related to preference drift.

  • Applicability and Limitations, we discuss the theoretical scope in discrete tasks such as FTN, where multiple preferences may induce identical policies and nominal returns. We also clarify that reward perturbations may violate the shared-reward assumption required by the degradation analysis. Therefore, although the preference-drift definition and empirical metric remain applicable, the derived bound is conditional rather than universally guaranteed.

Q3: We have reorganized Related Works to provide a clearer comparison of uncertainty assumptions, robustness objectives, and preference correspondence. In particular, we acknowledge that RMORL already considers both observational and environmental uncertainties. We also distinguish methods addressing dynamically changing preferences from our setting, where the requested preference remains fixed while uncertainty alters its realized return correspondence. Unlike approaches primarily targeting policy robustness or Pareto Set approximation, DROPS explicitly formulates preference drift through the inverse nominal return correspondence and introduces contrastive alignment as an embedding-based regularizer. These revisions clarify the distinctive problem formulation and alignment mechanism of DROPS without overstating its theoretical guarantees.

Q4: We appreciate the reviewer’s attention to this detail. The character is actually a small quadruped robot icon, intentionally included in the title to reflect the robotic-control setting of our work. It is a graphical element rather than an unreadable text character or an encoding error. We apologize for any confusion caused by its appearance in the PDF.

Reviewer_P4p4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper defines preference drift as the phenomenon in which a preference-conditioned policy realizes a trade-off different from the requested preference under uncertainty such as observation perturbations or environmental changes. The magnitude of drift is measured by identifying the preference whose return in the nominal (unperturbed) return map is closest to the return obtained under uncertainty, and then measuring its distance from the requested preference. The proposed method, DROPS, augments hypernetwork-based Pareto Set Learning with three components: (i) sampling nearby and distant preferences using a trust region defined by the KL divergence between policies; (ii) reducing the upper bound on performance degradation through an adversarial loss for step-level observation perturbations and episode-level environmental perturbations; and (iii) aligning the preference-to-policy mapping through an InfoNCE-based contrastive loss and a KDE-entropy-based diversity loss. The paper reports better results than existing MORL methods in terms of Hypervolume (HV) and the preference drift metric (U) on five MO-MuJoCo tasks and the discrete FTN benchmark.

Critical strengths

S1. Formulation of Preference Drift under Uncertainty

While prior robust MORL studies primarily focus on preventing return degradation under perturbations, this paper identifies the misalignment between the requested preference and the realized trade-off correspondence as a distinct problem. By mathematically defining this drift as the distance to the preference that yields the closest nominal return to the uncertain return, the authors introduce a metric that captures displacement in the preference space rather than mere degradation in the objective space. Preserving the user-specified trade-off during practical deployment of multi-objective policies is an important yet underexplored issue, making the problem formulation itself valuable.

S2. Temporal Hierarchy of Uncertainties and Consistent Empirical Results

Decomposing uncertainty into step-level observation corruption (ν\nu) and episode-level fixed environment parameter variation (ξ\xi) allows the framework to bound the performance degradation of reactive policies via separate policy and transition deviation terms without solving a full belief-state POMDP. Empirically, DROPS achieves the best results in terms of HV and U across all five MO-MuJoCo tasks and all three depths of FTN. Moreover, evaluations in non-perturbed environments show HV performance comparable to or better than baselines, demonstrating that robustness is achieved without substantially sacrificing nominal performance.

Critical weaknesses

W1. Additional analysis of contrastive drift alignment

Table 3 provides useful evidence that the contrastive component contributes to the overall performance. Explaining how the out-of-region positive and in-region negatives preserve the correspondence in Definition 1 would help connect the training objective to the stated notion of preference drift. A comparison with aligning nominal and perturbed trajectories under the same preference, together with an analysis of the interaction between the contrastive and diversity losses, would further clarify the role of this design.

W2. Further clarification of the theoretical analysis and experimental reporting

Providing a derivation of the preference-drift bound, specifying the region of the inverse-stability assumption, and discussing its applicability to the FTN setting would help clarify the scope of the theoretical results. Explaining which terms can be affected by policy optimization and illustrating the bound at the experimental discount factor would help relate the analysis to the training objective. Stating the combined loss and explaining how the coefficient in Table 7 applies to its components, together with confidence intervals or seed-level comparisons for the preference-drift metric, would make the implementation and empirical comparisons easier to assess.

Questions for authors

Q1. Handling of δp(u)\delta_{p}(u) in Ladv\mathcal{L}_{\mathrm{adv}} and Actual Implementation of the Episode-level Adversary

Since δp(u)\delta_{p}(u) is independent of the policy parameters Θ\Theta and cannot be reduced via policy optimization, does the claimed effect of “tightening the degradation bound” apply solely to the δπ\delta_{\pi} term? If so, defense against environmental variation ξ\xi relies entirely on LRL(Θ;u)\mathcal{L}_{\mathrm{RL}}(\Theta;u) inside max⁡u∈U\max_{u\in\mathcal{U}}. How does the episode-level adversary actually search for the worst-case ξ\xi? Given that Table 5 and Table 6 in Appendix A.3 describe the perturbations as uniform sampling ϵ∼U(⋅,⋅)\epsilon\sim\mathcal{U}(\cdot,\cdot), is the actual implementation based on domain randomization rather than adversarial perturbation?

Q2. Validity of Lipschitz-based Sampling for ωi\omega_{i} and Definition of Policy Divergence

While ωo\omega_{o} is explicitly filtered by checking Dˉ(πω~o∥πω)>ρ\bar{D}(\pi_{\tilde{\omega}_{o}}\Vert\pi_{\omega}) > \rho, ωi\omega_{i} is generated simply by adding noise satisfying ∥ϵ∥2≤ρ/L\lVert\epsilon\rVert_{2}\leq\rho/L without verifying the KL divergence. How were the unknown Lipschitz constant LL and radius ρ\rho chosen in practice, and have you measured what fraction of generated ωi\omega_{i} samples actually reside within the trust region? Furthermore, judging by the hyperparameters in Table 7 (e.g., policy update delay, noise clipping), the underlying RL algorithm appears to use a deterministic actor (TD3-style); how are the KL divergence in Dˉ\bar{D} and the total variation distance in δπ\delta_{\pi} computed for deterministic policies?

8 comments
AuthorsOpenReview ↗

We thank the reviewer for the constructive comments. We have strengthened the theoretical analysis and clarified the implementation of DROPS.

We thank the reviewer for the constructive comments. We have strengthened the theoretical analysis and clarified the implementation of DROPS. Our revisions are detailed below.

W1: Connection to Definition 1. We have clarified the relationship between contrastive drift alignment and preference drift in Section 3.3 and added Appendix B. Definition 1 characterizes preference drift through the inverse correspondence between uncertain and nominal returns, and the contrastive objective provides a tractable behavioral surrogate. Appendix B establishes the behavioral separation induced by KL-based pairing through the mean-squared action discrepancy of Gaussian-smoothed policies.

W1: Pairing strategy and diversity interaction. We added experiments comparing DROPS with Distance-only and Reversed pairing in Section 4.2 (Table 4). DROPS achieves the highest HV and lowest U across all five MO-MuJoCo tasks, providing empirical evidence supporting the proposed pairing strategy. We also extended Table 3 with joint ablations of the contrastive and diversity losses. These results indicate an empirical interaction between the two components.

W2: Derivation and theoretical scope. We have expanded Appendix A to provide the complete derivation of the preference-drift bound. Combining it with Definition 1, the triangle inequality, and local inverse stability yields dPD(ω;u)≤2mκμ[δπ(ω,u)+δp(u)]d_{\mathrm{PD}}(\omega;u) \leq \frac{2\sqrt{m}\kappa}{\mu} [\delta_\pi(\omega,u)+\delta_p(u)] and κ=2Rmax⁡(1−γ)2\kappa=\frac{2R_{\max}}{(1-\gamma)^2}. We also derive a return-space bound that does not require inverse stability and clarify the limitations of converting return displacement into preference-space drift.

W2: Relation to optimization. For a fixed uncertainty realization, δp(u)\delta_p(u) describes exogenous transition discrepancies and cannot be directly reduced through policy optimization. In contrast, δπ(ω,u)\delta_\pi(\omega,u) characterizes policy-dependent sensitivity and motivates the policy-consistency regularizer. The inverse-stability constant is not directly optimized. These distinctions are clarified in Section 3.2 and Appendix A.

W2: Overall training objective and statistical reporting. We add Section 3.5 to explicitly define the overall training objective. Appendix D (Table 8) specifies the KL, contrastive, and diversity coefficients as λ=15\lambda=15, λcl=1\lambda_{cl}=1, and λds=1\lambda_{ds}=1. Furthermore, we report approximate 95% Student-t confidence intervals for U based on seven independent runs, improving statistical reporting without implying unestablished significance.

Q1: Handling of δp(u)\delta_p(u) and Episode-Level Adversary. Although δp(u)\delta_p(u) is independent of policy parameters for a fixed uncertainty realization and cannot be directly reduced through policy optimization, environmental variations affect training through the RL objective and the policy-consistency regularizer targets policy-dependent sensitivity. We clarify this distinction: Section 3.2 identifies the inner maximization as an idealized worst-case objective, and Appendix C (Tables 6–7) specifies that episode-level configurations are sampled from predefined ranges and remain fixed within each episode.

Q2: Lipschitz condition and sampling. We have corrected the sufficient condition in Section 3.1 to |ϵ2\epsilon_2|≤ρ/L\leq\rho/L, subject to the specified local neighborhood. Appendix D consolidates the sampling and divergence evaluation procedures and specifies the trust-region threshold ρ=0.05\rho=0.05 and a local empirical Lipschitz estimate L^ω\widehat{L}_\omega, constructed from sampled preference-policy pairs with a safety factor cL=1.25c_L=1.25. Since this estimate is not a certified Lipschitz constant, candidates are additionally filtered through minibatch-based divergence tests, and these tests verify membership under the empirical divergence measure.

Q2: Policy divergence for deterministic actors. We agree that deterministic Dirac action distributions generally have infinite KL divergence when their actions differ. Appendix D therefore introduces an auxiliary Gaussian-smoothed policy with fixed covariance. Its KL divergence equals the squared difference between deterministic actor outputs divided by 2σdiv22\sigma_{\mathrm{div}}^2. The actor remains deterministic, and the Gaussian representation is used solely for divergence evaluation.

Q2: Implementation and theoretical limitations. Appendix D further clarifies minibatch-based divergence estimation, policy-consistency regularization, and the relationship to total variation distance. Pinsker's inequality applies to auxiliary Gaussian-smoothed distributions, not directly to the original deterministic policies. Thus, the resulting KL provides a tractable behavioral regularizer, and the return-degradation bound remains conditional on the assumptions stated in Section 3.2 and Appendix A .

Thank you for the thoughtful rebuttal. The revision addresses most of my theoretical concerns by deriving the preference drift bound and stating in the main text that inverse stability must hold on a region containing both the requested preference and its nearest nominal match. It also specifies the combined loss and its coefficients and clarifies that episode-level perturbations are sampled from a fixed distribution.

It remains unclear why contrastive drift alignment pulls the out-of-region preference ωo\omega_o toward the anchor and pushes the in-region preference ωi\omega_i away. The appendix acknowledges that this loss does not guarantee reduced drift. The only direct evidence for this choice comes from comparisons with Distance-only and Reversed pairing. Their fairness is difficult to assess because the paper does not specify whether the variants used the same temperature and number of negatives as DROPS. It also reports no sensitivity analysis for the contrastive loss weight, temperature, or ρ\rho.

For the deterministic actor used by DROPS, even a slight change in action under an observation perturbation yields the maximum total variation distance of 11. Describing the degradation bound as conservative understates how little information it provides in this setting. On all five MO-MuJoCo tasks, the 95% confidence interval for DROPS’s preference drift UU overlaps substantially with that of the baseline with the second-lowest mean drift. These intervals alone do not establish a statistical advantage for DROPS.

Given the sampling procedure and the bound’s limited applicability to deterministic actors, I suggest qualifying the abstract’s claim that hierarchical adversarial perturbation tightens the bound and adjusting the contribution summary accordingly. When discussing the Lipschitz guarantee for generating ωi\omega_i, the main text could also mention the divergence check used to filter candidates, currently described only in the appendix.

Thank you again for the care you took in responding to my comments.

AuthorsOpenReview ↗

Response to Reviewer P4p4 regarding Remaining Questions of DROPS (1/4)

A. Rationale and Theoretical Interpretation of Contrastive Drift Alignment

We sincerely thank the reviewer for the continued attention to the rationale behind our contrastive pairing strategy. We have strengthened this distinction in Section 3.3 and Appendix B.

1. Motivation for asymmetric contrastive pairing. Our pairing strategy is designed to regularize preference-conditioned trajectory representations according to policy-level behavioral discrepancies, rather than preference-space proximity alone. The trust region Tω(ρ)T_\omega(\rho) distinguishes candidates using the expected policy divergence from the anchor preference ω\omega. For deterministic actors, the auxiliary Gaussian-smoothed divergence satisfies

[\|a_{\omega'}(s)-a_\omega(s)\|_2^2]}{2{\sigma}^2}.$$ Accordingly, an accepted in-region candidate exhibits an empirical mean-squared action discrepancy no greater than $2\sigma^2\rho$, whereas an accepted out-of-region candidate exceeds this threshold. This provides a behavioral criterion for distinguishing locally similar policies from more divergent candidates. The out-of-region preference $\omega_o$ is therefore treated as a ***hard alignment candidate***. Because its policy differs more substantially from the anchor under the divergence criterion, encouraging similarity between their uncertain trajectory embeddings introduces an alignment objective across behaviorally distinct preferences. In contrast, the in-region preferences $\omega_i$ serve as local references for maintaining discrimination among preference-conditioned representations. Assigning them as negatives encourages the encoder to retain distinctions within the local region. Importantly, this does not require nearby policies to produce different actions or returns. The contrastive objective operates on trajectory embeddings and regulates their relative similarities. This asymmetric assignment deliberately differs from conventional proximity-based contrastive pairing. Its purpose is to construct a challenging alignment objective while retaining local representation discrimination. The diversity regularizer $\mathcal L_{ds}$ further complements this objective by discouraging degenerate concentration of trajectory embeddings across the learned Pareto Set. **2. Theoretical interpretation.** We have clarified the optimization behavior of InfoNCE in ***Appendix B***. Let $s_o=\mathcal S(z,z_o)$ denote the anchor-positive similarity and $s_i^j=\mathcal S(z,z_i^j)$ denote the anchor-negative similarities. The contrastive objective can be rewritten as $$\mathcal L_{cl}= \log\left(1+\sum_{j=1}^{K} \exp\left(\frac{s_i^j-s_o}{t}\right)\right).$$ Its partial derivatives satisfy $\partial\mathcal L_{cl}/\partial s_o=-(1-q_o)/t\leq0$ and $\partial\mathcal L_{cl}/\partial s_i^j=q_i^j/t\geq0$. These relationships establish that minimizing the objective favors greater anchor-positive similarity relative to anchor-negative similarities. Thus, the theoretical analysis characterizes the intended direction of contrastive optimization and the resulting relative-similarity preference. $\mathcal L_{cl}$ is an embedding-level surrogate for regularizing the preference-conditioned mapping, rather than a direct optimization of $d_{\mathrm{PD}}(\omega;u)$. We explicitly distinguish this optimization property from a guarantee of preference-drift reduction. **3. Empirical support and manuscript clarification.** The pairing comparisons in Table 4 provide complementary empirical evidence for the asymmetric assignment. On MO-HalfCheetah-v2, DROPS achieves HV/U of $2.45/0.18$, compared with $1.59/0.36$ for Distance-only and $1.83/0.28$ for Reversed pairing. The proposed strategy also achieves higher mean HV and lower mean U across the remaining four MO-MuJoCo tasks. Furthermore, the joint ablation results in Table 3 show that removing the contrastive and diversity components degrades both Pareto Set quality and preference robustness, supporting their complementary empirical contributions. We have refined Section 3.3 to describe $\omega_o$ as a behaviorally distinct alignment candidate and $\omega_i$ as local discriminative references. Appendix B now explicitly separates KL-based behavioral characterization and InfoNCE optimization properties. We sincerely appreciate the reviewer's concern. The revised explanation clarifies ***why the asymmetric pairing is adopted and what its contrastive objective mathematically encourages***, providing a more precise interpretation of the proposed mechanism.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.