- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates the observation uncertainty and the variant environment in MORL. To this end, the paper proposes a Preference-Drift-Aware approach for learning a Robust Pareto Set (DROPS). DROPS includes several components, such as drift-aware trust region for each preference, hierarchical adversarial perturbations, and contrastive drift alignment. Extensive numerical results on MO-MuJoCo are reported.
Critical strengths
This paper considers an interesting and emerging scenario in MORL. Besides, the numerical results are rich.
Critical weaknesses
- My main concern is that the paper remains largely heuristic, and no theoretical analysis is provided to ensure the correctness of the approaches and claims.
- Even though the problem is interesting (variant environment and uncertain observations), the solutions seem insufficient to address it. Specifically: (1) the policy class considered in the paper remains a stationary policy class, which cannot characterize the dynamics of the environment; (2) when the observation is uncertain, the partially observable MDP (POMDP), rather than the conventional MDP, is usually the better model; and (3) although the authors claim that "the formulation does not require the policy to solve the general belief state POMDP", they do not provide clear support.
- Most experiments only include 2 objectives, and the MO-MuJoCo setting is restrictive, making it inconvincible to verify the effectiveness of the approach in large-scale or practical applications. An LLM-based experiments with more objectives would strengthen the experiments, and validate the scalability of the method.
Questions for authors
Please try to address the Weaknesses.
Moreover, could the authors provide the codes of experiments?
We thank the reviewer. We have expanded the theory, clarified partial observability, and added higher-objective continuous-control experiments. The experimental code is provided in the ***Supplementary Material*** with detailed instructions.
We thank the reviewer. We have expanded the theory, clarified partial observability, and added higher-objective continuous-control experiments. The experimental code is provided in the Supplementary Material with detailed instructions.
Q1. Theoretical justification. To address your concern, we have substantially strengthened the theoretical analysis by providing rigorous derivations, clarifying the conditions under which the results hold, and establishing additional propositions. Specifically, the revisions include:
Q2. Stationary policies and POMDP modeling.
Stationarity. DROPS learns a family of stationary, preference-conditioned reactive policies, not an explicitly time-varying controller. However, the observation mechanism and transition kernel vary with the uncertainty realization. Proposition 2 establishes that when an environment configuration is sampled at reset and remains fixed within an episode, the expected return is an average of configuration-conditioned returns, which generally differs from evaluation using an averaged transition kernel.
Partial observability. We clarify that observation corruption motivates a POMDP. Sections 2 and 3.2 explicitly describe uncertainty-induced partially observable environments using latent states, observation mechanisms, and environment-dependent transitions. The restriction concerns the policy class: DROPS uses reactive observation-conditioned policies within this model, without assuming that corrupted observations reveal the latent state.
Belief states. Proposition 1 proves that under conditionally memoryless observations and a fixed episode configuration, a reactive policy induces an effective latent-state action kernel and Markov transition kernel. The corresponding value function satisfies a Bellman equation. This supports executing and evaluating the specified policy without explicit belief-state inference, not solving the general POMDP.
Q3. Higher-objective evaluation. The existing study includes three-objective MO-Hopper-v3 and six-objective FTN. To strengthen validation in continuous control, we added Appendix "Higher-Objective Continuous-Control Evaluation", covering MO-Ant-v5 (3 objectives) and MO-Lunar-Lander-Continuous-v3 (4 objectives). At uncertainty intensity ϑ=1, we compare DROPS against Hyper-MORL, PD-MORL, PSL-MORL, and PG-MORL using step-level observation and episode-level environment perturbations over seven independent runs. DROPS obtains the highest mean HV and the lowest mean preference-drift U on both tasks. These results support the broader continuous-control applicability of our proposed DROPS.