- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes OW-CMORL for multi-objective RL under time-varying operating conditions. The method encodes recent context history and forecasts with a Transformer, retrieves previously stored MORL policy snapshots from an expert bank, performs short local refinement, and updates an online Pareto archive. The paper also evaluates the method on four dynamic control domains.
Critical strengths
- The paper considers a practically relevant problem: maintaining good multi-objective policies under changing operating conditions.
- I think the overall system is fairly complete, with context inference, memory, retrieval, local adaptation, and Pareto archive maintenance.
- The experiments cover four control domains and include several useful ablations on the context window, expert bank, and retrieval mechanism.
Critical weaknesses
-
The methodological novelty appears quite limited. Dynamic MORL with time-varying Pareto sets and online refinement is already studied in MORL-CA [1]. Continual MORL and catastrophic forgetting are explicitly studied in CORe3 [2], while Q-Pensieve [3] already introduces snapshot-based knowledge reuse in MORL. Context-driven non-stationarity and catastrophic forgetting without task labels are also studied in LCPO [4]. Against these works, OW-CMORL mainly adds a Transformer context encoder, context-indexed snapshot retrieval, HNSW, and short local refinement. In fact, the paper itself describes OW-CMORL as an “orchestration layer rather than a replacement for the MORL learner.”
-
I also find the theoretical contribution weak. The main expectation statement in Theorem 1 essentially follows directly from Assumption A3, which already assumes the corresponding embedding population error is bounded. Theorem 3 similarly assumes the relevant loss decomposition and monotonic decrease of eviction loss with bank capacity, and then derives the corresponding long-run average bound. These results seem closer to sanity checks for the proposed architecture than substantive theoretical guarantees.
-
The “implicit context inference” claim seems overstated. OW-CMORL directly receives the environment-side dynamic-factor vector z_t; what is unavailable is only an explicit stage/change-point label. This is considerably easier than inferring a latent operating regime from observations alone.
-
Theorem 2 appears incorrect as stated, beyond the factor-of-two issue. Its assumptions B1–B4 do not assume that the retrieved snapshot is optimal at its original context. However, the proof suddenly uses J(hat{pi};hat{z}) = J(pi^*{hat{z}};hat{z}), i.e. exact optimality of the retrieved policy. But the expert bank stores arbitrary training-stage snapshots, not guaranteed optimal policies. Without an additional source-context suboptimality term, the claimed bound does not follow.
Overall, I see a useful engineering combination of existing ideas, but not a sufficiently new problem formulation or learning principle for ICLR.
[1] Gan et al., “MORL-CA: Dynamic Multi-Objective Reinforcement Learning for Chlor-Alkali Process Optimization Under Time-Varying Conditions,” IJCAI 2026. [2] Li et al., “Continual Multi-Objective Reinforcement Learning via Reward Model Rehearsal,” IJCAI 2024. This work formalizes continual MORL and explicitly addresses adaptation and catastrophic forgetting. IJCAI [3] Hung et al., “Q-Pensieve: Boosting Sample Efficiency of Multi-Objective RL Through Memory Sharing of Q-Snapshots,” ICLR 2023. [4] Hamadanian et al., “Online Reinforcement Learning in Non-Stationary Context-Driven Environments,” ICLR 2025.
Questions for authors
- What is the main methodological novelty over MORL-CA? Both methods address changing operating conditions, time-varying Pareto solutions, and online refinement. Is the main difference essentially context-based retrieval of historical experts?
- How is the expert bank fundamentally different from existing snapshot/memory reuse mechanisms such as Q-Pensieve [2], beyond indexing snapshots by an environmental-context embedding?
- What substantive guarantee does Theorem 1 provide beyond Assumption A3, which already assumes that the embedding mismatch is bounded? Similarly, what does Theorem 3 establish beyond the assumed loss decomposition and monotonicity of the eviction loss?
3 comments
Thank you for clarifying how OW-CMORL differs from MORL-CA and Q-Pensieve. My main concern about Theorem 2 remains unresolved. Even with an exact context match and no fine-tuning, a retrieved snapshot can still be suboptimal, yet the stated bound is zero. I appreciate the clarification on the method, but my concerns about the scope of the contribution and the correctness of the bound lead me to maintain my original assessment.
Theorem 2 does contain genuine flaws, as you and other reviewers pointed out. The original proof of Theorem 2 implicitly assumes that the retrieval strategy is exactly optimal on the source context, and this assumption was not stated explicitly. We introduce the source-side suboptimality error , and re-derive the revised upper bound:
$ \mathbb E\left[V_\omega^*(z_{\mathrm{curr}})-J_\omega(\hat\pi';z_{\mathrm{curr}})\right] \le\mathbb E[\varepsilon_{\mathrm{src}}] +2L_J\mathbb E[\delta_{\mathrm{ret}}] +L_VL_\Pi\Delta_{\mathrm{max}} $.
We added this assumption and updated the theorem and its proof in the revised manuscript.
Q1:
The core distinction indeed lies in the context-aware retrieval and reuse mechanism for historical experts, yet there is more to it. MORL-CA focuses on incrementally updating a single policy online. Upon arrival of each new context, it performs local optimization from scratch and cannot reuse previously learned Pareto policies, leading to substantial redundant computation.
In contrast, OW-CMORL learns a context metric and binds historical Pareto expert snapshots to corresponding contexts. When a new context arrives, it first retrieves and directly reuses well-trained matching policies, followed by lightweight local refinement. It is further equipped with a capacity-aware expert eviction mechanism to maintain memory under limited storage and mitigate forgetting.
Q2:
Q-Pensieve stores critic snapshots and relies on a set of critics to collectively guide the gradient update of the current actor, which serves as a training-side experience replay auxiliary. By contrast, the expert bank in OW-CMORL stores full Pareto policy snapshots bound to their corresponding context distributions. After retrieval, these policies can be directly restored for decision-making or lightweight local fine-tuning, targeting cross-context policy reuse. Furthermore, our framework integrates context metric learning and a capacity-aware eviction mechanism, which is specially tailored for time-varying multi-objective settings.
Q3:
Theorem 1 adds a finite sample exponential concentration bound quantifying deviation of empirical risk from the population risk given in Assumption A3, without providing per sample deterministic guarantees.
For Theorem 1, we add a constraint term, ∣dcos(Eϕ(Xa),Eϕ(Xb))−dZ(za,zb)∣, to the Transformer loss to align the cosine distance in the embedding space with the distance in the original context space. This ensures the encoder learns the geometric proximity among contexts and fulfills the achievability required by Assumption A3.
For Theorem 3, its conclusion is built upon loss decomposition and the monotonicity of eviction loss. Finite expert bank capacity introduces forgetting loss caused by eviction, and enlarging the capacity reduces this long-term loss. In the revised manuscript, we marked the scope of applicability for these two sets of assumptions and elaborate on their boundary conditions.