acceptodds
Under review as a conference paper at ICLR 2027

OW-CMORL: Online Window-Conditioned Multi-Objective Reinforcement Learning for Time-Varying Optimization

Abstract

This paper investigates non-stationary multi-objective reinforcement learning (MORL) for industrial control, where drifting exogenous operating conditions alter the Pareto frontier. Targeting this non-stationary online deployment scenario, we identify three practical obstacles: catastrophic forgetting, implicit context inference, and near-real-time constraints. We formalize this problem as a time-varying multi-objective contextual Markov decision process and derive three complementary theoretical upper-bounds bounding context embedding mismatch, local fine-tuning approximation, and long-run forgetting-induced performance degradation. We propose Online Window-Conditioned MORL (OW-CMORL): a context-indexed expert-bank stores restorable policy snapshots to mitigate catastrophic forgetting; a Transformer-based context-window encoder generates query embeddings from factor history and short-horizon forecasts for implicit unlabeled-condition representation; indexed retrieval fetches matched snapshots for lightweight local refinement, whose outputs refresh the online Pareto archive and avoid costly full retraining to satisfy near-real-time constraints. Experiments across four dynamic control systems under a unified evaluation protocol demonstrate that OW-CMORL preserves high-quality Pareto frontier while adapting to non-stationary operating conditions under near-real-time constraints. Open-sourced codes at: https://github.com/hoshinojyunn/OW-CMORL

Primary area:
reinforcement learning
Keywords:
Time‑varying multi‑objective reinforcement learning, Context‑aware adaptation, Catastrophic forgetting, Non‑stationary control, Operating‑condition drift

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_KXHJOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes OW-CMORL for multi-objective RL under time-varying operating conditions. The method encodes recent context history and forecasts with a Transformer, retrieves previously stored MORL policy snapshots from an expert bank, performs short local refinement, and updates an online Pareto archive. The paper also evaluates the method on four dynamic control domains.

Critical strengths
  1. The paper considers a practically relevant problem: maintaining good multi-objective policies under changing operating conditions.
  2. I think the overall system is fairly complete, with context inference, memory, retrieval, local adaptation, and Pareto archive maintenance.
  3. The experiments cover four control domains and include several useful ablations on the context window, expert bank, and retrieval mechanism.
Critical weaknesses
  1. The methodological novelty appears quite limited. Dynamic MORL with time-varying Pareto sets and online refinement is already studied in MORL-CA [1]. Continual MORL and catastrophic forgetting are explicitly studied in CORe3 [2], while Q-Pensieve [3] already introduces snapshot-based knowledge reuse in MORL. Context-driven non-stationarity and catastrophic forgetting without task labels are also studied in LCPO [4]. Against these works, OW-CMORL mainly adds a Transformer context encoder, context-indexed snapshot retrieval, HNSW, and short local refinement. In fact, the paper itself describes OW-CMORL as an “orchestration layer rather than a replacement for the MORL learner.”

  2. I also find the theoretical contribution weak. The main expectation statement in Theorem 1 essentially follows directly from Assumption A3, which already assumes the corresponding embedding population error is bounded. Theorem 3 similarly assumes the relevant loss decomposition and monotonic decrease of eviction loss with bank capacity, and then derives the corresponding long-run average bound. These results seem closer to sanity checks for the proposed architecture than substantive theoretical guarantees.

  3. The “implicit context inference” claim seems overstated. OW-CMORL directly receives the environment-side dynamic-factor vector z_t; what is unavailable is only an explicit stage/change-point label. This is considerably easier than inferring a latent operating regime from observations alone.

  4. Theorem 2 appears incorrect as stated, beyond the factor-of-two issue. Its assumptions B1–B4 do not assume that the retrieved snapshot is optimal at its original context. However, the proof suddenly uses J(hat{pi};hat{z}) = J(pi^*{hat{z}};hat{z}), i.e. exact optimality of the retrieved policy. But the expert bank stores arbitrary training-stage snapshots, not guaranteed optimal policies. Without an additional source-context suboptimality term, the claimed bound does not follow.

Overall, I see a useful engineering combination of existing ideas, but not a sufficiently new problem formulation or learning principle for ICLR.

[1] Gan et al., “MORL-CA: Dynamic Multi-Objective Reinforcement Learning for Chlor-Alkali Process Optimization Under Time-Varying Conditions,” IJCAI 2026. [2] Li et al., “Continual Multi-Objective Reinforcement Learning via Reward Model Rehearsal,” IJCAI 2024. This work formalizes continual MORL and explicitly addresses adaptation and catastrophic forgetting. IJCAI [3] Hung et al., “Q-Pensieve: Boosting Sample Efficiency of Multi-Objective RL Through Memory Sharing of Q-Snapshots,” ICLR 2023. [4] Hamadanian et al., “Online Reinforcement Learning in Non-Stationary Context-Driven Environments,” ICLR 2025.

Questions for authors
  1. What is the main methodological novelty over MORL-CA? Both methods address changing operating conditions, time-varying Pareto solutions, and online refinement. Is the main difference essentially context-based retrieval of historical experts?
  2. How is the expert bank fundamentally different from existing snapshot/memory reuse mechanisms such as Q-Pensieve [2], beyond indexing snapshots by an environmental-context embedding?
  3. What substantive guarantee does Theorem 1 provide beyond Assumption A3, which already assumes that the embedding mismatch is bounded? Similarly, what does Theorem 3 establish beyond the assumed loss decomposition and monotonicity of the eviction loss?
3 comments

Q1:

The core distinction indeed lies in the context-aware retrieval and reuse mechanism for historical experts, yet there is more to it. MORL-CA focuses on incrementally updating a single policy online. Upon arrival of each new context, it performs local optimization from scratch and cannot reuse previously learned Pareto policies, leading to substantial redundant computation.

In contrast, OW-CMORL learns a context metric and binds historical Pareto expert snapshots to corresponding contexts. When a new context arrives, it first retrieves and directly reuses well-trained matching policies, followed by lightweight local refinement. It is further equipped with a capacity-aware expert eviction mechanism to maintain memory under limited storage and mitigate forgetting.

Q2:

Q-Pensieve stores critic snapshots and relies on a set of critics to collectively guide the gradient update of the current actor, which serves as a training-side experience replay auxiliary. By contrast, the expert bank in OW-CMORL stores full Pareto policy snapshots bound to their corresponding context distributions. After retrieval, these policies can be directly restored for decision-making or lightweight local fine-tuning, targeting cross-context policy reuse. Furthermore, our framework integrates context metric learning and a capacity-aware eviction mechanism, which is specially tailored for time-varying multi-objective settings.

Q3:

Theorem 1 adds a finite sample exponential concentration bound quantifying deviation of empirical risk from the population risk given in Assumption A3, without providing per sample deterministic guarantees.

For Theorem 1, we add a constraint term, ∣dcos⁡(Eϕ(Xa),Eϕ(Xb))−dZ(za,zb)∣\left|d_{\cos}\left(E_{\phi }\left(X_{a}\right),E_{\phi }\left(X_{b}\right)\right)-d_{Z}\left(z_{a},z_{b}\right)\right|, to the Transformer loss to align the cosine distance in the embedding space with the distance in the original context space. This ensures the encoder learns the geometric proximity among contexts and fulfills the achievability required by Assumption A3.

For Theorem 3, its conclusion is built upon loss decomposition and the monotonicity of eviction loss. Finite expert bank capacity introduces forgetting loss caused by eviction, and enlarging the capacity reduces this long-term loss. In the revised manuscript, we marked the scope of applicability for these two sets of assumptions and elaborate on their boundary conditions.

Thank you for clarifying how OW-CMORL differs from MORL-CA and Q-Pensieve. My main concern about Theorem 2 remains unresolved. Even with an exact context match and no fine-tuning, a retrieved snapshot can still be suboptimal, yet the stated bound is zero. I appreciate the clarification on the method, but my concerns about the scope of the contribution and the correctness of the bound lead me to maintain my original assessment.

Theorem 2 does contain genuine flaws, as you and other reviewers pointed out. The original proof of Theorem 2 implicitly assumes that the retrieval strategy is exactly optimal on the source context, and this assumption was not stated explicitly. We introduce the source-side suboptimality error εsrc{\varepsilon}_{\text{src}}, and re-derive the revised upper bound:

$ \mathbb E\left[V_\omega^*(z_{\mathrm{curr}})-J_\omega(\hat\pi';z_{\mathrm{curr}})\right] \le\mathbb E[\varepsilon_{\mathrm{src}}] +2L_J\mathbb E[\delta_{\mathrm{ret}}] +L_VL_\Pi\Delta_{\mathrm{max}} $.

We added this assumption and updated the theorem and its proof in the revised manuscript.

Reviewer_hr5COpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies multi-objective control under recurring changes in operating conditions. OW-CMORL encodes recent context and a short forecast, retrieves saved learner checkpoints, performs local adaptation, and maintains a Pareto archive. It proposes three theoretical bounds and evaluates the framework on four industrial simulators, including context, memory, retrieval, and out-of-distribution experiments.

Critical strengths

Preserving complete, context-indexed learner snapshots is a sensible response to recurring operating regimes: it can retain specialists covering different trade-offs without requiring one continually updated policy to preserve them all. The separation between retrieval and the underlying MORL operators also makes the proposed architecture potentially useful across heterogeneous controllers.

Critical weaknesses
  1. The reported significance tests are incompatible with the reported confidence intervals. For EV charging EU, Table 1 gives OW-CMORL as 1.1956 ± 0.5932 and CAPQL as 0.9520 ± 0.4690. Under the stated 95% t-intervals over ten seed means, even perfect positive pairing gives a difference standard error of at least (0.5932 − 0.4690)/2.262 = 0.0549. Thus the paired statistic cannot exceed 4.44, giving a two-sided p-value of at least approximately 0.0016, before any Holm correction. Table 6 instead reports a corrected p-value of 4.83 × 10⁻⁹. This cannot be explained by pairing and prevents me from trusting the claimed statistical evidence without a reconstruction from the seed-level results.

  2. The theoretical foundation does not establish the claimed guarantees. Theorem 2 is false under B1–B4: setting retrieval mismatch and update magnitude to zero makes its bound zero, although a same-context snapshot can still be suboptimal. The proof silently equates the retrieved policy with an exact optimum; neither the assumptions nor the implementation guarantee this. An initial optimization-error term is necessary. Moreover, Theorem 1 assumes its expected mismatch bound in A3, and Theorem 3 assumes the loss decomposition and capacity monotonicity it is meant to explain. These results therefore do not substantiate the claimed benefits of the learned retrieval geometry or the particular eviction rule.

Questions for authors
  1. Can you provide the ten paired seed-level aggregates and the calculations producing Tables 1 and 6, reconcile the incompatibility above, and identify which comparative conclusions survive the corrected analysis?

  2. What bound holds for the actual, potentially suboptimal checkpoints used by OW-CMORL? Can you quantify their initial optimization error and derive an advantage of context-local replacement over FIFO without assuming that advantage in the loss decomposition or capacity dependence?

1 comment

Q1:

Thank you for pointing out the inconsistency between the confidence intervals and the significance results after Holm correction. After careful examination, this discrepancy arises from two separate mistakes:

  1. Different experimental datasets were used for the two tables. Table 1 and Table 6 were not computed from evaluations with the same set of seeds. Table 1 was aggregated from an earlier batch of results, while Table 6 was generated using a new dataset obtained from subsequent retraining and evaluation. This leads to the numerical mismatch you observed. This is an error in our experimental documentation.

  2. Inconsistent definition of the testing family for multiple testing (Holm procedure) between the manuscript text and implementation. In our code, we divided tests into 20 independent families grouped by environment and metric, with 4 hypotheses per family. However, the paper incorrectly states that all 80 hypotheses belong to a single testing family. The stringency of Holm correction depends on the number of hypotheses within each family. Mismatched family definitions substantially alter adjusted p-values and result in erroneous statistical descriptions.

We re-conducted the experiments under a unified setup and preserved the raw metrics for every seed. All p-values and confidence intervals have been cross-verified in Excel, and the verification spreadsheet is available at https://github.com/hoshinojyunn/OW-CMORL/tree/main/log/p_bounds.xlsx.

We have revised the manuscript accordingly:

• Corrected the incorrect statement that Table 1 and Table 6 share the same dataset;

• Updated the section on multiple testing to align with the final testing-family configuration;

• Replaced the confidence intervals in Table 1 and the significance results in Table 6 with the newly computed values from this unified dataset.

Supplementary note: With the unified dataset and the corrected testing pipeline, the confidence intervals and significance test results are now consistent, resolving the discrepancy raised by the reviewer.

The updated results are presented in Tables 1 and 6 in the revised paper.

Reviewer 1 raised the similar concerns.

Q2:

Theorem 2 does contain genuine flaws as you and other reviewers pointed out. The original proof of Theorem 2 implicitly assumes that the retrieval strategy is exactly optimal on the source context, and this assumption was not stated explicitly. We introduce the source-side suboptimality error εsrc{\varepsilon}_{\text{src}}, and re-derive the revised upper bound:

$ \mathbb E\left[V_\omega^*(z_{\mathrm{curr}})-J_\omega(\hat\pi';z_{\mathrm{curr}})\right] \le\mathbb E[\varepsilon_{\mathrm{src}}] +2L_J\mathbb E[\delta_{\mathrm{ret}}] +L_VL_\Pi\Delta_{\mathrm{max}} $.

We added this assumption and updated the theorem and its proof in the revised manuscript.

Regarding Assumption A3 (embedding distortion) for Theorem 1: the standard Transformer predictive loss cannot guarantee geometric alignment of embeddings. We add a metric regularization term to improve the empirical validity of this assumption and clarify that A3 is a conditional assumption.

Regarding Assumptions C1 and C3 for Theorem 3: the original loss decomposition omitted error sources including expert suboptimality, ANN retrieval, Pareto selection, and evaluation noise. We will supplement these error terms. C1 and C3 are conditional assumptions; the updated Theorem 3 still provides a conditional theoretical guarantee, which holds approximately under the test scenarios considered in this work.

Without introducing loss decomposition or capacity-dependent assumptions, we are currently unable to provide a rigorous theoretical proof showing that local context replacement outperforms FIFO. The performance difference between the two is supported by ablation experiments. We will state this theoretical limitation and add the corresponding ablation results in the revised manuscript.

Reviewer_Lj1ROpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses non-stationary multi-objective reinforcement learning (MORL) under changing operating conditions. The authors propose OW-CMORL, which combines Transformer-based context encoding, a context-indexed expert bank, HNSW retrieval, lightweight local policy refinement, and online Pareto archive maintenance. Three theoretical bounds are presented to characterize context embedding mismatch, policy refinement error, and long-term performance degradation. The framework is evaluated across four dynamic control environments using 100 operating conditions and 10 random seeds, alongside out-of-distribution and ablation studies. The authors report improvements in preference-aware utility and adaptation performance while claiming near-real-time optimization capabilities.

Critical strengths

S1. Practical Relevance. The paper addresses an important problem in non-stationary multi-objective reinforcement learning, where changing and recurring operating conditions can alter Pareto trade-offs and cause catastrophic forgetting. Reusing previously learned policies is a reasonable approach to this challenge. S2. Coherent Framework Design. OW-CMORL integrates context encoding, expert-bank memory, HNSW retrieval, local policy refinement, and Pareto archive maintenance into a unified framework. Its modular design allows the reuse of existing MORL learners and offers potential practical flexibility. S3. Comprehensive Evaluation Design. The evaluation covers four dynamic control environments and includes comparisons with multiple MORL baselines, out-of-distribution adaptation experiments, and component ablation studies. The use of shared operating conditions and evaluation protocols is a positive aspect of the experimental design, although concerns regarding the reliability of certain reported results remain.

Critical weaknesses

The reliability of the reported experimental results raises substantial concerns. Section 5.5 and Appendix J.1 present Table 2 as a context-window ablation conducted across four environments under a shared 100-shift protocol. However, the publicly available script run_context_key_ablation_100_short_budget.py reproduces all 48 reported values using predefined performance degradation factors and synthetic trajectories, without executing environment rollouts or policy updates. This raises serious questions about the provenance of the reported ablation results. Furthermore, the confidence intervals in Table 1 appear mathematically inconsistent with the Holm-adjusted significance results in Table 6 under the stated statistical procedure. For example, the reported Chlor-alkali EU comparison between OW-CMORL and CAPQL implies an unadjusted two-sided p-value of at least 0.316, whereas Table 6 reports an adjusted p-value of 0.017. These discrepancies require clarification and supporting experimental evidence.

W2. Insufficient Theoretical Justification. The theoretical analysis does not adequately support several of the paper's claims. In particular, Theorem 2 does not hold under its stated assumptions B1–B4, since an exact context match and zero parameter updates do not guarantee that the retrieved policy is optimal. Its proof additionally assumes optimality under the source context without explicitly including this condition in the theorem. Theorem 1 largely restates an assumed embedding-error bound, while Theorem 3 relies on assumptions concerning loss decomposition and memory capacity rather than deriving these properties from the proposed algorithm. Consequently, the theoretical guarantees require substantial clarification and revision.

W3. Limited Methodological Novelty. The proposed framework primarily integrates established techniques, including Transformer-based context encoding, HNSW retrieval, expert-bank memory, local policy refinement, and Pareto archive maintenance. Moreover, Appendix B.1 explicitly describes OW-CMORL as an orchestration layer built upon existing MORL learners. Although this integration is reasonable, the paper does not sufficiently distinguish its novel algorithmic contributions from the combination of existing components. Stronger justification and controlled comparisons are needed to establish which specific design choices provide benefits beyond straightforward integration.

W4. Insufficient Evidence for Near-Real-Time Performance. The claimed near-real-time capability is not adequately supported by end-to-end measurements. Table 5 reports expert-bank capacities of only 18–32 snapshots, whereas the HNSW efficiency study in Appendix J.3 uses a substantially different setting with one million snapshots. More importantly, retrieval latency alone does not capture the computational costs of context encoding, checkpoint restoration, local policy updates, evaluation, and Pareto archive maintenance. Without measurements of the complete online adaptation pipeline, it remains unclear whether the proposed method satisfies practical near-real-time constraints.

W5. Figure Readability and Presentation. Figures 1 and 6 contain excessively small text and thin lines, making the architecture and information flow difficult to follow at normal reading size. Improving the font sizes, line weights, and overall visual clarity would make the proposed framework easier to understand.

Questions for authors

Q1. Provenance and Reproducibility of the Ablation Results. Section 5.5 and Appendix J.1 describe Table 2 as a context-window ablation conducted under a shared 100-shift protocol. However, the publicly available script run_context_key_ablation_100_short_budget.py can reproduce all 48 reported values using predefined performance degradation factors and synthetic trajectories, without executing environment rollouts or policy updates. Could the authors clarify how the results in Table 2 were generated and explain the relationship between this script and the reported experiments? Please provide the original experimental logs, relevant checkpoints, and the corresponding result-generation procedure.

Q2. Statistical Consistency of the Reported Results. The confidence intervals in Table 1 appear inconsistent with the Holm-adjusted significance results in Table 6 under the stated statistical procedure. For example, the reported Chlor-alkali EU results for OW-CMORL and CAPQL imply an unadjusted two-sided p-value of at least approximately 0.316, whereas Table 6 reports an adjusted p-value of 0.017. Could the authors explain this discrepancy and provide the underlying seed-level results and statistical calculations?

Q3. Validity of the Theoretical Guarantees. Theorem 2 appears to require an additional assumption concerning the optimality of the retrieved policy under its source context. Without this assumption, an exact context match and zero parameter updates do not guarantee optimality. Could the authors clarify this assumption and revise the corresponding bound? Furthermore, how are the key assumptions underlying Theorems 1 and 3 justified for the actual implementation, particularly those concerning embedding approximation, loss decomposition, and expert-bank capacity?

Q4. Distinct Algorithmic Contributions. OW-CMORL integrates several established techniques, including Transformer-based context encoding, HNSW retrieval, expert-bank memory, local policy refinement, and Pareto archive maintenance. Appendix B.1 also describes the framework as an orchestration layer built upon existing MORL learners. Could the authors clarify which components constitute the main novel algorithmic contributions? What evidence demonstrates that the proposed design offers advantages beyond a straightforward integration of existing techniques?

Q5. Evidence Supporting Near-Real-Time Performance. The expert-bank capacities reported in Table 5 range from 18 to 32 snapshots, whereas Appendix J.3 evaluates HNSW retrieval using one million snapshots. Could the authors explain how these experimental settings relate to the actual online deployment scenario? More importantly, could they report the end-to-end adaptation latency, including context encoding, checkpoint restoration, local policy updates, evaluation, and Pareto archive maintenance, along with the hardware configuration and practical latency requirements?

Q6. Readability of Figures 1 and 6. The text and connecting lines in Figures 1 and 6 are difficult to read at normal viewing size. Could the authors improve these figures by increasing font sizes, using thicker lines, and clarifying the connections between components to make the proposed architecture easier to understand?

4 comments

Q1:

Thanks for the reviewer’s comments. We clarify the workflow of our ablation experiments and the purpose of the two scripts:

The main ablation experiments for Table 2 are executed by run_owcmorl_ablation_study.py. This script trains all ablated policy variants and runs environment interactions under non-stationary environments, collects outcomes across multiple repeated trials, and outputs the averaged relative drop/rise ratios from these repeated runs into results.json. In contrast, run_context_key_ablation_100_short_budget.py is a post-processing aggregation script rather than an experiment runner. It loads raw metrics stored in results.json, computes group-wise means and the relative performance change of each ablated variant against OW-CMORL, and finally produces the 48 reported metrics shown in Table 2.

The synthetic trajectory module inside this post-processing script serves only as an independent sanity check to verify whether our metric calculation logic behaves as expected. It is not involved in generating, modifying or calibrating any values reported in Table 2. All results presented in the paper originate from genuine policy training and environment rollouts.

Regarding experimental artifacts: our initial open-source release only archived aggregated summary results, without releasing full per-run training logs and intermediate checkpoints. We have re-run this set of ablation experiments and supplemented fully verifiable logs, checkpoints and related materials. Consistent with most reinforcement learning benchmarks, random variance exists in our experiments. Exact values from a single re-run will fluctuate within a reasonable range, yet the core conclusions of the ablation study, the direction and relative magnitude of performance degradation upon module removal, remain stable across repeated trials.

The reproduced results are presented as Table 2 in the revised paper.

Context-window ablation

Under the shared 100-shift protocol, each entry reports mean HV / EU / Regret. Higher is better for HV and EU; lower is better for Regret. H1 uses only the current factor vector; H2 removes forecast or history.

EnvironmentFull windowH1: current factorH2: no forecastH2: no history
Building1.6106e+10 / 9.4550e+3 / 3.2469e+01.4162e+10 / 7.8337e+3 / 3.0364e+01.5812e+10 / 8.6437e+3 / 3.8344e+01.5499e+10 / 8.6071e+3 / 3.4961e+0
EV charging7.3320e-2 / 1.1602e+0 / 4.9014e-15.5832e-2 / 9.1995e-1 / 5.8434e-16.3741e-2 / 1.0346e+0 / 5.3897e-16.7737e-2 / 1.1375e+0 / 5.8415e-1
Cogen5.1830e+23 / -1.6349e+6 / 9.1183e+34.9638e+23 / -1.8027e+06 / 1.0408e+45.6801e+23 / -1.6577e+06 / 9.5508e+35.0714e+23 / -1.7935e+06 / 1.2662e+04
Chlor-alkali1.0076e+12 / -7.7895e+4 / 9.7075e+19.1382e+11 / -9.1055e+04 / 1.1157e+21.0937e+12 / -8.0952e+04 / 1.2145e+21.0344e+12 / -8.2491e+04 / 1.1701e+2

Logs and related records are available in the anonymous online repository: https://github.com/hoshinojyunn/OW-CMORL/tree/main/log/

Q2:

Thank you for pointing out the inconsistency between the confidence intervals and the significance results after Holm correction. After careful examination, this discrepancy arises from two separate mistakes:

  1. Different experimental datasets were used for the two tables. Table 1 and Table 6 were not computed from evaluations with the same set of seeds. Table 1 was aggregated from an earlier batch of results, while Table 6 was generated using a new dataset obtained from subsequent retraining and evaluation. This leads to the numerical mismatch you observed. This is an error in our experimental documentation.

  2. Inconsistent definition of the testing family for multiple testing (Holm procedure) between the manuscript text and implementation. In our code, we divided tests into 20 independent families grouped by environment and metric, with 4 hypotheses per family. However, the paper incorrectly states that all 80 hypotheses belong to a single testing family. The stringency of Holm correction depends on the number of hypotheses within each family. Mismatched family definitions substantially alter adjusted p-values and result in erroneous statistical descriptions.

We re-conducted the experiments under a unified setup and preserved the raw metrics for every seed. All p-values and confidence intervals have been cross-verified in Excel, and the verification spreadsheet is available at https://github.com/hoshinojyunn/OW-CMORL/tree/main/log/p_bounds.xlsx.

We have revised the manuscript accordingly:

• Corrected the incorrect statement that Table 1 and Table 6 share the same dataset;

• Updated the section on multiple testing to align with the final testing-family configuration;

• Replaced the confidence intervals in Table 1 and the significance results in Table 6 with the newly computed values from this unified dataset.

Supplementary note: With the unified dataset and the corrected testing pipeline, the confidence intervals and significance test results are now consistent, resolving the discrepancy raised by the reviewer.

The updated results are presented in Tables 1 and 6 in the revised paper.

Reviewer 2 raised the similar concerns.

Q3:

Thank you for the valuable comments. The original proof of Theorem 2 implicitly assumes that the retrieval strategy is exactly optimal on the source context, and this assumption was not stated explicitly. We introduce the source-side suboptimality error εsrc{\varepsilon}_{\text{src}}, and re-derive the revised upper bound:

$ \mathbb E\left[V_\omega^*(z_{\mathrm{curr}})-J_\omega(\hat\pi';z_{\mathrm{curr}})\right] \le\mathbb E[\varepsilon_{\mathrm{src}}] +2L_J\mathbb E[\delta_{\mathrm{ret}}] +L_VL_\Pi\Delta_{\mathrm{max}} $.

We added this assumption and updated the theorem and its proof in the revised manuscript.

Regarding Assumption A3 (embedding distortion) for Theorem 1: the standard Transformer predictive loss cannot guarantee geometric alignment of embeddings. We add a metric regularization term to improve the empirical validity of this assumption and clarify that A3 is a conditional assumption.

Regarding Assumptions C1 and C3 for Theorem 3: the original loss decomposition omitted error sources including expert suboptimality, ANN retrieval, Pareto selection, and evaluation noise. We will supplement these error terms. C1 and C3 are conditional assumptions; the updated Theorem 3 still provides a conditional theoretical guarantee, which holds approximately under the test scenarios considered in this work.

Q4:

The main innovations of OW-CMORL lie in its Transformer context encoder and the expert memory bank module. With the Transformer context encoder, we can formally quantify the similarity between different contexts. When a new context arrives, the framework can retrieve an approximately optimal policy from the expert bank, so that previously trained policies matching the current context can be directly deployed for solving the task. Although the framework orchestrates existing MORL learners, each module addresses a core challenge. This expert-bank-based paradigm offers a new avenue to tackle catastrophic forgetting in dynamic optimization problems. Moreover, prior works along this line rarely investigate dynamic multi-objective optimization settings.

Q5:

The bank size of 18–32 snapshots in Table 5 denotes the steady state expert bank capacity used in our main performance comparison experiments. It represents the online memory upper bound maintained via continuous eviction under dynamic environments. By contrast, Appendix J.3 uses one million snapshots for a standalone retrieval stress test, which validates the retrieval scalability of HNSW under a large snapshot inventory. These two settings serve distinct purposes: the former simulates the controlled memory capacity during normal algorithm operation, while the latter verifies that retrieval does not break down as snapshots keep accumulating after long term deployment. Note that existing experiments in J.3 only measure latency for the retrieval stage alone and do not cover the full end to end pipeline.

Admittedly, end to end latency is indeed the most intuitive metric for practical real world performance. However, one million policies incur substantial storage overhead, and rebuilding the expert bank together with the HNSW index imposes heavy computational costs and long experimental turnaround. Such resources are not available under our laboratory setup, so we have not yet conducted these end to end experiments.

Q6:

We improved them.

Thank you for the responses. I have considered the replies and found they have not meaningfully resolved my concerns. I will maintain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.