acceptodds
Under review as a conference paper at ICLR 2027

Leveraging Generative Uncertainty Modeling for Distributionally Robust Offline Reinforcement Learning

19 views

Abstract

Distributionally robust offline RL addresses the challenge of distribution shift of deployment environment by optimizing policies against an uncertainty set of environment dynamics. The uncertainty-set design is critical: it should capture diverse distribution shifts for robustness while excluding unrealistic shifts that lead to overly pessimistic policies. Existing methods often assume rectangular uncertainty, which can combine locally worst-case transitions that cannot coexist in a single environment. Many also rely on standard distribution discrepancy measures: -divergences ignore support-space geometry, while Wasserstein sets can be computationally challenging in high-dimensional continuous settings. To capture non-rectangular, diverse yet plausible shifts, we construct a generative uncertainty set using flow models which can generate diverse and plausible trajectories. The proposed set captures temporally coupled dynamics shifts through shared model parameters while maintaining consistency with offline data via a tractable flow-matching constraint that implies a Wasserstein bound. Building on this uncertainty model, we develop Generative Uncertainty-Augmented Robust Offline RL (GURO-RL), which addresses value evaluation and adversarial optimization under flow-based uncertainty and finite offline data. Experiments across multiple applications demonstrate that GURO-RL improves policy robustness under diverse distribution shifts.

Primary area:
reinforcement learning
Keywords:
Distributionally Robust Learning, Generative Model, Offline Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3qtFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces GURO-RL for offline reinforcement learning under dynamics shifts, representing trajectory distributions with conditional flow models. It defines uncertainty through a flow-matching loss constraint and combines adversarial generator fine-tuning, importance weighting, and multistep critic targets. The paper establishes an average conditional squared Wasserstein-2 bound under stated assumptions. Experiments cover six MuJoCo task–dataset configurations, three offline RL backbones, and an EPANET simulation.

Critical strengths
  1. Combining flow-matching constraints with adversarial trajectory generation addresses uncertainty modeling for implicit world models.
  2. The core proof of Proposition 4.1 establishes a connection between fitting loss and distributional distance under its stated assumptions.
  3. Reported aggregate improvements across varied settings suggest practical value beyond a single task or backbone.
Critical weaknesses
  1. Tables 12–13 report actuator-family standard deviations of 5.5 and 2.1 despite matching scenario-level summaries; under matched seed-wise averaging, Table 12 also exceeds the bound implied by component standard deviations. Raw-run recomputation and a specified aggregation procedure are necessary to assess this evidence.
  2. Initial-action importance weighting generally cannot correct behavior-policy continuations in multistep targets, and using an optimized pretrained policy as the denominator introduces another mismatch. Shared generator parameters also do not ensure consistency with one stationary MDP kernel, leaving the central robust-MDP interpretation unsupported.
  3. Evaluated OOD trajectories inform uncertainty-budget selection, while early-stopping rules and validation/test separation remain unspecified. Untouched test perturbations and explicit, equal-information tuning protocols are needed to establish generalization.
Questions for authors
  1. What justifies interpreting exponentiated flow-matching loss differences as endpoint density ratios or an equivalent optimization reformulation?
  2. With identical generator conditioning and training budgets, does adversarial fine-tuning improve worst-perturbation returns while satisfying the flow-matching constraint?
6 comments

We thank the reviewer for the insightful feedback. Below, we provide detailed responses to the questions.

W1. Tables 12–13 report actuator-family standard deviations of 5.5 and 2.1 despite matching scenario-level summaries; under matched seed-wise averaging, Table 12 also exceeds the bound implied by component standard deviations. Raw-run recomputation and a specified aggregation procedure are necessary to assess this evidence.

Thank you for identifying this inconsistency. The two standard deviations referred to the same actuator-family aggregate. The 5.55.5 entry in Table 12 (Table 14 in revised manuscript) was a reporting error carried over from an earlier draft, which we have corrected. We recomputed the WDN summaries and verified that the GURO-RL column of Table 12 (Table 14 in revised manuscript) and the H=5H=5 column of Table 13 (Table 15 in revised manuscript) agree across all 11 evaluation settings (one nominal and ten perturbed). The revised tables consistently report means and sample standard deviations for individual settings, and means for aggregate rows. Both tables report an actuator-family mean of 150.9150.9.

W2. Initial-action importance weighting generally cannot correct behavior-policy continuations in multistep targets, and using an optimized pretrained policy as the denominator introduces another mismatch. Shared generator parameters also do not ensure consistency with one stationary MDP kernel, leaving the central robust-MDP interpretation unsupported.

Eqn. (8) provides a policy-dependent adversarial objective by reweighting the initial action toward the current policy. We use the pretrained policy as a proxy for the behavior policy, so we agree that the objective is approximate. For H=1H=1, the generated score has an approximate one-step Bellman form. A multi-step state-reward expansion with H>1H>1 can improve the value evaluation accuracy by reducing reliance on the terminal bootstrap of value function VψV_{\psi} as is also demonstrated by Ding et al. (2024), but intermediate actions do not follow π\pi, introducing policy-mismatch error. Nevertheless, the horizon HH can be properly selected to balance the benefits of multi-step expansion against the accumulated policy-mismatch error, as illustrated by the horizon ablation in Figure 3, supporting the practical usefulness of this surrogate. We have clarified this point in Section 5.2.1 in our revised manuscript.

Comparison with an explicit policy-aware variant. We implement GURO-RL-Seq, which constructs trajectories using one-step flow predictions and the current policy. Starting from a logged pair (s0,a0)(s_0,a_0), it recursively samples (rh,sh+1)∼Pθ(⋅∣sh,ah),h=0,…,H−1,(r_h,s_{h+1}) \sim P_\theta(\cdot\mid s_h,a_h), \qquad h=0,\ldots,H-1, where PθP_\theta is the flow model's first-step reward-state distribution. For h<H−1h<H-1, it msamples ah+1∼π(⋅∣sh+1)a_{h+1}\sim\pi(\cdot\mid s_{h+1}). The same θ\theta is held fixed throughout each rollout. Thus, this construction explicitly uses one stationary reward–transition kernel and separates policy sampling from transition sampling. It removes intermediate-action mismatch, although the initial-weighting approximation remains.

Table R1: Task and benchmark-level averages: MuJoCo normalized scores and WDN episodic returns.

BenchmarkSettingGURO-RLGURO-RL-Seq
MuJoCoH-SM65.854.9
W-SM63.062.0
Avg64.458.5
WDNAvg171.7169.5

Table R1 compares the two methods using the same per-task hyperparameter settings. GURO-RL achieves the higher overall average on both benchmarks. GURO-RL-Seq explicitly accounts for intermediate policy actions, but queries the model at predicted states and policy-sampled actions. These queries may enter regions with limited offline coverage, making accumulating extrapolation error a plausible contributor to the performance gap. These results support multi-step generation as a practical surrogate for value learning. The use of joint prediction is also consistent with Ding et al. (2024), whose Diffusion World Model generates multi-step states and rewards to avoid recursive model queries.

GURO-RL-Seq alternates between predicting one transition and sampling the next action from the current policy (Appendix D.3.1). Both methods retain the initial-action weight in Eq. (8). GURO-RL reduces flow-model calls per rollout from HH to one and improves mean OOD performance from 52.452.4 to 58.358.3 across H-SM and W-SM on MuJoCo ((Table 11) and from 157.7157.7 to 160.3160.3 on WDN (Table 16). We have discussed these results in Section D.3.2 and E.5.2.

W3. Evaluated OOD trajectories inform uncertainty-budget selection, while early-stopping rules and validation/test separation remain unspecified. Untouched test perturbations and explicit, equal-information tuning protocols are needed to establish generalization.

Thank you for raising this question. For each task, we select the uncertainty budget ϵ\epsilon and determine early stopping using a separate validation dataset with environment-parameter configurations different from those used for testing. In MuJoCo, we scale morphology, friction, or gravity one at a time by 0.10.1, 0.70.7, 1.31.3, or 1.91.9, giving 12 OOD validation environments per task. In WDN application, we use building demand traces at J3–J6 that are disjoint from the test traces, newly generated demand profiles at J7, and pipe-property and actuator-scale settings distinct from those used for testing. Appendices C.4 and E.3 of the revised manuscript describe the full validation configurations. All methods use the same validation dataset and have equal access to tuning information. We have clarified this protocol in the revision.

Reviewer_3pxeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies offline reinforcement learning under changes in environment dynamics. The goal is to train a policy from recorded data that still works when conditions such as friction or gravity change. The authors use a flow model to generate future states and rewards. Shared model parameters couple the generated changes, and a flow-matching loss limits how far the generated distribution can move from the data. A theoretical bound connects this loss to Wasserstein distance. They then propose GURO-RL, which adjusts the generator toward unfavorable outcomes and uses these sequences to update the policy's value estimates. On simulated robot locomotion and a water-distribution network, GURO-RL improves average performance under changed conditions.

Critical strengths
  1. The uncertainty constraint is useful. It gives a practical way to limit adversarial changes to a flow model without computing its probability density. The Wasserstein bound provides some theoretical support for controlling how far the generated distribution moves from the data.
  2. The results are encouraging. Across six MuJoCo settings, GURO-RL reaches an average OOD score of 57.1, compared with 50.4 for the strongest baseline average.
Critical weaknesses
  1. The generated futures may not match the current policy's actions. The generator learns from recorded sequences and uses only the starting state and first action. It generates later steps without explicitly following the current policy's later actions, which may differ from those in the data.
  2. The comparison and model selection need clarification. FWM receives an extra input describing future rewards, so its comparison with GURO-RL changes more than adversarial training alone. A comparison using identical inputs is missing.
  3. It is worth clarifying and discussing the proposed method's limitations or effectiveness in other adversarially settings, for example if an adversary can change the environment dynamics in ways that are not captured by the generator or the adversary can change one of the MuJoCo agents like the attack studies in the papers [1-2], how would the method perform?

[1] https://www.usenix.org/conference/usenixsecurity21/presentation/wu-xian [2] https://www.usenix.org/conference/usenixsecurity23/presentation/guo-wenbo

Questions for authors
  1. How do the generated sequences account for the current policy's actions after the first step? How does a mismatch affect the value estimates, and when is this approximation reasonable?
  2. Can you compare against the same generator with identical inputs, sequence length and policy-training budget, but without adversarial fine-tuning?
  3. What OOD information was used to select the uncertainty budget, sequence length and stopping point? Were final evaluation runs separate from tuning, and did the baselines receive comparable tuning access?
5 comments

We thank the reviewer for the insightful feedback. Below, we provide detailed responses to the questions.

W1. The generated futures may not match the current policy's actions. The generator learns from recorded sequences and uses only the starting state and first action. It generates later steps without explicitly following the current policy's later actions, which may differ from those in the data.

Q1. How do the generated sequences account for the current policy's actions after the first step? How does a mismatch affect the value estimates, and when is this approximation reasonable?

We agree that Eqn. (8) reweights the initial action toward π\pi but does not ensure that subsequent generated steps follow π\pi when the rollout horizon H>1H>1. For H=1H=1, it still yields an approximate policy-aware one-step Bellman target. A multi-step state-reward expansion with H>1H>1 can improve the value evaluation accuracy by reducing reliance on the terminal bootstrap of value function VψV_{\psi} as is also demonstrated by Ding et al. (2024), but intermediate actions do not follow π\pi, introducing policy-mismatch error. Nevertheless, the horizon HH can be properly selected to balance the benefits of multi-step expansion against the accumulated policy-mismatch error, as illustrated by the horizon ablation in Figure 3, supporting the practical usefulness of this surrogate. We have clarified this point in Section 5.2.1 in our revised manuscript.

To justify the effectiveness of this surrogate, we implement GURO-RL-Seq, which recursively samples one-step transitions and rewards, with each intermediate action explicitly sampled from π\pi. Appendix D.3.1 of the revised manuscript describes the implementation of GURO-RL-Seq. Table 10 summarizes its generation and scoring procedure. This ensures that the action at each roll-out step strictly follows the current policy π\pi. Under the same per-task hyperparameter settings, GURO-RL achieves higher average performance than GURO-RL-Seq on the evaluated MuJoCo settings and the WDN benchmark, as shown in Table R1.

Table R1: Comparison of MuJoCo normalized scores and WDN episodic returns. The MuJoCo average covers H-SM and W-SM.

BenchmarkSettingGURO-RLGURO-RL-Seq
MuJoCoH-SM65.854.9
W-SM63.062.0
Avg64.458.5
WDNAvg171.7169.5

This is because recursive queries at predicted states and policy-sampled actions can enter regions with limited offline coverage, causing model errors to accumulate over steps. The results support that multi-step state–reward generation, with initial actions reweighted toward π\pi, is a useful practical surrogate for value evaluation. This interpretation is consistent with the motivation for multi-step prediction in Ding et al. (2024). Appendices D.3.2 and E.5.2 of the revised manuscript discuss this comparison. Table 11 reports the results for H-SM and W-SM, and Table 16 reports the results for WDN.

W2. The comparison and model selection need clarification. FWM receives an extra input describing future rewards, so its comparison with GURO-RL changes more than adversarial training alone. A comparison using identical inputs is missing.

Q2. Can you compare against the same generator with identical inputs, sequence length and policy-training budget, but without adversarial fine-tuning?

Thank you for this comment. The default comparison of FWM and GURO-RL in Table 2 already controls for conditioning inputs, sequence length, and policy-training budget and other conditions, isolating the effect of adversarial fine-tuning. FWM keeps the pretrained generator frozen, whereas GURO-RL adversarially fine-tunes it. For each task, FWM and GURO-RL start from the same pretrained flow-matching model and Stage 1 checkpoints. Both generators receive the logged initial state–action pair and the return-to-go (RTG) label. They use identical generation window lengths T\mathcal{T} (1212 for Hopper and WDN; 1818 for Walker2d and HalfCheetah), the same critic-target horizon H=5H=5, and the same policy-training budget.

The main formulation denotes the generator context by (s,a)(s,a) to emphasize the initial pair whose value target is constructed. Appendices C.4.1 of the revised manuscript documents the shared RTG conditioning used by FWM and GURO-RL.

W3. It is worth clarifying and discussing the proposed method's limitations or effectiveness in other adversarially settings, for example if an adversary can change the environment dynamics in ways that are not captured by the generator or the adversary can change one of the MuJoCo agents like the attack studies in the papers [1-2], how would the method perform?

[1] https://www.usenix.org/conference/usenixsecurity21/presentation/wu-xian

[2] https://www.usenix.org/conference/usenixsecurity23/presentation/guo-wenbo

Thank you for suggesting these relevant works. Wu et al. [1] train adversarial policies that control an adversarial agent to exploit a victim policy in two-player games. Guo et al. [2] propose PATROL, a game-theoretic defense against such adversarial-policy attacks.

Our targeted problem differs. GURO-RL focuses on robust offline policy learning under uncertain environment dynamics, aiming to improve generalization across potential deployment environments, whereas security-oriented works explicitly consider strategic adversarial policies. In contrast, our adversary searches over transition–reward distributions represented by the flow model, without explicitly accounting for strategic adversarial behavior. Under its stated assumptions, our analysis bounds the average squared conditional Wasserstein-2 distance to the offline reference distribution for models in the uncertainty set.

A potential extension is to augment the uncertainty set with transition–reward distributions induced by adversarial policies that perturb the environment dynamics. This could be achieved by formulating an adversarial optimization problem over the space of such adversarial policies. We have clarified the scope of GURO-RL in Section 2 (Related Work) of the revised manuscript.

Reviewer_EvGVOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GURO-RL, a distributionally robust offline reinforcement learning framework that uses conditional guided flow models to construct a non-rectangular uncertainty set over state-reward trajectories. The proposed uncertainty set is constrained by flow-matching loss, with a theoretical bound relating it to Wasserstein-2 distance. GURO-RL adversarially fine-tunes the flow world model to generate challenging trajectories for robust policy learning. Experiments on MuJoCo and water-distribution-network environments demonstrate improved OOD robustness over baselines.

Critical strengths
  1. The paper introduces a non-rectangular generative uncertainty set based on conditional guided flow models. Shared model parameters naturally induce temporal coupling across state-action transitions, addressing a key limitation of rectangular uncertainty sets.
  2. The core technical contribution is the adversarial fine-tuning of a pretrained conditional flow model to generate policy-dependent adverse trajectories, which are then used for conservative value learning.
  3. Proposition 4.1 establishes that the tractable flow-matching constraint implies a Wasserstein-2 bound under stated assumptions, providing distributional consistency while remaining computationally practical. The framework also integrates cleanly with standard offline RL backbones.
Critical weaknesses
  1. The paper argues that non-rectangular uncertainty sets can capture dependencies across state-action transitions and avoid overly pessimistic combinations of worst-case dynamics. However, no controlled comparison with an equivalent rectangular uncertainty set is provided. Therefore, it remains unclear whether the robustness improvements arise from non-rectangular modeling, adversarial fine-tuning, or multi-step trajectory generation.
  2. Adversarial trajectories are generated only from initial state-action pairs observed in the offline dataset. Although this reduces extrapolation error, the method does not explicitly model uncertainty for unseen state-action regions that may arise under deployment distribution shifts.
  3. The method relies on a task-dependent uncertainty budget, but the paper does not clearly explain how this parameter should be selected when deployment dynamics are unknown.
Questions for authors
  1. Appendix C.2 indicates that the Wasserstein-2 distances between nominal and OOD environments were used to inform the selection of the uncertainty budget. Could the authors clarify the exact selection procedure and whether the uncertainty parameters of the baselines were tuned using comparable information? How would the uncertainty budget be selected when only offline training data are available, and would the reported improvements remain under a purely offline selection procedure?
  2. RAMBO-RL (Robust Adversarial Model-Based Offline Reinforcement Learning, NeurIPS 2022) already studies adversarial world-model optimization for offline RL, while Flow Matching Policy Gradients introduces the flow-matching-loss-based PPO-style optimization mechanism. Could the authors clarify the conceptual and algorithmic contributions of GURO-RL beyond these existing approaches and consider including RAMBO-RL as a baseline?
  3. The conditional flow model generates an H-step state-reward trajectory conditioned only on the initial state-action pair. As acknowledged in Section 4, the pretrained trajectory distribution is induced by both the environment dynamics and the behavior policy. However, the IPW correction in Equation (8) accounts only for the initial action distribution, while subsequent transitions implicitly depend on the behavior policy. Under what assumptions does the proposed H-step target provide a valid estimate of the target policy's value when the intermediate action distributions differ? Furthermore, since the adversarial flow model does not explicitly separate actions from environment transitions, how can the authors distinguish dynamics uncertainty from changes in the behavior-policy-induced trajectory distribution?
  4. The paper motivates non-rectangular uncertainty by arguing that shared physical factors induce dependencies across state-action transitions. However, shared flow-model parameters do not necessarily guarantee physically consistent dynamics. Beyond aggregate OOD performance, can the authors demonstrate whether the adversarially generated trajectories exhibit coherent dynamics shifts across different state-action pairs rather than arbitrary low-value perturbations? For example, could the authors compare generated transitions against simulator transitions under controlled friction, gravity, or morphology changes using matched initial state-action pairs?
7 comments

We thank the reviewer for the insightful feedback. Below, we provide detailed responses to the questions.

W1. The paper argues that non-rectangular uncertainty sets can capture dependencies across state-action transitions and avoid overly pessimistic combinations of worst-case dynamics. However, no controlled comparison with an equivalent rectangular uncertainty set is provided. Therefore, it remains unclear whether the robustness improvements arise from non-rectangular modeling, adversarial fine-tuning, or multi-step trajectory generation.

We thank the reviewer for rasing this question. GURO-RL's robustness improvements reflect the design of its flow-based uncertainty set. First, it does not require rectangular factorization, allowing dependencies across state–action transitions. Second, flow-matching (FM) loss provides a tractable discrepancy measure for expressive generators of diverse, realistic, high-dimensional trajectories. Under its stated assumptions, Proposition 4.1 establishes an averaged conditional Wasserstein discrepancy bound, anchoring adversarial adaptation to nominal data.

To examine the benefit of non-rectangular uncertainty structure, we evaluate GURO-RL-Seq-Rec, a stagewise rectangular variant, on MuJoCo (H-SM and W-SM) and water distribution. Starting from a logged state–action pair (s0,a0)(s_0,a_0), it recursively generates τ=(r0,s1,…,rH−1,sH)\tau=(r_0,s_1,\ldots,r_{H-1},s_H). At each stage h=0,…,H−1h=0,\ldots,H-1, it samples (rh,sh+1)∼Pθh(1)(⋅∣sh,ah)(r_h,s_{h+1})\sim P_{\theta_h}^{(1)}(\cdot\mid s_h,a_h), where Pθh(1)P_{\theta_h}^{(1)} denotes the first-step marginal of the flow distribution. For h<H−1h<H-1, the next action is sampled as ah+1∼π(⋅∣sh+1)a_{h+1}\sim\pi(\cdot\mid s_{h+1}). The parameters θh\theta_h are adversarially reoptimized using the sample-averaged reward expansion in (7), subject to the same FM-loss budget ϵ\epsilon. Let Uϵ\mathcal U_\epsilon denote the resulting family of admissible one-step kernels. This approximates the stagewise rectangular family UϵH\mathcal U_\epsilon^H, allowing a separate kernel at each step while retaining coupling across state–action pairs within each step. We will provide implementation details in the appendix.

Table R1: Mean performance on MuJoCo (H-SM and W-SM) and WDN. Both methods use H=5H=5.

BenchmarkAverageGURO-RLGURO-RL-Seq-Rec
MuJoCoAll64.459.6
OOD58.353.5
WDNAll171.7169.0
OOD160.3157.1

Table R1 shows that GURO-RL outperforms GURO-RL-Seq-Rec by 4.84.8 OOD points on MuJoCo and 3.23.2 OOD return on WDN. These results are consistent with the benefit of jointly modeling dependencies under persistent physical shifts. Joint generation also avoids recursive model queries, which can accumulate prediction errors. Thus, this comparison supports joint uncertainty modeling, although differences in generation and adversarial optimization prevent isolating non-rectangularity as the sole source of improvement.

W2. Adversarial trajectories are generated only from initial state-action pairs observed in the offline dataset. Although this reduces extrapolation error, the method does not explicitly model uncertainty for unseen state-action regions that may arise under deployment distribution shifts.

Thank you for this comment. Much of the existing distributionally robust offline RL literature (e.g., Blanchet et al., 2023; Shi & Chi, 2024; Zhang et al., 2025) targets transition shifts while treating the evaluation initial-state distribution d0d_0 as fixed. This focus reflects a distinctive challenge of RL: transition changes propagate through sequential decisions, altering future rewards and policy-dependent state visitation. GURO-RL follows this conventional setting, generating adverse continuations from logged state–action pairs.

At the formulation level, incorporating initial-distribution shifts is straightforward through standard distributionally robust optimization. Introducing an ambiguity set U0(d0)\mathcal U_0(d_0) yields max⁡πinf⁡d~0∈U0(d0)inf⁡ν∈UEs0∼d~0[V^νπ(s0)].\max_{\pi} \inf_{\widetilde d_0\in\mathcal U_0(d_0)} \inf_{\nu\in\mathcal U} \mathbb E_{s_0\sim\widetilde d_0} \left[\widehat V^\pi_\nu(s_0)\right]. The additional minimization accounts for uncertainty in initial conditions, with initial state–action distribution d~0(s0)π(a0∣s0)\widetilde d_0(s_0)\pi(a_0\mid s_0), while the transition uncertainty set on handles transition shifts. We have clarified the fixed initial-state distribution and discussed how to extend the formulation to capture initial-state distribution shifts in Section 3.2.

W3. The method relies on a task-dependent uncertainty budget, but the paper does not clearly explain how this parameter should be selected when deployment dynamics are unknown.

Q1. Appendix C.2 indicates that the Wasserstein-2 distances between nominal and OOD environments were used to inform the selection of the uncertainty budget. Could the authors clarify the exact selection procedure and whether the uncertainty parameters of the baselines were tuned using comparable information? How would the uncertainty budget be selected when only offline training data are available, and would the reported improvements remain under a purely offline selection procedure?

We tuned each algorithm separately for each task using the same validation sets and settings.For our method, we swept ϵ∈{0.05,0.1,0.2,0.35,0.5,0.7,1.0}\epsilon\in\{0.05, 0.1, 0.2, 0.35, 0.5, 0.7, 1.0\} for each of the six MuJoCo task–dataset settings and selected the budget that achieved the highest validation Avg for that setting. on the validation nominal and OOD environments. Table R2 compares the measured Wasserstein distances with the selected budgets. Within each robot, larger measured shifts correspond to larger selected ϵ\epsilon. This analysis provides a post-hoc interpretation of the selected budgets, rather than a direct calibration rule.

Table R2: Measured state-distribution shifts and selected uncertainty budgets.

H-SMH-MEW-SMW-MEHC-SMHC-ME
Dˉ\bar D0.0570.1150.1070.1781.5782.029
Selected ϵ\epsilon0.351.00.050.500.100.50

Our framework takes ϵ\epsilon as a supplied adversarial budget intended to reflect the estimated deployment shift. When only nominal offline data are available, selecting a deployment-relevant budget requires an additional uncertainty-quantification procedure and assumptions about potential shifts. This calibration problem is outside the scope of our work. We have yet to evaluate a fully offline selection procedure. The revision clarifies these distinctions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Revisions

We thank all reviewers for their feedback. The main changes are marked in blue in the revised manuscript. All section and table numbers below refer to that version.

Main text

  • Scope and assumptions (Sections 2, 3.2, and 7). We clarified how GURO-RL relates to RAMBO-RL and prior work on strategic adversarial policies, made the fixed initial-state distribution assumption explicit, and discussed extensions to uncertain initial conditions and physically structured shifts.
  • Adversarial objective (Section 5.2.1). We clarified that Eq. (8) reweights the initial action and explained the trade-off between reduced reliance on bootstrapping and policy mismatch in longer rollouts. We also explained why Eq. (11) uses an exponentiated flow-matching loss difference as a likelihood-ratio proxy.

Experiments and appendices

  • RAMBO-RL baseline (Section 6 and Appendices C.3, C.4.1, and D.1). We added RAMBO-RL to the MuJoCo comparison (Tables 2 and 8) and documented its training setup.
  • Training and validation (Appendices C.2, C.4.1, and E.3). We documented shared return-to-go conditioning for FWM and GURO-RL in MuJoCo, separate OOD validation settings for MuJoCo and WDN, and MuJoCo Stage 2 early stopping based on nominal returns. We clarified how the ambiguity budget is selected for each MuJoCo task–dataset setting and added the selected budgets to Table 5 alongside Wasserstein shift measurements for post-hoc interpretation.
  • Sequential variants (Appendices D.3 and E.5). We added matched comparisons with GURO-RL-Seq and GURO-RL-Seq-Rec to study joint generation and the uncertainty structure across rollout steps on MuJoCo and WDN (Tables 10, 11, and 16).
  • WDN results reporting (Appendix E.4, Tables 14 and 15). We corrected an inconsistency in the reported standard deviation for the actuator-scale aggregate. Both tables now report means and sample standard deviations for individual settings and means for aggregate rows.
AuthorsOpenReview ↗

A Brief Follow-Up on Our Rebuttal

Dear Reviewers and Area Chairs,

We thank the reviewers for their thoughtful feedback and the chairs for their time and care in overseeing the discussion.

In our rebuttal and revised manuscript, we provide additional experiments, document our evaluation procedures, and clarify the assumptions and approximations underlying GURO-RL.

  • For Reviewer EvGV — We examined joint generation and uncertainty-set structure by comparing GURO-RL with two sequential variants, including a stagewise rectangular variant, added RAMBO-RL as a MuJoCo baseline, and clarified how initial-action weighting incorporates the current policy into the multi-step adversarial objective, along with the scope of the uncertainty set and how we selected the uncertainty budget.
  • For Reviewer 3pxe — We explained how intermediate-action mismatch affects multi-step value estimates, documented the matched comparison with the frozen-generator baseline (FWM), and clarified our validation and early-stopping procedures and the scope of GURO-RL’s robustness claims.
  • For Reviewer 3qtF — We corrected the reported standard deviation for the WDN actuator-scale aggregate, evaluated a sequential rollout that uses a stationary kernel, explained the flow-matching likelihood-ratio proxy, clarified the separation between validation and testing, and reported the lowest OOD-family mean scores in the matched FWM comparison.

With the discussion period ending on November 18 (AoE), we would be grateful to hear whether our responses address your main concerns. If any point would benefit from further clarification, we would be happy to discuss it.

Thank you again for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.