- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces GURO-RL for offline reinforcement learning under dynamics shifts, representing trajectory distributions with conditional flow models. It defines uncertainty through a flow-matching loss constraint and combines adversarial generator fine-tuning, importance weighting, and multistep critic targets. The paper establishes an average conditional squared Wasserstein-2 bound under stated assumptions. Experiments cover six MuJoCo task–dataset configurations, three offline RL backbones, and an EPANET simulation.
Critical strengths
- Combining flow-matching constraints with adversarial trajectory generation addresses uncertainty modeling for implicit world models.
- The core proof of Proposition 4.1 establishes a connection between fitting loss and distributional distance under its stated assumptions.
- Reported aggregate improvements across varied settings suggest practical value beyond a single task or backbone.
Critical weaknesses
- Tables 12–13 report actuator-family standard deviations of 5.5 and 2.1 despite matching scenario-level summaries; under matched seed-wise averaging, Table 12 also exceeds the bound implied by component standard deviations. Raw-run recomputation and a specified aggregation procedure are necessary to assess this evidence.
- Initial-action importance weighting generally cannot correct behavior-policy continuations in multistep targets, and using an optimized pretrained policy as the denominator introduces another mismatch. Shared generator parameters also do not ensure consistency with one stationary MDP kernel, leaving the central robust-MDP interpretation unsupported.
- Evaluated OOD trajectories inform uncertainty-budget selection, while early-stopping rules and validation/test separation remain unspecified. Untouched test perturbations and explicit, equal-information tuning protocols are needed to establish generalization.
Questions for authors
- What justifies interpreting exponentiated flow-matching loss differences as endpoint density ratios or an equivalent optimization reformulation?
- With identical generator conditioning and training budgets, does adversarial fine-tuning improve worst-perturbation returns while satisfying the flow-matching constraint?
6 comments
W2. Initial-action importance weighting generally cannot correct behavior-policy continuations in multistep targets, and using an optimized pretrained policy as the denominator introduces another mismatch. Shared generator parameters also do not ensure consistency with one stationary MDP kernel, leaving the central robust-MDP interpretation unsupported.
Eqn. (8) provides a policy-dependent adversarial objective by reweighting the initial action toward the current policy. We use the pretrained policy as a proxy for the behavior policy, so we agree that the objective is approximate. For , the generated score has an approximate one-step Bellman form. A multi-step state-reward expansion with can improve the value evaluation accuracy by reducing reliance on the terminal bootstrap of value function as is also demonstrated by Ding et al. (2024), but intermediate actions do not follow , introducing policy-mismatch error. Nevertheless, the horizon can be properly selected to balance the benefits of multi-step expansion against the accumulated policy-mismatch error, as illustrated by the horizon ablation in Figure 3, supporting the practical usefulness of this surrogate. We have clarified this point in Section 5.2.1 in our revised manuscript.
Comparison with an explicit policy-aware variant. We implement GURO-RL-Seq, which constructs trajectories using one-step flow predictions and the current policy. Starting from a logged pair , it recursively samples where is the flow model's first-step reward-state distribution. For , it msamples . The same is held fixed throughout each rollout. Thus, this construction explicitly uses one stationary reward–transition kernel and separates policy sampling from transition sampling. It removes intermediate-action mismatch, although the initial-weighting approximation remains.
Table R1: Task and benchmark-level averages: MuJoCo normalized scores and WDN episodic returns.
| Benchmark | Setting | GURO-RL | GURO-RL-Seq |
|---|---|---|---|
| MuJoCo | H-SM | 65.8 | 54.9 |
| W-SM | 63.0 | 62.0 | |
| Avg | 64.4 | 58.5 | |
| WDN | Avg | 171.7 | 169.5 |
Table R1 compares the two methods using the same per-task hyperparameter settings. GURO-RL achieves the higher overall average on both benchmarks. GURO-RL-Seq explicitly accounts for intermediate policy actions, but queries the model at predicted states and policy-sampled actions. These queries may enter regions with limited offline coverage, making accumulating extrapolation error a plausible contributor to the performance gap. These results support multi-step generation as a practical surrogate for value learning. The use of joint prediction is also consistent with Ding et al. (2024), whose Diffusion World Model generates multi-step states and rewards to avoid recursive model queries.
GURO-RL-Seq alternates between predicting one transition and sampling the next action from the current policy (Appendix D.3.1). Both methods retain the initial-action weight in Eq. (8). GURO-RL reduces flow-model calls per rollout from to one and improves mean OOD performance from to across H-SM and W-SM on MuJoCo ((Table 11) and from to on WDN (Table 16). We have discussed these results in Section D.3.2 and E.5.2.
W3. Evaluated OOD trajectories inform uncertainty-budget selection, while early-stopping rules and validation/test separation remain unspecified. Untouched test perturbations and explicit, equal-information tuning protocols are needed to establish generalization.
Thank you for raising this question. For each task, we select the uncertainty budget and determine early stopping using a separate validation dataset with environment-parameter configurations different from those used for testing. In MuJoCo, we scale morphology, friction, or gravity one at a time by , , , or , giving 12 OOD validation environments per task. In WDN application, we use building demand traces at J3–J6 that are disjoint from the test traces, newly generated demand profiles at J7, and pipe-property and actuator-scale settings distinct from those used for testing. Appendices C.4 and E.3 of the revised manuscript describe the full validation configurations. All methods use the same validation dataset and have equal access to tuning information. We have clarified this protocol in the revision.
We thank the reviewer for the insightful feedback. Below, we provide detailed responses to the questions.
W1. Tables 12–13 report actuator-family standard deviations of 5.5 and 2.1 despite matching scenario-level summaries; under matched seed-wise averaging, Table 12 also exceeds the bound implied by component standard deviations. Raw-run recomputation and a specified aggregation procedure are necessary to assess this evidence.Thank you for identifying this inconsistency. The two standard deviations referred to the same actuator-family aggregate. The 5.5 entry in Table 12 (Table 14 in revised manuscript) was a reporting error carried over from an earlier draft, which we have corrected. We recomputed the WDN summaries and verified that the GURO-RL column of Table 12 (Table 14 in revised manuscript) and the H=5 column of Table 13 (Table 15 in revised manuscript) agree across all 11 evaluation settings (one nominal and ten perturbed). The revised tables consistently report means and sample standard deviations for individual settings, and means for aggregate rows. Both tables report an actuator-family mean of 150.9.