- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces GIRO, a robust reinforcement learning method that defines an uncertainty set of generative world models through a training-loss constraint, using diffusion models or conditional VAEs. GIRO alternates between an inner loop that adversarially updates the world model to minimize the current policy’s return while learning a value function from generated trajectories, and an outer loop that improves the policy under the resulting model. The method is evaluated on D4RL tasks with observation and action perturbations, as well as changes to body mass and friction.
Critical strengths
-
The method design is well aligned with its goal of learning robust policies. Updating the policy using the worst-case generative model within an uncertainty set, along with its corresponding value function, provides a convincing rationale for improved generalization. The method consistently follows this objective.
-
The paper provides clear theoretical insights into how GIRO mitigates world-model overoptimization differently from KL-regularized optimization and characterizes the convergence bounds for both the value function and the policy.
Critical weaknesses
-
Modest gains over a strong baseline. In Table 1, GIRO with VAE improves the aggregate worst-case score over MOBILE by only approximately 1% (714.2 vs. 707.1), while GIRO-Diffusion underperforms MOBILE in Table 7 (691.0 vs. 707.1). These marginal gains raise doubts about whether GIRO’s benefits justify the complexity of its adversarial updates.
-
Lower average performance. Averaging the reported scores in Table 6 equally across the eight perturbations and nine task–dataset combinations yields 90.37 for MOBILE, 85.80 for GIRO-Diffusion, and 84.46 for GIRO-VAE. Thus, both GIRO variants underperform MOBILE on average, indicating weak robustness capability across diverse perturbations.
-
Limited perturbation range. The mass and friction evaluations only use scaling factors of 0.95 and 1.05. A broader sweep, such as 0.8–1.2, would better characterize how performance degrades as the dynamics shift becomes more severe. The current narrow range provides limited evidence of robustness beyond small deviations from the nominal environment.
Questions for authors
Please see the weakness section.
4 comments
Thank you for the detailed rebuttal and additional experiments. I understand the worst-case objective of GIRO, but remain unconvinced that its robustness benefits justify the lower average returns relative to RAMBO and MOBILE. Lower variance alone does not establish this benefit. For example, the RAMBO variance of approximately 85 on HC-ME corresponds to a standard deviation of 9.2, which alone does not indicate severe degradation given the original score of RAMBO (93.7).
Under the wider perturbation range, the original GIRO-VAE underperforms RAMBO on both metrics. The fine-tuned variant achieves a improvement, but its settings were specifically adjusted for the expanded range. The rebuttal does not establish whether the baselines received comparable tuning, leaving uncertainty about the comparability of these results.
Given these remaining concerns, I maintain my original rating.
Thank you for the follow-up. We agree that comparing a fine-tuned GIRO with baselines that were not tuned for the wider range is not a fair comparison, so we fine-tuned RAMBO and MOBILE with the same protocol.
Protocol. Starting from the released OfflineRL-Kit checkpoints used in the paper (seeds 1–3), each baseline is trained for another 200K steps, as for GIRO. Only the robustness hyperparameters are changed; all other settings follow the released per-dataset configurations.
- RAMBO: adversarial weight (released: or , depending on the dataset), and a larger adversary learning rate ( vs. ).
- MOBILE: uncertainty penalty at , and the released per-dataset values.
As for GIRO, each method uses one configuration for all nine datasets, selected by total worst-case return on screening runs, and we report the final checkpoint. This selects for RAMBO ( the largest released value) and the original for MOBILE.
| Method | Nominal | Mean (perturbed) | Worst-case |
|---|---|---|---|
| RAMBO (released) | 734.3 | 708.4 | 530.5 |
| RAMBO (fine-tuned, ) | 746.1 | 681.5 | 515.2 |
| MOBILE (released) | 860.1 | 766.3 | 483.6 |
| MOBILE (fine-tuned, original ) | 815.2 | 763.6 | 499.7 |
| GIRO-VAE (original) | 777.2 | 731.4 | 476.4 |
| GIRO-VAE (fine-tuned) | 815.8 | 792.0 | 603.4 |
Normalized scores summed over the nine datasets (3 seeds, 20 episodes). Body mass and friction are scaled by ; worst-case is the minimum over these 12 settings of the 3-seed mean.
Larger robustness hyperparameters do not help the baselines. Fine-tuned RAMBO stays below its released checkpoint under perturbation, and doubling or quadrupling MOBILE's penalty lowers its worst-case return on the screening runs (). Training also becomes less stable: RAMBO diverges on hopper-medium-expert with the larger adversary learning rate, and MOBILE collapses on halfcheetah- and walker2d-medium-expert with the penalty. Under the same protocol, GIRO-VAE has the highest worst-case and mean perturbed return, and its nominal return is on par with fine-tuned MOBILE.
Why GIRO benefits where RAMBO does not. RAMBO adds the adversarial term to the model's supervised loss with a fixed weight , so how far the model may move from the data is never set directly. It depends on , on the adversary's step size and number of updates, and on the scale of the supervised loss, which differs between datasets. This is why the released varies by dataset, and why a larger or step size gives inconsistent gains or divergence. GIRO instead keeps the model's fit loss on the offline data within a budget relative to the nominal model's loss, so its uncertainty set is explicitly the set of models that explain the data almost as well as the nominal one, and one relative budget works for all nine datasets.
Why fine-tuning. In the paper, GIRO's budget had to be kept small: with a large budget, training from scratch diverges, because the critic is still inaccurate early on and the adversary can push transitions far from the data. That set does not cover $\pm$20% perturbations, which explains the original results in the wider range. Starting from a converged checkpoint, the critic is stable enough for a larger budget, so fine-tuning works as a two-stage budget schedule. Nothing in it is specific to GIRO, which is why we applied it to the baselines as well.
On variance and large perturbations. We agree that variance alone is not sufficient, which is why worst-case return is our main metric. With fine-tuning, GIRO-VAE is now the strongest method at the largest perturbations. At body mass and friction and , its worst-case sum is 603.4, compared with at most 530.5 for the baselines (released or fine-tuned), and its mean return over these four settings is 754.5, compared with at most 670.8. It has the highest worst-case return at these settings on 5 of the 9 datasets, with the largest margins on hopper-medium-expert (102.6 vs. 61.0 for RAMBO and 40.9 for MOBILE) and walker2d-medium-expert (86.1 vs. 76.1). HalfCheetah, including HC-ME, remains the exception: the baselines lose little there under these perturbations, and GIRO does not improve on them.
We have added these results and the corresponding implementation details to Appendix C.3 of the revised manuscript.
Dear Reviewer 7Mf4, as the discussion period is ending soon, we would appreciate it if you could review our follow-up experiments with fine-tuned baselines and let us know whether they address your concerns. Thank you for your time.
Response to Reviewer 7Mf4
We thank the reviewer for the careful assessment and for recognizing the alignment between GIRO’s design and its robustness objective, as well as the value of its theoretical analysis. We address the three empirical concerns below.
Response to Weakness 1: Modest Gains Over a Strong Baseline
We agree that the improvement in aggregate worst-case return over MOBILE is modest (714.2 versus 707.1). However, GIRO (VAE) also reduces the sum of disturbance variances from 496.09 to 137.78, a reduction of approximately 72%, with lower variance on seven of nine datasets. This stability is consistent with the intended benefit of our minimax formulation: optimizing against adverse dynamics encourages policies that remain effective across plausible model variations. Thus, the results show substantially greater consistency across disturbances alongside the modest improvement in aggregate worst-case return.
Response to Weakness 2: Lower Average Performance
We acknowledge that MOBILE and RAMBO achieve higher overall average returns. As discussed in our response to Weakness 1, GIRO achieves a higher aggregate worst-case return than MOBILE and substantially lower disturbance variance, consistent with its minimax objective.
If average performance across disturbed transition models were the primary goal, the corresponding objective would be maxπm1∑j=1mVπPj(ρ). GIRO instead optimizes against the worst admissible dynamics. The average-return comparison identifies a limitation of our method, but alone does not establish weaker worst-case robustness.
Response to Weakness 3: Limited Perturbation Range
We thank the reviewer for this suggestion. We evaluated all methods under body-mass and friction scaling factors of 0.8, 0.9, 0.95, 1.05, 1.1, and 1.2, using 20 episodes per condition and three seeds. Our original hyperparameters were tuned for small perturbations. Under this wider range, the original GIRO (VAE) checkpoints achieve a lower sum of worst-case returns than RAMBO (476.4 versus 530.5).
To adapt GIRO to the broader perturbation range, we fine-tuned the existing GIRO (VAE) checkpoints for an additional 200,000 training steps. We used the same settings across all datasets: a larger uncertainty budget (budget scale 1.02, compared with the original 0.94–1.00) and a larger adversarial update step size (3×10−6, compared with 10−7).
For every dataset, we report the checkpoint after these additional 200,000 steps. The fine-tuned GIRO checkpoints use 1.2 million training steps in total, whereas the evaluated CQL, MOPO, IQL, and RAMBO checkpoints use 2 million steps. Experiments with GIRO (Diffusion) are still running, and we will include those results once available.
Fine-tuned GIRO (VAE) achieves the highest sum of worst-case returns among the evaluated methods (603.4, compared with 530.5 for RAMBO). Relative to the original GIRO checkpoints, fine-tuning improves worst-case returns on seven of nine datasets. The largest gains occur on Hopper-ME (37.5 → 102.6), Walker2d-ME (63.8 → 86.1), Walker2d-M (65.8 → 84.4), and Walker2d-MR (49.2 → 61.1). Performance is essentially unchanged on HalfCheetah-M (62.9 → 62.8) and decreases slightly on HalfCheetah-ME (59.3 → 57.1).
These results demonstrate improved robustness after adapting GIRO’s budget and update settings to the broader perturbation range. We include the detailed per-task, per-disturbance results and the fine-tuning configuration in Appendix C.3.