- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the issue that current robust RL methods usually failed to adapt to new cases and thus made it difficult for performance comparison. The authors propose RobustRLlib, a robust RL benchmark with a unified algorithm library and evaluation platform. The work integrates 16 robust online, offline, and safe reinforcement learning methods and classifies them by robust mechanisms, combined with a unified perturbation interface. The benchmark conducts large-scale evaluation on standard control tasks and further demonstrates interface extension on Isaac Lab and a VLA task. According to the result of the evaluation, the paper reports that robust algorithms generally outperform standard RL under perturbations, but their advantage decreases as perturbations become more severe; the perturbation types that algorithms claim to target do not reliably predict their empirical advantage; and online robust mechanisms usually yield larger gains over their base algorithms than offline mechanisms. Overall, the work provides a unified and extensible platform for robust RL evaluation together with empirical findings on the strengths and limitations of existing robust methods.
Critical strengths
The unified implementation and perturbation interface have high potential value as it fills the gap in robust RL benchmarking, and the paper provides broad model coverage with relatively sufficient reproduction details. Integrating multiple online, offline, and safe robust RL methods into an extensible interface covering different types of perturbations is a meaningful and labor-intensive systems engineering effort. The paper also provides complete tasks, datasets, perturbation parameters, training budgets and hyperparameters to help reproduction. In addition, reporting both performance and constraint satisfaction in Isaac Lab further reflects a correct understanding that high return does not equal practical deployment.
Critical weaknesses
- The comparison protocol cannot cleanly identify the contribution of robust mechanisms. Different training budgets, base algorithms, training distributions, and customized adaptations all vary simultaneously, yet the paper draws mechanism-family and robustness-capability conclusions from this. In particular, ATLA receives about 2.5× the environment interactions of PPO; RFQI uses an independent best-checkpoint rule. The current experiments are not sound enough to draw the conclusions that the paper reported.
- The paper’s capability boundary and cross-shift conclusions are not adequately supported by the metric design and evidence coverage. The robustness score measures absolute performance under shifted conditions. In this case, a method that starts from a stronger nominal policy may still receive a higher robustness score even if it degrades more severely under shift. Adding results and analysis with metric subtracting the nominal performance would strengthen the evidence of the claimed observations. Moreover, the experiments mix two different types of shifts: frozen-policy perturbations (observation, action, dynamics, execution latency) during deployment and perturbations (reward corruption, credit latency) during training, which are not comparable within the same figure/analysis.
Questions for authors
- If robustness is defined as degradation relative to nominal, e.g., Δ = S_shift − S_nominal or retention/AUC, do the main conclusions about robust-vs-standard, online-vs-offline, and mechanism families still hold?
- Can you provide compute-matched results for the different algorithms? Will the new result still report the same gain?
- Q1–Q4 reflect within-axis parameter distances from nominal, which is not a unified measure of difficulty and cannot be compared across axes. Is there a better way to characterize the overall difficulty of each perturbation axis?
- Compound-shift results cover only two tasks and one selected best method per task. Was method selection based on the same evaluation results? How do other methods rank under compound shifts?
4 comments
Response to Reviewer cg9S: Part Two
[Q2] Can you provide compute-matched results for the different algorithms? Will the new result still report the same gain?
We conducted additional experiments that align the training budget and checkpoint rule of every method (General Response G2):<!-- We have aligned the training budget and checkpoint rule of every method: --> ATLA-PPO and ATLA-SA-PPO are now read at PPO's 2M-step budget, RFQI at the last checkpoint inside its budget instead of its best one, and RORL and RAMBO at 1M updates instead of their official 3M and 2M. Our main conclusions remain largely unchanged. The pooled robust–standard gap moves from +7.9 to +6.7, robust methods stay ahead at every severity level (+8.4 at Q1 to +4.3 at Q4), and every mechanism family keeps the sign of its gain. Although the updated results lead to minor changes in individual algorithm rankings, the overall patterns and key insights remain consistent. Importantly, our analysis focuses on capability maps across algorithms and robustness mechanisms, rather than relying on the performance of a few specific methods.
Please refer to General Response G2 for the detailed revisions.
[Q3] Interpreting perturbation severity across different axes.
Thank you for this insightful question.
-
We agree: Q1–Q4 only indicate the relative shift magnitude of a given parameter, not a unified difficulty scale. The shift parameters differ across tasks and shift modes, and so do their units and magnitude ranges. The shift levels of every task and mode are listed in Table 4 (Appendix C).
-
We do not intend to compare difficulty across axes; Q1–Q4 only give a general sense of how far a shift moves from nominal. Such a comparison is hard to be fair as the reviewer points out, since difficulty also depends on the task and the policy: at the same gravity ×1.2, IQL keeps 50% of its nominal score on HalfCheetah but only 25% on Hopper. We therefore avoid definitive conclusions about the difficulty of different axes and only report the measured performance under each shift.
-
A better difficulty measure is beyond the scope of this work, but RobustRLlib provides exactly the platform for such analysis, with parameterized shift sweeps and a unified evaluation across tasks. One possible direction is to calibrate severity by performance instead of parameter distance: for each task and axis, define the levels as the shifts at which a fixed set of reference policies retains, e.g., 75%, 50%, and 25% of its nominal score. This would place all axes on a common, performance-based scale, although the scale would still depend on the chosen reference policies.
Response to Reviewer cg9S: Part Three
[Q4] Method selection and broader coverage of compound-shift evaluations.
Thank you for this question. The reviewer is correct that one method per task, selected on the same compound results, gives only a limited view. We conducted additional experiments that extend the compound evaluation to more tasks and more algorithms, training all algorithms at the unified budget of General Response G2:
- More tasks: we add Walker2d and HalfCheetah to the OmniH2O-based compound scenario, with the same factor set and severity as Hopper.
- More algorithms: Instead of one selected method per task, we report all 20 algorithms (8 online, 12 offline) on Hopper, the 8 online algorithms on Walker2d, the 12 offline algorithms on HalfCheetah, and six online algorithms on Pusher.
The results(https://robust-rllib.site/results/compound/) report the nominal score, compound score, and retention of every algorithm. The main findings are:
- Composition remains hard across the library. On Hopper, Walker2d, and Pusher, no method retains half of its nominal score (best retention 32.0%, 39.9%, and 45.3%), so the difficulty of composition holds for every method, not only the two shown in the submission.
- No single method leads on every task. The best online retention comes from SAC, DR-SAC and PPO on Hopper, Walker2d and Pusher, respectively, and no method keeps half of its nominal score on these three tasks. Robustness to compound shifts thus remains an open challenge for the community.
The compound score measures how well a policy performs under combined perturbations, while retention measures how much of its nominal performance it preserves. Several robust methods achieve leading compound scores, although this advantage is task-dependent and does not imply the highest retention. Retention therefore adds useful information about relative degradation, but should be interpreted alongside absolute performance, particularly when nominal scores are low. We report both metrics to distinguish policies that remain effective under perturbations from those that preserve a larger fraction of their starting performance.
Follow-up on our responses
Dear Reviewer,
Thank you again for your thoughtful feedback that helped us strengthen our paper. We hope our explanations and additional results have addressed the concerns you highlighted. As the discussion period draws to a close, we would welcome any further comments and would be happy to address any remaining questions.
Best regards,
The Authors
Response to Reviewer cg9S: Part One
We appreciate the reviewer's careful evaluation of our work and recognition of the value of a unified, extensible framework for robust RL research. We have taken these concerns seriously and conducted additional experiments and analysis to provide stronger evidence for our conclusions. We respond to the specific concerns and questions below.
Thank you for raising this insightful question; we agree that the submission let these factors vary together. We conducted additional experiments with aligned settings (General Response G2) and added them to the revision:
Please refer to General Response G2 for the full results of these additional experiments; our main conclusions generally hold: the pooled robust–standard gap changes only from +7.9 to +6.7, and every mechanism family keeps the sign of its gain.
We also conducted additional analysis to isolate the contribution of each robustness mechanism from its base algorithm; please see General Response G3.
Thank you for these insightful suggestions.
Consistent shift protocol. Thank you for raising this. We conducted additional experiments in which every shift is applied to the frozen policy at test time, so all shift sources now follow the same protocol and the training-time shifts (reward corruption and credit latency) are no longer part of the evaluation. The 50 configurations are listed in Table 5 (Appendix C). The results are in Figures 7 and 8 and, per condition, in Figures 30 and 31, and the results are also at https://robust-rllib.site/results/isolated/.
Robustness relative to nominal: as the reviewer suggested, we conducted additional analysis with retention, S_shift / S_nominal, i.e., the fraction of its nominal score that a policy keeps under shift. Unlike the absolute score, retention does not credit a method for a high nominal score alone. We recompute on retention the three comparisons the reviewer asked about (robust vs. standard, online vs. offline, and the five mechanism families), each as a curve over the severity quartiles (https://robust-rllib.site/results/retention/, "Retention across shift severity"); the per-algorithm results are in the tables below. Read together, the two metrics refine our conclusions by showing where robustness comes from:
Online (8 algorithms).
Offline (12 algorithms).