acceptodds
Under review as a conference paper at ICLR 2027

RobustRLlib: A Unified Library and Benchmark for Robust Reinforcement Learning Algorithms

Abstract

Robust reinforcement learning (RL) is critical for reliable real-world deployment, where policies may fail under diverse shifts and disruptions. Existing robust RL algorithms offer a rich set of mechanisms and principles, yet two major barriers limit their broader potential: limited algorithm reusability and an unclear empirical capability boundary. We introduce RobustRLlib, an algorithm-centric library and benchmark that addresses both challenges. It integrates 16 representative robust online, offline, and safe RL methods under a unified interface for substantially improved reusability, together with an enhanced evaluation platform spanning diverse real-world shifts and advanced humanoid and vision-language-action tasks. Systematic evaluation over 273 perturbation configurations provides an up-to-date capability map of current robust RL algorithms, revealing where they remain effective and where substantial robustness gaps persist. As an open-source, user-friendly platform, RobustRLlib offers a reproducible and extensible codebase for applying existing methods to new problems and developing new robust RL algorithms.

Primary area:
datasets and benchmarks
Keywords:
Robust reinforcement learning, benchmark, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_cg9SOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the issue that current robust RL methods usually failed to adapt to new cases and thus made it difficult for performance comparison. The authors propose RobustRLlib, a robust RL benchmark with a unified algorithm library and evaluation platform. The work integrates 16 robust online, offline, and safe reinforcement learning methods and classifies them by robust mechanisms, combined with a unified perturbation interface. The benchmark conducts large-scale evaluation on standard control tasks and further demonstrates interface extension on Isaac Lab and a VLA task. According to the result of the evaluation, the paper reports that robust algorithms generally outperform standard RL under perturbations, but their advantage decreases as perturbations become more severe; the perturbation types that algorithms claim to target do not reliably predict their empirical advantage; and online robust mechanisms usually yield larger gains over their base algorithms than offline mechanisms. Overall, the work provides a unified and extensible platform for robust RL evaluation together with empirical findings on the strengths and limitations of existing robust methods.

Critical strengths

The unified implementation and perturbation interface have high potential value as it fills the gap in robust RL benchmarking, and the paper provides broad model coverage with relatively sufficient reproduction details. Integrating multiple online, offline, and safe robust RL methods into an extensible interface covering different types of perturbations is a meaningful and labor-intensive systems engineering effort. The paper also provides complete tasks, datasets, perturbation parameters, training budgets and hyperparameters to help reproduction. In addition, reporting both performance and constraint satisfaction in Isaac Lab further reflects a correct understanding that high return does not equal practical deployment.

Critical weaknesses
  1. The comparison protocol cannot cleanly identify the contribution of robust mechanisms. Different training budgets, base algorithms, training distributions, and customized adaptations all vary simultaneously, yet the paper draws mechanism-family and robustness-capability conclusions from this. In particular, ATLA receives about 2.5× the environment interactions of PPO; RFQI uses an independent best-checkpoint rule. The current experiments are not sound enough to draw the conclusions that the paper reported.
  2. The paper’s capability boundary and cross-shift conclusions are not adequately supported by the metric design and evidence coverage. The robustness score measures absolute performance under shifted conditions. In this case, a method that starts from a stronger nominal policy may still receive a higher robustness score even if it degrades more severely under shift. Adding results and analysis with metric subtracting the nominal performance would strengthen the evidence of the claimed observations. Moreover, the experiments mix two different types of shifts: frozen-policy perturbations (observation, action, dynamics, execution latency) during deployment and perturbations (reward corruption, credit latency) during training, which are not comparable within the same figure/analysis.
Questions for authors
  1. If robustness is defined as degradation relative to nominal, e.g., Δ = S_shift − S_nominal or retention/AUC, do the main conclusions about robust-vs-standard, online-vs-offline, and mechanism families still hold?
  2. Can you provide compute-matched results for the different algorithms? Will the new result still report the same gain?
  3. Q1–Q4 reflect within-axis parameter distances from nominal, which is not a unified measure of difficulty and cannot be compared across axes. Is there a better way to characterize the overall difficulty of each perturbation axis?
  4. Compound-shift results cover only two tasks and one selected best method per task. Was method selection based on the same evaluation results? How do other methods rank under compound shifts?
4 comments
AuthorsOpenReview ↗

Response to Reviewer cg9S: Part One

We appreciate the reviewer's careful evaluation of our work and recognition of the value of a unified, extensible framework for robust RL research. We have taken these concerns seriously and conducted additional experiments and analysis to provide stronger evidence for our conclusions. We respond to the specific concerns and questions below.

[W1 + Q2] Identifying the contribution of robust mechanisms by controlled comparisons under matched training budgets and checkpoint rules.

Thank you for raising this insightful question; we agree that the submission let these factors vary together. We conducted additional experiments with aligned settings (General Response G2) and added them to the revision:

  • Training budget. ATLA and ATLA-SA originally followed the official ATLA protocol (5M environment steps vs. 2M for PPO), since adversarial training needs more steps to converge. We have added matched-budget comparisons that evaluate them at the saved checkpoint closest to PPO's 2M steps, and similarly aligned the training budgets of RORL and RAMBO.
  • Checkpoint-selection rule. All methods, including RFQI (previously an exception), are now evaluated at the last checkpoint within their training budget.

Please refer to General Response G2 for the full results of these additional experiments; our main conclusions generally hold: the pooled robust–standard gap changes only from +7.9 to +6.7, and every mechanism family keeps the sign of its gain.

We also conducted additional analysis to isolate the contribution of each robustness mechanism from its base algorithm; please see General Response G3.

[W2+Q1]Robustness metric and shift protocol: absolute score versus retention, and inconsistant shifts protocol.

Thank you for these insightful suggestions.

  • Consistent shift protocol. Thank you for raising this. We conducted additional experiments in which every shift is applied to the frozen policy at test time, so all shift sources now follow the same protocol and the training-time shifts (reward corruption and credit latency) are no longer part of the evaluation. The 50 configurations are listed in Table 5 (Appendix C). The results are in Figures 7 and 8 and, per condition, in Figures 30 and 31, and the results are also at https://robust-rllib.site/results/isolated/.

  • Robustness relative to nominal: as the reviewer suggested, we conducted additional analysis with retention, S_shift / S_nominal, i.e., the fraction of its nominal score that a policy keeps under shift. Unlike the absolute score, retention does not credit a method for a high nominal score alone. We recompute on retention the three comparisons the reviewer asked about (robust vs. standard, online vs. offline, and the five mechanism families), each as a curve over the severity quartiles (https://robust-rllib.site/results/retention/, "Retention across shift severity"); the per-algorithm results are in the tables below. Read together, the two metrics refine our conclusions by showing where robustness comes from:

    • Online: robust mechanisms preserve more of the nominal score under shift, generally retaining more than their base algorithms (e.g., ATLA-SA-PPO 0.75 and ATLA-PPO 0.70 vs. PPO 0.63; DR-SAC 0.74 vs. SAC 0.70).
    • Offline: robust methods largely inherit the degradation profile of their base learner (e.g., ATLA-IQL and RSC-IQL retain 0.53, the same as IQL), so their scores under shift mainly track their nominal performance.
    • The online-vs-offline conclusion therefore holds and becomes sharper under retention, consistent with the separated comparison in General Response G3.

Online (8 algorithms).

MethodFamilyNominalScore under shiftRetention
ATLA-SA-PPOLearner-centric (online)72.455.50.75
DR-SACEnvironment-centric91.166.80.74
ATLA-PPOLearner-centric (online)78.655.50.70
SACStandard RL78.656.70.70
RARL-TRPOEnvironment-centric72.849.80.69
RSC-SACLearner-centric (online)77.952.70.67
RARL-PPOEnvironment-centric76.349.40.65
PPOStandard RL71.246.20.63

Offline (12 algorithms).

MethodFamilyNominalScore under shiftRetention
RFQILearner-centric (offline)62.342.00.68
MOPOStandard RL24.614.00.64
RORLLearner-centric (offline)63.839.40.61
TD3+BCStandard RL89.451.30.57
RAMBOData-centric85.746.70.56
PLR-PVLGenerative model102.755.00.55
FWMGenerative model105.956.60.54
ATLA-IQLLearner-centric (offline)108.056.90.53
IQLStandard RL102.553.10.53
RSC-IQLData-centric100.453.00.53
ROMBGenerative model106.654.90.52
SynthERStandard RL104.053.20.52
AuthorsOpenReview ↗

Response to Reviewer cg9S: Part Two

[Q2] Can you provide compute-matched results for the different algorithms? Will the new result still report the same gain?

We conducted additional experiments that align the training budget and checkpoint rule of every method (General Response G2):<!-- We have aligned the training budget and checkpoint rule of every method: --> ATLA-PPO and ATLA-SA-PPO are now read at PPO's 2M-step budget, RFQI at the last checkpoint inside its budget instead of its best one, and RORL and RAMBO at 1M updates instead of their official 3M and 2M. Our main conclusions remain largely unchanged. The pooled robust–standard gap moves from +7.9 to +6.7, robust methods stay ahead at every severity level (+8.4 at Q1 to +4.3 at Q4), and every mechanism family keeps the sign of its gain. Although the updated results lead to minor changes in individual algorithm rankings, the overall patterns and key insights remain consistent. Importantly, our analysis focuses on capability maps across algorithms and robustness mechanisms, rather than relying on the performance of a few specific methods.

Please refer to General Response G2 for the detailed revisions.

[Q3] Interpreting perturbation severity across different axes.

Thank you for this insightful question.

  • We agree: Q1–Q4 only indicate the relative shift magnitude of a given parameter, not a unified difficulty scale. The shift parameters differ across tasks and shift modes, and so do their units and magnitude ranges. The shift levels of every task and mode are listed in Table 4 (Appendix C).

  • We do not intend to compare difficulty across axes; Q1–Q4 only give a general sense of how far a shift moves from nominal. Such a comparison is hard to be fair as the reviewer points out, since difficulty also depends on the task and the policy: at the same gravity ×1.2, IQL keeps 50% of its nominal score on HalfCheetah but only 25% on Hopper. We therefore avoid definitive conclusions about the difficulty of different axes and only report the measured performance under each shift.

  • A better difficulty measure is beyond the scope of this work, but RobustRLlib provides exactly the platform for such analysis, with parameterized shift sweeps and a unified evaluation across tasks. One possible direction is to calibrate severity by performance instead of parameter distance: for each task and axis, define the levels as the shifts at which a fixed set of reference policies retains, e.g., 75%, 50%, and 25% of its nominal score. This would place all axes on a common, performance-based scale, although the scale would still depend on the chosen reference policies.

AuthorsOpenReview ↗

Response to Reviewer cg9S: Part Three

[Q4] Method selection and broader coverage of compound-shift evaluations.

Thank you for this question. The reviewer is correct that one method per task, selected on the same compound results, gives only a limited view. We conducted additional experiments that extend the compound evaluation to more tasks and more algorithms, training all algorithms at the unified budget of General Response G2:

  1. More tasks: we add Walker2d and HalfCheetah to the OmniH2O-based compound scenario, with the same factor set and severity as Hopper.
  2. More algorithms: Instead of one selected method per task, we report all 20 algorithms (8 online, 12 offline) on Hopper, the 8 online algorithms on Walker2d, the 12 offline algorithms on HalfCheetah, and six online algorithms on Pusher.

The results(https://robust-rllib.site/results/compound/) report the nominal score, compound score, and retention of every algorithm. The main findings are:

  • Composition remains hard across the library. On Hopper, Walker2d, and Pusher, no method retains half of its nominal score (best retention 32.0%, 39.9%, and 45.3%), so the difficulty of composition holds for every method, not only the two shown in the submission.
  • No single method leads on every task. The best online retention comes from SAC, DR-SAC and PPO on Hopper, Walker2d and Pusher, respectively, and no method keeps half of its nominal score on these three tasks. Robustness to compound shifts thus remains an open challenge for the community.

The compound score measures how well a policy performs under combined perturbations, while retention measures how much of its nominal performance it preserves. Several robust methods achieve leading compound scores, although this advantage is task-dependent and does not imply the highest retention. Retention therefore adds useful information about relative degradation, but should be interpreted alongside absolute performance, particularly when nominal scores are low. We report both metrics to distinguish policies that remain effective under perturbations from those that preserve a larger fraction of their starting performance.

AuthorsOpenReview ↗

Follow-up on our responses

Dear Reviewer,

Thank you again for your thoughtful feedback that helped us strengthen our paper. We hope our explanations and additional results have addressed the concerns you highlighted. As the discussion period draws to a close, we would welcome any further comments and would be happy to address any remaining questions.

Best regards,
The Authors

Reviewer_uCaROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents RobustRLlib, a library and benchmark for robust RL. It is designed to address two barriers: algorithm reusability and systematic characterization of the capability boundary. It consists of a robust RL algorithm library and an expandable disruptor toolbox. The library includes robust offline, online, and safe RL algorithms for users to apply. The toolbox provides various types of shifts for users to test their policies. It integrates a large set of robust algorithms and evaluates algorithms across 273 perturbation configurations. The authors report that robust methods outperform standard ones but degrade as shifts intensify.

Critical strengths
  • [S1] It targets a real problem. Robust RL implementations are fragmented. A common benchmark makes it easier to compare offline, online, and safe algorithms.
  • [S2] The evaluation protocol is standardized. Performance scores are normalized, and the evaluation grids and shift severities are predefined.
Critical weaknesses
  • [W1] The contribution is narrower than claimed. According to Table 2, compared with Robust- Gymnasium it adds only the latency shift and more robust algorithms.
  • [W2] Environments and baselines are not validated against their originals. There is no test that the wrapped environments behave like the original simulators. Many baselines are reimplemented within RobustRLlib, but none is compared with its published results.
  • [W3] Training budgets are not unified across methods. For example, ATLA and ATLA-SA are trained for ~5M steps while PPO is trained for 2M (Appendix F.2). Since the benchmark is meant to let users compare algorithms, unequal budgets can make the comparisons unfair.
  • [W4] Section 3.1 states that the advantage narrows from +10.3 points in Q1 to +5.1 in Q4 (Figure 6, left). However, reading from the Figure 6, the gains stay at about 16-18% across Q1-Q4 relative to the standard RL score.
Questions for authors
  • [Q1] What is each baseline’s nominal score compared with its original paper?

  • [Q2] Could the authors report the robust comparisons at a fixed training budget, or show learning curves for robust methods?

  • [Q3] Could the authors report the results excluding Q1? Domain randomization is trained on the range [0.9, 1.1] (Table 21), which overlaps the Q1 test values in Table 4.

7 comments
AuthorsOpenReview ↗

Response to Reviewer uCaR: Part One

We thank the reviewer for recognizing RobustRLlib’s practical value and standardized benchmark for offline, online, and safe algorithms. Below, we address each concern one by one.

[W1] Contribution beyond existing environment-centric benchmarks.

As stated in Our contributions at the beginning of the General Response, RobustRLlib is a fully complementary effort to environment-centric benchmarks: Robust-Gymnasium focuses on perturbed environments and provides only 4 algorithms, whereas RobustRLlib focuses on the algorithm side with three contributions:

  1. A unified library of 17 representative robust RL algorithms, covering online, offline, and safe RL, integrated under a unified interface as reproducible, reusable baselines.
  2. A taxonomy of robustness mechanisms: existing algorithms gain robustness by changing either how the policy learns (learner-centric, e.g., adversarial training, worst-case Bellman updates, smoothness regularization) or what it learns from (data-/environment-centric, e.g., perturbed-environment rollouts, data augmentation, generative world models). Our attribution analysis further separates the contribution of each mechanism from that of its base learner (General Response G3).
  3. A capability map of existing robust RL algorithms over 279 perturbation configurations, showing where current methods remain effective and where robustness gaps persist, as a roadmap for the robust RL community.

[W2 + Q1] Validation against the originals: wrapped environments versus the original simulators, and baseline scores versus published results.

We address the two parts of this weakness separately.

  • Environments: we do not modify the simulators, and our additional experiment verifies that the nominal environments are identical to the original papers. Every task runs on the unmodified physics engine and the standard Gymnasium task definition (MuJoCo for the locomotion and manipulation tasks); a shift is an explicit wrapper around the environment, or a declared change of a model parameter, and with no shift specified the environment is the original one. To verify this, we stepped each wrapped nominal environment and the upstream Gymnasium environment side by side from the same reset seed and with the same action sequence, and compared the observation, reward and termination signals at every step: they are bit-for-bit identical on Hopper, Walker2d, HalfCheetah, Pusher, AdroitHandDoor, PointMaze, FetchReach, CarRacing and LunarLander.
TaskUpstream referenceResult
Hopper, Walker2d, HalfCheetah, PusherGymnasium 1.2bit-for-bit identical
AdroitHandDoor, PointMaze, FetchReachGymnasium-Robotics 1.4bit-for-bit identical
CarRacingGymnasium 1.1bit-for-bit identical (image observations)
LunarLanderGymnasium 1.2bit-for-bit identical
  • Baselines [Q1]: we conducted additional experiments, and our results are broadly consistent with the original papers. We reproduced the available baseline results under the experimental settings of their original papers; General Response G1 gives the full comparison and explains how we implement each method and why some methods are adapted.

[W3 + Q2] Fair comparisons under matched training budgets(ATLA and ATLA-SA ~5M vs. PPO 2M).

As explained in General Response G2, the original 5M-versus-2M budgets follow ATLA’s published training protocol, because adversarial training requires more steps to converge. To address both [W3] and [Q2], we conducted additional experiments at a matched 2M-step budget, evaluating ATLA and ATLA-SA alongside PPO, and our conclusions generally hold. Please see General Response G2 for the detailed rationale and results.

AuthorsOpenReview ↗

Response to Reviewer uCaR: Part Two

[Q3] Robustness results excluding the training-overlapping Q1 range. Could the author report others except Q1 range?

We conducted additional analysis that recomputes every quantity of Section 3.1 on Q2–Q4 only, with the same aggregation as Figure 17/18, at the unified training budget described in General Response G2.

Q1 partially overlaps with DR-SAC’s training distribution: its locomotion test values (×0.9\times0.9, ×1.1\times1.1; Table 4) fall within the training randomization range [0.9,1.1][0.9,1.1] (Table 21), whereas Q2–Q4 lie outside it. This could could make robust methods look better at the mildest level.

Excluding Q1 preserves our conclusions: the pooled robust–standard gap is +6.1 points (+15.2%), versus +6.7 across Q1–Q4, and remains positive at each level and within every mechanism family. DR-SAC outperforms SAC by +12.3/+9.3/+5.2 points at Q2/Q3/Q4 and remains the strongest online method on Q2–Q4 (60.1 vs. 51.8 for SAC).

Per-method scores on Q2–Q4 (normalized score, mean over tasks; same aggregation as Figure 5).

MethodQ2Q3Q4Q2–Q4
DR-SAC76.958.348.260.1
SAC64.649.043.051.8
ATLA-SA-PPO61.652.342.451.7
ATLA-PPO62.251.240.250.7
RSC-SAC61.844.138.247.2
RARL-TRPO51.945.337.744.8
RARL-PPO49.443.838.443.8
PPO47.741.237.241.8
ATLA-IQL61.344.340.448.4
FWM62.145.337.047.9
PLR-PVL60.245.437.347.4
ROMB59.744.036.246.3
SynthER56.243.538.545.9
TD3+BC57.144.532.944.6
IQL57.042.135.444.6
RSC-IQL56.242.233.343.5
RAMBO50.739.632.340.6
RFQI48.036.730.238.0
RORL44.135.827.435.5
MOPO13.812.69.611.8

Mechanism families on Q2–Q4 (Table 16 construction).

Mechanism familyPaired gainPooled gain
Learner-centric (online)+4.8+9.8
Environment-centric+5.2+9.5
Learner-centric (offline)+3.8+0.5
Data-centric-1.0+2.0
Generative model+2.6+7.1

Severity decay on Q2–Q4 (pooled Standard RL vs Robust RL, as in Figure 6 left).

Q2Q3Q4Q2–Q4
Standard RL49.438.832.840.1
Robust RL57.644.937.146.1
Gap+8.2+6.1+4.3+6.1
Relative gain+16.6%+15.6%+13.2%+15.2%

[W4] Distinguishing absolute gains from relative improvements.

We agree with the reviewer’s interpretation. We have revised the paragraph "Robust RL helps at every severity, but both degrade as shifts intensify" in Section 3.1 to report both measures.

AuthorsOpenReview ↗

Follow-up on our responses

Dear Reviewer,

We appreciate your thoughtful comments and the time you have devoted to reviewing our work. We hope the clarifications and additional experiments in our responses have been helpful. Before the discussion period ends, please let us know if any concerns remain and we would be glad to address them.

Best regards,
The Authors

Reviewer_SScqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents RobustRLlib, a library that brings together online, offline, and robust-safe RL methods through a shared perturbation interface. The experiments compare these methods across tasks and distribution shifts, with additional examples in Isaac Lab and with VLA policies. The main contribution is the integration of existing algorithms and evaluation tools into a common framework.

Critical strengths
  1. The library addresses a practical problem. Robust RL methods are often released in separate codebases, making them difficult to compare or reuse. Bringing online, offline, and safe methods together is useful, and the algorithm coverage is a strong point of the paper.

  2. The experiments cover a broad range of tasks, perturbations, severity levels, and combined shifts. I appreciated the detailed configurations and per-task results, which help readers understand what is behind the aggregate scores. Multiple seeds and bootstrap intervals for aggregate gains also strengthen the evaluation.

  3. The Isaac Lab and VLA experiments give concrete examples of how the framework can be extended.

Critical weaknesses
  1. Keeping each method’s original architecture and hyperparameters is understandable, but differences in training setups make it hard to isolate the contribution of robustness mechanisms. For example, ATLA and ATLA-SA use about 5M environment interactions, compared with 2M for PPO. A few matched-budget comparisons and targeted ablations would help clarify how much these mechanisms contribute to the reported gains.

  2. The observation-shift averages in Table 5 are computed over different conditions. ATLA and ATLA-SA exclude the two MAD attacks, while other methods include all five conditions. This makes the averages difficult to compare fairly.

  3. The paper lacks systematic validation against original implementations or published results, especially for adapted methods.

  4. The novelty should be framed primarily around integration and algorithm coverage. Many perturbation mechanisms already exist in Robust Gymnasium and other benchmarks, including prior support for latency perturbations. The strongest contribution is therefore their integration with a broader algorithm library and systematic evaluation, rather than the individual perturbation operators.

Questions for authors
  1. Do the main conclusions about robustness mechanisms still hold when methods use the same base learner, comparable training budgets, and the same checkpoint-selection rule?

  2. Appendix E describes several adaptations. How were these validated against the original methods? Can you provide representative comparisons with the original implementations or published results?

Minor Correction Figure 1 appears to contain 17 robust configurations, while the text reports 16 methods. Please clarify the distinction between methods and variants and make the count consistent.

2 comments
AuthorsOpenReview ↗

Response to Reviewer SScq

We sincerely thank the reviewer for the constructive feedback and for recognizing the practical value of our unified library. Below, we address each of the raised weaknesses and questions point by point.

[W1 + Q1] Isolating the contribution of robustness mechanisms under aligned training setups: same base learner, matched training budgets, and a common checkpoint-selection rule.

Thanks for raising this insightful question. We conducted additional experiments with aligned settings (General Response G2) and added them to the revision:

  • Training budget. ATLA and ATLA-SA originally followed the official ATLA protocol (5M environment steps vs. 2M for PPO), since adversarial training needs more steps to converge. We have added matched-budget comparisons that evaluate them at the saved checkpoint closest to PPO's 2M steps, and similarly aligned the training budgets of RORL and RAMBO.
  • Checkpoint-selection rule. All methods, including RFQI (previously an exception), are now evaluated at the last checkpoint within their training budget.

Please refer to General Response G2 for the full results of these additional experiments; our main conclusions generally hold.

We also conducted additional analysis to isolate the contribution of each robustness mechanism from its base algorithm; please see General Response G3.

[W2] Inconsistent evaluation conditions in the observation-shift averages: ATLA/ATLA-SA exclude the two MAD attacks

Thank you for catching this inconsistency. We fixed this issue by conducting additional evaluations of ATLA-PPO and ATLA-SA-PPO under the two missing MAD attack conditions, using the same attack, five training seeds, and 20 episodes per condition as for every other method. Every method's observation score now averages the same five conditions, and our observations still hold with the corrected scores below.

MethodSubmission(3 conditions)Revision(5 conditions)
ATLA-PPO33.0 ± 19.035.3 ± 18.9
ATLA-SA-PPO87.0 ± 12.289.6 ± 9.8

In the revised PDF, Table 5 lists the two MAD conditions within the shared five-condition observation source; Figures 7 and 8 in the revised PDF and Appendix G in the revised PDF include both MAD conditions for every method; and the discussion in Section 3.2 reflects the corrected scores.

[W3 + Q2] Systematic validation against original implementations and published results, especially for the adapted methods.

As detailed in General Response G1, we validate the implementations of all our methods, mainly can be divided into the following two categories:

  • Original methods: we conducted additional experiments and our results are broadly consistent with those reported in their original papers.
  • Our adaptations: published results do not exist for these exact variants, but our experiments show that they are workable, with full algorithm details in Appendix E and all implementations in our released code for further validation.

Please see General Response G1 for more details.

[W4] Clarification of novelty of RobustRLlib

Thank you so much for recognizing our contributions. Yes, we agree: the primary contribution of RobustRLlib is its algorithm-centric library, which integrates 17 representative robust online, offline, and safe RL algorithms under a unified interface as reproducible, reusable baselines (see Our contributions at the beginning of the General Response).

We appreciate this suggestion and will follow it by highlighting this distinction more clearly in the revision: the algorithm library is our core contribution, while the additional shifted environments and advanced robotics tasks (humanoid and vision-language-action) serve as a side contribution that extends existing robust RL environments.

[Other questions] Consistent counting of methods and algorithm variants in appendix.

Thank you for pointing out this inconsistency. We acknowledge that this was a counting error, as we previously counted DR-PPO and DR-SAC as a single method rather than two distinct algorithm variants. We have corrected the count from 16 to 17 throughout the manuscript to ensure consistency between Figure 1 and the text.

AuthorsOpenReview ↗

Follow-up on our responses

Dear Reviewer,

Thank you again for your time and constructive feedback. We hope our responses and additional results have addressed your concerns. Before the discussion period closes, we would welcome any further feedback and would be happy to clarify any remaining questions.

Best regards,
The Authors

Comments on the paper

9 comments
AuthorsOpenReview ↗

General Response (1/4): Contributions and comparison with prior benchmarks

We thank all four reviewers for their constructive feedback. The common concerns on algorithm reproducibility, training budgets, and the isolated contributions of robustness mechanisms are exactly the questions RobustRLlib is built to help answer: with all methods in one shared framework, these factors can be controlled explicitly, which is how we conducted the additional experiments and analysis below. We first briefly restate our contributions.

Highlight Our contributions: a unified and reproducible robust RL library.

  • Motivation. Existing robust RL algorithms are developed on diverse codebases and evaluated under different environment versions, training budgets, and protocols, making it difficult for the community to compare them fairly or to build new algorithms on clear baselines.
  • What RobustRLlib provides.
    1. A unified library of 17 representative robust RL algorithms, covering online, offline, and safe RL, integrated under a unified interface as reproducible, reusable baselines.
    2. A taxonomy of robustness mechanisms: existing algorithms gain robustness by changing either how the policy learns (learner-centric, e.g., adversarial training, worst-case Bellman updates, smoothness regularization) or what it learns from (data-/environment-centric, e.g., perturbed-environment rollouts, data augmentation, generative world models). This mechanism-level view shows which components can be transferred across base learners, fostering their integration and new algorithm design.
    3. A capability map of existing robust RL algorithms over 279 perturbation configurations, showing where current methods remain effective and where robustness gaps persist, as a roadmap for the robust RL community.
  • Comparisons to prior benchmarks. RobustRLlib is a fully complementary effort for the robust RL community: prior benchmarks such as Robust-Gymnasium (Gu et al., 2025) and RRLS (Zouitine et al., 2024) focus on environment side, providing perturbed tasks for evaluation, whereas RobustRLlib focuses on the algorithm side, providing unified, reusable implementations of abundant robust RL algorithms.
BenchmarkRobust algorithmsMDP components shiftedLatencySemanticCompoundTask expansion
RLlib (Liang et al., 2018)00/4✗✗✗✓
RRLS (Zouitine et al., 2024)42/4✗✗◐✓
ODRL (Lyu et al., 2024)01/4✗✗✗✗
Robust-Gymnasium (Gu et al., 2025)44/4✗✓✓✓
RWRL Suite (Dulac-Arnold et al., 2020)03/4✓✗✓◐
RoAd-RL (Mohan et al., 2026)01/4✗✗✗✓
RobustRLlib (ours)174/4✓✓✓✓

Table 2 of the paper. Robust algorithms counts the robust or robust safe RL algorithms provided by each benchmark. MDP components shifted counts, out of four, the components on which a shift is evaluated: observations, actions, transitions, and reward/cost signals. ✓ explicit, evaluated support; ◐ partial or scope-limited support; ✗ absent or not demonstrated in the cited release.

Below, we address the three common questions raised by reviewers (G1–G3).

AuthorsOpenReview ↗

General Response (2/4): G1. Algorithm reproducibility

G1. Algorithm reproducibility.

We thank the reviewers for raising this important question, indeed reproducible implementations are a central contribution of RobustRLlib.

Two categories of methods in RobustRLlib.

  • Original methods (all methods except the six adaptations below): We use the authors’ released code where available and otherwise reproduce the method from its paper, following the reported architectures, hyperparameters, and algorithmic procedures.
  • Our adaptations (FWM, ATLA-IQL, RSC-IQL, ROMB, RAMU-PPO, PLR-PVL): We keep each method's core robustness mechanism and only adapt the base learner or model backbone, which separates the effect of the mechanism from that of the base algorithm.

Original methods: we conducted additional experiments, and our results are consistent with the original papers. To validate our implementations, we compared them against the results reported in the original papers, retraining the offline methods from scratch on the original D4RL datasets for this comparison. As shown below, our results are broadly consistent with the reported ones, supporting the correctness of our implementations:

Online methodsOur resultsOriginal paper resultsOriginal paper
PPO23.4 ± 10.9 / 75.5 ± 8.3 / 77.1 ± 8.713.9 / 73.8 / 49.8Huang et al. (2022)
SAC64.2 ± 25.3 / 71.0 ± 27.8 / 104.8 ± 5.579.9 / 71.6 / 78.2Huang et al. (2022)
ATLA-PPO45.3 ± 5.0 / 67.5 ± 28.4 / 83.5 ± 8.345.9 ± 0.4 / 79.3 ± 29.4 / 68.3 ± 23.1Zhang et al. (2021)
ATLA-SA-PPO44.0 ± 5.5 / 102.2 ± 10.5 / 63.6 ± 23.051.8 ± 6.9 / 101.7 ± 18.4 / 83.7 ± 10.3Zhang et al. (2021)
RARL-TRPO39.0 ± 1.9 / 76.3 ± 24.7 / 97.9 ± 31.746.1 / 110.9 / 127.5Pinto et al. (2017)
RSC-SAC (Door)101.8 ± 34.7 / 65.8 ± 28.898 ± 5 / 61 ± 17Ding et al. (2023)

HalfCheetah / Hopper / Walker2d, mean ± s.d. over seeds, Gymnasium v5.

Offline methodsOur resultsOriginal paper resultsOriginal paper
RORL104.6 ± 0.7104.8 ± 0.1Yang et al. (2022)
RAMBO91.2 ± 7.692.8 ± 6.0Rigter et al. (2022)
SynthER64.2 ± 0.972.0 ± 4.5Lu et al. (2023)
TD3+BC56.9 ± 1.559.3 ± 4.2Fujimoto & Gu (2021)
IQL53.7 ± 2.766.3Kostrikov et al. (2021)
MOPO40.0 ± 23.748.0Rigter et al. (2022)
RFQI38.6 ± 7.541.0 ± 4.9‡Chen et al. (2025)

D4RL hopper-medium-v2, retrained on the original data and evaluated in the original D4RL environment, mean ± s.d. over seeds. ‡The original paper of RFQI (Panaganti et al., 2022) reports learning curves only. This is the result of Chen et al. (2025), Table 1, who evaluate under 5% measurement noise.

The small remaining gaps between re-implementation and original results are normal in RL [1–6]. Those gaps here mainly come from (i) environment versions: our online methods run on Gymnasium v5 rather than v2–v4 in original papers; and (ii) evaluation protocol: every method is evaluated at its last checkpoint over five seeds × 20 episodes, rather than with each paper's own checkpoint selection and reporting statistics.

[1] Tarasov, Denis, et al. "CORL: Research-oriented deep offline reinforcement learning library." Advances in Neural Information Processing Systems 36 (2023): 30997-31020.

[2] Qin, Rong-Jun, et al. "NeoRL: A near real-world benchmark for offline reinforcement learning." Advances in Neural Information Processing Systems 35 (2022): 24753-24765.

[3] Seno, Takuma, and Michita Imai. "d3rlpy: An offline deep reinforcement learning library." Journal of Machine Learning Research 23.315 (2022): 1-20.

[4] Raffin, Antonin, et al. "Stable-Baselines3: Reliable reinforcement learning implementations." Journal of Machine Learning Research 22.268 (2021): 1-8.

[5] Henderson, Peter, et al. "Deep reinforcement learning that matters." Proceedings of the AAAI Conference on Artificial Intelligence 32.1 (2018): 3207-3214.

[6] Engstrom, Logan, et al. "Implementation matters in deep RL: A case study on PPO and TRPO." International Conference on Learning Representations (2020).

AuthorsOpenReview ↗

General Response (3/4): G1 Algorithm reproducibility, our adaptations

G1. Algorithm reproducibility (continued).

Our adaptations: workable and fully documented. Published results do not exist for these exact variants, so a direct numerical comparison is not possible. Our experiments show that all of them are workable: their nominal (unshifted) performance is on par with their base learners, 106.3–119.6 for the five IQL-based adaptations vs. 117.7 for IQL on MuJoCo tasks (table below). Full algorithm details are in Appendix E, and all implementations and configurations are in our released code for inspection.

MethodHopperWalker2dHalfCheetahMean
IQL (base learner)106.8 ± 4.3125.1 ± 5.4121.3 ± 7.2117.7
ATLA-IQL107.2 ± 4.9130.9 ± 6.0112.3 ± 13.9116.8
RSC-IQL100.8 ± 11.8118.2 ± 19.399.9 ± 14.3106.3
ROMB99.1 ± 14.7133.1 ± 2.9120.4 ± 7.3117.6
FWM101.1 ± 14.6132.7 ± 2.7123.7 ± 5.0119.2
PLR-PVL106.7 ± 4.6134.1 ± 0.8117.9 ± 9.3119.6

Nominal normalized score (no shift) of adapted methods, mean ± s.d. over five seeds.

Summary: unifying previously fragmented implementations is itself a core contribution of RobustRLlib. With explicit configurations and a single evaluation protocol, implementation choices and discrepancies become auditable, and the community, including the original authors, can verify and improve each method on a shared codebase.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.