acceptodds
Under review as a conference paper at ICLR 2027

Provably Safe Sim-to-Real Transfer

Abstract

We address safe sim-to-real transfer, in which an agent leverages an imperfect simulator and limited real-world interaction while ensuring safety throughout data collection in the real system. This problem arises in applications such as robotics and healthcare: simulators provide cheap data, but sim-to-real mismatch makes direct transfer unreliable, and collecting real-world data to correct this mismatch must itself be safe. Moreover, deployment objectives may vary across tasks, making it costly to collect new data for each reward function. We therefore formulate safe sim-to-real transfer as a reward-free safe reinforcement learning (RL) problem, in which data are collected once and reused to plan for arbitrary reward functions. We develop a computationally efficient algorithm that identifies where the simulator and real dynamics differ, uses certified simulator transitions where they are reliable, and estimates mismatched transitions from safely collected data. With high probability, every policy deployed during learning is feasible, and the collected data support the computation of a feasible and near-optimal policy for any reward function. When the simulator is uninformative, our algorithm recovers online reward-free safe RL while improving the best-known sample complexity by a factor of , where is the safety margin of a baseline policy. When the simulator is accurate on most transitions, this improvement grows to , where $|B|$ denotes the size of the sim-to-real mismatch region.

Primary area:
reinforcement learning
Keywords:
sim-to-real, reinforcement learning, safe exploration, convergence analysis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nFNKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies safe sim-to-real transfer. The authors formulates safe-sim-to real transfer as a reward-model safe RL problem and develop computationally efficient algorithm. Their theoretical results argue that, with high probability, every policy deployed during learning is feasible, and the collected data support the computation of a feasible and near-optimal policy for any reward function.

Critical strengths

I think there are the following three strengths.

  1. I think that the problem setting formulated in this paper is interesting and important. This formulation is related to safety-critical domains such robotics or healthcare.
  2. Assuming that Assumption 2.4 holds, the theoretical results presented by the authors are useful and novel. I read through the proofs, and I did not find major issues.
  3. The proposed algorithm is technically reasonable.
Critical weaknesses

I also have some concerns.

  1. Though Assumption 2.3 (i.e., Slator's condition) is common and natural, I am not fully convinced that Assumption 2.4 is reasonable. I think that the sim-to-real transfer is difficult mainly because such state transition that satisfies Assumption 2.4 cannot be easily obtained. I feel making assumption 2.4 implicitly avoids the essential difficulty of sim-to-real RL.
  2. Additionally, if I understand correctly, I feel it is very difficult to learn such state transition dynamics that satisfies Assumption 2.4. This assumption would be related to out-of-distribution (OOD) in RL, I am not sure how we can learn such state transition functions.
  3. While I understand that this paper is theoretical, I consider that the empirical experiments in this paper is rather weak. I would like the authors to conduct some experiments in common safe RL benchmarks (e.g., safety-gymnasium). If the authors conduct some experiments in more complicated benchmark problems, I think the authors would understand my concerns regarding Assumption 2.4.
Questions for authors

This paper presents some interesting ideas with rigorous theoretical analyses. Although this paper presents a novel and interesting theoretical result, I feel it is largely based on Assumption 2.4. I am not fully confident that Assumption 2.4 is reasonable and how we can obtain such estimated state transitions that the assumption holds. Also, the empirical experiments are conducted only with toy environments and I do not feel the authors' claims are sufficiently supported.

My questions are:

  1. Could you elaborate more on why the authors consider Assumption 2.4 reasonable? Does it hold in real applications?

  2. How can we learn such state transition functions that satisfies Assumption 2.4 in complicated tasks (e.g., safety-gymnasium)?

3 comments

Thanks for your input and questions that help us clarify the setting and contributions.

  1. Choice of Assumption 2.4: If we are given only a simulator and without prior mismatch information, we cannot guarantee that learning with the simulator offers any benefit over online learning without it [1]. This is because the agent need to explore the entire state–action space to determine where the simulator can be trusted, and the cost of this exploration can eliminate any potential sample savings from the simulator. Therefore, a central challenge in efficient sim-to-real transfer is how to leverage prior mismatch information.

    Our paper studies efficient sim-to-real transfer under one form of prior mismatch information, specified by Assumption 2.4. Assumption 2.4 has also been used in hierarchical RL (Chua et al., 2023), meta-RL (Chen et al., 2022; Mutti & Tamar, 2024; Brunskill & Li, 2013), and sim-to-real RL (Qu et al., 2025). This assumption requires knowledge of ϵs\epsilon_s and σs\sigma_s. These parameters separate sim-to-real mismatches, measured in total variation distance, into two categories: negligible mismatches of at most ϵs\epsilon_s and significant mismatches of at least σs\sigma_s. It does not require knowledge of the mismatch region or the real transition dynamics. And our algorithm identifies these regions and safely learning the real dynamics.

    We would like to point out that the simulator is given, and we do not learn a simulator whose transitions must satisfy Assumption 2.4. Without knowledge of ϵs\epsilon_s and σs\sigma_s, we can set ϵs=σs=0\epsilon_s=\sigma_s=0. This recovers the fully online version discussed after Theorem 4.1, where our algorithm improves upon the sample-complexity bound of Huang et al. (2023). In such case, Assumption 2.4 then holds for any simulator, since the total-variation distance always lies in [0,1][0,1]. With σs=0\sigma_s=0, the mismatch region includes all time–state–action triples, and the algorithm learns real-world transitions without simulator reuse.

  2. Practicability of Assumption 2.4: The assumption captures situations where a simulator models some parts of the real-world dynamics accurately but others poorly. For example, in robotics, a simulator may be accurate in nominal operating regimes but less reliable during contact-rich interactions, such as grasping deformable objects. Here, ϵs\epsilon_s provides an upper bound on the small differences between simulated and real transition probabilities in nominal regimes, whereas σs\sigma_s provides a lower bound on the larger differences during these grasping interactions. Given ϵs\epsilon_s and σs\sigma_s, our algorithm uses (ϵs+σs)/2(\epsilon_s+\sigma_s)/2 as a threshold to distinguish between negligible and significant mismatches and estimate the mismatch region. Based on this estimate, we construct a hybrid model that combines simulated and learned dynamics and design a mixture policy to safely explore the mismatch region and collect data ti refine hybrid model for accurate reward-free planning.

    Furthermore, we provide Algorithm 4 in Appendix B.6 for estimating σs\sigma_s. If a safe baseline policy explores different grasping motions on these objects, Algorithm 4 can use this policy to collect real-world data and estimate σs\sigma_s. Similarly, if a safe baseline policy explores nominal regimes known to form the negligible-mismatch region, we can adapt Algorithm 4 to estimate ϵs\epsilon_s by taking the maximum upper confidence bound on their mismatch instead of the minimum lower confidence bound used to estimate σs\sigma_s.

  3. Gridworld and continuous-control benchmarks: Gridworld provides a controlled testbed that matches our finite-state, finite-action setting. Small changes to the environment allow us to vary ϵs\epsilon_s and σs\sigma_s and study their effects, enabling direct comparisons between our theoretical predictions and empirical results. Safety-Gymnasium focuses on continuous-control tasks, which fall outside the setting analyzed in this paper. We leave it as future work to extend our results to the continuous setting.

[1] Leveraging (Biased) information: Multi-armed bandits with offline data.

Thank you for the response. However, I remain concerned that the assumption shifts much of the difficulty of sim-to-real transfer into obtaining the required prior knowledge. In challenging tasks, it is still unclear whether acquiring this knowledge is substantially easier than solving the original problem. This is why I suggested more challenging experiments. I am not convinced whether the required prior knowledge is practically obtainable. Therefore, I will keep my original rating.

For challenging tasks like robotic grasping and manipulation, simulators may accurately predict free-space motion yet make substantial errors during interactions with objects [1]. Assumption~2.4 captures such cases where sim-to-real errors are negligible during free-space motion but substantially larger during contact-rich interactions with objects.

In the following, we provide a possible approach to estimating σs\sigma_s and ϵs\epsilon_s in a robotic grasping task. Under a well-calibrated stochastic transition model, such as Gaussian transitions with a known noise covariance, bounds on the one-step physical prediction error $e_h(s,a) =\left|f_h^{\mathrm{sim}}(s,a)-f_h^{\mathrm{real}}(s,a)\right|_2, $ where fhsimf_h^{\mathrm{sim}} and fhrealf_h^{\mathrm{real}} denote the mean next states, can be translated into TV mismatch bounds for estimating σs,ϵs\sigma_s, \epsilon_s.

These physical errors can be estimated by comparing measurements from the simulator and the real world. For contact-related mismatches, [2] identifies contact parameters and evaluates discrepancies between simulated and real trajectories, while [3] quantifies discrepancies in post-impact velocities. In contrast, small discrepancies have been observed during free-space motion: [4] reports an average joint-position discrepancy of approximately 0.00250.0025 rad between simulated and real UR5e trajectories during free-space motion.

[1] Focused Adaptation of Dynamics Models for Deformable Object Manipulation.

[2] Validating Robotics Simulators on Real-World Impacts.

[3] Impact-Aware Robotic Manipulation: Quantifying the Sim-To-Real Gap for Velocity Jumps.

[4] Experimental Evaluation of Collision Avoidance Techniques for Collaborative Robots.

Reviewer_jKNJOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses safe sim-to-real transfer, whose objective is to use an imperfect simulator and a reduced number of real-world interactions to learn safe and near-optimal policies. Assuming a strictly feasible baseline policy and a separation gap between negligible and large mismatches of simulated and real transitions, the author(s) propose a computationally efficient algorithm that relies on real dynamics only in regions where the simulator is inaccurate. High-probability guarantees for safe exploration and accurate planning are provided. The real-world sample complexity is shown to scale with mismatch region size, and the proposed exploration policy even improves over the sample complexity bound obtained in (Huang et al. 2023) for simulator-free settings. A grid world experiment confirms the scheme stays safe while using far fewer samples than a simulator-free baseline.

Overall, the paper is well-written and presents a novel algorithm along with theoretical guarantees that will be of great interest to the safe RL theory audience.

Critical strengths
  • The paper is the first to address the problem of reward-free safe RL with an imperfect simulator, and the proposed algorithm addresses the problem efficiently with clear guarantees. Although the approach is designed for basic tabular settings, it is useful for a theoretical RL audience and can provide insights into safe RL exploration for general settings.

  • The computational improvement over RF-RL (Huang et al. 2023), obtained by optimizing a linear bonus over empirically feasible policies and then mixing the candidate with the safe baseline, is quite interesting. Shifting pessimism from the optimization to the execution step turns RF-RL’s nonconvex exploration problem into a standard CMDP solvable as an LP, and the margin-dependent mixing weight preserves both safety and exploration progress.

Critical weaknesses
  • The paper's motivation emphasizes real-world, safety-critical systems, but the discussion of how the approach extends beyond tabular settings is limited. It would be interesting for the reader to discuss which components of the proposed algorithm would carry over to more general settings.
Questions for authors
  • The experiments use two baselines: reward-free safe RL, which ignores the simulator, and unconstrained sim-to-real RL, which drops safety entirely. Algorithm 1 drives efficient exploration by optimizing the candidate policy over feasible policies and ensures safety by mixing the candidate policy with π0\pi^0 using an adaptive weight αt\alpha^t. Ablations with the ‘constraint without mixing’ and ‘mixing without the constraint’ would isolate the benefits of each mechanism.

  • The mechanism used to guarantee safe exploration (falling back on, or mixing with, a strictly safe baseline policy when the candidate policy cannot be certified feasible; Lines 5 and 13 of Algorithm 1) is closely related to the safe-policy deployment techniques of Kitamura et al. (NeurIPS 2025) in linear CMDPs and Pacchiano et al. (AISTATS 2021) in safe linear bandits. The authors should discuss this line of work, at least in the related work section.

Kitamura, Toshinori, et al. "Provably efficient RL under episode-wise safety in constrained MDPs with linear function approximation." Advances in Neural Information Processing Systems, 2025.

Pacchiano, Aldo, et al. "Stochastic bandits with linear constraints." International conference on artificial intelligence and statistics. PMLR, 2021.

3 comments

Thanks for your input and questions that help us clarify the setting and contributions.

  • Additional experiments: Thanks for the suggestion to include ablation studies to isolate the benefits of each mechanism in our algorithm. We have conducted additional experiments on our gridworld experiments, as described below, and will include these results in the revised manuscript.

    1. Constraint without mixing. We directly deploy the candidate policy

      πˉt∈arg⁡max⁡π∈ΠfeasP^tVb,1P^t,π(s1),\bar{\pi}^t \in \arg\max_{\pi\in\Pi_{\mathrm{feas}}^{\widehat P^t}} V_{b,1}^{\widehat P^t,\pi}(s_1),

      without the mixing step in Line 13. This tests the mixing strategy's role in ensuring safe exploration.

    2. Mixing without the constraint. We remove the empirical feasibility constraint when computing the candidate policy:

      πˉt∈arg⁡max⁡π∈ΠVb,1P^t,π(s1),\bar{\pi}^t \in \arg\max_{\pi\in\Pi} V_{b,1}^{\widehat P^t,\pi}(s_1),

      while retaining the adaptive mixture

      πt=αtπˉt+(1−αt)π0.\pi^t=\alpha^t\bar{\pi}^t+(1-\alpha^t)\pi^0.

      The weight αt\alpha^t is computed using Equation (9) for this variant's candidate. This tests whether restricting candidate selection to empirically feasible policies improves the sample efficiency of the algorithm while still ensuring safe exploration.

    The results are summarized below.

    AlgorithmSamples from Mreal\mathcal{M}^{\mathrm{real}}Suboptimality gapUnsafe iterations
    Safe sim-to-real RL39,168±67239{,}168 \pm 6720.003±0.0080.003 \pm 0.0080.0%
    Reward-free safe RL135,456±545135{,}456 \pm 5450.003±0.0070.003 \pm 0.0070.0%
    Unconstrained sim-to-real RL21,114±27621{,}114 \pm 2760.003±0.0090.003 \pm 0.00951.4%
    Constraint without mixing20,970±28020{,}970 \pm 2800.003±0.0090.003 \pm 0.00941.5%
    Mixing without constraint42,072±79042{,}072 \pm 7900.003±0.0080.003 \pm 0.0080.0%

    Constraint without mixing requires approximately 0.7%0.7\% fewer real-world samples than unconstrained sim-to-real RL, and has a lower unsafe iteration rate. However, neither approach guarantees safe exploration. Mixing without the constraint ensures safe exploration but requires approximately 7.4%7.4\% more real-world samples than the original algorithm (Safe sim-to-real RL). These results reveal complementary roles of the two mechanisms: mixing ensures safe exploration, while restricting candidate selection to empirically feasible policies improves sample efficiency.

  • Related work: Thank you for pointing out two missing references on online learning with safe exploration guarantees in the setting of linear CMDPs (Kitamura et al., NeurIPS 2025) and linear bandits (Pacchiano et al., AISTATS 2021). We will incorporate both into the related work section, in the paragraph following our discussion of meta-RL, as follows:

    To address safety during real-world data collection, a separate line of work studies online safe learning, including model-based (Yu et al., 2025; Bura et al., 2022; Liu et al., 2021; Wendl et al., 2026) and model-free (Ni & Kamgarpour, 2025b; Kitamura et al., 2025) RL methods, as well as methods for constrained linear bandits (Pacchiano et al., 2021). These methods provide high-probability guarantees for learning a near-optimal policy under safe exploration.

    We agree that the check of the baseline's safety margin in Line 5 also appears in safe RL methods, including DOPE (Bura et al., 2022) and DOPE+ (Yu et al., 2025), as well as Kitamura et al. (2025). We have acknowledged these connections when introducing this step in the revised manuscript. The mixing strategy in Line 13 differs from the policy computation in Kitamura et al. (2025). Their algorithm does not deploy a mixture policy. Instead, it constructs a softmax policy parameterized by a constraint multiplier λ\lambda, which controls the weight of the pessimistic constraint value relative to the optimistic reward objective. They use bisection over λ\lambda to select an exploration policy satisfying the pessimistic constraint. This approach could serve as a potential alternative to our mixing strategy for designing safe exploration policies.

Thank you for your response. It would be great if you could also comment on the raised weakness: 'The paper’s motivation emphasizes real-world, safety-critical systems, but the discussion of how the approach extends beyond the tabular setting is limited. It would benefit readers if the authors discussed which components of the proposed algorithm would carry over to more general settings'.

Our algorithm provides a model-based approach to safe sim-to-real transfer in discrete settings. More importantly, it quantifies the real-world sample complexity and makes explicit the prior mismatch information under which safe and sample-efficient transfer can be guaranteed. One possible extension to continuous settings is to replace the tabular transition model with a well-calibrated statistical model, such as a Gaussian process [1,2] or neural-network ensemble [3,4,5]. Under appropriate assumptions, such a model could provide confidence bounds on discrepancies between predicted and real-world trajectories, connecting directly to our confidence-based approach to identifying mismatch regions.

Given such a well-calibrated model, our algorithm suggests three mechanisms for this extension: (a) identifying unreliable simulator regions using confidence bounds on estimated real-world dynamics, (b) constructing a hybrid model that combines simulated and learned dynamics, and (c) using a mixture policy for safe exploration. Parts (a) and (b) could be adapted by replacing tabular confidence bounds with those provided by the statistical model and using them to determine estimated mismatch region. For part (c), we first need to extend Lemma~3.1 to continuous settings to support the design of a candidate policy that is empirically feasible under the hybrid model while exploring regions relevant to reducing planning error. We can then adapt the mixture policy to compensate for potential constraint violations caused by model uncertainty, like [6]. We will include this discussion in the revised manuscript and leave it as future work to extend our results to continuous settings.

[1] Efficient Model-based Multi-agent Reinforcement Learning via Optimistic Equilibrium Computation.

[2] Safe Model-based Reinforcement Learning with Stability Guarantees.

[3] Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning.

[4] Accurate uncertainties for deep learning using calibrated regression.

[5] Calibrated Model-Based Deep Reinforcement Learning.

[6] On the Robustness of Safe Reinforcement Learning under Observational Perturbations.

Reviewer_kq7GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The work investigates theoretical guarantees on real-world sample complexity for safe sim-to-real world transfer using reward- free Reinforcement learning. Furthermore, several experiments are conducted in a gridworld setting for demonstrating the feasibility of the approach. The main contribution is provable safety guarantees for sim-to-real transfer in a reward-free setting. By enabling the reuse of collected data for arbitrary reward functions, the method claims to reduce the task-specific data collection and it can be adapted to real-world settings.

Critical strengths
  1. The theoretical guarantees and efficient computation of safe exploration for safe sim-to-real transfer are provided.
  2. The refinement of the model-plant mismatch using the hybrid models is a pratical and effective way to handle model-plant mismatch.
Critical weaknesses
  1. The theoretical work heavily relies on several assumptions about the dynamics and mode-plant mismatchs. This may lead to limited applicability of the theoretical gurantees. The discussion on assumption is not adequately address in the paper.
  2. There is no benchmarking against other safe sim-to-real transfer approaches discussed in the introduction. Table 1 only provides the comparison.
  3. The experiments are limited to grid world environments and hence, the limited empirical evaluation of the proposed method.
  4. The sample complexity related analysis does not discuss the computational cost of estimating σs\sigma_s.
Questions for authors
  1. What will be the impact of violating the assumptions on the theoretical guarantees?
  2. Is it possible to provide any example of specifying ϵs\epsilon_s and σs\sigma_s using the domain knowledge? Also, will it change depending on the type of simulators (physics-grounded or data-driven)? How can this knowledge be incorporated in Alogrithm 4 for estimating σs\sigma_s? How will the data quality impact on estimation of σs\sigma_s in the real-world applications?
  3. How does this proposed safe sim-to-real approach compare to other safety-critical transfer learning paradigms, such as those based on domain adaptation or meta-learning?
  4. Does the Horizon HH have a role in refinement of the model-plant mismatch region?
  5. The experiments do not explicitly address the computational cost of the linear uncertainty certificate maximization in Algorithm 1. How does this cost scale with the dimensionality of the state and action spaces?
5 comments

Thanks for your input and questions that help us clarify the setting and contributions.

  1. Discussion of the dynamics and model–plant mismatch assumptions (Assumption 2.4): Please note that we discuss Assumption 2.4 immediately after introducing it, explain how to estimate σs\sigma_s in Appendix B.6, and study parameter misspecification in Appendix B.7. Here, we restate and further elaborate on these discussions for your convenience.

    (i). Necessity of the assumption: If we are given only a simulator and without prior mismatch information, we cannot guarantee that learning with the simulator offers any benefit over online learning without it [1]. This is because the agent need to explore the entire state–action space to determine where the simulator can be trusted, and the cost of this exploration can eliminate any potential sample savings from the simulator. Therefore, prior mismatch information is needed for efficient sim-to-real transfer.

    Our paper studies sim-to-real transfer under one form of prior mismatch information, specified by Assumption 2.4. Assumption 2.4 has also been used in hierarchical RL (Chua et al., 2023), meta-RL (Chen et al., 2022; Mutti & Tamar, 2024; Brunskill & Li, 2013), and sim-to-real RL (Qu et al., 2025). This assumption requires knowledge of ϵs\epsilon_s and σs\sigma_s. These parameters separate sim-to-real mismatches, measured in total variation distance, into two categories: negligible mismatches of at most ϵs\epsilon_s and significant mismatches of at least σs\sigma_s. It does not require knowledge of the mismatch region or the real transition dynamics. And our algorithm identifies these regions and safely learning the real dynamics.

    (ii). Consequences of misspecifying the separation parameters: A conservative choice of these parameters, namely an underestimate of σs\sigma_s and an overestimate of ϵs\epsilon_s, reduces the separation gap σs−ϵs\sigma_s-\epsilon_s and leads to higher sample complexity for estimating the mismatch region, as shown in term (ii)(ii) of the sample-complexity bound in Theorem 4.1. The guarantees for safe exploration and accurate reward-free planning remain valid. By contrast, overestimating σs\sigma_s and underestimating ϵs\epsilon_s can cause true mismatch triples to be removed from the estimated mismatch region. Thus, the hybrid model may rely on simulator transitions that do not accurately approximate the real-world dynamics. Consequently, safe exploration and accurate reward-free planning are no longer guaranteed. Therefore, we recommend a conservative choice of these parameters.

    (iii). Availability and estimation of the parameters: Our main sample-complexity result assumes knowledge of ϵs\epsilon_s and σs\sigma_s, and Algorithm 1 does not include a parameter-estimation stage. Algorithm 4 in Appendix B.6 provides an auxiliary option for estimating σs\sigma_s when the safe baseline policy π0\pi^0 sufficiently covers the mismatch region B\mathcal{B}, namely,

    dhPreal,π0(s,a)≥qmin⁡>0,∀(h,s,a)∈B.d_h^{P^{\mathrm{real}},\pi^0}(s,a)\geq q_{\min}>0,\quad\forall(h,s,a)\in\mathcal{B}.

    We have updated the paper and added a short proof of the sample complexity for estimating σs\sigma_s in Appendix B.6. By Proposition B.1 in the revised manuscript, Algorithm 4 returns a conservative and ϵ\epsilon-accurate estimate of σs\sigma_s with high probability using O~(H∣S∣qmin⁡−1ϵ−2)\widetilde{\mathcal{O}}(H|\mathcal{S}|q_{\min}^{-1}\epsilon^{-2}) real-world samples. Thus, poorer coverage of the mismatch region, meaning a smaller value of qmin⁡q_{\min}, requires more real-world samples to estimate σs\sigma_s accurately. We also empirically validate Algorithm 4 in Appendix B.6 through case studies with different mismatch regions in the gridworld example.

    (iv). The reviewer asks whether domain knowledge and simulator type can help estimate ϵs\epsilon_s and σs\sigma_s. For example, in robotics, a simulator may be accurate in nominal operating regimes but less reliable during contact-rich interactions, such as grasping deformable objects. If a safe baseline policy explores different grasping motions on these objects, we can use it in Algorithm 4 to collect real-world data and estimate σs\sigma_s. Similarly, if a safe baseline policy explores nominal regimes known to form the negligible-mismatch region, we can adapt Algorithm 4 to estimate ϵs\epsilon_s by taking the maximum upper confidence bound on their mismatch instead of the minimum lower confidence bound used to estimate σs\sigma_s. Both procedures apply to physics-grounded and data-driven simulators.

  1. Additional experiments: Thank you for suggesting additional comparisons with the methods in Table 1, which includes meta-learning approaches. We also compare our work with the safe domain adaptation approach in [2], which uses real-world data to fine-tune simulator-trained policies and empirically improves performance while reducing safety risks. We have conducted additional experiments against methods in table 1 and [2], with the results summarized in the table below.

    AlgorithmSamples from Mreal\mathcal{M}^{\mathrm{real}}Suboptimality gapUnsafe iterations
    Safe sim-to-real RL39,168±67239{,}168 \pm 6720.003±0.0080.003 \pm 0.0080.0%
    Reward-free safe RL135,456±545135{,}456 \pm 5450.003±0.0070.003 \pm 0.0070.0%
    Unconstrained sim-to-real RL21,114±27621{,}114 \pm 2760.003±0.0090.003 \pm 0.00951.4%
    [2](12.84±10.91)×106(12.84 \pm 10.91)\times10^60.0157±0.01770.0157 \pm 0.01770.0%
    [4](112.20±140.34)×106(112.20 \pm 140.34)\times10^60.0033±0.00570.0033 \pm 0.00570.0%
    [6](5.26±3.90)×106(5.26 \pm 3.90)\times10^60.0274±0.01160.0274 \pm 0.01160.0%
    [7](29.55±55.36)×106(29.55 \pm 55.36)\times10^60.0308±0.01740.0308 \pm 0.01740.0%
    [3]9,916±289{,}916 \pm 280.0056±0.02060.0056 \pm 0.020648.7%
    [5]209,652±293,607209{,}652 \pm 293{,}6070.0301±0.05020.0301 \pm 0.050211.9%
    [8]12,680±5,65712{,}680 \pm 5{,}6570.0015±0.00090.0015 \pm 0.000919.3%
    [9]855,480±207,446855{,}480 \pm 207{,}4460.0371±0.01160.0371 \pm 0.011633.1%
    [10]30,300±18,03730{,}300 \pm 18{,}0370.0218±0.01410.0218 \pm 0.014135.0%

    Our method requires fewer real-world samples than the other methods while ensuring safe exploration and accurate planning.

  2. Role of HH: The bonus function btb^t in Algorithm 1 also scales linearly with HH, and we terminate the algorithm (line 10) when the value VbπtV_b^{\pi^t} is sufficiently small. Therefore, a larger HH can require more samples for learning. As shown in Theorem 4.1, our sample complexity is polynomial in HH, with a worst-case horizon dependence of O~(H7)\widetilde O(H^7).

  3. Computational cost: Solving line 8 in Algorithm 1 is a CMDP problem with known hybrid transition dynamics, reward btb^t, and constraint cc. It can be formulated as a linear program with H∣S∣∣A∣H|\mathcal{S}||\mathcal{A}| decision variables and solved in polynomial time. Alternatively, we can use existing CMDP algorithms, such as [11,12,13].

  4. Gridworld experiments: Gridworld provides a controlled testbed that matches our finite-state, finite-action setting. Small changes to the environment allow us to vary ϵs\epsilon_s and σs\sigma_s and study their effects, enabling direct comparisons between our theoretical predictions and empirical results.

References

[1] Leveraging (Biased) information: Multi-armed bandits with offline data.

[2] Safe Continual Domain Adaptation after Sim2Real Transfer of Reinforcement Learning Policies in Robotics.

[3] Fast Active Learning for Pure Exploration in Reinforcement Learning.

[4] Constrained meta reinforcement learning with provable test-time safety.

[5] A Simple Reward-Free Approach to Constrained Reinforcement Learning.

[6] Improved Regret Bound for Safe Reinforcement Learning via Tighter Cost Pessimism and Reward Optimism.

[7] A Safe Exploration Approach to Constrained Markov Decision Processes.

[8] Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RL.

[9] Hybrid Transfer Reinforcement Learning: Provable Sample Efficiency from Shifted-Dynamics Data.

[10] A Unified Algorithmic Framework for Hybrid Reinforcement Learning in Tabular MDPs with Shifted Transition Dynamics.

[11] Convergence and sample complexity of natural policy gradient primal-dual methods for constrained MDPs.

[12] Constrained Policy Optimization.

[13] CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guarantee.

I would like to thank the authors for providing the responses to my questions and providing additional baselines.  My questions are partially addressed. I have a couple of  follow-up questions and minor comments. 

1.  Assumption 2.4 remains physically unrealistic because real-world dynamics mismatches vary continuously rather than leaving a gap. Also, "prior mismatch information" is required as mentioned in Point 1 (i) and (iv). Then, what does it mean by  "estimating the mismatch region" in Algorithm 1?  How do we pick conservative thresholds that eliminate the sample-efficiency gains of using a simulator?

2.  In the point 4, the computational cost with the linear certificate requires solving Linear programs with ∣S∣∣A∣H|S||A|H variables at every episode. This will scale exponentially with state-action dimensionality even for sizable discrete  domain tasks. Also, will the approximation of CMDP solving approaches   violate the provable safety   and sample-complexity bounds proposed in the manuscript? 

  1. Restricting to a synthetic Gridworld constructed for demonstrating the central results does not help readers to understand the claims of sim-to-real applicability in even continuous tasks approximated to discretized settings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.