- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies safe sim-to-real transfer. The authors formulates safe-sim-to real transfer as a reward-model safe RL problem and develop computationally efficient algorithm. Their theoretical results argue that, with high probability, every policy deployed during learning is feasible, and the collected data support the computation of a feasible and near-optimal policy for any reward function.
Critical strengths
I think there are the following three strengths.
- I think that the problem setting formulated in this paper is interesting and important. This formulation is related to safety-critical domains such robotics or healthcare.
- Assuming that Assumption 2.4 holds, the theoretical results presented by the authors are useful and novel. I read through the proofs, and I did not find major issues.
- The proposed algorithm is technically reasonable.
Critical weaknesses
I also have some concerns.
- Though Assumption 2.3 (i.e., Slator's condition) is common and natural, I am not fully convinced that Assumption 2.4 is reasonable. I think that the sim-to-real transfer is difficult mainly because such state transition that satisfies Assumption 2.4 cannot be easily obtained. I feel making assumption 2.4 implicitly avoids the essential difficulty of sim-to-real RL.
- Additionally, if I understand correctly, I feel it is very difficult to learn such state transition dynamics that satisfies Assumption 2.4. This assumption would be related to out-of-distribution (OOD) in RL, I am not sure how we can learn such state transition functions.
- While I understand that this paper is theoretical, I consider that the empirical experiments in this paper is rather weak. I would like the authors to conduct some experiments in common safe RL benchmarks (e.g., safety-gymnasium). If the authors conduct some experiments in more complicated benchmark problems, I think the authors would understand my concerns regarding Assumption 2.4.
Questions for authors
This paper presents some interesting ideas with rigorous theoretical analyses. Although this paper presents a novel and interesting theoretical result, I feel it is largely based on Assumption 2.4. I am not fully confident that Assumption 2.4 is reasonable and how we can obtain such estimated state transitions that the assumption holds. Also, the empirical experiments are conducted only with toy environments and I do not feel the authors' claims are sufficiently supported.
My questions are:
-
Could you elaborate more on why the authors consider Assumption 2.4 reasonable? Does it hold in real applications?
-
How can we learn such state transition functions that satisfies Assumption 2.4 in complicated tasks (e.g., safety-gymnasium)?
3 comments
Thank you for the response. However, I remain concerned that the assumption shifts much of the difficulty of sim-to-real transfer into obtaining the required prior knowledge. In challenging tasks, it is still unclear whether acquiring this knowledge is substantially easier than solving the original problem. This is why I suggested more challenging experiments. I am not convinced whether the required prior knowledge is practically obtainable. Therefore, I will keep my original rating.
For challenging tasks like robotic grasping and manipulation, simulators may accurately predict free-space motion yet make substantial errors during interactions with objects [1]. Assumption~2.4 captures such cases where sim-to-real errors are negligible during free-space motion but substantially larger during contact-rich interactions with objects.
In the following, we provide a possible approach to estimating and in a robotic grasping task. Under a well-calibrated stochastic transition model, such as Gaussian transitions with a known noise covariance, bounds on the one-step physical prediction error $e_h(s,a) =\left|f_h^{\mathrm{sim}}(s,a)-f_h^{\mathrm{real}}(s,a)\right|_2, $ where and denote the mean next states, can be translated into TV mismatch bounds for estimating .
These physical errors can be estimated by comparing measurements from the simulator and the real world. For contact-related mismatches, [2] identifies contact parameters and evaluates discrepancies between simulated and real trajectories, while [3] quantifies discrepancies in post-impact velocities. In contrast, small discrepancies have been observed during free-space motion: [4] reports an average joint-position discrepancy of approximately rad between simulated and real UR5e trajectories during free-space motion.
[1] Focused Adaptation of Dynamics Models for Deformable Object Manipulation.
[2] Validating Robotics Simulators on Real-World Impacts.
[3] Impact-Aware Robotic Manipulation: Quantifying the Sim-To-Real Gap for Velocity Jumps.
[4] Experimental Evaluation of Collision Avoidance Techniques for Collaborative Robots.
Thanks for your input and questions that help us clarify the setting and contributions.
Choice of Assumption 2.4: If we are given only a simulator and without prior mismatch information, we cannot guarantee that learning with the simulator offers any benefit over online learning without it [1]. This is because the agent need to explore the entire state–action space to determine where the simulator can be trusted, and the cost of this exploration can eliminate any potential sample savings from the simulator. Therefore, a central challenge in efficient sim-to-real transfer is how to leverage prior mismatch information.
Our paper studies efficient sim-to-real transfer under one form of prior mismatch information, specified by Assumption 2.4. Assumption 2.4 has also been used in hierarchical RL (Chua et al., 2023), meta-RL (Chen et al., 2022; Mutti & Tamar, 2024; Brunskill & Li, 2013), and sim-to-real RL (Qu et al., 2025). This assumption requires knowledge of ϵs and σs. These parameters separate sim-to-real mismatches, measured in total variation distance, into two categories: negligible mismatches of at most ϵs and significant mismatches of at least σs. It does not require knowledge of the mismatch region or the real transition dynamics. And our algorithm identifies these regions and safely learning the real dynamics.
We would like to point out that the simulator is given, and we do not learn a simulator whose transitions must satisfy Assumption 2.4. Without knowledge of ϵs and σs, we can set ϵs=σs=0. This recovers the fully online version discussed after Theorem 4.1, where our algorithm improves upon the sample-complexity bound of Huang et al. (2023). In such case, Assumption 2.4 then holds for any simulator, since the total-variation distance always lies in [0,1]. With σs=0, the mismatch region includes all time–state–action triples, and the algorithm learns real-world transitions without simulator reuse.
Practicability of Assumption 2.4: The assumption captures situations where a simulator models some parts of the real-world dynamics accurately but others poorly. For example, in robotics, a simulator may be accurate in nominal operating regimes but less reliable during contact-rich interactions, such as grasping deformable objects. Here, ϵs provides an upper bound on the small differences between simulated and real transition probabilities in nominal regimes, whereas σs provides a lower bound on the larger differences during these grasping interactions. Given ϵs and σs, our algorithm uses (ϵs+σs)/2 as a threshold to distinguish between negligible and significant mismatches and estimate the mismatch region. Based on this estimate, we construct a hybrid model that combines simulated and learned dynamics and design a mixture policy to safely explore the mismatch region and collect data ti refine hybrid model for accurate reward-free planning.
Furthermore, we provide Algorithm 4 in Appendix B.6 for estimating σs. If a safe baseline policy explores different grasping motions on these objects, Algorithm 4 can use this policy to collect real-world data and estimate σs. Similarly, if a safe baseline policy explores nominal regimes known to form the negligible-mismatch region, we can adapt Algorithm 4 to estimate ϵs by taking the maximum upper confidence bound on their mismatch instead of the minimum lower confidence bound used to estimate σs.
Gridworld and continuous-control benchmarks: Gridworld provides a controlled testbed that matches our finite-state, finite-action setting. Small changes to the environment allow us to vary ϵs and σs and study their effects, enabling direct comparisons between our theoretical predictions and empirical results. Safety-Gymnasium focuses on continuous-control tasks, which fall outside the setting analyzed in this paper. We leave it as future work to extend our results to the continuous setting.
[1] Leveraging (Biased) information: Multi-armed bandits with offline data.