acceptodds
Under review as a conference paper at ICLR 2027

Can Safety Transfer Across Embodiments? Corrective Supervision for Offline Safe Reinforcement Learning

Abstract

Offline safe reinforcement learning relies on safety-relevant experience from a fixed dataset, which may be limited for a particular embodiment. Other embodiments may have demonstrated useful corrective behavior, but differences in actions and dynamics make this experience difficult to reuse. Can their observed motion provide safety supervision beyond what the target can learn from its own data? We investigate this question through cross-embodiment corrective supervision (CECS), an offline approach that transfers observed source motion into target-native training signals. The method retrieves safety-relevant source motions in locally matched task contexts, translates them into target-native actions via target inverse dynamics, filters the resulting corrections with target-side safety information, and uses them to refine the target policy. This formulation transfers corrective behavior rather than directly sharing actions or policies across embodiments. Experiments across different morphologies and task families show improved safety over pretrained target policies and competitive safety–reward trade-offs against offline safe RL baselines. Controlled comparisons at shared contexts and equal correction counts show that source supervision improves safety beyond equally trained target-only retrieval. Shuffling the corrective labels substantially degrades performance, highlighting the importance of matching corrections to the context. A matched-state analysis further shows that the refined policy reduces accumulated cost through sustained control while retaining most task progress. These results demonstrate the value of cross-embodiment motion as corrective supervision and identify context matching as an important factor in its effective use.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Offline Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ubWUOpenReview ↗
Rating
2
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether safe behavior from one embodiment can improve another embodiment’s policy in offline safe reinforcement learning. It proposes Cross-Embodiment Corrective Supervision (CECS), which retrieves context-matched safe source motion, translates it into target-native actions through a target-specific inverse dynamics model, filters corrections using local data support and a target cost critic, and uses the accepted labels to refine the target policy. Experiments across four simulated morphologies and two task families demonstrate improved safety and competitive safety–reward trade-offs against offline safe RL baselines in pairwise and multi-source settings. Matched-training controls show benefits beyond target-only retrieval at identical contexts and correction counts, while shuffled-label comparisons establish the importance of context matching. Matched-state rollouts further show sustained cost reductions while retaining most task progress. The validation is limited to simulation with shared task-defined motion interfaces and provides empirical safety improvements rather than formal guarantees.

Critical strengths
  • The paper investigates an insightful question: can embodiments facing similar safety risks benefit from one another’s corrective behavior despite differences in morphology and dynamics? This question is consequential for offline safe RL, where a target embodiment may have limited examples of useful safety corrections in its own dataset.
  • The evaluation is extensive within the chosen simulated benchmarks. It covers four morphologies, two task families, pairwise and multi-source transfer, and comparisons with offline safe RL baselines. The matched-training controls, shuffled-label comparisons, and matched-state rollouts are particularly useful for assessing whether source corrections provide value beyond additional target-only training.
Critical weaknesses
  • The methodological advance relative to the complexity of the pipeline needs clearer justification. At a high level, the method retrieves source motion, translates it into target-native actions, and uses those actions as corrective supervision. The predictive variant additionally trains a shared encoder with prediction, reconstruction, and alignment objectives; the target-side pipeline also requires inverse-dynamics models before policy refinement. While the experiments support the usefulness of source corrections, the paper should explain more clearly what constitutes the technical advance in this combination and whether the added preparation stages are justified by their benefits relative to the simpler explicit-retrieval variant.
  • Sensitivity to the loss weights is insufficiently evaluated. The predictive-retrieval objective in Eq. (3) combines future prediction, reconstruction, and alignment, while Eq. (9) combines native supervision, confidence-weighted corrective supervision, and entropy regularization. The paper reports settings and normalization procedures, but I could not find an empirical sensitivity analysis for the reconstruction and alignment weights or the relative strength of corrective supervision. Such an analysis would help assess robustness across source–target pairs and the manual tuning required to apply the method to a new setting.

Minor Comments

  • The AI Use Statement describes assistance with drafting and editing the manuscript. If generative AI was also used to create or edit Figure 1, please clarify that use in the disclosure.
  • Figure 1 is embedded as a raster image rather than a vector graphic. Please export the diagram in vector format to keep its text, lines, and arrows sharp when zoomed or printed.
Questions for authors
  • What is the computational overhead of each preparation and training stage: predictive-encoder training, the five-fold target inverse-dynamics models, target cost-critic training, retrieval and correction construction, and policy pretraining and refinement? Please report hardware, elapsed time, and whether each component must be retrained for a new source–target pair, and distinguish the costs of explicit and predictive retrieval.
  • [Bonus] The experiments cover distinct morphologies, but all transfer occurs within Run or Circle using shared planar motion coordinates. How far can the method generalize to robotic arms with different kinematic structures, action dimensions, and feasible workspaces? In particular, what shared motion interface and target-data coverage would be needed for the inverse model to translate source corrections reliably, and when should the method abstain because the source motion is infeasible for the target?
Reviewer_o1FpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether observed motion from one embodiment can supply safety supervision for an offline safe RL policy on another embodiment, whose actions and dynamics differ. It proposes cross-embodiment corrective supervision (CECS). At target contexts that later violate the cost budget, CECS retrieves source continuations from locally supported nearest neighbours in a shared task frame. A target inverse-dynamics model translates each continuation into a target-native action. A target cost critic then keeps a correction only when the proposed action is predicted to stay within the remaining budget allowance while the logged action is not. Accepted corrections refine a pretrained B2R policy through a confidence-weighted likelihood term. The contribution of the paper is a new framework, called CECS, implementing pairwise and multi-source transfer across multiple morphologies (Car, Ball, Ant, Drone in its numerical studies) and multiple task families (Run, Circle in its numerical studies.

The paper tried to validate its claim using Bullet-Safety-Gym with DSRL datasets, in which CECS keeps mean normalized cost within budget on all reported targets and retains more reward than FISOR. Matched-training controls show gains beyond target-only retrieval and show that pairing each correction with its context matters. Gate ablations, an independent predictor audit, policy-adoption measures, and matched-state replay suggest the policy adopts the corrections and reduces boundary drift while preserving most task progress.

Critical strengths
  1. The paper introduces a compelling perspective on data scarcity in offline safe RL by treating cross-embodiment motion as corrective supervision. Translating observed source behaviors into target-native actions via inverse dynamics and screening them through a target-side risk critic avoids complex joint-policy learning and requires zero retrieval or source overhead at deployment.

  2. The empirical validation is exceptionally thorough. By benchmarking against target-only self-retrieval under identical contexts and counts, introducing shuffled-action controls, and conducting matched-state replays, the authors convincingly isolate the specific value of context-matched source corrections and rule out simple data-augmentation artifacts.

Critical weaknesses
  1. The empirical validation is limited to two simulated benchmark domains (Run and Circle) using manually designed low-dimensional task-space coordinates. Furthermore, the method struggles when morphologies diverge substantially: AntCircle yielded virtually zero viable corrections (0.08% retention rate) and had to be excluded from pair transfer, highlighting that task-space motion similarity alone does not reliably capture dynamic feasibility or guarantee cross-embodiment transferability.

  2. Candidate actions are vetted using a target cost critic trained offline via IQL on target datasets. While the independent audit in Section 5.3 evaluates candidate agreement, the paper provides limited analysis of whether the critic remains reliable when evaluating counterfactual pseudo-actions a~tT\tilde{a}_t^T that fall near or beyond the support of the target's offline data distribution.

  3. In several settings, the method reduces cost primarily by inducing overly conservative behavior at the expense of task reward (e.g., in multi-source BallRun, reward drops near zero or becomes negative, and on BallCircle, baselines like B2R achieve higher reward while still satisfying the cost budget). Additionally, the authors do not evaluate the sensitivity of policy performance to the specific formulation of the reference budget allowance btγb_t^\gamma, leaving it unclear under what conditions the filtering balances progress rather than over-penalizing risk.

  4. The pipeline terminology can be difficult to track across sections. Overlapping terms are used for spatial trajectories (continuation, command, requested motion) and generated actions (proposal, pseudo-action, corrective label). Defining a unified nomenclature in Section 3 and Figure 1 would improve clarity.

Questions for authors
  1. The local support gate relies on task-space kinematic similarity to retrieve source continuations. How well does this kinematic distance correlate with dynamic feasibility and downstream policy improvement? Under what formal criteria or diagnostics can practitioners determine whether a source embodiment’s motion repertoire is transferable before running the full pipeline?

  2. The inverse dynamics model and the target cost critic are both trained on offline target data. When retrieved source continuations fall near or outside the target’s support, how vulnerable is the framework to compounding translation errors or critic overestimation? Does the framework incorporate any conservative regularization or epistemic uncertainty checks on the proposed pseudo-actions?

  3. The current evaluation fixes the safety budget at B=10B=10. How do the safety–reward trade-offs of CECS compare against baselines across a spectrum of budget thresholds (from strict zero-tolerance to relaxed constraints)? Does the reference allowance btγb_t^\gamma tend to induce overly conservative behavior when the remaining budget is tight?

  4. The task-space retrieval query uses a fixed 200 ms observation window across all four morphologies. Given the significant disparities in inertia, control frequencies, and response latencies among systems (e.g., Car vs. Drone vs. Ant), how sensitive is retrieval fidelity to this window length, and does a uniform duration adequately capture equivalent dynamic contexts?

  5. In settings like DroneCircle →\to BallCircle, only ~1.2% of candidate contexts survive support gating and critic screening (624 out of 50,947). What is the computational and memory footprint of the nearest-neighbor retrieval, inverse translation, and critic inference steps during offline labeling? How well does this pipeline scale to larger offline multi-source datasets?

  6. The formulation assumes that the source and target share identical task frames and cost definitions. To what extent can the framework be adapted to settings where embodiments operate under different physical constraints or asymmetric safety budgets (e.g., obstacle collision vs. tilt limits)?

Reviewer_FzJsOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

To what extend safety-related behavior can be transferred between different embodiments in offline reinforcement learning is investigated. The presented approach retrieves motions from a source embodiment in similar task contexts, converts the retrieved motion into actions of the target embodiment using a learned inverse dynamics model, and uses a learned target-side cost critic to decide whether the resulting action should be retained as corrective supervision. The accepted actions are utilized to refine an offline target policy. Numerical experiments investigate pairwise as well as multi-source transfer against several offline safe-RL baselines. The results show that transferring motions from other embodiments can reduce empirical costs in several of the considered settings.

Critical strengths
  • Sharing data between different embodiments is a relevant research problem.

  • Uisng a common task representation is a reasonable approach.

  • The paper provides several ablations and controlled experiments. In particular, the comparison between target-only retrieval and cross-embodiment retrieval at matched target contexts is useful for isolating whether the source data add information beyond additional target-side training.

  • The substantial deterioration when corrective actions are assigned to different contexts indicates that the correspondence between context and retrieved motion matters.

Critical weaknesses
  • It is unclear how "safety" is defined throughout the paper. Pure offline reinforcement learning cannot by itself provide a formal guarantee that unsafe behavior will not occur after deployment. The proposed cost critic is also learned from data and therefore cannot serve as a formally correct safety mechanism. Indeed, the authors acknowledge in the conclusion that the critic provides empirical screening rather than formal safety guarantees. This should be addressed earlier in the abstract and the introduction.

  • In a safety-critical system requiring certification, one typically requires a formally correct safety layer to prevent unsafe actions, e.g., through verified shielding, reachability-based safety filtering, or similar methods. Once such a layer is present, however, the role of CECS likely changes. Cross-embodiment transfer does not need to guarantee safety itself. It can instead learn actions that preserve performance while reducing how often the safety layer has to intervene. This should be discussed explicitly.

  • The conceptual novelty of cross-embodiment safety transfer compared with existing cross-embodiment transfer is unclear to me. The related-work section already discusses learning from observation, cross-domain correspondence, cross-embodiment skill representations, and offline cross-embodiment RL. At the technical level, CECS retrieves motion from another embodiment, represents the motion in a shared task space, and translates it into a target action. It is not sufficiently clear why transferring a "safety-relevant" motion poses a fundamentally different transfer problem from transferring a normal task-relevant motion. Which component is necessary that is not required in previous work?

  • Furthermore, a precise definition of safety-relevant motions is missing.

  • The experiments show empirical improvements in the considered environments, but they do not resolve the above conceptual issues. The paper demonstrates that certain source trajectories can be useful for reducing target cost. However, I am not yet convinced that the results establish a distinct problem of transferring "safety" rather than transferring useful behavior under a safety-related training objective.

Questions for authors
  1. What exactly do the authors mean by "safety-relevant motion"? Could this be defined mathematically? In particular, is safety relevance a property of a trajectory segment itself, its initial state, its future cost, or its relationship to a particular safety constraint?

  2. How is cross-embodiment transfer of safety-relevant motion technically different from existing cross-embodiment transfer of ordinary task-relevant motion? Which part of CECS would not be required if the objective were simply to transfer high-performance motions between embodiments?

  3. What is actually transferred: safety or performance-improving behavior? The formal safety assessment remains target-specific because the translated action is screened by a target-side cost critic trained entirely on target data.

  4. How should CECS be used in a safety-critical physical system? Since the learned policy and critic do not provide formal guarantees, I would expect an independent safety layer at deployment. Do the authors envisage CECS being combined with such a mechanism?

  5. If a formally correct safety layer is used, shouldn't CECS primarily optimize performance rather than safety? For example, one could evaluate whether transferred corrective behavior reduces safety-layer interventions, reduces conservatism, or maintains higher task performance near safety boundaries.

  6. Can the authors clarify the distinction between satisfying the expected-cost constraint and guaranteeing safety? Equation (1) constrains expected episodic cost, while the feasible rate measures the probability of satisfying an episodic budget.

  7. Could the main idea be evaluated against a generic cross-embodiment transfer approach followed by exactly the same target-side risk filtering? Such a baseline would help determine whether CECS introduces something specific to safety transfer or whether its benefit mainly comes from combining existing cross-embodiment motion transfer with a cost critic.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.