acceptodds
Under review as a conference paper at ICLR 2027

Adapting Learned Robot Simulators: Policy Visitation Beats Error Magnitude

Abstract

Policies trained inside learned robot simulators exploit the simulators' errors: on the public NeRD model of the Ant robot, jump policies score higher in the model than in the physics engine it imitates. The prevailing remedy is to adapt the simulator with data collected where the model errs most, but in learned robot simulators this has not been tested against collecting where the policy goes. We provide that test with two tools: a controlled testbed in which seven acquisition rules, including an adversary given the model's true error, adapt the released NeRD models under identical data budgets and fine-tuning; and a paired-perturbation diagnostic that checks whether a model reproduces the engine's contact-event changes. The adversary finds errors – larger than random actions, yet yields policies no better than random data, whereas collecting along the trajectories of model-trained policies recovers – of the return of a policy trained in the engine on the jump task. Fine-tuning that cuts one-step error fivefold leaves most contact-event changes unreproduced, and the rule with the lowest boundary error trains worse policies than policy-trajectory collection. The value of adaptation data thus depends on whether the policy will visit its states, not on how wrong the model is there, and prediction accuracy, contact fidelity, and policy performance must be evaluated separately.

Primary area:
applications to robotics, autonomy, planning
Keywords:
learned simulators, robot learning, model-based reinforcement learning, simulator adaptation, contact dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wxuAOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how data-selection strategies affect the fine-tuning of learned robot simulators. It argues that, during policy training, a state’s policy visitation frequency is more informative than the magnitude of the model error at that state for assessing how improving the state will affect policy optimization outcomes. The paper’s main contributions can be broadly divided into two aspects. First, it validates the central claim that, when fine-tuning a model for policy optimization, policy visitation frequency is more important than model error magnitude. It also shows that prediction accuracy, contact fidelity, and policy performance are relatively independent simulator metrics. Second, it designs and implements a controlled testbed and a contact-event diagnostic tool for evaluating robot simulators, providing reusable evaluation tools for the fine-tuning process.Using the same NeRD model, the same number of state-transition samples collected from the same physics engine, and the same fine-tuning procedure, the paper quantitatively compares seven distinct complete data-acquisition pipelines across multiple Ant robot tasks. It evaluates policy returns and prediction errors in both the learned simulator and the ground-truth environment, providing empirical support for its conclusions.

Critical strengths
  1. The overall experimental design is sufficiently rigorous. Except for the data-selection strategy, the experimental procedures are kept consistent to control for irrelevant variables. The evaluation metrics cover multiple dimensions, including policy return, prediction error, and contact fidelity, and statistical analyses are used to reduce the influence of experimental variability. The experiments compare seven different data-selection strategies across multiple tasks on the Ant robot and on ANYmal, while also reporting metrics from the ground-truth environment as references. This rigorously designed experimental framework can provide useful guidance for evaluating other learned simulators.
  2. The paper’s discussion and validation of the basic design principles for data selection during fine-tuning can provide guidance for future fine-tuning of learned simulators. The paper conducts a detailed comparison between policy-visitation-based and model-error-based data-selection strategies, and uses statistical results to demonstrate which type of data selection can better support the policy optimization process.
Critical weaknesses
  1. The experiments use only the Ant and ANYmal robots, both of which are legged robots. They do not include other robot morphologies, such as wheeled or aerial robots, nor do they further validate the conclusions on real robots. Therefore, it is unclear whether the conclusions generalize to real-world settings or other types of robots. In addition, the paper evaluates only the combination of the NeRD model and PPO, making it likewise difficult to determine whether the conclusions generalize to other models and policy-optimization procedures. Overall, the empirical evidence is limited in breadth, making it difficult to establish the broad applicability of the claims. The implications for real-robot deployment also remain insufficiently clear.
  2. All of the algorithms evaluated in this paper are existing methods, and the idea of prioritizing policy visitation is also not novel. Apart from comparing policy-visitation-based methods with model-error-based methods, the paper’s apparent innovation seems limited to improving the contact-diagnosis method. This raises concerns about whether the paper provides sufficient novelty.
Questions for authors
  1. Table1 shows that each data-acquisition procedure consists of three components: starting states, collection actions, and selection criterion. Is the claim that policy visitation is superior to model error intended to apply to the data-acquisition procedure as a whole, or only to the selection criterion component?
2 comments

Thank you for recognizing the controlled design. Following your comments, we:

  • W1.1: added wheeled tracking and manipulation;
  • W1.2: added cost-aware simulated transfer and clarified physical-robot evaluation as future work;
  • W1.3: added GRU-SAC and residual ensemble-SAC;
  • W1.4: stated conclusions for the tested configurations;
  • W2: connected paired contact evidence to controlled acquisition–reuse comparisons;
  • Q1: separated procedure, component, and weighting comparisons.

W1.1–W1.4 address the distinct concerns within your first weakness.

[W1.1 & W1.3 & W1.4] Evaluation breadth and scope

We compare seven procedures on Ant, ANYmal, Jackal, and Franka Wipe, with three independently trained models per task. The implementations cover NeRD-PPO, GRU-SAC, and residual ensemble-SAC. Figure 2 includes all planned policies, averaged within models. VS-U has the highest mean on Jackal; PR leads on Wipe at 71.98, versus 36.24 for EA and 29.62 for VS-C. We report conclusions for the tested configurations and identify crossed evaluation as future work (§5).

[W1.2] Physical deployment and simulated transfer

Physical-robot evaluation is future work. The added transfer study uses simulated source and target environments, replacing nominal PD dynamics with a learned ANYdrive actuator. Each condition receives 100,000 cost units: one per selected transition and up to 600 per actual termination for recovery.

PR exceeds BASE and EA-online in target mean return on all three models. Both contrasts exceed the larger within-model two-policy return gap in 2/3 models; the source-only contrast does so in 1/3, while contrasts with random-action collection vary in sign. This is a descriptive training-variation criterion. Two EA-online collectors acquire no further transitions after their first update. We also specify reset-access and error-feedback requirements (§4.3).

[W2] Paired contact evidence and contribution

On Wipe, GT and EA each respond to 19/1,096 probes, but share only one responding probe. Paired measurements therefore distinguish response locations even when aggregate counts match.

The protocol applies opposite velocity perturbations with identical actions over ten controls, comparing contact-vector sequences after removing consecutive duplicates. Shared responses indicate activation in both simulators; agreement of their sequences is separate.

Independent GT replays reproduce 16 responding and 24 nonresponding probes, using one initial model per task. Scale, horizon, and contact-criterion checks characterize sensitivity. Figure 3 reports prediction error; Table 3 reports shared/extra responses (§4.2). We connect these diagnostics to controlled acquisition-reuse comparisons, building on policy-relevant learning and contact-aware modeling (§2).

[Q1] Procedures versus selection criteria

EA-versus-visitation compares complete procedures; component controls vary starts, actions, and selection separately. Within visited candidates, VS-C minus VS-U estimates joint contact-error weighting: +0.2 SD return, 95% CI [-0.5, +0.9], Holm p=1. This estimates the combined rule, not an isolated contact benefit. Selected-distribution analyses retain all 12 models and are exploratory (§4.3).

Thank you for your detailed response. The Wipe example illustrates the paired-measurement procedure, but I am more interested in how this diagnostic differs from existing diagnostic methods and what is distinctive about its design. Could the authors explain why the paper’s contribution is sufficiently important, given that the evaluated algorithms and the idea of prioritizing policy visitation already exist? The authors have clarified that the paper’s main comparison concerns complete data-acquisition procedures rather than the selection criterion alone. I would appreciate further clarification on the extent to which the observed performance differences can be attributed to an emphasis on policy-visited states, rather than to other differences between the procedures.

Reviewer_21UXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper

Critical strengths

The observed problem is correct and important, that a policy trained in a learned simulator learns to exploit simulator errors, and policies trained in it usually work worse in the real robot / simulator. And use policy visitation as a criteria to augment data is a solution to this problem. (yet, this idea is not new, and has been explored a lot from different aspects)

Critical weaknesses

The paper does not propose a new method. Its aim is to compare two common strategies to collect data for the adaptation of a learned simulator: collection on area where the model error is large, and collection on where the policy really visits. For a comparison paper, the range of the experiments is the main contribution. But the main text gives the core experiments for only one robot (Ant) and two tasks (run and jump). The ANYmal results are only descriptive as mentioned by the authors themselves. And even these two tasks don't have consistent results. The main result (0.81–0.88) comes from one task, Ant jump. On Ant run, the gain of VS-C over EA is approximately half (+0.37, against +0.74 on jump). Thus, the experiments do not support the general claim that “policy visitation matters more than error magnitude.”

Questions for authors
  1. Generalization to other robots and tasks. The main comparison result (Fig.3 and Table 2) uses only Ant run and Ant jump. How does the claim generalize to other more complicated robots and tasks? If the results do not hold, which part of it stays true?
  2. Comparison with visitation-weighted baselines. Some methods use policy-visitation to weight the data, how do they compare to collecting new data if given the same data budget?
  3. Task information vs. visitation. PR and VS rules use policy that are trained for the task, so they use the task information through the reward. EA, RA, BS do not use the task. So how does the comparison separate between the effect of policy visitation and the task information?
1 comment

Thank you for the helpful suggestions. Following your comments, we:

  • W1: clarified the empirical contribution beyond comparing established strategies;
  • W2: expanded the main comparison beyond Ant;
  • W3: reported task-specific gains alongside initial performance;
  • W4: revised the claims to reflect task-dependent outcomes;
  • Q1: expanded the task and model coverage;
  • Q2: compared new collection with weighted replay and mixtures;
  • Q3: added controls for starts, actions, and selection.

[W1] Empirical contribution

Our contribution is the controlled comparison of acquisition and reuse, their task-dependent outcomes, and complementary policy, prediction, and paired contact measurements on the same adapted models (§§2, 4).

[W2 & Q1] Evaluation across tasks

We compare seven procedures on Ant Run, ANYmal Forward, Jackal tracking, and Franka Wipe, using three independently trained models per task, 100,000 retained transitions, and 2,000 adaptation updates. Policy repetitions are averaged within each model (Table 1; Figure 2).

All four visitation procedures exceed EA in mean return on the first three tasks. On Wipe, PR, VS-U, and VS-V exceed EA, while VS-C falls below it. The highest means belong to VS-C, VS-V, VS-U, and PR, respectively.

[W3 & W4] Task differences and scope of claims

The separate released-model Run/Jump gains for VS-C over EA are +0.37/+0.74 reference return, from initial returns of 0.70/0.15. The revised claims focus on task-dependent acquisition outcomes and direct collection–reuse comparisons (§4.1).

[Q2] Collection versus weighted replay

The reuse study compares new policy data, visitation-weighted replay, and equal old–new mixtures, using eight initial models and five policies per model/condition under matched retained-data and update budgets.

New policy data minus visitation-weighted replay is +0.024/-0.007 for rich/poor replay, with model-paired 97.5% CIs [-0.806, +0.854]/[-0.661, +0.647]. These are close point estimates, not an equivalence result. Returns are divided by 500. Equal mixtures have the highest observed means, 1.594/1.571; their mean advantage over weighted replay remains positive when any one model is omitted.

Collection and visitation-query costs are reported separately from the matched budget. Replay policies use 4M training steps; the four-task comparison uses 12M for Ant, ANYmal, and Jackal, and 1M for Wipe (Table 1; §4.3).

[Q3] Task information and components

The component controls vary reset/visited starts, random/policy actions, and uniform/error-ranked selection. Visited-start contrasts are +164.3 and -4.4 return units in the separate three- and five-model cohorts. Across two fixed evaluation banks, policy differences exceed bank differences in 68/72 cells.

PR and the visited-state rules receive task information through task-trained policies. We distinguish whole-procedure comparisons from these component controls and selection within a fixed candidate pool (§4.3).

Reviewer_De4SOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This work centers around controlled experiments to show that across many data acquisition methods, the utilization of policy visitation outperforms model error-based methods when adapting learned simulators. The authors show this by deploying various data acquisition methods across four tasks in the pretrained NeRD simulator and using the PPO RL algorithm. The authors make these comparisons using matched data budgets, fine-tuning schedules, and identical pre-trained simulator checkpoints. They find that policy visitation strategies outperform methods that seek out state-action pairs of maximum model error in enabling downstream policy performance using RL. Furthermore, they introduce a contact-event diagnostic and show that even with improvements in one-step model prediction accuracy during contact events, fidelity across the contact event is not necessarily improved, nor is downstream policy performance.

Critical strengths

Overall I really appreciate the clear motivation in the introduction into why a reader should care about this paper. It comes early and is quite clear in establishing the need to study how to adapt learned simulators. The discrepancy in policy performance for learned models versus those trained in the underlying physics engine is a highlight alongside the effective description of why we care more about where the policy wants to go being useful for data acquisition as opposed to simply relying on sampling data that has high model errors. This naturally leads into the rest of the paper and the motivation behind the many acquisition strategies used for comparisons later.

The authors do a great job of developing controlled experiments and a methodology to best isolate the benefits of one data acquisition strategy over the others. These include matching data budgets, identical pretrained simulator checkpoints, and matching fine-tuning schedules. These combined components allow for an effective targeted empirical study. I also appreciated their explanation and inclusion of an oracle model-error based adversary as a strong comparative baseline and the contact event diagnostics later in the results.

Critical weaknesses

In the title, section titles, and contributions, as well as throughout the paper, the authors make broad claims about the generality and relative importance of policy visitation as a data acquisition policy over choosing data with max model prediction error when doing learned simulator updates. There is a certain amount of qualification at times, but it is far exceeded by the stronger headlines. The evidence from the experiments on the other hand is quite limited. The authors use only the NeRD learned simulator, two robots, and PPO as the only policy optimization algorithm. The authors evaluate on a total of four tasks, two ANYmal tasks and two Ant tasks, but restrict their formal analysis to the Ant tasks. Since the effectiveness of data acquisition is closely tied to the underlying MDP that the simulator describes I think that making broad claims about policy visitation beating model error based on only four tasks total and a single RL algorithm is insufficient.

The paper has quite limited conceptual novelty overall. The central claim of the paper is that policy visitation as a data acquisition policy is much more useful than model error when deciding what data to ingest for model adaptation. Having higher accuracy along policy relevant state-action pairs is already a well established concept in model-based RL and the authors cite some of these works (e.g. Ross and Bagnell). While I very much appreciate the controlled experimental design of this paper, I don't think the authors sufficiently distinguish their contributions from what is already known in the model-based RL field. As a result, especially when combined with the limited evaluation protocol discussed in weakness 1 (1 RL algorithm, 2 tasks used for formal analysis), the general finding of this paper and the headline title is stronger than the novelty of this work warrants.

Questions for authors

Since there is substantial prior work (e.g. Ross and Bagnell) on policy relevant model data acquisition and the importance of model accuracy along the policy visitation distribution, what would the authors say the contribution of this work is in terms of novelty? The controlled experiments for comparing data acquisition strategies?

As highlighted in the weaknesses, there seem to be broad claims being made from very narrow experimental tasks and a single RL algorithm. What evidence is there to make broader and more general claims about the utility of policy aware data acquisition when compared to model error-based methods?

1 comment

Thank you for recognizing the clear motivation and controlled experimental design. Following your comments, we:

  • W1: aligned the claims with the evidence from the tested configurations;
  • W2: expanded the robot, dynamics-model, and policy-learning coverage;
  • W3: reported task-dependent outcomes and between-model variation;
  • W4: clarified the empirical contribution relative to established policy-relevant model learning;
  • W5: revised the title, abstract, and conclusions accordingly;
  • Q1: added direct acquisition–reuse comparisons and component controls;
  • Q2: presented the broader comparison in the main text.

[W1 & W3 & W5] Claim scope and task dependence

PR, VS-U, VS-C, and VS-V all exceed EA in mean return on the locomotion and tracking tasks. On Wipe, PR, VS-U, and VS-V exceed EA, while VS-C falls below it. The highest-mean procedure changes across tasks. These mean rankings, together with the between-model variation, characterize the tested robot–model–learner combinations. A fully crossed design is a direction for extending the comparison (§5).

The revised title, “Adapting Learned Robot Simulators: Comparing Data Acquisition and Reuse,” reflects this contribution. The abstract, introduction, and conclusion now center on the observed acquisition outcomes and the direct collection–reuse comparison. Section 5 outlines extensions through crossed designs and physical-robot evaluation.

[W2 & Q2] Evaluation across tasks and model families

The complete seven-procedure comparison now covers Ant running, ANYmal walking, Jackal tracking, and Franka Wipe, using NeRD–PPO, GRU–SAC, and residual ensemble–SAC. Each task has three independently trained initial models. Within each model, procedures share the initial checkpoint, retain 100,000 additional transitions, and receive 2,000 adaptation updates. Figure 2 reports GT policy returns and between-model variation, averaging policy repetitions within models.

Figure 2 includes all initially planned policy evaluations. Exploratory repeats and earlier partial comparisons are reported separately (§4.1).

[W4 & Q1] Empirical contribution

Our contribution is a controlled empirical comparison of acquisition and reuse choices, building on established policy-relevant model learning. The added fresh-data-versus-replay experiment yields close mean returns under matched retained-data and update budgets, with uncertainty quantified by model-paired intervals. Separate controls examine starting states, actions, and selection, while the four-task comparison identifies task-dependent procedure rankings.

The same-model measurements add a further distinction: Jackal BS* achieves the lowest mean prediction error, while visitation procedures yield higher mean policy return. Paired contact probes identify shared and extra responses on those adapted models. The contribution is therefore the evidence connecting acquisition choices to these different outcomes (§§2, 4.1–4.3).

Section 2 clarifies the comparison with prior work. Our EA is an oracle error-seeking control using exact normalized one-step L2 error. PASL uses a learned Wasserstein critic with a task-return constraint; GALILEO performs offline adversarial reweighting; PROWL combines behavior constraints with prediction error, action fidelity, and learning progress. These objectives and information sources distinguish EA from those methods.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their careful comments and for recognizing the clear motivation (De4S), the importance of simulator exploitation (21UX), and the controlled design and complementary evaluations (De4S, wxuA).

The revised manuscript marks changes in blue. The added experiments show how acquisition outcomes depend on the task and quantify fresh collection against visitation-weighted reuse. Prediction, paired contact responses, and policy return reveal different properties of the same adapted models. Together, these comparisons provide a broader empirical basis for choosing adaptation data and assessing what the resulting model improves.

W# denotes distinct weakness points in the order raised; Q# follows the questions in the original review. For wxuA, W1.1–W1.4 retain the first original weakness while separating its concerns; W2 retains the original novelty concern.

De4S

  • Evaluation scope (W2, Q2): Seven procedures are now compared across four robot tasks, three dynamics families, and two policy-learning algorithms. → Section 4.1, Figure 2.
  • Contribution (W4, Q1): Direct acquisition–reuse comparisons and component controls make the empirical contribution explicit. → Sections 1, 2, and 4.3.
  • Claim scope (W1, W3, W5): Task-dependent outcomes guide the revised title and conclusions for the tested configurations. → Sections 1, 4.1, and 5.

21UX

  • Task-dependent outcomes (W2, W3, W4, Q1): The main text presents the complete four-task comparison and the task-specific effects. → Section 4.1, Figure 2.
  • Replay comparison (W1, Q2): We added matched-budget comparisons of fresh data, visitation-weighted replay, and mixtures, with model-paired estimates. → Section 4.3.
  • Task information (Q3): Component controls separate starting states, collection actions, and selection within the acquisition procedure. → Section 4.3.

wxuA

  • Broader evaluation and deployment (W1.1–W1.4): Wheeled locomotion, manipulation, additional dynamics models, and transfer under a collection-and-recovery budget extend the evaluation. → Sections 4.1 and 4.3.
  • Contact analysis (W2): Paired probes distinguish responding states even when aggregate response counts match, adding information alongside prediction and policy measurements. → Section 4.2, Table 3.
  • Procedure and selection (Q1): We report whole-procedure outcomes, within-candidate weighting, and component contrasts separately. → Section 4.3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.