- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how data-selection strategies affect the fine-tuning of learned robot simulators. It argues that, during policy training, a state’s policy visitation frequency is more informative than the magnitude of the model error at that state for assessing how improving the state will affect policy optimization outcomes. The paper’s main contributions can be broadly divided into two aspects. First, it validates the central claim that, when fine-tuning a model for policy optimization, policy visitation frequency is more important than model error magnitude. It also shows that prediction accuracy, contact fidelity, and policy performance are relatively independent simulator metrics. Second, it designs and implements a controlled testbed and a contact-event diagnostic tool for evaluating robot simulators, providing reusable evaluation tools for the fine-tuning process.Using the same NeRD model, the same number of state-transition samples collected from the same physics engine, and the same fine-tuning procedure, the paper quantitatively compares seven distinct complete data-acquisition pipelines across multiple Ant robot tasks. It evaluates policy returns and prediction errors in both the learned simulator and the ground-truth environment, providing empirical support for its conclusions.
Critical strengths
- The overall experimental design is sufficiently rigorous. Except for the data-selection strategy, the experimental procedures are kept consistent to control for irrelevant variables. The evaluation metrics cover multiple dimensions, including policy return, prediction error, and contact fidelity, and statistical analyses are used to reduce the influence of experimental variability. The experiments compare seven different data-selection strategies across multiple tasks on the Ant robot and on ANYmal, while also reporting metrics from the ground-truth environment as references. This rigorously designed experimental framework can provide useful guidance for evaluating other learned simulators.
- The paper’s discussion and validation of the basic design principles for data selection during fine-tuning can provide guidance for future fine-tuning of learned simulators. The paper conducts a detailed comparison between policy-visitation-based and model-error-based data-selection strategies, and uses statistical results to demonstrate which type of data selection can better support the policy optimization process.
Critical weaknesses
- The experiments use only the Ant and ANYmal robots, both of which are legged robots. They do not include other robot morphologies, such as wheeled or aerial robots, nor do they further validate the conclusions on real robots. Therefore, it is unclear whether the conclusions generalize to real-world settings or other types of robots. In addition, the paper evaluates only the combination of the NeRD model and PPO, making it likewise difficult to determine whether the conclusions generalize to other models and policy-optimization procedures. Overall, the empirical evidence is limited in breadth, making it difficult to establish the broad applicability of the claims. The implications for real-robot deployment also remain insufficiently clear.
- All of the algorithms evaluated in this paper are existing methods, and the idea of prioritizing policy visitation is also not novel. Apart from comparing policy-visitation-based methods with model-error-based methods, the paper’s apparent innovation seems limited to improving the contact-diagnosis method. This raises concerns about whether the paper provides sufficient novelty.
Questions for authors
- Table1 shows that each data-acquisition procedure consists of three components: starting states, collection actions, and selection criterion. Is the claim that policy visitation is superior to model error intended to apply to the data-acquisition procedure as a whole, or only to the selection criterion component?
2 comments
Thank you for your detailed response. The Wipe example illustrates the paired-measurement procedure, but I am more interested in how this diagnostic differs from existing diagnostic methods and what is distinctive about its design. Could the authors explain why the paper’s contribution is sufficiently important, given that the evaluated algorithms and the idea of prioritizing policy visitation already exist? The authors have clarified that the paper’s main comparison concerns complete data-acquisition procedures rather than the selection criterion alone. I would appreciate further clarification on the extent to which the observed performance differences can be attributed to an emphasis on policy-visited states, rather than to other differences between the procedures.
Thank you for recognizing the controlled design. Following your comments, we:
W1.1–W1.4 address the distinct concerns within your first weakness.
[W1.1 & W1.3 & W1.4] Evaluation breadth and scope
We compare seven procedures on Ant, ANYmal, Jackal, and Franka Wipe, with three independently trained models per task. The implementations cover NeRD-PPO, GRU-SAC, and residual ensemble-SAC. Figure 2 includes all planned policies, averaged within models. VS-U has the highest mean on Jackal; PR leads on Wipe at 71.98, versus 36.24 for EA and 29.62 for VS-C. We report conclusions for the tested configurations and identify crossed evaluation as future work (§5).
[W1.2] Physical deployment and simulated transfer
Physical-robot evaluation is future work. The added transfer study uses simulated source and target environments, replacing nominal PD dynamics with a learned ANYdrive actuator. Each condition receives 100,000 cost units: one per selected transition and up to 600 per actual termination for recovery.
PR exceeds BASE and EA-online in target mean return on all three models. Both contrasts exceed the larger within-model two-policy return gap in 2/3 models; the source-only contrast does so in 1/3, while contrasts with random-action collection vary in sign. This is a descriptive training-variation criterion. Two EA-online collectors acquire no further transitions after their first update. We also specify reset-access and error-feedback requirements (§4.3).
[W2] Paired contact evidence and contribution
On Wipe, GT and EA each respond to 19/1,096 probes, but share only one responding probe. Paired measurements therefore distinguish response locations even when aggregate counts match.
The protocol applies opposite velocity perturbations with identical actions over ten controls, comparing contact-vector sequences after removing consecutive duplicates. Shared responses indicate activation in both simulators; agreement of their sequences is separate.
Independent GT replays reproduce 16 responding and 24 nonresponding probes, using one initial model per task. Scale, horizon, and contact-criterion checks characterize sensitivity. Figure 3 reports prediction error; Table 3 reports shared/extra responses (§4.2). We connect these diagnostics to controlled acquisition-reuse comparisons, building on policy-relevant learning and contact-aware modeling (§2).
[Q1] Procedures versus selection criteria
EA-versus-visitation compares complete procedures; component controls vary starts, actions, and selection separately. Within visited candidates, VS-C minus VS-U estimates joint contact-error weighting: +0.2 SD return, 95% CI [-0.5, +0.9], Holm p=1. This estimates the combined rule, not an isolated contact benefit. Selected-distribution analyses retain all 12 models and are exploratory (§4.3).