- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper uses RL to train a policy to control a robot arm at the joint level to reach a target pose. Methods are proposed for both real (off-line RL) and sim (on-line RL). A staged start-state curriculum is used to gradually learn the workspace. The learned controller achieves high accuracy of < 1 mm and < 1 degree.
Critical strengths
- IK and joint control are important robotic problems
- The learned controller leverages deep neural networks, leading to a simple method that learns from the data.
- The paper is thorough in its experimentation, covering both sim and real.
- The achieved accuracy is better than what can be achieved with traditional controllers.
Critical weaknesses
- The paper is not self-contained. Some of the basic concepts (like the reward function) can only be found in the supplemental.
- Notation is used without defining it.
- The order in which details are presented makes them hard to follow. The reward function, for example, is scattered throughout the paper. Of the terms shown in Eq (6) L261, two appear later in the section, while the other three are in the appendix.
- The reward function contains 5 terms, several of which are themselves composed of multiple terms. All in all, there are approx. 10 terms in the reward function. This brings up the question of reward shaping, and sensitivity to parameters.
- The work highly overlaps with "Joint Space Control via Deep Reinforcement Learning" (IROS 2021, https://arxiv.org/abs/2011.06332), which proposes a very similar approach (and also avoids obstacles). This paper should be cited, and the differences clearly explained.
Questions for authors
- How do you measure accuracy on a real robot?
- Why are no videos included showing the simulated and real robots executing the policy?
- How does the method handle changes in weight distribution - like picking up an object?
- How hard would it be to modify the controller to avoid obstacles?
- Table 2 highlights the last row, but the best number is actually DLS (hand-tuned).
- L154 The "joint reference z_t" could be better explained. The paper assumes quite a bit of controls knowledge to make sense of it.
- L190 The vee operator ('v') needs to be defined / explained, to make the paper self-contained. Too much knoweldge is assumed.
- L357 What is a "single attempt"?
- L357-360 Your method achieves 98.9% but the closest baseline achieves 0.04%? This gap is too large for the baseline to be considered credible.
7 comments
Response to Reviewer iKpW (Part 2/4)
W5 Differences from JAiLeR
W5. The work highly overlaps with "Joint Space Control via Deep Reinforcement Learning" (IROS 2021, https://arxiv.org/abs/2011.06332), which proposes a very similar approach (and also avoids obstacles). This paper should be cited, and the differences clearly explained.
Response
JAiLeR is cited in the submission (Introduction and Related Work), and we further clarify the differences from JAiLeR below. Although PolicyIK and JAiLeR both use reinforcement learning for joint-space control, the two methods differ in major ways. PolicyIK reaches full 6-D pose goals within 1 mm and 0.2° and can be trained directly on the physical arm without simulation, whereas JAiLeR evaluates 3-D position goals within 1 cm and transfers its policy from simulation. The major aspects are listed below.
Table R1: Comparison between JAiLeR and PolicyIK.
| Aspect | JAiLeR | PolicyIK |
|---|---|---|
| Goal evaluated | 3-D position | Full 6-D pose |
| Arm and task redundancy | 6-DoF Jaco with a 3-D goal: 3 redundant DoF | 7-DoF OpenArm with a 6-D goal: 1 redundant DoF |
| Reported accuracy | Simulation: 96.5% within 1 cm, mean error 4 mm | Simulation: 93.5% within 1 mm and 0.2° in one attempt, 98.9% with the deployment protocol |
| Physical-arm training | Sim-to-real transfer, with additional real-arm fine-tuning | Direct real-arm training with FlashSAC |
JAiLeR avoids obstacles, but only for position goals. We extended PolicyIK to avoid obstacles while reaching full 6-D pose goals, and report the results in response to Q4.
Q1 Accuracy Measurement
Q1. How do you measure accuracy on a real robot?
Response
We compute it from the joint encoders. The measured joint angles are passed through the robot’s forward kinematics, and the resulting gripper-base pose is compared with the goal (Section 5.2). This follows the standard definition of IK error [1] and is consistent with the evaluation metrics used in prior IK-solver studies, including IKFlow [2] and CppFlow [3]. It is not absolute positioning accuracy as defined by ISO 9283 [4], which relies on independent pose measurements and therefore also captures errors in the robot's kinematic model.
We added such an external measurement in Appendix L. Two calibrated cameras track an AprilTag cube mounted in place of the gripper, independently of the encoders. We evaluated PolicyIK and DLS on 100 tasks. The cameras confirm that tasks counted as held are physically still.
Table 17: Tasks held (of 100) by the joint encoders, and median errors at 25 s. Parentheses: held tasks that the cameras also measure as still.
| Controller | 15 s, held 1 s | 15 s, held 5 s | 25 s, held 1 s | 25 s, held 5 s | mm | ° |
|---|---|---|---|---|---|---|
| DLS | 42 (41) | 22 (19) | 52 (49) | 38 (34) | 0.55 | 0.11 |
| PolicyIK | 65 (64) | 45 (41) | 74 (72) | 68 (65) | 0.56 | 0.12 |
[1] Lynch and Park. Modern Robotics: Mechanics, Planning, and Control, Sec. 6.2. Cambridge University Press, 2017.
[2] Ames et al. IKFlow: Generating Diverse Inverse Kinematics Solutions. IEEE RA-L, 2022.
[3] Morgan et al. CppFlow: Generative Inverse Kinematics for Efficient and Robust Cartesian Path Planning. ICRA, 2024.
[4] International Organization for Standardization. ISO 9283:1998 — Manipulating Industrial Robots: Performance Criteria and Related Test Methods. ISO, 1998.
Q2 Videos
Q2. Why are no videos included showing the simulated and real robots executing the policy?
Response
We have added videos to the supplementary material. On the physical OpenArm, they show PolicyIK reaching four consecutive goals within 1 mm and 0.2°, a comparison with a classic IK controller, and the improvement of the policy over real-arm training. In simulation, they show examples of PolicyIK reaching held-out goals and a pick-and-place task in which PolicyIK executes the pose commands of π0.5.
Response to Reviewer iKpW (Part 3/4)
Q3 Robustness to payload changes
Q3. How does the method handle changes in weight distribution - like picking up an object?
Response
We thank the reviewer for this question. PolicyIK maintains high strict success across tool payloads of 0–500 g, with less than a one-percentage-point decrease in both configurations. With arm-gravity feedforward, it achieves 94.0% at 500 g, compared with 40.7% for DLS, 49.7% for QP, and 49.5% for MPC. We evaluate on the same 4,120 tasks, modeling the payload as a point mass rigidly attached at the tool center point (TCP). We added this experiment to Appendix H.1 (Table 10) of the revised manuscript.
Table R1: Strict success (%) in simulation. All controllers receive gravity compensation for the known payload. The two PolicyIK policies are trained without and with arm-gravity feedforward, respectively.
| Controller | Arm-gravity feedforward | 0 g | 100 g | 300 g | 500 g |
|---|---|---|---|---|---|
| DLS | Yes | 52.06 | 50.29 | 45.46 | 40.70 |
| QP | Yes | 56.21 | 55.44 | 52.96 | 49.73 |
| MPC | Yes | 56.33 | 55.78 | 53.28 | 49.52 |
| PolicyIK | No | 93.49 ± 0.20 | 93.37 ± 0.10 | 93.31 ± 0.07 | 92.89 ± 0.17 |
| PolicyIK | Yes | 94.34 ± 0.32 | 94.44 ± 0.29 | 94.17 ± 0.40 | 94.00 ± 0.36 |
Q4 Obstacle-avoidance extension
Q4. How hard would it be to modify the controller to avoid obstacles?
Response
PolicyIK extends naturally to obstacle avoidance by adapting its observation, constraint penalties, and training curriculum. These changes follow the same principles used in our simulation and real-robot settings, without changing the core control approach. In the spherical-obstacle evaluation setup of JAiLeR [1], our extension achieves 97.5%, 93.9%, and 90.6% collision-free episodes with one, two, and three obstacles under full-pose goals. Appendix H.3 and Table 12 of the revised manuscript describe the extension and report the results.
Table R1: Collision-free episodes (%) with full-pose goals and 8 cm-radius spheres. Evaluation includes tasks with collision-free start and goal arm configurations.
| Obstacle-avoidance training | 1 obstacle | 2 obstacles | 3 obstacles |
|---|---|---|---|
| JAiLeR recipe | 77.86 | 61.91 | 50.54 |
| PolicyIK obstacle extension | 97.49 ± 0.28 | 93.91 ± 0.48 | 90.58 ± 0.65 |
JAiLeR demonstrates obstacle avoidance for position goals using link-to-obstacle vectors and a proximity penalty [1]. However, full-pose goals additionally constrain orientation, leaving less freedom to adjust the arm posture around obstacles. We adapt three parts of PolicyIK:
- Make nearby obstacles more visible to the policy. Each link receives the direction to its nearest obstacle, weighted more strongly as the link gets closer and set to zero beyond 10 cm. The observation also includes the predicted minimum distance to obstacles over the next three control steps at the current joint velocity. This follows our existing approach of emphasizing nearby constraints in the observation.
- Penalize approaching and contacting obstacles. We add a proximity penalty and terminate training episodes on collision, extending the constraint-penalty design used for joint limits.
- Include obstacles throughout the curriculum. Obstacles are present from the first training stage, so the policy learns reaching and avoidance together. We retain 25% obstacle-free episodes to continue training precise reaching without obstacles.
[1] Kumar et al. Joint Space Control via Deep Reinforcement Learning. IROS, 2021.
Q5 Best-result marking in Table 2
Q5. Table 2 highlights the last row, but the best number is actually DLS (hand-tuned).
Response
Thank you for pointing this out. As in our other tables, the shaded row marks our method rather than the best result. Table 2 had omitted the bold and underline used elsewhere for the best and second-best numbers. We have added them for clarity.
Response to Reviewer iKpW (Part 1/4)
We thank the reviewer for the careful reading and detailed feedback. We have revised the paper accordingly and addressed each of the raised weaknesses and questions in detail below. All revisions are marked in blue in the revised manuscript, and a blue section title indicates that we refined the writing of the whole section.
W1 & W3 Presentation of the reward function
Response
Thank you for suggesting a clearer presentation. We have consolidated the definitions of all five reward groups in Section 4.3 (Reward Design), following their order in Eq. (7) (Eq. (6) in the submission), with implementation details retained in the appendix.
W2 & L154 & L190 & L357 Notations
Response
Thank you for the detailed suggestions. We now define these terms where they first appear in the revised version.
W4 Reward-weight sensitivity
Response
We thank the reviewer for raising this question. We evaluated sensitivity by scaling the weights within each of the five reward groups together while keeping the other groups unchanged, then retraining using the recipe of Table 1 in the paper. We added these results to Appendix F.2 (Table 8) of the revised manuscript.
The same reward terms are also effective for training PolicyIK directly on physical OpenArm.
Table R1: Strict success (%) under reward-group weight scaling.