acceptodds
Under review as a conference paper at ICLR 2027

PolicyIK: High-Precision Inverse Kinematics for Robot Arms with Reinforcement Learning

Abstract

Inverse kinematics (IK) converts end-effector pose commands into joint motion, and many vision-language-action (VLA) policies rely on it to act on a given arm and to transfer across embodiments. Precise IK remains laborious: redundancy and singularities complicate the inverse mapping, execution depends on hand-tuned controllers, and training data on physical robots are costly to collect. We present PolicyIK, a goal-conditioned reinforcement learning policy that updates joint-position references from pose feedback and observes its accumulated reference relative to the measured joints, so the arm can rest on the goal rather than millimeters away. Two-scale pose errors with an error-dependent action gain let it correct millimeter residuals without slowing large motions, and rewards for progress and for staying within tolerance teach it to both reach and retain the target. In simulation, a single attempt keeps the mean errors over the final second within 1 mm and on 93.5% of held-out tasks, and a deployment protocol raises success to 98.9%. On physical hardware, PolicyIK achieves comparable or higher holding success than the evaluated classical IK controllers under matched execution budgets, with higher success rates under longer holding requirements. In simulated pick-and-place, it executes the end-effector commands of the VLA policy with 95.7% success. These results show that reinforcement learning can provide high-precision inverse kinematics for robot arms.

Primary area:
applications to robotics, autonomy, planning
Keywords:
reinforcement learning, robotics, high-precision robotic control, inverse kinematics

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_iKpWOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper uses RL to train a policy to control a robot arm at the joint level to reach a target pose. Methods are proposed for both real (off-line RL) and sim (on-line RL). A staged start-state curriculum is used to gradually learn the workspace. The learned controller achieves high accuracy of < 1 mm and < 1 degree.

Critical strengths
  • IK and joint control are important robotic problems
  • The learned controller leverages deep neural networks, leading to a simple method that learns from the data.
  • The paper is thorough in its experimentation, covering both sim and real.
  • The achieved accuracy is better than what can be achieved with traditional controllers.
Critical weaknesses
  • The paper is not self-contained. Some of the basic concepts (like the reward function) can only be found in the supplemental.
  • Notation is used without defining it.
  • The order in which details are presented makes them hard to follow. The reward function, for example, is scattered throughout the paper. Of the terms shown in Eq (6) L261, two appear later in the section, while the other three are in the appendix.
  • The reward function contains 5 terms, several of which are themselves composed of multiple terms. All in all, there are approx. 10 terms in the reward function. This brings up the question of reward shaping, and sensitivity to parameters.
  • The work highly overlaps with "Joint Space Control via Deep Reinforcement Learning" (IROS 2021, https://arxiv.org/abs/2011.06332), which proposes a very similar approach (and also avoids obstacles). This paper should be cited, and the differences clearly explained.
Questions for authors
  • How do you measure accuracy on a real robot?
  • Why are no videos included showing the simulated and real robots executing the policy?
  • How does the method handle changes in weight distribution - like picking up an object?
  • How hard would it be to modify the controller to avoid obstacles?
  • Table 2 highlights the last row, but the best number is actually DLS (hand-tuned).
  • L154 The "joint reference z_t" could be better explained. The paper assumes quite a bit of controls knowledge to make sense of it.
  • L190 The vee operator ('v') needs to be defined / explained, to make the paper self-contained. Too much knoweldge is assumed.
  • L357 What is a "single attempt"?
  • L357-360 Your method achieves 98.9% but the closest baseline achieves 0.04%? This gap is too large for the baseline to be considered credible.
7 comments
AuthorsOpenReview ↗

Response to Reviewer iKpW (Part 1/4)

We thank the reviewer for the careful reading and detailed feedback. We have revised the paper accordingly and addressed each of the raised weaknesses and questions in detail below. All revisions are marked in blue in the revised manuscript, and a blue section title indicates that we refined the writing of the whole section.

W1 & W3 Presentation of the reward function

W1. The paper is not self-contained. Some of the basic concepts (like the reward function) can only be found in the supplemental.

W3. The order in which details are presented makes them hard to follow. The reward function, for example, is scattered throughout the paper. Of the terms shown in Eq (6) L261, two appear later in the section, while the other three are in the appendix.

Response

Thank you for suggesting a clearer presentation. We have consolidated the definitions of all five reward groups in Section 4.3 (Reward Design), following their order in Eq. (7) (Eq. (6) in the submission), with implementation details retained in the appendix.


W2 & L154 & L190 & L357 Notations

W2. Notation is used without defining it.

L154. The "joint reference z_t" could be better explained. The paper assumes quite a bit of controls knowledge to make sense of it.

L190. The vee operator ('v') needs to be defined / explained, to make the paper self-contained. Too much knoweldge is assumed.

L357. What is a "single attempt"?

Response

Thank you for the detailed suggestions. We now define these terms where they first appear in the revised version.

  • Joint reference ztz_t (L154). ztz_t is the target joint positions sent to the arm's position controller. The arm does not reach this target exactly, so ztz_t differs from the measured joint positions qtq_t. Section 3 now states this.
  • Vee operator (L190). Log⁡(⋅)∨\operatorname{Log}(\cdot)^\vee converts a rotation matrix into a rotation vector, whose direction is the rotation axis and whose length is the rotation angle. Section 3 now defines it where it first appears.
  • Single attempt (L357). A single attempt is one 160-step policy rollout (5.3 s) toward a goal, without retries. Results with retries follow the deployment protocol and are reported separately. Section 5.1 now defines this.

W4 Reward-weight sensitivity

W4. The reward function contains 5 terms, several of which are themselves composed of multiple terms. All in all, there are approx. 10 terms in the reward function. This brings up the question of reward shaping, and sensitivity to parameters.

Response

We thank the reviewer for raising this question. We evaluated sensitivity by scaling the weights within each of the five reward groups together while keeping the other groups unchanged, then retraining using the recipe of Table 1 in the paper. We added these results to Appendix F.2 (Table 8) of the revised manuscript.

  • Most weight changes have little effect. Halving or doubling any group’s weights keeps strict success within 1.3 percentage points of the default. Dwell, motion cost, and limit cost remain within one percentage point of the default across the ×0.25–×4 range. High success therefore does not require precisely tuned group weights.
  • Only extreme tracking or progress weights reduce precision. At tracking ×0.25 and progress ×4, training can plateau at centimeter-level errors instead of continuing toward precise reaching.

The same reward terms are also effective for training PolicyIK directly on physical OpenArm.

Table R1: Strict success (%) under reward-group weight scaling.

Reward group×0.25×0.5×1 (default)×2×4
Tracking30.9793.7593.5392.2489.92
Progress92.4493.2493.5392.820.00
Dwell92.6693.4993.5393.6493.50
Motion cost93.3793.0393.5392.9093.02
Limit cost93.0292.9993.5394.1693.98
AuthorsOpenReview ↗

Response to Reviewer iKpW (Part 2/4)

W5 Differences from JAiLeR

W5. The work highly overlaps with "Joint Space Control via Deep Reinforcement Learning" (IROS 2021, https://arxiv.org/abs/2011.06332), which proposes a very similar approach (and also avoids obstacles). This paper should be cited, and the differences clearly explained.

Response

JAiLeR is cited in the submission (Introduction and Related Work), and we further clarify the differences from JAiLeR below. Although PolicyIK and JAiLeR both use reinforcement learning for joint-space control, the two methods differ in major ways. PolicyIK reaches full 6-D pose goals within 1 mm and 0.2° and can be trained directly on the physical arm without simulation, whereas JAiLeR evaluates 3-D position goals within 1 cm and transfers its policy from simulation. The major aspects are listed below.

Table R1: Comparison between JAiLeR and PolicyIK.

AspectJAiLeRPolicyIK
Goal evaluated3-D positionFull 6-D pose
Arm and task redundancy6-DoF Jaco with a 3-D goal: 3 redundant DoF7-DoF OpenArm with a 6-D goal: 1 redundant DoF
Reported accuracySimulation: 96.5% within 1 cm, mean error 4 mmSimulation: 93.5% within 1 mm and 0.2° in one attempt, 98.9% with the deployment protocol
Physical-arm trainingSim-to-real transfer, with additional real-arm fine-tuningDirect real-arm training with FlashSAC

JAiLeR avoids obstacles, but only for position goals. We extended PolicyIK to avoid obstacles while reaching full 6-D pose goals, and report the results in response to Q4.


Q1 Accuracy Measurement

Q1. How do you measure accuracy on a real robot?

Response

We compute it from the joint encoders. The measured joint angles are passed through the robot’s forward kinematics, and the resulting gripper-base pose is compared with the goal (Section 5.2). This follows the standard definition of IK error [1] and is consistent with the evaluation metrics used in prior IK-solver studies, including IKFlow [2] and CppFlow [3]. It is not absolute positioning accuracy as defined by ISO 9283 [4], which relies on independent pose measurements and therefore also captures errors in the robot's kinematic model.

We added such an external measurement in Appendix L. Two calibrated cameras track an AprilTag cube mounted in place of the gripper, independently of the encoders. We evaluated PolicyIK and DLS on 100 tasks. The cameras confirm that tasks counted as held are physically still.

Table 17: Tasks held (of 100) by the joint encoders, and median errors at 25 s. Parentheses: held tasks that the cameras also measure as still.

Controller15 s, held 1 s15 s, held 5 s25 s, held 1 s25 s, held 5 smm°
DLS42 (41)22 (19)52 (49)38 (34)0.550.11
PolicyIK65 (64)45 (41)74 (72)68 (65)0.560.12

[1] Lynch and Park. Modern Robotics: Mechanics, Planning, and Control, Sec. 6.2. Cambridge University Press, 2017.

[2] Ames et al. IKFlow: Generating Diverse Inverse Kinematics Solutions. IEEE RA-L, 2022.

[3] Morgan et al. CppFlow: Generative Inverse Kinematics for Efficient and Robust Cartesian Path Planning. ICRA, 2024.

[4] International Organization for Standardization. ISO 9283:1998 — Manipulating Industrial Robots: Performance Criteria and Related Test Methods. ISO, 1998.


Q2 Videos

Q2. Why are no videos included showing the simulated and real robots executing the policy?

Response

We have added videos to the supplementary material. On the physical OpenArm, they show PolicyIK reaching four consecutive goals within 1 mm and 0.2°, a comparison with a classic IK controller, and the improvement of the policy over real-arm training. In simulation, they show examples of PolicyIK reaching held-out goals and a pick-and-place task in which PolicyIK executes the pose commands of π0.5.

AuthorsOpenReview ↗

Response to Reviewer iKpW (Part 3/4)

Q3 Robustness to payload changes

Q3. How does the method handle changes in weight distribution - like picking up an object?

Response

We thank the reviewer for this question. PolicyIK maintains high strict success across tool payloads of 0–500 g, with less than a one-percentage-point decrease in both configurations. With arm-gravity feedforward, it achieves 94.0% at 500 g, compared with 40.7% for DLS, 49.7% for QP, and 49.5% for MPC. We evaluate on the same 4,120 tasks, modeling the payload as a point mass rigidly attached at the tool center point (TCP). We added this experiment to Appendix H.1 (Table 10) of the revised manuscript.

Table R1: Strict success (%) in simulation. All controllers receive gravity compensation for the known payload. The two PolicyIK policies are trained without and with arm-gravity feedforward, respectively.

ControllerArm-gravity feedforward0 g100 g300 g500 g
DLSYes52.0650.2945.4640.70
QPYes56.2155.4452.9649.73
MPCYes56.3355.7853.2849.52
PolicyIKNo93.49 ± 0.2093.37 ± 0.1093.31 ± 0.0792.89 ± 0.17
PolicyIKYes94.34 ± 0.3294.44 ± 0.2994.17 ± 0.4094.00 ± 0.36

Q4 Obstacle-avoidance extension

Q4. How hard would it be to modify the controller to avoid obstacles?

Response

PolicyIK extends naturally to obstacle avoidance by adapting its observation, constraint penalties, and training curriculum. These changes follow the same principles used in our simulation and real-robot settings, without changing the core control approach. In the spherical-obstacle evaluation setup of JAiLeR [1], our extension achieves 97.5%, 93.9%, and 90.6% collision-free episodes with one, two, and three obstacles under full-pose goals. Appendix H.3 and Table 12 of the revised manuscript describe the extension and report the results.

Table R1: Collision-free episodes (%) with full-pose goals and 8 cm-radius spheres. Evaluation includes tasks with collision-free start and goal arm configurations.

Obstacle-avoidance training1 obstacle2 obstacles3 obstacles
JAiLeR recipe77.8661.9150.54
PolicyIK obstacle extension97.49 ± 0.2893.91 ± 0.4890.58 ± 0.65

JAiLeR demonstrates obstacle avoidance for position goals using link-to-obstacle vectors and a proximity penalty [1]. However, full-pose goals additionally constrain orientation, leaving less freedom to adjust the arm posture around obstacles. We adapt three parts of PolicyIK:

  • Make nearby obstacles more visible to the policy. Each link receives the direction to its nearest obstacle, weighted more strongly as the link gets closer and set to zero beyond 10 cm. The observation also includes the predicted minimum distance to obstacles over the next three control steps at the current joint velocity. This follows our existing approach of emphasizing nearby constraints in the observation.
  • Penalize approaching and contacting obstacles. We add a proximity penalty and terminate training episodes on collision, extending the constraint-penalty design used for joint limits.
  • Include obstacles throughout the curriculum. Obstacles are present from the first training stage, so the policy learns reaching and avoidance together. We retain 25% obstacle-free episodes to continue training precise reaching without obstacles.

[1] Kumar et al. Joint Space Control via Deep Reinforcement Learning. IROS, 2021.


Q5 Best-result marking in Table 2

Q5. Table 2 highlights the last row, but the best number is actually DLS (hand-tuned).

Response

Thank you for pointing this out. As in our other tables, the shaded row marks our method rather than the best result. Table 2 had omitted the bold and underline used elsewhere for the best and second-best numbers. We have added them for clarity.

Reviewer_9Jw9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents PolicyIK, a goal-conditioned RL method for high-precision inverse kinematics via a joint-position interface. By innovating in reference accumulation, multi-scale pose errors, and progress rewards, PolicyIK attains millimeter-level accuracy without classical solvers, as comprehensively demonstrated in extensive simulations and real-world tests. The ablations are thorough and the appendices provide strong reproducibility. However, the baseline comparisons and evaluation breadth require strengthening before publication. I recommend acceptance after revisions.

Critical strengths

By innovating in reference accumulation, multi-scale pose errors, and progress rewards, PolicyIK attains millimeter-level accuracy without classical solvers, as comprehensively demonstrated in extensive simulations and real-world tests. The ablations are thorough and the appendices provide strong reproducibility.

Critical weaknesses

The baseline comparisons and evaluation breadth require strengthening before publication. I recommend acceptance after revisions.

Questions for authors

(1) The only RL baseline (UMI-RL) performs near zero, inflating PolicyIK’s apparent advantage. Include comparisons with recent learned IK methods and the hybrid RL+DLS approach to situate PolicyIK properly in the literature. (2) Please evaluate classical controllers over a longer settling time to confirm PolicyIK’s edge is not merely a time-window effect. (3) Provide a brief sensitivity analysis for the hand-tuned error-dependent gain thresholds, or discuss learning them adaptively. (4) Insufficient coverage of recent work (2025-2026): The literature review misses key recent advances in learned IK and VLA execution layers. Please add discussions on recent generative IK approaches and RL-based IK methods. (5) Writing polish. Please sharpen Related Work distinctions vs. residual/integrating methods and shorten overly long paragraphs.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 9Jw9 (Part 1/2)

We thank the reviewer for the insightful and positive evaluation of our work. We revised the paper accordingly and addressed the concerns in detail below. All revisions are marked in blue in the revised manuscript.

Q1 Comparison with learned IK and hybrid RL + DLS

Q1. The only RL baseline (UMI-RL) performs near zero, inflating PolicyIK's apparent advantage. Include comparisons with recent learned IK methods and the hybrid RL+DLS approach to situate PolicyIK properly in the literature.

Response

We evaluated CycleIK, IKFlow, and the hybrid RL + DLS method MPDIK on the same 4,120 held-out tasks as Table 1 in the paper. With the same arm-gravity compensation and the deployment protocol, PolicyIK achieves the highest strict success rate in this comparison, 99.35%.

Table R1: Comparison with learned and hybrid IK methods in simulation. All methods use exact arm-gravity compensation. Strict/coarse tolerances are 1 mm / 0.2° and 1 cm / 2°, respectively. Best results are in bold and second-best results are underlined.

MethodStrict (%)Coarse (%)Median pos (mm)Median ori (°)Mean attempts
CycleIK0.01 ± 0.0160.81 ± 6.525.13 ± 0.831.61 ± 0.151.000
CycleIK + DLS finisher78.19 ± 1.5295.54 ± 0.830.310 ± 0.0350.091 ± 0.0051.000
IKFlow8.16 ± 0.9797.26 ± 1.731.19 ± 0.040.472 ± 0.0181.000
IKFlow + DLS finisher92.62 ± 1.1099.65 ± 0.120.146±0.003‾\underline{0.146 \pm 0.003}0.054 ± 0.0011.000
MPDIK22.23 ± 0.1599.02 ± 0.080.894 ± 0.0040.401 ± 0.0041.000
MPDIK + DLS finisher94.17 ± 0.2999.60 ± 0.040.137 ± 0.0020.052±0.000‾\underline{0.052 \pm 0.000}1.000
PolicyIK94.34±0.32‾\underline{94.34 \pm 0.32}96.76 ± 0.140.234 ± 0.0080.037 ± 0.0011.000
PolicyIK + deployment protocol99.35 ± 0.0399.63±0.03‾\underline{99.63 \pm 0.03}0.229 ± 0.0080.037 ± 0.0011.163 ± 0.007

We selected CycleIK and IKFlow for their reported millimeter-level accuracy and publicly available implementations. CycleIK predicts joint angles directly from a target pose, while IKFlow uses a conditional normalizing flow to generate candidate joint solutions. MPDIK combines an RL policy with DLS refinement on the kinematic model. Our reimplementations produce joint solutions that the arm tracks with gravity compensation. We additionally evaluated all three with the same online DLS finisher, which uses the measured arm state and Jacobian to correct the pose during execution. The revised manuscript adds CycleIK, IKFlow, and MPDIK to Table 1 (Section 5.1.1) and describes their implementation in Appendix E.3.

The DLS-augmented baselines fail more often on targets closer to kinematic singularities. PolicyIK succeeds on most tasks that all three DLS-augmented baselines fail, demonstrating its ability to handle challenging targets through learned feedback control.


Q2 Longer settling time

Q2. Please evaluate classical controllers over a longer settling time to confirm PolicyIK's edge is not merely a time-window effect.

Response

PolicyIK's lead is not a time-window effect. Our hardware trials were recorded for 40 s for all methods, but the submission reported only the 15 s and 25 s budgets for space. With the extra page of the revision, we now add the 40 s budget to Table 3 (Table R1 below).

The longer budget does not change the picture. The classic controllers do benefit from the extra time, mostly in the 5 s hold, yet none of them catches up with PolicyIK. Even MPC, the strongest of them at 40 s, matches PolicyIK only for the 1 s hold and still falls clearly behind for the 5 s hold. The table also shows that PolicyIK reaches the goal faster: within 15 s, it already holds more tasks for the final 5 s than any classic controller does within 25 s. The extra time therefore mainly helps the classic controllers arrive, which PolicyIK does much earlier.

This pattern is what Section 5.2 and Figure 6 describe. The classic controllers keep correcting small residuals and move in and out of the tolerance, so more time does not translate into holding. PolicyIK, in contrast, stays within the tolerance once it arrives. Its lead therefore comes from sustained holding rather than from the choice of time window.

Table R1: Tasks held (of 60) on the physical OpenArm for the final 1 s or 5 s of each budget, with the 1 mm and 0.2° tolerance. The 15 s and 25 s columns are those of Table 3, and the 40 s column is new.

Controller15 s: 1 s15 s: 5 s25 s: 1 s25 s: 5 s40 s: 1 s40 s: 5 s
DLS21629172721
Resolved-rate QP21624162517
Kinematic MPC10326153024
Adapted OpenArm QP8116112117
PolicyIK252029263030
AuthorsOpenReview ↗

Response to Reviewer 9Jw9 (Part 2/2)

Q3 Sensitivity of gain thresholds

Q3. Provide a brief sensitivity analysis for the hand-tuned error-dependent gain thresholds, or discuss learning them adaptively.

Response

We thank the reviewer for this suggestion and added the sensitivity analysis to Appendix F.2 (Table 8) of the revised manuscript. We varied the threshold windows and minimum gain separately. Performance remains stable when any parameter setting is halved or doubled. Strict success decreases only when the gain remains high near the target, which occurs when a threshold window is quartered or the minimum gain is raised from 0.1 to 0.4.

  1. Moderate parameter changes have little effect. Across all ×0.5 and ×2 settings, strict success remains within 1.6 percentage points of the default.
  2. Reducing gain too late or retaining a high minimum gain reduces precision. Quartering the position or orientation window shifts gain reduction to smaller errors. The controller therefore retains larger reference updates closer to the target, leaving less room for fine corrections. Because the overall gain takes the maximum of the position- and orientation-dependent gains, either component can keep the update large. Strict success decreases by 5.3 and 6.6 percentage points, respectively. Increasing the minimum gain from 0.1 to 0.4 instead makes the smallest reference update four times larger for the same policy output, reducing strict success by 3.5 points.

Table R1: Strict success (%). Both endpoints of each threshold window are scaled together, and other parameters remain unchanged.

Parameter varied×0.25×0.5×1 (default)×2×4
Position window88.2092.3593.5393.8893.62
Orientation window86.8992.2993.5393.3093.28
Minimum gain93.2992.6393.5391.9490.00
  1. Adjusting gain with error outperforms using a fixed gain. The error-dependent schedule achieves 93.49% strict success, compared with 85.62–89.39% for the three tested fixed gains.

Table R2: Strict and coarse success (%) with fixed and error-dependent gains.

GainStrict (%)Coarse (%)
Fixed 0.2589.39‾\underline{89.39}94.20‾\underline{94.20}
Fixed 0.587.5993.33
Fixed 1.085.6293.37
Error-dependent93.4996.56

Q4 Coverage of recent related work

Q4. Insufficient coverage of recent work (2025-2026): The literature review misses key recent advances in learned IK and VLA execution layers. Please add discussions on recent generative IK approaches and RL-based IK methods.

Response

We thank the reviewer for pointing out these missing discussions. We expanded Related Work to discuss recent generative IK, RL-based IK, and VLA execution methods, and clarified their relationship to PolicyIK.

  • Generative and learned IK. We added IKDiffuser for kinematic trees and MorphIK for unseen robot morphologies, and expanded the discussion of MimicIK, which learns joint-displacement commands from teleoperation. The revised discussion distinguishes learning from kinematic data or demonstrations from PolicyIK’s learning through interaction with robot dynamics for precise pose reaching and holding. (Section 2, “Inverse kinematics and task-space control”)
  • RL-based IK. We added a recent method that learns residual corrections to IK commands, and expanded the discussion of MPDIK, which combines RL with DLS refinement. We clarify how these hybrid approaches differ from PolicyIK’s use of a single learned policy for joint-reference updates without a separate IK solver. (Section 2, “Reinforcement learning for reaching and pose tracking”)
  • VLA execution layers. We added a paragraph covering simulation-trained residual action correction, adaptive command execution, and velocity feedforward. We explain how PolicyIK converts VLA-derived pose references into joint-reference updates, as demonstrated in our pick-and-place experiment. (Appendix A, “Execution layers for VLA policies”)

Q5 Related Work distinctions and concision

Q5. Writing polish. Please sharpen Related Work distinctions vs. residual/integrating methods and shorten overly long paragraphs.

Response

Thank you for these writing suggestions. We clarified both distinctions and condensed the paragraphs. Unlike methods that learn residual corrections to IK solutions, PolicyIK requires no nominal IK solution. Like integrating methods, it accumulates joint-reference updates. It also observes the accumulated reference relative to the measured joints and scales its updates with the pose error. The revised paragraph “Learning for execution-error compensation” in Section 2 states these distinctions.

Dear Reviewer,

Thanks for your time and constructive feedback. As the discussion period closes in two days, we want to confirm whether our responses have addressed your concerns.

We would be happy to clarify any remaining questions. Thank you again for your time and consideration.

Best regards, The Authors

Reviewer_vmdEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors tackle the problem of full-pose end-effector reaching with robot arms using goal-conditioned reinforcement learning. Through careful empirical approach and well-designed observation space, they show that its comparable or better than existing solutions. The accuracy achieved in simulation and results on hardware are promising. Some of the key ingredients are - representing pose errors in multiple scales, reduction of action magnitude when approaching near the goal and rewarding both movement towards the goal and staying there.

Critical strengths
  1. The experimental set up and ablations present interesting findings - The usefulness of command-state is technically sound and the experimental results back it up - without command-state feedback the success rate falls noticeably.
  2. The focus on evaluating the differences in reaching and holding an end-effector pose is well thought out. The dwell ablation provides useful insights.
  3. The integration with hardware and a upstream VLA gives the paper practical relevance.
Critical weaknesses
  1. The results on the hardware needs more clarifications - were the end-effector pose errors measured on the real robot verified by some ground-truth measurements? For example, like motion-capture system?
  2. More details on the hindsight experience replay implementation will be helpful. In the proposed method, changing the goal in hindsight would have an impact on the action accumulation? How does it impact equation 1?
  3. Integration with VLA does not need the pose accuracy requirements the authors strived to achieve. Showing a pick and place task with PolicyIK is helpful, however the task does not need the level of accuracy the policy is capable off. It might strengthen the paper if the chosen task requires the sub-mm accuracy to really showcase the practical use of the proposed method.
Questions for authors

Answering some of the questions raised in the weakness section can be helpful improving the paper.

  1. It seems like the sample budget for training the model in simulation is vastly greater than real world experiments, from the numbers in the appendix, simulation uses 10000 times more samples than real robot experiments. Yet the performance in simulation is a modest improvement on the reported real robot results - is it correct to state that performance does not proportionally improve with sample count? I understand the algorithms used are different - PPO vs FlashSAC, yet it brings up the question of the number of samples actually needed to learn PolicyIK.
  2. A psuedo-code describing the PPO and FlashSAC will be helpful to a reader and someone trying to reproduce these results.
6 comments
AuthorsOpenReview ↗

Response to Reviewer vmdE (Part 1/3)

We thank the reviewer for the thoughtful and constructive feedback. We have revised the paper accordingly and addressed each of the raised weaknesses and questions in detail below. All revisions are marked in blue in the revised manuscript, and a blue section title indicates that we refined the writing of the whole section.

W1 Real-Robot End-Effector Pose Error Measurement

W1. The results on the hardware needs more clarifications - were the end-effector pose errors measured on the real robot verified by some ground-truth measurements? For example, like motion-capture system?

Response

The errors are computed from the joint encoders through forward kinematics, so they are relative to the robot’s kinematic model. We added an external measurement in Appendix L. Two calibrated cameras track an AprilTag cube mounted in place of the gripper, independently of the encoders. We evaluated PolicyIK and DLS on 100 tasks with the same goals.

The holds are physical. In each cell, cameras and encoders agree on whether the gripper is still in 92–96 of the 97–98 tasks both cameras see (Table 17), and they measure similar motion (Table 18).

Table 17: Tasks held (of 100) by the joint encoders, and median errors at 25 s. Parentheses: held tasks that the cameras also measure as still.

Controller15 s, held 1 s15 s, held 5 s25 s, held 1 s25 s, held 5 smm°
DLS42 (41)22 (19)52 (49)38 (34)0.550.11
PolicyIK65 (64)45 (41)74 (72)68 (65)0.560.12

Table 18: Median motion (mm) during the final window, joint encoders / cameras.

ControllerTasks15 s, final 1 s15 s, final 5 s25 s, final 1 s25 s, final 5 s
DLSHeld0.03 / 0.150.42 / 0.460.01 / 0.160.18 / 0.35
DLSNot held0.71 / 0.872.06 / 2.060.45 / 0.521.79 / 1.74
PolicyIKHeld0.01 / 0.090.04 / 0.110.01 / 0.080.02 / 0.10
PolicyIKNot held0.08 / 0.210.80 / 0.890.01 / 0.100.02 / 0.16

W2 Hindsight relabeling and Equation 1

W2. More details on the hindsight experience replay implementation will be helpful. In the proposed method, changing the goal in hindsight would have an impact on the action accumulation? How does it impact equation 1?

Response

Appendix I.3 (“Goal relabeling”) of the revised manuscript now describes HER implementation with more details. We draw hindsight goals from sources similar to those of OGBench [1], with our own sampling probabilities. Each transition sampled from the online buffer receives its goal from one of three sources:

  • the commanded goal of its episode (probability 0.4), which keeps training on the attempted task, including failed attempts;
  • a future goal (0.4), the pose reached a geometrically distributed number of steps later in the same episode, which turns every trajectory into successful examples for the poses it passes through;
  • the current goal (0.2), the pose reached at the transition itself, which trains holding. In 30% of these draws, the pose is perturbed by 1 mm and 0.2°. These perturbed goals serve the same purpose as the random goals of OGBench: they are goals the arm did not reach, placed at the scale of the tolerance to train small corrections.

Relabeling does not change the accumulated reference. A relabeled transition keeps the measured joint states and the references zt−1z_{t-1} and ztz_t of the physical transition, so it still describes the motion the arm actually executed. Relabeling affects Eq. (2) (Eq. (1) in the submission) only through the gain β\beta, which is computed from the pose error to the goal.

Under a relabeled goal, the gain becomes β′\beta', and we rescale the stored action by β/β′\beta/\beta', so that Eq. (2) becomes

zt=clip⁡[zt−1+ηβ′ββ′at,qlo,qhi].\displaystyle z_t = \operatorname{clip}\left[z_{t-1} + \eta \beta' \frac{\beta}{\beta'} a_t, q^{\mathrm{lo}}, q^{\mathrm{hi}}\right].

The other goal-dependent quantities are recomputed under the relabeled goal: the goal pose and the pose-error channels of the observation (keypoint offsets and coarse and fine errors), the running-best features hth_t, and all reward terms. The running-best error is recomputed from the stored trajectory as the minimum tolerance-normalized error under the new goal over the earlier steps of the episode.

[1] Park et al. OGBench: Benchmarking Offline Goal-Conditioned RL. ICLR, 2025.

AuthorsOpenReview ↗

Response to Reviewer vmdE (Part 2/3)

W3 Sub-millimeter peg insertion

W3. Integration with VLA does not need the pose accuracy requirements the authors strived to achieve. Showing a pick and place task with PolicyIK is helpful, however the task does not need the level of accuracy the policy is capable off. It might strengthen the paper if the chosen task requires the sub-mm accuracy to really showcase the practical use of the proposed method.

Response

We thank the reviewer for this suggestion. We added a precision assembly task in simulation using the round-peg geometry of the NIST Assembly Task Board. NIST-board assembly is also studied in Factory [1], and peg insertion is used to evaluate contact-rich manipulation in FORGE [2]. The arm picks up a Ø12 mm peg, carries it to the board, and inserts it into holes with diametral clearances of 3, 1, and 0.5 mm. Success requires the peg to remain grasped and at least 15 mm inside the hole throughout the final second. Peg and hole positions are randomized, and all controllers receive the same target pose trajectory. Appendix H.2 and Table 11 of the revised manuscript describe the task and report the results.

PolicyIK maintains high insertion success across both carry distances and outperforms DLS and QP at tighter clearances. All controllers succeed at 3 mm, but their performance separates as the clearance narrows. PolicyIK maintains over 91% success at 0.5 mm for both carry distances, demonstrating its effectiveness on an assembly task requiring sub-millimeter alignment.

Table R1: Peg-insertion success (%) in simulation at decreasing diametral clearances. All controllers use the same arm-gravity compensation.

Carry distanceController3 mm1 mm0.5 mm
≈10 cmDLS100.096.743.3
≈10 cmQP100.097.146.1
≈10 cmPolicyIK100.0100.091.3
≈17 cmDLS100.096.529.4
≈17 cmQP100.096.330.4
≈17 cmPolicyIK100.0100.092.5

[1] Narang et al. Factory: Fast Contact for Robotic Assembly. RSS, 2022.

[2] Noseworthy et al. FORGE: Force-Guided Exploration for Robust Contact-Rich Manipulation under Uncertainty. arXiv:2408.04587, 2024.

AuthorsOpenReview ↗

Response to Reviewer vmdE (Part 3/3)

Q1 Number of samples needed

Q1. It seems like the sample budget for training the model in simulation is vastly greater than real world experiments, from the numbers in the appendix, simulation uses 10000 times more samples than real robot experiments. Yet the performance in simulation is a modest improvement on the reported real robot results - is it correct to state that performance does not proportionally improve with sample count? I understand the algorithms used are different - PPO vs FlashSAC, yet it brings up the question of the number of samples actually needed to learn PolicyIK.

Response

Yes, in simulation performance does not improve in proportion to the sample count. Strict success rises from 9.0% to 85.9% within the first 3.1×1083.1\times10^8 samples, but the final doubling to 1.28×1091.28\times10^9 adds only 2.1 percentage points (Table R1). We also note that the hardware result is well below the simulation result rather than close to it: 48% of tasks are held within 25 s on the arm (Table 3), against 93.5% strict success within a 5.3 s attempt in simulation (Table 1).

The 10410^4 gap in sample count comes from the training setups rather than from the task itself. First, the simulation workspace is deliberately broad. It surrounds a stand-alone arm on all sides, including where the torso and the second arm of the physical robot would be, whereas hardware goals lie in the practical working region in front of the arm, about one fifth of that volume (Appendix I.1). Second, on-policy PPO is itself less sample-efficient than off-policy FlashSAC: in the FlashSAC benchmarks, FlashSAC matches or outperforms PPO with a quarter of its environment steps [1]. Third, PPO with 4,096 parallel environments makes about 2×10−42\times10^{-4} gradient updates per transition, whereas FlashSAC on the arm makes three, so both settings use a similar total of about 3×1053\times10^5 updates. On the arm, HER and a start-state curriculum that keeps near-goal experience frequent further focus these updates where precision is decided (Appendix I.3).

As a result, PPO needs 6.4×1086.4\times10^8 samples for 91.3% strict success over the broad simulation workspace, while FlashSAC learns from 1.0×1051.0\times10^5 transitions a hardware policy whose holding success matches or exceeds that of four classic IK controllers. More samples still help on the arm. Under the new hardware recipe of the revision (Appendix K), training from 1.0×1051.0\times10^5 to 1.2×1051.2\times10^5 transitions raises the tasks reached within 25 s from 38 to 54 of 60, and those held for the final 5 s from 27 to 43 (Table R2).

Table R1: Strict and coarse success (%) in simulation after each training budget, with the PPO recipe of Table 1. Mean ± standard deviation over 3 seeds.

Training epochsSamples (M)Strict (%)Coarse (%)
80078.69.0 ± 7.584.5 ± 11.7
1,600157.357.9 ± 14.493.2 ± 0.1
3,200314.685.9 ± 1.294.4 ± 0.2
6,500639.091.3 ± 0.795.8 ± 0.3
13,0001,278.093.4 ± 0.496.5 ± 0.2

Table R2: Tasks reached and held (of 60) on the physical OpenArm within the 25 s budget, after 1.0×1051.0\times10^5 and 1.2×1051.2\times10^5 transitions under the new training recipe (Appendix K), with the 1 mm and 0.2° tolerance.

TransitionsReachedHeld 1 sHeld 5 s
1.0×1051.0\times10^5383327
1.2×1051.2\times10^5544543

[1] Kim et al. FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control. RSS, 2026.


Q2 Pseudo-code for PPO and FlashSAC

Q2. A psuedo-code describing the PPO and FlashSAC will be helpful to a reader and someone trying to reproduce these results.

Response

The revised manuscript adds pseudo-code for both training loops. Algorithm 1 in Appendix C describes PPO training in simulation, and Algorithm 2 in Appendix I.3 describes FlashSAC training on the physical arm.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their thoughtful and constructive feedback. We have incorporated the suggested experiments, discussions, and analyses into the revised manuscript, with revisions highlighted in blue. Below, we summarize our core contributions and the major updates introduced during the rebuttal.

Core contributions of PolicyIK

  1. Precise full-pose reaching without a classic IK solver (highlighted by 9Jw9, iKpW): PolicyIK formulates inverse kinematics as goal-conditioned RL and trains a single policy that reaches and holds full-pose goals within 1 mm and 0.2° through a joint-position interface. It succeeds on 93.5% of 4,120 held-out tasks in a single attempt and 98.9% with the deployment protocol.
  2. Design choices supported by ablations (highlighted by vmdE, 9Jw9): Ablations show that command-state feedback, multiscale pose feedback, and the progress and dwell rewards each contribute to precision, with separate effects on reaching and holding.
  3. Deployment on physical hardware and with VLA policies (highlighted by vmdE, 9Jw9, iKpW): PolicyIK is trained directly on the physical OpenArm with FlashSAC and holds the poses it reaches more reliably than classic IK controllers. It also executes the pose commands of π0.5 in simulated pick-and-place with 95.7% success.

Major updates of experimental results during rebuttal

  • Section 5.1.1 & Appendix E : Comparisons with classic IK controllers (DLS, QP, MPC), learned IK methods (CycleIK, IKFlow), the hybrid RL + DLS method MPDIK, and a further tuned UMI-RL, where PolicyIK shows a clear advantage in strict success (addressing 9Jw9's Q1 and iKpW's L357–360).
  • Section 5.2 & Table 3 : A 40 s budget on the physical arm, where PolicyIK still holds the most tasks for the final 5 s compared with the classic controllers (addressing 9Jw9's Q2).
  • Appendix H.2 : Sub-millimeter peg insertion, where PolicyIK succeeds in over 91% of trials at 0.5 mm clearance compared with 29–46% for DLS and QP (addressing vmdE's W3).
  • Appendix H.1 : Robustness to tool payloads of 0–500 g, where PolicyIK reaches 94.0% strict success at 500 g compared with 40.7–49.7% for the classic controllers (addressing iKpW's Q3).
  • Appendix H.3 : Obstacle-avoidance extension under full-pose goals, achieving 97.5%, 93.9%, and 90.6% collision-free episodes with one, two, and three obstacles (addressing iKpW's Q4).
  • Appendix F.2 : Sensitivity analysis of the gain thresholds and reward-group weights, showing that strict success stays close to the default when any setting is halved or doubled (addressing 9Jw9's Q3 and iKpW's W4).
  • Appendix L : External measurement on the physical arm with two calibrated cameras tracking an AprilTag cube, which confirms that the tasks counted as held by the joint encoders are physically still (addressing vmdE's W1 and iKpW's Q1).
  • Appendix K : Extended hardware training, where a second policy trained for 1.2×10⁵ transitions with an extended recipe holds 43 of 60 tasks.

Major updates of discussions and clarifications during rebuttal

  • Section 2 & Appendix A : Expanded related work on generative IK, RL-based IK, and VLA execution layers, with clearer distinctions from PolicyIK (addressing 9Jw9's Q4 & Q5 and iKpW's W5).
  • Section 4.3 : Clearer definitions of all five reward groups (addressing iKpW's W1 & W3).
  • Sections 3 & 5.1 : Clearer definitions of key notation and terms where they first appear (addressing iKpW's W2, L154, L190 & L357).
  • Table 2 & Supplementary Material : Best and second-best marking in Table 2, and videos of the simulated and physical robots (addressing iKpW's Q2 & Q5).
  • Appendix I.3 : Details of hindsight goal relabeling and how a relabeled goal enters Eq. (2) (addressing vmdE's W2).
  • Appendix C & I.3 : Pseudo-code of PPO training in simulation and FlashSAC training on the physical arm (Algorithms 1 and 2) (addressing vmdE's Q2).

We believe these revisions address the reviewers' concerns and strengthen our manuscript. The individual responses below provide the full tables and details for each point.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.