- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper provides theoretical and empirical evidence that velocity-based reward terms are unnecessary for policy-gradient RL on stabilization control tasks. The authors show that learning performance can be sensitive to the coefficient of the velocity-based term in the reward function, and the PPO can learn stabilization control tasks without the velocity-based reward term with minimal or no performance degradation.
Critical strengths
The illustrative example in Figure 1 clearly demonstrates the sensitivity of performance to the coefficient of the velocity-based reward term. I appreciate the details of the reward functions of all tested tasks in table 1. Results in table 2 and Figure 5 provide insight why capping the value of the velocity-based term can recover some loss of performance, particularly for large coefficient values.
Critical weaknesses
The second part of section 4 provides theoretical explanation why velocity-based terms remain necessary in the agent’s observation. However, the explanation in lines 216–255 is unclear, hard to understand, and the writing clarity can be improved significantly. The following are a list of unclear technical phrases used in the explanation that have not been defined anywhere: “canonical phase-space state vector, dynamics seen from the policy action interface, dissipative force outside the policy, divergence in canonical phase space, force field, contract phase-space volume, full-state stabilizability, asymptotic attraction, passive damping, lower-level PD control, dissipative contact, dominance condition, structural obstruction, zeroth-order actor.”
Since Figure 3 shows the results of an ablation, the plot should include velocity-observation results to make comparison with no-velocity observation easier.
It is not clear why changing eq 11 to eq 12 makes it partial. The provided explanation is hard to understand: "Partial explicitly constrains the complete end-effector pose (position and orientation) but not the remaining null-space coordinate. Thus, “Partial” is complete in task space but incomplete in the robot’s generalized configuration."
The abstract sentence “We theoretically and empirically demonstrate that policy gradient methods can successfully solve stabilization tasks without first-order reward terms” is an overclaim. The paper empirically demonstrates only with PPO not all policy gradient methods.
The conclusion sentence “Our work provides a principled framework for reward design in deep reinforcement learning for robotics” is an overclaim. The scope of the work is much narrower.
Questions for authors
No questions.
2 comments
Fig. 3 should include velocity-observation results. The next version will use paired bars for every task. For the tasks below, dwell in seconds:
| Task | Full observation | No velocity |
|---|---|---|
| Cartpole | 7.81 | 0.63 |
| Double Cart Pendulum | 8.88 | 0.02 |
| Quadcopter | 10.09 | 0.00 |
| Franka | 14.86 | 11.54 |
| Humanoid | 6.74 | 5.28 |
The comparison also shows that dwell understates what these policies do. On Double Cart Pendulum, the no-velocity policy holds the configuration (configuration-only dwell about 9.7 s) and fails only the velocity check. On Franka it keeps the pose (pass rate 0.975) while its velocity pass rate falls from 0.609 to 0.046 between the best and final checkpoints.
Why Eq. 12 is "partial". The Franka arm has 7 joints, while the end-effector pose has 6 degrees of freedom. Fixing the hand's position and orientation therefore still leaves one free direction: the elbow can swing while the hand stays exactly still. Eq. 12 rewards only the hand, so that motion costs nothing and the reward never distinguishes a still arm from a moving one. Eq. 11 also rewards the joint posture , so all 7 joints are pinned and elbow motion lowers the reward. "Partial" means the reward constrains the hand but not the whole arm. This matches Fig. 6: Partial holds the pose, but its joint-speed pass rate is 0.088, compared with 0.759 for Full. We will use this explanation and state it as a short proposition.
Abstract and conclusion overclaim. Only PPO was tested. The sentence will read: "We show that a configuration-only reward can be sufficient for stabilization, and test this with PPO on simulated robotic tasks, where large velocity weights can cause collapse." The sufficiency theorem does not depend on the algorithm, but every empirical statement will now say PPO.
Thank you for the specific list.
Lines 216-255 and the undefined terms. This passage was too compressed. The next version will state the system as an explicit equation, and each term will either be defined at first use or replaced:
The argument in plain words: a position-only policy can push the system with forces that depend on where it is, but it cannot remove energy according to how fast it is moving. Without friction-like forces, a set of nearby states therefore keeps its volume over time, and it cannot collapse onto the target. With damping, the convergence rate is limited by the damping itself.