- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper shows why standard Eikonal-based physics-informed GCRL fails in robotic manipulation and establishes that physics-informed value learning must account for which state dimensions the robot can actually control at each interaction stage, rather than imposing uniform geometric constraints across the entire state space.
Critical strengths
- This paper provides a formal analysis explaining why full-state Eikonal regularization can conflict with locally feasible dynamics.
- They demonstrate encouraging improvements over baselines and has real world validation on physical UR5e robot.
Critical weaknesses
- The algorithmic novelty is limited as the transition-based HJB residual was previously proposed, while the hierarchical approach combines existing flow-based subgoal generation and value-learning techniques.
- Figure 4 shows that excluding object poses from the low-level state improves even non-physics-informed hierarchical methods. Then how much of the performance improvement comes specifically from the proposed formulation?
- Physical experiments involve only one object and a relatively simple pick-and-place task, with 20 evaluation episodes.
Questions for authors
- Since Equation 4 uses observed transitions rather than optimizing over feasible actions, under what conditions is the resulting regularizer consistent with the optimal HJB equation? How sensitive is performance to suboptimal offline trajectories?
- Could the authors provide uncertainty estimates for the physical evaluation and discuss whether the improvements remain consistent across different object configurations?
2 comments
W2: hierarchical methods
Figure 4 shows that excluding object poses from the low-level state improves even non-physics-informed hierarchical methods. Then how much of the performance improvement comes specifically from the proposed formulation?
We thank the reviewer for highlighting this observation. We agree and already acknowledge (lines 451-454) that the controllability-aware decomposition improves both physics-informed and unregularized algorithms. This is itself an interesting finding, demonstrating that our theoretically motivated decomposition benefits GCRL beyond physics-informed regularization.
We also want to acknowledge that, in Figure 4, H-Flow-GCIVL and Pi-H-Flow-HJB-GCIVL achieve comparable performance, with a marginal difference in mean peak success rate (+2% percentage points) and very similar learning-curve performance. Thus, the additional benefit of HJB regularization is limited in this setting.
However, two other results demonstrate its effectiveness:
-
Figure 3: Without hierarchy, HJB-GCIVL outperforms GCIVL, particularly in the challenging Triple and Quadruple environments.
-
Real-world experiments (Section 6): Pi-H-Flow-HJB-GCIVL achieves higher success rates than H-Flow-GCIVL across all three evaluation thresholds.
These findings suggest that controllability-aware decomposition and HJB regularization offer complementary, but not necessarily additive, benefits. We have clarified this distinction in the revised manuscript (lines 455-461, highlighted in blue).
W3, Q2: real-world experiments
Physical experiments involve only one object and a relatively simple pick-and-place task, with 20 evaluation episodes.
Could the authors provide uncertainty estimates for the physical evaluation and discuss whether the improvements remain consistent across different object configurations?
We thank the reviewer for this question. We would like to emphasize that our real-world experiments already involve randomized initial object poses (lines 484-485), and we did not observe systematic performance differences across these configurations. Therefore, the reported success rates (lines 502-509) reflect evaluations across different initial configurations rather than a single fixed pick-and-place scenario.
Regarding statistical uncertainty, we acknowledge that 20 evaluation episodes per method provide limited ability to establish statistically significant performance differences at conventional thresholds (e.g., with a p-value less than 0.05). Nevertheless, we find it encouraging that Pi-H-Flow-HJB-GCIVL consistently achieves higher observed success rates across all three evaluation thresholds in real-world experiments, suggesting that the benefits observed in simulation can also translate to physical manipulation.
Finally, while we were unable to extend the physical experiments to multi-object rearrangement tasks, our simulated multi-object experiments (Fig. 3) suggest that similar benefits may extend to these settings.
We have revised the discussion of our real-world results accordingly (lines 508-512).
We thank the reviewer for the thorough review and valuable feedback, which provided an opportunity to improve our work. We have carefully addressed your concerns and revised the manuscript accordingly, with changes highlighted in blue. We have also conducted additional experiments to further investigate the robustness of our proposed regularizer. We hope that our responses and revisions satisfactorily address your concerns. Thank you!
W1: Algorithmic novelties
We acknowledge that our formulations build on existing components, as explicitly discussed in the manuscript (lines 285-290 and 304-307, 322–323). However, our main contribution is to explain why existing physics-informed regularization methods struggle in manipulation due to state-dependent controllability, and to derive principled solutions that overcome these limitations.
Two important distinctions from prior work are:
Giammarino et al. (2026): This work primarily focused on navigation and introduced the HJB-based regularizer as an underperforming method compared to the Eikonal regularization. In contrast, our work shows that this relationship can be reversed in robotic manipulation and explains why this occurs. Specifically, Theorems 4.1 and 4.2 establish when and why Eikonal regularization becomes inconsistent with the local dynamics and when it remains a reasonable approximation. This provides new insights into the design and application of physics-informed GCRL algorithms beyond navigation.
SHARSA (Park et al., 2026): While we adopt flow-based subgoal generation and critic-based selection, our specific hierarchical decomposition is motivated by our theoretical analysis. This leads to separating object-centric high-level reasoning from robot-centric low-level control, allowing physics-informed regularization to operate on a representation more closely aligned with the robot's controllable dynamics.
These distinctions, together with the components shared with prior work, are highlighted in Section 5 (Lines 281-284 and 293-309).
Q1 : HJB regularizer consistency and sensitivity to suboptimal trajectories
We thank the reviewer for raising this point. The empirical HJB residual approximates the optimal HJB condition when observed transitions are sufficiently close to optimal goal-reaching directions. Indeed, the HJB equality holds exactly along optimal directions, while suboptimal transitions may introduce additional approximation errors.
As highlighted in lines 273-277, the HJB regularizer is computed using future goals sampled along the same trajectory as the observed transition. Furthermore, the OGBench Play datasets, used for the experiments in Fig. 3, are collected using expert policies, providing goal-directed trajectories that make the approximation reasonable for many local transitions [1].
[1] Park S, Frans K, Eysenbach B, Levine S. Ogbench: Benchmarking offline goal-conditioned rl. In International Conference on Learning Representations 2025 May 1 (Vol. 2025, pp. 94937-94982).
To answer this question, we conducted additional experiments comparing GCIVL and HJB-GCIVL on the OGBench Noisy manipulation datasets, which contain substantially more action noise and suboptimal trajectories than the Play datasets. These new experiments are summarized in Fig. 5 and lines 461–480 (both highlighted in blue in the revised manuscript).
The results show that the benefits of HJB regularization can persist under noisy trajectories, although its effectiveness varies across environments. In Triple, HJB-GCIVL loses its advantage over GCIVL, with both methods achieving comparable performance. On the other hand, in the more challenging Quadruple environment, HJB-GCIVL outperforms GCIVL (approximately 45% versus 20% final success rate). These findings suggest that while HJB regularization is sensitive to data quality, suboptimal trajectories do not undermine its benefits, highlighting opportunities for further improvements.