When Interaction Changes Controllability: Physics-informed Goal-Conditioned Reinforcement Learning for Manipulation

Learning to reach arbitrary goals requires agents to infer a rich notion of reachability across state-goal pairs, which becomes increasingly difficult over long decision horizons and in systems with complex dynamics. Physics-informed Goal-Conditioned Reinforcement Learning (Pi-GCRL) addresses this problem primarily via Eikonal-based losses, achieving strong results in navigation while remaining less effective in robotic manipulation. In this paper, we show that this gap stems from the state-dependent controllability of manipulation, where object coordinates are not locally controllable throughout the task. We first establish that full-state Eikonal regularization becomes inconsistent whenever goal-relevant coordinates are locally uncontrollable. This insight leads to two complementary approaches: a loss that regularizes value gradients only along feasible directions, and a hierarchical decomposition that restricts physics-informed learning to a controllable low-level space. We evaluate both approaches on offline manipulation benchmarks and a physical robot, showing substantial improvements over naive full-state Eikonal regularization and demonstrating that accounting for local controllability improves the effectiveness of Pi-GCRL in robotic manipulation.