Fly-by-Code: Embodied Coding Agents for Aerial Manipulation with Active Visual and Physical Feedback
Abstract
Embodied coding agents emerge as a promising approach to construct robot policies from natural-language instructions. However, extending these agents to physically demanding settings introduces two fundamental challenges: partial observability and limited interaction robustness. These challenges can leave task-critical outcomes unobserved and failure causes unresolved, making execution feedback insufficient for completion assessment and program revision. To address these gaps, we introduce Active Feedback (AF), an active "View and Probe" framework. After each task execution, the agent synthesizes a separate diagnostic program to acquire informative viewpoints or perform targeted physical tests in simulation, returning evidence for completion assessment and program revision. We study AF in aerial manipulation, where payload constraints limit onboard sensing and exacerbate partial observability, while strong dynamic coupling and limited control margins make physical interactions fragile. Across six aerial manipulation tasks, AF increases final task success from 31.7% to 75% relative to terminal feedback, using the same coding-agent configuration and task-execution limit. The task programs synthesized in simulation transfer directly to a physical aerial manipulator via shared APIs without task-program source-code modifications and successfully execute multi-stage tasks in the real world. These results show that executable diagnosis helps embodied coding agents address gaps in execution feedback, improving both completion assessment and program refinement for autonomous aerial manipulation.
Then back it, or bet against it.
Related papers
Open the market on this paper to see 7 more related papers.