acceptodds
Under review as a conference paper at ICLR 2027

Fly-by-Code: Embodied Coding Agents for Aerial Manipulation with Active Visual and Physical Feedback

Abstract

Embodied coding agents emerge as a promising approach to construct robot policies from natural-language instructions. However, extending these agents to physically demanding settings introduces two fundamental challenges: partial observability and limited interaction robustness. These challenges can leave task-critical outcomes unobserved and failure causes unresolved, making execution feedback insufficient for completion assessment and program revision. To address these gaps, we introduce Active Feedback (AF), an active "View and Probe" framework. After each task execution, the agent synthesizes a separate diagnostic program to acquire informative viewpoints or perform targeted physical tests in simulation, returning evidence for completion assessment and program revision. We study AF in aerial manipulation, where payload constraints limit onboard sensing and exacerbate partial observability, while strong dynamic coupling and limited control margins make physical interactions fragile. Across six aerial manipulation tasks, AF increases final task success from 31.7% to 75% relative to terminal feedback, using the same coding-agent configuration and task-execution limit. The task programs synthesized in simulation transfer directly to a physical aerial manipulator via shared APIs without task-program source-code modifications and successfully execute multi-stage tasks in the real world. These results show that executable diagnosis helps embodied coding agents address gaps in execution feedback, improving both completion assessment and program refinement for autonomous aerial manipulation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotics, Embodied AI, Agentic AI, Aerial Manipulation, Coding Agent

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_oqzPOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an active feedback (AF) mechanism that generates a diagnostic program separately after each task execution. It obtains new observation perspectives through View, examines physical issues through Probes in the simulation, and then decides whether to terminate or modify the task code.

Critical strengths

The research problem has practical significance, and the system implementation and experimental results are also worthy of recognition.

Critical weaknesses
  1. The novelty different to existing active-probing methods need to be shown clearer. The proposed Fly-by-Code primarily differs by placing diagnosis after a complete task attempt, excluding diagnostic actions from task progress, and resetting the environment before the next attempt. The paper must verify the additional contributes beyond probing during execution or using fixed verification routines.
  2. The experiments do not directly compare to other SOTA methods. The CaP-X baseline is an adaptation that excludes visual-differencing module and parallel ensemble; CaP-X+Trace borrows ASPIRE’s execution-recording mechanism without incorporating its skill accumulation or evolutionary program search.These are not direct comparisons to complete CaP-Agent0, ASPIRE, or other existing systems.
  3. The ablations do not sufficiently separate the contributions of the diagnostic components. It does not isolate the effects of additional observations, physical probing, and the independent diagnostic structure.
  4. The hardware experiments do not establish the benefit of active diagnosis on real robots.
  5. Physical task completion and completion-assessment performance should be reported separately.
Questions for authors

None.

5 comments
AuthorsOpenReview ↗

Response to Reviewer oqzP (1/3): W1

Dear Reviewer oqzP,

Thank you for your time and valuable feedback. We have conducted additional experiments and analyses in response to your comments and address each point below.


(W1) Differences from existing active-probing methods

Fly-by-Code generates a separate post-task diagnostic program to collect new evidence for completion assessment and program revision.

1. Comparison with existing methods

Matcha [1] selects predefined sensing actions. FORCA [2] generates contact probes within task programs and revises them using recorded feedback. PhysCaP [3] generates pre-task exploration programs using physical-property measurement modules.

Method  Generates probing code    Main probing stage    Separate probing program  
Matcha [1]—Before task—
FORCA [2]✓During task—
PhysCaP [3]✓Before task✓
Fly-by-Code (Ours)✓After task✓

2. Benefits of separate post-task diagnosis

Our additional comparisons with pre-task and inline probing show that Fly-by-Code achieves higher success rates on all three evaluated tasks. We evaluated an aerial adaptation of PhysCaP for pre-task probing and an ablation of our framework for inline probing, as used in FORCA. We used the same robot APIs and task-attempt budget across these comparisons, with the environment reset between task attempts.

Task  Pre-task probing (PhysCaP style)    Inline probing (FORCA style)    Fly-by-Code  
Cabinet2/102/104/10
Align1/101/107/10
Payload2/102/1010/10
  • Benefit over inline probing

    Separating the diagnostic stage from scored task execution lets the agent prioritize informative probing over immediate task progress. In several inline-probing trials, the agent chose conservative plans while physical constraints remained unknown. In contrast, AF can try informative tests even when the diagnostic actions may fail and use the results to guide more effective program revisions.

  • Benefit over pre-task probing

    Post-task diagnosis focuses probing on information needed for task success and program revision. Pre-task exploration must anticipate useful information before execution, so it may probe unnecessarily broadly or target the wrong information. In contrast, AF uses observations and measurements from the current attempt to identify unverified outcomes and form failure hypotheses. Its diagnostic program then starts from the resulting state and focuses directly on these unresolved questions, without replaying the whole task.

3. Benefits over fixed verification routines

AF generates diagnostic tests in response to observed failures, whereas fixed routines rely on predefined checks that may not provide the evidence needed to diagnose an unexpected failure.

In Align, the robot must push two boxes back so they line up with the other boxes on the shelf. Repeated pushes caused one box to rotate. A routine that only checks final box positions could detect the misalignment, but would not reveal which contact strategy would correct it. The agent generated six tests comparing contact points and sideways slides, measuring how each affected translation and rotation. The revised program used these measurements to align both boxes. A fixed routine would need such tests built in beforehand, whereas AF generated them in response to the observed failure. This reduces the need to anticipate and hand-code task- and failure-specific checks.

[1] Zhao et al. Chat with the Environment: Interactive Multimodal Perception Using Large Language Models. IROS, 2023.

[2] Tan et al. FORCA: A Force-Aware Coding Agent for Contact-Rich Robot Manipulation. arXiv, 2026.

[3] Lin et al. PhysCaP: Grounding Code-as-Policy Agent with Physics-Informed Exploration. arXiv, 2026.

AuthorsOpenReview ↗

Response to Reviewer oqzP (2/3): W2–W3

(W2) Direct comparisons with CaP-Agent0 and ASPIRE

Our additional comparisons show that Fly-by-Code achieves higher overall success at lower estimated LLM cost per success than CaP-Agent0 and ASPIRE on the evaluated tasks.

Our original evaluation used CaP-X and CaP-X+Trace to isolate the effect of the feedback mechanism under a controlled experimental setup. We additionally evaluated the released implementations of CaP-Agent0 and ASPIRE for our aerial manipulation environment, retaining CaP-Agent0’s visual-differencing module (VDM) and multi-model ensemble and ASPIRE’s shared, expanding skill library. Each method was evaluated on three tasks, with ten trials per task, using the same robot APIs and task prompts.

Task  CaP-Agent0    ASPIRE    Fly-by-Code  
Cabinet2/10 ($449.8)5/10 ($67.0)4/10 ($61.5)
2-Objects pick-and-place5/10 ($59.2)8/10 ($65.0)9/10 ($11.3)
Payload3/10 ($347.4)2/10 ($371.0)10/10 ($16.5)
Total successes10/3015/3023/30

Costs in parentheses are estimated total LLM costs across all ten trials divided by the number of successes, in USD.

These results highlight the need to actively acquire missing outcome evidence and physical information in ego-view-only manipulation.

  • Limitations of visual differencing and ensembles (CaP-Agent0):

    VDM cannot detect changes outside the recorded egocentric views, and false completion declarations remained on the pick-and-place task. Ensembling adds code-generation cost but still relies on task-execution feedback. CaP-Agent0 achieved lower success on all three tasks at higher estimated LLM cost per success.

  • Limitations of skill accumulation (ASPIRE):

    Even with its shared skill library, ASPIRE achieved 2/10 successes on Payload, where transport planning requires determining which loads the robot can lift. Building its shared skill library also requires additional time and LLM cost; Fly-by-Code requires no separate learning stage.

  • The advantage of active diagnosis:

    AF generates targeted physical tests, such as lifting proposed loads, and moves to additional viewpoints to verify outcomes. This provides new evidence for completion assessment and program revision. Fly-by-Code combines the highest overall success with strong cost-efficiency, using as little as 1/21 of CaP-Agent0’s estimated LLM cost per success.

We will document the model configurations, aerial adaptations, and cost accounting in the appendix.


(W3) Contributions of additional observations (View), physical probing (Probe), and the independent diagnostic structure

Our new ablations support both the synergy of View and Probe and the benefit of our independent diagnostic structure.

W3-1 : Contributions and Synergy of View and Probe

We evaluated View-only and Probe-only configurations to isolate each component’s contribution. During AF, View-only disables physical tests, while Probe-only withholds AF images from the next decision.

Task  CaP-X    View-only    Probe-only    Fly-by-Code  
Payload3/105/104/1010/10
Cabinet1/101/101/104/10
Align0/102/102/107/10
2-Objects Pick-and-place4/107/108/109/10
Total Success8/40 (20.0%)15/40 (37.5%)15/40 (37.5%)30/40 (75.0%)
  • View-only (Lacks physical testing):

    Fails to safely measure physical constraints (e.g., payload limits) during the unscored AF stage, forcing exploratory actions into the scored execution.

  • Probe-only (Lacks visual confirmation):

    Executes physical tests but leaves outcomes only as text logs plus one terminal image, so it cannot visually check what a test actually did, and this leads to lower success.

  • The Synergy (Full AF):

    Probe safely acquires missing physical information outside the scored execution, while View visually confirms the physical outcomes. Both are required for effective refinement. This synergy shows in the results: View-only and Probe-only each add only +7 successes over CaP-X, whereas full AF adds +22, far more than the two gains combined (+14).

W3-2 : Contribution of the independent diagnostic structure

In Fly-by-Code (Ours), the diagnostic program runs as a separate, unscored stage after the task attempt, so it can test without having to complete the task.

To explicitly isolate the contribution of this structure, we ran an inline probing ablation that places observation and testing inside the task program, with all other settings identical to Fly-by-Code. Across the three tasks (Payload, Cabinet, Align), inline probing achieved 5/30 successes, compared with 21/30 for Fly-by-Code. Further analysis is provided in our response to W1.

AuthorsOpenReview ↗

Response to Reviewer oqzP (3/3): W4–W5

(W4) Active Feedback on the physical robot

In our new hardware experiment, Active Feedback corrected a false completion assessment and guided three real-world feedback turns, yielding a program that achieved both goals of a 2-Objects pick-and-place task.

The original hardware experiments demonstrated zero-shot deployment of programs refined in simulation. In this additional experiment, we used diagnostic evidence collected on the real aerial manipulator to revise the task program after physical execution failures.

The task was to place a hammer in a hanging basket and a wrench on the upper cabinet shelf. We started from a CaP-X program that had succeeded in simulation. During its initial hardware execution, the hammer struck the basket handle and fell outside. CaP-X incorrectly returned FINISH based on the execution log and terminal image.

In Fly-by-Code, the agent generated a post-task diagnostic program to check whether the hammer was inside the basket. The real robot executed this program, flying to new viewpoints and collecting wrist-camera images that revealed the empty basket and its handle geometry. This additional evidence prompted REGENERATE and task-program revision. These real-world diagnostic flights required no human intervention and respected the collision-avoidance constraints in the low-level robot APIs.

Each of the three feedback turns followed diagnostic flight → completion assessment and program revision → hardware re-execution:

Turn Diagnostic evidenceCode revisionHardware execution result
1Empty basket and handle geometryAdjust hammer release pose and locationHammer placed correctly, but wrench misplaced on lower shelf
2Hammer placement verifiedPreserve hammer code and revise shelf-selection codeWrench still misplaced on lower shelf
3Full-cabinet view from farther away, revealing roof and shelf heightsRevise shelf-selection algorithmBoth goals achieved

This experiment demonstrates two benefits of active diagnosis on the physical robot:

  • Correcting completion errors:

    AF acquired outcome evidence missing from the execution record and replaced an incorrect FINISH with REGENERATE. This enabled further refinement instead of terminating with an incomplete task.

  • Evidence-guided program refinement:

    AF did more than detect failure. Its observations helped preserve successful hammer-placement code and provided the shelf information needed to correct the remaining placement error. The final program achieved both goals, demonstrating how on-robot diagnosis can improve the real-world performance of a simulation-refined program.

We will document the diagnostic observations, code revisions, and hardware execution results in the appendix.


(W5) Separating physical completion from completion assessment

Separate reporting shows that AF improves both physical task completion (75.0%) and completion-assessment precision (90.0%).

Physical task completion is evaluated from the final task-execution state before AF, independently of the agent’s completion declaration. Completion-assessment precision measures the fraction of FINISH declarations that correspond to actual task completion.

Each method was evaluated over 60 trials. “Iteration limit reached” denotes trials that ended after the maximum number of task attempts without declaring FINISH.

Method   Physical task completionFINISH declarationsFalse FINISHIteration limit reachedCompletion-assessment precision
CaP-X19/60 (31.7%)44251643.2%
CaP-X+Trace29/60 (48.3%)48191260.4%
Fly-by-Code45/60 (75.0%)5051090.0%

These results demonstrate two distinct benefits of AF:

  • Higher physical task completion:

    AF achieved 75.0% physical completion, compared with 31.7% for CaP-X and 48.3% for CaP-X+Trace. Its benefits extend to actually completing tasks, not only assessing their outcomes.

  • More reliable completion assessment:

    False FINISH declarations decreased from 25 for CaP-X and 19 for CaP-X+Trace to 5 for AF. Its 90.0% completion-assessment precision reflects fewer incorrect terminations with an incomplete task.

Across these 180 trials, all physically completed final task states were accompanied by FINISH, so physical completion rates match the success rates reported in the paper. We will revise the evaluation description and table captions to clearly define and separately report physical task completion and completion-assessment precision.

Reviewer_pfRPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  1. A code-as-policy design for UAV manipulation tasks.
  2. Active feedback mechanism for results verification and failure recovery.
Critical strengths
  1. A code-as-policy design for UAV manipulation tasks.
  2. Active feedback mechanism for results verification and failure recovery.
Critical weaknesses
  1. The proposed framework largely builds upon the existing Code-as-Policy paradigm and extends it to aerial manipulation. The introduction of Active Feedback (AF) is a useful addition that distinguishes the approach from manipulation methods without explicit feedback. However, the authors could further clarify the methodological novelty of AF and highlight its contributions beyond existing code-based robotic frameworks.
  2. The experimental evaluation is relatively limited, with only 60 trials across six tasks. A broader evaluation involving more diverse tasks, environments, and manipulation scenarios would help better demonstrate the robustness and generalizability of the proposed approach.
  3. The current results mainly demonstrate the effectiveness of the overall AF framework. Additional ablation studies comparing View-only, Probe-only, and their combination would help clarify the individual contributions of these components and provide a better understanding of how they complement each other.
Questions for authors
  1. What are the inference latency and end-to-end execution time?
3 comments
AuthorsOpenReview ↗

Response to Reviewer pfRP (1/2): W1-W2

Dear Reviewer pfRP,

Thank you for your valuable and insightful comments. We have carefully considered your feedback and provided our responses to each point below.


(W1) Methodological contribution of Active Feedback

Our methodological contribution is separate, post-task diagnosis for completion assessment and program revision.

Aerial manipulation can span large workspaces, so we cannot assume access to an external camera overlooking the entire environment. In our ego-view-only setting, outcomes may leave view as the robot moves, while contact forces can disturb the flying base. Execution feedback can therefore leave both outcomes and physical failure causes unresolved.

In Fly-by-Code, the agent investigates these uncertainties without having to complete the task during diagnosis. After a task attempt, it uses the execution evidence to generate a diagnostic program. To assess these design choices, we conducted additional comparisons on the Cabinet, Align, and Payload tasks, with ten trials per task and the same robot APIs, task-attempt budget, and between-attempt resets.

  • Separate diagnosis: Task performance is scored before AF, so diagnostic actions do not change that attempt’s recorded score. To test this separation, our new inline-probing ablation placed observation and testing inside the task program. It achieved 5/30 successes, compared with 21/30 for Fly-by-Code. This separation removes the need to recover from a test and complete the task within the same execution. In our inline-probing analysis, concerns about recovery discouraged informative tests.
  • Post-task diagnosis: AF uses the post-execution state to check outcomes and test hypotheses about observed failures. Its findings guide FINISH / REGENERATE and program revision. To compare with pre-task exploration, we evaluated an aerial implementation of PhysCaP [1], which generates exploration programs before task execution. It achieved 5/30 successes, compared with 21/30 for Fly-by-Code. Execution evidence helps focus further probing on actual difficulties, whereas pre-task exploration may pursue cues that turn out to be uninformative for the task.

We will revise the introduction and method description to clarify the contributions of separate, post-task diagnosis. 

[1] Lin et al. PhysCaP: Grounding Code-as-Policy Agent with Physics-Informed Exploration. arXiv, 2026.


(W2) Generalizability Across Diverse Tasks and Embodiments (Humanoid)

To explicitly demonstrate the broad generalizability and robustness of Active Feedback (AF), we expanded our evaluation to two complex, long-horizon household tasks (Chopping Wood, Hanging Pictures) from the BEHAVIOR-1K benchmark, since, to our knowledge, there is no established benchmark or simulation suite for long-horizon aerial manipulation. We also transferred our framework to a different embodiment (R1Pro Humanoid) under a head-camera-only setup, using the same AF framework as in our aerial manipulation experiments.

As shown in the table below (60 additional trials), AF outperforms both baselines in these new environments and removes false FINISH on Chopping Wood (3 and 2 → 0):

Task  Metric    CaP-X    CaP-X + Trace    Ours (AF)  
Chopping Wood  Success  6/107/109/10
False FINISH  320
Hanging Pictures  Success  0/100/103/10
False FINISH  000

These new scenarios highlight how AF adapts to different embodiment constraints:

  • Chopping Wood (Resolving Visual Ambiguity): Baselines declared false completions more often than Ours (3 and 2 of 10 trials vs. 0) when unchopped pieces fell outside the humanoid's head-camera view. AF navigated to acquire missing viewpoints, reducing false declarations to zero (0 false FINISH).

  • Hanging Pictures (Physical Hypothesis Testing): AF generated diagnostic programs to physically test candidate hanging positions and angles in the unscored stage, informing subsequent program revisions and enabling successes (3/10) where baselines failed (0/10).

These results support the AF stage’s utility for completion assessment and task-program refinement across a broader range of tasks, environments, and manipulation scenarios, extending even to a different robot embodiment. We will include these results and experimental details in the paper.

AuthorsOpenReview ↗

Response to Reviewer pfRP (2/2): W3, Q1

(W3) Individual and complementary contributions of View and Probe

To explicitly isolate the contributions of visual observations and physical probing, we evaluated View-only and Probe-only configurations. As shown in the table below, neither component alone is sufficient; their combination shows a clear synergy, adding +22 successes over the baseline.

  • View-only: AF permits observation but no physical tests. Newly captured images are passed to the next decision.

  • Probe-only: AF permits physical interactions and measurements, but images captured during AF are not passed to the next decision.

Task    CaP-X    View-only    Probe-only    Fly-by-Code  
Payload  3/105/104/1010/10
Cabinet  1/101/101/104/10
Align  0/102/102/107/10
2-Objects Pick-and-place  4/107/108/109/10
Total Success    8/40 (20.0%)    15/40 (37.5%)    15/40 (37.5%)    30/40 (75.0%)  
  • View-only (Lacks physical testing): Fails to safely measure physical constraints (e.g., payload limits) during the unscored AF stage, forcing exploratory actions into the scored execution.

  • Probe-only (Lacks visual confirmation): Executes physical tests but leaves outcomes only as text logs plus one terminal image, so it cannot visually check what a test actually did, and this leads to lower success.

  • The Synergy (Full AF): Probe safely acquires missing physical information outside the scored execution, while View visually confirms the physical outcomes. Both are required for effective refinement. This synergy shows in the results: View-only and Probe-only each add only +7 successes over CaP-X, whereas full AF adds +22, far more than the two gains combined (+14).


(Q1) Inference latency and end-to-end execution time

AF (Active Feedback, Ours) achieves the lowest total wall-clock time per success in our evaluation, despite the additional time spent on diagnosis.

We re-aggregated timing measurements from 180 main evaluation trials and 1,760 LLM calls. Each entry below reports the median / mean time in seconds, including extended thinking and program generation where applicable.

Method    Initial generation    AF request    Decision: FINISH    REGENERATE decision + program revision  
CaP-X  214 / 238—22 / 28190 / 212
CaP-X+Trace  168 / 197—22 / 28182 / 206
Fly-by-Code  162 / 202109 / 11112 / 17165 / 192

LLM inference occurs between program executions, outside the real-time flight-control loop. Its latency therefore does not delay ongoing flight-control commands.

The following table reports mean wall-clock times per turn in simulation. LLM time share is calculated using the end-to-end duration of each turn.

Method   LLM inference (s)  Task + AF execution (s)  End-to-end time per turn (s)  LLM time share  Wall-clock time per success (h) 
CaP-X2261,3081,55514.5%9.8
CaP-X+Trace  2141,2791,53513.9%6.2
Fly-by-Code3121,642 (1,275 + 367)1,99715.6%4.5

These measurements highlight two aspects of execution efficiency:

  • Faster decision-making and program revision: AF showed lower median and mean times for both completion assessment (FINISH) and regeneration decisions with program revision (REGENERATE). In particular, median FINISH time fell from 22 seconds to 12 seconds, a reduction of approximately 45%. These shorter assessment and revision times are consistent with diagnostic evidence reducing ambiguity and the model’s reasoning burden.

  • A small LLM time share: LLM inference accounts for only 15.6% of AF’s end-to-end time per turn. Most elapsed time is spent executing robot programs in simulation. More powerful computing resources can reduce this execution time and the overall wall-clock time.

AF’s higher success rate translates into better end-to-end time efficiency per successful outcome.

Dear Reviewer pfRP,

Thank you for your careful review and valuable suggestions.

We have added videos videos and explanations of the BEHAVIOR-1K experiments described in our response to W2 to our anonymous website (https://fbycode.github.io/flybycode/#behavior-1k).

We have also added a video and explanations of a new hardware experiment evaluating Active Feedback for real-world task-program refinement (https://fbycode.github.io/flybycode/#real-world-feedback).

As the discussion period draws to a close, please let us know if any further clarification would be helpful.

If our responses and new results address your concerns, we would be grateful if you would consider revisiting your assessment.

Reviewer_LU8xOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Fly by Code, which introduces Active Feedback into an embodied coding agent. The framework enables an aerial manipulator to generate an independent diagnostic program after task execution. It uses View to acquire additional visual evidence or Probe to test unknown physical properties. The framework then uses this evidence for completion assessment and program refinement. The authors evaluate the method on six simulated tasks and a real aerial manipulator.

Critical strengths
  1. The authors separate task execution from diagnostic evidence acquisition. The framework uses View and Probe to actively address missing visual evidence and unknown physical constraints.

  2. The experimental evaluation is relatively comprehensive. It covers six aerial manipulation tasks, completion assessment, different model and interface configurations, No Harness analysis, semantic map ablation, and resource analysis.

Critical weaknesses
  1. Active Feedback mainly relies on the LLM to decide when and how to execute View or Probe based on prompts. The method lacks explicit Uncertainty Estimation, Information Gain, or a learned diagnostic policy. Therefore, the diagnostic strategy remains largely heuristic.

  2. View and Probe are two core components of the proposed method. However, the paper lacks direct component ablations, such as View only and Probe only configurations.

  3. The authors should include component ablations for View and Probe. They should also validate Active View or safety constrained diagnostic interactions on real hardware. This evaluation helps establish the effectiveness of the core Active Feedback mechanism in real deployments.

Questions for authors

Please refer to the Critical Weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer LU8x (1/2): W1–W2

Dear Reviewer LU8x,

We sincerely appreciate your time and constructive feedback. We have conducted additional experiments and analyses to clarify your comments, as detailed below.


(W1) Analysis of LLM-generated diagnostics

While Active Feedback (AF) does not require a separately trained diagnostic policy or explicit mathematical uncertainty estimation, our VLM-based formulation acquires useful diagnostic evidence in 80% of executions.

By leveraging the model’s reasoning, AF targets unresolved questions and synthesizes diagnostic code to acquire the missing evidence needed for completion assessment and program revision.

To empirically validate the practical utility of this LLM-guided strategy, we comprehensively analyzed all 326 diagnostic-program executions across 60 evaluation trials. Even without an explicit information-gain objective, the generated diagnostic programs returned actionable evidence in most executions:

Diagnostic utility metricRate
Acquired sufficient evidence to answer at least one stated diagnostic question80% (260/326)
Evidence was clearly used in a subsequent decision or task program79% (259/326)

Partial use of evidence was not counted as clear use in the second metric.

This high rate of actionable information gathering supports Fly-by-Code’s strong task-level performance. Fly-by-Code achieves a 75.0% overall success rate across the six original tasks, compared with 31.7% and 48.3% for terminal and trace feedback baselines. Furthermore, additional experiments conducted during the rebuttal show that these gains extend to a different robot embodiment (the R1Pro humanoid robot) on the BEHAVIOR-1K tasks. Our method achieved 60% overall success versus 30–35% for the baselines. Qualitative examples of how diagnostic evidence informs completion assessment and program revision are provided in Appendices D.1 and D.2, respectively.


(W2) Component ablations of View and Probe

To explicitly isolate the contributions of View and Probe, we evaluated View-only and Probe-only configurations. As shown in the table below, neither component alone is sufficient; their combination shows a clear synergy, adding +22 successes over the baseline.

  • View-only: AF permits observation but no physical tests. Newly captured images are passed to the next decision.
  • Probe-only: AF permits physical interactions and measurements, but images captured during AF are not passed to the next decision.
TaskCaP-XView-onlyProbe-onlyFly-by-Code
Payload3/105/104/1010/10
Cabinet1/101/101/104/10
Align0/102/102/107/10
2-Objects Pick-and-place4/107/108/109/10
Total Success8/40 (20.0%)15/40 (37.5%)15/40 (37.5%)30/40 (75.0%)
  • View-only (Lacks physical testing): Fails to safely measure physical constraints (e.g., payload limits) during the unscored AF stage, forcing exploratory actions into the scored execution.
  • Probe-only (Lacks visual confirmation): Executes physical tests but leaves outcomes only as text logs plus one terminal image, so it cannot visually check what a test actually did, and this leads to lower success.
  • The Synergy (Full AF): Probe safely acquires missing physical information outside the scored execution, while View visually confirms the physical outcomes. This synergy shows in the results: View-only and Probe-only each add only +7 successes over CaP-X, whereas full AF adds +22, far more than the two gains combined (+14).
AuthorsOpenReview ↗

Response to Reviewer LU8x (2/2): W3

(W3) Active Feedback on real hardware

On our real aerial manipulator, Active View collected useful evidence within collision-avoidance constraints and guided program refinement that achieved both goals of a 2-Objects pick-and-place task.

For the View/Probe component ablations, please see our response to W2.

The task required placing a hammer in a hanging basket and a wrench on the upper cabinet shelf. We started from a CaP-X program that had succeeded in simulation. On hardware, the hammer struck the basket handle and fell outside, but CaP-X incorrectly returned FINISH based on the execution log and terminal image.

The robot then executed an agent-generated diagnostic program during Active Feedback (AF), moving to new viewpoints and capturing wrist-camera images. These revealed the empty basket and its handle geometry, prompting REGENERATE and program revision.

Across three real-world feedback turns, the loop followed diagnostic flight → completion assessment and program revision → hardware re-execution:

TurnDiagnostic evidenceCode revisionHardware execution result
1Empty basket and handle geometryAdjust hammer release pose and locationHammer placed correctly, but wrench misplaced on lower shelf
2Hammer placement verifiedPreserve hammer code and revise shelf-selection codeWrench still misplaced on lower shelf
3Full-cabinet view from farther away, revealing roof and shelf heightsRevise shelf-selection algorithmBoth goals achieved

These results demonstrate two practical benefits of Active View on real hardware:

  • Correcting completion errors: Additional viewpoints revealed a failure that the execution log and terminal image had not resolved, preventing premature termination.
  • Program refinement through safety-constrained diagnosis: Simulation and hardware share a collision-avoidance planner through the low-level robot APIs, which reject motions that violate obstacle-avoidance constraints. The real-world diagnostic flights collected useful evidence within these constraints without human intervention, guiding program revisions that achieved both task goals.

We will document the diagnostic observations, code revisions, and hardware execution results in the appendix.

Thank you to the authors for the thoughtful response and additional experiments. I will keep my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.