- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes an active feedback (AF) mechanism that generates a diagnostic program separately after each task execution. It obtains new observation perspectives through View, examines physical issues through Probes in the simulation, and then decides whether to terminate or modify the task code.
Critical strengths
The research problem has practical significance, and the system implementation and experimental results are also worthy of recognition.
Critical weaknesses
- The novelty different to existing active-probing methods need to be shown clearer. The proposed Fly-by-Code primarily differs by placing diagnosis after a complete task attempt, excluding diagnostic actions from task progress, and resetting the environment before the next attempt. The paper must verify the additional contributes beyond probing during execution or using fixed verification routines.
- The experiments do not directly compare to other SOTA methods. The CaP-X baseline is an adaptation that excludes visual-differencing module and parallel ensemble; CaP-X+Trace borrows ASPIRE’s execution-recording mechanism without incorporating its skill accumulation or evolutionary program search.These are not direct comparisons to complete CaP-Agent0, ASPIRE, or other existing systems.
- The ablations do not sufficiently separate the contributions of the diagnostic components. It does not isolate the effects of additional observations, physical probing, and the independent diagnostic structure.
- The hardware experiments do not establish the benefit of active diagnosis on real robots.
- Physical task completion and completion-assessment performance should be reported separately.
Questions for authors
None.
5 comments
Response to Reviewer oqzP (2/3): W2–W3
(W2) Direct comparisons with CaP-Agent0 and ASPIRE
Our additional comparisons show that Fly-by-Code achieves higher overall success at lower estimated LLM cost per success than CaP-Agent0 and ASPIRE on the evaluated tasks.
Our original evaluation used CaP-X and CaP-X+Trace to isolate the effect of the feedback mechanism under a controlled experimental setup. We additionally evaluated the released implementations of CaP-Agent0 and ASPIRE for our aerial manipulation environment, retaining CaP-Agent0’s visual-differencing module (VDM) and multi-model ensemble and ASPIRE’s shared, expanding skill library. Each method was evaluated on three tasks, with ten trials per task, using the same robot APIs and task prompts.
| Task | CaP-Agent0 | ASPIRE | Fly-by-Code |
|---|---|---|---|
| Cabinet | 2/10 ($449.8) | 5/10 ($67.0) | 4/10 ($61.5) |
| 2-Objects pick-and-place | 5/10 ($59.2) | 8/10 ($65.0) | 9/10 ($11.3) |
| Payload | 3/10 ($347.4) | 2/10 ($371.0) | 10/10 ($16.5) |
| Total successes | 10/30 | 15/30 | 23/30 |
Costs in parentheses are estimated total LLM costs across all ten trials divided by the number of successes, in USD.
These results highlight the need to actively acquire missing outcome evidence and physical information in ego-view-only manipulation.
-
Limitations of visual differencing and ensembles (CaP-Agent0):
VDM cannot detect changes outside the recorded egocentric views, and false completion declarations remained on the pick-and-place task. Ensembling adds code-generation cost but still relies on task-execution feedback. CaP-Agent0 achieved lower success on all three tasks at higher estimated LLM cost per success.
-
Limitations of skill accumulation (ASPIRE):
Even with its shared skill library, ASPIRE achieved 2/10 successes on Payload, where transport planning requires determining which loads the robot can lift. Building its shared skill library also requires additional time and LLM cost; Fly-by-Code requires no separate learning stage.
-
The advantage of active diagnosis:
AF generates targeted physical tests, such as lifting proposed loads, and moves to additional viewpoints to verify outcomes. This provides new evidence for completion assessment and program revision. Fly-by-Code combines the highest overall success with strong cost-efficiency, using as little as 1/21 of CaP-Agent0’s estimated LLM cost per success.
We will document the model configurations, aerial adaptations, and cost accounting in the appendix.
(W3) Contributions of additional observations (View), physical probing (Probe), and the independent diagnostic structure
Our new ablations support both the synergy of View and Probe and the benefit of our independent diagnostic structure.
W3-1 : Contributions and Synergy of View and Probe
We evaluated View-only and Probe-only configurations to isolate each component’s contribution. During AF, View-only disables physical tests, while Probe-only withholds AF images from the next decision.
| Task | CaP-X | View-only | Probe-only | Fly-by-Code |
|---|---|---|---|---|
| Payload | 3/10 | 5/10 | 4/10 | 10/10 |
| Cabinet | 1/10 | 1/10 | 1/10 | 4/10 |
| Align | 0/10 | 2/10 | 2/10 | 7/10 |
| 2-Objects Pick-and-place | 4/10 | 7/10 | 8/10 | 9/10 |
| Total Success | 8/40 (20.0%) | 15/40 (37.5%) | 15/40 (37.5%) | 30/40 (75.0%) |
-
View-only (Lacks physical testing):
Fails to safely measure physical constraints (e.g., payload limits) during the unscored AF stage, forcing exploratory actions into the scored execution.
-
Probe-only (Lacks visual confirmation):
Executes physical tests but leaves outcomes only as text logs plus one terminal image, so it cannot visually check what a test actually did, and this leads to lower success.
-
The Synergy (Full AF):
Probe safely acquires missing physical information outside the scored execution, while View visually confirms the physical outcomes. Both are required for effective refinement. This synergy shows in the results: View-only and Probe-only each add only +7 successes over CaP-X, whereas full AF adds +22, far more than the two gains combined (+14).
W3-2 : Contribution of the independent diagnostic structure
In Fly-by-Code (Ours), the diagnostic program runs as a separate, unscored stage after the task attempt, so it can test without having to complete the task.
To explicitly isolate the contribution of this structure, we ran an inline probing ablation that places observation and testing inside the task program, with all other settings identical to Fly-by-Code. Across the three tasks (Payload, Cabinet, Align), inline probing achieved 5/30 successes, compared with 21/30 for Fly-by-Code. Further analysis is provided in our response to W1.
Response to Reviewer oqzP (3/3): W4–W5
(W4) Active Feedback on the physical robot
In our new hardware experiment, Active Feedback corrected a false completion assessment and guided three real-world feedback turns, yielding a program that achieved both goals of a 2-Objects pick-and-place task.
The original hardware experiments demonstrated zero-shot deployment of programs refined in simulation. In this additional experiment, we used diagnostic evidence collected on the real aerial manipulator to revise the task program after physical execution failures.
The task was to place a hammer in a hanging basket and a wrench on the upper cabinet shelf. We started from a CaP-X program that had succeeded in simulation. During its initial hardware execution, the hammer struck the basket handle and fell outside. CaP-X incorrectly returned FINISH based on the execution log and terminal image.
In Fly-by-Code, the agent generated a post-task diagnostic program to check whether the hammer was inside the basket. The real robot executed this program, flying to new viewpoints and collecting wrist-camera images that revealed the empty basket and its handle geometry. This additional evidence prompted REGENERATE and task-program revision. These real-world diagnostic flights required no human intervention and respected the collision-avoidance constraints in the low-level robot APIs.
Each of the three feedback turns followed diagnostic flight → completion assessment and program revision → hardware re-execution:
| Turn | Diagnostic evidence | Code revision | Hardware execution result |
|---|---|---|---|
| 1 | Empty basket and handle geometry | Adjust hammer release pose and location | Hammer placed correctly, but wrench misplaced on lower shelf |
| 2 | Hammer placement verified | Preserve hammer code and revise shelf-selection code | Wrench still misplaced on lower shelf |
| 3 | Full-cabinet view from farther away, revealing roof and shelf heights | Revise shelf-selection algorithm | Both goals achieved |
This experiment demonstrates two benefits of active diagnosis on the physical robot:
-
Correcting completion errors:
AF acquired outcome evidence missing from the execution record and replaced an incorrect
FINISHwithREGENERATE. This enabled further refinement instead of terminating with an incomplete task. -
Evidence-guided program refinement:
AF did more than detect failure. Its observations helped preserve successful hammer-placement code and provided the shelf information needed to correct the remaining placement error. The final program achieved both goals, demonstrating how on-robot diagnosis can improve the real-world performance of a simulation-refined program.
We will document the diagnostic observations, code revisions, and hardware execution results in the appendix.
(W5) Separating physical completion from completion assessment
Separate reporting shows that AF improves both physical task completion (75.0%) and completion-assessment precision (90.0%).
Physical task completion is evaluated from the final task-execution state before AF, independently of the agent’s completion declaration. Completion-assessment precision measures the fraction of FINISH declarations that correspond to actual task completion.
Each method was evaluated over 60 trials. “Iteration limit reached” denotes trials that ended after the maximum number of task attempts without declaring FINISH.
| Method | Physical task completion | FINISH declarations | False FINISH | Iteration limit reached | Completion-assessment precision |
|---|---|---|---|---|---|
| CaP-X | 19/60 (31.7%) | 44 | 25 | 16 | 43.2% |
| CaP-X+Trace | 29/60 (48.3%) | 48 | 19 | 12 | 60.4% |
| Fly-by-Code | 45/60 (75.0%) | 50 | 5 | 10 | 90.0% |
These results demonstrate two distinct benefits of AF:
-
Higher physical task completion:
AF achieved 75.0% physical completion, compared with 31.7% for CaP-X and 48.3% for CaP-X+Trace. Its benefits extend to actually completing tasks, not only assessing their outcomes.
-
More reliable completion assessment:
False
FINISHdeclarations decreased from 25 for CaP-X and 19 for CaP-X+Trace to 5 for AF. Its 90.0% completion-assessment precision reflects fewer incorrect terminations with an incomplete task.
Across these 180 trials, all physically completed final task states were accompanied by FINISH, so physical completion rates match the success rates reported in the paper. We will revise the evaluation description and table captions to clearly define and separately report physical task completion and completion-assessment precision.
Response to Reviewer oqzP (1/3): W1
Dear Reviewer oqzP,
Thank you for your time and valuable feedback. We have conducted additional experiments and analyses in response to your comments and address each point below.
(W1) Differences from existing active-probing methods
Fly-by-Code generates a separate post-task diagnostic program to collect new evidence for completion assessment and program revision.
1. Comparison with existing methods
Matcha [1] selects predefined sensing actions. FORCA [2] generates contact probes within task programs and revises them using recorded feedback. PhysCaP [3] generates pre-task exploration programs using physical-property measurement modules.
2. Benefits of separate post-task diagnosis
Our additional comparisons with pre-task and inline probing show that Fly-by-Code achieves higher success rates on all three evaluated tasks. We evaluated an aerial adaptation of PhysCaP for pre-task probing and an ablation of our framework for inline probing, as used in FORCA. We used the same robot APIs and task-attempt budget across these comparisons, with the environment reset between task attempts.
Benefit over inline probing
Separating the diagnostic stage from scored task execution lets the agent prioritize informative probing over immediate task progress. In several inline-probing trials, the agent chose conservative plans while physical constraints remained unknown. In contrast, AF can try informative tests even when the diagnostic actions may fail and use the results to guide more effective program revisions.
Benefit over pre-task probing
Post-task diagnosis focuses probing on information needed for task success and program revision. Pre-task exploration must anticipate useful information before execution, so it may probe unnecessarily broadly or target the wrong information. In contrast, AF uses observations and measurements from the current attempt to identify unverified outcomes and form failure hypotheses. Its diagnostic program then starts from the resulting state and focuses directly on these unresolved questions, without replaying the whole task.
3. Benefits over fixed verification routines
AF generates diagnostic tests in response to observed failures, whereas fixed routines rely on predefined checks that may not provide the evidence needed to diagnose an unexpected failure.
In Align, the robot must push two boxes back so they line up with the other boxes on the shelf. Repeated pushes caused one box to rotate. A routine that only checks final box positions could detect the misalignment, but would not reveal which contact strategy would correct it. The agent generated six tests comparing contact points and sideways slides, measuring how each affected translation and rotation. The revised program used these measurements to align both boxes. A fixed routine would need such tests built in beforehand, whereas AF generated them in response to the observed failure. This reduces the need to anticipate and hand-code task- and failure-specific checks.
[1] Zhao et al. Chat with the Environment: Interactive Multimodal Perception Using Large Language Models. IROS, 2023.
[2] Tan et al. FORCA: A Force-Aware Coding Agent for Contact-Rich Robot Manipulation. arXiv, 2026.
[3] Lin et al. PhysCaP: Grounding Code-as-Policy Agent with Physics-Informed Exploration. arXiv, 2026.