- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that a robot program that reaches the correct final outcome may still violate physical constraints while it runs, and it proposes RoboPhys to evaluate this separately. RoboPhys is an evaluation protocol that complements terminal task predicates with trajectory-level physical monitoring under diverse physical conditions. Across five code-generation systems and 300 programs, the results show that trajectory-level diagnoses can guide effective repair.
Critical strengths
-
RoboPhys clearly distinguishes task completion from physical execution quality and uses held-out re-execution without giving feedback to the generator.
-
The trajectory audit provides useful mechanism-level analysis, while complete-run validation exposes downstream failures and repair regressions.
Critical weaknesses
-
The physical oracle is tied to a narrow, hand-specified Panda/MuJoCo setup. Its thresholds, overlapping F/A predicates, and treatment of rejected requests limit the generality of the conclusions.
-
The repair evaluation is based on inspected cases and uses task-specific rules and unequal information access. It therefore does not establish a general repair method or isolate which design choices produce the reported gains.
Questions for authors
-
Can you validate the physical monitors outside the current Panda/MuJoCo stack, or provide independent expert/hardware labels and event-level error rates? Please also report the headline results after separating the predicates shared by F and A, and after treating rejected requests L separately from executed physical violations.
-
Can you evaluate one frozen repair policy on tasks, geometries, and generation sources that were not used for the trajectory audit or pilot repairs, with matched information access and a simple no-repair baseline? Please report paired uncertainty estimates and all F, A, L, strict-success, and regression outcomes.
2 comments
Thanks for the responses by the authors. The event-label reanalysis resolves my concern that overlapping predicates might explain the main finding, and the iiwa experiment adds useful evidence beyond Panda. Both setups still share the simulator and monitor code, however, so monitor accuracy remains independently unverified. The repair evidence is also limited: the main results concern inspected cases, and the transfer pilots show mixed gains alongside functional regressions. These additions address part of my concerns, but stronger validation and evidence of transfer are still needed. I therefore maintain my original rating.
Response to Reviewer PeNt
Thank you for the specific validation requests. We added event-label analysis, paired cross-arm execution, and repair tests.
We reanalysed all 2,736 saved executions, retaining the programs, thresholds, completion labels, refusal records, and experimental groups. These are existing records, not additional runs.
Alongside the original A label, A_exec retains events supported by executed states. It excludes release_instability, which overlaps with terminal stability, and the request/protocol flags premature_release and unverified_transport. The latter can arise before IK or gateway acceptance and does not establish executed motion.
Nominal violation counts fall from 46 to 44, while all 38 unsafe successes among 136 functional successes remain. The two held-out groups retain 94 and 493 executions with actual violations, including 63 and 455 unsafe successes. No saved execution contains release_instability. Among 123 nominal completions without gateway refusal, 31 still contain executed violations. Separating request flags and overlapping predicates therefore preserves the main finding.
We also replayed 29 unchanged programs on Panda and iiwa. Reference programs completed all six nominal tasks on each arm without violations or refusals. Each arm had four unsafe successes, with only two violating sources shared. Base, home configuration, and controller were adapted before freezing the setup. Joint-speed limits were arm-specific; Panda's TCP limit was not applied to iiwa because no verified global iiwa bound was available. All violating runs involved collision, monitored on both arms.
Both setups still share MuJoCo, the gripper, and monitor code. Independent expert or hardware labels, event-level error rates, and threshold-sensitivity measurements remain missing. Cross-arm replay broadens the evidence but does not validate detector accuracy independently.
The original 42/57 and 6/57 results compare complete repair strategies with different observations, candidate construction, and edit locations. They cannot isolate the effect of task knowledge. The former uses rules developed from inspected trajectories; the latter is a state-dependent geometric prototype. Rule application and evaluation are automatic, while rule design occurs during development.
We froze the prototype before testing five existing programs under two new geometries each. Against the unchanged-program baseline, strict successes rose from 3 to 4 across ten configurations, while completions fell from 7 to 5. Strict success requires completion without violations or rejected requests. A separate pilot used one project-aware assistant-written source across five tasks and two settings each; strict successes remained at four. This was one shared source, not ten independently generated programs.
An additional release test gave geometric, transition, and complete-run selectors the same candidate bank. Transition and complete-run checking selected the same safe 30-mm adjustment on the inspected Panda C1 source. Neither found a safe candidate on iiwa. Thus, this test shows no extra recovery from complete-run selection over transition checking. Candidate selection consumed execution feedback and was not an independent held-out performance estimate.
A safe B1 control required no edit, while a B2 program that aborted before release remained unrepaired. Fixed release edits preserved completion for two known C1 sources on both arms but removed all recorded violations only on Panda.
We report both recoveries and regressions. The experiments support repair of some inspected cases, while a frozen policy tested on unused tasks and generation sources, with matched information and paired uncertainty estimates, is still needed to establish general repair performance.