acceptodds
Withdrawn from ICLR 2027

Coding Agents with Harness for Safe Robot Control

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Coding agents have emerged as a promising paradigm for robot manipulation: a language model writes the robot controller as a program, and agents built in this way now operate robots without robot-specific training. Whether this paradigm is also safe, however, has not been asked. We evaluate coding agent under a safety constraint, where each task pairs a manipulation goal with an obstacle the robot must not touch. The agent pursues the goal but collides with the obstacle in most cases, treating task completion as its sole objective while neglecting safety. The agent reasons about the obstacle in its traces, and the prompt already forbids touching it, so neither perception nor instruction is at fault; the fault lies in the planning, where the stated constraint never becomes a priority. By decomposing manipulation into a route phase and a contact-rich moment, we locate the source of the failure. Along the route, the model cannot prioritize the safety constraint, having no notion of a clearing route and none of replanning once a chosen route becomes infeasible. At the contact, it is unaware that contact execution is bounded by the same constraint. To close this gap, we present SafeHarness, which equips the model with two obstacle-aware harnesses that enable it to prioritize the safety constraint. Obstacle-aware route planning grounds the objects as bounding boxes and draws candidate routes over them as sequences of waypoints. The agent then plans a route in advance, verifies it, replans when necessary, and only then executes it. Obstacle-aware contact execution instead selects the contact position so that the contact itself avoids the obstacle. SafeHarness attains task success and collision avoidance with GPT-6-astra, surpassing the previous SOTA by and points, respectively. Compared with the same agent without harnesses, these results represent gains of and points, respectively.

Primary area:
foundation or frontier models, including LLMs
Keywords:
robot manipulation, safety, coding agent

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_hpCvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SafeHarness adds obstacle-aware route planning and contact execution to a coding-agent robot controller. Routes are proposed, checked externally, and replanned when needed; contact strategies depend on obstacle adjacency. On 32 SafeLIBERO tasks, the strongest configuration achieves 81.2% task success, 91.9% collision avoidance, and 78.1% safe success, with complementary real-robot results and component ablations.

Critical strengths
  1. Same-backbone and component ablations show substantial improvements in joint safe success, supporting the value of both route and contact protection.
  2. Real-robot tests complement simulation and demonstrate that the proposed execution procedure can work beyond the benchmark simulator.
Critical weaknesses
  1. Method novelty: The main gains come from explicit collision checks, replanning, and obstacle-dependent contact rules. The paper compares against VLAs and AEGIS, but not a conventional motion planner inside the same coding-agent harness. It therefore does not establish the benefit of LLM waypoint generation over an established geometric planning solution.
  2. Safety coverage: CAR counts an obstacle as untouched when its displacement never exceeds 1 mm; contact without displacement can therefore be missed. Route checks describe the gripper and payload, without sufficiently specifying full-arm coverage or image-to-metric geometry. These gaps matter because the method is presented as enforcing collision avoidance, rather than only reducing this surrogate.
  3. Failure attribution: Verbal recognition of an obstacle does not establish accurate perception or feasible control. Likewise, adding geometric tools and execution checks does not isolate whether failures come from prioritization, skill replay, or planning limitations. The claims about safety priorities and growing-context failures need interventions on those factors, beyond the current harness ablations.
  4. Practical validation: Real-robot evidence covers only two tasks with 20 trials each; simulation primarily tests one static obstacle. There are no uncertainty estimates or characterization of planning latency/API cost. This limits conclusions about robustness and practical operation under varied clutter, perception error, or time constraints.
Questions for authors
  1. How does the same harness perform with an established motion planner replacing LLM waypoint proposals, retaining contact rules and collision verification?
  2. Which robot bodies and swept volumes are checked, how is metric geometry obtained, and how do results change under direct contact detection and perturbed obstacle estimates?
  3. What interventions support the priority/context explanation, and what are real-robot uncertainty estimates and planning-time costs?
Reviewer_au4XOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SafeHarness, a framework for improving obstacle avoidance in coding-agent-based robot manipulation. The system coordinates motion primitives and a frozen VLA through obstacle-aware route planning and contact execution. Route planning uses segmentation-derived bounding boxes to guide waypoint generation, check proposed paths, and trigger replanning. Contact execution selects approach directions and gripper orientations based on nearby obstacles. Experiments on SafeLIBERO and two real-robot pick-and-place tasks show improvements in task success and collision avoidance over VLA and coding-agent baselines.

Critical strengths
  • The paper addresses a timely and relevant topic: safety in agentic robot manipulation. Evaluating these systems beyond task completion is important.
Critical weaknesses
  • Limited technical contribution and insufficient safety analysis. The method largely integrates collision checking and motion planning into existing agentic robotic framework, with limited methodological novelty. Given this, I would expect the paper’s contribution to come from a deeper understanding of safety failures in existing systems or a more comprehensive and diagnostic evaluation. However, the paper largely treats safety as obstacle avoidance while making broad claims about safe robot control. It does not sufficiently situate this formulation within the broader safety literature or distinguish between spatial and temporal constraints, predefined geometric rules and commonsense or user-specified restrictions, and empirical safety improvements versus guarantees. These distinctions need not all become experimental settings, but they should inform the scope and discussion. Although the component ablations demonstrate improvements, they primarily show that adding obstacle-aware mechanisms improves obstacle avoidance. They provide limited insight into the causes of failures, the assumptions under which the safeguards remain effective, and the failures that persist despite them. The limited methodological novelty is therefore not accompanied by sufficient analytical depth or empirical insight to support the paper’s broad positioning.
  • Missing technical details hinder comprehension and reproducibility. The paper does not provide any appendix, prompts, videos, or code. More importantly, central operations remain unclear: how visual observations (2D/3D) are converted into object geometries, how the external verifier checks the robot and held object, how contact strategies are defined and selected, and how these configurations are passed to or enforced during VLA execution. The real-robot VLA checkpoint and its training history are also insufficiently documented. These omissions make it difficult to reconstruct the method or assess its assumptions.
  • The presentation is poor and space is not used efficiently. Considerable space is devoted to defensive and repetitive prose while essential technical explanations remain missing. For example, the two “Before any motion is issued,” or “The proposal obtained this way is not yet a route.” The methodological sections would benefit from concrete algorithms, interface definitions, and implementation details. The main obstacle to understanding the paper is missing technical information, which repeated high-level justification does not resolve.

Other comments:

  • The section organization could be improved. It’s a bit odd to include preliminaries and discussion in the method section. In my opinion, only 3.2 and 3.3 are the actual method sections.
  • Sections 2–4 read to me as largely AI-generated, especially section 2-4. The paper contains many repeated, defensive phrases (~20 times of “rather than”) and strange technical terms (“economy of observation”, “route is free to bend”). This wastes the space and contributes to the poor presentation.
Questions for authors
  • How is the external verifier implemented in simulation and on the real robot? What geometric representations does it use, and does it access privileged information such as object models or ground-truth states? If these quantities are estimated, please describe the perception pipeline and how its outputs support metric collision checking.
  • Which VLA checkpoint is used for the PiPER-X experiments? Since Harness VLA uses a different real-robot platform, please clarify the checkpoint’s prior training, any robot-specific fine-tuning, and the observation/action mapping used for PiPER-X. Does “without fine-tuning” mean no additional fine-tuning within SafeHarness?
Reviewer_8iAeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows that LLM coding agents controlling robot arms on SafeLIBERO usually finish the task but hit the obstacle they were told to avoid, even when they mention it in their reasoning. SafeHarness splits each task into a free-space route and a contact step. For the route, the agent grounds target and obstacle as bounding boxes, proposes waypoints, has them checked against the obstacle box outside the LLM, and replans when the check fails or execution stalls. For contact, an adjacency test picks the approach side or wrist rotation. The method beats AEGIS and the same agent without the harness on both task success and collision avoidance, with a small real-robot study showing the same trend.

Critical strengths

The question is timely, since coding agents for manipulation are scored almost only on success. The most interesting finding to me is that adding skill memory makes the agent less safe near the target, because it replays skills learned in obstacle-free scenes. That points to a real tension between competence and safety in this paradigm.

Keeping the obstacle in a stateless tool outside the conversation, rather than hoping a prompt constraint survives a long context, is a clean design. The gains over AEGIS are large, and the harness is simple enough for others to adopt.

Critical weaknesses

The experiments do not show which part of the harness does the work, or that the LLM is needed for it. The agent without the harness seems to get no bounding boxes, so the gain mixes obstacle grounding, LLM route proposal, the box check, and replanning. The paper says the route proposer could be replaced by any planner, which makes a classical planner over the same boxes, driving the same frozen VLA, the natural baseline. The "first" claims should also be softened, since earlier work enforces safety constraints on LLM-driven robot agents [1, 2], though at the task level.

The supporting evidence is also thin. The claim that prompt constraints fade as the context grows rests on citations and one rollout. The baselines appear to come from a different protocol and lack safe-success rates, and there are no confidence intervals although several ablation gaps are small. It is also unclear how the 2D boxes become metric clearance checks, and how the contact rule steers a frozen VLA.

[1] Z. Yang, S. S. Raman, A. Shah, and S. Tellex. Plug in the Safety Chip: Enforcing Constraints for LLM-driven Robot Agents.
[2] Z. Ravichandran, A. Robey, V. Kumar, G. J. Pappas, and H. Hassani. Safety Guardrails for LLM-Enabled Robots.

Questions for authors
  1. How does a classical planner (e.g., RRT) over the same boxes, with the same frozen VLA, compare with the LLM route proposer?
  2. Can you separately ablate grounding, the verifier, and replanning, and report how often verification fails or execution halts?
  3. Were the baselines re-run under your protocol, and what are their safe-success rates with confidence intervals?
  4. Is there direct evidence for context dilution, such as collision rate against episode length, or a baseline that restates the constraint every turn?

Clarification: how are the 2D boxes lifted to 3D, and how are the two obstacles in the real-robot scenes handled?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.