- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
TR-Graph aims to repair defective robot commands. An LLM parses the command into a structured task spec and grounds it to a scene graph. An LLM safety evaluation then produces a small causal graph (task variables → hazard → violation sink). A LinUCB bandit allocates the generation budget across six repair operators (substitute, constrain, retarget, reorder, relax, decompose). Candidates pass gated validation and are ranked by a fidelity score minus an intervention cost. The paper evaluates on SafeAgentBench and EARBench, 28 real-robot scenarios on a Franka arm, case studies, and ablations. It claims safer, more executable repairs than all baselines.
Critical strengths
- Repair is a useful alternative to simply accepting or rejecting a defective command. Explicitly balancing safety, feasibility, and user intent makes the problem practically relevant. The system also produces grounded action sequences and required skills, rather than generic safety-grounded sentences. Checking these outputs before passing them to the robot is a good design choice.
- The diagnosis, graph, candidate edits, and validation outcomes provide useful debugging and explanation artifacts.
Critical weaknesses
- The paper’s central claim is that repairs preserve the user’s intent, yet it never reports a fidelity score for any method, relying instead on a low, self-optimized affinity proxy while its own case studies show core intent being dropped (e.g., “Cut the carrot” becoming “Place the carrot”).
- The real-robot evidence measures qualification, not success rate. The paper does not provide a corresponding table of completed physical tasks, execution failures, or observed safety violations. The criterion for successful repair is also ambiguous, with some examples provided seem to replace the user's original intent.
- The bandit’s contribution is unclear because the paper inconsistently states that it resets per experiment versus per task; resetting per task would prevent learning across commands, undermining the claimed benefit of online adaptation.
- Some numbers in the table and text contradict each other, please detect them and fix.
- It's a bit unclear to me whether such as complex system to just do command repair is necessary, seems like something a SOTA LLM can handle.
Questions for authors
- Where are the fidelity results? Appendix E.8 defines Task Fidelity and Repair Quality scores, but these are not reported. Can you provide them for TR-Graph and the baselines?
- What's criterion for a successful repair? It seems pretty subjective for now. For example, why is removing “cut” or “wash” considered a successful repair rather than partial task completion?
- When should the system ask for clarification? If several safe alternatives preserve different aspects of the command, how does it decide whether to select one or ask the user which requirement matters most?
- Do users accept the proposed repairs? Can you report acceptance rates and whether users believe the repaired task achieves their original objective?
- How does the results compared to simply prompting an LLM? What are the cases where LLM fails and the proposed method good at.