- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
PhysSense introduces a benchmark for sensor-grounded safety decisions in embodied-agent scenarios. It contains 580 synthetic instances from 200 templates across ten categories, with the action labels Safe, Unsafe, and Ask. The inputs are textualized observations and sensor readings rather than raw visual perception or physical robot execution. The paper also proposes PhysReason, a training-free system that routes sensor fields to an explicit rule bank and uses an LLM to interpret the evidence and produce a decision. Across eight models, PhysReason raises mean accuracy from 67.74% to 85.19% and Ask recall from 83.96% to 97.36%, while some robot-operational-state measures regress substantially.
Critical strengths
- The benchmark frames safety as a three-way decision, including abstention, rather than forcing every case into safe/unsafe classification. Counterbalanced safe/unsafe cases are a useful way to probe whether a model attends to sensor values.
- The paper reports heterogeneous model outcomes and multiple error analyses, exposing that aggregate gains do not guarantee improvement on every safety-relevant subtask. The operational-state regression is important evidence for readers to consider.
- PhysReason makes its rule bank and sensor-routing logic explicit, which is a practical and inspectable alternative to relying only on a long free-form prompt.
Critical weaknesses
- The benchmark’s Ask label is structurally leaked. All 180 Ask instances have an empty sensor dictionary, while all Safe/Unsafe instances provide the required readings (Sec. 3.2.2; Appendix A.1–A.2). Thus, an empty-dictionary check identifies Ask without reasoning about which measurement is absent or whether remaining evidence suffices. There are no partial-missingness, conflicting-sensor, or sensor-present-but-uninformative cases. High Ask recall and the triplet score cannot establish robust evidence-sufficiency reasoning under this design.
- The reference labels rely on hidden, template-specific numerical thresholds whose provenance and independent validation are not established clearly enough. The paper states that standards motivate hazard coverage but do not prescribe every numerical boundary (Appendix A.5–A.6); the release uses representative low/high values rather than dense sweeps over sensor ranges (Appendix A.2). Since evaluated models never see the thresholds, a disagreement may reflect a different reasonable operating limit, a point the authors themselves acknowledge. Without documented derivation and independent safety-expert validation, it is unclear what the measured error means beyond reproducing these particular annotations.
- The main Direct-versus-PhysReason experiment is confounded and under-replicated. Direct is asked for a short JSON answer, while PhysReason gets a staged evidence-audit prompt and a larger output cap: 200 versus 700 tokens for default endpoints and 1,200 versus 2,400 for Claude. The instructions, response schema, and parsing path also differ (Sec. 4.1; Appendix B.3–B.6). The authors explicitly characterize the results as comparisons of full configurations, not an isolated RuleBank test. Every model-method pair has one run over 580 examples, with no across-run confidence intervals, and actual latency/token use is not measured. The +17.46-point mean gain therefore cannot be attributed to the RuleBank or judged for run-to-run stability.
- PhysReason is a bundle of a 27-rule knowledge artifact, a router, a selected core-rule projection, and a longer reasoning prompt, but no ablation isolates these components or compares against a token-matched stronger direct prompt. The failure on robot operational state is especially revealing: the category contains motor temperature, battery temperature, and tilt, but the routed method selects one motion/tilt rule whose listed critical states omit the thermal signals (Table 9). Seven of eight models remain at or below 62.50% accuracy in this category; whether joint thermal-and-stability retrieval fixes the failure is untested (Sec. 5.2; Appendix C.1).
- Aggregate accuracy obscures safety-critical regressions, and the evaluation scores only final labels—not the correctness of a requested sensor, intermediate evidence audit, or actual execution. Although mean hazardous approval falls from 12.75% to 8.44%, it rises for GPT-5.4, Claude Opus 5, and Qwen 3.5 35B; Qwen 3.5 35B increases from 18.50% to 22.50% (Table 2). Robot-state accuracy falls by 26.56 points, and all-variant correctness on the 20 robot-state pairs falls from 70.63% to 17.50% (Tables 3–4). The evidence does not support a general safety-improvement claim.
Questions for authors
- Can you compare Direct and PhysReason with identical prompts, token caps, response schemas, and parsers, then ablate the RuleBank, routing, rule projection, and evidence-audit steps separately? Please report actual token consumption and latency.
- Can you provide repeated runs or another justified uncertainty analysis for every model-method pair, and report uncertainty on hazardous approvals and safe rejections rather than only aggregate accuracy?
- How were the per-template thresholds derived and independently validated by robotics/safety experts? Can you test continuous or held-out sensor values, and jointly route the thermal and stability evidence in the robot-state category?
I am willing to discuss with the authors.