- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PhysCogSafe diagnoses risk-conditioned behavior using matched benign, hazardous, and control scenes plus an independently verified safe reference. It covers geometry, object properties, and multi-subgoal constraints across five robot policies and 11 scenario families. Among baseline-successful trials, macro-averaged safe completion falls from 69.5% in controls to 11.7% under risk, with distinct failure profiles across dimensions.
Critical strengths
- Matched controls, unchanged goals, and verified safe references address important confounds in robot safety evaluation and make the diagnostic framework useful.
- Capability conditioning and the four joint outcomes distinguish unsafe task completion from harmful failure; explicit exclusions and cell-level denominators improve transparency.
Critical weaknesses
- Diagnostic attribution: Success on the native task establishes ordinary manipulation competence, not competence at the safe alternative required by a probe. A safe-reference controller demonstrates feasibility but not that the evaluated policy possesses that skill. The results identify failed adaptation, without fully distinguishing risk cognition from perception or control limitations.
- Construction dependence: Trajectory-conditioned probes are selected so the policy's original actions become harmful under risk. This is a valid stress test, but its difficulty depends on the policy and chosen trajectory. Even with the stated within-policy interpretation, pooled failure rates need stability checks using separate construction trajectories and frozen/shared probes to establish how much is specific to this selection.
- Statistical evidence: Several cells have only two baseline-successful trials (Tables 10–12), yet receive full weight in cell-level macro-averages. Paired confidence intervals and sensitivity to sparse cells are missing. The large overall gap is plausible, but exact headline values and dimension comparisons are less secure than their point estimates suggest.
- Dimension interpretation: D1, D2, and D3 contain only four, four, and three scenario families, with different tasks and physical interventions. Larger deficits in D2/D3 may partly reflect probe difficulty or additional motor/sequencing demands. Matched-difficulty or factorial probes are needed before treating these differences as evidence about the relative cognitive demands of the dimensions.
- Incremental contribution: Existing robot-safety evaluations already establish that task success does not imply safe execution. The matched-control protocol is a useful refinement, but the paper does not demonstrate that its taxonomy reliably separates failure causes or yields diagnostic insights unavailable from simpler outcome-based evaluation. This limits the significance of the claimed cognitive diagnosis.
Questions for authors
- Can a risk-aware comparison policy with the same observations/action interface complete the safe alternatives, and do explicit risk cues separate recognition failures from execution limitations?
- How stable are the results on independently constructed trajectories or frozen shared probes, and what are paired intervals and sparse-cell sensitivity results?
- Do the D1-versus-D2/D3 differences persist after controlling task difficulty and additional motor/sequencing requirements?
Q1. Can a risk-aware comparison policy with the same observations/action interface complete the safe alternatives, and do explicit risk cues separate recognition failures from execution limitations?
A1. We agree with the diagnostic-attribution boundary. Native-task success shows only ordinary manipulation competence; the safe reference shows that a safe solution exists under the same robot interface, not that the evaluated policy possesses that skill. Our estimand is therefore failed risk-conditioned adaptation on a feasible probe, not a process-level separation of risk cognition, perception, or control. A same-interface risk-aware comparison policy would strengthen the positive- control story, but it is not required to interpret the present negative result once Πsafe already establishes solvability. Explicit linguistic risk cues similarly change the construct from native-prompt vision-conditioned behavior to language-conditioned safety, so they cannot cleanly separate recognition from execution under our protocol. We will state both as valuable future diagnostics and remove language that implies a fully decomposed cognitive attribution.
Q2. How stable are the results on independently constructed trajectories or frozen shared probes, and what are paired intervals and sparse-cell sensitivity results?
A2. We agree that trajectory-conditioned probes are policy- and trajectory- dependent stress tests, so pooled rates must not be read as a shared-item score. Primary claims will be within-policy / within-cell directional contrasts (Q(ER)<Q(EC), H(ER)>H(EC)), with the aggregate 69.5→11.7 demoted to a descriptive summary of heterogeneous cells. We will add paired bootstrap CIs and sparse-cell sensitivity (nB≥5/10), so that cells with nB=2 remain in the ledger but do not drive headlines. Full stability on independently reconstructed trajectories and frozen shared cross-policy probes is important; we will report what the current artifacts support and mark broader reconstruction as a limitation / future check rather than claim it is already established.
Q3. Do the D1-versus-D2/D3 differences persist after controlling task difficulty and additional motor/sequencing requirements?
A3. No: under the present design we cannot treat D1 versus D2/D3 as evidence of relative cognitive demand. The dimensions contain only 4/4/3 families with different tasks, interventions, and motor/sequencing burdens, so larger ΔQ may partly reflect probe difficulty rather than dimensional cognition. What the data do support is a qualitative outcome-composition contrast: D2 more often shows harmful completion (p11), D3 more often harmful failure (p01), while D1 is more mixed. We will remove headline rankings such as 41.7/70.5/63.4 as cognitive- difficulty scores, report family-level results, and state that matched- difficulty or factorial probes are future work. Correspondingly, the contribution is the matched EB/ER/EC plus safe- reference attribution protocol, not a taxonomy that already separates failure causes beyond simpler outcome evaluation.