- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether large language models internally distinguish content danger (CD, harm expressed in the wording itself) from physical danger (PD, harm arising from executing an instruction in the physical world, e.g., "microwave a metal fork"). Fitting separate L2-regularized logistic probes to last-token hidden states from single layers of six LLMs (Qwen2.5 3B/7B/14B/32B, Phi-3.5-mini, SmolLM2-1.7B), the authors find that CD- and PD-associated directions are statistically separable: probe-weight angles of 71.1–77.3° exceed all 999 size-matched random-split nulls in every model (p=0.001), stable across three coordinate conventions, and content-only readouts detect only 9.6–38.0% of held-out physical danger. Because existing embodied-safety benchmarks suffer lexical shortcuts (word TF-IDF reaches 79.8% on SafeAgentBench), the authors construct PD-Bench: 650 human-annotated object-placement cells (48 objects × 38 sites, three annotators, explicit danger words excluded, object/site/phrasing marginals balanced) instantiated as 2,000 paraphrased instructions. On PD-Bench under a five-seed, cell-grouped nested protocol, a frozen linear probe reaches 0.802–0.841 cell-AUC on unseen combinations versus 0.652–0.682 for the models' own verbal judgments—a "representation–verbalization gap" that also appears on SafeText (+0.03), SafeAgentBench (+0.03), and EARBench (+0.30), persists across 22 judge configurations up to 72B, and is not closed by extended thinking. The gap is localized causally: an exact output-layer graft of the probed layer raises verbal AUC to 0.801–0.845, a rank-16 deep bridge recovers 24–75% of the gap (embedding-sourced controls stay at 0.650–0.691), and a transferred Azaria–Mitchell truth probe achieves only 0.599–0.656—supporting a routing-failure account rather than missing knowledge or generic truthfulness. Building on the gap, the authors introduce PRISM, a d+1-parameter single-layer probe applied to a frozen judge-prompt forward pass, which on SafeAgentBench/SafeText/EARBench reduces false-positive rates from 27.3%/46.3%/49.1% to 13.7%/11.7%/17.6% without modifying the base model, at roughly half the measured latency of verbal scoring in the authors' (unoptimized) implementations.
Critical strengths
(I) A well-posed representational question answered with exemplary measurement hygiene. The paper asks whether content danger and physical danger are the same internal signal, and answers it with a battery of controls that collectively make the finding hard to dismiss: size-matched random-split nulls for the angle statistic (p=0.001 in six models), replication across three coordinate conventions, cross-category detection collapse as behavioral corroboration, and a purpose-built benchmark (PD-Bench) that removes both danger-word shortcuts and marginal leakage—so the separability and gap results cannot be explained by vocabulary. This matters because it converts a vague worry ("embodied safety is different from text safety") into a measurable, replicated property of LLM representations, and it does so with a methodological template (exact nulls, cell-grouped folds, unanimous strata, permutation controls) that the probing literature routinely lacks. (II) The causal attribution section goes beyond correlation and polices its own conclusions. Rather than stopping at "the probe beats the model's words," §5 asks why and localizes the gap: the exact output-layer graft shows the information is legible to the output head but unused; the authors then demonstrate—via an oracle scalar late-fusion sweep—that the graft cannot support a routing claim, and rest that claim solely on the rank-16 deep bridge whose signal must traverse a full decoder block; embedding-sourced controls rule out added capacity; and the Azaria–Mitchell truth-probe transfer with register-matched declarative re-rendering rules out a generic truthfulness direction. Combined with the honest reporting of the mean-difference ablation's unresolved collapse mechanism, this is one of the more intellectually careful intervention sections I have read in a safety-probing paper, and it is consequential for my recommendation because it elevates the paper from "a good probe detector" to "evidence about why verbal safety judgments fail," which is the scientifically durable contribution.
Critical weaknesses
The deployment story is narrower than the headline suggests. Two scoping decisions bound the evidence. First, the paper restricts itself to P0 risk—categorical object–site compatibility under default conditions—by its own three-tier taxonomy, excluding P1 (parameter/device-state dependent, e.g., 30 s vs. 10 min in a microwave) and P2 (accumulation over time), which are precisely the regimes where embodied deployment actually fails; all results are also instruction-level text only, with no scene state or perceptual grounding, so the situational safety cases that flip an instruction's safety with the scene are out of scope. Second, PRISM is bound to its representation context: cross-prompt transfer collapses (0.46), cross-dataset frozen transfer underperforms zero-shot verbal baselines on SafeText, and every external-benchmark result uses within-dataset refitting. The paper is commendably transparent about both limits, but together they mean the demonstrated artifact is a per-domain, per-prompt static-text filter for household object-placement risk—valuable as evidence, but far from the "physical-risk detection in embodied agents" framing of the abstract. I would like the limitations to be surfaced in the introduction rather than only in §6.
Questions for authors
(I) PD-Bench labels have Fleiss κ=0.34 overall (356/650 unanimous). Did you consider a larger annotator pool or an adjudication round for split cells? How sensitive are the probe–verbal gap estimates to restricting to cells where at least two annotators report high confidence? (II) You evaluate unseen object–site combinations and leave-one-mechanism-out transfer, but no adversarial evaluation. Since probing-based detectors are known to fail under distribution shift crafted by an adversary (e.g., Wang et al., 2026, which you cite), how does PRISM behave under paraphrase attacks or instruction obfuscation designed to push probes below threshold? Even a small red-team study would substantially strengthen the safety claim. (III) The three annotators' mean confidences are 1.73/2.36/2.65 (of 3). Did annotators receive any training or calibration on pilot items beyond the written protocol? And since the per-cell vote table will be released: does the lowest-confidence annotator's vote pattern differ systematically on the split cells, and would the probe–verbal gap estimates change if the released labels were restricted to cells where at least two annotators report high confidence? (IV) Concurrent interpretability work reads silent internal states directly: the Jacobian lens / J-space line (Anthropic's "A global workspace in language models," July 2026, and the J++ Lens follow-up, October 2026) reads verbalizable internal representations that the model never expresses in output, for monitoring goals such as detecting eval awareness and hidden plans. Your representation–verbalization gap is conceptually adjacent: the model "knows" physical risk internally but fails to verbalize it. Could you comment on the relationship—for example, whether the CD/PD directions you isolate lie inside or outside such "workspace" representations, and whether unsupervised lens-style readouts could complement your supervised probe on this task?
2 comments
3. The three annotators' mean confidences are 1.73/2.36/2.65 (of 3). Did annotators receive any training or calibration on pilot items beyond the written protocol? And since the per-cell vote table will be released: does the lowest-confidence annotator's vote pattern differ systematically on the split cells, and would the probe–verbal gap estimates change if the released labels were restricted to cells where at least two annotators report high confidence?
We apologize for not describing annotator preparation clearly enough. All three annotators discussed the annotation guidelines together and completed a pilot annotation round before formal labeling.
On the 294 split cells, the lowest-confidence annotator (A) cast the minority vote on 92 cells (31.3%), compared with 109 (37.1%) for B and 93 (31.6%) for C. Thus, A was not the most frequent minority voter. Voting tendencies nevertheless differed, the unsafe-vote rates on these same cells were 45.2%, 32.7%, and 83.7%, respectively. A’s disagreements with the majority more often favored safe, as 70/181 majority-unsafe cells versus 22/113 majority-safe cells. The lowest-confidence annotator showed no systematic voting differences on split cells.
The high-confidence analysis in 1(b) supports the robustness of our main finding. Restricting evaluation to 279 cells with at least two maximum-confidence ratings preserves positive probe–verbal gap estimates across all four models. The 95% credible intervals remain no less than zero for all models.
4. Concurrent interpretability work reads silent internal states directly: the Jacobian lens / J-space line (Anthropic's "A global workspace in language models," July 2026, and the J++ Lens follow-up, October 2026) reads verbalizable internal representations that the model never expresses in output, for monitoring goals such as detecting eval awareness and hidden plans. Your representation–verbalization gap is conceptually adjacent: the model "knows" physical risk internally but fails to verbalize it. Could you comment on the relationship—for example, whether the CD/PD directions you isolate lie inside or outside such "workspace" representations, and whether unsupervised lens-style readouts could complement your supervised probe on this task?
Thank you for this insightful question and highlighting this connection. The workspace framework raises a concrete question about the physical-risk information that our probes recover but verbal judgments fail to express.
CD/PD separation and workspace membership concern different properties: our analysis distinguishes risk readout directions, whereas the workspace framework characterizes representations available for verbal report and flexible internal use. Both CD and PD could therefore occur within the same workspace while remaining directionally separated.
The Jacobian lens and J++ Lens could complement our probes by revealing risk-related concepts without training a risk-specific classifier. Our probes quantify how reliably hidden states distinguish safe from unsafe instructions; lens readouts could help interpret the concepts supporting that distinction. If risk-related concepts appear when verbal judgments fail, this would provide additional evidence that the missed information is available in a verbalizable form.
5. Scope and Deployment Limitations
Sorry for the confusion and thank you for your advice. We understand your concerns and agree that the boundaries of our findings should be better communicated. We will revise the Introduction and Abstract to better define our research scope and clarify the boundaries of our conclusions. In future work, we will explore more complex physical danger scenarios and generalization to broader settings. We will clarify this point in the final version of the manuscript.
6. Cross-prompt and cross-dataset transfer
Thank you for this constructive suggestion. We apologize for not making our transfer protocol clear enough. PRISM supports cross-dataset use by adapting only the linear readout using target-domain labels, while keeping the entire LLM frozen. On SafeText, this achieves approximately 0.984 AUC for both 3B and 7B, exceeding zero-shot verbal judgments of 0.964/0.973. The 0.46 AUC concerns direct cross-prompt transfer of a fixed judge-trained probe and fitting the readout within the planner context yields 0.778 AUC. We will foreground this distinction and clarify how readout adaptation enables cross-dataset use without updating the underlying LLM.
Thank you so much for your exceptionally thoughtful, generous, and encouraging evaluation. We are genuinely grateful for the care and attention you have devoted to understanding our work. We are equally grateful for your insightful and constructive suggestions, which will help us further refine and strengthen our work.
1. PD-Bench Annotation Reliability and Sensitivity Analysis
a. PD-Bench labels have Fleiss κ=0.34 overall (356/650 unanimous). Did you consider a larger annotator pool or an adjudication round for split cells?
Thank you for your valuable suggestion. We will involve independent adjudicators to review all cells with annotator disagreement and resolve disputed labels. We will report the results in final version.
b. How sensitive are the probe–verbal gap estimates to restricting to cells where at least two annotators report high confidence?
Thanks for your valuable insights. Our sensitivity analysis supports the probe’s advantage on high-confidence examples.
Restricting evaluation to cells where at least two annotators report maximum confidence (3) retains 279 cells (35 safe, 244 unsafe). Probe–verbal gap estimates remain positive for all four models, with 95% credible intervals above zero for three.
& & \begin{array}{ccccc} \hline \rule{0pt}{2.6ex} \textbf{Model} & \textbf{Probe AUC} & \textbf{Verbal AUC} & \textbf{Gap} & \textbf{95\\% credible interval} \\\\[5pt] \hline \rule{0pt}{2.6ex} \text{Qwen2.5-3B} & 0.906 & 0.850 & +0.055 & [-0.005,\\,0.116] \\\\[4pt] \text{Qwen2.5-7B} & 0.913 & 0.845 & +0.068 & [0.012,\\,0.126] \\\\[4pt] \text{Qwen3-8B} & 0.872 & 0.745 & +0.128 & [0.052,\\,0.209] \\\\[4pt] \text{Phi-3.5-mini} & 0.908 & 0.829 & +0.078 & [0.024,\\,0.137] \\\\[4pt] \hline \end{array} & & & \end{flalign*}$$ We retain the original labels, fitted models, and averaging over 25 within-fold cell AUCs. Intervals use 10,000 paired cell-level Bayesian-bootstrap draws. **2. You evaluate unseen object–site combinations and leave-one-mechanism-out transfer, but no adversarial evaluation. Since probing-based detectors are known to fail under distribution shift crafted by an adversary (e.g., Wang et al., 2026, which you cite), how does PRISM behave under paraphrase attacks or instruction obfuscation designed to push probes below threshold? Even a small red-team study would substantially strengthen the safety claim.** Thank you for this suggestion. We conducted a small score-guided red-team study on Qwen2.5-3B, and the results show that PRISM retained an 85% detection rate (34/40) under paraphrase attacks and instruction obfuscation. We randomly selected 40 instructions from PD-Bench. All 40 were unanimously labeled as unsafe by all three annotators at the highest confidence level and were correctly identified as unsafe by the model before perturbation. We then generated up to 6 perturbations, covering paraphrasing, structured formatting, and character/whitespace obfuscation and apply them to the original instructions, forming 40 candidate instructions. One human reviewer assessed all 40 original and candidate pairs for preservation of the action, conditions, and physical risk. Rejected or uncertain candidates were removed and were replaced by other instructions until all 40 instructions met the inclusion criteria. We review these attacks using frozen probes, fixed-shape FP32 inference, and single-instruction thresholds calibrated exclusively from training-inner-OOF predictions, targeting a nominal 10% FPR. After perturbation, 34 instructions were still correctly detected, while six were misclassified as safe, reducing the detection rate from 100% to 85%. This corresponds to a conditional attack success rate of 15% (6/40). Five failures resulted from lexical substitutions and one from the word-order change.