acceptodds
Under review as a conference paper at ICLR 2027

Safe in Text, Unsafe in Action: Distinguishing Content and Physical Danger in LLM Hidden States

Abstract

Deploying large language models (LLMs) in embodied agents extends safety concerns from generated content to physical actions. Although content safety has been extensively studied, it does not guarantee physical safety, as seemingly benign instructions can become hazardous when executed in the physical world (e.g., “microwave a metal fork”). We identify content danger (CD) and physical danger (PD) as distinct safety dimensions and investigate whether LLMs internally distinguish them. Across six LLMs spanning 1.7B32B parameters and three model families, we find statistically separable CD- and PD-associated directions in hidden-state representations (, the 999-permutation floor). This separation exposes a gap between internal safety representations and verbal safety judgments. Across four benchmarks, a linear readout of hidden states outperforms the model’s own verbal judgment by 0.030.30 AUC, with the largest gap occurring for physically grounded danger. A rank-16 path from the probed layer into the final decoder block recovers 2475% of this gap without changing any base weight, while the same construction at the embedding layer does not, meaning the signal is present but under-used. A probe transferred from a true/false benchmark reaches only 0.600.66, so it is not generic truthfulness. Building on this finding, we introduce PRISM (Probing Representations for Integrated Safety Monitoring), a lightweight detector that identifies physical risks using a linear probe of a single hidden layer without modifying the base model. Across SafeAgentBench, SafeText, and EARBench, PRISM reduces false positives by 5075% compared with verbal safety judges. These results show that LLMs internally encode physical danger as a distinct safety dimension, but fail to fully route this information into their own safety judgments, enabling hidden-state representations to serve as an effective signal for physical-risk detection in embodied agents.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM safety, embodied AI safety, physical safety, hidden-state probing, linear probes, interpretability, embodied agents, safety benchmark, over-refusal, representation-verbalization gap

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Mh2jOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether large language models internally distinguish content danger (CD, harm expressed in the wording itself) from physical danger (PD, harm arising from executing an instruction in the physical world, e.g., "microwave a metal fork"). Fitting separate L2-regularized logistic probes to last-token hidden states from single layers of six LLMs (Qwen2.5 3B/7B/14B/32B, Phi-3.5-mini, SmolLM2-1.7B), the authors find that CD- and PD-associated directions are statistically separable: probe-weight angles of 71.1–77.3° exceed all 999 size-matched random-split nulls in every model (p=0.001), stable across three coordinate conventions, and content-only readouts detect only 9.6–38.0% of held-out physical danger. Because existing embodied-safety benchmarks suffer lexical shortcuts (word TF-IDF reaches 79.8% on SafeAgentBench), the authors construct PD-Bench: 650 human-annotated object-placement cells (48 objects × 38 sites, three annotators, explicit danger words excluded, object/site/phrasing marginals balanced) instantiated as 2,000 paraphrased instructions. On PD-Bench under a five-seed, cell-grouped nested protocol, a frozen linear probe reaches 0.802–0.841 cell-AUC on unseen combinations versus 0.652–0.682 for the models' own verbal judgments—a "representation–verbalization gap" that also appears on SafeText (+0.03), SafeAgentBench (+0.03), and EARBench (+0.30), persists across 22 judge configurations up to 72B, and is not closed by extended thinking. The gap is localized causally: an exact output-layer graft of the probed layer raises verbal AUC to 0.801–0.845, a rank-16 deep bridge recovers 24–75% of the gap (embedding-sourced controls stay at 0.650–0.691), and a transferred Azaria–Mitchell truth probe achieves only 0.599–0.656—supporting a routing-failure account rather than missing knowledge or generic truthfulness. Building on the gap, the authors introduce PRISM, a d+1-parameter single-layer probe applied to a frozen judge-prompt forward pass, which on SafeAgentBench/SafeText/EARBench reduces false-positive rates from 27.3%/46.3%/49.1% to 13.7%/11.7%/17.6% without modifying the base model, at roughly half the measured latency of verbal scoring in the authors' (unoptimized) implementations.

Critical strengths

(I) A well-posed representational question answered with exemplary measurement hygiene. The paper asks whether content danger and physical danger are the same internal signal, and answers it with a battery of controls that collectively make the finding hard to dismiss: size-matched random-split nulls for the angle statistic (p=0.001 in six models), replication across three coordinate conventions, cross-category detection collapse as behavioral corroboration, and a purpose-built benchmark (PD-Bench) that removes both danger-word shortcuts and marginal leakage—so the separability and gap results cannot be explained by vocabulary. This matters because it converts a vague worry ("embodied safety is different from text safety") into a measurable, replicated property of LLM representations, and it does so with a methodological template (exact nulls, cell-grouped folds, unanimous strata, permutation controls) that the probing literature routinely lacks. (II) The causal attribution section goes beyond correlation and polices its own conclusions. Rather than stopping at "the probe beats the model's words," §5 asks why and localizes the gap: the exact output-layer graft shows the information is legible to the output head but unused; the authors then demonstrate—via an oracle scalar late-fusion sweep—that the graft cannot support a routing claim, and rest that claim solely on the rank-16 deep bridge whose signal must traverse a full decoder block; embedding-sourced controls rule out added capacity; and the Azaria–Mitchell truth-probe transfer with register-matched declarative re-rendering rules out a generic truthfulness direction. Combined with the honest reporting of the mean-difference ablation's unresolved collapse mechanism, this is one of the more intellectually careful intervention sections I have read in a safety-probing paper, and it is consequential for my recommendation because it elevates the paper from "a good probe detector" to "evidence about why verbal safety judgments fail," which is the scientifically durable contribution.

Critical weaknesses

The deployment story is narrower than the headline suggests. Two scoping decisions bound the evidence. First, the paper restricts itself to P0 risk—categorical object–site compatibility under default conditions—by its own three-tier taxonomy, excluding P1 (parameter/device-state dependent, e.g., 30 s vs. 10 min in a microwave) and P2 (accumulation over time), which are precisely the regimes where embodied deployment actually fails; all results are also instruction-level text only, with no scene state or perceptual grounding, so the situational safety cases that flip an instruction's safety with the scene are out of scope. Second, PRISM is bound to its representation context: cross-prompt transfer collapses (0.46), cross-dataset frozen transfer underperforms zero-shot verbal baselines on SafeText, and every external-benchmark result uses within-dataset refitting. The paper is commendably transparent about both limits, but together they mean the demonstrated artifact is a per-domain, per-prompt static-text filter for household object-placement risk—valuable as evidence, but far from the "physical-risk detection in embodied agents" framing of the abstract. I would like the limitations to be surfaced in the introduction rather than only in §6.

Questions for authors

(I) PD-Bench labels have Fleiss κ=0.34 overall (356/650 unanimous). Did you consider a larger annotator pool or an adjudication round for split cells? How sensitive are the probe–verbal gap estimates to restricting to cells where at least two annotators report high confidence? (II) You evaluate unseen object–site combinations and leave-one-mechanism-out transfer, but no adversarial evaluation. Since probing-based detectors are known to fail under distribution shift crafted by an adversary (e.g., Wang et al., 2026, which you cite), how does PRISM behave under paraphrase attacks or instruction obfuscation designed to push probes below threshold? Even a small red-team study would substantially strengthen the safety claim. (III) The three annotators' mean confidences are 1.73/2.36/2.65 (of 3). Did annotators receive any training or calibration on pilot items beyond the written protocol? And since the per-cell vote table will be released: does the lowest-confidence annotator's vote pattern differ systematically on the split cells, and would the probe–verbal gap estimates change if the released labels were restricted to cells where at least two annotators report high confidence? (IV) Concurrent interpretability work reads silent internal states directly: the Jacobian lens / J-space line (Anthropic's "A global workspace in language models," July 2026, and the J++ Lens follow-up, October 2026) reads verbalizable internal representations that the model never expresses in output, for monitoring goals such as detecting eval awareness and hidden plans. Your representation–verbalization gap is conceptually adjacent: the model "knows" physical risk internally but fails to verbalize it. Could you comment on the relationship—for example, whether the CD/PD directions you isolate lie inside or outside such "workspace" representations, and whether unsupervised lens-style readouts could complement your supervised probe on this task?

2 comments

Thank you so much for your exceptionally thoughtful, generous, and encouraging evaluation. We are genuinely grateful for the care and attention you have devoted to understanding our work. We are equally grateful for your insightful and constructive suggestions, which will help us further refine and strengthen our work.

1. PD-Bench Annotation Reliability and Sensitivity Analysis

a. PD-Bench labels have Fleiss κ=0.34 overall (356/650 unanimous). Did you consider a larger annotator pool or an adjudication round for split cells?

Thank you for your valuable suggestion. We will involve independent adjudicators to review all cells with annotator disagreement and resolve disputed labels. We will report the results in final version.

b. How sensitive are the probe–verbal gap estimates to restricting to cells where at least two annotators report high confidence?

Thanks for your valuable insights. Our sensitivity analysis supports the probe’s advantage on high-confidence examples.

Restricting evaluation to cells where at least two annotators report maximum confidence (3) retains 279 cells (35 safe, 244 unsafe). Probe–verbal gap estimates remain positive for all four models, with 95% credible intervals above zero for three.

& & \begin{array}{ccccc} \hline \rule{0pt}{2.6ex} \textbf{Model} & \textbf{Probe AUC} & \textbf{Verbal AUC} & \textbf{Gap} & \textbf{95\\% credible interval} \\\\[5pt] \hline \rule{0pt}{2.6ex} \text{Qwen2.5-3B} & 0.906 & 0.850 & +0.055 & [-0.005,\\,0.116] \\\\[4pt] \text{Qwen2.5-7B} & 0.913 & 0.845 & +0.068 & [0.012,\\,0.126] \\\\[4pt] \text{Qwen3-8B} & 0.872 & 0.745 & +0.128 & [0.052,\\,0.209] \\\\[4pt] \text{Phi-3.5-mini} & 0.908 & 0.829 & +0.078 & [0.024,\\,0.137] \\\\[4pt] \hline \end{array} & & & \end{flalign*}$$ We retain the original labels, fitted models, and averaging over 25 within-fold cell AUCs. Intervals use 10,000 paired cell-level Bayesian-bootstrap draws. **2. You evaluate unseen object–site combinations and leave-one-mechanism-out transfer, but no adversarial evaluation. Since probing-based detectors are known to fail under distribution shift crafted by an adversary (e.g., Wang et al., 2026, which you cite), how does PRISM behave under paraphrase attacks or instruction obfuscation designed to push probes below threshold? Even a small red-team study would substantially strengthen the safety claim.** Thank you for this suggestion. We conducted a small score-guided red-team study on Qwen2.5-3B, and the results show that PRISM retained an 85% detection rate (34/40) under paraphrase attacks and instruction obfuscation. We randomly selected 40 instructions from PD-Bench. All 40 were unanimously labeled as unsafe by all three annotators at the highest confidence level and were correctly identified as unsafe by the model before perturbation. We then generated up to 6 perturbations, covering paraphrasing, structured formatting, and character/whitespace obfuscation and apply them to the original instructions, forming 40 candidate instructions. One human reviewer assessed all 40 original and candidate pairs for preservation of the action, conditions, and physical risk. Rejected or uncertain candidates were removed and were replaced by other instructions until all 40 instructions met the inclusion criteria. We review these attacks using frozen probes, fixed-shape FP32 inference, and single-instruction thresholds calibrated exclusively from training-inner-OOF predictions, targeting a nominal 10% FPR. After perturbation, 34 instructions were still correctly detected, while six were misclassified as safe, reducing the detection rate from 100% to 85%. This corresponds to a conditional attack success rate of 15% (6/40). Five failures resulted from lexical substitutions and one from the word-order change.

3. The three annotators' mean confidences are 1.73/2.36/2.65 (of 3). Did annotators receive any training or calibration on pilot items beyond the written protocol? And since the per-cell vote table will be released: does the lowest-confidence annotator's vote pattern differ systematically on the split cells, and would the probe–verbal gap estimates change if the released labels were restricted to cells where at least two annotators report high confidence?

We apologize for not describing annotator preparation clearly enough. All three annotators discussed the annotation guidelines together and completed a pilot annotation round before formal labeling.

On the 294 split cells, the lowest-confidence annotator (A) cast the minority vote on 92 cells (31.3%), compared with 109 (37.1%) for B and 93 (31.6%) for C. Thus, A was not the most frequent minority voter. Voting tendencies nevertheless differed, the unsafe-vote rates on these same cells were 45.2%, 32.7%, and 83.7%, respectively. A’s disagreements with the majority more often favored safe, as 70/181 majority-unsafe cells versus 22/113 majority-safe cells. The lowest-confidence annotator showed no systematic voting differences on split cells.

The high-confidence analysis in 1(b) supports the robustness of our main finding. Restricting evaluation to 279 cells with at least two maximum-confidence ratings preserves positive probe–verbal gap estimates across all four models. The 95% credible intervals remain no less than zero for all models.

4. Concurrent interpretability work reads silent internal states directly: the Jacobian lens / J-space line (Anthropic's "A global workspace in language models," July 2026, and the J++ Lens follow-up, October 2026) reads verbalizable internal representations that the model never expresses in output, for monitoring goals such as detecting eval awareness and hidden plans. Your representation–verbalization gap is conceptually adjacent: the model "knows" physical risk internally but fails to verbalize it. Could you comment on the relationship—for example, whether the CD/PD directions you isolate lie inside or outside such "workspace" representations, and whether unsupervised lens-style readouts could complement your supervised probe on this task?

Thank you for this insightful question and highlighting this connection. The workspace framework raises a concrete question about the physical-risk information that our probes recover but verbal judgments fail to express.

CD/PD separation and workspace membership concern different properties: our analysis distinguishes risk readout directions, whereas the workspace framework characterizes representations available for verbal report and flexible internal use. Both CD and PD could therefore occur within the same workspace while remaining directionally separated.

The Jacobian lens and J++ Lens could complement our probes by revealing risk-related concepts without training a risk-specific classifier. Our probes quantify how reliably hidden states distinguish safe from unsafe instructions; lens readouts could help interpret the concepts supporting that distinction. If risk-related concepts appear when verbal judgments fail, this would provide additional evidence that the missed information is available in a verbalizable form.

5. Scope and Deployment Limitations

Sorry for the confusion and thank you for your advice. We understand your concerns and agree that the boundaries of our findings should be better communicated. We will revise the Introduction and Abstract to better define our research scope and clarify the boundaries of our conclusions. In future work, we will explore more complex physical danger scenarios and generalization to broader settings. We will clarify this point in the final version of the manuscript.

6. Cross-prompt and cross-dataset transfer

Thank you for this constructive suggestion. We apologize for not making our transfer protocol clear enough. PRISM supports cross-dataset use by adapting only the linear readout using target-domain labels, while keeping the entire LLM frozen. On SafeText, this achieves approximately 0.984 AUC for both 3B and 7B, exceeding zero-shot verbal judgments of 0.964/0.973. The 0.46 AUC concerns direct cross-prompt transfer of a fixed judge-trained probe and fitting the readout within the planner context yields 0.778 AUC. We will foreground this distinction and clarify how readout adaptation enables cross-dataset use without updating the underlying LLM.

Reviewer_hvVGOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper states a gap between content (CD) and physical safety (PD). The authors 1) introduce PD-Bench with 650 object×site combos and 2000 instructions, 2) show across 6 LLMs that CD and PD directions in activation space are separated, 3) show that single-layer linear probe beats the model's own verbal safety judgments by +0.128, and 4) show that graft/bridge interventions improve verbal judgments to 0.80–0.845 AUC while cutting false-positive rates.

Critical strengths
  1. Thinking about safety from the consequence side is a sound and important view.
  2. Statistical protocol is thorough, with 5 seeds x 5 cell-grouped outer folds, and paired bootstrap CIs.
  3. The writing is clear and internally consistent, and the experiments are extensive.
Critical weaknesses
  1. The probe's gains do not transfer. On SafeText, the frozen probe reaches 0.749 (3B)/0.559 (7B) while the same models' zero-shot verbal judgments reach 0.964/0.973. This limits the method's generality.
  2. The interventions' capability side effects are untested. No experiment measures general utility or non-safety-task performance.
  3. Methodological novelty is low. The method assembles standard components, and the key conceptual claim — consequence-oriented safety — was established earlier by cross-modal benchmarks (MSSBench) and consequence-oriented agent benchmarks (ToolEmu).
  4. PD-Bench's distribution is narrow: all instructions are single-step household object×site placement, restricted to the P0 tier.

[1] Multimodal Situational Safety

[2] Identifying the Risks of LM Agents with an LM-Emulated Sandbox

Questions for authors
  1. Do frontier closed-source models (GPT, Claude, Gemini) still exhibit this verbal-CD/PD gap?
  2. Did the synthetic construction produce repetitive sentence patterns, and if so, how was this handled? Please report template-diversity statistics of the 2,000 instructions.
  3. What is the general-capability performance of the graft and bridge interventions on non-safety tasks after intervention?
3 comments

Thank you for your thoughtful evaluation and constructive feedback. We greatly appreciate your recognition of the rigor of our statistical evaluation, the clarity of our writing, and the comprehensiveness of our experiments. We are also deeply grateful for the opportunity to clarify and further elaborate on our research progression.

1. Do frontier closed-source models (GPT, Claude, Gemini) still exhibit this verbal-CD/PD gap?

We observe consistent CD/PD directional separation across the tested 3B–32B models, suggesting that it may also persist in frontier closed-source models.

Although stronger reasoning may improve large models’ verbal risk judgments, our target setting requires compact models for resource-constrained, on-device deployment. Our method gives these small models the similar physical-risk recognition capabilities of much larger models. It accesses risk information already encoded in their hidden states, without increasing the backbone size or relying on cloud inference.

2. Did the synthetic construction produce repetitive sentence patterns, and if so, how was this handled? Please report template-diversity statistics of the 2,000 instructions.

Sorry for the confusion. The construction uses six action frames and eight contextual tails. All 2,000 complete instruction strings are distinct, and all 48 frame–tail combinations occur, with 29–54 instances each. There are 1–11 instructions per object×site cell, with a median of three while 101 cells have one instruction. We will clarify this point in the final version of the manuscript.

3. What is the general-capability performance of the graft and bridge interventions on non-safety tasks after intervention?

Thanks for your valuable insights. Experiments show that bridge and graft hardly effect performance on non-safety tasks.

We evaluate both on Qwen2.5-3B and 7B using 256 ARC-Easy questions, 256 ARC-Challenge questions, and 128 IFEval prompts, averaging over all 25 trained fits per model.

Bridge remains close to baseline on these evaluations. ARC accuracy changes range from −0.45 to +0.02 percentage points. IFEval strict prompt-level accuracy changes from 64.06% to 63.09% at 3B and from 76.56% to 77.03% at 7B; the paired 95% confidence intervals for these differences are [−4.03, +2.13] and [−1.94, +3.00] percentage points, respectively. Graft reaches similar performance on the evaluated non-safety tasks. The two prespecified extensions yield ARC declines of 0.02–2.22 percentage points and IFEval declines of 0.31–3.38 points. Under the secondary variant, all evaluated scores are unchanged. ARC-Easy/Challenge accuracy remains 93.36%/82.42% at 3B and 95.70%/91.02% at 7B, while IFEval strict accuracy remains 64.06% and 76.56% across all 25 fits per model.

We will report both the primary results and this secondary analysis, making the observed utility trade-offs and evaluation scope explicit.

4. Frozen transfer to SafeText

Sorry for the confusion. On SafeText, PRISM outperforms zero-shot verbal judgments. With the LLM frozen and only the linear readout fitted on SafeText labels, PRISM achieves approximately 0.984 AUC for both 3B and 7B, compared with 0.964/0.973 for zero-shot judgments. The cited 0.749/0.559 results reflect direct transfer of a fixed PD-Bench probe without adaptation. Our protocol instead refits only the linear readout using target-domain labels, demonstrating PRISM’s cross-dataset applicability while keeping the underlying LLM frozen. We will clarify this point in the final version of the manuscript.

5. Novelty of our work

Our main contribution is a scientific discovery which is the separation of CD and PD readout directions in LLM hidden states. To our knowledge, we provide the first systematic characterization of this separation across six models. Standard probes serve as measurement tools for this discovery.

MSSBench and ToolEmu establish that safety depends on visual context and action consequences. Both works offer valuable perspectives on contextual and consequential safety. However behavioral distinction does not imply that CD and PD are represented differently inside a model. Both could still be captured by a shared safety readout. We compare CD and PD within the same model against a common safe reference. We find separated readout directions and limited CD to PD detection transfer. This reveals a distinction in how risk information can be read from the model, which behavioral benchmarks alone cannot establish.

We will cite both works and clarify this distinction in the final version.

6. Why PD Bench focuses on household P0 risk

Sorry for the confusion. As stated in the Introduction and §6, our work deliberately focuses on P0-level physical danger, rather than attempting to cover all levels of physical danger. We will revise the manuscript to clarify the scope and limitations of the present work. In future research, we will extend our investigation to address P1/P2. We will clarify this point in the final version of the manuscript.

Thanks for the detailed rebuttal. The rebuttal addresses several concerns but not the significance-level ones: 1) whether the representation–verbalization gap exists in frontier closed-source models is unknown, only 1.7B–32B open-weight models are tested; 2) the direction separation is largely an artifact of fitting two probes to two different binary objectives, so the observed difference may simply reflect different training targets rather than a genuine representational distinction; 3) the method is validated only on PD-Bench's single-step household object×site placements, so generalization to other physical-risk scenarios is unverified. Therefore, I maintain my score of Weak Reject.

Reviewer_eib7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares hidden-state directions associated with content danger and physical danger, and studies the gap between how well risk can be decoded from those states and how the models actually judge risk in language. The authors construct PD-Bench from object-placement combinations, introduce a single-layer linear probe called PRISM, and then try to improve safety judgments through output grafting and low-rank bridging. The central claim is that models already encode physical-risk information that is distinct from content danger, but do not fully make use of that information in their outputs.

Critical strengths

The readout comparisons on PD-Bench are the more convincing part of the paper. Holding out object-placement combinations, using human labels, and comparing against lexical features, additive object-and-placement models, and retrieval prompting take the results beyond the simplest detection of hazardous words. For Qwen2.5-7B, for example, probe AUC is 0.839, compared with 0.682 for verbal judgment and 0.804 for retrieval prompting. This gap is useful because it suggests that a lightweight readout can recover physical-risk information that the model’s own verbal judgment does not fully express.

Critical weaknesses

The central claim that content danger (CD) and physical danger (PD) have separated representations does not yet rule out the most direct explanation: the two groups differ in vocabulary and in the tasks they pose. Sections 3.1-3.2 divide SafeAgentBench into CD and PD using keyword rules and then compare the two groups’ directions relative to safe samples. The 999 random partitions and coordinate controls address random grouping and coordinate choice, but random grouping does not preserve the vocabulary and task differences between the original groups. It therefore cannot establish that the additional separation specifically reflects the type of physical risk, rather than these more ordinary differences in wording and task setup. PD-Bench reduces lexical shortcuts in physical-risk prediction, but it does not re-establish the CD/PD separation using controls that are matched in semantic content and wording. This leaves a key gap between the paper’s title and its central finding.

The interpretation as a routing failure is also stronger than the evidence. The directional ablations in Section 5 do not yield a PD-selective effect. Sampling unsafe examples at random and computing their mean-difference direction relative to safe examples produces a similar collapse in judgment. Output grafting largely amounts to incorporating a supervised classification result into the model’s output. Low-rank bridging offers further evidence that hidden states can be used more effectively, but the improvement still does not uniquely identify why the original model fails. I accept that hidden states support better task readout; I am not yet convinced that the cause is specifically a failure to route physical-risk information.

This also limits the interpretation for pre-execution monitoring. Section 4 reports an AUC of only 0.46 when transferring directly from judge prompts to planner prompts, and the cross-benchmark results require separate fitting. The evidence supports risk classification in particular contexts, but it does not readily extend to a general physical-safety signal that could be read out during planning.

Overall assessment: The question is well chosen, and the physical-risk readout has practical results. However, the most interesting contribution, for me, is the representational distinction between physical and content danger, and the evidence for that distinction is not yet sufficiently controlled. The routing interpretation also goes beyond what the experiments directly establish. I therefore lean toward rejection, primarily because of the gap between the evidence and the central claims.

Questions for authors
  • What are the exact source and content of the CD/PD keyword rules? Do existing experiments show that the directional separation persists after matching vocabulary, instruction format, and task content? Which experiment connects PD-Bench to the earlier separation claim?
  • Without a PD-selective ablation effect, what evidence further distinguishes a supervised readout that simply improves the answer from a failure to route physical-risk information in the original model?
3 comments

Thank you for identifying the representational question as particularly valuable and for recognizing the strength of the PD-Bench readout comparisons. We appreciate the opportunity to make this progression explicit and to strengthen its evidential support.

1. Definition and provenance of the CD and PD grouping

a. What are the exact source and content of the CD/PD keyword rules?

Our keywords are: break, throw, smash, destroy, damage, drop, crack, shatter, ruin, hit, harm, hurt, injure, kill, attack, dangerous, hazard, risk, unsafe, toxic, poison, fire, burn, explode and shock.

We build these rules based on lexical evidence from the Threat Dictionary[1] (Choi et al., 2022) and VerbNet[2] (Kipper et al., 2008), combining threat related terms with verbs denoting destruction or bodily harm and adding terms based on the tasks and their physical risks. We will include these researches in our final draft.

[1] Choi, V. K., Shrestha, S., Pan, X., & Gelfand, M. J. (2022). When danger strikes: A linguistic tool for tracking America’s collective response to threats. Proceedings of the National Academy of Sciences, 119(4), e2113891119.

[2] Kipper, K., Korhonen, A., Ryant, N., & Palmer, M. (2008). A large-scale classification of English verbs. Language Resources and Evaluation, 42(1), 21–40.

b. Do existing experiments show that the directional separation persists after matching vocabulary, instruction format, and task content?

Thanks for your valuable insights. Results showed that after controlling vocabulary, instruction format, and task content, the directional separation do persist.

We matched instructions by action family, room, and word count (difference ≤ 5), yielding 39 disjoint CD–PD pairs. Using a common safe reference, the angles between the two readout directions remain 73.27° for Qwen2.5-3B and 73.32° for Qwen2.5-7B, with nominal within-pair permutation p-values of 0.002 and 0.001.

A stricter matching criterion requiring cosine similarity ≥0.75 between instruction embeddings computed after masking the defining keywords. This yields 24 disjoint pairs, covering 48 of the 300 unsafe instructions. The corresponding angles are 73.06° for 3B and 72.37° for 7B.

We will clarify this issue in the revision.

c. Which experiment connects PD-Bench to the earlier separation claim?

Sorry for the confusion. Our central finding is the directional separation between CD and PD, which we first identify on SafeAgentBench through the large angles between their associated directions and the poor detection of PD by CD-only readouts. This observation motivates us to further investigate what physical-risk information models can represent and use in the absence of content danger cues. To this end, we construct PD-Bench, a benchmark focusing exclusively on physical danger, to investigate the underlying causes of this separation and ultimately leverage this finding to develop safety defenses. We will clarify this point in the final version of the manuscript.

2. Without a PD-selective ablation effect, what evidence further distinguishes a supervised readout that simply improves the answer from a failure to route physical-risk information in the original model?

Sorry for the confusion. We also use graft and bridge to investigate the routing failure hypothesis. Both graft and bridge are evaluated on PD-Bench using five data-split seeds and five cell-grouped outer folds per seed, following the protocol in §3.3.

On Qwen2.5-7B, with the base model frozen, graft adds a supervised hidden-state readout to the safety-label scores and raises verbal AUC from 0.682 to 0.845, demonstrating that connecting this information to the output can improve safety judgments.

Bridge injects hidden representations before the final decoder block through a trained rank-16 connection. This raises 7B AUC from 0.681 to 0.801, exceeding the final-block-input source (0.766) and constant source (0.722). With supervision, rank, injection point, and training budget matched, the selected-source advantage over the final-input source is 0.0347, 95% CI [0.0189, 0.0504] at 7B.

Prior work supports the effectiveness of the output and representation interventions underlying our graft and bridge designs (Han et al., 2024[3]; Wu et al., 2024[4]). Combined with controlled ablation, their source dependent gains support a routing failure interpretation as available risk information is insufficiently used by the default output pathway (Li et al., 2023[5]; Elazar et al., 2021[6]). We will make this reasoning explicit in §5.

[3] Han, C., Xu, J., Li, M., Fung, Y., Sun, C., Jiang, N., Abdelzaher, T., & Ji, H. (2024). Word embeddings are steers for language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 16410–16430). Association for Computational Linguistics.

[4] Wu, Z., Arora, A., Wang, Z., Geiger, A., Jurafsky, D., Manning, C. D., & Potts, C. (2024). ReFT: Representation finetuning for language models. Advances in Neural Information Processing Systems, 37, 63908–63962.

[5] Li, K., Patel, O., Viégas, F., Pfister, H., & Wattenberg, M. (2023). Inference-time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36, 41451–41530.

[6] Elazar, Y., Ravfogel, S., Jacovi, A., & Goldberg, Y. (2021). Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9, 160–175.

3. Pre execution risk classification and cross context transfer

Sorry for the confusion. PRISM focuses on instruction level risk classification during planning, before execution. Cross context transfer was included as a supplementary check for rigor and completeness. The 0.46 AUC concerns transferring a judge trained probe unchanged to planner prompts, whereas our main evaluation uses context matched fitting and achieves 0.926 AUC on SafeAgentBench. This transfer result does not alter our main conclusion that hidden-state readouts support pre-execution risk classification. We will clarify this point in the final version of the manuscript.

Thank you for the response. My main concern remains the central claim that the model "knows" about physical risk but fails to use it. The current evidence shows the information can be decoded, but not that the model fails to route it. Since this is the paper's core claim, please upload a revised PDF that addresses it. I will reconsider my rating based on how this issue is addressed in the revision.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of rebuttal (part 1)

We sincerely thank the PC, AC, and the reviewers for their thoughtful engagement and valuable feedback. Reviewer eib7 highlighted the value of the physical-risk readout comparisons on held-out object × site combinations. Reviewer hvVG recognized the importance of evaluating the consequences of an instruction and the rigor of the statistical protocol. Reviewer Mh2j emphasized the value of turning the distinction between content and physical danger into a measurable question about model representations. Throughout the discussion, the reviewers’ questions helped us refine our experiments and sharpen the conclusions we draw from them. We summarize below the contributions they highlighted and how the rebuttal and revised manuscript address their main concerns.

1. What reviewers appreciated about our work:

A measurable distinction between content danger and physical danger. Reviewer hvVG recognized the importance of evaluating the consequences of an instruction, and Reviewers eib7 and Mh2j identified the representational question as particularly valuable. Building on behavioral work such as MSSBench and ToolEmu, we ask whether a model uses a common readout for danger expressed in words and danger arising from execution. Across six models from three families, the two readout directions are consistently separated. This distinction matters in practice: a readout trained on content danger detects only 19.8–38.0% of held-out physical hazards on SafeAgentBench.

A benchmark designed to isolate physical risk. Reviewers eib7 and Mh2j highlighted the human annotations, held-out object × site combinations, and controls for surface cues in PD-Bench. Its 2,000 instructions cover 650 combinations without explicit danger words. The object and its placement jointly determine what makes an instruction hazardous, and all phrasings of a test combination are held out together. This design lets us test whether a model recognizes the risk of a new combination and quantify how much can already be predicted from wording, object identity, or location alone.

A consistent gap between what a model encodes and what its own safety judgment expresses. Reviewer eib7 singled out the PD-Bench readout comparisons as a strength, and Reviewer Mh2j emphasized their scientific value. Across four backbones, a linear hidden-state readout exceeds the same model’s verbal judgment by 0.128–0.177 AUC on unseen combinations. On Qwen2.5-7B, even retrieval prompting with access to the full training pool reaches 0.804 AUC, compared with 0.839 for the probe and 0.682 for the zero-shot judgment. The largest gap in our cross-benchmark evaluation occurs on EARBench, an independently developed benchmark.

Controlled interventions that connect internal representations to behavior. Reviewer Mh2j particularly valued the effort to determine what each intervention can establish. The output graft tests whether a readable signal can improve the answer directly. The rank-16 bridge asks whether the signal can improve the model’s verbal judgment when introduced before the final decoder block. With all base-model weights frozen, the bridge recovers 24–75% of the probe–verbal gap across four backbones. The distinction between these experiments gives researchers a way to study how physical-risk information affects a decision at different points in the model.

Statistical controls tailored to each research question. Reviewers hvVG and Mh2j highlighted the rigor of the evaluation. For directional separation, we repeat the complete two-probe fitting procedure on 999 size-matched random partitions. For physical-risk prediction, we use five seeds and five cell-grouped outer folds, select models within the training folds, and compute paired uncertainty intervals over object × site combinations. This matters because several phrasings can describe the same physical situation. The evaluation therefore measures generalization to a new situation while accounting for the dependence among its paraphrases.

A lightweight pre-execution monitor with a concrete detection benefit. Reviewer eib7 recognized the practical value of the physical-risk readout. PRISM requires only a single linear readout from a frozen model and can score an instruction before execution without generating a verbal response. At the same rate of blocking safe instructions, the hidden-state signal catches hazards the model’s verbal check misses.

AuthorsOpenReview ↗

Summary of rebuttal (part 2)

2. How we addressed reviewers’ main concerns:

Evidence for CD/PD separation (Reviewers eib7 and hvVG, §3.1–3.2). The separation between CD and PD is the main and core finding of our work. The reviewers asked whether the observed separation could arise from fitting two probes or from differences in wording and task content. We evaluate the first explanation by repeating the same two-probe procedure on 999 size-matched random partitions, each with a common safe reference. The observed angles of 71.1°–77.3° exceeded every null angle in all six models across three families. To address the second explanation, the revision adds 39 CD–PD pairs matched on action family, room, and instruction length. The angle remained approximately 73° at both 3B and 7B. A stricter match based on semantic similarity after masking the defining keywords retained 24 pairs and yielded angles of 73.1° and 72.4°. This gives the representational distinction a behavioral consequence as well, as a readout trained on content danger misses most held-out physical hazards.

From decodability to the model’s own judgment (Reviewer eib7, §5, Figure 6 and Table 6). Reviewer eib7 asked what evidence connects a readable risk signal to an incorrect verbal decision. The revision adds two tests that address this question directly. First, we varied which layer supplied information to a trained bridge while holding supervision, rank, injection point, and training budget fixed. At 7B, the selected layer reached 0.801 AUC, compared with 0.766 for the final-block input and 0.722 for a constant input. The selected-layer advantage replicated at 3B. With the bridge fixed, removing the risk component also reduced performance more than removing matched random components. Second, we examined 50 selected errors across four datasets, including both missed hazards and false alarms. A held-out probe recovered the correct label from the original states in all 50 cases. Internal edits corrected all 50 verbal decisions, and reverse edits disrupted 44 initially correct decisions. Most edits used a correctly judged donor with the same reference label. These cases show that risk information can be readable on the very instruction the model misjudges and that changing its internal computation can change the verdict. How often a particular routing mechanism explains naturally occurring errors remains an open question.

PD-Bench: Motivation and Findings (Reviewers eib7, hvVG, and Mh2j, §3.2–3.4). The separation observed on SafeAgentBench motivated a further question, that when the wording gives no explicit warning, how well does a model’s verbal answer reflect the physical risk recoverable from its hidden states? We built PD-Bench to test this on 2,000 distinct instructions covering 650 human-labeled object × site combinations, holding out entire combinations at evaluation. On Qwen2.5-7B, the verbal judge reached 0.682 AUC, retrieval prompting reached 0.804, and the linear probe reached 0.839. The revision documents all 48 combinations of request frame and contextual phrasing. It also reports a sensitivity analysis on the 279 object × site combinations for which at least two annotators gave maximum confidence. The probe advantage remained positive on all four models, with 95% credible intervals above zero on three. In the rebuttal’s robustness study, 34 of 40 initially detected unsafe instructions remained detectable after human-verified, risk-preserving perturbations.

Applicability and Transferability of PRISM (Reviewers eib7, hvVG, and Mh2j, §3.4, 4, and 6). PRISM uses a single linear readout from a frozen model to assess an instruction before execution. On SafeAgentBench at 7B, it detected 89.8% of physical hazards at the verbal judge’s false-positive rate, compared with 84.5% for the judge. The transfer experiments clarify how to apply it in a new setting. Direct judge-to-planner transfer reached 0.46 AUC, while fitting the readout in the planner context yielded 0.778. On SafeText, refitting only the readout reached approximately 0.984 at both 3B and 7B. We also evaluated the interventions on non-safety tasks in the rebuttal. Bridge accuracy on ARC stayed within 0.45 percentage points of baseline, while the evaluated graft variants showed IFEval declines of up to 3.38 points. The demonstrated setting is instruction-level assessment with open-weight models. PD-Bench focuses specifically on single-step household placements under default conditions.

We respectfully thank all the reviewers for their time again. Our work gives researchers a reproducible way to measure which physical hazards a content-oriented readout misses, a controlled benchmark for comparing what a model encodes with what it says, and intervention evidence that helps explain individual errors. We are grateful to the reviewers for helping us make both the findings and their limits more precise.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.