acceptodds
Under review as a conference paper at ICLR 2027

Word Substitution Bypasses Safety Monitors Despite Explicit Mappings

Abstract

Word substitution can bypass language-model safeguards by replacing sensitive terms with harmless-looking words, even in some of the recent advanced models. External safety monitors add another layer of defense, yet it remains unclear whether they can recognize harmful content even when told what the substituted words mean. To further investigate this issue, we conducted a comprehensive and systematic study of this phenomenon. Our empirical study first reveals that, across seven generation services, substitution-based attacks persist against some frontier models; seven monitors show 13.3–76.7 percentage-point alarm reductions on paired held-out inputs despite explicit meanings; and restoring questions increases alarms at least as much as restoring answers. Also, in an open-weight monitor, swapping codeword meanings while fixing vocabulary, questions, and answers yields distinguishable hidden states but unchanged safety decisions: mappings are linearly separable on unseen questions, while scores shift but remain below threshold. Explicit harmful concepts instead trigger some alarms. On generated answers, templates, output prefixes, and general instructions receive 92% of direct decision-position attention, with similar aggregate allocations across mappings. Demonstrations improve detection on unseen questions, and lightweight fine-tuning improves risk ranking under matched inference settings. Identical warnings yield different harmful-delivery outcomes depending on whether triggering text is withheld. Our findings show that explicit word meanings and distinguishable internal states are insufficient for reliable safety monitoring, informing how external monitors should be evaluated and improved.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, AI Safety, Jailbreak Attacks, Adversarial Robustness, Mechanistic Interpretability, Safety Evaluation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_TXwzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether external safety monitors detect harmful content expressed through word substitution, even when the substitutions are explicitly defined. It compares fixed generated answers under substituted and restored wording, separates request- and answer-wording effects, and examines mapping-dependent hidden states and decisions. Additional experiments investigate demonstrations, fine-tuning, sequential alarms, and output-release policies. Across seven monitor configurations, substituted inputs receive fewer alarms on independently selected requests, although the effects vary substantially across monitors and evaluation settings.

Critical strengths
  1. Careful controls make the behavioral finding convincing: Holding generated answers fixed avoids confusing wording effects with differences in generated content. The crossed request/answer comparisons and unrelated-answer-word control further show that an alarm under an explicit request does not establish interpretation of the substituted answer. On the 30 new requests, all seven monitors show lower alarm rates under substitution, with five comparisons surviving Holm correction.

  2. The paper separates internal sensitivity, detection, and actual protection: Mapping changes produce distinguishable hidden states and score changes without necessarily changing decisions. The authors avoid interpreting this as proof of comprehension and report that activation interventions fail to establish mapping specificity. Likewise, the release-policy experiment holds warnings fixed, isolating the consequences of withholding flagged text. This restraint makes the diagnostic conclusions more credible.

Overall, the paper addresses a critical and timely vulnerability in external safety guardrails. The empirical evaluation across seven safety monitors is thorough and well-controlled, and the internal probing experiments provide insightful evidence of a disconnect between internal concept representation and final safety decision thresholds. The inclusion of practical mitigation analyses further strengthens its value to the AI safety community.

Critical weaknesses
  1. The monitor evaluation is narrow despite covering several configurations - The independent harmful set contains only 30 responses, 26 from DeepSeek, and is selected using automated harmfulness judgments. Seven configurations represent four monitor families with different policy coverage and intended inputs. These results establish a vulnerability on the tested examples, but provide limited evidence about its prevalence across generators, harm categories, and realistic benign traffic.

  2. The internal analysis establishes distinguishability rather than a mechanism for mapping failure - The probe uses 24 constructed pairs with the target codeword always in the first mapping row, allowing classification from the concept at that position. Attention measurements are descriptive, and controlled activation replacements do not identify a mapping-specific effect. The authors acknowledge these limitations, but the analysis consequently leaves unresolved why contextual information fails to produce the appropriate decision.

  3. The mitigation evidence remains preliminary - Fine-tuning is evaluated on 24 harmful examples and 20 controls; benign controls are also much shorter than harmful answers. Under matched inference, only one of three seeds in each training scheme has an AUC-gain interval excluding zero. Paired substituted/restored training shows no clear advantage over repeated restored examples. Thus, the experiments suggest adaptation can help, but do not establish a robust substitution-specific defense.

  4. The harmful-delivery estimates depend on a potentially vulnerable evaluator - Released prefixes are judged by Llama Guard 3, which the paper itself shows is sensitive to substituted wording. The authors disclose this dependency, but it leaves uncertainty about absolute delivery counts and whether apparently safe released prefixes contain recoverable harmful content. An independent human audit would strengthen this part of the study.

Questions for authors
  1. Does the paired detection gap persist on a larger, generator-balanced set with human-verified harmfulness and monitor-policy applicability?

  2. Does probe performance persist when mapping-row order, codeword identity, and harmful-concept identity are held out or counterbalanced?

  3. Do adaptation gains survive length-matched benign controls and evaluation at a fixed false-positive rate?

  4. How often do human judgments disagree with Llama Guard’s released-prefix labels, particularly under substitution?

5 comments
AuthorsOpenReview ↗

Response 1 of 4

Thank you for your thoughtful review of our work. We appreciate your attention to generator balance and to the judgments of harmfulness and policy applicability. We have revised certain sections of the paper as well as the appendix based on the reviews.

W1 and Q1. Generator balance and qualification

We evaluated 132 native responses from 59 requests and 78 generator–request pairs across six generators. The response counts were DeepSeek (65), Grok (45), Qwen (13), GLM (5), GPT Sol (3), and GPT Terra (1). Two model judges independently assigned the highest harmfulness label to the exact restored prefix scored by the monitors. Human harmfulness and policy-applicability ratings were not collected for this comparison. Table 1 gives the following alarm counts.

| Monitor | Substitution alarms /132 | Restoration alarms /132 | Difference (pp) |

| Llama Guard 3 | 75 | 116 | +31.06 |

| Qwen3Guard-0.6B | 56 | 122 | +50.00 |

| Qwen3Guard-4B | 60 | 124 | +48.48 |

| Qwen3Guard-8B | 76 | 123 | +35.61 |

| ShieldGemma-2B | 33 | 67 | +25.76 |

| ReasoningShield-1B | 77 | 88 | +8.33 |

| ReasoningShield-3B | 86 | 99 | +9.85 |

All seven monitors flag more restored responses. Restricting the comparison to the 124 responses with explicit mappings gives increases of 8.06–50.81 percentage points. DeepSeek and Grok contribute 110 of the 132 responses, so we also selected three requests for each of the five generators with at least three distinct requests. This gives 22 responses from 15 generator–request pairs and 13 distinct requests. After weighting the five generators equally, the gain is 55.33 points for Llama Guard, 58.00/66.00/67.33 for the three Qwen3Guard sizes, 20.00/29.33 for the two ReasoningShield sizes, and 26.67 for ShieldGemma. Llama Guard and all three Qwen3Guard sizes show the same direction in each of the five generators (Appendix C.2).

We screened 4,985 published HarmBench and JailbreakBench responses for overlap and retained 2,225 across 119 request clusters. Both judges labeled 500 of the retained responses directly harmful. These 500 span 113 request clusters and eight generator versions. ShieldGemma flagged 285/500 (57.0%), Qwen3Guard-4B flagged 446/500 (89.2%), and ReasoningShield-1B flagged 415/499 valid outputs (one parsing failure). The 500 include three Claude-2.1 and four GPT-4-0125-preview responses. Each monitor received the released attack prompt and the full published answer (Table 7).

We evaluated 819 control pairs across 813 distinct request identifiers, including 240 synthetic benign, 156 refusal, 195 metaphor, and 228 definition/fictional pairs. Table 1 shows where wording affects benign alarms. Qwen3Guard-4B flags 75/195 substituted metaphors and 13/195 restored metaphors. On definitions and fictional contexts the counts are 126/228 and 77/228. Six monitors passed 8/8 control calibration checks. ReasoningShield-3B passed 7/8.

The seven monitor configurations span four families, including three Qwen3Guard scales and two ReasoningShield scales. Each paired comparison holds the policy and monitor fixed. Appendix C.1 gives the native alarm rules and explains how we used ReasoningShield's reasoning-trace interface to assess answers.

AuthorsOpenReview ↗

Response 2 of 4

W2 and Q2. Counterbalanced probe

We trained the probe with the target in one mapping row and evaluated it in the other. We also held out base pairs with unseen codewords and concepts. Separate tests held out entire behavior families or request templates.

The assay contains 216 base pairs from 18 behavior families × 12 templates, expanded to 1,728 factor conditions, plus 432 explicit-concept controls. The primary readout uses layer 22 of Qwen3-14B with a 20-dimensional PCA fitted within the training fold and a linear classifier.

The probe achieves an AUC of 0.9109 on held-out base pairs with unseen concepts and codewords. Cross-row transfer yields an AUC of 0.5949 when training on row 1 and evaluating on row 2, and 0.5811 in the reverse direction. Training on queries for codeword A and testing on B gives 0.9150, and the reverse direction gives 0.9017. Holding out behavior families and request templates gives AUCs of 0.8982 and 0.8988. Figure 4(a) shows these comparisons.

Using all 5,120 hidden dimensions gives cross-row AUCs of 0.4592 and 0.6320. Under both readouts, moving the target to the other row produces the largest drop in transfer performance.

In the input-intervention evaluation (Section 4.2), we held the query, answer, all non-table text, and the right-column concept order fixed. We exchanged only the two codewords in the left column. The harmful assignment scored higher in 864/864 pairs, by 10.6603 on average. The score follows which concept the queried codeword denotes when the concept positions are held fixed.

Reversing the rows preserved the mapping, query, answer, token counts, sequence length, and decision position. In the score-distribution pass, 102/432 harmful-target pairs and 0/432 benign-target pairs changed decisions. In the input-intervention baseline, the corresponding counts were 102/432 and 1/432, or 103/864 pairs in total (Appendix D.3). The 103 pairs span 77 base pairs and all 18 behavior families.

AuthorsOpenReview ↗

Response 3 of 4

W2 and Q2 (continued). Score calibration and state interventions

With the same monitor instruction and native alarm rule (d > 0), explicit harmful controls triggered 7/24 alarms in the single-template lexicon set and 210/216 in the 12-template scenario set. We selected both sets independently of monitor scores (Appendices D.1–D.3).

Figure 4(b) shows the score-distribution pass on the 216 counterbalanced base pairs. For coded targets, the harmful and benign medians are −2.00 and −12.75 (AUC 0.9618). For explicitly named targets, the medians are +14.25 and −19.00 (AUC 0.9971). At the original threshold of zero, coded harmful and benign targets trigger 290/864 and 0/864 alarms, compared with 210/216 and 2/216 for explicit targets (Table 13).

We recalibrated the threshold on these same scores (Appendix D.3). We selected a threshold on 17 behavior families and tested it on the held-out family, repeating this for each of the 18 families. At a 5% benign-alarm calibration target, 683/864 coded harmful targets and 43/864 coded benign targets trigger an alarm.

Moving the queried concept from row 1 to row 2 raises the midpoint of the harmful and benign scores by 2.3568 on average and widens their separation by 1.0353. Both scores rise on average. In the score-distribution pass, the native threshold gave 103/432 harmful-target alarms in row 1 and 187/432 in row 2. Ranking changes little across the two positions (AUC 0.9651 and 0.9656).

We also held 89 answer strings fixed and changed only the accompanying request. With the coded and original requests, respectively, Llama Guard 3 flags 52 and 74 answers. Under the coded and original questions, alarm counts were 36 and 86 for Qwen3Guard-0.6B, 46 and 80 for 4B, 50 and 75 for 8B, 8 and 37 for ShieldGemma, 57 and 61 for ReasoningShield-1B, and 62 and 67 for 3B (Table 1). In a separate comparison on all 132 paired responses, restoring only the question raises more alarms than restoring only the answer for every monitor. For Qwen3Guard-4B, the count rises from 60 with both fields substituted to 69 when only the answer is restored, 121 when only the question is restored, and 124 when both are restored (Figure 3). With the original question and unrelated words in place of the answer codewords, it still flags 117 responses.

At layer 22 we compared two donors that preserve the target class. One exchanges the codeword assignment and queried key together, and the other reverses the rows. Across 1,728 conditions, the mean assignment-minus-row donor contrast is 0.00362 when both donors are scored in the same direction. Replacing the mapping-span states leaves the recipient's other states at that layer intact.

We wrote identical layer-22 donor states at two destinations on 864 conditions from 108 base pairs and weighted the 18 families equally (Appendix D.6.3). Assignment and row-order donors produce mean score changes of +0.3692 and +0.3695 at the original mapping span, compared with −0.2520 and −0.2653 at an equal-length output-template span. The differences between destinations are +0.6212 and +0.6348, positive in all 18 families. Moving the recipient's own mapping states into the template also changes the score (−0.3947). The effect of the replacement therefore depends on where the states are written for both donor types.

Reviewer_hhKAOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines whether safety monitors detect harmful answers expressed through word substitution when the mapping to the original terms is explicitly provided. The central experiment compares the same generated answers under substituted and restored wording across seven monitor configurations. Further experiments vary request and answer wording separately, examine internal representations in Qwen3-14B, and evaluate demonstrations, fine-tuning, and streaming release policies.

Critical strengths
  1. The paired evaluation provides convincing evidence of sensitivity to wording. Reusing each generated answer allows a direct comparison between substituted and explicit expressions of its content. On the independent set of 30 questions, restoration increases alarm rates by 13.3–76.7 percentage points, with five of seven comparisons remaining significant after Holm correction (Table 1). Replication on questions selected before monitor evaluation strengthens the central finding.

  2. The paper includes useful controls and reports their implications carefully. Full-answer and token-level evaluations extend the analysis beyond the initial truncated inputs, while the request/answer experiments show how much detection depends on the surrounding request. The negative results are also informative. Under matched precision and scoring, paired-wording augmentation has no established advantage over repeating restored examples. The activation interventions likewise fail to identify an effect specific to mapping positions once donor and replacement-count differences are controlled.

Critical weaknesses
  1. The internal analysis provides limited evidence about the source of the detection failure. As acknowledged in Section 4.1, the queried codeword always occupies the first mapping row. A probe can therefore distinguish the conditions using the concept in that position, leaving the perfect discrimination result compatible with a positional shortcut. The monitor also detects only 7 of the 24 explicitly harmful controls, which makes decision calibration a plausible contributor to the observed behavior. These experiments leave open whether the monitor fails to resolve the substitution or fails to use the resolved meaning in its safety judgment.

  2. The mitigation experiments have evaluation limitations that affect their practical interpretation. The adaptation test contains harmful responses from only 12 held-out questions, and the benign answers are much shorter than the harmful answers: median lengths are 107 and 696 characters, respectively (Appendix E.1). Answer construction could therefore contribute to the measured discrimination. In the streaming experiment, Llama Guard 3 judges the released prefixes (Appendix F.3). Its sensitivity to substituted wording is established earlier in the paper, so it may also miss harmful content in those prefixes. Independent adjudication would strengthen the estimated benefit of buffering.

Questions for authors
  1. Does the probe result persist after counterbalancing both mapping-row order and the queried codeword? Score distributions on these counterbalanced pairs would also help distinguish discrimination from threshold calibration.

  2. Do the adaptation gains persist when benign and harmful answers are matched for length and construction, and detection is compared at a common false-positive rate?

  3. Can the released prefixes responsible for the difference between buffering and immediate release be checked independently, through another judge or manual adjudication?

7 comments
AuthorsOpenReview ↗

Response 1 of 3

Thank you for your careful review of our paper. We appreciate your suggestion to vary the target codeword’s position and examine the monitor’s scores alongside the probe results. This lets us ask two distinct questions. Does the monitor track which concept the codeword denotes? How does that assignment affect its safety decision? We have revised certain sections of the paper as well as the appendix based on the reviews.

W1 and Q1. Probe transfer and alarm calibration

We tested the probe on held-out codewords, concepts, and row positions. We then used the monitor’s own scores to measure the effect of changing the mapping and of calibrating its alarm threshold.

We crossed concept assignment, queried codeword, and row order for 216 base pairs (18 behavior families × 12 request templates), giving 1,728 coded inputs and 432 explicit-concept controls. At layer 22 of Qwen3-14B, we fitted PCA and a linear classifier within each training fold. With 20 components, AUC was 0.9109 on held-out base pairs, where both named concepts and both codewords were unseen. Holding out behavior families gave 0.8982, and holding out request templates gave 0.8988. Training on queries for codeword A and testing on B gave 0.9150, and the reverse gave 0.9017.

Training on targets in the first row and testing on targets in the second gave AUC 0.5949; reversing the rows gave 0.5811. Keeping all 5,120 hidden dimensions gave 0.4592 and 0.6320 on the same two tests. The first-row construction you identified allowed the probe to use the concept at a fixed position. Figure 4a now shows the two cross-row tests alongside the other holdouts, and Appendix D.2 gives the training and test split for each.

At the monitor’s native alarm threshold (score > 0), the judgments were:

| Condition | Harmful-concept alarms | Benign-concept alarms |

| Mapping factor conditions | 290 / 864 | 0 / 864 |

| Explicit-concept controls | 210 / 216 | 2 / 216 |

The monitor’s own scores rank harmful above benign coded targets with AUC 0.9618. Their median scores are −2.00 and −12.75, respectively. We kept those scores fixed and fitted an alarm threshold on 17 behavior families, testing on the held-out family in turn. At a 5% benign-alarm calibration target, 683/864 harmful and 43/864 benign targets triggered alarms. We added this held-out calibration to Appendix D.3 to show what changes when only the decision rule varies.

Moving the queried concept from row 1 to row 2 raised the mean midpoint of the paired harmful and benign scores by 2.3568, and their mean separation by 1.0353. At the native threshold, 103/432 harmful targets triggered alarms in row 1 and 187/432 in row 2. Their row-specific AUCs were 0.9651 and 0.9656. The shift in score midpoint helps explain the higher alarm rate in row 2.

AuthorsOpenReview ↗

Response 2 of 3

W1 and Q1 (continued). Mapping and state interventions

To test whether the declared association affects the monitor’s own score, we held the query, answer, all non-table text, and the right-column concept order fixed and changed only the left-column code-to-concept assignment. All 864 distinct unordered pairs move in the target-concept direction, with a mean score difference of 10.6603. The monitor therefore assigns a higher score to the harmful meaning when the concept words stay in the same positions.

Within that evaluation, a row-order comparison uses 864 pairs that preserve the mapping, query, answer, total length, decision position, and token multiset. Changing only the row order produces different binary decisions in 103 pairs, spanning 77 base groups and all 18 behavior families. These comprise 102 harmful-target pairs and one benign-target pair; the mean absolute score difference is 3.0388. Section 7 now separates the score change due to assignment from the change due to reordering equivalent definitions.

On generated answers, we held 89 answer strings fixed across 50 requests and changed only the accompanying question. The coded question reduced alarms by 24.72 percentage points for Llama Guard 3, by 56.18, 38.20, and 28.09 points for Qwen3Guard-0.6B, 4B, and 8B, by 32.58 points for ShieldGemma, and by 4.49 and 5.62 points for ReasoningShield-1B and 3B. Table 1 reports the alarm counts. Five of the seven paired comparisons remained significant after Holm correction. Changing the question alone was sufficient to change detection of identical answers.

At layer 22, replacing mapping-span states with a row-order donor or an assignment donor flipped 8 and 9 alarms, respectively, across 1,728 conditions. Using the same counterfactual direction for both donors, the mean assignment-minus-row contrast was 0.00362. Changing an input recomputes the entire representation. A local replacement retains the recipient’s other states at that layer.

As a control on the local intervention, we tested identical layer-22 donor states at two destinations on a subset of 864 conditions from 108 base pairs spanning all 18 families. Using the same counterfactual direction for both donor types, mapping and row-order donors produce mean score changes of +0.3692 and +0.3695 at the original mapping span, compared with −0.2520 and −0.2653 at an equal-length output-template span. The destination contrasts are +0.6212 and +0.6348, positive in all 18 families. Moving the recipient's own mapping states into the template also gives −0.3947. The effect of the same states thus depends on where they are written. Appendix D.6.3 gives the score and alarm changes for each destination, including the own-state control that shows the template write has an effect of its own.

AuthorsOpenReview ↗

Response 3 of 3

W2 and Q2. Adaptation on matched responses

The native ranking evaluation uses 12 held-out questions. Its median answer lengths are 107 characters for benign controls and 696 for harmful responses. To address this imbalance, we evaluated 247 length-matched safe/unsafe pairs from PKU-SafeRLHF. Each pair contains two answers to the same request from the same generator, with publisher-provided labels. Alpaca-7B contributes 81 pairs, Alpaca2-7B 94, and Alpaca3-8B 72. Relative differences are at most 15% in answer tokens, 10% in monitor-input content tokens, and 20% in characters, using the longer answer as denominator. A further 238 pairs also match on list versus prose format and the presence of an opening refusal. A stricter length check retains 63 pairs. On this subset, the adapters show no consistent gain over base.

Base and six adapters were evaluated on all 494 responses. Base AUC is 0.8406; adapter AUCs range from 0.8494 to 0.8622. Across three seeds, mean AUC is 0.8577 for paired-wording training (P) and 0.8552 for restored-repeat training (G).

To answer the operating-point question, we used five-fold request-disjoint calibration at 5% and 10% target FPR. Thresholds are fitted exclusively on benign scores from other folds; both arms of each test request are excluded. At the 5% target:

| Configuration | Realised FPR | TPR | ΔTPR vs base (pp) |

| base | 12/247 = 4.86% | 86/247 = 34.82% | — |

| P0 | 13/247 = 5.26% | 94/247 = 38.06% | +3.24 |

| G0 | 17/247 = 6.88% | 83/247 = 33.60% | −1.21 |

| P1 | 13/247 = 5.26% | 95/247 = 38.46% | +3.64 |

| G1 | 11/247 = 4.45% | 89/247 = 36.03% | +1.21 |

| P2 | 13/247 = 5.26% | 87/247 = 35.22% | +0.40 |

| G2 | 12/247 = 4.86% | 63/247 = 25.51% | −9.31 |

At the 10% target, base detects 130/247 unsafe responses (52.63%) at an FPR of 21/247 (8.50%). The adapters detect 113–145/247 at realized FPRs of 8.50%–10.53%. In Appendix E.4 we now give the results after recalibrating on the 238 pairs matched for format as well as length. At the 5% target, base detects 82/238 with 12 false alarms. Section 5.3 reports the means across training seeds. Paired-wording training gives the highest mean detection at both FPR targets on both cohorts.

At the native threshold, base detects 40/49 substituted harmful responses (81.63%). Paired-wording and restored-repeat training average 94.56% and 95.24%, respectively. For the 37 responses with both wordings scored, all configurations flag every restored response. On 181 complete benign responses, base produces 22/181 false alarms (12.15%), and G0 produces the most among the six adapters at 38/181 (20.99%). Each adapter flags more benign responses than base. Transferring thresholds from this benign cohort to the external pairs produces realized FPRs of 40.08%–48.99% at the 5% target and 46.15%–59.92% at the 10% target.

W2 and Q3. Independent judgments of released prefixes

We had three additional model judges assess the released text and added their findings to Section 6.2. Across five wording conditions on 100 requests, 307 distinct request–prefix pairs account for 343 nonempty policy outputs. We counted a shared prefix once for judging and used its label for each policy that released it.

Of the 287 policy outputs Llama Guard 3 labeled safe, GLM, Qwen, and DeepSeek respectively judged 32, 25, and 22 to give direct harmful assistance. Including hedged assistance raised the counts to 36, 38, and 25. Pairwise three-class Cohen’s κ across these judges was 0.745–0.815. Two local Qwen judges gave direct-harm counts of 21 and 29; including hedged assistance gave 64 and 46. Their three-class κ was 0.244.

Appendix F.4 now gives the harmful-delivery counts for each judge under both release policies. With warning times and full-answer labels held fixed, Llama Guard gives 53 harmful deliveries under buffering and 95 under immediate release. With direct harmful assistance as the criterion, the three additional judges give 59 versus 79 (GLM), 59 versus 72 (Qwen), and 56 versus 72 (DeepSeek). All three find fewer harmful deliveries under buffering. Assigning every nonempty prefix a safe label, then every such prefix a harmful label, gives the extreme-case ranges of 46–122 under buffering and 46–313 under immediate release.

We will publicly release the code and raw model outputs.

Thank you again for your review and suggestions.

Reviewer_Zz8COpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how word substitution can bypass external safety monitors even when the substituted terms are explicitly defined. Through controlled experiments across several monitors, the authors show that replacing harmful terms substantially reduces detection, and that alarms often depend more on the original request than on the meaning of the generated response. Further analysis reveals that monitors can internally distinguish different word mappings without changing their safety decisions. The paper also explores few-shot prompting, fine-tuning, and output buffering as possible defenses. Overall, the work provides a systematic analysis of word-substitution failures and highlights the gap between recognizing harmful content and effectively preventing its delivery.

Critical strengths
  • The paper provides convincing evidence that word substitution can bypass safety monitors even when explicit mappings are provided.

  • The controlled experiments help identify why monitors fail, showing that detection often relies on question wording rather than understanding the substituted content.

Critical weaknesses
  • The main evaluation set is small and heavily dominated by DeepSeek (26 out of 30 harmful responses) (Appendix C.1). It is unclear whether the findings generalize to other models.

  • The detection improvements are also evaluated on a small validation set, with substantial length differences between harmful and benign responses. This raises concerns about whether the improvements partly rely on superficial cues rather than recognizing harmful content. (Appendix E.1)

  • The internal analysis does not establish why the monitors fail. The probe may rely on fixed mapping positions rather than semantic understanding, while the activation experiments do not identify a mapping-specific causal effect.

Questions for authors
  • Does the probe still perform well when the mapping order is randomized? The target word always appears in the first row, so the probe may rely on position cues rather than understanding the mapping.

  • Do the detection improvements remain when harmful and benign responses are matched for length and style, and methods are compared at the same false-positive rate?

6 comments
AuthorsOpenReview ↗

Response 1 of 4

Thank you for taking the time to review our paper. We appreciate your close reading of the generator breakdown. We have revised certain sections of the paper as well as the appendix based on the reviews.

W1. Detection across generators

DeepSeek supplied 26 of the 30 responses in the four-judgment, full-response cohort. We tested the restoration effect in a generator-balanced sample. All seven monitors flagged more responses after restoration when we weighted the five generators equally. Appendix C.2 reports this comparison. We selected three requests each for DeepSeek, Grok, Qwen, GLM, and GPT Sol using source metadata. Together they contributed 22 responses across 15 generator–request pairs and 13 distinct requests. We averaged the paired change in alarms within each request, then across requests and generators. The increase ranged from 20.00 to 67.33 percentage points across monitors. Llama Guard 3 and all three Qwen3Guard sizes showed the increase for each of the five generators individually.

Section 3.1 and Table 1 report the full paired evaluation. It covers 132 responses from 59 requests and 78 generator–request pairs. DeepSeek contributed 65 responses, Grok 45, Qwen 13, GLM five, GPT Sol three, and GPT Terra one. Two judges independently identified direct harmful assistance in the exact restored prefix used for each comparison. We then scored the substituted and restored wordings of that same response.

| Monitor | Substitution alarms /132 | Restoration alarms /132 | Difference (pp) |

| Llama Guard 3 | 75 | 116 | +31.06 |

| Qwen3Guard-0.6B | 56 | 122 | +50.00 |

| Qwen3Guard-4B | 60 | 124 | +48.48 |

| Qwen3Guard-8B | 76 | 123 | +35.61 |

| ShieldGemma-2B | 33 | 67 | +25.76 |

| ReasoningShield-1B | 77 | 88 | +8.33 |

| ReasoningShield-3B | 86 | 99 | +9.85 |

Among the 124 responses accompanied by an explicit mapping, restoration increased alarms by 8.06 to 50.81 percentage points across the seven monitors.

We screened 4,985 published HarmBench and JailbreakBench responses for overlap and retained 2,225 across 119 request clusters. Both judges labeled 500 of the retained responses directly harmful. These 500 span 113 request clusters and eight generator versions. ShieldGemma flagged 285/500 (57.0%), Qwen3Guard-4B flagged 446/500 (89.2%), and ReasoningShield-1B flagged 415/499 valid outputs (one parsing failure). Each monitor received the released attack prompt and the full published answer (Table 7).

Table 1 also reports 819 control pairs across 813 distinct requests, covering 240 benign examples, 156 refusals, 195 metaphors, and 228 definitions or fictional contexts. For example, Qwen3Guard-4B flagged 75 of the 195 metaphors under substitution and 13 after restoration.

AuthorsOpenReview ↗

Response 2 of 4

W2 and Q2. Length and format matching at common false-positive rates

We compared paired-wording training, restored-repeat training, and the base monitor on 247 length-matched safe/unsafe pairs and on the 238 that also matched in answer format. Paired-wording training had the highest three-seed mean detection rate on both cohorts at the 5% and 10% target FPRs.

We constructed the 247 pairs from PKU-SafeRLHF. Alpaca-7B contributed 81, Alpaca2-7B 94, and Alpaca3-8B 72. Matching uses at most 15% difference in answer tokens, 10% in monitor-content tokens, and 20% in characters, each normalized by the longer arm. A stricter 63-pair subset is reported separately.

The table reports AUC across thresholds and alarm counts at the native threshold of zero for the 247 pairs. P denotes paired-wording training, G denotes restored-repeat training, and the digit identifies the training seed.

| Configuration | AUC | Unsafe alarms /247 | Safe false alarms /247 | FPR at threshold 0 |

| base | 0.8406 | 240 | 163 | 66.0% |

| P0 | 0.8494 | 238 | 143 | 57.9% |

| G0 | 0.8535 | 240 | 145 | 58.7% |

| P1 | 0.8613 | 243 | 151 | 61.1% |

| G1 | 0.8597 | 243 | 142 | 57.5% |

| P2 | 0.8622 | 236 | 128 | 51.8% |

| G2 | 0.8524 | 237 | 126 | 51.0% |

Both answers in each pair come from the same generator and answer the same request. We used the publisher’s safe/unsafe labels. On the 247 pairs, the three-seed mean AUC in Section 5.2 is 0.8577 after paired-wording training and 0.8552 after restored-repeat training, compared with 0.8406 for base.

We used five-fold request-disjoint calibration at 5% and 10% target FPR, with 5% as the primary comparison. Each method's threshold was fitted only to benign scores from the other four folds, excluding both arms of every test request.

At the 5% target, the seven configurations gave the following false-positive and true-positive rates.

| Configuration | Realised FPR | TPR | ΔTPR vs base (pp) |

| base | 12/247 = 4.86% | 86/247 = 34.82% | — |

| P0 | 13/247 = 5.26% | 94/247 = 38.06% | +3.24 |

| G0 | 17/247 = 6.88% | 83/247 = 33.60% | −1.21 |

| P1 | 13/247 = 5.26% | 95/247 = 38.46% | +3.64 |

| G1 | 11/247 = 4.45% | 89/247 = 36.03% | +1.21 |

| P2 | 13/247 = 5.26% | 87/247 = 35.22% | +0.40 |

| G2 | 12/247 = 4.86% | 63/247 = 25.51% | −9.31 |

At the 5% target on 247 requests, averaging P0, P1, and P2 gave 37.25% TPR and 5.26% realized FPR. Base gave 34.82% and 4.86%, and restored-repeat training averaged 31.71% and 5.40% (Table 3(b)). At the 10% target, base realizes an FPR of 21/247 = 8.50% and a TPR of 130/247 = 52.63%. The adapters realize FPRs of 8.50%–10.53% and detect 113–145 of 247 unsafe responses. The stricter 63-pair and per-generator analyses reuse the 247-pair calibration thresholds.

The paired-wording mean minus base was +2.43 percentage points in TPR (95% interval −7.15 to +12.82) and +0.40 points in realized FPR (−0.81 to +2.70). Across 5,000 bootstrap draws we resampled requests within generator and fold, kept both answers and all configurations together, refitted the thresholds, and averaged the three trained adapters within each draw (Appendix E.4).

Of the 247 pairs, 238 also matched on list versus prose format and on whether the answer opens with a refusal (Appendix E.4). Selection used only the answer text. Of the 63 pairs passing tighter length tolerances, 61 also match in format. We recalibrated within the 238 pairs using the same five folds. At the 5% target, base detects 82/238 unsafe responses with 12/238 false alarms. Each of the three paired-wording adapters detects 91–93 with 11–12 false alarms. Across all six adapters, TPR differences from base range from −8.82 to +4.62 percentage points at the 5% target and from −4.20 to +7.14 at the 10% target.

At the original score threshold of zero, base flags 22/181 complete benign responses (12.15%). All six adapters have higher FPRs on these responses, with G0 highest at 38/181 (20.99%). When thresholds fitted on these 181 responses are transferred to the external 247-pair cohort, realized FPRs are 40.08%–48.99% at the 5% target and 46.15%–59.92% at the 10% target, with corresponding TPR ranges of 91.50%–93.12% and 92.71%–95.95%.

AuthorsOpenReview ↗

Response 3 of 4

W3 and Q1. Mapping position and probe transfer

We evaluated each assignment in both row orders. We also tested the monitor's own scores directly by changing the assignment while keeping the concept words in their original positions.

In the counterbalanced probe test (Section 4.1), we used 216 base pairs from 18 behavior families × 12 request templates. Varying the mapping assignment, queried key, and row produces 1,728 factor conditions, with 432 explicit-concept controls evaluated separately. The readout uses layer 22 of Qwen3-14B, a 20-dimensional PCA fitted within each training fold, and a linear classifier.

Training with the target in row 1 and testing on row 2 gives AUC 0.5949. Training on row 2 and testing on row 1 gives 0.5811. The same probe gives 0.9109 on held-out base pairs, 0.8982 on held-out behavior families, and 0.8988 on held-out request templates. Training on queries for codeword A and testing on B gives 0.9150, and the reverse gives 0.9017. Row position is the largest transfer sensitivity. The same pattern appears with the full hidden state as input to the probe (cross-row AUCs 0.4592 and 0.6320).

In the input-intervention evaluation (Section 4.2), we held the query, answer, all non-table text, and the right-column concept order fixed. We exchanged only the two codewords in the left column. The harmful assignment scored higher in 864/864 pairs, by 10.6603 on average. Reading the concept at a fixed position would predict no change under this exchange.

Reversing the rows preserved the mapping, query, answer, token counts, sequence length, and decision position. In the score-distribution pass, 102/432 harmful-target pairs and 0/432 benign-target pairs changed decisions. In the input-intervention baseline, the corresponding counts were 102/432 and 1/432, or 103/864 pairs in total, spanning all 18 behavior families (Appendix D.3).

With the same monitor instruction and native alarm rule (d > 0), explicit harmful controls triggered 7/24 alarms in the single-template lexicon set and 210/216 in the 12-template scenario set. We selected both sets independently of monitor scores (Appendices D.1–D.3).

The score-distribution pass on the 216 base pairs gave the following native-threshold alarm counts (Table 13).

| Condition | Harmful-concept alarms | Benign-concept alarms |

| Coded targets | 290 / 864 | 0 / 864 |

| Explicit-concept controls | 210 / 216 | 2 / 216 |

For the native scores in the score-distribution pass, harmful versus benign AUC is 0.9618 with codewords and 0.9971 with explicit concepts. The median score for harmful targets was +14.25 when named explicitly and −2.00 when coded. We tested the role of the score threshold by calibrating on 17 behavior families and evaluating the held-out family, rotating through all 18. At a 5% benign-alarm calibration target, the same scores yield 683/864 harmful-target alarms and 43/864 benign-target alarms.

In Section 7 we separated the score shift shared by both target concepts from the change in their separation. Moving the target to row 2 raises the harmful score by 2.87 units on average and the benign score by 1.84. Their midpoint rises by 2.36 units, and half the change in their separation is 0.52. Most of the mean harmful-score shift is shared with the benign target. In the score-distribution pass, the native threshold gave 103/432 harmful-target alarms in row 1 and 187/432 in row 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.