acceptodds
Under review as a conference paper at ICLR 2027

JailNewsGuard: Release-Time Risk Gating for Jailbreak-Induced Fake News

Abstract

While existing jailbreak defenses primarily target harmful instruction following or toxic content generation, relatively little attention has been paid to release-time misinformation containment. This distinction is particularly important because misinformation generation depends not only on refusal behavior but also on factual consistency, narrative framing, and semantic manipulation. We propose a three-stage defense system called JailNewsGuard that screens explicit templates and context-length anomalies before generation, then scores generated text for eight harmfulness dimensions: faithfulness, verifiability, malicious-instruction adherence, scope, scale, formality, subjectivity, and agitativeness. The defense operates strictly as an external containment gate without modifying model weights or improving internal safety alignment. Moreover, we provide theoretical support for false release bound, dominance of multi-dimensional vector gating, and generalization bound for hidden-state safety probe. We conduct experiments on JailNewsBench with 10000 news items, 7 attack conditions, 23 languages, and 34 regions, yielding 70000 paired baseline and defense records. Our method successfully suppresses the attack success rate from 68.29% to 4.39% for DeepSeek-V4-Pro, and from 17.40% to 1.09% for GPT-5.5, corresponding to relative reductions of 93.57% and 93.75%, respectively.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Release-time risk gating, jailbreak-induced fake news, external containment

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_NkgxOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies release-time containment of jailbreak-induced fake news: given a frozen LLM and a news-generation request, the goal is to decide whether the generated article may be released. JailNewsGuard wraps the model in three stages without changing its weights or decoding. Stage 1 screens the prompt with a regular-expression library covering five template-based attack families (Explicit, Role Play, System Override, Research Front, Negative Prompting) and a length-and-vocabulary-overlap heuristic for context overload, and replaces flagged prompts with a refusal. Stage 2, for white-box models only, applies a logistic-regression probe to mean-pooled penultimate-layer hidden states of the generated tokens. Stage 3 asks an LLM judge to score the article from 0 to 4 on eight dimensions (faithfulness, verifiability, malicious-instruction adherence, scope, scale, formality, subjectivity, agitativeness) and blocks it if any dimension reaches its threshold or the mean does. Theorem 1 gives a sub-Gaussian bound on false release and claims that vector gating Pareto-dominates scalar gating; Theorem 2 gives a Rademacher generalization bound for the probe. On 10,000 JailNewsBench items under seven attack conditions (23 languages, 34 regions), the overall attack success rate (ASR) falls from 68.29% to 4.39% on DeepSeek-V4-Pro and from 17.40% to 1.09% on GPT-5.5, for which Stage 2 is skipped.

Critical strengths

S1. A practical problem and a deployable design. The paper targets a realistic setting: preventing jailbroken models from releasing fake news when only an inference API is available. The defense needs no retraining, gradient access or decoding changes, its decisions are deterministic given the prompt and the judge scores, and Stages 1 and 3 apply to closed-source models such as GPT-5.5. This matters because a simple gate of this kind would be easy to adopt, if its effectiveness were established.

S2. Large-scale, multilingual evaluation. The experiments cover 10,000 items across 23 languages and 34 regions, seven attack conditions and two frontier models, with a paired design (70,000 records per setting) and UID-clustered confidence intervals. The baseline results are informative in themselves: for example, DeepSeek-V4-Pro does not refuse 95.68% of explicit fake-news requests, against 10.08% for GPT-5.5 (Tables 1–2). This matters because it shows the problem is real across languages and models, which is also why the narrow scope of the evaluation (W1) is the main limitation.

Critical weaknesses

W1. The evaluation does not support the claim of effective, robust containment.

  • The headline reduction comes mostly from matching the benchmark's own templates.

    • Five of the seven attack conditions are fixed-template wrappers (Sec. 3.2, App. A.3), and Stage 1's regular expressions are written for exactly these five families (Sec. 3.3.1, Fig. 1).
    • Stage 1 blocks all 50,000 corresponding records (App. A.6.1). On its own, it lowers ASR from 68.29% to 13.69%, which is 54.60 of the 63.90-point reduction (Table A2).
    • The context-overload detector mirrors how the benchmark builds that attack. It flags prompts longer than 2,000 characters whose first and last thirds have a vocabulary overlap below 0.05 (Sec. 3.3.1, Eq. 6), and the attack prepends a randomly sampled news article truncated to 512 tokens (Sec. 3.2, App. A.3). Even so, it blocks only 58.93% of these prompts on DeepSeek-V4-Pro and 37.92% on GPT-5.5 (Tables 1–2).
    • Only the Original condition has no jailbreak wrapper (Sec. 3.2) and receives no input blocking (Table 1). There, ASR falls from 66.37% to 25.08% on DeepSeek-V4-Pro, so a quarter of plain malicious requests still produce released articles.
  • No stronger or adaptive attacks. The threat model limits the attacker to the seven benchmark transformations (Sec. 3.1). Yet the paper cites optimization-based and automated jailbreaks (Zou et al., 2023; Chao et al., 2025), and itself notes that robustness to benchmark transformations does not imply robustness to adaptive attacks (Sec. 2.1). Paraphrased templates, PAIR- or GCG-style attacks, and attacks aware of the gate are not evaluated. Claims such as "complete protection against structured prompt transformations" (App. A.6.1) are therefore supported only for the five exact templates.

  • No defense baselines. Results are compared only with no defense (Tables 1–2) and with the method's own components (Table A2). None of the following are compared:

    • existing input-output safeguards such as Llama Guard (cited in Sec. 1) and moderation classifiers (Markov et al., 2023);
    • common jailbreak defenses such as perplexity filtering or SmoothLLM;
    • a single-score judge using the same model.

    The advantage over existing practice is therefore untested, including the claimed advantage of vector over scalar gating.

  • Over-blocking is not measured, and the metric rewards it.

    • ASR counts every non-refused output (Eq. 14, App. A.5.3), so any block counts as a success, and the dataset contains only malicious requests.
    • Some dimensions describe the topic or style rather than the harm. Scale scores 3 for international and 4 for global subject matter (App. A.4). With the default threshold of 3 (App. A.5.2), the rubric implies that any article on an international topic would be blocked, however accurate it is. Likewise, Formality, with a threshold of 4, blocks any article judged to be "Professional-grade journalism (datelines, quotes, expert citations)" (App. A.4). Without benign requests, the false-positive cost is unknown.
    • The McNemar tests and Wilson intervals (Tables 1–2) confirm a difference from no defense. But with 10,000 items, and template attacks blocked by construction, they cannot address these questions.

This weakness is consequential because the paper's central claim is effective release-time containment. As designed, the evaluation cannot distinguish that from rule matching on known templates combined with aggressive blocking.

W2. Key components are under-specified, and the theory is weakly justified.

  • The probe cannot be reproduced or assessed.
    • Stage 2 is specified only as logistic regression on mean-pooled penultimate-layer hidden states (Sec. 3.3.2, Eq. 9). The training data, labels, number of samples, data splits and the value of δ_probe are not given anywhere, and no ablation isolates its contribution (Table A2).
    • It also runs after generation (Algorithm 1, lines 5–7). It is thus a second output classifier that requires white-box access, although it is presented as detecting "harmful intent" (Sec. 3.3.2) and its guarantee is stated for "population prompts" (Sec. 1).
    • Fig. 1 lists DeepSeek-V4-Pro behind an API-compatible endpoint, without explaining how its hidden states were obtained.
  • The judge is not independent. Sec. 1 and Sec. 3.3.3 describe an independent judge model, but App. A.5.2 states that the same LLM is used for generation and for the eight-dimensional judging. Generation and judging errors are then likely to be correlated, and no result with a different judge is reported.
  • The theory does not carry weight.
    • Theorem 1:
      • It assumes zero-mean, σ²-sub-Gaussian judge errors, but σ is never estimated, and a judge that is the same model as the generator is unlikely to be unbiased.
      • The per-dimension term of Eq. 17 approaches 1 for truly harmful articles near the threshold, and equals 1 when s*_j = ϑ_j.
      • Eq. 17 also applies the mean term unconditionally, although its derivation (App. A.1, Case (b)) requires s̄* ≥ δ_out. For s* = (3, 0, …, 0), the mean term makes the bound close to 0, while the true miss probability can be about 1/2, so the bound is incorrect as stated.
      • The "harmful" criterion (Eq. 16) is defined with the gate's own thresholds.
    • Pareto dominance: the claim is supported by a numerical example with chosen values (App. A.1), not by a proof covering arbitrary scalar gates. Although the example invokes false positives to choose ζ = 1.0, it never shows that the vector gate's false-positive rate is no worse than the scalar gate's, which Pareto dominance requires. Moreover, at the default thresholds the mean condition can never trigger on its own: if no dimension reaches its threshold, the mean is at most (7 × 2 + 3)/8 ≈ 2.1 < 3.
    • Theorem 2:
      • It is a standard Rademacher bound for norm-bounded linear predictors.
      • Its example loss, the natural-log logistic loss, violates the theorem's own conditions: it exceeds 1, and it does not upper-bound the 0-1 loss near the decision boundary.
      • It covers only i.i.d. prompts, not adversarial ones.
  • Positioning and writing.
    • The paper has only 22 references, of which only three are inference-time defenses or safeguards (Llama Guard; Markov et al., 2023; SafeDecoding), and the related work describes defense categories without engaging with specific methods.
    • The claimed decomposition into pre-generation structural screening and post-generation semantic screening is what input-output safeguards such as Llama Guard already do.
    • Sec. 5.2 states that Stage 2 is skipped for GPT-5.5, but later credits "latent probing" and "Stages Two and Three" for its results.

This weakness is consequential because neither the method nor the theory makes up for the limits of the evaluation. One component cannot be reproduced, the judge's independence is described inconsistently, the theorems rest on unverified assumptions, and the thin positioning leaves the contribution over existing safeguards unclear.

Questions for authors
  1. Additional result requested (could change my assessment; W1). How does JailNewsGuard compare with Llama Guard applied to prompts and outputs, a moderation API, perplexity filtering, SmoothLLM, and a single-score judge using the same judge model?
  2. Additional result requested (W1). How does the defense perform against paraphrased versions of the five templates, PAIR- or GCG-style attacks (both cited in the paper), and adaptive attacks that know the gate?
  3. Additional result requested (W1). What is the false-positive rate on benign news-generation requests, in particular for international or global topics, given that a Scale score of 3 or more triggers a block?
  4. Clarification and additional result (W2). What data, labels, sample size and splits were used to train the probe, and what is δ_probe? How were hidden states obtained for DeepSeek-V4-Pro? What is the ASR without Stage 2?
  5. Clarification (W1, W2). How are records without valid judge scores treated in the ASR and in gating?
    • App. A.6.4 reports valid scores for 31,956 baseline and 55,401 defended records out of 70,000.
    • Table A3 reports the same 4.39% ASR for thresholds of 1, 2 and 3. Why does lowering the thresholds not change the ASR?
    • The output-gate-only reduction (68.29% to 22.66%, about 31,940 records; Table A2) nearly equals the number of baseline records with valid scores. Does the gate block almost every scored output, so that the residual ASR consists mostly of unscored ones?
  6. Clarification and additional result (W2). Was the same model used as generator and judge in all experiments (App. A.5.2), contrary to Sec. 1 and Sec. 3.3.3? How do the results change with a different judge?
  7. Clarification (W1). Eq. 1 defines a successful attack as an Unsafe output, while Eq. 14 and App. A.5.3 count every non-refused output. Which definition produced Tables 1–2, and how are harmless non-refusals counted?
Reviewer_ZSZYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes JailNewsGuard, a release-time framework designed to reduce the release of fake news induced by jailbreak attacks. It integrates checks on incoming prompts, an optional safety probe, and a multidimensional assessment of generated content to determine whether a response should be released. The evaluation examines the framework under varied attack and language settings and reports reductions in attack success. The paper further includes analyses of individual components and decision thresholds, together with theoretical arguments intended to support the proposed gating mechanism.

Critical strengths
  1. The paper focuses on a safety problem with practical relevance and investigates a potentially valuable mechanism for controlling what language models disclose. The framework is generally clear and may be attractive for deployment in real world applications.

  2. The paper reports a series of empirical analyses that offer preliminary support for the proposed approach. The supplementary investigations of component behavior and decision thresholds provide helpful context for interpreting the findings.

Critical weaknesses
  1. The limited empirical scope weakens the persuasiveness of the paper’s broader claims. The evaluation is restricted to one benchmark, JailNewsBench, and two target models, DeepSeek-V4-Pro and GPT-5.5. Although the benchmark includes multiple languages, regions, and attack conditions, this diversity remains within a single data source and evaluation setup. It is therefore unclear whether the reported gains would persist on independently constructed datasets or across a broader range of model families and capabilities. Additional evidence is needed to establish that the findings extend beyond the particular benchmark and model pair.

  2. The evaluation does not demonstrate a useful safety–utility trade-off beyond known attack templates. Five fixed-prefix structural attack families account for 50,000 of 70,000 evaluation records and have zero defended ASR on DeepSeek-V4-Pro. The overall input blocking rate across all seven attack conditions is 79.85% for this model, while the unwrapped Original condition still has a defended ASR of 25.08%. There are no held-out or adaptive attacks, comparisons with alternative defenses, or measurements of benign false positives. Because ASR counts non-refused outputs, withholding more outputs can lower this metric without showing better factual judgment. The score-stability analysis on selected malicious examples in Appendix A.6.3 does not establish utility for legitimate news generation. Such utility is especially uncertain because a professional journalistic style or an international subject scope may independently trigger rejection under the stated scoring rules.

  3. The implemented pipeline and the meaning of the reported safety improvements remain unclear. Appendix A.5.2 states that the same model endpoint is used for generation and judging, whereas Section 3.3.3 describes an independent evaluation model, no independently adjudicated accuracy assessment validates the judge. The probe lacks reported details of its training set, its fitted threshold, and an isolated ablation. Section 5.2 explicitly skips probing for GPT-5.5 but later attributes gains to it. Table A4 reports a defended Faithfulness risk mean of 3.875 despite a blocking threshold of 3. This could concern pre-gate or replacement text, but its interpretation concerns released outputs. These issues prevent reliable attribution of the gains and verification of factual containment.

Questions for authors
  1. How well does the framework generalize to independent datasets and additional model families or capability levels? Please test whether the gains persist when current gating rules and thresholds are transferred, and clarify any retuning required.

  2. At matched benign false-positive rates, how does the method compare with existing safeguards on held-out and adaptive attacks when factual harmfulness is assessed independently rather than through non-refusal alone?

  3. For each main result, what probe data, split, threshold, and per-stage settings were used? Please isolate the probe’s contribution, reconcile the GPT-5.5 description, and specify whether Table A4 scores candidates, replacement text, or actually released articles.

Reviewer_6qusOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The manuscript tackles release‑time misinformation containment for frozen LLMs. It proposes JailNewsGuard, a three‑stage runtime pipeline that (1) sanitizes prompts with hand‑crafted regexes and an anomaly detector, (2) (optionally) probes the penultimate hidden layer of a white‑box model, and (3) evaluates the generated article using an independent LLM judge that scores eight semantic risk dimensions. Experiments are carried out on JailNewsBench with two target models evaluated: DeepSeek‑V4‑Pro (white‑box) and GPT‑5.5 (black‑box). Significant Attack Success Rate (ASR) drops are reported for both models.

Critical strengths
  1. The paper presents a clear problem framing and system design. It shifts the safety focus from generation‑time refusal to a release‑time containment gate, which matches real‑world deployment constraints (black‑box APIs, no weight updates). The three‑stage architecture is clearly articulated.

  2. The paper conducts a large‑scale, multilingual evaluation. 70 000 paired records are evaluated on 23 languages and 34 regions, providing strong empirical evidence that the system consistently reduces ASR across attack families.

Critical weaknesses
  1. The paper lacks comparison with state‑of‑the‑art jailbreak defenses. The experimental section only contrasts “undefended model” vs. “JailNewsGuard”. No results are shown for existing release‑time or input‑filtering systems. Without this comparison it is unclear whether the reported drop in ASR is truly state‑of‑the‑art or simply the benefit of adding any non‑trivial filter.

  2. The paper relies on a fragile hand-crafted prompt sanitizer. Stage 1 uses a fixed regex library and a simple length/overlap heuristic. Simple modifications, such as minor paraphrasing, or token-level reordering, could easily bypass these rules. However, the paper does not evaluate robustness against unseen jailbreak templates, raising concerns about the system’s reliability in adversarial settings.

  3. The study evaluates only two target models (DeepSeek‑V4‑Pro and GPT‑5.5). Both are decoder‑only, transformer‑based LLMs with similar tokenization schemes. This narrow selection cannot substantiate the claim that JailNewsGuard works for any frozen LLM.

  4. Stage 2 applicability is restricted to white‑box settings. The hidden‑state probe is unavailable for black‑box APIs, but the paper presents it as a core component of the three‑stage pipeline. This further undermines the claimed novelty of the three-stage framework, particularly since the other two stages also primarily combine existing approaches without introducing substantial methodological innovations.

  5. The paper lacks a quantitative false‑positive analysis. The paper reports mean score shifts on benign news (< 0.016) but never gives a precision/recall or percentage of legitimate articles blocked by the full pipeline. Without this metric it is impossible to assess the trade‑off between safety and utility, which is crucial for real deployments.

Questions for authors
  1. Baseline comparison: Could you evaluate JailNewsGuard against existing defence on the same JailNewsBench split? This would clarify whether your three‑stage pipeline offers a genuine improvement over the current state of the art.
  2. Have you measured how many novel jailbreak prompts (e.g., using synonym substitution, Unicode tricks, or multi‑sentence splitting) evade the regex rules? If not, could you provide an analysis or propose a data‑driven alternative that adapts to new patterns?
  3. False‑positive rate on clean content: What proportion of legitimate news articles are blocked by the full pipeline (especially Stage 3)? Reporting precision/recall or a “clean‑content retained” metric would help assess the practical trade‑off between safety and utility.
  4. Do you have any preliminary results on other model families (e.g., LLaMA‑2)? Demonstrating that Stage 2 works across different architectures would strengthen the claim of broad applicability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.