acceptodds
Under review as a conference paper at ICLR 2027

MISSING MASS, MISPLACED CONFIDENCE: CALIBRATING GENERATIVE SAFETY GUARDS

Abstract

Generative safety guards produce scores that downstream systems threshold, rank, and audit as probabilities. A commonly used score renormalizes the guard's next-token distribution over the safe and unsafe label tokens, discarding their total probability mass, which we call verdict mass, . For a fixed input prompt, the resulting conditional distribution differs from the full output distribution by exactly in total variation, and the guard's first-token log loss separates into mass and verdict-choice terms. Across three Llama Guard checkpoints, the released prompt templates concentrate mass near one, while removing their safety instruction exposes substantial variation across inputs. For the same input texts, safety-assessment instructions yield higher verdict mass than length-matched general-task instructions, and attack rewrites that the guard misses flip its verdict while largely preserving this mass. We combine the instruction-free mass with the native logit margin in verdict-mass calibration (VMC), a closed-form correction requiring no guard retraining, in a parameter-free form and an extension with one fitted temperature. On a held-out mixture of benign, harmful, and attack inputs, the extension reduces negative log-likelihood by approximately 9–38% relative to temperature scaling and improves Brier score and expected calibration error across all three checkpoints; it also improves all three metrics on ShieldGemma 9B.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
calibration, verdic mass, guard model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8WyeOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper claims the usual way of turning a generative guard’s output into a probability by renormalising over the safe and unsafe tokens discards how much total probability the guard puts on those two tokens. Specifically, if we rerun the guard with the safety instruction removed, the total probability varies, and the authors observe it being higher on harmful prompts and attacks, and it mostly holds up on attacks the guard misses. So it boils down to this: when mass is high, the score gets nudged toward unsafe, and when mass is low, it mostly fall back on the temperature-scaled native score. With one fitted temperature, it beats temperature scaling on NLL, Brier and ECE for Llama Guard 2, 3 and 4, and on ShieldGemma.

Critical strengths

The observation that a second, unprompted look at the guard still picks up jailbreaks is new to me, plus the method needs no retraining and very few labels.

Critical weaknesses

The mass that renormalization actually throws away is the mass under the released template, and since that’s close to one for every input, the native score loses almost nothing. The paper instead measures mass under an instruction-free prompt the guard was presumably never trained on. That quantity isn’t part of any score anyone computes. So the method is really asking the guard a second question and using the answer as a feature, and the identities in section 2 don’t explain why that answer should track harmfulness, which the authors concede in section 5. There’s no argument for why this particular question is the right one beyond finding that it correlates with harmful inputs.

The main comparison is with temperature scaling, which can’t fix the one-sided bias visible in Figure 2. The paper’s own Appendix F shows a spline on the native margin alone outperforms VMC_T on all three guards (and with no second forward pass), and adding mass to it gains very little. In table 10, Shuffling mass across inputs within margin and length bins still keeps most of the improvement on Guard 3 and almost all of it on Guard 2. So it looks like much of the gain comes from a general upward shift rather than information specific to each input. Meanwhile the method needs a second full forward pass, its gains come almost entirely from AdvBench-derived attacks in a test mix that’s 39% unsafe, and by the authors' analysis it does worse than temperature scaling once unsafe traffic drops below ~20%.

Questions for authors

How does VMC_T compare with two-parameter Platt scaling and with the contextual and batch calibration baselines from Liu et al. (2025a)? Given that a native-only spline already beats VMC_T, what’s the case for VMC beyond the regime with very few labels?

Why measure mass under an instruction-free prompt rather than another second query, such as a reworded instruction or an ensemble of prompts? Is there a reason to expect the guard’s behaviour on a prompt it wasn’t trained on to stay stable across models?

Can you report AUROC, the TP rate at a low FP rate (like 5%), and false-positive rates on XSTest-safe and OR-Bench for LlamaGuards? I want to know whether mass improves detection or just reshapes the probabilities, and how much FP it causes.

How does VMC do at realistic, low unsafe prevalence, on harmful data not derived from AdvBench, and on response moderation?

7 comments

Why mass tracks harmfulness and why this prompt (W1, Q2)

The answer to the “second question” is expected to track harmfulness because of how guards are trained, and the fit confirms it. Guards are fine-tuned on red-teaming requests of both labels [1], and fine-tuned models produce their trained format on inputs resembling their training data even without an instruction [2], so harmful requests keep a high verdict mass when the instruction is removed.

To check the direction, we fit on labels the constants that VMC_0 (Eq. 7) fixes, which gives general VMC:

p=σ(βmN+(1−β)logit⁡(α+(1−α)ME)T+c)(A)p=\sigma\Big(\frac{\beta m_N+(1-\beta) \operatorname{logit}(\alpha+(1-\alpha)M_E)}{T}+c\Big)\quad\text{(A)}

Here mNm_N is the native margin and MEM_E the verdict mass computed from the user message without the safety instruction (m and M in the paper), β weights the native margin against the mass term, T is a temperature, c is an intercept, and α + (1 − α)MEM_E is the paper’s a(M) (Eq. 6) with its constant ½ written as α; VMC_0 sets α = β = ½, T = 1, and c = 0. The fitted weight on the mass term is positive on every checkpoint (table below; added as Table 9).

Fitted Eq. (A)Guard 3 / 4 / 2
Weight on mass term, 1 − β0.32 / 0.31 / 0.32

“Why this particular question.” It is not a particular question we chose: it is the user message alone in a plain Llama chat turn, with the whole guard template removed (Appendix B), so it requires no prompt design. The removal is needed because with the template, as the reviewer notes, the native mass MN≈1M_N \approx 1 for every input (Figure 1b), which is what the training objective in Eq. (3) rewards. The signal also does not depend on this choice: with any of five other second prompts, including two rewordings of the safety instruction, VMC_T still has lower NLL than temperature scaling on every checkpoint (table below; added as Table 5). The direction is also stable across models: mass is higher on unsafe inputs on Llama Guard 2, 3, and 4 (Table 2), and VMC transfers to ShieldGemma (Appendix G).

MethodNLL, Guard 3 / 4 / 2
Temperature scaling0.270 / 0.286 / 0.353
VMC_T, worst of five other prompts0.178 / 0.262 / 0.321
VMC_T, ours0.168 / 0.240 / 0.321

Platt, contextual, batch, and spline baselines (W2, Q1)

There is a “case for VMC beyond the regime with very few labels”: the verdict mass M matches or improves every calibrator the reviewer names on every checkpoint (table below; added as Table 20), namely two-parameter Platt scaling, contextual and batch calibration [3], and the native-only spline. The right comparison is each calibrator with and without M, because our claim is that M improves any calibrator of the native score, not that one form of VMC is best; VMC is a plug-in term, orthogonal to the choice of calibrator. For Platt scaling, adding M gives general VMC, Eq. (A) above.

MethodNLL, Guard 3 / 4 / 2
Platt0.167 / 0.197 / 0.211
Platt + M0.155 / 0.180 / 0.209
Contextual0.178 / 0.257 / 0.214
Contextual + M0.156 / 0.257 / 0.211
Batch0.168 / 0.196 / 0.211
Batch + M0.168 / 0.186 / 0.210
Spline0.161 / 0.187 / 0.189
Spline + M0.151 / 0.173 / 0.185

“Adding mass to it gains very little.” The gain from M concentrates on the inputs that matter: on attacks the native guard misses, adding M to Platt scaling lowers NLL four to six times more than on average on Guard 3 and Guard 4 (table below).

InputsNLL change from adding M to Platt, Guard 3 / 4 / 2
All evaluation inputs−0.012 / −0.017 / −0.002
Attacks the native guard misses−0.048 / −0.095 / −0.002

Shuffled mass (W2)

The gain from M is “information specific to each input,” not “a general upward shift”: on top of Platt scaling, which already captures any shift, M shuffled across inputs gives almost no gain while the real M does (table below; added to Table 12, Table 10 in the submission).

MethodNLL, Guard 3 / 4 / 2
Platt0.167 / 0.197 / 0.211
Platt + shuffled M0.165 / 0.196 / 0.211
Platt + M0.155 / 0.180 / 0.209

References

[1] Inan et al. Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations. arXiv:2312.06674, 2023.

[2] Wang et al. Two-stage LLM Fine-tuning with Less Specialization and More Generalization. ICLR, 2024.

[3] Liu et al. On Calibration of LLM-based Guard Models for Reliable Content Moderation. ICLR, 2025.

Non-AdvBench data, low prevalence, and cost (W2, Q4)

The gains are not confined to “AdvBench-derived attacks in a test mix that’s 39% unsafe”: they hold on harmful data unrelated to AdvBench and at realistic low prevalence.

Harmful data unrelated to AdvBench. VMC_T also improves on OR-Bench-toxic, which is not derived from AdvBench, on every checkpoint (Figure 5).

MethodNLL on OR-Bench-toxic, Guard 3 / 4 / 2
Temperature scaling0.371 / 0.612 / 0.388
VMC_T (ours)0.204 / 0.497 / 0.361

Low prevalence. VMC does not do “worse than temperature scaling” at low prevalence once the calibrator is adjusted to the deployment prevalence: with the standard prior shift, general VMC matches or beats both baselines at 1%, 2%, and 5% unsafe (table below; added as Table 19). The loss in Appendix E.1 came from evaluating at lower prevalence a calibrator fitted on data with 40% unsafe inputs, without this adjustment. General VMC is Eq. (A) above.

NLL, Guard 3 / 4 / 21% unsafe2% unsafe5% unsafe
Temperature scaling0.065 / 0.073 / 0.0920.071 / 0.079 / 0.1000.090 / 0.097 / 0.123
Platt0.028 / 0.029 / 0.0480.042 / 0.045 / 0.0670.071 / 0.077 / 0.105
General VMC (ours)0.025 / 0.029 / 0.0460.038 / 0.044 / 0.0660.065 / 0.073 / 0.103

Cost. The second pass is not “full”: it drops the guard template, including its policy text, and reads only first-token logits, costing about a third of the native pass.

Per inputNative guardAdded by VMC
Mean input tokens256–271+76
Overhead, relative to native—≈0.3×

Response moderation. Response moderation is outside our focus: the jailbreak attacks we study act on prompts.

Detection metrics (Q3)

Mass “improves detection” rather than “just reshapes the probabilities”: adding M raises the TPR at 1% and 5% FPR on every checkpoint (table below; added as Table 15), which is what matters most for a guard, since a missed attack does more harm than an over-refused benign request. Only M can do this, since any monotone recalibration of the native margin, whatever its fitting budget, keeps the native ranking.

Method, Guard 3 / 4 / 2AUROCTPR @ 1% FPRTPR @ 5% FPR
Native, temperature, or Platt scaling0.984 / 0.977 / 0.9770.755 / 0.703 / 0.5610.903 / 0.897 / 0.890
General VMC (ours)0.985 / 0.981 / 0.9770.776 / 0.720 / 0.5750.922 / 0.909 / 0.891

“How much FP it causes.” Over all benign inputs, adding M lowers the false-positive rate at 90% recall on Guard 3 and Guard 4; the rates on XSTest-safe and OR-Bench that the reviewer requested are in the same table (table below; added as Table 16).

FPR @ 90% recall, Guard 3 / 4 / 2All benignXSTest-safeOR-Bench
Native, temperature, or Platt scaling4.7% / 5.3% / 5.3%10.4% / 14.6% / 19.8%17.6% / 15.1% / 18.3%
General VMC (ours)3.9% / 4.4% / 5.5%7.3% / 14.6% / 19.8%14.6% / 18.8% / 18.6%

Dear Reviewer 8Wye, As the discussion period is nearing its end, we would like to kindly follow up on our responses. We hope our clarifications have adequately addressed your concerns, and we would greatly appreciate it if they could be reflected in your final assessment, if you find them satisfactory. Thank you again for your time and consideration.

Reviewer_oHsKOpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies probability calibration in safety guard models and proposes Verdict-Mass Calibration (VMC), which combines the original guard score with verdict-token probability mass measured in a second forward pass without the safety instruction. Experiments on Llama Guard 2, 3, and 4 and ShieldGemma show improvements over temperature scaling in NLL, Brier score, and ECE. However, the evidence for these gains is design dependent and inconsistent across settings.

Critical strengths

The paper gives a reasonable mathematical analysis of the information lost during binary normalization. Additionally, the behavioral experiments show that verdict mass differs across input types and can remain informative on some attacks missed by the original guard.

The method does not require retraining, and the authors provide insightful ablations across several settings, such as a transfer evaluation on ShieldGemma.

Critical weaknesses

The method requires a second forward pass without the original safety instruction, which increases computational cost. The prompt choice is also unclear. Llama Guard 2 uses a different wrapper, and ShieldGemma required selecting one of 11 prompt (Appendix B, G). This raises concerns about the method's reliability and portability.

Results of the method vary across settings. For example, on Guard 4, sensitive benign inputs have higher mean mass (0.566) than general harmful inputs (0.478) and attacks (0.486), potentially increasing false positives (Appendix C, Table 2). At 5% unsafe prevalence, VMCT{VMC}_T has higher weighted NLL than temperature scaling on all three guards (Appendix E.1), limiting its benefit in real settings where most queries are benign.

VMC{VMC} performs inconsistently, for example, VMC0{VMC}_0 has very high NLL of 0.486 on Guard 4, compared with 0.240 for VMCT{VMC}_T and 0.286 for temperature scaling (Table 3). Additionally, a flexible calibrator using only the native margin achieves lower NLL than VMCT{VMC}_T on all three models with the full calibration set (Appendix F, Table 15). The paper does not clarify which method should be preferred in practice.

The attack evaluation has limited coverage of adaptive attacks. Prompts were generated against Llama-3.1-8B-Instruct rather than the guards, and the robustness test selects the worst suffix from only three fixed candidates (Appendix D.4). On Guard 3, VMCT{VMC}_T's attack detection drops by 44.5 percentage points, versus 32.6 points for the native guard. This raises questions about the robustness of the method.

The paper discusses recent methods, such as, contextual and batch calibration [1] in Related Works section but does not empirically compare against these methods.

Reference

[1] Liu et al., On Calibration of LLM-based Guard Models for Reliable Content Moderation., ICLR 2025.

Questions for authors
  • Is there a clear explanation for why instruction-free verdict mass should be higher for unsafe inputs?
  • How does VMC compare with contextual and batch calibration, and how robust is it to attacks optimized against both passes?
6 comments

Which VMC to use (W3)

VMC does not perform “inconsistently”: its variants are one equation, Eq. (A) below, that differs only in which constants are fitted, so the one that “should be preferred in practice” is set by the labels available: general VMC, which fits all of them, when labels exist, and VMC_0, which fits none, when they do not. The paper’s contribution is the verdict mass M that every variant uses, not any one variant. With the native margin mNm_N and the verdict mass MEM_E computed from the user message without the safety instruction (m and M in the paper), general VMC is

p=σ(βmN+(1−β)logit⁡(α+(1−α)ME)T+c)(A)p=\sigma\Big(\frac{\beta m_N+(1-\beta) \operatorname{logit}(\alpha+(1-\alpha)M_E)}{T}+c\Big)\quad\text{(A)}

where β weights the native margin against the mass term, T is a temperature, c is an intercept, and α + (1 − α)MEM_E is the paper’s a(M) (Eq. 6) with its constant ½ written as α. VMC_0 (Eq. 7) fixes α = β = ½, T = 1, and c = 0 and needs no labels, VMC_T fits one constant, and general VMC fits all four; without the mass term, Eq. (A) is affine calibration.

VMC_0 on Guard 4. VMC_0 fits nothing, so its reference is the native score, and it improves on the native score on every checkpoint, including Guard 4 (Table 3). Temperature scaling fits one constant, so its reference is VMC_T, which beats it on every checkpoint (Table 3).

“A flexible calibrator using only the native margin.” Adding M improves it as well, on all three checkpoints, making the spline with M the best calibrator we tested (table below; added to Table 3); a flexible calibrator outperforming VMC_T compares two fitting budgets, not a calibrator with and without M. The gain from M is not a budget effect either way: with M shuffled across inputs, the same form gives almost no gain over affine calibration (table below; added to Table 12).

MethodNLL, Guard 3 / 4 / 2
Affine0.167 / 0.197 / 0.211
Affine + shuffled M0.165 / 0.196 / 0.211
General VMC (affine + M)0.155 / 0.180 / 0.209
Spline0.161 / 0.187 / 0.189
Spline + M0.151 / 0.173 / 0.185

Cost and prompt choice (W1)

Computational cost. The second pass adds about a third of the native cost (table below): the instruction-free prompt has no policy text, only first-token logits are needed, and no additional model or retraining is required.

Per inputNative guardAdded by VMC
Mean input tokens256–271+76
Overhead, relative to native—≈0.3×

Prompt choice. The instruction-free prompt is not tuned: for every Llama Guard checkpoint it is the user message alone in a plain Llama chat turn, with the whole guard template removed (Appendix B). For ShieldGemma, whose template differs, the prompt was likewise fixed by label-free measurement gates rather than by calibration results (Appendix G). The “reliability and portability” of the gain do not depend on this prompt: with any of five other prompts in its place, VMC_T still has lower NLL than temperature scaling on every checkpoint (table below; added as Table 5).

MethodNLL, Guard 3 / 4 / 2
Temperature scaling0.270 / 0.286 / 0.353
VMC_T, worst of five other prompts0.178 / 0.262 / 0.321
VMC_T, ours0.168 / 0.240 / 0.321

Sensitive benign inputs and low prevalence (W2)

Sensitive benign inputs on Guard 4. Adding M lowers the false-positive rate over all benign inputs even on Guard 4, where sensitive benign inputs receive more mass than harmful ones (0.566 vs. 0.478; Appendix C): from 5.3% to 4.4% (table below; added as Table 16). This is because M recognizes ordinary benign inputs better: their false-positive rate falls from 4.0% to 2.7%.

Guard 4, FPR @ 90% recallAll benignOrdinary benignSensitive benign
Native5.3%4.0%15.0%
+ M4.4%2.7%18.0%

Ordinary benign: Dolly and Alpaca; sensitive benign: OR-Bench and XSTest-safe.

5% prevalence. The benefit holds “in real settings where most queries are benign” once the calibrator is adjusted to the deployment prevalence: with the standard prior shift, general VMC matches or beats both baselines at 1%, 2%, and 5% unsafe (table below; added as Table 19). The loss in Appendix E.1 came from evaluating at lower prevalence a calibrator fitted on data with 40% unsafe inputs, without this adjustment.

NLL, Guard 3 / 4 / 21% unsafe2% unsafe5% unsafe
Temperature scaling0.065 / 0.073 / 0.0920.071 / 0.079 / 0.1000.090 / 0.097 / 0.123
Affine0.028 / 0.029 / 0.0480.042 / 0.045 / 0.0670.071 / 0.077 / 0.105
General VMC (ours)0.025 / 0.029 / 0.0460.038 / 0.044 / 0.0660.065 / 0.073 / 0.103

Adaptive attacks (W4, Q2)

Even under “attacks optimized against both passes,” VMC_T flags at least as many attacks as the native guard at every step (table below). The attack optimizes a GCG suffix for each prompt directly against the VMC_T score on the guard itself, with both prompts and the calibrator known.

Attacks flagged, Guard 3 / 4 / 2Step 0Step 100Step 400
Native guard0.965 / 0.915 / 0.9500.501 / 0.108 / 0.0090.090 / 0.011 / 0.001
VMC_T (ours)1.000 / 0.965 / 0.9500.689 / 0.129 / 0.0150.132 / 0.014 / 0.001

Detection drop on Guard 3. VMC_T flags more attacks than the native guard both before and after the suffix (Table 4); its drop is larger only because it starts higher, and robustness is measured by detection after the attack, not by the size of the drop.

Contextual and batch calibration (W5, Q2)

We now “empirically compare against these methods”: adding the verdict mass M to contextual and to batch calibration, implemented following [1], matches or improves each on every checkpoint (table below; added as Table 20). The comparison is with and without M because our claim is that M improves any calibrator of the native score, not that one form of VMC is best; VMC is a plug-in term, orthogonal to the choice of calibrator.

MethodNLL, Guard 3 / 4 / 2
Contextual calibration0.178 / 0.257 / 0.214
Contextual calibration + M0.156 / 0.257 / 0.211
Batch calibration0.168 / 0.196 / 0.211
Batch calibration + M0.168 / 0.186 / 0.210

Why mass is higher on unsafe inputs (Q1)

Prior work explains it: guards are fine-tuned on red-teaming requests of both labels [2], and fine-tuned models produce their trained output format on inputs resembling their training data even without an instruction [3], so harmful requests and attacks, unlike ordinary benign requests, keep high verdict mass. We have added this explanation to Section 8.

References

[1] Liu et al. On Calibration of LLM-based Guard Models for Reliable Content Moderation. ICLR, 2025.

[2] Inan et al. Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations. arXiv:2312.06674, 2023.

[3] Wang et al. Two-stage LLM Fine-tuning with Less Specialization and More Generalization. ICLR, 2024.

Thanks for the detailed responses and additional experiments.

Most of my concerns are addressed. However, I am not fully convinced about the false-positive rate. On sensitive benign queries, adding M increases false positives. Since verdict mass is higher on these queries and the mass term can only raise the unsafe probability, how would you calibrate the system without compromising sensitive cases?

Reviewer_yn2SOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces verdict mass, the total probability assigned to safe and unsafe label tokens, as an auxiliary signal for calibrating generative safety guards. The proposed VMC combines mass measured under a modified prompt with the native logit margin, improving calibration over temperature scaling across three Llama Guard checkpoints and ShieldGemma.

Critical strengths
  1. The distinction between verdict mass and verdict choice is interpretable, and the observation that missed attacks retain substantial mass is interesting.
  2. The evaluation includes connected-group splits, instruction and length controls, permutation tests, and limited-label experiments.
  3. Improvements over temperature scaling are consistent across several calibration metrics and guard checkpoints.
Critical weaknesses
  1. The theory does not directly justify the method. The distributional identities concern normalization under a fixed prompt, whereas VMC combines quantities from different prompt conditions. They also do not establish the proposed monotonic mapping from mass to unsafe evidence.
  2. The baseline comparison needs strengthening. Conditional permutation preserves much of the gain on Guard 3 and Guard 2, leaving the contribution of systematic bias correction unclear. Separately reported affine calibration and constant-offset baselines would help. Flexible native-margin calibrators already outperform VMC with sufficient labels.
  3. Practical benefits remain distribution-dependent. Gains diminish at low unsafe prevalence, while sensitive benign inputs can receive high mass. Recall at matched false-positive rates and cross-source or cross-attack-family evaluation are needed to establish deployment value.
  4. Robustness evidence is limited. The nonnegative correction structurally preserves native unsafe decisions at threshold 0.5, so increased detection should be assessed alongside false positives. Three fixed task suffixes provide only a limited adaptive stress test.
Questions for authors

None

4 comments

Theory and the method (W1)

The theory in Section 2 directly justifies the two choices that define the method: adding a quantity beyond the native margin, and computing it without the safety instruction. The submitted paper stated the steps from the theory to the method compactly, so the connection may not have been clear; we spell them out below and have added them to Sections 2 and 5, and the method itself is unchanged.

Adding a quantity: Proposition 1. Proposition 1 justifies combining “quantities from different prompt conditions,” because it never requires the two to share a prompt. It shows that a second quantity computed from the same input (user message) can lower the expected loss below every recalibration of the native margin alone, and that quantity may be computed under a different prompt, as VMC’s verdict mass is, with the safety instruction removed.

Computing it without the instruction: the identities. The identities “concern normalization under a fixed prompt,” and show that with the safety instruction, fine-tuning rewards a high mass (Eq. 3), so the native mass MN≈1M_N \approx 1 for every input (Figure 1b) and cannot tell inputs apart. The mass therefore has to be computed without the safety instruction, which gives the verdict mass MEM_E that VMC adds.

Form of the combination. How VMC adds MEM_E to the native margin follows from Bayes’ rule: the paper’s Eq. 7 is the sum that Bayes’ rule gives once each term is modeled as in affine calibration, and Eq. (B) below writes its constants as parameters.

Step 1. With mNm_N the native margin, computed with the safety instruction (m and M in the paper are mNm_N and MEM_E), the log-odds of unsafe splits into the evidence from mNm_N and the evidence that MEM_E adds beyond it:

logit⁡P(Y=1∣mN,ME)=logit⁡P(Y=1∣mN)+log⁡P(ME∣mN,Y=1)P(ME∣mN,Y=0)(A)\operatorname{logit} P(Y{=}1\mid m_N,M_E)=\operatorname{logit} P(Y{=}1\mid m_N)+\log\frac{P(M_E\mid m_N,Y{=}1)}{P(M_E\mid m_N,Y{=}0)}\quad\text{(A)}

Step 2. VMC models each term of Eq. (A) as linear in a log-odds. The first is linear in mNm_N, itself the log-odds of the renormalized verdict, as in affine calibration; the second is linear in the log-odds of a verdict mass, chosen in Step 3.

Step 3. VMC chooses the verdict mass of a single output distribution, because Eq. (2) is stated for one distribution: renormalizing it over the two verdict tokens discards the probability 1 − M outside them. Mixing the output distributions of the two passes with weights α and 1 − α gives the verdict mass below, which depends only on MEM_E because MN≈1M_N \approx 1; the paper’s a(M) (Eq. 6) is the case α = ½:

αMN+(1−α)ME≈α+(1−α)ME\alpha M_N+(1-\alpha)M_E\approx\alpha+(1-\alpha)M_E

Step 4. The two slopes, β/T and (1 − β)/T, and the intercept c are fitted by minimizing log loss:

logit⁡p=βTmN+c⏟≈ first term of (A)+1−βTlogit⁡(α+(1−α)ME)⏟≈ second term of (A)(B)\operatorname{logit} p=\underset{\approx\ \text{first term of (A)}}{\underbrace{\frac{\beta}{T} m_N+c}}+\underset{\approx\ \text{second term of (A)}}{\underbrace{\frac{1-\beta}{T}\operatorname{logit}\big(\alpha+(1-\alpha)M_E\big)}}\quad\text{(B)}

VMC0\mathrm{VMC}_0 (Eq. 7) sets α = β = ½, T = 1, and c = 0, the natural choice without labels: the two prompts and the two terms weigh equally and no prior is assumed; with labels, all four are fitted, which we call general VMC.

“Monotonic mapping from mass to unsafe evidence.” This direction is expected from how guards are trained, and the fit confirms it. Guards are fine-tuned on red-teaming requests of both labels [1], and fine-tuned models produce their trained output format on inputs resembling their training data even without an instruction [2], so harmful requests keep a high mass. Fitting Eq. (B) on labels makes the weight on the mass term positive on all three checkpoints (table below; added as Table 9).

Fitted Eq. (B)Guard 3 / 4 / 2
Weight on mass term, 1 − β0.32 / 0.31 / 0.32

This derivation is now in Section 5, where Eqs. (9) and (11) are Eqs. (A) and (B) above.

References

[1] Inan et al. Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations. arXiv:2312.06674, 2023.

[2] Wang et al. Two-stage LLM Fine-tuning with Less Specialization and More Generalization. ICLR, 2024.

Baselines and permutation (W2)

The comparison now includes the requested “affine calibration and constant-offset baselines”: affine calibration fits a slope and a constant offset, so it contains the constant-offset baseline, and adding the verdict mass M to it lowers NLL on all three checkpoints, as it does for a flexible calibrator (tables below; added to Table 3 and Appendix F). The right comparison is each baseline with and without M, because our claim is that M improves any native-margin calibrator, not that one calibrator form is best. Eq. (A) above is why M can be added to any calibrator: the calibrator estimates the first term from the native margin, and M enters only through the second, so VMC is a plug-in term that is orthogonal to the choice of calibrator.

“Flexible native-margin calibrators already outperform VMC.” They outperform VMCT\mathrm{VMC}_T only “with sufficient labels,” through a larger fitting budget, and M improves them as well: the cross-validated spline with M is the best calibrator we tested (table below). With few labels, VMCT\mathrm{VMC}_T outperforms the flexible calibrator (Table 13; Table 11 in the submission).

MethodNLL, Guard 3 / 4 / 2
Affine0.167 / 0.197 / 0.211
Affine + M (general VMC)0.155 / 0.180 / 0.209
Spline0.161 / 0.187 / 0.189
Spline + M0.151 / 0.173 / 0.185

“Systematic bias correction.” With the bias correction given to an intercept, M shuffled across inputs gives almost no gain over affine calibration, while the real M does (table below; added to Table 12, Table 10 in the submission), so the gain from M is information specific to each input.

MethodNLL, Guard 3 / 4 / 2
Affine0.167 / 0.197 / 0.211
Affine + shuffled M0.165 / 0.196 / 0.211
Affine + M0.155 / 0.180 / 0.209

Distribution dependence (W3)

The practical benefits of adding M are not “distribution-dependent”: they hold at low unsafe prevalence, at matched false-positive rates, and on held-out attack families (tables below).

“Gains diminish at low unsafe prevalence.” The gains remain once the calibrator is adjusted to the deployment prevalence: with the standard prior shift, general VMC matches or beats both baselines at 1%, 2%, and 5% unsafe (table below; added as Table 19). The loss in Appendix E.1 came from evaluating at lower prevalence a calibrator fitted on data with 40% unsafe inputs, without this adjustment.

NLL, Guard 3 / 4 / 21% unsafe2% unsafe5% unsafe
Temperature scaling0.065 / 0.073 / 0.0920.071 / 0.079 / 0.1000.090 / 0.097 / 0.123
Affine0.028 / 0.029 / 0.0480.042 / 0.045 / 0.0670.071 / 0.077 / 0.105
General VMC (ours)0.025 / 0.029 / 0.0460.038 / 0.044 / 0.0660.065 / 0.073 / 0.103

“Recall at matched false-positive rates.” Adding M raises recall at matched false-positive rates on every checkpoint (table below; added as Table 15). Only M can do this: any monotone recalibration of the native margin, whatever its fitting budget, keeps the native ranking.

Method, Guard 3 / 4 / 2TPR @ 1% FPRTPR @ 5% FPR
Temperature scaling or affine (native ranking)0.755 / 0.703 / 0.5610.903 / 0.897 / 0.890
General VMC (ours)0.776 / 0.720 / 0.5750.922 / 0.909 / 0.891

“Sensitive benign inputs can receive high mass.” Adding M still lowers the false-positive rate over all benign inputs, even on Guard 4, where these inputs receive more mass than harmful ones (Appendix C): from 5.3% to 4.4% (table below; added as Table 16). This is because M recognizes ordinary benign inputs better: their false-positive rate falls from 4.0% to 2.7%.

Guard 4, FPR @ 90% recallAll benignOrdinary benignSensitive benign
Native5.3%4.0%15.0%
+ M4.4%2.7%18.0%

Ordinary benign: Dolly and Alpaca; sensitive benign: OR-Bench and XSTest-safe.

“Cross-attack-family evaluation.” On every attack family held out from calibration, VMCT\mathrm{VMC}_T, the variant reported in the paper, beats temperature scaling, which fits the same single parameter (table below).

Held-out NLL, Guard 3 / 4 / 2AutoDANPAIR / TAPRoleplayGCG
Temperature scaling1.09 / 1.00 / 1.220.74 / 0.77 / 0.780.44 / 0.47 / 0.870.24 / 0.21 / 0.72
VMCT\mathrm{VMC}_T (ours)0.60 / 0.69 / 1.050.40 / 0.65 / 0.710.13 / 0.30 / 0.740.09 / 0.13 / 0.60

Adaptive attacks (W4)

Beyond the “three fixed task suffixes,” VMCT\mathrm{VMC}_T flags at least as many attacks as the native guard at every step of a white-box attack optimized against its own score (table below). The attack optimizes a GCG suffix for each prompt directly against the VMCT\mathrm{VMC}_T score, with both prompts and the calibrator known.

Attacks flagged, Guard 3 / 4 / 2Step 0Step 100Step 400
Native guard0.965 / 0.915 / 0.9500.501 / 0.108 / 0.0090.090 / 0.011 / 0.001
VMCT\mathrm{VMC}_T (ours)1.000 / 0.965 / 0.9500.689 / 0.129 / 0.0150.132 / 0.014 / 0.001

“Structurally preserves native unsafe decisions at threshold 0.5.” The gain is not only structural: at matched false-positive rates, where no threshold is fixed, adding M raises recall (table under W3).

Dear Reviewer yn2S, As the discussion period is nearing its end, we would like to kindly follow up on our responses. We hope our clarifications have adequately addressed your concerns, and we would greatly appreciate it if they could be reflected in your final assessment, if you find them satisfactory. Thank you again for your time and consideration.

Comments on the paper

1 comment

Overview

We thank all reviewers for their constructive feedback. They found the distinction between verdict mass and verdict choice interpretable (yn2S), noted that verdict mass still picks up attacks the guard misses (yn2S, 8Wye), found the analysis of binary normalization reasonable (oHsK), and valued that the method requires no retraining (oHsK, 8Wye) and improves on temperature scaling consistently across metrics and checkpoints (yn2S). yn2S and 8Wye asked whether the theory justifies adding the instruction-free verdict mass, and oHsK why it is higher on unsafe inputs. All three asked whether the gain remains over stronger calibrators and at low unsafe prevalence; yn2S and 8Wye asked for recall at fixed false-positive rates, and yn2S and oHsK about adaptive attacks.

Summary of reply

Theory (yn2S, 8Wye). Section 2 justifies the two choices that define the method. Proposition 1 shows why a second quantity is added: computed from the same input, even under a different prompt, it can lower the expected loss below every recalibration of the native margin alone. The identities show why verdict mass has to be computed without the safety instruction. The form of VMC follows from Bayes' rule, which Section 5 now derives.

Why verdict mass is higher on unsafe inputs (all). Guards are fine-tuned to answer with a verdict on red-teaming requests, and a fine-tuned model produces its trained output format on inputs resembling its training data even without an instruction, so harmful requests keep a high verdict mass. Consistent with this, the fitted weight on the mass term is positive on all three checkpoints.

Stronger calibrators (all). Adding verdict mass matches or improves every calibrator of the native score that the reviewers name, on every checkpoint: Platt (affine) scaling, contextual and batch calibration, and a flexible spline. The gain comes from each input's own mass, not from a uniform shift: it almost disappears when mass is shuffled across inputs (yn2S, 8Wye).

Low prevalence and detection. At 1%, 2%, and 5% unsafe prevalence, Platt scaling with verdict mass (general VMC) matches or beats temperature scaling and Platt scaling alone once the calibrator is adjusted to the deployment prevalence (all). Adding verdict mass raises the TPR at 1% and 5% FPR on every checkpoint, which temperature and Platt scaling cannot do, since they keep the native ranking (yn2S, 8Wye).

Generalization and adaptive attacks. VMCT\mathrm{VMC}_T beats temperature scaling on every checkpoint on OR-Bench-toxic, which is not derived from AdvBench (8Wye), and on every attack family held out from calibration (yn2S). Under a white-box GCG attack optimized against its own score, it flags at least as many attacks as the native guard at every step (yn2S, oHsK).

Cost and prompt (oHsK, 8Wye). The second pass adds about a third of the native cost. The instruction-free prompt requires no prompt design: it is the user message alone, with the whole guard template removed. With any of five other prompts in its place, VMCT\mathrm{VMC}_T still has lower NLL than temperature scaling on every checkpoint.

Summary of revision

We have uploaded a revised PDF, with changes marked in red.

Main

  • Section 1. State that the form of VMC follows from Bayes' rule and has a general form (yn2S).
  • Section 2. Define the instruction-free prompt as the user message alone in a plain chat turn, with the whole guard template removed (oHsK, 8Wye). Explain why verdict mass has to be computed without the safety instruction, and that Proposition 1 does not require the two quantities to share a prompt (yn2S, 8Wye).
  • Section 5, Eqs. 9–11. Derive the form of VMC from Bayes' rule and define general VMC, of which VMC0\mathrm{VMC}_0 and VMCT\mathrm{VMC}_T are special cases (all).
  • Section 6, Table 3. Add Platt (affine) scaling and general VMC (all).
  • Section 7. Distinguish recalibrating the guard's score from adding a second measurement of the input (oHsK).
  • Section 8. Explain from guard training why verdict mass is higher on unsafe inputs (all), and state which variant to use with ample and with few labels (oHsK).

Appendix

  • B, Table 5. Add VMCT\mathrm{VMC}_T with five other second prompts (oHsK, 8Wye).
  • D, Table 9. Report the fitted weight on the mass term in general VMC (yn2S, 8Wye).
  • D.1, Table 10. Add paired differences between general VMC and Platt scaling (yn2S, 8Wye).
  • D.2, Table 12. Add the permuted-mass control on top of Platt scaling (yn2S, 8Wye).
  • D.4, Tables 15–16. Add AUROC, TPR at 1% and 5% FPR, and FPR at 90% recall by benign group, including XSTest-safe and OR-Bench (all).
  • E.1, Table 19. Add results at 1%, 2%, and 5% unsafe prevalence (all).
  • F, Table 20. Add verdict mass to Platt scaling, contextual and batch calibration, and a flexible spline (all).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.