- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Conf-Guard, a dynamic defense framework for detecting multi-turn jailbreak attacks in large language model conversations. Unlike conventional guardrails that independently classify each turn, Conf-Guard maintains an evolving safety score over the conversation and uses a running minimum so that previously observed risk cannot be “reset” by later benign-looking turns. The conversation is terminated once this score falls below a calibrated threshold. The key contribution is the use of conformal prediction to calibrate this stopping threshold, providing a formal guarantee that, under an exchangeability assumption between calibration and test conversations, at least a user-specified proportion \gamma of malicious conversations will be stopped before a successful jailbreak occurs, corresponding to an attack success rate of at most 1-\gamma. The framework combines signals from existing jailbreak classifiers with an embedding-based outlier score derived from benign reference conversations, and supports both prompt-response and prompt-only settings. Experiments across GPT-, Gemini-, and human-derived multi-turn datasets show that Conf-Guard can reduce jailbreak success rates while maintaining low false-refusal rates and empirically satisfying its conformal guarantees. Its main limitation is that these guarantees may weaken under distribution shift or adaptive attacks that differ substantially from the calibration data.
Critical strengths
- Conf-Guard can be combined with different existing jailbreak detectors without modifying the underlying LLM, making it easy to integrate into various guardrail systems.
- By using a running minimum of safety scores, Conf-Guard preserves earlier risk signals and prevents later benign turns from masking suspicious behavior.
Critical weaknesses
- The multi-turn mechanism mainly relies on the running minimum of historical safety scores. While simple and effective, it does not explicitly model intent evolution or complex interactions across turns, and an early false alarm cannot be corrected by later context.
- Conformal calibration improves threshold selection but does not inherently improve the detector itself. If the underlying safety score is unreliable, Conf-Guard may still suffer from poor detection performance or excessive false refusals.
- Conf-Guard requires malicious multi-turn conversations together with annotations indicating when a jailbreak first occurs. Such data can be costly to collect and annotate, which may limit practical deployment.
Questions for authors
Please see weakness
3 comments
response to rebuttal
The rebuttal partially solved my concern. The second and third weakness are not solved. For the second, if the underlying safety score is unreliable, how to ensure the security. For the third, collecting a dataset for multi-turn conversations together with annotations indicating when a jailbreak first occurs is not an easy work since in real-world it is hard to find such conversation.
Thank you for engaging with us! We are glad we addressed your first concern.
For the second concern, it seems to us that you are asking for a method which simultaneously (1) maintains our formal guarantee, and (2) achieves good empirical results (low FRR), even when the underlying method to flag jailbreaks (i.e., the score function) performs poorly. This is impossible to achieve, it is not a weakness of our method. Can you please clarify what you expect as a way to address this perceived weakness?
For the third concern, targeted red teaming is a common practice to collect such conversations before deployment. We use both automated attacks and human red-teaming data. We also want to highlight that our turn-level annotations are generated automatically by the attacker and responder agents, reducing the need for expensive manual annotation. We assess the reliability of these annotations through human validation on dataset subsets, as described in Appendix C.1, “Manual Validation of Benchmarks.” Finally, our experiments show that calibration can work with a small number of malicious conversations in the settings we evaluate.
Thank you very much for reviewing our work. Below we address the weaknesses you mentioned in your review, please let us know if you have any further questions. We have modified the manuscript (changes are highlighted in blue) to better convey the points discussed below. If we addressed all your concerns, we kindly ask that you consider increasing your score.
Note that our theory does not actually rely on the running minimum. Our theorem actually still holds even if we remove the monotonicity assumption. We originally added this requirement to convey the intuition of how we expect the function to behave, but we agree that this made our presentation unclear. We have now modified the manuscript to address this.
Note also that our main contribution is the conformal methodology behind Conf-Guard, which we think of as a wrapper which takes any existing jailbreak classifier or score function and produces a rule for halting conversations satisfying a formal safety guarantee. The score functions we used in the paper are meant to be sensible choices to highlight the usefulness of Conf-Guard, but they are not meant to be set in stone for practitioners nor to represent the best possible score functions.
We agree this is true, but we respectfully disagree with the characterization that this property is a limitation. Our main goal is not to improve upon the underlying detector that is used, but to obtain from it a halting mechanism that satisfies a formal safety guarantee. Thus, our main contribution is orthogonal to the performance of the underlying detector. We see this orthogonality as a strength, as any improvements made to the detector do not void our guarantee and require only re-running calibration. We also note that the exact same argument holds for conformal prediction: it is only as good as its underlying score function, and yet it remains a highly used tool in practice.
First, the need for labelled data is widespread across machine learning, and we believe that listing this need as a weakness of our method is unnecessarily harsh. Second, while we agree that collecting this data can be somewhat expensive, in applications where safety is critical, this cost is typically easily worth the investment. Lastly, we highlight that our method works well even when the calibration dataset is small: see the ablations in Appendix F.1 (formerly Appendix F in the original submission).