- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3(+1)
- Soundness
- 2
Read review
Paper summary
This paper studies cases where benchmarks report a model successfully being jailbroken and determines whether people who aren't experts in the type of harm in question might find it easier to implement the harm if needed. It calls this criterion operational enablement. To answer this, the authors have lay human raters label all 29 instances, finding that only a few successful jailbreaks have this effect. It also finds that conclusions about which attacks are best depend on which evaluator is chosen to make the assessment. There is also an extension questioning the impact of judge choice and a new judge with false negative labels.
Critical strengths
I find the question that this paper asks on the validity of jailbreaking evaluation quite interesting. By separating automated success scores from human judgments of operational enablement on the same fixed outputs, the experimental design makes it possible to move beyond treating a judge’s positive label as evidence of usable harmful capability and asking only whether evaluators agree with each other, toward asking whether a score supports the specific inference it is used to make. I appreciate the carefulness with which the evaluators and calibration are analyzed, as well as the transparency of the negative results and the handling of missing/contested labels.
Critical weaknesses
This paper questions whether StrongReject scores support the inference of usable harm by introducing operational enablement to evaluate jailbreak attempts. However, it doesn’t discuss why this is actually appropriate in StrongReject’s common use cases, which is critical — otherwise, the primary result only shows that 25/29 results are different, which is hard to evaluate on its own. The argument behind this paper seems to be that the choice of jailbreak evaluator can result in poor decisions, but it’s unclear that the operational enablement approach would change outcomes in practice, such as the rank of models. The external replay doesn’t test the new construct, only compares against the source dataset’s labels. While using construct language throughout, this paper does not draw on the rich literatures of measurement theory/validity (e.g. Jacobs & Wallach 2021) nor on harm classifier disagreement (e.g. Fasching & Lelkes 2025) or related annotator disagreement on tasks related to judging harms (e.g. Sap et al. 2022). This creates gaps in the methodology such as the lack of systematically and rigorously creating and validating the rubric, and failing to analyze disagreement, which has been found to be fruitful for such work.
Questions for authors
- Where is operational enablement an appropriate criterion, and why?
- How was the rubric developed?
- In cases with disagreement, are there any qualitatively observable patterns?
2 comments
Thank you for your reassessment
Thank you for taking the time to reassess our revision and for your positive update. Your questions helped us improve the paper’s clarity, clarify the methodological scope, and better motivate its contribution. We appreciate your careful reading and constructive feedback.
Thanks for your response! I'm broadly satisfied with the changes as enhancing clarity, which resolved some methods concerns, and contextualizing the paper to motivate it better. I've bumped my rating.