- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduce a two-dimensional benchmark that systematically varies lexical and semantic diversity of both trigger and behavior and evaluate representation-based detection methods, DPO alignment and one inference-time defenses. The association analyses show that attack success is associated with several dataset-level properties, while defense and detection outcomes are substantially less predictable; mechanistic analyses further reveal that backdoors vary in how diffusely they are represented across model components, with localized backdoors being more amenable to targeted steering.
Critical strengths
The experiment results offered some useful information. The analysis of how distributional differences between backdoored and clean examples relate to attack success, defense efficacy, and detectability. These associations provide useful context for interpreting performance differences across datasets and models.
Critical weaknesses
The contribution of this paper is limited and narrow in scope. The evaluation focuses on conventional backdoor configurations and a limited set of detection and defense methods, with defenses only covering general DPO and CleanGen. It remains unclear whether the findings generalize to more sophisticated attacks or a broader range of countermeasures.
The current experiment is not sufficient to support some of the claims made in the article. For example, the conclusion that “preference-based safety training is not a reliable backdoor defense” too broad given the current DPO setup. The results do not clearly distinguish whether the observed failure stems from the preference optimization objective, the coverage of the training data, the capacity of the LoRA updates, or the optimization settings. And some of the hyperparameter choices and experimental settings are not clearly explained or analyzed.
Questions for authors
see Critical Weaknesses above
Response to Reviewers
We thank the reviewers for their constructive feedback. We plan to improve our paper by including multiple instantiations of each category, add safety training baselines such as BEEEAR and strengthening our analysis.
We would also like to clarify that we provided the data and code as supplementary materials with the original submission. Our benchmark extends prior work by systematically varying both trigger and behavior characteristics and studying the resulting backdoors through detection, defense, and mechanistic analysis.