- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies new kinds of benign controls that one should use when benchmarking auditing techniques, and in particular when trying to determine whether an auditing technique has important false positives.
Critical strengths
The paper provides extensive results, I may have been able to get more value out of it and evaluate its soundness better if the presentation were better.
Critical weaknesses
The paper has hard-to-parse vague nice-sounding sentences that are hard to read (that sound AI-generated to me, though they would be extremely bad even if a human wrote them). This starts in the abstract "Three findings, one per blank. Against decoys, the stages that read the dictionary respond to text, not installation. Against random directions, the gate’s steering half passes blanks as often as the canary’s own features, 99.8% against 99.2% for entity canaries at the gate’s steering scale" (what is a gate? what is the gate steering scale?) but is present throughout the paper. These made the paper extremely hard for me to parse.
The idea is not original, and controls have extensively been used in prior work, such as https://www.aisi.gov.uk/blog/auditing-games-for-sandbagging-detection or https://alignment.anthropic.com/2026/auditing-overt-saboteur/. To the extent that this work introduces new kinds of controls, it fails to justify the ways in which they differ from previous kinds of controls, and why they are better than controls introduced by prior work. And in particular, the conclusion of Section 4 on what audits showing null results should provide does not seem sound to me: careful controls are essential when claiming a positive result. But only recall on positive examples is essential when claiming a negative one. To the extent your canaries are also positives, I do not understand how this contribution goes beyond the one from AuditBench.
Questions for authors
- Can you please rewrite the paper in a way that is more human readable?
- In what ways are your controls different from those used by prior work? In what ways are they better?