acceptodds
Under review as a conference paper at ICLR 2027

FEATURE CANARIES DECOYS, TWINS AND THE LIMIT OF DETECTION OF AN SAE AUDIT PIPELINE

Abstract

Audits that use sparse autoencoders to look for hidden behaviours in language models often end with a null result, and the null says nothing about what could have been found. Measurement science has a fix for this. A procedure states a limit of detection and earns it on blanks, samples known to carry nothing. We build the audit analogue. Canaries, small harmless behaviours, are trained into language models at controlled doses, beside three kinds of blank. Decoys carry the canary's text and no behaviour. Clean twins and the public base carry the training and no canary. Random directions carry an intervention and no feature. The experiment then failed, instructively. Containment caught canary text at the lowest dose we placed in the dictionary's corpus, so its limit could not be located, and the blanks show what the pipeline found instead. Three findings, one per blank. Against decoys, the stages that read the dictionary respond to text, not installation. Against random directions, the gate's steering half passes blanks as often as the canary's own features, 99.8% against 99.2% for entity canaries at the gate's steering scale. The gate's limit is its prompt budget. A behaviour present in 0.35% of cued prompts is caught half the time at 200 prompts, and one present in 1.5% is caught 95% of the time. Against twins and reference models, what separates implants from decoys is training exposure, from about 40 occurrences, not behaviour. The scope is narrow, only nonce-anchored entity canaries installed on Pythia, the easiest case, so these failures are upper bounds. What transfers is the testbed and a short checklist of what any audit should report.

Primary area:
interpretability and explainable AI
Keywords:
sparseautoencoders, ExplainableAI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HZe2OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies new kinds of benign controls that one should use when benchmarking auditing techniques, and in particular when trying to determine whether an auditing technique has important false positives.

Critical strengths

The paper provides extensive results, I may have been able to get more value out of it and evaluate its soundness better if the presentation were better.

Critical weaknesses

The paper has hard-to-parse vague nice-sounding sentences that are hard to read (that sound AI-generated to me, though they would be extremely bad even if a human wrote them). This starts in the abstract "Three findings, one per blank. Against decoys, the stages that read the dictionary respond to text, not installation. Against random directions, the gate’s steering half passes blanks as often as the canary’s own features, 99.8% against 99.2% for entity canaries at the gate’s steering scale" (what is a gate? what is the gate steering scale?) but is present throughout the paper. These made the paper extremely hard for me to parse.

The idea is not original, and controls have extensively been used in prior work, such as https://www.aisi.gov.uk/blog/auditing-games-for-sandbagging-detection or https://alignment.anthropic.com/2026/auditing-overt-saboteur/. To the extent that this work introduces new kinds of controls, it fails to justify the ways in which they differ from previous kinds of controls, and why they are better than controls introduced by prior work. And in particular, the conclusion of Section 4 on what audits showing null results should provide does not seem sound to me: careful controls are essential when claiming a positive result. But only recall on positive examples is essential when claiming a negative one. To the extent your canaries are also positives, I do not understand how this contribution goes beyond the one from AuditBench.

Questions for authors
  • Can you please rewrite the paper in a way that is more human readable?
  • In what ways are your controls different from those used by prior work? In what ways are they better?
Reviewer_kZZ9OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper develops a controlled testbed for evaluating SAE-based audits of learned behaviors. It trains benign canaries into small language models at specified doses and introduces three complementary controls: text-only decoys, clean model twins, and random intervention directions. Applying these controls to an assembled audit pipeline shows that feature selection and explanation can detect canary text without establishing behavioral installation, strong steering can suppress behavior through general model degradation, and reference-model comparisons can identify training exposure without establishing the corresponding behavior. The paper also examines the role of query budgets and decoding choices and proposes a reporting checklist for interpreting audit results.

Critical strengths
  1. The controls address distinct and consequential sources of misleading evidence. Text-only decoys distinguish representations of a string from an installed behavior; random directions test whether an intervention is behavior-specific; and clean twins establish a baseline for model comparisons. Together, these controls provide a useful framework for assessing what an audit result actually demonstrates.

  2. The study investigates its initial findings through weaker steering interventions, explicit loss budgets, alternative baselines, and additional model configurations. The distinction between training exposure and behavioral installation is particularly informative. The authors also distinguish prespecified analyses from post-hoc results and corrections, making the empirical evidence easier to assess.

Critical weaknesses
  1. The abstract and introduction describe the negative results as upper bounds because the installed canaries represent an especially easy case. This inference is not established: easily cued, nonce-anchored behaviors need not be the easiest case for every feature-selection or intervention method. Moreover, the evaluated pipeline is the authors’ own assembly of published components. The discussion’s scope qualifications are helpful, but they do not establish the broader upper-bound claim. The experiments support the value of the controls and identify failures of particular stages; the extent to which those failures constrain other SAE-based audits remains unclear.

  2. The evaluation of the rebuilt causal gate is incomplete for ablation. Table 1 explicitly states that held-out loss was not measured for this configuration, although the proposed gate includes a 2% loss budget. Consequently, the reported 4.5% pass rate for tested features and 0.2% worst-blank pass rate do not establish that the intervention satisfies the full operating criterion. Table 17 reports loss changes for joint ablations, but those measurements concern a different configuration. This is a limited validation gap that appears addressable without changing the study’s design.

Questions for authors
  1. What assumptions justify interpreting these negative results as upper bounds beyond the evaluated pipeline? In particular, does “easiest case” refer to eliciting the behavior or to identifying its causal representation? Please clarify which conclusions are intended to generalize and which remain specific to the selected stages and canary construction.

  2. What is the held-out loss change for the exact ablation configuration evaluated in Table 1? After applying the stated 2% loss budget, do the tested and blank pass rates—and the characterization of ablation as specific but insensitive—remain unchanged?

Reviewer_gw1VOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes borrowing the "limit of detection / blanks" idea from analytical chemistry to evaluate SAE-based audits for hidden behaviours. The authors train harmless "canaries" (mostly made-up entities, plus triggers and styles) into Pythia models at controlled doses. They add three kinds of blank: decoys (text in the SAE corpus but never trained in), clean twins, and random steering directions. They run a four-stage pipeline (latent detection and causal gate, LLM explanation, agent surfacing, discovery) and report that the original goal failed: the limit of detection could not be located, because even the lowest dose was detected. Instead they report three findings. (1) The dictionary-reading stages respond to canary text, not installation. (2) The steering half of the causal gate passes random directions about as often as real latents, so its limit is essentially the prompt budget. (3) A simple loss comparison against a reference model separates implants from decoys, reflecting training exposure rather than behaviour. They close with a five-item reporting checklist and a promised testbed release.

Critical strengths
  1. A well-motivated idea: audit nulls should come with blanks and a stated detection limit. The decoy design (same text in the dictionary corpus, never trained in) cleanly separates "text" from "installation", and the untrained-dictionary result makes this point vividly.
  2. Unusually transparent reporting, with evidence labels, disclosed corrections and negative results, plus a practical checklist and a testbed that could be useful if released.
Critical weaknesses

1.The title and framing promise a limit of detection but the authors state that it was not located. The three findings are largely true by construction or restate known results. Decoys sit in the dictionary corpus at the same dose as implants, so text-based lead selection cannot distinguish them. The loss-ratio result is membership inference, as the authors acknowledge, and closely related work on reference-model and activation differencing (Abu Baker et al.; Minder et al.) and on audit agents versus simpler tools (AuditBench's tool-to-agent gap) should be positioned against more directly. The "limit equals prompt budget" result is essentially a binomial/rule-of-three argument. 2. Only nonce-anchored entities install on small models (410M to 1.4B). while triggers and styles never install, so this is not an audit of hidden behaviours in the usual sense. The paper claim that the failures are "upper bounds" is asserted but not shown. The steering gate is by the authors' own account non-standard and saturates at −5× (99.2% vs 99.8%).

Questions for authors
  1. Can you test the causal gate with norm-matched controls or mean ablation at scales inside your loss budget? Does any specific operating point exist?
  2. Why does ablation fail on installed entities? Is it feature splitting/absorption or redundancy?
  3. How do your findings relate to AuditBench's tool-to-agent gap and its observation that synthetic-document models are easier to audit?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.