- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Natural Language Autoencoders (NLAs) translate LLM activations into natural-language explanations and have been proposed as auditing tools for unverbalized properties such as evaluation awareness. But the verbalizer is itself an LM trained for reconstruction, not faithfulness, so a specific-sounding explanation is not by itself evidence about the activation. This paper builds a ground-truth calibration harness for released NLA checkpoints (Gemma-3-12B/27B; earlier Qwen2.5-7B): reporting targets are facts or conditions the authors themselves specify -- or a word the model later reveals -- scored under preregistered controls (wrong-activation/shuffled control, visible-text baseline, blinded judge, supervised readability probes). Findings: apparent lie detection on famous facts is indistinguishable from verbalizer prior-completion under a truthful-instruction control and does not recur with invented facts (Exp. A/B, exploratory; positive control thin -- one cleanly reported fact per scale). In preregistered Exp. C, an assigned inspected-vs-ordinary mode is linearly decodable at both scales (LOFO accuracy 1.000), yet the NLA reports it in 0/31 and 0/32 items after blinded human review (placebos leave token echo / instruction residue unresolved). In preregistered Exp. D, no early-position NLA explanation names the word the model later reveals (0/96), while the target model predicts its own reveal from visible text alone in 61/96 (McNemar p < 0.001);
Critical strengths
- The right question, asked the right way: ground truth by construction plus preregistration is exactly how auditing instruments should be validated, and the control stack is a reusable methodological contribution.
- Decision-relevant results: the Exp. C probe–verbalizer gap (perfect linear decodability, zero verified reporting) and the Exp. A truthful-rung control change how any NLA output should be read.
- Exemplary claim discipline — the thin Exp. B positive control, unresolved Exp. C placebos, the failed judge-validation gate, and the "not a refutation" scoping are all disclosed rather than buried.
- Not a one-sided takedown: the Exp. D wrong-activation asymmetry (147 vs. 5) is a real positive result showing where reports do follow the activation.
- Full reproducibility artifact: preregistrations, frozen decision rules, traces, and an independently implemented audit script for the confirmatory statistics.
Critical weaknesses
- Scope vs. motivation: the systems making the headline auditing claims (Claude NLAs) are unreleased and untested; the two confirmatory scales are both Gemma-3 (within-family), and the 70B checkpoint gets only a qualitative pilot.
- Exp. B's null is weak evidence given a positive control of one fact per scale -- "no demonstrable recovery at the instrument's resolution" cuts both ways.
- Exp. C's central gap is ambiguous by the authors' own admission: if the probe reads instruction residue, the gap may be over an artifact rather than a hidden model state.
- Exp. D's headline contrast (0/96 vs. 61/96) compares differently elicited outputs, and the probe null at the early position means the experiment cannot distinguish "explainer cannot read it" from "nothing yet to read."
- Process deviations from preregistration (un-run judge step in Exp. C, substituted readers; judge validation gate failed in Exp. D) are disclosed but accumulate.
- Task families are narrow by design (single-token facts, binary modes, word reveals); whether harness verdicts generalize to richer auditing targets (deception, goals) is unknown.
Questions for authors
-
The Claude NLAs behind the headline evaluation-awareness demonstrations are unreleased. How much should readers update on those from null results on Gemma/Qwen checkpoints, if verbalizer quality scales with the base model?
-
In Exp. C, with token echo and instruction residue unresolved (at 12B the residue placebo probe matched the main probe exactly), what does the probe–verbalizer gap establish? If the probe is reading the instruction, is the NLA arguably correct not to report an abstract mode?
-
Exp. B's positive control is one fact per scale. Was a stronger positive control -- facts pre-selected because the AV reports them cleanly -- considered, and would it change the interpretation of the null?
-
Exp. D's confirmatory comparison elicits outputs asymmetrically (free-form explanation, strictly scored, vs. a forced choice between two candidates). Would a candidate-conditioned AV close the gap -- and if so, is the finding about NLAs or about elicitation format?
-
Reports become activation-dependent mainly at the pre-reveal position. Is the fair summary that the AV is a lagging indicator -- reporting what the target has effectively already decided? A position-resolved analysis relative to target commitment would help.
-
The Exp. D judge failed its preregistered human-validation gate (0.60/0.76 vs. required 0.8). Why report assertion-level secondary results (e.g., 55 vs. 1 donor assertions) at all, rather than restricting to judge-independent literal occurrence?
-
Exp. C's frozen blinded-judge step did not run and was replaced by two household readers under amendment. How do the authors bound the risk that this substitution biased the survival decision on the two nominations?
-
For the harness to be used as a gate, what quantitative pass bar would the authors set (e.g., detection rate at a specified false-positive rate under the wrong-activation control)? An explicit standard would make the harness much more actionable.
2 comments
Response to Reviewer Pu1b (2/2)
Q6. Judge-dependent results. The revision does what you suggest: apart from the registered test, its main text scores explanations by literal occurrence only. Judge-classified assertion counts are in the appendix, labeled as conditional on a judge that failed its validation gate.
Q7. The substituted review. We can bound the bias for the nominated items only. Counting every keyword nomination as a detection gives at most 1/31 at 12B and 1/32 at 27B as submitted, so the substituted readers could have moved each result by at most one item. For cells excluded before any reader saw them, a new analysis removes every shuffle condition from the screen and nominates the same items. At the documented layer the screen nominates no 12B item and two 27B items. The 27B texts have not been assessed, so the corresponding bounds there are 0/31 and 2/32, and we report the two as unresolved keyword nominations. The revision leads with the registered screen and reports the human reads as a secondary, amended step.
In checking the screen we found that the analysis code omitted the preregistered ≤ 2/8 cap on the shuffled rate. No reported count changes. It is one more deviation, and it is disclosed.
Q8. A pass bar. The right form is a detection rate on known positives at a fixed false-positive rate, under a wrong-activation control with opposite-condition donors. The requirement should be set in advance from the intended use and checked on separate validation data. We lack positive-control data validating this complete reporting criterion, so we cannot show that any explainer meets such a bar or justify a numerical threshold from these experiments. We did compute one property of the rule we registered. An idealized keyword reporter that always emits the keywords for inspected-mode vectors and never for ordinary-mode vectors would be nominated on about 20 of 32 items, because donors were not restricted to the opposite mode. When the criterion requires a low target-positive rate after substitution, donors should come from the opposite condition; alternatively, each substituted explanation can be evaluated against its donor's actual condition. The revision says so.
Weakness 1 (scope). Both confirmatory models are Gemma-3, and the 70B checkpoint had only a qualitative pilot. The limitations say so.
Weakness 3 (the gap may be over an artifact). It may; see Q2. The revision withdraws the gap framing.
Weakness 5 (accumulating deviations). They do accumulate, and the list is now longer: the input-layer error, the duplicated start token, the early positions in A and B, and the omitted cap. The revision collects documented deviations and departures in one appendix table.
Weakness 6 (narrow task families). Whether any of this transfers to richer targets is unknown, and the limitations say so.
On the strengths you listed. Strength 2 called the Exp. C probe–verbalizer gap and the Exp. A truthful-rung control decision-relevant. The revision withdraws the first as stated (Q2) and qualifies the second (Q3). We would rather you assess the paper on the narrower claims in the general response.
Presentation. The revision gives each experiment's setup, results and supported conclusion in that order, with a results table in each experiment section, and moves the amendment history to the appendix.
Response to Reviewer Pu1b (1/2)
Thank you for the detailed review and the eight questions. The general response describes an input-layer error that we found after submission, and two corrective reruns. New analyses are post hoc and were written down before they were run. Below, numbers are from the submitted runs; a number in square brackets is the documented-layer value where a rerun covers it.
Q1. The unreleased Claude NLAs. Readers should update very little. Our results concern three released checkpoints on narrow tasks. The submitted runs gave them activations one block before their documented layer, and only Exp. C (both Gemma scales) and Exp. D (27B) were rerun at the documented layer. Section 3.1 and the limitations say this.
Q2. What the Exp. C gap establishes. Much less than the submission suggested, and we withdraw the "gap" framing. If the probe reads the instruction, the explainer is arguably correct not to report an abstract mode, and the placebos do not rule that out. The task also does not establish that a faithful explanation must report the assigned mode at the scored positions. A new analysis shows that the explanations are not blank about the mode either. Explanations of inspected-mode activations contain the screening keywords more often than explanations of ordinary-mode activations of the same item (mean rate difference 0.063 at 12B [0.151], 0.165 at 27B [0.185]), and the contrast stays positive when the same vectors are verbalized in shuffled slots (0.082 [0.175], 0.193 [0.167]). The contrasts establish mode-associated wording. They do not by themselves establish that explanations correctly state the assigned condition, or distinguish an abstract evaluation-state representation from features of the instructions. The revision reports the probe, this contrast and the registered screen side by side.
Q3. A stronger positive control for Exp. B. We did not run one. Pre-selecting facts that the explainer reports cleanly under truthful instructions is the right design. Without it the null carries little weight in either direction, as your Weakness 2 says. The audit adds a limitation that applies to A and B. In the Gemma runs the key truthful-condition positions were at token indices 16–42, below the minimum index of 50 used by the released web-text extraction pipeline. We have not established the position coverage of the checkpoints' complete training data, so we cannot say whether this contributed to the thin positive control or to the truthful-rung pattern in A. The revision keeps A and B as motivation only.
Q4. Elicitation format. The released explainer takes only an activation vector, so a candidate-conditioned version would be a different method. We did not test one, and the question is open. What we can report is literal naming in the recorded outputs. At
Ready.no explanation contains any candidate word [none]. Immediately before the reveal, 76 of 96 contain the revealed word [80], 48 contain both allowed words [49], and of the 38 that name exactly one, 28 name the revealed word [31 of 42]. AtReady.the specified probe did not establish recovery of the eventual answer, so the result does not demonstrate a failure to report information shown to be recoverable, and the comparison with the predictor concerns two differently elicited procedures. The revision reports the registered comparison as registered and no longer interprets it as isolating explanation faithfulness or the benefit of activation access (your Weakness 4).Q5. A lagging indicator? On these data, somewhat less than that. Candidate naming is concentrated at the token before the answer, and even there 48 of 96 explanations name both allowed words [49]. A plain logit lens on the same vectors selects the revealed word in 63 of 96 items at that position [67] and in 52 and 51 of 96 at the two earlier ones [53 and 55]. We have three positions and no measure of when the target commits, so this is a coarse position-resolved analysis and we make no claim about timing.
The substitution result (the donor's word in 145 of 192 texts and the recipient's in one [152 and none]) shows activation-dependent candidate naming. Donor and recipient candidate pairs are disjoint, so it does not isolate the eventual choice from the pair. Your Strength 4 should be read with that qualification.