acceptodds
Under review as a conference paper at ICLR 2027

Auditing the Auditor: Ground-Truth Controls for Natural-Language Explanations of LLM Activations

Abstract

Natural Language Autoencoders (NLAs) generate natural-language explanations of language-model activations and have been proposed as an auditing tool for unverbalized properties such as evaluation awareness. Because the explainer is itself a language model trained for reconstruction rather than faithfulness, a specific-sounding explanation is not by itself evidence about the activation, and the method's authors do not establish calibration of their awareness measurements. We contribute a ground-truth calibration harness for selected released NLA checkpoints (Gemma-3-12B/27B; earlier, Qwen2.5-7B) that scores explanations against facts or conditions we specify, or against a word the model later reveals, under preregistered controls: a wrong-activation (shuffled) control, a visible-text baseline, a blinded judge, and supervised readability probes. Exploratory experiments show that apparent successes on famous facts are indistinguishable from verbalizer prior-completion under a truthful-instruction control and do not recur when the facts are invented (a thin positive control: one cleanly reported fact per scale). In a preregistered hidden-condition experiment, an assigned evaluation-vs-ordinary mode was linearly decoded at both scales, although placebo controls do not rule out token echo or instruction residue; keyword screening nominated 1/31 (12B) and 1/32 (27B) items, and neither met an amended blinded-human reporting criterion (two independent readers; nominated cells not resampled). In a preregistered prospective-reveal experiment on Gemma-3-27B, no early-position free-form NLA explanation contained a candidate word (0/96), while the target model, choosing between the two allowed words from the visible conversation, predicted the later reveal in 61/96 cases (exact McNemar ); a supervised probe did not establish that the reveal was recoverable at that position. Under the wrong-activation control, explanations contained the substituted item's word far more often than the item's own (147 vs. 5 of 576 texts, by literal occurrence), almost entirely immediately before the reveal. We present the control stack as a reusable harness so that future explainers can be evaluated against the same standard.

Primary area:
interpretability and explainable AI
Keywords:
interpretability, natural language autoencoders, explanation faithfulness, activation-based explanations, model auditing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Pu1bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Natural Language Autoencoders (NLAs) translate LLM activations into natural-language explanations and have been proposed as auditing tools for unverbalized properties such as evaluation awareness. But the verbalizer is itself an LM trained for reconstruction, not faithfulness, so a specific-sounding explanation is not by itself evidence about the activation. This paper builds a ground-truth calibration harness for released NLA checkpoints (Gemma-3-12B/27B; earlier Qwen2.5-7B): reporting targets are facts or conditions the authors themselves specify -- or a word the model later reveals -- scored under preregistered controls (wrong-activation/shuffled control, visible-text baseline, blinded judge, supervised readability probes). Findings: apparent lie detection on famous facts is indistinguishable from verbalizer prior-completion under a truthful-instruction control and does not recur with invented facts (Exp. A/B, exploratory; positive control thin -- one cleanly reported fact per scale). In preregistered Exp. C, an assigned inspected-vs-ordinary mode is linearly decodable at both scales (LOFO accuracy 1.000), yet the NLA reports it in 0/31 and 0/32 items after blinded human review (placebos leave token echo / instruction residue unresolved). In preregistered Exp. D, no early-position NLA explanation names the word the model later reveals (0/96), while the target model predicts its own reveal from visible text alone in 61/96 (McNemar p < 0.001);

Critical strengths
  1. The right question, asked the right way: ground truth by construction plus preregistration is exactly how auditing instruments should be validated, and the control stack is a reusable methodological contribution.
  2. Decision-relevant results: the Exp. C probe–verbalizer gap (perfect linear decodability, zero verified reporting) and the Exp. A truthful-rung control change how any NLA output should be read.
  3. Exemplary claim discipline — the thin Exp. B positive control, unresolved Exp. C placebos, the failed judge-validation gate, and the "not a refutation" scoping are all disclosed rather than buried.
  4. Not a one-sided takedown: the Exp. D wrong-activation asymmetry (147 vs. 5) is a real positive result showing where reports do follow the activation.
  5. Full reproducibility artifact: preregistrations, frozen decision rules, traces, and an independently implemented audit script for the confirmatory statistics.
Critical weaknesses
  1. Scope vs. motivation: the systems making the headline auditing claims (Claude NLAs) are unreleased and untested; the two confirmatory scales are both Gemma-3 (within-family), and the 70B checkpoint gets only a qualitative pilot.
  2. Exp. B's null is weak evidence given a positive control of one fact per scale -- "no demonstrable recovery at the instrument's resolution" cuts both ways.
  3. Exp. C's central gap is ambiguous by the authors' own admission: if the probe reads instruction residue, the gap may be over an artifact rather than a hidden model state.
  4. Exp. D's headline contrast (0/96 vs. 61/96) compares differently elicited outputs, and the probe null at the early position means the experiment cannot distinguish "explainer cannot read it" from "nothing yet to read."
  5. Process deviations from preregistration (un-run judge step in Exp. C, substituted readers; judge validation gate failed in Exp. D) are disclosed but accumulate.
  6. Task families are narrow by design (single-token facts, binary modes, word reveals); whether harness verdicts generalize to richer auditing targets (deception, goals) is unknown.
Questions for authors
  1. The Claude NLAs behind the headline evaluation-awareness demonstrations are unreleased. How much should readers update on those from null results on Gemma/Qwen checkpoints, if verbalizer quality scales with the base model?

  2. In Exp. C, with token echo and instruction residue unresolved (at 12B the residue placebo probe matched the main probe exactly), what does the probe–verbalizer gap establish? If the probe is reading the instruction, is the NLA arguably correct not to report an abstract mode?

  3. Exp. B's positive control is one fact per scale. Was a stronger positive control -- facts pre-selected because the AV reports them cleanly -- considered, and would it change the interpretation of the null?

  4. Exp. D's confirmatory comparison elicits outputs asymmetrically (free-form explanation, strictly scored, vs. a forced choice between two candidates). Would a candidate-conditioned AV close the gap -- and if so, is the finding about NLAs or about elicitation format?

  5. Reports become activation-dependent mainly at the pre-reveal position. Is the fair summary that the AV is a lagging indicator -- reporting what the target has effectively already decided? A position-resolved analysis relative to target commitment would help.

  6. The Exp. D judge failed its preregistered human-validation gate (0.60/0.76 vs. required 0.8). Why report assertion-level secondary results (e.g., 55 vs. 1 donor assertions) at all, rather than restricting to judge-independent literal occurrence?

  7. Exp. C's frozen blinded-judge step did not run and was replaced by two household readers under amendment. How do the authors bound the risk that this substitution biased the survival decision on the two nominations?

  8. For the harness to be used as a gate, what quantitative pass bar would the authors set (e.g., detection rate at a specified false-positive rate under the wrong-activation control)? An explicit standard would make the harness much more actionable.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Pu1b (1/2)

Thank you for the detailed review and the eight questions. The general response describes an input-layer error that we found after submission, and two corrective reruns. New analyses are post hoc and were written down before they were run. Below, numbers are from the submitted runs; a number in square brackets is the documented-layer value where a rerun covers it.

Q1. The unreleased Claude NLAs. Readers should update very little. Our results concern three released checkpoints on narrow tasks. The submitted runs gave them activations one block before their documented layer, and only Exp. C (both Gemma scales) and Exp. D (27B) were rerun at the documented layer. Section 3.1 and the limitations say this.

Q2. What the Exp. C gap establishes. Much less than the submission suggested, and we withdraw the "gap" framing. If the probe reads the instruction, the explainer is arguably correct not to report an abstract mode, and the placebos do not rule that out. The task also does not establish that a faithful explanation must report the assigned mode at the scored positions. A new analysis shows that the explanations are not blank about the mode either. Explanations of inspected-mode activations contain the screening keywords more often than explanations of ordinary-mode activations of the same item (mean rate difference 0.063 at 12B [0.151], 0.165 at 27B [0.185]), and the contrast stays positive when the same vectors are verbalized in shuffled slots (0.082 [0.175], 0.193 [0.167]). The contrasts establish mode-associated wording. They do not by themselves establish that explanations correctly state the assigned condition, or distinguish an abstract evaluation-state representation from features of the instructions. The revision reports the probe, this contrast and the registered screen side by side.

Q3. A stronger positive control for Exp. B. We did not run one. Pre-selecting facts that the explainer reports cleanly under truthful instructions is the right design. Without it the null carries little weight in either direction, as your Weakness 2 says. The audit adds a limitation that applies to A and B. In the Gemma runs the key truthful-condition positions were at token indices 16–42, below the minimum index of 50 used by the released web-text extraction pipeline. We have not established the position coverage of the checkpoints' complete training data, so we cannot say whether this contributed to the thin positive control or to the truthful-rung pattern in A. The revision keeps A and B as motivation only.

Q4. Elicitation format. The released explainer takes only an activation vector, so a candidate-conditioned version would be a different method. We did not test one, and the question is open. What we can report is literal naming in the recorded outputs. At Ready. no explanation contains any candidate word [none]. Immediately before the reveal, 76 of 96 contain the revealed word [80], 48 contain both allowed words [49], and of the 38 that name exactly one, 28 name the revealed word [31 of 42]. At Ready. the specified probe did not establish recovery of the eventual answer, so the result does not demonstrate a failure to report information shown to be recoverable, and the comparison with the predictor concerns two differently elicited procedures. The revision reports the registered comparison as registered and no longer interprets it as isolating explanation faithfulness or the benefit of activation access (your Weakness 4).

Q5. A lagging indicator? On these data, somewhat less than that. Candidate naming is concentrated at the token before the answer, and even there 48 of 96 explanations name both allowed words [49]. A plain logit lens on the same vectors selects the revealed word in 63 of 96 items at that position [67] and in 52 and 51 of 96 at the two earlier ones [53 and 55]. We have three positions and no measure of when the target commits, so this is a coarse position-resolved analysis and we make no claim about timing.

The substitution result (the donor's word in 145 of 192 texts and the recipient's in one [152 and none]) shows activation-dependent candidate naming. Donor and recipient candidate pairs are disjoint, so it does not isolate the eventual choice from the pair. Your Strength 4 should be read with that qualification.

AuthorsOpenReview ↗

Response to Reviewer Pu1b (2/2)

Q6. Judge-dependent results. The revision does what you suggest: apart from the registered test, its main text scores explanations by literal occurrence only. Judge-classified assertion counts are in the appendix, labeled as conditional on a judge that failed its validation gate.

Q7. The substituted review. We can bound the bias for the nominated items only. Counting every keyword nomination as a detection gives at most 1/31 at 12B and 1/32 at 27B as submitted, so the substituted readers could have moved each result by at most one item. For cells excluded before any reader saw them, a new analysis removes every shuffle condition from the screen and nominates the same items. At the documented layer the screen nominates no 12B item and two 27B items. The 27B texts have not been assessed, so the corresponding bounds there are 0/31 and 2/32, and we report the two as unresolved keyword nominations. The revision leads with the registered screen and reports the human reads as a secondary, amended step.

In checking the screen we found that the analysis code omitted the preregistered ≤ 2/8 cap on the shuffled rate. No reported count changes. It is one more deviation, and it is disclosed.

Q8. A pass bar. The right form is a detection rate on known positives at a fixed false-positive rate, under a wrong-activation control with opposite-condition donors. The requirement should be set in advance from the intended use and checked on separate validation data. We lack positive-control data validating this complete reporting criterion, so we cannot show that any explainer meets such a bar or justify a numerical threshold from these experiments. We did compute one property of the rule we registered. An idealized keyword reporter that always emits the keywords for inspected-mode vectors and never for ordinary-mode vectors would be nominated on about 20 of 32 items, because donors were not restricted to the opposite mode. When the criterion requires a low target-positive rate after substitution, donors should come from the opposite condition; alternatively, each substituted explanation can be evaluated against its donor's actual condition. The revision says so.

Weakness 1 (scope). Both confirmatory models are Gemma-3, and the 70B checkpoint had only a qualitative pilot. The limitations say so.

Weakness 3 (the gap may be over an artifact). It may; see Q2. The revision withdraws the gap framing.

Weakness 5 (accumulating deviations). They do accumulate, and the list is now longer: the input-layer error, the duplicated start token, the early positions in A and B, and the omitted cap. The revision collects documented deviations and departures in one appendix table.

Weakness 6 (narrow task families). Whether any of this transfers to richer targets is unknown, and the limitations say so.

On the strengths you listed. Strength 2 called the Exp. C probe–verbalizer gap and the Exp. A truthful-rung control decision-relevant. The revision withdraws the first as stated (Q2) and qualifies the second (Q3). We would rather you assess the paper on the narrower claims in the general response.

Presentation. The revision gives each experiment's setup, results and supported conclusion in that order, with a results table in each experiment section, and moves the amendment history to the appendix.

Reviewer_YzG2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper audits Anthropic's released NLA checkpoints, mainly Gemma-3-12B and 27B, on synthetic tasks with targets fixed by construction. It uses a wrong-activation control, a visible-text baseline, supervised probes with placebo arms, and blinded scoring. The analysis reveals several interesting findings: apparent lie detection is matched by the truthful-instruction control, and disappears with invented facts, and the NLA's outputs follow the substituted activation only at the token immediately before the reveal.

Critical strengths
  • Some of the reported results support direct evidence of unfaithful mechanisms and need to be studied in detail to understand where the current setup is vulnerable. At the pre-reveal token, the donor's word appears 145 times and the recipient's once. This is direct evidence that the explanations depend on the activation there.
  • NLAs are gaining a lot of traction for audits, and analysing how faithful they are and these released checkpoints themselves need independent testing
  • The authors report and disclose number for everything including failed gates, annotator conditions
Critical weaknesses
  • Overall, the density and presentation make the paper hard to evaluate throughout. Nearly every claim is hedged, Heavy internal terminology is used before or without definition: "rungs", “persistence positions”. Details essential to intepreting the main results like probe positions, donor composition, response format are deferred to the appendix. The experiment results by themselves are not uniformly presented, and analysed upon.
  • The Exp. D confirmatory endpoint cannot inform NLA quality. The NLA is never asked the question, while the predictor is handed both words. The probe at "Ready." is at chance (0.542), so the choice may not yet be encoded. The paper concedes the test does not isolate a benefit from activation access, yet presents it with p = 8.7×10⁻(¹⁹)
  • The harness is shown not to discriminate, but no known-good readout is shown passing it, and no known-bad explainer is shown failing it. Section 4 omits the detection criteria, whether shuffle controls were used, and any results table, there is no activation ablation
  • The Exp. C shuffle criterion penalises genuine detection. Shuffled donors may come from either mode, so a faithful NLA would also report "inspected" whenever its donor happened to be inspected-mode. It would then fail the ≤2/8 shuffled requirement. At 27B, 7 shuffled cells already reach the ≥5/8 keyword rate. The 0/31 and 0/32 counts are therefore biased toward zero by construction.

Minor inconsistencies

There are a few typos and structural inconsisncies eg: (a assigned -> an assigned, line 087), blinded LLM judging is reported a part of the harness even with failed experiments. Fundamental experiments like logit lens, tuned lens and model reports are not cited. The current set of experiments also heavily rely on existing and documented approaches, so the novelty of the propsed method is not very exciting.

Questions for authors

Same as weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer YzG2 (1/2)

Thank you for the review, and in particular for the point about the shuffle criterion, which led to the most informative analysis of this period. The general response describes an input-layer error that we found after submission, and two corrective reruns. Below, numbers are from the submitted runs; a number in square brackets is the documented-layer value where a rerun covers it.

The Exp. C shuffle criterion. The mechanism you describe is real. Donors were not restricted to the opposite mode, so an explainer that reported the mode of its input would, given an inspected-mode donor, fail the shuffle conditions. We ran three post hoc analyses, written down before they were run. The donor assignment had not been logged, so we reconstructed it from the archived code, seed and inputs. Donor-dependent results are conditional on that reconstruction.

  1. How much the rule can cost. Take an idealized keyword reporter that always emits the keywords for inspected-mode vectors and never for ordinary-mode vectors. Over simulated donor assignments the registered rule nominates it on a mean of 19.6 of 31 items at 12B and 20.2 of 32 at 27B (20 and 20 under the reconstructed assignment). The rule does not force zero. It does miss this idealized reporter on over a third of items.
  2. What it decided here. With every shuffle condition removed, the screen nominates the same items in the submitted runs and at the documented layer. In the submitted 27B run the registered conditions would exclude two position-level cells, both with inspected-mode donors, and no items.
  3. Whether shuffled explanations track donor mode. At 27B all 7 shuffled cells that reach 5/8 have inspected-mode donors (7 of 52 [6 of 52; 5 of 51 if the one inexactly reconstructed control input is omitted]). None of the 44 cells with ordinary-mode donors does [0 of 44]. At 12B no shuffled cell reaches 5/8 as submitted; at the documented layer 6 do, all with inspected-mode donors [6 of 52; 0 of 44]. Across items, the keyword rate is higher for inspected-mode than for ordinary-mode vectors in the real explanations (mean difference 0.063 at 12B [0.151], 0.165 at 27B [0.185]) and in the shuffled-slot explanations of the same vectors (0.082 [0.175], 0.193 [0.167]). At each layer, all four contrasts are positive (Holm-adjusted p approximately 0.0004 across the four contrasts, at the resolution of 10,000 Monte Carlo draws).

Two things therefore hold together. The rule is defective in the way you describe. In these data, though, the shuffle conditions changed no item nomination, so they did not produce the reported counts. What the analysis adds is that the explanations carry mode-associated wording that follows the supplied activation. The contrasts do not by themselves establish that explanations correctly state the assigned condition, or distinguish an abstract evaluation-state representation from features of the instructions. We withdraw the interpretation that the explanations contain no information associated with the mode. The original nomination counts stand as recorded, and so does the failure of the nominated items to meet the amended human-reading criterion. In checking the screen we also found that our analysis code omitted the preregistered ≤ 2/8 cap. No reported count changes, and the revision discloses it.

AuthorsOpenReview ↗

Response to Reviewer YzG2 (2/2)

No known-good readout passes and no known-bad explainer fails. The complete procedure was not validated against known-good and known-bad explainers. That is the paper's main limitation. Exp. B's truthful-rung control succeeded for one of four facts per scale, which is not that validation. The revision states this in the abstract and no longer describes the controls as a validated harness. Section 4 now states its detection criteria and has a results table. Exps. A and B had no wrong-activation control and no activation ablation, and the revision says so.

The Exp. D confirmatory endpoint. The comparison concerns two differently elicited procedures (the explainer is not asked the question; the predictor is given both words) and does not isolate explanation faithfulness or the benefit of activation access. The probe did not establish recovery of the eventual answer at Ready., so the absence of candidate words there does not demonstrate a failure to report information shown to be recoverable. The revision reports the registered test as registered and removes the p-value from the abstract.

The pre-reveal result you listed as a strength. Under substitution at that position, the donor's revealed word occurs in 145 of 192 texts and the recipient's in one [152 and none]. This supports activation-dependent candidate naming. It needs a qualification that the submission did not make. Donor and recipient candidate pairs are disjoint, so the contrast does not isolate reporting the eventual choice within a pair from reporting the pair's candidate words. In the unsubstituted explanations at that position, 48 of 96 name both allowed words [49].

Presentation. The revision defines "rung" and "persistence position" at first use. It moves the response format, the scored positions and the donor composition into the main text, gives each experiment section a results table, states the supported conclusion after each result, and moves the amendment history to the appendix.

Minor points. The typo is fixed. Blinded LLM judging is no longer listed as a working component: it failed validation in Exp. D and did not run in Exp. C. Logit lens and tuned lens are cited. If by "model reports" you mean work on models' self-reports of their internal states, we have added a citation there; please tell us if you meant something else. On novelty, we agree that the individual controls are standard.

I thank the authors for the detailed response, and for taking account for my concerns and suggestions. The engagement with the shuffle-criterion point was productive, I’m glad the mechanism was confirmed and that it led somewhere useful with the mode-contrast finding, I believe this also led to a real contribution that wasn’t in the original submission. The logit lens comparison was also run and framed appropriately.

That said, I maintain my original score at 3, due to the gaps that still persist. I rated a weak acceptance in my original review because I think this kind of independent auditing of released NLA checkpoints is something that is useful for the community, and the paper takes a careful, pre-registered approach to a genuinely hard problem. The revision is more honest, better scoped, and more useful than the original. However, the errors and limitations are real and the input-layer error and the unresolved procedure validation gap are non-trivial gaps that the authors themselves name plainly. I appreciate the effort that went into this paper and the revision, and I wish the authors the best with the work.

Reviewer_iQShOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a set of controls for evaluating Natural Language Autoencoder (NLA) explanations: a shuffled-activation control, a visible-text baseline, a blinded LLM judge, and supervised probes. It applies them to released Gemma-3-12B/27B (and Qwen2.5-7B) NLA checkpoints in four small experiments: instructed lies about famous facts (A) and invented facts (B), an assigned “inspected vs. ordinary” mode (C), and a delayed word reveal (D). The NLA does not report the planted targets. The only activation-dependent reporting occurs at the token immediately before the model outputs the answer.

Critical strengths

Preregistration, with deviations and failed validation gates reported openly.

Critical weaknesses
  • The experiments cannot distinguish a faithful explainer from an unfaithful one.

    • Exp C. is inert by design: the output is a fixed one-sentence verdict, and the scored positions are tokens inside it. A faithful verbalizer has no reason to mention an irrelevant system-prompt detail there since it might not be useful, so 0/31 and 0/32 says little about NLAs. The probe result is expected given the different prompts. The placebo probes confirm it cannot be separated from prompt differences. There is no positive control.

    • Exp D. At “Ready.”, the authors’ own ranking probe is at chance (0.542, p=0.27), so there is no evidence the choice exists yet, and the NLA null is uninterpretable. At the pre-reveal position the next token is the answer, so a logit/tuned-lens baseline is the obvious comparison, and it is not run (Sec. 6).

    • Exps A/B. They only use four facts each. Exp B’s positive control works for one fact per model, so B is uninformative (acknowledged).

-Weak measurement and evidence.

  • The Exp D judge failed its preregistered validation gate (κ = 0.60 and 0.76, below the 0.8 threshold), and the required escalation was not carried out.

  • The Exp C semantic review was added post hoc, after outputs were known. It used two readers from the same household, and the nominated cells were not resampled.

  • Both confirmatory models are Gemma-3, so the replication is within one family.

  • Presentation. The main text is dominated by qualifiers and amendment history, which makes the setups and findings hard to extract.

Questions for authors
  • How does the NLA compare to a logit or tuned lens at the pre-reveal position?

  • Is there a setting where the hidden condition affects behavior, with a positive control showing the harness can detect a true positive?

2 comments
AuthorsOpenReview ↗

Response to Reviewer iQSh (1/2)

Thank you for the review. The revision adopts its main objections, in the terms below. The general response describes an input-layer error that we found after submission, and two corrective reruns. Below, numbers are from the submitted runs; a number in square brackets is the documented-layer value where a rerun covers it.

"The experiments cannot distinguish a faithful explainer from an unfaithful one." The complete procedure was not validated against known-good and known-bad explainers, so its null results do not by themselves establish a failure of faithfulness. Exp. B's truthful-rung control succeeded for one of four facts per scale and does not supply that validation.

Exp. C. The analyzed items have matched visible answers across modes, the scored positions are the verdict token, the final period, and the end-of-turn token, and the task does not establish that a faithful explanation must report the assigned mode there. The probe result may reflect differences between the prompts, the placebo probes do not separate the mode from the instruction's wording, and Exp. C has no positive control. The revision states each of these in the main text and withdraws the interpretation of 0/31 and 0/32 as demonstrating a gap between what the probe recovers and what the explainer reports.

A post hoc analysis changes what we can say. Explanations of inspected-mode activations contain the screening keywords more often than explanations of ordinary-mode activations of the same item (mean rate difference 0.063 at 12B [0.151], 0.165 at 27B [0.185]). The contrast is also positive when the same vectors are verbalized in shuffled slots (0.082 [0.175], 0.193 [0.167]). The contrasts establish mode-associated wording. They do not by themselves establish that explanations correctly state the assigned condition, or distinguish an abstract evaluation-state representation from features of the instructions. We withdraw the broader interpretation that the explanations contain no information associated with the mode.

Exp. D. At the early position, none of the 96 explanations contained a candidate word. The specified probe did not establish recovery of the eventual answer there. This result therefore does not demonstrate a failure to report information independently shown to be recoverable. The comparison with the candidate-conditioned predictor concerns two differently elicited procedures and does not isolate explanation faithfulness or the benefit of activation access.

Question 1, logit lens. We ran it, post hoc, on the archived vectors.

  • Lens. At the pre-reveal position, the final norm and unembedding applied to the same vector select the revealed word over the other allowed word in 63 of 96 items [67]. At the two earlier positions the counts are 52 and 51 of 96 [53 and 55], which does not establish above-chance accuracy.
  • Explanations at pre-reveal. 76 of 96 contain the revealed word [80]. 48 of 96 name both allowed words in the submitted run and 49 of 96 at the documented layer. Among the 38 that name exactly one allowed word, 28 name the revealed word [31 of 42].
  • Under substitution at pre-reveal. The lens prefers the donor's word in 80 of 96 slots [89]. The donor's word occurs in 145 of 192 explanation texts [152].
  • Reading. Occurrence in free text and forced choice are different endpoints. These analyses do not establish a comparative advantage, disadvantage or equivalence. The 96 items share candidate sets and categories, so we report counts on this fixed item set and make no population claim. We did not run a tuned lens. Both lenses are now cited.
AuthorsOpenReview ↗

Response to Reviewer iQSh (2/2)

Exps. A and B. Four facts each, no wrong-activation control, and a positive control in B that succeeded for one fact per model. The audit adds a limitation: in the Gemma runs the key truthful-condition positions were at token indices 16–42, below the minimum index used by the released web-text extraction pipeline. The revision gives the criteria and results of A and B in one table and keeps them as motivation only.

Measurement.

  • Judge gate. The Exp. D judge failed its gate and the escalation was not carried out. Apart from the registered test, whose 0 of 96 equals the literal count, the revised main text scores explanations only by literal occurrence. Judge-classified counts are in the appendix, labeled as conditional on an unvalidated judge.
  • Exp. C review. It was post hoc, by two readers from one household, without resampling. The revision leads with the registered keyword screen (1/31 and 1/32 as submitted) and reports the reads as a secondary, amended step. At the documented layer the screen nominates no item at 12B and two at 27B. Those texts have not been assessed for whether they state the condition, and we report them as keyword nominations whose status is unresolved.
  • A deviation we found. Our analysis code omitted the preregistered ≤ 2/8 cap on the shuffled rate. No reported count changes. The revision discloses it.
  • One family. Both confirmatory models are Gemma-3. The limitations now say so without qualification.

Question 2. We do not provide the requested combination of a behaviorally consequential hidden-condition task and a validated positive control for the reporting criterion. In C, the analyzed items have matched visible answers; A/B's instruction-dependent answers and limited truthful controls do not supply that validation. We could not add such an experiment in this period. The revision names it as the missing experiment.

Presentation. The revision gives each experiment's setup, results and supported conclusion in that order, with a results table in each experiment section, and moves the amendment history to the appendix.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response (1/2): an error in our pipeline and corrective reruns

We thank all three reviewers for careful reviews. Before responding individually, we must report that after submission we identified an error in our activation-capture pipeline.

The error. After submission, we found that the NLA inputs in all submitted experiments (A–D) were Hugging Face hidden_states[K], one block earlier than the checkpoints' documented extraction layer, decoder block K's output (hidden_states[K+1]). The submitted NLA results therefore describe the released verbalizers applied one block before that documented layer. The released extraction code and the checkpoints' metadata define the layer. Checking our capture code against them was our responsibility.

Corrective reruns. In a post hoc corrective rerun of the Gemma-3-27B conditions from Experiments C and D, we replayed reconstructed archived prompts and completions and saved both hidden-state indices 41 and 42 in each forward pass. A hook on decoder block 41 exactly matched index 42 for all 216 sequences. At index 41, 1,007 of 1,008 vectors were bit-identical to their archived counterparts; one C control vector differed because a stripped whitespace token could not be reconstructed exactly. The original duplicated start-of-sequence token was retained. In a second post hoc corrective rerun, declared after the 27B results had been seen, we did the same for Experiment C at Gemma-3-12B (hidden-state indices 32 and 33): the hook on decoder block 32 exactly matched index 33 for all 120 sequences, and all 720 vectors at index 32 were bit-identical to their archived counterparts. The endpoints and comparison code specified in each rerun's declaration were fixed before any of its new explanations was generated. Neither ran a new probe, predictor, judge or human scoring.

EndpointSubmitted runs (original layer)Documented layer
D (27B), Ready.: first-draw explanations containing any candidate word0 of 960 of 96
D (27B), pre-reveal: first-draw explanations containing the archived revealed word; containing both allowed words76; 48 of 9680; 49 of 96
D (27B), pre-reveal, substituted activation: texts containing the donor's / the recipient's archived revealed word145 / 1 of 192152 / 0 of 192
C (27B): items nominated by the registered keyword screen1 of 322 of 32 (new texts not assessed)
C (12B): items nominated by the registered keyword screen1 of 310 of 31
C (27B): inspected-minus-ordinary keyword rate, real / shuffled-slot explanations0.165 / 0.1930.185 / 0.167
C (12B): the same contrast0.063 / 0.0820.151 / 0.175

At 27B the tested conditions reproduce the early-position literal absence and retain positive mode-keyword contrasts, while several counts change. At 12B the contrasts are positive and larger, all inspected-mode real cells reaching 5/8 are at the final period, and the one original nomination does not recur. These historical comparisons do not establish equivalence at either scale and do not isolate the effect of changing the layer. In the descriptive repeatability summaries, prespecified for 12B and added after the 27B results, repeated verbalizations at 12B agreed more closely within each run than across runs; the corresponding 27B summaries overlapped. The Qwen and exploratory conditions not included in these reruns retain the layer limitation.

Two further findings from the same audit. The layer correction preserved a duplicated start-of-sequence token in the Gemma target-model inputs; this departs from standard chat tokenization, and its effect here has not been measured. In the Gemma runs of A and B, the key truthful-condition positions were at token indices 16–42, below the minimum index of 50 used by the released web-text extraction pipeline. We have not established the position coverage of the checkpoints' complete training data.

The revised PDF states the error in the Introduction and in Section 3.1. It keeps every submitted number, labeled "original layer" (one block early), and reports the documented-layer results beside them. We also rewrote the abstract: it now describes the two main tasks, what the explanations contained, and what our results do not establish. The reruns do not address the reviewers' concerns about validation of the procedure or construct validity. We take those up in part 2 and in the individual responses.

AuthorsOpenReview ↗

General response (2/2): limits, new analyses, and the revised claims

Limits the revision adopts. Three recur across the reviews.

  1. The procedure is not validated. The complete evaluation procedure was not validated against known-good and known-bad explainers. Experiment B's truthful-rung positive control succeeded for only one of four facts per scale; it does not establish adequate sensitivity for the main C/D claims. Null results under the procedure therefore do not by themselves establish a failure of explanation faithfulness. The revision says this in the abstract and no longer presents the controls as a "calibration harness" or as a standard for other explainers.
  2. Exp. C. The analyzed items have matched visible answers across modes, and the task does not establish that a faithful explanation must report the assigned mode at the scored positions. The probe result may reflect differences between the prompts. The revision withdraws the reading of 0/31 and 0/32 as a gap between what the probe recovers and what the explainer reports.
  3. Exp. D. At the early position, none of the 96 explanations contained a candidate word. The specified probe did not establish recovery of the eventual answer there. This result therefore does not demonstrate a failure to report information independently shown to be recoverable. The comparison with the candidate-conditioned predictor concerns two differently elicited procedures and does not isolate explanation faithfulness or the benefit of activation access. The revision reports the registered test as registered and removes it from the abstract's claims.

New analyses. Each was written down, with its interpretation rules, before it was run. All are post hoc, on outputs we had already seen. Numbers are from the submitted runs; square brackets give the documented-layer value where a rerun covers it.

  • Exp. C, mode contrast (YzG2). Explanations of inspected-mode activations contain the screening keywords more often than explanations of ordinary-mode activations of the same item: mean rate difference 0.063 at 12B [0.151] and 0.165 at 27B [0.185]. A second verbalization of the same vectors in shuffled recipient slots gives 0.082 [0.175] and 0.193 [0.167], conditional on a donor map that we reconstructed from the archived code, seed and inputs. The contrasts establish mode-associated wording. They do not by themselves establish that explanations correctly state the assigned condition, or distinguish an abstract evaluation-state representation from features of the instructions.
  • Exp. C, the shuffle rule (YzG2). Removing every shuffle condition leaves the observed nominations unchanged, so those conditions did not produce the reported counts. The rule is still defective: an idealized keyword reporter, which always emits the keywords for inspected-mode vectors and never for ordinary-mode vectors, is nominated on only about 20 of 32 items, because donors were not restricted to the opposite mode.
  • Exp. D, literal naming and a logit lens (iQSh, Pu1b). Immediately before the reveal, 76 of 96 first-draw explanations contain the revealed word [80] and 48 contain both allowed words [49]. A plain logit lens on the same vectors selects the revealed word over the other allowed word in 63 of 96 items [67], and in 52 and 51 of 96 at the two earlier positions [53 and 55]. These analyses do not establish a comparative advantage, disadvantage or equivalence between the explainer and the lens.

What the revision claims.

  1. On the tested items at 27B, explanations contain no candidate word at the early position. Immediately before the reveal they contain candidate words that follow the supplied activation. Donor and recipient candidate pairs are disjoint, so this does not isolate reporting the eventual choice from reporting the pair.
  2. In Exp. C the explanations carry mode-associated wording at both scales and both layers. Neither item nominated in the submitted runs met the amended human-reading criterion. At the documented layer the screen nominates no item at 12B and two at 27B, which have not been assessed.
  3. When a criterion requires a low target-positive rate after substitution, donors should come from the opposite condition, or each substituted explanation should be scored against its donor's actual condition. Our registered rule did neither and can reject an idealized keyword reporter.
  4. None of this validates or refutes the explainer, and the procedure itself is not validated.

Appendix G maps each main claim of the submission to its status in the revision.

Presentation. All three reviews found the paper hard to follow. The revision gives each experiment's setup, results and supported conclusion in that order, a results table per experiment section, key terms defined at first use, and the amendment history in the appendix.

This is a large change for a discussion-period revision. Our aim is an accurate paper and public record.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.