- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper compares topic recovery by Natural Language Autoencoders and linear probes to measure the verbalization gap. It also tests prompt-level attacks, different monitor windows, and resampling to assess monitoring reliability.
Critical strengths
This paper evaluate three released checkpoints. They also use positive controls to check whether the verbalizer reads the activation it receives.
Critical weaknesses
- The matched closed-set readout also uses the reconstructor and cosine similarity. The authors do not rule out their effects, but claim that “Most of the gap is in distinguishing, not in generating.”
- The abstract highlights that the preamble attack reduces detection on Qwen-2.5-7B from “42 to 0 percent,” but this result strongly depends on the monitor window. Table 8 shows that, with a window of 10 tokens, detection under the preamble attack is close to clean detection: the gap is only 4percentage points.
- In Table 5, when Transfer = 0, Same as base ranges from 48.6% to 83.3%. However, the authors claim that activation patching “changes nothing about the model’s output.” The claim that the target model no longer uses the topic information also lacks sufficient evidence.
Questions for authors
The probe is refit on attacked activations. How do you rule out representation changes as a cause of the verbalizer’s failure? Can you test a probe trained on clean activations under attack without refitting it?