- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes Maximally-Activating Example Metamodels (i.e., MAEMs). A MAEM is a language model trained to take a direction in another model's internal activations and write text that strongly activates that direction, a work that normally requires a quite long procedure when, for instance, trying to interpret SAE directions. The authors fine-tune a copy of the model of interest, first with supervised training and then with reinforcement learning. The reward is how closely the generated text's activations match the target direction, so it needs no human or LLM labels, and every output can be checked with a forward pass. Results are strong: a single trained MAEM beats corpus search on activating held-out directions and generalizes to unseen languages and code. It is then applied to four interpretability tasks: identifying the concept behind steering vectors, reading hidden backdoor payloads, reading model "workspace" content, and activating and explaining SAE features.
Critical strengths
- Well-motivated idea that solves a concrete problem in interpretability: instead of searching a fixed corpus for text that activates a direction, the authors train a model to generate such text. This removes the dependence on what the corpus happens to contain and scales well, since one forward pass replaces scanning millions of tokens.
- Strong and general results: a single checkpoint outperforms corpus search, GCG and EPO, and generalizes to unseen languages and code. I also appreciated that the authors present thorough results in the context of four different interpretability applications. The same model handles all four without being trained on most of these direction types.
Critical weaknesses
Not critical:
- Generality and cost: a MAEM is initialized from the model's weights, trained on activations from one layer, and rewarded by that same model. All experiments use a single model and mainly a single layer. Does this mean a new MAEM has to be trained for every model, or even every layer? Training seems expensive. Could the authors report the overall training cost and comment on whether a MAEM might transfer to other layers or models? Not required for acceptance, but for future directions: it'd be great to run a small experiment about transfer to different architectures.
- Role of RL: the main difference from prior activation inverters, IIUC, seems to be the RL stage with the activation-based reward. However, the authors mention that SFT and RL are not compared at matched compute. How much of the improvement comes from the RL objective itself, as opposed to scale or training data?
Questions for authors
/
2 comments
I thank the authors for their response and maintain my positive score.
Rebuttal
We thank the reviewer for taking the time to read our work and providing an engaging discussion.
W1. Generality across models and layers, and cost.
Transfer across layers. We run a cross-layer experiment where the layer-42 MAEM reads activations of the same subject model from 10 layers between 6 and 62. We find that
Table R1. Zero-shot transfer of a layer-42 MAEM to other layers. Fidelity (held-out): the paper's centred cosine between the target activation and the activation of the MAEM's text read at the target's own layer, mean over 512 held-out activations. Fidelity (control): same score against a different target activation. WorkspaceBench (WB): pass rate under the benchmark's protocol, judged by Claude Sonnet 5. In association (WB.a, n = 91, chance 0.125) the readout must name a concept the prompt implies but never states; in basic readout (WB.b, n = 100, chance 0.20) it must state the one-word answer the model has not yet given.
Transfer across models. We have not tested this and leave it for future work. It would require addressing the issue of the second model having its own residual stream (different basis). One solution, other than a new MAEM for the second model, would be a learned map between the two activation spaces.
Cost. The released MAEM (SFT and RL) took roughly 120 GPU-hours on B200s.
W2. Role of the RL stage.
Yes, the RL stage is the main difference from prior activation inverters. We now measure and report the effect of the RL objective on the activation inversion task, starting RL from SFT checkpoints at 2M, 4M and 8M examples, and comparing at matched total compute against MAEM SFT-only, InversionView, and InverseScope.
At 1e20 FLOPs, compute-matched SFT reaches 0.54 held-out cosine and SFT+RL reaches 0.77-0.78. SFT-only held-out cosine is highest early in training (0.64 at 1M examples) and declines with more examples. RL improves on SFT on every metric, with the largest gain on SAE features (0.51 to 0.83-0.85 on unseen features). The final result is nearly the same from every SFT starting point.
InversionView and InverseScope trained on the same compute budget reach 0.39-0.42 held-out cosine, which is below the MAEM SFT.
See the new Appendix K for the full training curves. Note in Table R2 we present InversionView on GPT-2 Medium which performed better than InversionView on a 2-layer transformer. In our response to Reviewer 2mwT’s W2b, we report the SFT checkpoint on the Section 5 experiments, where SFT+RL outperforms it on most tasks.
Table R2. All methods at a matched budget of 1e20 FLOPs (including activation collection). MAEM SFT-only is a separate run extended to 14.8M examples to match the budget.
[1] Camila Blank, Agam Bhatia, Euan Ong, Neel Nanda. WorkspaceBench: Evaluating Interpretability Methods for the Global Workspace. 2026