- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper asks whether an LLM's state immediately before generation can predict a representation of its eventual response. It fits a linear map from the last prompt-token state to the mean activation over response tokens. On held-out chat prompts, the predicted vectors are close to the actual answer vectors, although this is a prediction of an averaged representation rather than the response text. The authors use the map to ask which aspects of a response are predictable and whether this signal can help anticipate unwanted behavior.
Critical strengths
-
The paper establishes a convincing core empirical result. The linear map on its own predicts held-out mean-answer activations substantially better than simple controls. This supports a real relationship between the pre-generation state and the later answer representation in the setting studied.
-
The paper helps explain what its high prediction score captures. The map recovers broad speaker-related qualities more reliably than topic or specific token information, making clear that predicting an answer representation is different from foreseeing the detailed answer.
-
The authors examine whether this relationship holds beyond the main model. In the tested OLMo checkpoint sequence, the map is already predictive before post-training and changes more across SFT than across the later stages. Although descriptive, this comparison helps locate when the relationship changes. Taken together, these experiments are why I favor a weak rather than clear rejection.
Critical weaknesses
-
The application comparisons do not isolate the effect of the metamodel. Figure 6 includes direct context projections and the mapped readout usually wins, but the methods use different layers. The metamodel also sees 6,468 or 16,000 trait-eliciting pairs in addition to generic chat pairs, while the context-derived baseline does not receive an independently selected layer (Appendix H.1). The gain may therefore depend on these choices rather than on a reusable generic map. For retrieval, Figure 20 inspects the top 100 prompts from the metamodel's training pool without reporting their behavioral hit rate or a quantitative comparison with direct context rankings (Appendix H.2).
-
The proposed generality has not yet been demonstrated as a downstream advantage. If I understand correctly, ParaScopes already maps current states linearly to representations of future text, and Persona Vectors already uses answer-derived directions to forecast these unwanted traits from the final prompt state. In this paper, projecting the predicted answer onto a behavior direction is algebraically another linear score of the prompt state; the inverse-map retrieval also ranks prompts against a fixed context-space direction. The test is whether one metamodel trained on generic data predicts or retrieves unseen behaviors better, or with much less behavior-specific data, than direct context readouts at matched layers. Figure 6 fits metamodels with trait-eliciting pairs, and Figure 20 does not report top-k behavioral hit rates against such readouts. A positive result on this comparison would address my concern.
Questions for authors
-
Do the Figure 6 gains persist at a common layer when the metamodel is trained only on generic chat pairs? For retrieval, how often do top-ranked held-out prompts actually elicit the behavior compared with direct context-direction ranking?
-
Just out of curiosity, could the much larger Base→SFT loss of metamodel transfer in Figure 3C be related to the density of parameter updates? Mukherjee et al. (2025) report denser SFT updates than DPO/RL updates in the models they study. Can you compare the sparsity and magnitude of the updates across your OLMo checkpoints with the corresponding transfer losses?