- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a Representational Effective Theory (RET) to abstract large language model computations into macrostates learned from hidden activations using a BYOL/JEPA-style self-supervised objective. These macrostates are used to represent long reasoning traces as sequences of higher-level states, which the authors interpret as distinct phases of the model's reasoning process. The authors demonstrate that these representations yield coherent and interpretable trajectories, enable early prediction of downstream behaviors such as sycophancy and alignment faking, and provide causal handles for steering model generation.
Critical strengths
-
The paper demonstrates that RET can identify coherent reasoning trajectories from a model's internal computations, which have multiple interesting applications, such as the demonstrated prediction of sycophancy and alignment faking, and interventions that influence model generation. The general capturing of model behavior in these macro states seems very useful for understanding and evaluating the complex computations of large language models.
-
The evaluations indicate that RET captures the semantic context of tokens rather than primarily their lexical identity. For example, occurrences of the same token (such as "and" in Figure 3) are represented according to their surrounding context, unlike the evaluated SAE baselines. This suggests that RET captures higher-level contextual and temporal structure, which is also supported by its improved next-macrostate prediction performance.
Critical weaknesses
- The paper claims that the macro-state trajectories are interpretable but does not evaluate this sufficiently. It provides a lot of qualitative examples, but leaves them for the reader to judge. The LLM-based MIS score provides some insights; however, this is based on LLM-generated descriptions for LLM-annotated states, so it remains unclear how much of this is LLMs reinforcing themselves and how much is useful insight for humans. Since interpretability is one of the central claims of the paper, the evaluations in this regard are not fully convincing to me.
- The paper focuses mostly on the NuminaMath domain. I agree that mathematical reasoning is interesting, but it would be good to validate the method on other domains and types of reasoning as well, e.g., coding, planning, or common-sense reasoning, which might follow different patterns. It remains open whether RET would capture similar meaningful states in these domains.
Questions for authors
- Q1: The macro-states shown in figures 1, 7, etc. do not appear one after the other, but there is some jumping around, oscillating between G10 and G7, for example, in figure 1. While this generally makes sense, how do you handle it for next-state prediction? It seems this could be better captured as a stochastic process, or even as an explicit automaton with multiple possible next states.
- Q2: Why do you choose the 50-token window for calculating the MIS score? Isn't this abstracting such oscillations away, or maybe even losing macro-states completely if they appear only a few times?
- Q3: Have you ablated the choice of K and G? Would be interesting to see if there are some macro-state types that stay consistent despite the choice of K and G.
- Q4: Since it is my main weakness point: how do you define interpretability for RET? The macro-states are a simplification of the thinking traces, for sure, yes, but it still takes a long time to understand the mapping of states to the tokens and the general trend of which state follows which. It feels like a human still needs to do a lot of work, and then the states are not even guaranteed to be correct interpretations. Have you considered applying process mining to macro-state sequences to discover more concise representations of the underlying reasoning processes, such as recurring transition patterns or higher-level state machines?
Response to Authors
Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my concerns. In particular, I find the copy-forward baseline and longer-horizon prediction results convincing.
Overall, I appreciate the rebuttal and maintain my positive assessment of the paper.