acceptodds
Under review as a conference paper at ICLR 2027

Goal Encoding in LLMs

Abstract

We investigate whether and how LLMs encode goals: internal neural representations that precede and guide action, that are flexible enough to be realized through many different execution paths. We construct a coding setting with multi-step tasks that allows us to evaluate goal-seeking behavior and the selection of sub-goals. In this setting, we localize goal-seeking causal effects in the output of a sparse set of attention heads. The same heads can be used to decode the model's future actions in natural language: e.g., "I am counting unique values,” or "I'm checking if a word is palindrome.” We verify that this encoding is causal for model's next action, persists across long token sequences, and is plastic: transplanting it into a different environment (e.g., another programming language) triggers the same operation, realized with a different sequence of tokens. Although these heads are extracted using a coding task, we find that their goal-seeking and goal-verbalization properties generalize to natural text and few-shot ICL settings. Finally, we discuss how transformer LMs, whose computation is bounded by layer depth, can maintain such a goal encoding across multiple tokens through frequent re-computations.

Primary area:
interpretability and explainable AI
Keywords:
llm, goal pursuit, interpretability, causal mediation analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6REGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks for an internal encoding of the next sub-task while an LLM writes short, multi-step code. Sub-tasks are listed in the prompt, and generation is prefilled up to the point where the next one starts. In Qwen2.5-Coder-32B, a sparse set of attention heads carries this encoding: it can be read out in words before the code is written, and transplanted from Python into C++ to trigger the same operation. The authors also argue that the model keeps the encoding by rereading the task and its progress, rather than storing one state. Checks on other models, few-shot tasks, and chat are more limited.

Critical strengths

The cross-language result is the main strength. An encoding taken from a Python run can make the model carry out the same operation in C++, instead of only copying the next Python tokens. The encoding is also readable a few tokens before the code starts. That is direct evidence for a portable representation of the next specified operation, not just a correlation with the next word. The second strength is the explanation of how this can last. A transformer cannot pass an upper-layer state forward for many tokens. The authors show that changing what these heads read in the task description can redirect the next sub-task, which supports their claim that the model rereads the task rather than carrying one stored goal.

Critical weaknesses

The findings are interesting, but they do not support the title. "Goal Encoding in LLMs" suggests a general property of LLMs. The positive evidence comes from a narrow setup: short coding tasks whose sub-tasks are already written in the prompt, with generation prefilled to the next step, mostly on one code model. Checks on other models use the same task bank. Outside it, few-shot verbalization is weak, and the chat examples are qualitative. A limited coding setup is not enough for this broad claim. The title and conclusion should be narrowed to the tasks actually tested. The reported success rates are also soft. An LLM judge counts a partial match as success, with no human agreement reported. The 122 selected heads are enough to carry the next sub-task in one ablation, but blinding them does not remove the behavior; a larger set is needed for that. So they are not shown to be a unique bottleneck.

Questions for authors

Q1: The title sounds like a claim about LLMs in general. The experiments mostly use one kind of coding task: the steps are already written in the prompt, and the code is filled in up to the next step. Which result shows the same thing outside this task? If the written steps are removed from the prompt, can the model still say what it is about to do? Can a signal taken from Python still make it do the same thing in C++? These answers would change my score. Q2: The paper uses another model to decide if a readout is correct, and a partial match counts. Would a person agree, on a sample of cases? How often is a “correct” readout the full next step, not just a vague phrase like “counting” or “checking a list”?

Suggestion: while the model is writing one step, replace its internal signal with the signal for a different step, and see if the code changes.

3 comments
AuthorsOpenReview ↗

Response to Reviewer `6REG`

Thank you for your review of our work. We appreciate your acknowledgement of the frequent re-read insight from our NRGS account. Below we address your questions and concerns. Please also see the general response above for a summary of the major additions and revisions to the paper.

(Q1) The experiments mostly use one kind of coding task, where the steps are already written in the prompt.

In our paper we test whether our insights generalize beyond the controlled setup in several ways:

  1. Code where the next operation is not in the prompt. Appendix L.1 includes qualitative examples of GoalScope readouts in settings where the next operation appears nowhere in the prompt. The readout can often verbalize the pending operation correctly.
  2. Goals at different levels of the hierarchy. In Appendix I.1.1 we test whether the frequent re-read insight holds across levels of the goal hierarchy. In Figure 24, while the LM is in the middle of implementing one goal (quick sort), we swap the k/v states of the corresponding tokens with those of another goal (merge sort). The LM switches to the next operation of the transplanted goal ("dividing the array into two halves" for merge sort), a sub-goal that is not specified anywhere in the prompt. Table 8 in the updated submission quantifies this effect.
  3. Non-code settings. In Section 6 and Appendix L.2 we test whether GoalScope can verbalize the LM's next action beyond coding setups. These investigations use the same goal heads localized in the controlled coding setup.
    • Appendix L.2.1: GoalScope verbalizes the implicit rule in few-shot ICL prompts in 80%+ of cases for relations such as antonym, country-capital, and person-occupation. However, this performance depends on the relation: for translation tasks such as english-french and english-spanish, the verbalization performance drops below 20%.
    • Appendix L.2.2: GoalScope can often verbalize ahead of time the LM's "thought" that shapes its upcoming action. It can capture the effect of steering vectors and of emergent-misalignment (EM) LoRA weights in chat settings.

We also include the anonymous website https://goalscope.pages.dev, which demonstrates GoalScope verbalizations across different setups: code where the next step is not specified in the prompt, and the non-code settings. We hope this helps you better assess the quality of the results.

Can a signal taken from Python still make it do the same thing in C++?

This is exactly the requirement for plasticity: can the same (goal) encoding guide action in very different environments (Python vs. C++)? We designed our evaluations throughout the paper with this in mind. For example, our causality and causality-shift scores measures exactly what you are asking here. We take an encoding from a Python context and transplant it into an information-free C++ code stub. Success requires the LM to implement the same operation in the C++ environment, with different syntax, libraries, variable names, etc. All of our intervention experiments include cross-language results (py → cpp and cpp → py).

This transfer works in situations where the next operation is unspecified in the prompt. We have checked this qualitatively in notebooks/demos/01.3_unspec_caus_target.ipynb of our submitted codebase. We transplant an encoding into the C++ code stub below:

"""auto func(std::vector<int> arr) {
    if (arr.size() <= 1) { return arr; }
   """

Without any injection, the LM generates:

""" std::sort(arr.begin(), arr.end());
    return arr;
}"""

We can take the goal encoding from a Python context implementing a goal and transplant it into the C++ code above. The LM then implements the next sub-goal of the transplanted goal in C++. For example, when transplanted from merge sort, the next action divide into two sub-arrays is triggered, and transferring from "find the longest increasing sub-sequence" triggers the LM to initialize the memory for a dynamic programming approach. The results can be reproduced at notebooks/demos/01.3_unspec_caus_target.ipynb.

We are happy to add these results to the appendix if the reviewers think it would improve the paper.

The title sounds like a claim about LLMs in general.

Our paper does test general settings. In this paper we have conducted 26 experiments spanning diverse settings and eight different transformer LMs, including code generation where the next operation is not specified; non-code settings such as verbalizing the implicit rule in few-shot ICL; reading out the effect of steering vectors; and emergently misaligned LoRA weights in chat settings. Across this wide range of settings we found overwhelming evidence that a small set of goal heads encode an explicit representation of "intentions" that shape the LM's future actions. We appreciate your feedback on this. However, we keep the framing and title of the paper, as we believe they are appropriate given the strength of the evidence we have found.

(Q2)

The paper uses another model to decide whether a readout is correct, and a partial match counts.

Our operationalization of a goal requires persistence: the encoding guides the LM's actions across multiple generation steps. Evaluating a multi-token rollout is a non-trivial challenge in interpretability, which is why we use LLM oracles to evaluate the readouts. We count a partial match with the reference as success only when it is non-trivial. We allow partial matches because a goal can be genuinely present while its pursuit fails for other reasons: the LM may not be competent enough, or the intervention or steering may interfere with the LM's language-modeling capabilities.

  • The non-trivial criterion prompts the judge to check whether the readout says something genuinely meaningful about the next sub-goal. Vague phrases such as "counting" or "checking a list" are considered trivial and are not counted as success.

We again invite you to visit https://goalscope.pages.dev/, where we include faithfulness and causality readouts for all 253 specifications and their corresponding oracle judgments.

Would a person agree, on a sample of cases?

Based on your feedback, we have also measured how well human annotators agree with the oracle's faithfulness judgments reported in the paper. The precision is high: in 97% of the cases where the LLM judge marked a readout as a success, a human annotator also considered the verbalization a successful readout of the upcoming operation. We have included the agreement results in Appendix D.3 of the updated submission.

We invite you to try annotating a small sample yourself at https://goalscope-human-annotation.pages.dev/. At the end of the annotation you will be shown your agreement with the oracle judges (claude-sonnet-4.6 and gpt-5). Your annotations are kept completely anonymous. We also invite you to check https://goalscope.pages.dev, where we include all 253 specifications used to evaluate the readouts.

We hope this information will help you better assess the quality of the readouts and of the oracle evaluation pipeline.

(Weakness) The 122 selected heads are not shown to be a unique bottleneck.

Language models are flexible systems with redundant computations. Ablating one computation is often not enough to change the LM's behavior, because the model can compensate with backup computations (Wang et al., 2023; McGrath et al., 2023). We are observing the same phenomenon here. We selected the 122 heads with a threshold on the mask values. We chose the threshold of 0.9 arbitrarily, as a systematic grid search over all the hyperparameters at these LM scales would be computationally infeasible. In our ablations, we find that blinding these 122 heads is not enough to remove the behavior. However, blinding a broader set of 250 heads does remove the LM's capability. Figure 4 includes control experiments for each ablation.

Suggestion: while the model is writing one step, replace its internal signal with the signal for a different step, and see if the code changes.

We have done this experiment in Appendix I.1.1 (see Figure 23) of the updated submission. We create sub-sub-goals by breaking a step into smaller sub-steps joined by a conjunction or disjunction. At the connective points, we swap the key-value states that the goal heads read from, and the LM implements the pending sub-sub-goal of the step transplanted from the source. The causal-shift is 79%, and the redirection still holds across language directions.

However, transplanting the goal only has the desired causal effect when it does not contradict syntactic constraints. For example, assume the LM is implementing checking if any word is a palindrome and the progress is any(w == w[::-. At the last token (in Qwen2.5 it is [::-), if we swap in a different task, count the number of vowels across the words, it is already too late to completely suppress the current action. In these situations, depending on the intervention strength, the generation either breaks, or the LM finishes the current step with 1] and picks up the transplanted task later (with an and or on the next line). When this transfer is performed at syntactically consistent positions, we get strong causal effects, confirming our findings.


We hope we have been able to address your questions and concerns. Let us know if you have any follow-up questions or suggestions for improvements.

Reviewer_i9zyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies internal representations of upcoming operations in language models. Using coding tasks with explicitly specified subgoals, the authors identify a sparse set of attention heads whose outputs can be verbalized through GoalScope and transferred across programming languages to influence subsequent generation. They also examine how these representations change during an operation and propose a non-recurrent goal-seeking mechanism in which the model repeatedly compares task information with its progress. Additional experiments explore readouts in in-context learning and natural-language generation.

The evidence for portable operation representations is interesting. My main reservation concerns the proposed mechanism for selecting and maintaining goals, which is less well supported than the paper's framing suggests.

Critical strengths
  1. The controlled coding setup makes several useful distinctions possible. Explicit subtask boundaries and multiple pre-implementation pivots allow the authors to separate an upcoming operation from the broader task. Transferring representations between Python and C++ also provides a meaningful test of whether the representation extends beyond the particular tokens used to implement an operation.

  2. The attention-head interventions provide substantial evidence that the selected heads carry behaviorally relevant information. With access preserved for 250 selected heads, frontier implementation remains at 91.3%, compared with 92.5% without intervention and 8.7% when an equal number of random heads are spared. Conversely, restricting those 250 selected heads to the first token reduces performance to 15.8%. These results make a stronger case than readout accuracy alone.

  3. The appendix includes useful controls and negative results. In particular, the authors investigate alternative interpretations of query transfer, test additional model variants, and report failures under misleading code prefixes. These experiments help identify the actual limits of the proposed account, even though those limits need to be reflected more clearly in the main conclusions.

Critical weaknesses
  1. Appendix I.3 is difficult to reconcile with the main mechanistic claim. Query transfer selects the source frontier in 84% of cases, while the operation predicted from transferred progress appears in only 5%. This suggests that the query already carries a selected operation. The authors acknowledge this result, but the conclusion still presents goal-progress comparison as an established mechanism.

  2. The depth argument bounds the number of serial updates to a latent state, not how long that state remains accessible. Later tokens can directly attend to an earlier cached representation without relaying it through every intervening token. The experiments do not distinguish this possibility from fresh goal reconstruction, so the argument that persistence requires repeated recomputation is incomplete.

  3. The "causality" score credits partial semantic agreement without checking successful code execution (Appendix D.2). Table 13 labels 0 + 1 for w in set(words)) as ADAPTED, although it is syntactically invalid as printed. This example needs clarification. Compilation and functional tests on a subset would make the transfer claim more convincing than the reported judge spot checks.

  4. The contribution beyond existing task and function vectors needs clearer definition, since those works already establish portable task representations. Appendix E compares selected heads with averaged attention outputs and residual states under one steering configuration. Appendix J then obtains useful residual readouts through an OV projection. The baseline gap therefore partly depends on how the representation is decoded.

  5. The broader claims about goal pursuit exceed the tested settings. Persistence is measured over implementations of 10–40 tokens. ICL verbalization reaches 36.2% overall and 9% on extractive tasks, while the chat analysis presents four selected examples from eight prompts. These are useful exploratory results, but they do not establish a shared causal mechanism across domains.

Questions for authors
  1. How should the 84% source-frontier selection and 5% progress-consistent selection in Table 10 change the interpretation of Section 5? Is there evidence locating the earlier computation that selects the frontier?

  2. Can an intervention distinguish repeated reconstruction from the original task description from repeated retrieval of a goal representation stored at an earlier generated position?

  3. Appendix I amplifies the patched heads' output contributions by a factor of two, and query transfer repeatedly injects the same source-pivot queries. How do the results change without amplification and with intervention restricted to the pivot?

  4. What fraction of successful cross-language transfers compile and implement the complete target operation? Please also clarify the ADAPTED example in Table 13 and quantify agreement between the oracle and human judgments.

  5. How many examples and predicates are used to localize the heads, and how are they separated from the evaluation corpus? Does performance hold for predicates excluded from localization?

8 comments
AuthorsOpenReview ↗

Response to Reviewer `i9zy`

Thank you for your feedback and questions. We appreciate your recognition that our tests are designed to isolate encodings of higher-level goals from the specific token sequences that implement them. We have incorporated answers to all of the concerns you raised in the paper. Below we engage with your concerns in detail. Please also see the general response above for a summary of the major additions and revisions to the paper.

(W1, Q1) Query transfer selects the source's next sub-goal in 84% of cases. How should this change the interpretation of Section 5? Is there evidence locating the earlier computation that selects it?

Transformer LMs are flexible systems that can carry redundant computations for the same thing. The redundancy can be parallel or sequential (McGrath et al., 2023; Gur-Arieh et al., 2025). With sequential redundancy, the LM gets multiple opportunities to perform a computation, here selecting the next sub-goal. Query transfer on the goal heads triggers the target sub-goal in the patched run ~57% of the time. This is non-trivial evidence that the goal heads' queries carry the progress so far, and that this progress encoding actively participates in selecting the next sub-goal. To understand the complete picture, we stress test the insight in different situations.

  • In Appendix I.2.2 (previously I.3), when the source's next sub-goal is also in the destination's task list, query transfer consistently triggers that sub-goal instead of the task NRGS predicts. As you point out, and as we have also candidly acknowledged in our paper, this suggests the next sub-goal has already been selected before the goal heads enter the computation.
  • In the newly added investigations in Appendix I.2.1, we find that other signals, such as task order, also influence which sub-goal is selected. A mixture of different computations participates in selecting the next action.

These results are complementary to our finding in Section 5.1.2. They help complete the picture, revealing additional computations involved in the next sub-goal selection that can be separately measured with targeted interventions. We also note that these results do not affect the other NRGS prediction: that the LM frequently re-reads the goal from the task span. That prediction is validated with a different set of targeted causal interventions (see Section 5.1, Appendix I.1, and Appendix J).

\

(W2, Q2) Can an intervention distinguish repeated reconstruction from the task description from repeated retrieval of a goal stored at an earlier generated position?

Yes! We continued our investigation during the review period and added Appendix J, which directly addresses this question. We briefly describe the setup and findings here, and invite you to read the details in Appendix J.

We segment the prompt into two parts: the task span and the progress span. We then transfer the encoding of the task span, the progress span, or both, from either the source run or the clean destination run. This gives a 2×2 grid of which run (source vs. destination) contributes the encoding of which span (task vs. progress) in the patched run. We measure the causal-shift (whether the patched run implements the source's next sub-goal) under three types of encoding transfer: (1) all residual states (Figure 27); (2) key-value transfer on the goal heads; (3) key-value transfer on all attention heads.

A goal stored at earlier generated positions lives in the code-progress span, so this design pits the alternative you describe against the prediction that the LM frequently re-reads the goal from the task span. Transplanting the progress span alone has a much lower success rate in every intervention. With residual patching, the source task span alone redirects the LM in 89% of cases, and the code-progress span alone in only 6%. The progress span does hold traces of the upcoming operation: when the value states in the KV transfer are amplified by 4.0, the success rate starts to climb (Figure 28). This pre-committed signal is much weaker, and is easily suppressed when it competes against the dominant computation path that frequently re-reads the goal from the task span.

(Q3) Results without amplification, and with the intervention restricted to the pivot.

Effect of amplification on KV transfer. The updated Figure 28 in Appendix J shows that even without amplification, KV transfer triggers the source's next sub-goal in 77% of cases. Amplification is therefore not necessary to support the frequent re-read insight. It does improve performance (93% with ×2 and 97% with ×4) by strengthening the signal's effect on downstream computation.

Repeated injection of the same source-pivot queries. We found this to be necessary. When the query transfer is applied only at the pivot, the first generated token often does align with the predicted sub-goal, but on the very next step the LM goes back to implementing its native next sub-goal. For example:

  • native next sub-goal b\mathrm{b} = count the number of words that start with a vowel
  • target next sub-goal c\mathrm{c} = check if any word is a palindrome

With query transfer applied only at the first token, the LM starts with any(, presumably the start of the target implementation any(w == w[::-1] for w in words). After that first token, it returns to the native sub-goal, and the final generation is any(w[0] in 'aeiou' for w in words). This behavior is consistent with our frequent re-read prediction: after any(, the LM re-reads the task span, sees that the native sub-goal is still pending, and tries to go back to implementing it (in this case imperfectly). Repeated injection was needed to sustain the target sub-goal across generation steps.

Our revision uses a cleaner query-transfer intervention for multi-token generation, which accommodates frequent re-reading by allowing the progress-so-far information to evolve in the goal-head queries. We transfer the source query once at the pivot, without any amplification. This causes the goal heads at the pivot to shift their attention to a different task. During generation, we restrict the goal heads' attention to the task that received the most attention at the pivot (see Figure 25), while all the non-goal heads operate as usual. We have updated all the query-transfer scores in the new submission with this cleaner intervention.

(Q4, W3)

What fraction of successful cross-language transfers compile and implement the complete target operation?

We have checked this on the cross-language transfer results (py → cpp and cpp → py) reported in the paper:

ExperimentDirectionCausal shift (out of 128)Compiles
KV transfer (Figure 5 / Table 7)Python → C++115111 (96.5%)
KV transfer (Figure 5 / Table 7)C++ → Python121121 (100%)
Q transfer, order-diff (Figure 6 / Table 10)Python → C++7166 (93.0%)
Q transfer, order-diff (Figure 6 / Table 10)C++ → Python6564 (98.5%)
Q transfer, order-matched (Table 9)Python → C++10896 (88.9%)
Q transfer, order-matched (Table 9)C++ → Python112111 (99.1%)

Nearly all transfers compile when the destination language is Python, and the rate is moderately lower when it is C++. We have inspected some of the C++ failures and found they are mostly minor syntax errors, such as a missing parenthesis or semicolon. In a few C++ cases, the LM called a library or helper function that was not present in the prefilled prompt. For example, with the target check if a number is a palindromic number, prefilled up to ... auto b =, the LM generated:

...
    auto b = (n == reverseNumber(n));
...

int reverseNumber(int n) { ... }

We have also uploaded our full code base as supplementary material. The transfer intervention demos notebooks/demos/04_kv_transfer.ipynb and notebooks/demos/05_q_transfer.ipynb show the cross-language transfer experiments and the compilation checks.

The ADAPTED example in Table 13.

This example comes from Appendix K.2, where we plant a misleading prefix (here 0 +) to check the LM's ability to adapt to planted obstacles. The target sub-goal is count the number of distinct words. After the planted obstacle 0 +, the LM generated 1 for w in set(words)). This is indeed syntactically invalid, but it is only missing the sum( before the 1. We label it as ADAPTED because it is still a genuine attempt to implement the target sub-goal, although an imperfect one. A goal guides future actions, but it does not guarantee success (Bratman, 1987; Cohen & Levesque, 1990).

We have clarified this in the caption of the updated Table 15 (previously Table 13). We would also like to note that this experiment is tangential to the core investigations and does not affect any of the scores presented in the main text.

Agreement between the oracle and human judgments.

Based on your feedback, we have also measured how well human annotators agree with the oracle's faithfulness judgments reported in the paper. The precision is high: in 97% of the cases where the LLM judge marked a readout as a success, a human annotator also considered the verbalization a successful readout of the upcoming operation. We have included the agreement results in Appendix D.3 of the updated submission.

We invite you to try annotating a small sample yourself at https://goalscope-human-annotation.pages.dev/. At the end of the annotation you will be shown your agreement with the oracle judges (claude-sonnet-4.6 and gpt-5). Your annotations are kept completely anonymous. We also invite you to check https://goalscope.pages.dev, where we include all 253 specifications used to evaluate the readouts.

Reviewer_kcuvOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies whether LLMs encode internal representations of goals that precede and guide their actions. The authors introduce GoalScope - a method to identify a sparse set of attention heads carrying information about an upcoming sub-goal and verbalizes this information in natural language. The main experiments are on multistep coding tasks. Transplanting these representations between prompts can induce the same operation in a different programming language, suggesting that they encode what operation to perform independently of its token-level implementation. The authors also hypothesize that transformers maintain goals by repeatedly recomputing the intended sub-goal from the task description and history.

Critical strengths

None. I do not work in interpretability and do not understand the domain well.

Critical weaknesses
  • Generalization beyond controlled code generation is limited.
  • GoalScope repeatedly injects the same activation during every generation step sometimes with amplification. It shows that the representation can steer behavior, but may overstate what can be decoded from an unmodified forward pass. How well does a non-recurrent, single position probe predict the model’s future behavior?
  • Evaluation depends on an LLM judge accepting any “non-trivial” partial agreement. This is arbitrary.
Questions for authors

Please see weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer `kcuv`

Thank you for your review. We appreciate your candor about your familiarity with the area, and we have tried to keep our answers below to the point and self-contained. Please also see the general response above for a summary of the major additions and revisions to the paper.

(W1) Generalization beyond controlled code generation is limited.

In our paper we test whether our insights generalize beyond the controlled setup in several ways:

  1. Code where the next operation is not in the prompt. Appendix L.1 includes qualitative examples of GoalScope readouts in settings where the next operation appears nowhere in the prompt. The readout can often verbalize the pending operation correctly.
  2. Goals at different levels of the hierarchy. In Appendix I.1.1 we test whether the frequent re-read insight holds across levels of the goal hierarchy. In Figure 24, while the LM is in the middle of implementing one goal (quick sort), we swap the key-value states of the corresponding tokens with those of another goal (merge sort). The LM switches to the next operation of the transplanted goal ("dividing the array into two halves" for merge sort), a sub-goal that is not specified anywhere in the prompt. Table 8 in the updated submission quantifies this effect.
  3. Non-code settings. In Section 6 and Appendix L.2 we test whether GoalScope can verbalize the LM's next action beyond coding setups. These investigations use the same goal heads localized in the controlled coding setup.
    • Appendix L.2.1: GoalScope verbalizes the implicit rule in few-shot ICL prompts in 80%+ of cases for relations such as antonym, country-capital, and person-occupation. However, this performance depends on the relation: for translation tasks such as english-french and english-spanish, the verbalization performance drops below 20%.
    • Appendix L.2.2: GoalScope can often verbalize ahead of time the LM's "thought" that shapes its upcoming action. It can capture the effect of steering vectors and of emergent-misalignment (EM) LoRA weights in chat settings.

We also include the anonymous website https://goalscope.pages.dev, which demonstrates GoalScope verbalizations across different setups: code where the next step is not specified in the prompt, and the non-code settings. We hope this helps you better assess the quality of the results.

(W2) The interventions may overstate what can be decoded from an unmodified forward pass.

We would like to clarify how our interventions relate to the unmodified forward pass. In causal mediation analysis, we take a representation from a source run of the model, which is an unmodified forward pass. We then plant it in a different run of the model with a different context, and check whether the transplanted encoding changes the LM's behavior. If it does, we can attribute the change to the transplanted encoding, which captures some information from the unmodified source run. The content we transplant is always read from an unmodified forward pass, and the intervention tests whether that content is enough to drive the LM's behavior in a different context in a predictable way.

This technique is long established in the causal inference literature (Pearl, 2000), and has become a standard technique in interpretability research (Vig et al., 2020; Geiger et al., 2021; Meng et al., 2022; Wang et al., 2023; see Mueller et al., 2024 for a survey).

How well does a non-recurrent, single-position probe predict the model's future behavior?

The non-recurrent goal-seeking (NRGS) account proposes a computational motif for how a depth-bounded transformer LM sustains a (goal) encoding without running out of layers. This is unrelated to probing the LM's future behavior, for which we rely on GoalScope (Figure 2; details in Appendix D). GoalScope reads the goal heads' outputs at a single token position of the unmodified source run. The readout prompt contains no information about the source context, so everything in the readout comes from the injected encoding (Appendix D.1.2). In our controlled code-generation setup, we evaluate whether GoalScope can read out the LM's next action before it begins the implementation. It faithfully reads out the next action 90%+ of the time at the indent pivot, 3 tokens before the implementation starts.

Note that this is different from a traditional classification probe which would need to be trained to predict among predefined behavior classes. GoalScope does not require any training or predefined classes; it simply patches the goal heads' outputs into another context.

In all of our intervention experiments we report two scores: (i) causal-shift, whether the LM changes its behavior in the predicted way, and (ii) readout-shift, whether GoalScope captures that change. We have also tested whether GoalScope reads out the LM's next action beyond the controlled coding setup (see our response to your previous question).

(W3) Evaluation depends on an LLM judge accepting any "non-trivial" partial agreement.

Our operationalization of a goal requires persistence: the encoding guides the LM's actions across multiple generation steps. Evaluating a multi-token rollout is a challenge in interpretability, which is why we use LLM oracles to evaluate the readouts. We count a partial match with the reference as success only when it is non-trivial.

Based on your feedback, we have also measured how well human annotators agree with the oracle's faithfulness judgments reported in the paper. The precision is high: in 97% of the cases where the LLM judge marked a readout as a success, a human annotator also considered the verbalization a successful readout of the upcoming operation. We have included the agreement results in Appendix D.3 of the updated submission.

We invite you to try annotating a small sample yourself at https://goalscope-human-annotation.pages.dev/. At the end of the annotation you will be shown your agreement with the oracle judges (claude-sonnet-4.6 and gpt-5). Your annotations are kept completely anonymous. We also invite you to check https://goalscope.pages.dev, where we include all 253 specifications used to evaluate the readouts.

We hope we have been able to address your questions and concerns. Let us know if you have any follow-up questions or suggestions for improvements.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We are grateful to the reviewers for their feedback. We have updated our submission accordingly, including additional experiments to sharpen our contributions. These updates have strengthened our submission. Below we summarize the major additions and revisions:

  • We have conducted new experiments that further confirm that the LM frequently re-reads the goal from the task specification, across different situations and levels of the goal hierarchy.
    • New Appendix J confirms that the goal is sustained and pursued with frequent re-reading, instead of being carried forward in the latents of earlier generated tokens [ i9zy ].
    • Figure 28 in Appendix J shows the effect of amplification on KV transfer [ i9zy , kcuv ].
    • Appendix I.1.1 shows that the LM sustains goals at different levels of the hierarchy with the same computational motif. The same goal heads re-read the broader goal (e.g., merge sort vs. quick sort, whose sub-goals are not written in the prompt) and the sub-sub-goals within one sub-goal. We validate this with targeted interventions [ 6REG , kcuv ].
  • We have also added evidence that progress encoding (among other things) dictates the next action. In Appendix I.2.1, we have looked further into which LLM computations participate in selecting the next operation. In our controlled setting, the dominant computation is the goal-head queries inhibiting the progress so far in the context to surface the pending operations. However, we also find non-trivial evidence for a few other signals, such as the order of the tasks and the number of completed tasks. These new experiments are complementary to our findings in Section 5.1.2 and help complete the picture [ i9zy ].
  • Our revision uses a cleaner query-transfer intervention for multi-token generation (Figure 25), avoiding the amplification. Our updated results confirm that our findings are consistent with this cleaner setup [ i9zy ].
  • In response to reviewer requests, we have had human annotators evaluate a sample of GoalScope readouts. We find high agreement between the human annotators and the oracle judge verdicts reported in the paper (Appendix D.3). In 97% of the cases the LLM judge marked YES, the human annotators agreed [ 6REG , i9zy , kcuv ].
  • In the newly added Appendix M, we confirm that the core insights from NRGS generalize across LMs of different families and sizes, up to GLM-4.5-Air with 106B parameters [ 6REG, i9zy ].
  • We improved clarity and readability by reorganizing and improving the writing in response to reviewer feedback. Appendix N and Section 5.2 further clarify the scope of our claims [ 6REG , i9zy , kcuv ].


We are also providing the following resources to help the reviewers and readers better assess the quality of our results:


We also respond to each reviewer's questions and concerns individually. We invite the reviewers to examine the updated submission. We welcome any additional questions, comments, or suggestions for further improvement.



Thanks,
Authors

Dear reviewers, please review the authors’ rebuttal and provide your responses as soon as possible. Your timely feedback is important for moving the discussion forward. Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.