- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper looks for an internal encoding of the next sub-task while an LLM writes short, multi-step code. Sub-tasks are listed in the prompt, and generation is prefilled up to the point where the next one starts. In Qwen2.5-Coder-32B, a sparse set of attention heads carries this encoding: it can be read out in words before the code is written, and transplanted from Python into C++ to trigger the same operation. The authors also argue that the model keeps the encoding by rereading the task and its progress, rather than storing one state. Checks on other models, few-shot tasks, and chat are more limited.
Critical strengths
The cross-language result is the main strength. An encoding taken from a Python run can make the model carry out the same operation in C++, instead of only copying the next Python tokens. The encoding is also readable a few tokens before the code starts. That is direct evidence for a portable representation of the next specified operation, not just a correlation with the next word. The second strength is the explanation of how this can last. A transformer cannot pass an upper-layer state forward for many tokens. The authors show that changing what these heads read in the task description can redirect the next sub-task, which supports their claim that the model rereads the task rather than carrying one stored goal.
Critical weaknesses
The findings are interesting, but they do not support the title. "Goal Encoding in LLMs" suggests a general property of LLMs. The positive evidence comes from a narrow setup: short coding tasks whose sub-tasks are already written in the prompt, with generation prefilled to the next step, mostly on one code model. Checks on other models use the same task bank. Outside it, few-shot verbalization is weak, and the chat examples are qualitative. A limited coding setup is not enough for this broad claim. The title and conclusion should be narrowed to the tasks actually tested. The reported success rates are also soft. An LLM judge counts a partial match as success, with no human agreement reported. The 122 selected heads are enough to carry the next sub-task in one ablation, but blinding them does not remove the behavior; a larger set is needed for that. So they are not shown to be a unique bottleneck.
Questions for authors
Q1: The title sounds like a claim about LLMs in general. The experiments mostly use one kind of coding task: the steps are already written in the prompt, and the code is filled in up to the next step. Which result shows the same thing outside this task? If the written steps are removed from the prompt, can the model still say what it is about to do? Can a signal taken from Python still make it do the same thing in C++? These answers would change my score. Q2: The paper uses another model to decide if a readout is correct, and a partial match counts. Would a person agree, on a sample of cases? How often is a “correct” readout the full next step, not just a vague phrase like “counting” or “checking a list”?
Suggestion: while the model is writing one step, replace its internal signal with the signal for a different step, and see if the code changes.
3 comments
The title sounds like a claim about LLMs in general.
Our paper does test general settings. In this paper we have conducted 26 experiments spanning diverse settings and eight different transformer LMs, including code generation where the next operation is not specified; non-code settings such as verbalizing the implicit rule in few-shot ICL; reading out the effect of steering vectors; and emergently misaligned LoRA weights in chat settings. Across this wide range of settings we found overwhelming evidence that a small set of goal heads encode an explicit representation of "intentions" that shape the LM's future actions. We appreciate your feedback on this. However, we keep the framing and title of the paper, as we believe they are appropriate given the strength of the evidence we have found.
(Q2)
The paper uses another model to decide whether a readout is correct, and a partial match counts.
Our operationalization of a goal requires persistence: the encoding guides the LM's actions across multiple generation steps. Evaluating a multi-token rollout is a non-trivial challenge in interpretability, which is why we use LLM oracles to evaluate the readouts. We count a partial match with the reference as success only when it is non-trivial. We allow partial matches because a goal can be genuinely present while its pursuit fails for other reasons: the LM may not be competent enough, or the intervention or steering may interfere with the LM's language-modeling capabilities.
- The non-trivial criterion prompts the judge to check whether the readout says something genuinely meaningful about the next sub-goal. Vague phrases such as "counting" or "checking a list" are considered trivial and are not counted as success.
We again invite you to visit https://goalscope.pages.dev/, where we include faithfulness and causality readouts for all 253 specifications and their corresponding oracle judgments.
Would a person agree, on a sample of cases?
Based on your feedback, we have also measured how well human annotators agree with the oracle's faithfulness judgments reported in the paper. The precision is high: in 97% of the cases where the LLM judge marked a readout as a success, a human annotator also considered the verbalization a successful readout of the upcoming operation. We have included the agreement results in Appendix D.3 of the updated submission.
We invite you to try annotating a small sample yourself at https://goalscope-human-annotation.pages.dev/. At the end of the annotation you will be shown your agreement with the oracle judges (claude-sonnet-4.6 and gpt-5). Your annotations are kept completely anonymous. We also invite you to check https://goalscope.pages.dev, where we include all 253 specifications used to evaluate the readouts.
We hope this information will help you better assess the quality of the readouts and of the oracle evaluation pipeline.
(Weakness) The 122 selected heads are not shown to be a unique bottleneck.
Language models are flexible systems with redundant computations. Ablating one computation is often not enough to change the LM's behavior, because the model can compensate with backup computations (Wang et al., 2023; McGrath et al., 2023). We are observing the same phenomenon here. We selected the 122 heads with a threshold on the mask values. We chose the threshold of 0.9 arbitrarily, as a systematic grid search over all the hyperparameters at these LM scales would be computationally infeasible. In our ablations, we find that blinding these 122 heads is not enough to remove the behavior. However, blinding a broader set of 250 heads does remove the LM's capability. Figure 4 includes control experiments for each ablation.
Suggestion: while the model is writing one step, replace its internal signal with the signal for a different step, and see if the code changes.
We have done this experiment in Appendix I.1.1 (see Figure 23) of the updated submission. We create sub-sub-goals by breaking a step into smaller sub-steps joined by a conjunction or disjunction. At the connective points, we swap the key-value states that the goal heads read from, and the LM implements the pending sub-sub-goal of the step transplanted from the source. The causal-shift is 79%, and the redirection still holds across language directions.
However, transplanting the goal only has the desired causal effect when it does not contradict syntactic constraints. For example, assume the LM is implementing checking if any word is a palindrome and the progress is any(w == w[::-. At the last token (in Qwen2.5 it is [::-), if we swap in a different task, count the number of vowels across the words, it is already too late to completely suppress the current action. In these situations, depending on the intervention strength, the generation either breaks, or the LM finishes the current step with 1] and picks up the transplanted task later (with an and or on the next line). When this transfer is performed at syntactically consistent positions, we get strong causal effects, confirming our findings.
We hope we have been able to address your questions and concerns. Let us know if you have any follow-up questions or suggestions for improvements.
Response to Reviewer `6REG`
Thank you for your review of our work. We appreciate your acknowledgement of the frequent re-read insight from our NRGS account. Below we address your questions and concerns. Please also see the general response above for a summary of the major additions and revisions to the paper.
(Q1) The experiments mostly use one kind of coding task, where the steps are already written in the prompt.
In our paper we test whether our insights generalize beyond the controlled setup in several ways:
antonym,country-capital, andperson-occupation. However, this performance depends on the relation: for translation tasks such asenglish-frenchandenglish-spanish, the verbalization performance drops below 20%.We also include the anonymous website https://goalscope.pages.dev, which demonstrates GoalScope verbalizations across different setups: code where the next step is not specified in the prompt, and the non-code settings. We hope this helps you better assess the quality of the results.
Can a signal taken from Python still make it do the same thing in C++?
This is exactly the requirement for plasticity: can the same (goal) encoding guide action in very different environments (Python vs. C++)? We designed our evaluations throughout the paper with this in mind. For example, our causality and causality-shift scores measures exactly what you are asking here. We take an encoding from a Python context and transplant it into an information-free C++ code stub. Success requires the LM to implement the same operation in the C++ environment, with different syntax, libraries, variable names, etc. All of our intervention experiments include cross-language results (py → cpp and cpp → py).
This transfer works in situations where the next operation is unspecified in the prompt. We have checked this qualitatively in notebooks/demos/01.3_unspec_caus_target.ipynb of our submitted codebase. We transplant an encoding into the C++ code stub below:
Without any injection, the LM generates:
We can take the goal encoding from a Python context implementing a goal and transplant it into the C++ code above. The LM then implements the next sub-goal of the transplanted goal in C++. For example, when transplanted from merge sort, the next action divide into two sub-arrays is triggered, and transferring from "find the longest increasing sub-sequence" triggers the LM to initialize the memory for a dynamic programming approach. The results can be reproduced at notebooks/demos/01.3_unspec_caus_target.ipynb.
We are happy to add these results to the appendix if the reviewers think it would improve the paper.