acceptodds
Under review as a conference paper at ICLR 2027

LLMs program using Reusable and Composable Abstract Representations

Abstract

When writing code, the same computation can be expressed in different programming languages and through different implementation strategies. We show that the activations of LLMs capture this shared structure in reusable and composable abstract representations when they generate code. These representations emerge in the middle layers, are reusable across programming languages, and afford the model an impressive amount of generalization ability when generating code under various interventions. Using activation steering, we show that function, programming language, and implementation strategy can be independently manipulated. The independent interventions on language and function can be flexibly composed to control code generation, even in cases when the prompt contains no information about language and function. We additionally find that finetuning on examples of arbitrarily defined functions in one programming language transfers to other languages, consistent with the models reusing abstract features of code generation that are shared across different examples. Finally, to study how such abstract representations come to be, we implement a minimal model of shared, reusable, and composable structure in a synthetic setting, and show that transformers trained on such data learn representations that reuse these components across contexts.

Primary area:
interpretability and explainable AI
Keywords:
Abstraction, Activation Steering, Generalization, Programming, Large Language Models, Factored Representations

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yvRqOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether coding models retain task information in reusable internal states that separate function, programming language, and implementation strategy. It first screens 26 models for their ability to generate code after attention to the task specification is removed. It then uses donor-derived state interventions to transport, install, and compose task factors across contexts in selected models. Additional experiments examine a synthetic source with known structure, cross-language transfer from program examples, and Python-only fine-tuning. The central finding is that persistent task states support substantial cross-context reuse, although the interpretation of these states as abstract programs is stronger than what some of the controls establish.

Critical strengths
  1. The intervention design provides meaningful evidence of reusable task information. Estimating a function edit outside the recipient language, and a language edit outside the recipient function, is substantially more informative than showing that task identity can be decoded from activations. Composing these edits in an otherwise underspecified prompt is especially compelling. The execution-based evaluation and negative controls make the main result difficult to dismiss as a generic perturbation effect. The distinction between two-factor composition and the more difficult three-factor setting is also useful rather than being hidden in an aggregate score.

  2. The computational-modifier experiments strengthen the contribution beyond task swapping. Appendix A.14.2 estimates modifications while excluding the recipient’s base function, language, and wording. Successful transfer in this setting is a stronger test of reuse than installing a function that already appears in the donor pool. The nontrivial success rates across all four tested models, together with unsuccessful random and wrong-modifier controls, provide a useful empirical result even without accepting a globally additive account of program representations.

Critical weaknesses
  1. Section 5.1 overstates what the experiments establish about the content of the retained state. Non-identical generated code in Table 1 rules out straightforward verbatim reproduction, but it does not rule out a compressed task description or an algorithmic sketch that the model subsequently realizes as code. The strategy-transfer example likewise shows that donor information can influence computational organization, not that a prompt-summary explanation has been excluded. The operational definition in Section 6—state from one context supporting predictable reuse in another—is supported. The stronger statement that the state is neither a prompt summary nor a copy of an implementation needs a discriminating test or narrower wording.

  2. The generalization evidence should be interpreted more narrowly. The high transplant success rates are conditional on finding donors whose task information survives masking. For example, Table 5 reports 13/90 qualifying pairs for Qwen on faromin, followed by 13/13 successful transplants. That is strong conditional reuse, but not broad unconditional transfer. The fine-tuning experiments also do not clearly distinguish learning a new computation from binding a new name to a familiar computation: vantril counts distinct digits, while thulep computes the difference between the largest and smallest digit. These experiments remain informative, but they provide weaker evidence for acquisition of previously unknown algorithms than the main narrative suggests.

Questions for authors
  1. Which result is intended to distinguish a reusable computational representation from a compressed instruction or algorithm-description state? If no existing experiment separates these explanations, would the authors narrow the conclusion in Section 5.1 to the operational claim made in Section 6?

  2. How should the SFT results be interpreted relative to the “untouched base, described in plain words” comparison in Figure 13? Do the results demonstrate acquisition of a computation the model previously could not implement, or mainly cross-language transfer of a newly learned name-to-computation association? This distinction affects the significance of the generalization claim.

1 comment
AuthorsOpenReview ↗

Response to Reviewer yvRq

We appreciate the reviewers' positive comments about the intervention/experimental design of our study, particularly with respect to the strength of our evidence when composing representation. We also think it’s quite compelling that we can estimate the edits without using the recipient’s context in the donor sequences!

The referee points out that some elements of section 5 are confusingly worded, and potentially obfuscate some results. We agree and propose to make significant changes to the section to remedy this. We will start by renaming 5.1 from “what is an abstraction?” to “How abstract are these program representations?” to better represent the actual content of the section. Additionally, we plan to rewrite the full section so that our exact contributions and experiments are clear. The role that our experiments play in establishing a general picture of “abstraction” will be confined to the conclusion section to clearly differentiate motivation/discussion from concrete results. Excepts of these rewrites can be found in the Overall Response.

The reviewer also correctly points out that not all tests show strong transplant success. However, we interpret the fact that the failures are correlated to donors that survive masking as a feature. If the donor doesn’t survive the masking operation, then it likely is not forming the structured reusable representation for the given computation (and unmasked generation would work likely primarily through re-reading the provided program and, perhaps, translating it). In light of this, we think of this evidence as supporting the generalization behavior being dependent on the existence of the representation. This point should be made clearer, and will use the extra room in the revision for it. As an additional point, the results of the fine tuning experiments depend strongly on how well the fine-tune tune is optimized. We have not optimized our fine-tune, since our point is simply that it is possible to generalize with very little data– not to develop a useful low-data training algorithm. Thus, we caution against interpreting the algorithm success rates as more than showing a qualitative pattern. This is discussed in the appendix, but we can use the extra room in the revision to clarify.

We discuss our thoughts about whether the model is truly learning a new computation versus binding an already established one in the question responses below.

Q1: We believe that the current tests do support that the representation is not a copy of the prompt (the prompt is 8X longer than the suffix, so it can't be a copy of the prompt). However, like the referee points out, we do not show that the representation is not a “summary” of the prompt. What is even meant by a summary is left somewhat open (admittedly by us, but also the reviewer). In fact, it might even be reasonably argued that an organized abstract representation itself is a type of summary (compression) of the prompt – or vice-a-versa, that a compressed summary can be considered an abstraction. As suggested, we will make the wording more narrow– as well as rewrite section 5.1 for clarity. While we plan more changes to section 5.1, the specific part in question will be edited as follows:

  • "figure 6 gives a generic picture of what the model stores at its suffix: a language-invariant form of a program that is neither a prompt summary not an exact copy of an implementation it has seen."

Q2: Any good coding model should be able to perform any invented algorithm if it is clearly explained in a prompt. All of these programming languages are universal, so any new algorithm can be thought of as a combination of existing primitives. This is all to say that, while we agree that “vantril” and “thulep” can be described in terms of existing algorithms, we think it is pretty unlikely that they exist in enough density in the training data for a model to have this particular combination of primitives already established as an abstract computation in the same way that the “sum of two squares” likely would be. Our experiments go somewhat towards showing this, in fact. For example, models find the more complicated invented algorithms “kanex” and “faromin” more difficult to transfer than the simpler ones, despite them all using similarly simple primitives. We can interpret this as evidence that the models are building new representations rather than linking up existing algorithmic combinations. Overall we agree with the reviewer that there is a risk that generalization of simpler algorithms works primarily through binding a new name to an existing computation, which is why we include the more invented algorithms that are unlikely to already exist under a specific name. Although, we also note that the more complicated functions potentially need more training to learn so this could just as easily be due to poor training optimization for these particular tasks, as task complexity and how it bears on generalization.

Reviewer_r9YnOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper applies the mech interp toolkit to the problem of understanding how LLMs represent syntax and semantics in code. A series of well-designed activation steering experiments establishes that function, programming language, and implementation strategy can be manipulated independently; that these dimensions are composable; and that finetuning enables transfer across languages. Lastly, the authors invent a toy language in order to study how transformers acquire these representations in pretraining.

Critical strengths
  • I really like the progression and breadth of experiments: starting from establishing the base premise that models represent programs (S3), showing that function, language, and strategy can be reused (S4), and then moving to a nuanced study of abstractions (S5). Each section feels scientifically well-motivated and builds on the last.
  • The paper is full of interesting nuanced examples showing the effects of steering interventions; e.g., Fig. 6 shows an interesting example where an intervention porting from Haskell to Java causes the model to produce functional-style code where it otherwise wouldn’t when solving the same task.
  • The paper is very clear, at the level of a high-quality technical blog post. Pedagogical examples are well-chosen. While the authors disclose AI use for drafting, the writing is mostly easily-readable; AI slop is largely confined to the appendices.
Critical weaknesses
  • While the central result is nuanced and well-argued, perhaps we shouldn’t be too surprised by the core result that LLMs trained on a lot of code learn factored representations from which syntax and semantics can be disentangled. As I’m not super familiar with this literature, I’m not in the best position to say what mech interp work has already been done to this effect.
  • S5.1 is titled “What is an abstraction?” but I didn’t feel like that question was answered after reading it.
  • At places the narrative is bit repetitive; e.g., the top of Section 3 rehashes much of the introduction’s points about cross-language reusability of graph search, the Factored World Hypothesis, etc. in the same order as the intro, but with more formal detail. Similarly, some of the notation feels a bit arbitrary/unnecessary; e.g., x=dθ∣∣σx = d_\theta \mid\mid \sigma (Eq. 4) is just a prefix/suffix concatenation and probably doesn’t need its own numbered equation.
Questions for authors
  1. To what extent is the general finding novel to this paper? What are the main works from mech interp that are closest to this one, and in what ways does this work go beyond existing findings? This is partly addressed in Section 2 but I would be curious to hear a more in-depth discussion of what is already known about syntax/semantic representation in LLMs.
  2. Maybe I missed this, but how did you choose the representative models (Fig. 2) to carry forward into the subsequent analyses?
  3. What is an abstraction?
2 comments
AuthorsOpenReview ↗

Response to Reviewer r9Yn

We thank the reviewer for their positive comments about the organization of the manuscript, the scientific quality of our examples and experiments, as well as the clarity in the pedagogical approach. We appreciate the referee's critiques as well, particularly with regard to the claims and analysis in section 5.1. We propose changing the section title to “How abstract are these program representations?”, which better represents the actual content of the section. Additionally, we think the main content paragraph is not as clear as it could have been. In the revision, we will replace the main content paragraph in section 5.1. Here is portion of the proposed draft:

  • In most cases, a transported strategy state behaves like a simple label swap in the prompt: when it changes the recipient's program, the result is often identical to the recipient's own directly prompted program for the target task. However, when the recipient and donor languages are quite different from each other, this does not necessarily hold. Figure 5 shows a case where steering Java from one strategy to another using a Haskell-derived steer results in a ``Haskell flavored" Java program. So, this steering carries more than a strategy label swap: it is also tied to some donor specific information tied to language. This reinforces our understanding of ZS(θ)≈ZS(s)Z_S(\theta) \approx Z_S(s) as only the dominant behavior. The appropriate strategy for implementing a given function is dependent on the tools available in the language, so a finer grained picture takes this into account, allowing for a correction term ZS(θ)≈ZS(s)+ZS(s,λ)Z_S(\theta) \approx Z_S(s)+Z_S(s,\lambda).

The reviewer is also correct when referring to the repetitive nature of the intro and section 3. We believe, however, that this reiteration in a more formal setting serves an important pedagogical purpose. Not all readers will find it to be as redundant, and we aim to make this work easily understood by a general interpretability audience. With regard to, for example, the display equation for what amounts to concatenation– the reviewer is absolutely right. In the revision, we will remove this equation as a separate numbered equation, and give a pass over all other similar definitions to ensure they earn their keep.

Q1:We agree with the reviewer, that perhaps it is not too surprising! As a matter of fact, that is why we picked programming as a test-bed. Where we contribute beyond showing that there is separation of syntax and semantics, is in establishing a much more specific causal result: computational representations can be re-used across languages and combined with independently estimated language representations, even when neither factor is specified. The degree of reuse and composition does not obviously follow from multilingual coding ability alone. Additionally, we believe that our strategy experiments and our controlled experiments on synthetic data further characterize these representations beyond what is already established in the literature. We also note that we plan to include an expanded conclusion about the relevance of our work in the scope of broader interpretability work, which helps clarify both what we think the principal contributions are and also their relationship to future and ongoing work.

Q2: Our representative models are those that showed the presence of the compiled state at the suffix position. This selection process is very important, because it is only in these models that we have a straightforward candidate representation to test. This is not to say that the other models don’t learn similarly reusable representations– it’s just that we have no evidence of how/where they would store these abstractions. These points are currently described in section 3.1 and briefly in the conclusion section. We will use the expanded room in the revision to make these points more prominent.

Q3: Fair point. “Abstraction” can be an rather unscientific word. For our purpose it refers to a shared concept that transcends superficial syntactic differences, and we believe that use is warranted. However, our use of the word should be understood as colloquial. We are not defining "abstraction" formally here, and its colloquial use should be made obvious. We will make sure that the revision is clear about this. The changing of section 5.1’s title (and the text within) is a step towards this; we will look through the rest of the text and make changes accordingly. As part of this, we will use the extra space in the revision to include an expanded discussion section. A sample excerpt:

"LLMs can recognize what distinct code implementations share, and separate the abstract computation from the details of its realization. They do not relearn coding concepts in various contexts as independent things, but instead reuse them in disparate situations"

We cannot show the full draft, given character limits on the response, but more of the expanded discussion is shown in the overall response.

Thanks for the detailed reply. I enjoyed reading this paper and will be glad to see my suggestions incorporated into the revised manuscript. At this stage I would be happy to see this paper accepted. I will maintain my score, but have increased my confidence from 2 --> 3 to reflect the authors' constructive rebuttal response.

Reviewer_C5PWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies code-generating LLMs. After prefill, the specification is masked from attention. Strong coding models can still write correct code from the suffix state alone, which the authors call a "compiled" state. Then the authors add difference-of-means edits to the suffix KV-cache. Edits for function, language, and strategy (recursion vs. iteration) are estimated from other contexts. They claim these edits can transport, install, and compose each factor. A toy transformer trained on synthetic "shared mechanism" data shows similar behavior. Finally, a state harvested from one example transfers across languages. Fine-tuning on one invented Python algorithm transfers to held-out languages.

Critical strengths

The question is interesting. The leave-one-context-out design is a clean causal test. Controls are included (random direction, wrong value, Pythia). The toy model has an exact Bayesian reference. Several architectures are tested, including a recurrent model. The authors are open about limits, such as the failure on Mistral and the need to choose the right intervention site.

Critical weaknesses
  1. The evidence is narrow compared with the claim. The main interventions use only 4 digit-arithmetic tasks, 10 languages, and one strategy pair. These are simple, well-known functions. There are 36–72 trials per cell and no uncertainty estimates. The three-factor composition is weak, as the paper says. The title says "LLMs", but the results cover three models selected by the screen, plus RWKV. Mistral-Small largely fails (A.11). The authors say they chose models "addressable at the suffix" and designed interventions to fit. Models and tasks were thus selected where the effect shows up.
  2. Success of steering does not show "abstract" or additive representations. The paper never tests Eq. 2 directly. There is no interaction analysis and no residual check. The geometry analysis is in the appendix, on one model. A difference-of-means vector for "digit sum vs. digit product" averaged over languages will work whenever the suffix state holds a task label plus a language label. That is not strong evidence of language-invariant abstraction. Donors and recipients share exact wording, and paraphrasing lowers success (A.16). Fig. 6 shows the strategy edit carries recipient syntax. Retention under masking also tracks architecture (linear or hybrid attention) and model recency (Fig. 2). It may reflect where information is stored, not abstraction. Pythia is a weak control because it also cannot code well.
  3. The toy model does not support the claim about where these representations come from. Contribution 3 says shared-mechanism data is enough. But a dense model with the same loss fails (0.09 vs 0.99). The windowed model succeeds mainly because its 8-token window forces the information into the blank suffix. There is no ablation where the mechanism is not shared, so sharing is never shown to be the cause. The dimensional-efficiency argument is theoretical and never measured.
  4. The generalization results are weak evidence for "reuse of abstract features". In the transplant test, only donors that survive masking are used. For some models and nonce algorithms this is few (13/90 for Qwen, 27/90 for Qwen kanex). The mean-over-donors state fails on nonce algorithms. The fine-tuning test uses 9 examples, seed 0, and one control model. Cross-language transfer is already a known behavior, and the model can simply translate Python into another language. No baseline separates this from reuse of a shared representation. The claim that the state is "neither a summary nor a copy" rests on programs differing from the example. That does not show the state is language-invariant.
Questions for authors
  1. Can you test additivity directly (interaction terms, residuals) and extend to more functions, non-arithmetic tasks, and more strategies?
  2. Can you add a toy ablation with non-shared mechanisms, and a stronger code-capable control model?
  3. Can you add a fine-tuning baseline that separates reuse from plain translation ability?
1 comment
AuthorsOpenReview ↗

Response to Reviewer C5PW

Given the response character count limit, we respond primarily to the questions only. For the listed weaknesses: the quantity of issues raised combined with the terse writing style makes it hard to interpret exactly what each weakness is referring to. We would be happy to discuss the "key weaknesses" in detail as the conversation develops, as we believe each can be rebutted or addressed.

First we'd like to quickly list some weaknesses that are already addressed in the manuscript:

  • Figure 19 already decomposes variance as “interactions + residual” across several LLMs
  • A reusable function label is a possible interpretation for some tests. But, in A.14.2, we estimate modifiers such as “square the aggregate,” “increment,” or “double” while excluding the recipient’s base function, language, and wording
  • On paraphrasing, we interpret the result positively: successful transfer survives when the recipient wording is held out from donor estimation, so identical wording is not required.
  • Pythia is not a capability-matched coding control, but our inference does not depend on its failure. We also do interventions that test specificity within the capable models, holding their coding ability fixed.

Q1: We think the existing coverage is appropriate for the causal claim: A.14.2 tests four modifiers across five base computations and four models, holding out the recipient’s function, language, and wording. More tasks would broaden the demonstrated scope, but would not materially change the interpretation. Fig. 19 already provides an interaction/residual analysis, although it does not causally test Eq. 2. The advantage to digit functions is that they are easily verifiable and exist cleanly in all languages, whereas some programming languages for instance do not have proper strings. We ran an additional experiment mixing 2 digit functions (sum and prod) with 2 list-based functions (count_distinct and count_ascents) on Gemma4-31B in the Transport setting. In addition to the setting from the paper, we additionally test the settings where we hold out the function type (digit or list) such that the steering vector is derived from a different class of function. We still get 60/72 hits here, so the intervention largely works. Specifically, we get 16/18 digit→digit, 17/18 digit→list, 13/18 list→digit, 14/18 list→list. We will add the full experiment across all modes and intervention models to the appendix and reference them in the main text. We do agree that it improves our claims of generalization and thank the reviewer for the suggestion.

Q2: We would love to add a stronger code-capable control model, but it is difficult to find one. The ideal control model is one which is a strong coder, but completely lacks multilingual generalization. We do not have such a model and would have to train one, likely by creating a very overparameterized coder.

We have run additional toy model experiments. We trained new toy models using training data in which the different functions do not appear in the context of each language, but only in one language each. This removes shared structure in the data, as requested. The model still survives masking due to the windowed attention, so we can test the same interventions. We find that the representation of language and function here are not composable: function and language can no longer be composed/transported outside of the context of a donor sequence. This is consistent with our claims that flexible reuse is connected to a particular type of structure in the training data. We will add this experiment to the appendix in the revision. Thanks for the suggestion!

Q3: We note that the reviewer’s distinction cannot be fully drawn in our setting. With reasoning off and code-only output, there is no Python text to translate from, so “silently translates in one pass” and “reuses a shared representation” are largely the same thing, unless we pin down the definition of “translation”. As the reviewer suggested, we evaluated an explicit translation baseline for vantril using the unfine-tuned Gemma-4-31B-it model. Given one of the three Python training implementations, it passes the target-language tests in 21/27 cases across nine languages. With the same name-only prompt used in the fine-tuning evaluation, it passes 0/9. The fine-tuning result shows that information learned through Python examples can subsequently be accessed through that name in held-out languages. We agree that this does not determine whether the model retrieves a Python-like implementation and translates it internally, or uses a more language-independent representation. Our main claim as pertains to SFT is that our intervention experiments suggest that SFT should lead to generalization beyond a specific language with very few examples, which we do observe. To elucidate exactly how this mechanism works, more follow-up experiments would be required. We will state this in the paper.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall Review Response

We thank the reviewers for their careful and constructive reviews. All three count the intervention experiments among the paper’s strengths, and our central claim rests on them: a state change estimated in one context produces the corresponding change in another. The edits are built without access to the recipient’s context, and they compose in a prompt that specifies neither factor. This is also the novelty: prior work shows that syntax and semantics can be separated, whereas we show that independently estimated factors can be causally reused and recombined into working programs, which does not obviously follow from multilingual coding ability. We group the reviewer's comments into two groups, which we address broadly here; details are in the individual responses.

Scope of the evidence. The concern is that models and tasks were selected where the effect appears. We see the selection differently. The masking screen identifies where a model’s task state can be addressed, not whether that state is reusable; reusability is what the interventions then test. A model that fails the screen is one we cannot address at the suffix, not one shown to lack the effect. When we intervene during generation on such a model the same reuse appears in each one (the recurrent model, whose state can live in only one place, shows it strongly). We will make sure that the rationale behind our selection is made more clear in the revision. One reviewer also mentions the breadth of our tasks. However, no finite set of tasks settles generality, so the useful question is whether reuse survives when pushed in new directions. It does: it extends to a different class of functions (we’ve implemented a new experiment prompted by reviewer C5PW to show this).

Interpretation of the states. The concern here is that successful steering does not by itself establish abstract or additive representations, and that Section 5.1 claims more about the content of the state than the experiments show. We agree with some of this. Our notion of reuse is operational, and Eq. 2 should be treated as a hypothesis whose causal predictions we test rather than a claim of exact additive geometry. We do push back on the reading that our edits might merely be swapping task labels: it does not explain why an operation estimated from other functions, languages, and wordings can be applied to a new computation. In the revision we will rename and rewrite Section 5.1 and narrow its “neither a summary nor a copy” claim, and expand the conclusion to state the limits and interpretation of our study plainly. Below are a few representative samples from our draft of this expanded discussion section:

Clarifying what we mean by abstractions:

  • "We show that function, language, and strategy support separable causal interventions through Z⁡(θ), a “compiled” state that is induced by natural language coding prompts and then is strongly attended to during generation in LLMs. In-context experiments then use this state to show that a single code implementation supplies a Z⁡(θ) that is reusable across output languages. These reusable “abstractions” support different realizations of the same computation in distinct contexts, while also allowing reuse of information acquired from one of those realizations."

Being clear about the scope of the evidence:

  • "We then propose a structure in data that would favor such reusable representations, and validate this prediction using a minimal toy task that displays the same behavioral signatures of compositionality that we observe in LLMs. We also show that accurate prediction does not guarantee an accessible, reusable suffix state, with a similarly performant toy model that uses dense attention only. It is worth noting, however, that these results do not establish definitively that the same data structure causes the analogous representations in LLMs."

Contextualizing how the work fits into the bigger picture of interpretability:

  • "Although we have studied code, little in our analysis depends on it. Programming is simply a domain that makes the shared structure easy to see and test, since one verifiable algorithm is written in many languages and one language carries many algorithms. Much of the natural language corpus that LLMs learn from in pretraining also has the same shape: a relatively small number of underlying mechanisms expressed and recombined into a vast number of distinct forms. The same pressure toward reusable, composable states should apply whenever that is true. From this perspective, understanding where these shared representations form and how they compose is key to a robust understanding of how models organize what they know."

As the feedback round closes, we thank each reviewer for their comments and the productive discussion.

We are still editing the manuscript, but we have added an updated manuscript draft that takes advantage of the extra page and already implements many changes promised. Here is a high level list of what has been implemented so far:

  • Section 5.1 has been renamed and rewritten, wording around abstractions narrowed, and claims that the representation is not a “summary” have been removed

  • The conclusion has been rewritten completely to emphasize points/confusions brought up in the review process Contribution 3 has been reworded to emphasize that a controlled setting was necessary for the toy model to display analogous composable representations, and not just the right dataset.

  • We turned two of the numbered equations into inline equations

We want to note specifically to reviewer CP5W that we have already included one of their suggested control experiments in the appendices to show that a toy model trained on a data source without the shared mechanism does not display composability of function and language. The toy model details in the appendix have been completely rewritten to be more clear and self contained.

We added an additional paragraph before discussing our interventions that makes it clear the models that survive masking are chosen for clarity in the manuscript and not because other models fail to display the same qualitative behavior when properly addressed: “Before moving on, we want to emphasize that we do not believe that this representation at the suffix position is the only mechanism that LLM's use for composable representations. Reusable and composable representations are present in other models as well. We show this in Appendix A.11. The compiled suffix representations provide a clear and consistent interface to design interventions that work across model architectures, and so we focus on them in the main text for clarity.”

Tasks from the revision that we have yet to do:

  • Include more experiments to back up our claim that the suffix is merely a useful place to address several models– not the core mechanism of abstraction. We currently rely on mistral-small as our primary evidence, but will add additional models.
  • For reviewer CP5W: early results indicate that transfer between a new (non arithmetic) class of functions works; but we are still running additional experiments and have not yet added the relevant appendix.
  • Do a thorough edit over the wording to make sure that abstraction is clearly used colloquially, and not formally. Similarly make sure that our evidence of “new computation acquisition" is framed as evidence of “task information transfer” throughout the text and appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.