acceptodds
Under review as a conference paper at ICLR 2027

Latent Assembly: Causal Discovery and Cross-Task Linking of Executable Operations in Language Models

Abstract

Can computations learned by language models be decompiled as executable operations and assembled into new programs? We introduce a causal framework that recovers typed program states, discovers state-dependent operations, and neural interfaces through which one operation’s output becomes another’s input. The framework identifies operations by their causal transformations rather than their physical locations and tests reuse through predicted intermediate states, causal execution, and recompilation. In controlled transformers, our framework discovers ground-truth operation classes and composition laws starting from unlabeled neural activations. In pretrained Qwen, Gemma, and Llama models, we discover SELECT and LOAD operations connected by a reusable record-reference interface and validate their composition and recompilation. Most importantly, we show that three distinct operations discovered and qualified independently from separate tasks using our framework can be linked to execute semantically distinct tasks with no target-task fitting. We discover the frozen neural calling conventions through which the operations communicate. The three-operation chain achieves 84% accuracy, far exceeding baseline controls and Qwen 14B's direct zero-shot performance. Furthermore, interventions on upstream arguments produce the downstream behavior implied by the altered program. These results establish a concrete form of neural programmability: independently discovered computations can exchange arguments and execute together. They open a path toward making computations acquired during training accessible as components that can be inspected, tested, and deliberately recombined.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic Interpretability, Causal Abstraction, Language Models, Compositional Generalization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_iXTMOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Latent Assembly, a framework for recovering state-dependent operations from neural activations and composing them through causal interfaces. The method fits read/write maps and operation-specific linear transformations, then uses producer return maps and consumer casts to connect independently recovered operations. Experiments on Tracr-compiled transformers validate operation discovery and composition against known programs. On pretrained Qwen, Gemma, and Llama models, the authors study SELECT and LOAD operations. Finally, they link LOAD, ADD, and COMPARE in Qwen2.5-14B to execute new two- and three-operation tasks without target-task fitting.

Critical strengths
  1. The paper provides extensive causal validation, including activation interventions, matched controls, and counterfactual tests. In particular, the cross-task experiments show that changing an upstream argument can produce the corresponding downstream behavior, rather than simply degrading prediction accuracy.

  2. The controlled experiments provide ground-truth validation of operation discovery and composition. The authors also distinguish executability from native mediation and document negative results, which helps establish the scope of their findings.

Critical weaknesses
  1. Why not directly compose the original operations? The framework is entirely post-hoc: the operations and their semantics are predefined, and the authors collect activations from the pretrained model to fit linear transformations and interfaces. In Section 6, the program decomposition is also given, and each operation runs through a separate forward pass of the original LLM. This raises a fundamental question: why not simply execute these operations through prompts and compose their outputs directly? Such an approach requires no additional training, labeled data for interface fitting, or activation access. Yet this most natural baseline is missing. The existing results further weaken the motivation: CoT achieves 100% on both the quota and sensor tasks, while linked execution reaches 66.1% and 58% overall, respectively. Direct prompting also outperforms linked execution on rover routing (92.5% vs. 71.3%). While these are not matched comparisons with staged prompt composition, they suggest that simpler approaches may already be more effective. Without such a baseline, it is difficult to justify the additional fitting and engineering required by the proposed framework, or identify what is gained by recovering these post-hoc approximations instead of executing the known operations directly.

  2. Limited evidence for general operation discovery and scalable composition. The pretrained-model experiments start from predefined operations and supervised source tasks, making the method closer to fitting known computations than discovering new operations. The strongest blind-discovery results are obtained on Tracr-compiled transformers with planted programs. Moreover, cross-task composition in Section 6 uses native LLM executions connected through fitted interfaces, rather than directly composing the recovered linear transformations. Scalability is also unclear. The longest sequential composition contains only three operations, and its accuracy drops to 58% overall, with just 6% on equality cases. The paper already reports an additional performance penalty from adding a third operation. As the operation library and program length grow, the need for further interface calibration and the accumulation of errors become substantial concerns. The experiments do not establish whether this approach remains viable for longer programs, particularly compared with direct prompt-based composition that requires no interface training.

Presentation: Several figures are too blurry to read, particularly the text and labels in the diagrams. The extensive notation and terminology also make the paper unnecessarily difficult to follow.

Questions for authors
  1. Why not directly compose the operations? Given that the operations and program decomposition are already specified, what is gained by fitting neural transformations and calling conventions instead of executing the same operations through prompts and passing intermediate results as text? Can the authors provide this baseline under the same execution schedule, comparing accuracy, inference cost, and additional training requirements?

  2. Operation discovery: What does the framework discover in pretrained LLMs beyond fitting neural representations and transition maps for predefined operations? What evidence shows that these recovered operations reflect the model's native computation?

  3. Multi-hop composition: The longest sequential chain contains only three operations and already shows degraded performance. How does accuracy change with additional operations? Does the method require further calibration as the operation library grows, and what advantage would it retain over direct prompt-based composition for longer programs?

7 comments
AuthorsOpenReview ↗

Response, Part 1 (W1 & Q1)

Thank you for the careful review, and for recognizing our causal validation tests. We respond to each question and concern point to point.

Why not simply execute these operations through prompts and compose their outputs directly? [...] Can the authors provide this baseline under the same execution schedule, comparing accuracy, inference cost, and additional training requirements?

We ran exactly this: the same decomposition, items and prompts, with each operation's answer decoded and passed as text into the next prompt. Both routes use the same items and operation schedule. At depth 10, latent execution uses 11 prompt evaluations (about 2,950 tokens), versus 10 (about 2,560 tokens) for staged text. Only the interface maps are fitted using source-task activations, while the language-model weights remain frozen.

Text-level chaining results (all numbers reported are execution fidelity).

Programnumber of itemsDirectLatent hand-offText-level chain
LOAD → ADD → COMPARE, Qwen-2.5-14B1500.330.811.00
LOAD → ADD (rover routing), Qwen-2.5-14B800.930.710.95
ADD → COMPARE (production quota), Qwen-2.5-14B1800.580.830.97
LOAD → COMPARE (catalog task), Qwen-2.5-14B830.960.801.00
Model-decomposed programs (new experiment, model itself generates the operation decomposition), Qwen-2.5-14B4000.460.991.00
LOAD → ADD → COMPARE, Gemma-4-12B1500.320.810.81

Text-level chaining matches or exceeds the latent hand-off on every program. We agree that our original manuscript highlighted under-supported accuracy claims for the latent hand-off. We have revised affected sections accordingly (Abstract, §1, Table 2, Table 3), and we included the textual baseline results in the main tables.

what is gained by fitting neural transformations and calling conventions instead of executing the same operations through prompts and passing intermediate results as text?

We take this as the central concern, so we provide detailed clarifications and additional experiments.

We emphasize the distinction of Latent Assembly from textual mediation. The direct purpose of the latent hand-off is modeling computation in an interpretable and testable manner, not accuracy. Our core question is whether a model's internal states contain reusable values that independently discovered operations can exchange and interact with. A text chain cannot answer that, because it routes around those internal states by construction. This follows the paper's framing:

  • Decodability is not use (§2). A text chain decodes each value and re-reads it. Its success shows the values are decodable, not that any internal state is consumed.

  • Text chaining does not test claims of discovered operations. From an interpretability perspective, Latent Assembly tests claims about the model’s internal computations. Text-level chaining is a valid way to compose operations behaviorally, but causal composition generalization (§3) requires predicted internal states, generated one from another, to be executed downstream. It is a falsification test of the claim that our fitted operations describe the model's internal computation, which is a test that a text chain cannot provide.

  • Same behavior, different route (§4). Our compiled Mediated and Bypass transformers behave identically, yet only one consumes the intermediate state (64/64 vs 0/64). A text chain is a bypass in this sense: its accuracy cannot reveal which route the computation took, while the latent hand-off can. We ran the native-mediation experiment on pretrained models as wells. Please see “Response (Part 2)” or §5.1 (Lines 378 - 385) for more details.

  • Diagnosing failures (§7). Interventions let us distinguish an incorrect sum from a failure to pass that sum to COMPARE. On the same 150 fresh Qwen sensor items, replacing only the ADD read raised equality accuracy from 8% to 42% and strict-order accuracy from 87% to 100%, bringing overall accuracy from 61% to 81%. The staged text chain solved all 150 items. Text only showed that Qwen could execute the program, while changing the read identified one source of error in the latent hand-off.

New experiments distinguishing Latent Assembly from textual mediation:

In addition to the above clarifications, we run two new experiments further showcasing the distinction between Latent Assembly and textual mediation, particularly in pretrained models. (To be continued)

AuthorsOpenReview ↗

Response, Part 2 (W1 & Q1)

  • The latent hand-off carries a record reference instead of a name. We fit a frozen SELECT → LOAD map only on tables with unique names, where a record's name identifies it. We then test on tables where two records share a name. If the transferred state encoded only the selected name, accuracy would be at chance (0.50). It is not:
Setting (80 duplicate-name tables)Latent hand-offText: SELECT's output (name only)Text: name + selector ("the last Grace")
Qwen, city query0.9120.5001.000
Qwen, job query (map fitted on city queries only)0.7870.4751.000
Qwen, numeric age query0.9000.4881.000
Qwen, cities and jobs never seen in fitting0.8000.4881.000
Gemma, city query0.9630.5000.975

The state carries a reference to the selected record that the fitting data never required it to carry. A text hand-off can of course carry the same information if the format names it (e.g. "the last Grace" reaches 1.00), but then the designer chose what to transmit. The latent hand-off shows what the model's own state transmits.

  • The model's own computation runs through the interface (native mediation). §7 listed this as a limitation for pretrained models. We now test it on Qwen and Gemma with the logic of the Mediated/Bypass experiment from §4. Our discovered LOAD, ADD, and COMPARE operations write into a small subspace of the model's activations. To test whether the model itself reads numbers through that subspace, we take pairs of prompts that differ only in one place and have different correct answers. We give the model one prompt, with nothing installed, and swap in the other prompt's activation only within the subspace. All three operations correctly switch to the other prompt when mediated 89% - 99% of the time. When we instead swap only the rest of the activations, switching occurs only 0% - 0.6% of the time. This shows that the small subspace alone carries the answer, and that the pretrained models route through that subspace in its native computation. Importantly, a text chain cannot test this claim.

In short: the text chain is the right accuracy baseline, and we now report it in the revised manuscript. However, textual mediation is not a competitor. Latent Assembly asks which operations a model computes internally, what types of values they pass, and through which interfaces, and it makes each answer falsifiable by intervention. Linking is the strongest test of that account, because it only works if the recovered types and interfaces are correct. Text-level success cannot test any of this, because it never touches the internal states the account is about.

In Section 6, the program decomposition is also given, and each operation runs through a separate forward pass of the original LLM

The decomposition need not be given (new). Qwen2.5-14B now writes the program itself from the question, given only the operation signatures. Plans may not contain numbers, so the model cannot compute the answer while planning. The frozen operations then execute whatever program it wrote. It writes the correct program for 400/400 questions, compared with 98/400 when the question is removed from each prompt. The operations execute model-generated chains correctly under intervention (98.8% cases). Please find more details in our response “Response, Part 1” to Reviewer dyZh. We added this experiment and its results in §6.3 (Lines 479 - 494) of the revised manuscript.

  • Why separate forward passes. This follows from an architectural limitation and not convenience. In a single forward pass, a state computed at layer l can influence later tokens only at layers above l. In Qwen2.5-14B (48 layers), LOAD's result forms around L42 while COMPARE reads its operand at L8, so their composition cannot fit in one pass. This is also why the model fails the composite in one pass (direct 0.33 on the sensor task) and why chain-of-thought, itself a multi-pass route through generated tokens, succeeds. The textual mediation baseline is multi-pass as well. For interpretability purposes, this design is the stricter test: a value that crosses between separate computations with no shared context is evidence for a genuinely reusable argument in our neural calling conventions.

CoT achieves 100% on both the quota and sensor tasks, while linked execution reaches 66.1% and 58% overall, respectively.

The longest sequential composition contains only three operations, and its accuracy drops to 58% overall, with just 6% on equality cases.

On the numbers you cite: the sensor chain's 58% rises to 81% (strict order 1.00, equality 0.42), and the quota chain’s 66% rises to 83% (strict order 1.00, equality 0.48) once we replace the underfit return map on ADD’s output.

AuthorsOpenReview ↗

Response, Part 3 (W2 & Q2)

Limited evidence for general operation discovery [...] closer to fitting known computations than discovering new operations. [...] cross-task composition in Section 6 uses native LLM executions connected through fitted interfaces, rather than directly composing the recovered linear transformations.

Q2: What does the framework discover in pretrained LLMs beyond fitting [...]? What evidence shows that these recovered operations reflect the model's native computation?

Thank you for requesting this clarification. We reiterate our methods for validating discovered operations and present new experiments.

As in causal abstraction (Geiger et al., 2021; 2024), where researchers propose a high-level algorithm and test by intervention whether a network implements it, our task-level operation names are hypotheses that our interventions can refute. What the framework discovers, and what could have come out otherwise, is what type each intermediate value has and which interface carries it. Three findings separate this from fitting a known computation:

1. Types are decided by intervention. Nothing specified whether SELECT returns a value, a record–field combination or a reference. In §5, installing the donor's intermediate in a recipient with a different table and field returns the recipient's field of the donor's record (1.000 Qwen, 1.000 Gemma, 0.833 Llama; donor-field carryover 0.000).

2. What we recover generalizes along axes absent from fitting (New). We add three new experiments on pretrained models to further strengthen existing generalization experiments. Each map below was fitted on one set of conditions, frozen, and tested on a condition its fitting data contained no information about:

  • A fitted LOAD operation works on fields it was never fitted on. The LOAD transition is the fitted operation itself: a linear map that predicts the model's state after retrieval. We fit one transition on queries for birth city, former job, age and hours, both categorical and numerical, then freeze it. At test time we run a query it never saw, e.g. "The current city of Wendy is", apply the frozen transition to the L34 state, and write the result into a prompt about a different person in a different table. That prompt then answers with Wendy's value for held-out fields of both types: categorical “current city” 0.75 and numerical “commute” 0.93 (0.69 and 0.73 above the strongest controls, respectively).

  • Unseen answer types and vocabulary (a reference interface). The map from the duplicate-name experiment (Response, Part 2) was fitted only on city questions over a fixed vocabulary. On the same duplicate-name tables, it also selects the right record for a numeric age query (Qwen 0.90, Gemma 0.94) and for cities and jobs never seen in fitting (0.80, 0.98). It hands over which record, and the model's own LOAD then retrieves whatever that record holds.

  • Unfitted neural calling conventions. Recall from §6 that return maps and casts are fitted per operation, never per pair. We additionally test chaining ADD into ADD, a pair never fitted or tested in the 2-operation links, the hop returns the exact value 86% of the time. (More details in “Response, Part 4”).

3. The model's own computation runs through the interface (native mediation). This was detailed in “Response, Part 2”.

In short: which operations to look for is our starting hypothesis. What they pass, how far they generalize, and whether the model's own computation uses their interfaces are empirical findings. Each could have failed, and each is now tested in pretrained models.

Reviewer_dyZhOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether learned computations by LLMs can be decompiled and re-assembled into new executable programs like a decompiler + linker. The authors propose a causal framework that identifies operations using the state transformation rather than their locations, which are validated against Tracr-compiled transformers. Specifically they recover SELECT -> LOAD for table lookup in Qwen, Gemma, Llama, and then link LOAD, ADD and COMPARE are discovered separately on different tasks through frozen "neural calling conventions" to solve unseen multi-step tasks (49% -> 84% over the direct answer on a 3-step chain). Unlike the prior work (Hidden APIs, Ma+ 2026) which identifies reusable causal interfaces without chain operations, the main novelty here is the linking of independently discovered operations. The SELECT -> LOAD finding however seems to overlap with the known binding ID/positional pointer mechanisms (Feng & Steinhardt 2024, Prakash et al. 2025), which is not discussed in the paper. I think it's ambitious but interesting direction to treat learned computation as recombinable components. The controlled experiments are carefully designed as well: the bypass network gives a good look on whether an injected state is actually used or merely executable.

However, the abstract in its current wording overclaims that the model's own operations "execute" new programs, while in reality a manually crafted frontend reads the target prompt and decides which record to load etc, then finally each operation is run as a separate forward pass, suggesting that the program is not re-assembled by the model. Since numbers written directly in the prompt already work as well as the produced ones, it is unclear what the latent hand-off adds over just decoding each answer and putting it into the next prompt. Such text-level chain is the natural baseline here, rather than the direct answer which is close to the no-op level (CoT solves it perfectly). The appendix also shows LOAD does not transfer across field types and the said conventions do not transfer across layers, so the operations seems to be fits to specific formats rather than reusable components.

Critical strengths
  • An ambitious direction of turning learned computation into inspectable and recombinable components, potentially impactful.
  • The validation on compiled transformers with known answers and a behavior-matched bypass network is well thought out, and preregistering each test with the failed runs reported is quite transparent.
Critical weaknesses
  • The cross-task linking seems to be done manually, which can lower the novelty.
  • It's not clear if the latent hand-off is necessary without comparing against text-level chain baseline.
  • Narrow (3 ops on arguably toy tasks) and format-specific operations, which weakens the "neural programmability" claim. Also hard to follow, with many of key caveats only in the appendix.
Questions for authors
  • Have you tested chaining at text-level, i.e., decoding each operation's answer and write it as a number in the next prompt? Showing the benefit of latent-ness needs other non-latent baselines.
  • Can the model come up with the decomposition by itself, or is the frontend manually designed per task?
6 comments
AuthorsOpenReview ↗

Response, Part 1

Thank you for highlighting the potential impact of extracting and recombining learned computation from language models, and for recognizing our validation experiments on compiled transformers. Your questions and concerns prompted new experiments. We provide an overview of the results first:

  1. The manual decomposition of the linked program is not necessary. (Response, Part 1)
  2. Text-level chaining exceeds or matches our latent chains in every comparison. However, our framework exposes a shared internal interface that the model itself consumes and interacts with, something a text-level chain (where the model simply re-reads a number) cannot reveal or test. We conduct additional experiments to highlight this distinction. (Response, Part 2)

In addition, we test native mediation on Qwen-2.5-14B and Gemma-4-12B as on compiled transformers, showing that the model’s computation consumes the transplanted interfaces. (Response, Part 2)

The cross-task linking seems to be done manually, which can lower the novelty.

Can the model come up with the decomposition by itself, or is the frontend manually designed per task?

In the original manuscript: yes, we manually specify which operations are linked in what order to execute new tasks. We add a new experiment to test whether Qwen2.5-14B can come up with the decomposition by itself. Setup: We revisit the 3-operation sensor calibration task, but now use four different subtasks, each requiring a different program over the LOAD, ADD, and COMPARE operations:

ShapeThe question asksRequired program
S1Is station a above the threshold?COMPARE(LOAD(a), THRESHOLD)
S2Is a, after its correction, above the threshold?COMPARE(ADD(LOAD(a), CORRECTION), THRESHOLD)
S3What is a’s reading added with b’s?ADD(LOAD(a), LOAD(b))
S4Is a, after its correction, higher than b?COMPARE(ADD(LOAD(a), CORRECTION), LOAD(b))
  • The model writes the program. It sees only the available operation signatures, and we provide no example decompositions. It outputs the program for each question. We decode greedily, take one sample, and do no retries or repair. Test questions use phrasings and station names that never appear in the examples. A plan may name only operations, stations, CORRECTION, and THRESHOLD. A plan containing intermediate results or the final answer is rejected.
  • The frozen operations execute it. One generic executor runs whatever program the model wrote using the same three fitted operations as in the original manuscript.
  • Counterfactual check. With the plan held fixed, we edit one reading. If the plan depends on that reading, the answer should flip. If the plan ignores it, the answer should not change. Results (95% bootstrap CIs): | Endpoint | Result | Comparison | |---|---|---| | Model writes the correct program | 400/400 | Same prompt with question removed: 98/400 | | Edit a reading the plan depends on → answer follows | 0.988 [0.978, 0.997] (n = 600) | | | Edit a reading the plan ignores → answer unchanged | 0.995 [0.988, 1.00] (n = 400) | |

The model therefore successfully chooses the operations and their chains itself depending on different questions, and the discovered operations execute chains correctly under intervention. The counterfactual rows show the answer is not reached by some other route. It changes when and only when an input the model's program depends on changes. We added this experiment and its results in §6.3 (Lines 479 - 494) of the revised manuscript.

AuthorsOpenReview ↗

Response, Part 2

It's not clear if the latent hand-off is necessary without comparing against text-level chain baseline.

Have you tested chaining at text-level, i.e., decoding each operation's answer and write it as a number in the next prompt? Showing the benefit of latent-ness needs other non-latent baselines.

Thank you for this suggestion. We ran exactly the baseline you describe: the same decomposition, items and prompts, with each operation's answer decoded and written as a number into the next prompt.

1. Text-level chaining results (all numbers reported are execution fidelity).

Programnumber of itemsDirectLatent hand-offText-level chain
LOAD → ADD → COMPARE, Qwen-2.5-14B1500.330.811.00
LOAD → ADD (rover routing), Qwen-2.5-14B800.930.710.95
ADD → COMPARE (production quota), Qwen-2.5-14B1800.580.830.97
LOAD → COMPARE (catalog task), Qwen-2.5-14B830.960.801.00
Model-decomposed programs (new experiment, see “Response, Part 1”), Qwen-2.5-14B4000.460.991.00
LOAD → ADD → COMPARE, Gemma-4-12B1500.320.810.81

Text-level chaining matches or exceeds the latent hand-off on every program. We agree that our original manuscript highlighted under-supported accuracy claims for the latent hand-off. We have revised affected sections accordingly (Abstract, §1, Table 2, Table 3), and we included the textual baseline results in the main tables.

2. What the latent hand-off is for.

Nonetheless, we want to emphasize the distinction of Latent Assembly from textual mediation. The direct purpose of the latent hand-off is modeling computation in an interpretable and testable manner, not accuracy. Our core question is whether a model's internal states contain reusable values that independently discovered operations can exchange and interact with. A text chain cannot answer that, because it routes around those internal states by construction. This follows the paper's framing:

  • Decodability is not use (§2). A text chain decodes each value and re-reads it. Its success shows the values are decodable, not that any internal state is consumed.

  • Text chaining does not test claims of discovered operations. From an interpretability perspective, Latent Assembly tests claims about the model’s internal computations. Text-level chaining is a valid way to compose operations behaviorally, but causal composition generalization (§3) requires predicted internal states, generated one from another, to be executed downstream. It is a falsification test of the claim that our fitted operations describe the model's internal computation, which is a test that a text chain cannot provide.

  • Same behavior, different route (§4). Our compiled Mediated and Bypass transformers behave identically, yet only one consumes the intermediate state (64/64 vs 0/64). A text chain is a bypass in this sense: its accuracy cannot reveal which route the computation took, while the latent hand-off can.

  • Diagnosing failures (§7). Interventions let us distinguish an incorrect sum from a failure to pass that sum to COMPARE. On the same 150 fresh Qwen sensor items, replacing only the ADD read raised equality accuracy from 8% to 42% and strict-order accuracy from 87% to 100%, bringing overall accuracy from 61% to 81%. The staged text chain solved all 150 items. Text only showed that Qwen could execute the program, while changing the read identified one source of error in the latent hand-off.

3. New experiments distinguishing Latent Assembly from textual mediation.

In addition to the above clarifications, we run two new experiments further showcasing the distinction between Latent Assembly and textual mediation, particularly in pretrained models.

  • The latent hand-off carries a record reference instead of a name. We fit a frozen SELECT → LOAD map only on tables with unique names, where a record's name identifies it. We then test on tables where two records share a name. If the transferred state encoded only the selected name, accuracy would be at chance (0.50). It is not:
Setting (80 duplicate-name tables)Latent hand-offText: SELECT's output (name only)Text: name + selector ("the last Grace")
Qwen, city query0.9120.5001.000
Qwen, job query (map fitted on city queries only)0.7870.4751.000
Qwen, numeric age query0.9000.4881.000
Qwen, cities and jobs never seen in fitting0.8000.4881.000
Gemma, city query0.9630.5000.975

The state carries a reference to the selected record that the fitting data never required it to carry. A text hand-off can of course carry the same information if the format names it (e.g. "the last Grace" reaches 1.00), but then the designer chose what to transmit. The latent hand-off shows what the model's own state transmits.

AuthorsOpenReview ↗

Response, Part 3

Response (Part 3):

Have you tested chaining at text-level, i.e., decoding each operation's answer and write it as a number in the next prompt? Showing the benefit of latent-ness needs other non-latent baselines. (Continued)

  • The model's own computation runs through the interface (native mediation). §7 listed this as a limitation for pretrained models. We now test it on Qwen and Gemma with the logic of the Mediated/Bypass experiment from §4. Our discovered LOAD, ADD, and COMPARE operations write into a small subspace of the model's activations. To test whether the model itself reads numbers through that subspace, we take pairs of prompts that differ only in one place and have different correct answers. We give the model one prompt, with nothing installed, and swap in the other prompt's activation only within the subspace. All three operations correctly switch to the other prompt when mediated 89% - 99% of the time. When we instead swap only the rest of the activations, switching occurs only 0% - 0.6% of the time. This shows that the small subspace alone carries the answer, and that the pretrained models route through that subspace in its native computation. Importantly, a text chain cannot test this claim.

In short: the text chain is the right accuracy baseline, and we now report it in the revised manuscript. However, textual mediation is not a competitor. Latent Assembly asks which operations a model computes internally, what types of values they pass, and through which interfaces, and it makes each answer falsifiable by intervention. Linking is the strongest test of that account, because it only works if the recovered types and interfaces are correct. Text-level success cannot test any of this, because it never touches the internal states the account is about.

Narrow (3 ops on arguably toy tasks) and format-specific operations, which weakens the "neural programmability" claim. Also hard to follow, with many of key caveats only in the appendix.

Scope. We agree that the tasks are synthetic by design. Every claim needs ground-truth intermediate values, counterfactual items and leakage audits, which naturalistic data does not readily provide. The "concrete form of neural programmability" in our abstract means only that independently discovered operations can exchange arguments through frozen interfaces on these tasks. We have revised that sentence so it does not read as a general claim (Lines 33 - 37). Changing the demonstrations or templates reduced the frozen ADD read’s exact-sum recovery from 95.5% to 45.0% and 23.5%, respectively, even though Qwen still computed every sum correctly. Thus the record reference transfers across fields, but the numerical read remains sensitive to prompt format.

Readability. We moved the comparisons most relevant to interpreting the linking results into the main text. Tables 2 and 3 now report staged text alongside latent execution, and Table 3 separates strict-order, equality, and overall accuracy. We also state explicitly what the model and the runtime each do: Qwen chooses the operations and their connections, while the runtime places the operands and invokes each operation separately (§6.3, lines 474–488).

The SELECT -> LOAD finding however seems to overlap with the known binding ID/positional pointer mechanisms (Feng & Steinhardt 2024, Prakash et al. 2025), which is not discussed in the paper.

Thank you for pointing out this connection. We now discuss binding IDs and positional pointers in §2 (lines 127–131). Either could implement the reference we observe. Our contribution is to test whether SELECT passes LOAD a reusable argument specifying which record to read, while the recipient determines which field to retrieve. In §5, transplanting the state from a donor queried about Alice’s job into a recipient queried about Bob’s city makes the recipient return Alice’s city from its own table. LOAD also consumes a reference predicted by the fitted SELECT operation. Finally, an interface fitted only on unique-name city queries distinguishes two records with the same name on 91.2% of Qwen cases and 96.3% of Gemma cases, versus 50% for name-only text. This rules out a name-only handoff and remains consistent with a positional pointer.

Thank you for your time and valuable feedback! We hope these new results strengthen the validity of our operation discovery and linking framework, and provide additional evidence of its purpose beyond textual mediation. If our response addresses your concerns, we greatly appreciate it if you may consider increasing your rating of our work.

Reviewer_cyq4OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a framework for identifying state-dependent neural operations through typed read/write interfaces and testing their execution, composition, and reuse. Controlled Tracr experiments recover operation classes and composition laws and distinguish executable interventions from native mediation. Structured retrieval experiments in Qwen, Gemma, and Llama qualify SELECT and LOAD-related interfaces. In Qwen2.5-14B, independently fitted producer return maps and consumer casts link retrieval, addition, and comparison across separate source tasks without fitting the operations or interfaces on target-task examples. A staged sensor-calibration chain attains 84% accuracy on strict-order comparisons, 6% on equality, and 58% overall; chain-of-thought reaches 100%. These results primarily establish intervention-based interoperability of learned interfaces within the tested task and representation domains.

Critical strengths
  1. Recipient-local, cross-field reference interventions distinguish a reference from a transported answer, while predicted-intermediate composition tests require downstream consumption of generated states. The controlled MEDIATED/BYPASS experiment explicitly demonstrates that executability and native mediation are different properties. Together, these controls support a concrete causal interface claim rather than merely a correlation between hidden states and labels.

2.The producer returns and consumer casts are fitted separately on their own source tasks and frozen before linking; raw state transplantation fails on the tested pairwise edges. Successful links, upstream argument interventions, and the three-operation chain therefore provide useful evidence that compatible interfaces can support staged composition. Reporting equality failures, unsuccessful historical protocols, CoT performance, and heterogeneous cross-model outcomes also makes the limits of that evidence assessable.

Critical weaknesses
  1. Reusable operation discovery is not sufficiently distinguished from specialized transition fitting. Appendix C.6 shows that the blind selector×field arrangement is already recovered by a surface embedding baseline; the alternate-surface test was not adjudicated. More consequentially, Appendix C.8 finds zero LOAD transfer across categorical/numerical types and on some held-out categorical fields despite a successful pooled fit. Appendix D therefore qualifies a new numerical LOAD specialization. The cross-task runtime also relies on an external frontend, and, for example, the quota experiment executes operations on native source-form recipients (Appendix D.6). This is valid evidence for staged interface interoperability, but new target semantics alone do not show that the recovered operation transfers into the target task’s native computation. The discussion acknowledges the absence of a pretrained-model native-mediation guarantee. Taken together, these limitations leave the central decompilation/reusability interpretation under-supported, rather than merely in need of clearer wording.

  2. Reliability and the contribution of latent linking remain insufficiently isolated. Table 3’s 84% applies to 100 strict-order cases; equality is 6%, overall accuracy is 58%, and CoT is 100%. The pretrained cross-task library is tested only in Qwen2.5-14B, and Appendix C.5 reports a numerical SELECT handoff falling from 0.9917 with a fixed demonstration context to 0.5263 with recipients’ own demonstrations. Raw/random controls establish that the fitted interfaces matter, but there is no staged text-mediated comparison with the same external decomposition and number of calls to isolate what latent linking contributes beyond orchestration. Table 2 also places catalog linked accuracy on 53 strict-order cases beside direct accuracy on all 83 cases (Appendix D.7), which is not a matched comparison. Beating CoT is not necessary for a mechanistic contribution; the concern is that the limited transfer and mismatched or incomplete comparisons leave the strength of the proposed reusable abstraction uncertain.

Questions for authors
  1. Mechanistic clarification: For each cross-task result, which recipient prompts are in source-task form, and which require operation/interface transfer into the target task’s native neural computation? Please distinguish learned transition maps, native continuations, and frontend transformations in the execution trace. In light of Appendix C.8, what evidence supports reuse beyond the separately fitted field/domain specializations?

  2. Generalization request: With the interfaces frozen, what survives changes in few-shot context, prompt rendering, and numerical support? Please also stratify strict-order accuracy by numerical margin and whether the leading digits coincide. The existing equality and context failures make this a meaningful test of the claimed reusable argument representation, rather than simply an additional benchmark.

  3. Evaluation clarification: Please report catalog linked/direct performance on the same subsets and provide a staged text-mediated baseline using the same decomposition and number of calls as the neural linker. For the 99% wrong-station counterfactual result, please also report the subset on which the intervention changes the final relation and requires the intervening ADD step. These comparisons would clarify both the specific value of latent linking and the strength of downstream causal propagation.

8 comments
AuthorsOpenReview ↗

Response (Part 1)

Thank you for your thoughtful and detailed assessment of our work. We appreciate that you find our experiments qualifying discovered operations and neural calling conventions supportive of a concrete causal interface. We address your questions, present new experiments as per your requests, and highlight added clarifications below. We have included these points in the revised manuscript as well (modified locations listed below).

Reusable operation discovery is not sufficiently distinguished from specialized transition fitting.

As in causal abstraction (Geiger et al., 2021; 2024), where researchers propose a high-level algorithm and test by intervention whether a network implements it, our task-level operation names are hypotheses that our interventions can refute. What the framework discovers, and what could have come out otherwise, is what type each intermediate value has and which interface carries it. Three findings separate this from fitting a known computation:

1. Types are decided by intervention. Nothing specified whether SELECT returns a value, a record–field combination or a reference. In §5, installing the donor's intermediate in a recipient with a different table and field returns the recipient's field of the donor's record (1.000 Qwen, 1.000 Gemma, 0.833 Llama; donor-field carryover 0.000).

2. What we recover generalizes along axes absent from fitting (New). We add new experiments on pretrained models to further strengthen existing generalization experiments. Each map below was fitted on one set of conditions, frozen, and tested on a condition its fitting data contained no information about:

  • A fitted LOAD operation works on fields it was never fitted on. The LOAD transition is the fitted operation itself: a linear map that predicts the model's state after retrieval. We fit one transition on queries for birth city, former job, age and hours, both categorical and numerical, then freeze it. At test time we run a query it never saw, e.g. "The current city of Wendy is", apply the frozen transition to the L34 state, and write the result into a prompt about a different person in a different table. That prompt then answers with Wendy's value for held-out fields of both types: categorical “current city” 0.75 and numerical “commute” 0.93 (0.20 strongest control).

  • Unseen answer types and vocabulary (a reference interface). The map from the duplicate-name experiment (Response, Part 1) was fitted only on city questions over a fixed vocabulary. On the same duplicate-name tables, it also selects the right record for a numeric age query (Qwen 0.90, Gemma 0.94) and for cities and jobs never seen in fitting (0.80, 0.98). It hands over which record, and the model's own LOAD then retrieves whatever that record holds.

3. The model's own computation uses the recovered interface (new). This is the pretrained counterpart of §4's Mediated/Bypass test, run on Qwen and Gemma. Our discovered LOAD, ADD, and COMPARE operations write into a small subspace of the model's activations. To test whether the model itself reads numbers through that subspace, we take pairs of prompts that differ only in one place and have different correct answers. We give the model one prompt, with nothing installed, and swap in the other prompt's activation only within the subspace. All three operations correctly switch to the other prompt when mediated 89% - 99% of the time. When we instead swap only the rest of the activations, switching occurs only 0% - 0.6% of the time. This shows that the small subspace alone carries the answer, and that the pretrained models route through that subspace in its native computation.

AuthorsOpenReview ↗

Response (Part 2)

Appendix C.6 shows that the blind selector×field arrangement is already recovered by a surface embedding baseline

We agree, and C.6 says so. But a surface baseline recovering the same 2×4 arrangement is expected: the selector and the field are written in the prompt. The blind stage establishes something the surface cannot: the model's internal states keep this organization from the input through L34, where LOAD operates, on tables never used in discovery, and it predicts held-out cells. Which factor yields a reference and which parameterizes retrieval is then established by intervention, where a surface explanation predicts no effect.

In the donor–recipient test, the recipient's prompt is never changed. We replace only one internal state, and the recipient then reads the donor's record from its own table (1.000 in Qwen and Gemma, 0.833 in Llama). If that state merely encoded the donor's prompt, the recipient would repeat the donor's answer. It never does (0.000). The same holds when no donor prompt exists at all: in CCG, the installed state is predicted by the fitted SELECT operation, and LOAD still executes it (ρ = 0.74–0.87).

Appendix C.8 finds zero LOAD transfer across categorical/numerical types and on some held-out categorical fields despite a successful pooled fit.

Appendix D therefore qualifies a new numerical LOAD specialization.

1. These zeros are what C.8 was designed to detect. A pooled map that executes all four fields (1.000) does not show that the model runs one retrieval rule. It could equally be switching between type-specific rules. This is the question our shared-versus-separate test asks (§3), so C.8 preregistered holdouts to answer it. The answer is that retrieval is typed, but the reference it reads from is not:

  • A record reference read from a categorical query is consumed by a numerical query, and vice versa: 1.000 at every layer from L4 to L24, with carried field 0.000.
  • A LOAD transition fitted on one type does not execute the other (0.000).

So SELECT produces a type-general reference, and LOAD is implemented separately per type. Conventional instruction sets are built the same way: integer and floating-point loads are separate instructions that share one addressing mode. A method that reported a single universal LOAD here would have been wrong about the model.

Nonetheless, we ran a new experiment to show that we can indeed fit a generalized LOAD operation:

2. New results: within a type, LOAD transfers to fields never queried in fitting. We built a biography task with seven fields per record and fitted LOAD on four of them: birth city, former job, age and weekly hours. We then tested three fields never queried in fitting: current city, current job and commute time.

For each test, we take the donor's state at L34, apply the frozen map, install the result at the recipient's L42 final token, and score whether the recipient returns the donor's value for that field.

held-out fieldfit on all four fieldsfit on the two categorical fields onlystrongest control
current city0.750.78≤ 0.06
current job0.600.890.00
commute (numerical)0.93n/a0.20

The model's own accuracy on these fields is 0.98–1.00. In a longer 11-field record format, both maps again pass both held-out categorical fields (0.50–0.65, controls ≤ 0.07).

3. Pooling across types is what hides the transfer. Fitting on the categorical rows alone raises current-job transfer from 0.60 to 0.89 (+0.29 [0.20, 0.38]) with the same categorical rows. The pooled map's misses were near misses to the token that starts a numerical answer. In the 11-field format the gain replicates in direction, +0.13 [0.04, 0.22]. A "successful pooled fit" therefore blends two typed transitions, and that blending is what degrades held-out categorical fields. This is further evidence for point 1.

4. Why C.8's categorical holdouts were zero. Each categorical fold in C.8 fitted one categorical field and tested the other: CITY → JOB, JOB → CITY. The held-out field's answers (job titles vs city names) never appeared in fitting. In the new task, the held-out fields are new fields whose answers come from vocabularies the map has seen. Both results are consistent with one description: LOAD's transition generalizes across fields of its type, over the answer vocabularies it was fitted on.

In short: C.8's zeros reflect the model's organization, a shared reference with type-specific retrieval, and our holdouts are what revealed it. Within a type, the new experiments show that the fitted LOAD executes fields never queried in fitting, with controls near zero. We will add these results to C.8.

AuthorsOpenReview ↗

Response (Part 3)

The cross-task runtime also relies on an external frontend

1. The frontend places operands without computing anything. It never reads the answer, an intermediate value or the final relation. This is enforced in code by a provenance guard (sensor chain: 150 placements, 0 violations). With the frontend held fixed, the answer comes from the latent hand-off. Every arm below uses the same frontend, the same prompts and the same placements. Only what is passed between operations changes:

arm (sensor chain, 150 items)accuracy
linked chain0.81 (strict order 1.00)
nothing installed0.37
producer state passed without the calling convention0.37
random state of matched norm0.37
wrong station's value → answer implied by that value0.99

The frontend alone leaves the model at its base answer (0.37). When the passed value is changed, the answer follows that value (0.99) rather than anything the frontend placed.

2. New Experiment. Per request from reviewer dyZh, we ran a new experiment where the model comes up with the decomposition by itself, without relying on the frontend. Qwen writes the correct program for 400/400 questions, compared with 98/400 when the question is removed from each prompt. The operations execute model-generated chains correctly under intervention (98.8% cases). Please find more details in our response “Response (Part 1)” to Reviewer dyZh. We also added this experiment and its results in §6.3 (Lines 479 - 494) of the revised manuscript.

3. Staging is required, not a convenience. ADD's result becomes available near L36, while COMPARE's input port closes by L12 (Appendix D). One forward pass cannot hand one to the other, so operations run as separate calls, as chain-of-thought also does.

New target semantics alone do not show that the recovered operation transfers into the target task’s native computation

The discussion acknowledges the absence of a pretrained-model native-mediation guarantee.

We add two experiments that test exactly this, in the target task's own prompt rather than a source-form recipient:

First, the target task's native prompt consumes the linked value. The recipient is the unmodified catalog prompt. A price retrieved on the separate LOAD source task, passed through LOAD's frozen return map and COMPARE's frozen cast, is installed at one product's price tokens. The model's answer follows it: all four answer cells of a family are correct in 0.823 of 96 families, matching an edit of the price in the text itself (0.812). Installing the other product's real return instead gives 0.000.

Second, we now complete the native-mediation experiment on pretrained models, and show that the model's own computation uses the recovered interface, as reported in “Response (Part 1).”

Taken together, these limitations leave the central decompilation/reusability interpretation under-supported, rather than merely in need of clearer wording.

We agree these were the right tests to ask for. We hope that our clarifications and new experiments above address each of these limitations. What our results support is the interpretation we propose: the model's computation decomposes into typed operations, whose interfaces are recovered once per operation, reused across tasks without refitting, and used by the model's own computation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.