- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes Latent Assembly, a framework for recovering state-dependent operations from neural activations and composing them through causal interfaces. The method fits read/write maps and operation-specific linear transformations, then uses producer return maps and consumer casts to connect independently recovered operations. Experiments on Tracr-compiled transformers validate operation discovery and composition against known programs. On pretrained Qwen, Gemma, and Llama models, the authors study SELECT and LOAD operations. Finally, they link LOAD, ADD, and COMPARE in Qwen2.5-14B to execute new two- and three-operation tasks without target-task fitting.
Critical strengths
-
The paper provides extensive causal validation, including activation interventions, matched controls, and counterfactual tests. In particular, the cross-task experiments show that changing an upstream argument can produce the corresponding downstream behavior, rather than simply degrading prediction accuracy.
-
The controlled experiments provide ground-truth validation of operation discovery and composition. The authors also distinguish executability from native mediation and document negative results, which helps establish the scope of their findings.
Critical weaknesses
-
Why not directly compose the original operations? The framework is entirely post-hoc: the operations and their semantics are predefined, and the authors collect activations from the pretrained model to fit linear transformations and interfaces. In Section 6, the program decomposition is also given, and each operation runs through a separate forward pass of the original LLM. This raises a fundamental question: why not simply execute these operations through prompts and compose their outputs directly? Such an approach requires no additional training, labeled data for interface fitting, or activation access. Yet this most natural baseline is missing. The existing results further weaken the motivation: CoT achieves 100% on both the quota and sensor tasks, while linked execution reaches 66.1% and 58% overall, respectively. Direct prompting also outperforms linked execution on rover routing (92.5% vs. 71.3%). While these are not matched comparisons with staged prompt composition, they suggest that simpler approaches may already be more effective. Without such a baseline, it is difficult to justify the additional fitting and engineering required by the proposed framework, or identify what is gained by recovering these post-hoc approximations instead of executing the known operations directly.
-
Limited evidence for general operation discovery and scalable composition. The pretrained-model experiments start from predefined operations and supervised source tasks, making the method closer to fitting known computations than discovering new operations. The strongest blind-discovery results are obtained on Tracr-compiled transformers with planted programs. Moreover, cross-task composition in Section 6 uses native LLM executions connected through fitted interfaces, rather than directly composing the recovered linear transformations. Scalability is also unclear. The longest sequential composition contains only three operations, and its accuracy drops to 58% overall, with just 6% on equality cases. The paper already reports an additional performance penalty from adding a third operation. As the operation library and program length grow, the need for further interface calibration and the accumulation of errors become substantial concerns. The experiments do not establish whether this approach remains viable for longer programs, particularly compared with direct prompt-based composition that requires no interface training.
Presentation: Several figures are too blurry to read, particularly the text and labels in the diagrams. The extensive notation and terminology also make the paper unnecessarily difficult to follow.
Questions for authors
-
Why not directly compose the operations? Given that the operations and program decomposition are already specified, what is gained by fitting neural transformations and calling conventions instead of executing the same operations through prompts and passing intermediate results as text? Can the authors provide this baseline under the same execution schedule, comparing accuracy, inference cost, and additional training requirements?
-
Operation discovery: What does the framework discover in pretrained LLMs beyond fitting neural representations and transition maps for predefined operations? What evidence shows that these recovered operations reflect the model's native computation?
-
Multi-hop composition: The longest sequential chain contains only three operations and already shows degraded performance. How does accuracy change with additional operations? Does the method require further calibration as the operation library grows, and what advantage would it retain over direct prompt-based composition for longer programs?
7 comments
Response, Part 2 (W1 & Q1)
- The latent hand-off carries a record reference instead of a name. We fit a frozen SELECT → LOAD map only on tables with unique names, where a record's name identifies it. We then test on tables where two records share a name. If the transferred state encoded only the selected name, accuracy would be at chance (0.50). It is not:
| Setting (80 duplicate-name tables) | Latent hand-off | Text: SELECT's output (name only) | Text: name + selector ("the last Grace") |
|---|---|---|---|
| Qwen, city query | 0.912 | 0.500 | 1.000 |
| Qwen, job query (map fitted on city queries only) | 0.787 | 0.475 | 1.000 |
| Qwen, numeric age query | 0.900 | 0.488 | 1.000 |
| Qwen, cities and jobs never seen in fitting | 0.800 | 0.488 | 1.000 |
| Gemma, city query | 0.963 | 0.500 | 0.975 |
The state carries a reference to the selected record that the fitting data never required it to carry. A text hand-off can of course carry the same information if the format names it (e.g. "the last Grace" reaches 1.00), but then the designer chose what to transmit. The latent hand-off shows what the model's own state transmits.
- The model's own computation runs through the interface (native mediation). §7 listed this as a limitation for pretrained models. We now test it on Qwen and Gemma with the logic of the Mediated/Bypass experiment from §4. Our discovered LOAD, ADD, and COMPARE operations write into a small subspace of the model's activations. To test whether the model itself reads numbers through that subspace, we take pairs of prompts that differ only in one place and have different correct answers. We give the model one prompt, with nothing installed, and swap in the other prompt's activation only within the subspace. All three operations correctly switch to the other prompt when mediated 89% - 99% of the time. When we instead swap only the rest of the activations, switching occurs only 0% - 0.6% of the time. This shows that the small subspace alone carries the answer, and that the pretrained models route through that subspace in its native computation. Importantly, a text chain cannot test this claim.
In short: the text chain is the right accuracy baseline, and we now report it in the revised manuscript. However, textual mediation is not a competitor. Latent Assembly asks which operations a model computes internally, what types of values they pass, and through which interfaces, and it makes each answer falsifiable by intervention. Linking is the strongest test of that account, because it only works if the recovered types and interfaces are correct. Text-level success cannot test any of this, because it never touches the internal states the account is about.
In Section 6, the program decomposition is also given, and each operation runs through a separate forward pass of the original LLM
The decomposition need not be given (new). Qwen2.5-14B now writes the program itself from the question, given only the operation signatures. Plans may not contain numbers, so the model cannot compute the answer while planning. The frozen operations then execute whatever program it wrote. It writes the correct program for 400/400 questions, compared with 98/400 when the question is removed from each prompt. The operations execute model-generated chains correctly under intervention (98.8% cases). Please find more details in our response “Response, Part 1” to Reviewer dyZh. We added this experiment and its results in §6.3 (Lines 479 - 494) of the revised manuscript.
- Why separate forward passes. This follows from an architectural limitation and not convenience. In a single forward pass, a state computed at layer l can influence later tokens only at layers above l. In Qwen2.5-14B (48 layers), LOAD's result forms around L42 while COMPARE reads its operand at L8, so their composition cannot fit in one pass. This is also why the model fails the composite in one pass (direct 0.33 on the sensor task) and why chain-of-thought, itself a multi-pass route through generated tokens, succeeds. The textual mediation baseline is multi-pass as well. For interpretability purposes, this design is the stricter test: a value that crosses between separate computations with no shared context is evidence for a genuinely reusable argument in our neural calling conventions.
CoT achieves 100% on both the quota and sensor tasks, while linked execution reaches 66.1% and 58% overall, respectively.
The longest sequential composition contains only three operations, and its accuracy drops to 58% overall, with just 6% on equality cases.
On the numbers you cite: the sensor chain's 58% rises to 81% (strict order 1.00, equality 0.42), and the quota chain’s 66% rises to 83% (strict order 1.00, equality 0.48) once we replace the underfit return map on ADD’s output.
Response, Part 3 (W2 & Q2)
Limited evidence for general operation discovery [...] closer to fitting known computations than discovering new operations. [...] cross-task composition in Section 6 uses native LLM executions connected through fitted interfaces, rather than directly composing the recovered linear transformations.
Q2: What does the framework discover in pretrained LLMs beyond fitting [...]? What evidence shows that these recovered operations reflect the model's native computation?
Thank you for requesting this clarification. We reiterate our methods for validating discovered operations and present new experiments.
As in causal abstraction (Geiger et al., 2021; 2024), where researchers propose a high-level algorithm and test by intervention whether a network implements it, our task-level operation names are hypotheses that our interventions can refute. What the framework discovers, and what could have come out otherwise, is what type each intermediate value has and which interface carries it. Three findings separate this from fitting a known computation:
1. Types are decided by intervention. Nothing specified whether SELECT returns a value, a record–field combination or a reference. In §5, installing the donor's intermediate in a recipient with a different table and field returns the recipient's field of the donor's record (1.000 Qwen, 1.000 Gemma, 0.833 Llama; donor-field carryover 0.000).
2. What we recover generalizes along axes absent from fitting (New). We add three new experiments on pretrained models to further strengthen existing generalization experiments. Each map below was fitted on one set of conditions, frozen, and tested on a condition its fitting data contained no information about:
-
A fitted LOAD operation works on fields it was never fitted on. The LOAD transition is the fitted operation itself: a linear map that predicts the model's state after retrieval. We fit one transition on queries for birth city, former job, age and hours, both categorical and numerical, then freeze it. At test time we run a query it never saw, e.g. "The current city of Wendy is", apply the frozen transition to the L34 state, and write the result into a prompt about a different person in a different table. That prompt then answers with Wendy's value for held-out fields of both types: categorical “current city” 0.75 and numerical “commute” 0.93 (0.69 and 0.73 above the strongest controls, respectively).
-
Unseen answer types and vocabulary (a reference interface). The map from the duplicate-name experiment (Response, Part 2) was fitted only on city questions over a fixed vocabulary. On the same duplicate-name tables, it also selects the right record for a numeric age query (Qwen 0.90, Gemma 0.94) and for cities and jobs never seen in fitting (0.80, 0.98). It hands over which record, and the model's own LOAD then retrieves whatever that record holds.
-
Unfitted neural calling conventions. Recall from §6 that return maps and casts are fitted per operation, never per pair. We additionally test chaining ADD into ADD, a pair never fitted or tested in the 2-operation links, the hop returns the exact value 86% of the time. (More details in “Response, Part 4”).
3. The model's own computation runs through the interface (native mediation). This was detailed in “Response, Part 2”.
In short: which operations to look for is our starting hypothesis. What they pass, how far they generalize, and whether the model's own computation uses their interfaces are empirical findings. Each could have failed, and each is now tested in pretrained models.
Response, Part 1 (W1 & Q1)
Thank you for the careful review, and for recognizing our causal validation tests. We respond to each question and concern point to point.
We ran exactly this: the same decomposition, items and prompts, with each operation's answer decoded and passed as text into the next prompt. Both routes use the same items and operation schedule. At depth 10, latent execution uses 11 prompt evaluations (about 2,950 tokens), versus 10 (about 2,560 tokens) for staged text. Only the interface maps are fitted using source-task activations, while the language-model weights remain frozen.
Text-level chaining results (all numbers reported are execution fidelity).
Text-level chaining matches or exceeds the latent hand-off on every program. We agree that our original manuscript highlighted under-supported accuracy claims for the latent hand-off. We have revised affected sections accordingly (Abstract, §1, Table 2, Table 3), and we included the textual baseline results in the main tables.
We take this as the central concern, so we provide detailed clarifications and additional experiments.
We emphasize the distinction of Latent Assembly from textual mediation. The direct purpose of the latent hand-off is modeling computation in an interpretable and testable manner, not accuracy. Our core question is whether a model's internal states contain reusable values that independently discovered operations can exchange and interact with. A text chain cannot answer that, because it routes around those internal states by construction. This follows the paper's framing:
Decodability is not use (§2). A text chain decodes each value and re-reads it. Its success shows the values are decodable, not that any internal state is consumed.
Text chaining does not test claims of discovered operations. From an interpretability perspective, Latent Assembly tests claims about the model’s internal computations. Text-level chaining is a valid way to compose operations behaviorally, but causal composition generalization (§3) requires predicted internal states, generated one from another, to be executed downstream. It is a falsification test of the claim that our fitted operations describe the model's internal computation, which is a test that a text chain cannot provide.
Same behavior, different route (§4). Our compiled Mediated and Bypass transformers behave identically, yet only one consumes the intermediate state (64/64 vs 0/64). A text chain is a bypass in this sense: its accuracy cannot reveal which route the computation took, while the latent hand-off can. We ran the native-mediation experiment on pretrained models as wells. Please see “Response (Part 2)” or §5.1 (Lines 378 - 385) for more details.
Diagnosing failures (§7). Interventions let us distinguish an incorrect sum from a failure to pass that sum to COMPARE. On the same 150 fresh Qwen sensor items, replacing only the ADD read raised equality accuracy from 8% to 42% and strict-order accuracy from 87% to 100%, bringing overall accuracy from 61% to 81%. The staged text chain solved all 150 items. Text only showed that Qwen could execute the program, while changing the read identified one source of error in the latent hand-off.
New experiments distinguishing Latent Assembly from textual mediation:
In addition to the above clarifications, we run two new experiments further showcasing the distinction between Latent Assembly and textual mediation, particularly in pretrained models. (To be continued)