acceptodds
Under review as a conference paper at ICLR 2027

When Can One Transformer Use Another Transformer's Hidden State?

Abstract

When can separately optimized transformers directly use one another's hidden states without learned alignment? We study raw interoperability: whether a frozen recipient can causally use an independently learned donor state under the identity map. In controlled retrieval tasks, independently optimized models sharing externally specified coordinate scaffolds exhibit robust cross-model transplantation, including on a held-out two-hop compositional task. Decoder-subspace analysis shows that decoder-visible state carries substantial but incomplete transferable signal, while scaffold interventions identify shared output coordinates as the strongest tested anchor; recipient-specific decoder-null structure remains consequential. When output coordinates differ, identity-map transfer nearly disappears, whereas applying the exact known coordinate correspondence restores nearly all of the lost advantage as mapped compatibility. In a complementary boundary test, independently pretrained Pythia-70M seeds show no reliable cross-seed identity-map transfer at the tested interfaces, including a site subsequently verified in a post hoc diagnostic to be causally informative within individual models. These results identify raw cross-model interoperability as a scaffold-dependent interface property: it can emerge across independently optimized models, but depends on shared coordinate conventions and should not be assumed without direct testing in independently pretrained networks.

Primary area:
interpretability and explainable AI
Keywords:
model interoperability, representation learning, activation patching, mechanistic interpretability, representation alignment, transformers

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8Hf9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors explore when states between transformers can be stitched together using the identity tranformation. They mainly address this question in the context of one and two layer transformers on toy tasks, but they include an experiment on Pythia 70B. They find that without scaffolding during training, the models are not stitchable. They find this at scale as well, ultimately concluding that the "experiments identify conditions under which the identity map becomes usable across models and show that this compatibility is scaffold-induced and coordinate-relative."

Critical strengths
  • understanding model stitching is an important precursor for making good brain-computer interfaces.
  • the authors interpret their models to better understand the raw interoperability
Critical weaknesses
  • The authors mainly consider one and two layer models, which are known to be limited computational models due to lack of layer depth (Biran et al. 2024; Yang et al. 2024; Wang et al. 2024; Grant et al. 2025; Li, Guo & Andreas 2025). It is difficult to discern what to take away from such small-scale experiments that can be applied to models at scale.
  • There is a lack of theoretical rigor. The results in the paper are almost entirely empirical. There exists a lot of theoretical work showing what types of representational invariances could exist and be exploited for representational alignment (Williams et al. 2021). Much of the work in the reviewed paper could likely be explained in the context of existing frameworks.
  • Unclear or missing methodological detail:
    • The paper states that there are "two primary controlled regimes and one compositional bridge task" pg 4 line 206. Then it introduces the "first primary regime" pg 4 line 206 without ever mentioning a "second" regime. Is the second regime the "main regime" from pg 4 line 211?
    • What is a bridge task?
  • The answer to the motivating question is not very surprising. Given past work on model stitching, it would have been a surprising result if the identity operation was highly successful without constraining the training. It is not surprising that the raw transfer only worked when the models were constrained to a specific coordinate system. And, if I understood correctly, "worked" is generous, given that the recovery of the transfers only ever reached about ~0.75 at best. This work reads like the authors tried to paint negative results in a positive light.
Questions for authors
  • The paper states that there are "two primary controlled regimes and one compositional bridge task" pg 4 line 206. Then it introduces the "first primary regime" pg 4 line 206 without ever mentioning a "second" regime. Is the second regime the "main regime" from pg 4 line 211?
  • What is a bridge task?
  • Why should a general audience have predicted that the identity transform would work?
  • What new capabilities, understanding, or efficiencies have been unlocked by the results of this paper?
1 comment
AuthorsOpenReview ↗

Response to Reviewer 8Hf9

We thank the reviewer for the detailed feedback. We agree that unconstrained identity-map transfer should not generally be expected. Our contribution is to characterize the conditions under which raw interoperability becomes possible and its limitations.

Scientific insight and theory. Our experiments distinguish functional equivalence, compatibility under a known coordinate transformation, and direct identity-map interoperability. Exact function-preserving gauge transformations alter raw transfer without changing model behavior, while applying the known coordinate correspondence restores compatibility. Scaffold interventions further show that shared output coordinates support substantially more transfer than shared input coordinates under the tested conditions (advantages 0.644 versus 0.003). These controlled findings complement existing representational-invariance theory rather than establishing a new general theorem.

Beyond the shared-decoder intuition. Shared decoder coordinates make partial compatibility plausible but do not guarantee equivalent downstream causal effects. The full donor and decoder-matched surrogate have identical immediate fixed-decoder projections, yet achieve recovery of 0.746 and 0.489, respectively, after downstream recipient computation. Thus, matching decoder-visible information preserves substantial but incomplete transfer, demonstrating that immediate decoder agreement does not fully determine downstream compatibility.

Scale and recovery. We agree that shallow models limit generalization and that recovery around 0.75 is partial rather than complete. Nevertheless, recovery of 0.760 versus a matched corrupted-donor control of 0.076 demonstrates substantial causal transfer under the controlled shared-scaffold condition. The eight-block, approximately 1.6M-parameter experiment retains fixed content-addressing routing, while independently pretrained Pythia-70M models show no reliable cross-seed identity-map transfer at the tested interfaces. We therefore do not claim general LLM interoperability or practical efficiency gains.

Terminology. Associative retrieval is the second primary controlled regime. The "bridge task" refers to held-out two-hop compositional retrieval: given Ea→EbE_a \rightarrow E_b and Eb→EcE_b \rightarrow E_c, the model predicts EcE_c from EaE_a. Complete two-hop compositions are withheld during training, while their constituent atomic relations remain represented.

Motivation and significance. We do not assume identity-map transfer should have been predicted in general. Our question is which controlled training conditions make direct transfer possible despite independent optimization. The findings identify coordinate constraints supporting transfer, distinguish decoder-visible agreement from downstream causal interchangeability, and demonstrate that functional equivalence does not guarantee identity-map compatibility. These results provide a causal characterization of alignment-free interfaces, not a claim of immediate new capabilities or efficiencies.

Reviewer_MbtbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

Through controlled experiments, the authors find that hidden states can be directly transferred between independently trained models under specific task settings and architectures with a shared output decoder. However, this phenomenon is not observed when the output decoder is not shared or in the experiments on pretrained Pythia models.

Critical strengths
  1. The research question is interesting and closely related to representation alignment and mechanistic interpretability. While many existing works study whether hidden representations are similar across models, representational similarity does not necessarily imply that these representations are functionally interchangeable.

  2. The experiments are carefully designed, with relatively strong causal evidence supporting the main findings.

Critical weaknesses
  1. Highly artificial experimental settings and limited potential impact. Most of the positive results are obtained on very simple models and synthetic tasks. Although the authors conduct a larger-scale experiment, the 1.6M-parameter model still relies on a fixed, non-learned content-addressing router. More importantly, the main findings do not hold in less constrained settings, which substantially limits their potential impact.

  2. Limited novelty and scientific insight. I find most of the experimental findings fairly predictable and do not think they provide much new insight. The models are trained on the same task, share the same output decoder, and predict the same targets. As a result, the gradients with respect to their final hidden representations are constrained to the same subspace defined by the output decoder. It is therefore quite natural to expect some degree of representation compatibility near the output layer. I am not convinced that the experimental results reveal much beyond this straightforward expectation.

  3. Poor presentation of experimental results. Almost all the figures and tables supporting the paper's main findings are placed in the appendix. This makes the paper unnecessarily difficult to read and follow, as readers have to constantly refer to the appendix to understand and evaluate the experimental results.

Questions for authors

See weakness

1 comment
AuthorsOpenReview ↗

Response to Reviewer Mbtb

We thank the reviewer for recognizing the importance of the research question and the strength of our causal experimental design.

Novelty and scientific insight. We agree that a shared decoder makes partial representational compatibility plausible. However, this intuition alone does not establish which coordinate constraints support transfer or whether decoder-visible agreement guarantees downstream interchangeability.

Our experiments provide three complementary findings. First, independently changing input coordinates while retaining shared output coordinates preserves substantial transfer (advantage 0.644 versus 0.684 when both are shared), whereas retaining shared input coordinates but changing output coordinates reduces the advantage to 0.003. Second, the full donor and decoder-matched surrogate have identical immediate fixed-decoder projections, yet achieve different recovery after downstream recipient computation (0.746 versus 0.489). This shows that decoder-visible agreement does not fully determine downstream causal effects. Third, exact function-preserving gauge transformations alter identity-map compatibility without changing model behavior, while applying the known coordinate correspondence restores transfer.

Together, these experiments distinguish the roles of training-coordinate scaffolds, decoder-visible information, and whole-state causal interchangeability. These controlled causal distinctions constitute our scientific contribution beyond the shared-decoder intuition.

Scope and impact. We agree that the controlled settings limit generalization. The eight-block, approximately 1.6M-parameter experiment retains fixed content-addressing routing, while independently pretrained Pythia-70M models show no reliable cross-seed identity-map transfer at the tested interfaces. Our findings therefore do not establish general LLM interoperability or practical efficiency gains. Instead, they identify experimentally supported conditions for alignment-free hidden-state exchange and its limitations, with shared output-coordinate conventions suggesting a potential direction for future interface research.

Presentation. We appreciate the concern regarding the placement of experimental results. Given the main-text page limit, we prioritized the central quantitative findings and their interpretation in the main text, while providing detailed supporting figures, tables, and diagnostics in the appendix. This organization allows us to present the principal results while retaining comprehensive experimental documentation.

Reviewer_DsTdOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents very surprising new results. In particular, the authors find that one model can directly use another model's hidden states without any additional operations. This finding, while limited and preliminary, suggests that "raw interoperability" may have arisen in independently trained toy-sized transformer models, and this may point to some deeper reason explaining the unprecedented success of contemporary LLMs. However, the authors find no comparable evidence of interoperability in independently pretrained Pythia-70M models—models that more closely resemble real-world LLMs. I nevertheless consider this limitation acceptable for an initial investigation.

Critical strengths
  • The finding is very interesting in my opinion. The experimental design is simple but gets the job done.
  • The authors provide a good survey that lays out all the relevant context. Although I am not particularly familiar with the subject matter, I was able to grasp the full context quickly.
  • This paper is generally well-written and easy to follow.
Critical weaknesses
  • A major limitation of this work is that all of their findings are based on toy models trained in rather limited, controlled settings. Similar findings were not observed in models with more realistic sizes and capabilities, and therefore the conclusion may not scale to real models. Nevertheless, I would consider this to be an acceptable limitation to be addressed in the future.
  • It would be nice to see some investigation into whether vestiges of raw interoperability can be found in real models beyond these toy models, or at least a discussion of promising directions.
Questions for authors

See weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer DsTd

We thank the reviewer for the encouraging assessment and agree that extending these findings beyond controlled settings is an important challenge.

Our experiments progress from small transformers enabling mechanistic isolation to an eight-block, approximately 1.6M-parameter stress test with fixed content-addressing routing, and finally to independently pretrained Pythia-70M models, where reliable cross-seed identity-map transfer is not observed at the tested interfaces. These findings establish raw interoperability under specific controlled conditions while identifying important limitations on generalization.

Two promising directions are investigating whether shared output-coordinate conventions support transfer in larger models with learned routing and examining whether decoder-visible subspaces exhibit partial compatibility in pretrained models even when whole-state identity transfer fails. Our controlled decoder-subspace results motivate the latter: the decoder-matched surrogate achieves substantial but incomplete recovery relative to the full donor (0.489 versus 0.746).

We view the present work as providing a controlled causal foundation for these investigations, rather than demonstrating general LLM interoperability or immediate efficiency gains.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.