acceptodds
Under review as a conference paper at ICLR 2027

MemHub: Zero-Shot Latent Memory Transfer Across Heterogeneous LLMs

Abstract

Direct latent communication between language models currently takes two forms: pairwise projections, whose training cost grows quadratically with the number of models, and shared representation spaces, which admit new members solely through existing ones and have been shown only within one model family. We introduce MEMHUB, a memory hub in which each frozen model independently trains one Writer–Reader adapter pair against a small frozen anchor that fixes a shared latent coordinate system. Members never observe one another’s parameters, activations, or training, yet the persistent artifacts they write are mutually readable zero-shot. Across nine members from five heterogeneous families (0.5B–8B), MEMHUB recovers a median 96.1% of pair-specific readout gains over all 72 foreign directions, retains 97% of self-transfer under dataset shift, and onboards two held-out families with one adapter pair each and no incumbent retraining. Two hubs retrained from scratch read the first hub’s artifacts, and each other’s, at parity over 432 cross-hub directions: the interface belongs to the anchor, not to any training run. Zero-shot reads outperform a pair-trained C2C channel overall. The artifact is a fixed 64 KB object read at prefill cost: 21–33× faster to first token than a live text handoff and higher quality than a text channel trained at matched budget. Controlled factorials isolate the mechanism: the geometric constraint alone fixes the shared coordinate system, and anchoring two of 32 slots aligns the rest.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Latent memory, Zero-shot transfer

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ko3pOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MemHub, a shared latent-memory interface that enables heterogeneous LLMs to exchange persistent memory artifacts without pairwise training. Each model independently trains a Writer–Reader adapter pair against a fixed public anchor. This allows zero-shot cross-model reading and independent onboarding of new models. The authors validate the approach across multiple model families, independently rebuilt hubs, distribution shifts, and downstream tasks.

Critical strengths
  • Independent cross-model interoperability. The fixed-anchor design enables independently trained heterogeneous LLMs to exchange reusable latent memories without pairwise alignment or retraining existing members.

  • Strong experimental and mechanistic validation. The paper's evaluation covers complete directed pairs among nine models, distribution shifts, independent training runs, and alternative anchor configurations. The geometric and functional anchoring ablations and the representation-alignment analysis provide useful insight into why interoperability emerges.

Critical weaknesses
  • Practical advantages over persistent text memory remain insufficiently established. The reported 21–33× latency improvement primarily compares against live text generation, whereas the stored-text results show substantially smaller advantages and occasionally lower first-token latency for text.

  • The generality of reusable memory transfer is not yet fully demonstrated. The strongest results come from partitioned multi-hop QA, while the non-QA experiments require additional Reader training and exhibit larger cross-model performance gaps, particularly on classification and summarization.

Questions for authors

I'm intended to increase my score if below questions are well justified.

  1. In Table 1, the Empty baseline already shows a fairly large improvement over Half. How much of the reported 96.1% recovery actually comes from the transferred memory itself?

  2. The speedup over live text communication is impressive, but the advantage over re-reading stored text seems much smaller. Would MemHub still have a clear advantage over storing a short text summary in advance?

  3. Most of the main experiments focus on multi-hop QA with split contexts. How well would MemHub work when the same stored memory needs to answer different questions or preserve more detailed information?

Reviewer_UxuaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the O(M2)O(M^2) scaling bottleneck in cross-model latent communication across heterogeneous language models. Instead of training pairwise projectors or jointly training a shared bus, the authors propose MemHub, where each frozen backbone independently trains a single write/read adapter pair via a self-loop reconstruction objective anchored to a frozen public coordinate frame (using BGE-M3 and per-slot whitening). This design reduces onboarding complexity to O(M)O(M) and enables zero-shot cross-family latent handoffs (including incremental M+1M+1 onboarding and multimodal extensions) without touching existing adapters.

Critical strengths

Practical Problem & Scalable Formulation: Breaking the O(M2)O(M^2) pairwise training barrier via an O(M)O(M) independently trained hub protocol is well-motivated and practically valuable for modular multi-agent systems. The incremental onboarding property—allowing new models or modalities (VLMs) to join without retraining incumbents—is a very appealing feature.

Simple and Effective Anchor Mechanism: Explaining the cross-family coordinate gauge problem (invertible linear/orthogonal ambiguity) and resolving it via a frozen public reference frame (ZrefZ_{\text{ref}}) combined with per-slot whitening is clean, lightweight (~5 GPU-hours per model), and geometrically intuitive.

Thorough Empirical & Ablation Studies: The evaluation across four distinct model families (Qwen3, Llama-3.2, Phi-4, Gemma-2), onboarding experiments (SmolLM2, InternVL3), and progressive anchor ablations (Table 3) convincingly demonstrate that the anchor constraints are necessary and effective for zero-shot interoperability.

Critical weaknesses

Marginal Gains Over Simple Text-to-Text Handoff on Standard Tasks: On standard textual reasoning benchmarks (Table 1), the average performance of MemHub (60.6%) is only slightly higher than standard Text-to-Text communication (T2T-think at 60.0%). While MemHub clearly shines in vision-to-text handoffs (Table 2, where text serialization loses spatial details) and multi-read latency amortization, its accuracy advantage for pure text handoffs is relatively modest.

Fixed Bottleneck Capacity (K=32K=32): The shared memory interface uses a fixed small number of slots (K=32K=32). While sufficient for short reasoning traces (e.g., GSM8K, MATH-500), this fixed capacity might become an information bottleneck for long-context reasoning or document-level handoffs.

Dependence on External Anchor Quality: Because the entire shared coordinate system relies on a frozen external embedder (BGE-M3), the semantic upper bound of the hub may be constrained by the anchor model's capabilities, especially in specialized domains (e.g., complex code or formal math) where general text embedders might struggle.

Questions for authors

see weakness.

Reviewer_ViyoOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a standardized latent-memory format that independently trained LLMs can write to and read from, without pairwise alignment or joint training.

Each model is frozen and receives two trainable adapters. The Writer compresses hidden states into a shared representation. The Reader converts these vectors into a virtual KV-cache prefix in its own model's native architecture. The receiving model then answers a query using this prefix. It does not need the Writer's vocabulary, KV-cache layout, or model architecture.

The important ingredient is a frozen public anchor consisting of:

  • A text embedding model and an orthogonal projection that define canonical targets for selected memory slots.
  • A frozen judge LLM and Reader that encourage all Writers to produce functionally readable memories.

The most compelling result is is the demonstration that independently trained models from different random seeds can exchange latent memories without further alignment.

I would view the paper primarily as a contribution to representation standardization and decentralized model interoperability, with persistent latent memory as an important application rather than a fully established replacement for conventional memory systems.

Critical strengths
  1. Strength over joint shared-space learning methods

For joint shared-space learning, a new member requires access to some existing participating models or their adapters and associated computation. For MemHub, it requires only a fixed public reference.

This creates three benefits:

  1. Organizational independence: Different organizations can implement the same memory interface without exposing their proprietary models, activations, or training processes to one another.
  2. Asynchronous ecosystem growth: A model joining in 2028 could theoretically read artifacts written by a model trained in 2026, provided both conform to the same anchor and interface version. No coordination with the original Writer is required.
  3. Independent reproducibility: Different developers can train adapters against the same standard and exchange outputs without agreeing on the same training run or model pool.

In summary, MemHub's novelty over joint shared-space learning is: Independently trained models can implement a standardized, persistent latent-memory interface without access to incumbent models, at only a modest loss in task performance relative to joint cross-model training.

  1. The mechanism is thoroughly investigated.

The authors derive the basis-invariance issue, demonstrate failure without anchoring, and isolate geometric and functional constraints. The Procrustes experiment is particularly informative: a simple alignment fitted on 50 paired unlabeled texts largely restores interoperability between unanchored models. This supports the claim that representational incompatibility is a central obstacle.

  1. Extensive experiments with a diverse set of models.

The experimental coverage and controls are substantial. Nine primary models, two held-out families, multiple seeds, paired statistical tests, C2C and joint-space baselines, and studies of quantization, positional transport, and artifact composition make this much stronger than a demonstration on one or two model pairs.

Critical weaknesses
  1. Narrowness of the evaluation task set

The distribution-shift evaluation covers only two additional multi-hop QA datasets. While the results convincingly demonstrate that cross-model compatibility persists across these distributions, it remains unclear whether the same property holds under more substantial shifts in task structure or domain.

I would like to see:

  1. an offline persistent-text-memory baseline. Generate a high-quality, query-independent textual memory once, store it, and compare repeated reads against MemHub under controlled quality, latency, and storage budgets. This directly tests the paper's proposed use case.
  2. a genuinely task-agnostic memory benchmark. Train Writers on a general corpus, then evaluate multiple unforeseen downstream tasks without retraining the Writers, measuring not just whether foreign memories remain readable but how much useful source information is retained. Different retrieval, reasoning, and summarization demands would make this especially informative.
  3. a stronger shared-space comparison involving independent onboarding. Compare the total compute, operational dependencies, and final quality of MemHub against joint shared-space training with a frozen-incumbent onboarding protocol. This would quantify the actual price and benefit of independence, rather than conflating it with the benefits of escaping pairwise training.
  1. Future-proofness of the fixed public anchors

What if a bigger, better model comes out in the future, and the existing fixed public anchors cannot capture all of the semantics in the embeddings of the bigger model? A future model needs to encode semantic distinctions that the anchor's training objective never required or could not evaluate. The fixed anchor cannot guarantee that different models will encode those distinctions compatibly.

The authors have not discussed this limitation in the paper. However, the paper already contains hints of this limitation:

The paper's experiments show that Writers of very different sizes produce artifacts with surprisingly similar downstream utility. For example, on HotpotQA, a 0.5B Writer produces memories that are approximately as useful to the Readers as those from an 8B Writer. The authors interpret this as evidence of Writer invariance.

However, I see another possible interpretation: The evaluation task may be too simple to expose meaningful differences between Writers' semantic capabilities.

The benchmark generally requires transferring a missing supporting fact in a multi-hop QA problem. A small model may be perfectly capable of encoding that fact, even though a much larger model possesses substantially richer semantic representations.

Consider instead tasks involving:

  • Long mathematical proofs with subtle dependencies.
  • Code execution state and complex program semantics.
  • Scientific reasoning involving uncertain hypotheses.
  • Multi-step agent plans and internal uncertainty.
  • Concepts that were absent from the anchor's training distribution.

These might expose a much larger gap between Writers. The paper does not establish that the same degree of Writer invariance would persist in such settings.

Questions for authors
  1. Can there be a future-proof latent semantic standard, that solves weakness #2?
  2. How significant are the pair-specific gain and dataset-shift retention metrics? It seems like the pair-specific readout gain provides good evidence that MemHub eliminates most of the benefit of pair-specific Reader training. But is it informative about the intrinsic quality or completeness of the shared latent representation? Also, the dataset-shift retention metric seems to show that Writer independence survives the evaluated distribution shifts. But does it show that MemHub's absolute task performance is robust under distribution shift?

Comments on the paper

1 comment

Dear Reviewers, This is a gentle reminder that the discussion deadlines are approaching. Please make sure to engage with the authors and address any outstanding questions or concerns before the discussion period closes. Best regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.