acceptodds
Under review as a conference paper at ICLR 2027

Latent Bridge: Asking for Help Without Words

Abstract

Text-based model collaboration requires a large model to generate intermediate responses before a smaller model can use its guidance, adding large-model decoding cost to inference. We introduce POND (Prefill-Only, No-Decode), a method that uses a frozen large sender for prefill and a small receiver for all autoregressive decoding. A trainable bridge maps the sender’s hidden states to additional attention keys and values, allowing the receiver to use these representations throughout generation. POND also supports periodic review, in which the sender processes the receiver’s partial response and updates its guidance through additional prefill passes while preserving the receiver’s existing KV cache. Across reasoning and coding benchmarks, POND improves receiver performance with senders from different model families. In particular, GLM-5.3 provides gains comparable to those of Qwen3.8-Max when paired with a Qwen receiver, supporting the effectiveness of representation transfer between independently trained model families. POND further demonstrates its ability to generalize across domains. When trained on math and coding tasks, it transfers directly to general knowledge tasks without additional domain-specific training. These results demonstrate that large frontier model prefill representations can improve small-model generation without requiring large-model autoregressive decoding. Code and inference examples can be found at https://latent-1117.pages.dev.

Primary area:
foundation or frontier models, including LLMs
Keywords:
latent communication, model collaboration, hidden representations, representation alignment, adaptive inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_i8KgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes POND, a framework in which a frozen large model (Sender) only prefills the input and a small model (Receiver) performs all autoregressive decoding. A trainable bridge maps the Sender’s hidden states into additional attention keys/values of the Receiver, and an periodic-review mechanism lets the Sender re-read the Receiver’s partial output and refresh this guidance. With a Qwen3.5-4B Receiver and three frontier-scale Senders, the authors report improvements over the Receiver alone on math, code and knowledge benchmarks.

Critical strengths
  1. The problem setting is clear. It turns the asymmetry between parallel prefilling and sequential decoding into a division of labor across different models.
  2. The performance improvements are consistent, with all three senders improving the receiver on all nine benchmarks.
  3. To some extent, the results suggest that the accuracy gain does not come at the cost of efficiency: even without review, the outputs are shorter and the latency is lower compared with the receiver alone.
Critical weaknesses
  1. The paper lacks comparisons with other baseline methods, such as C2C[1], which is mentioned in the paper.
  2. Some experimental results mentioned in the appendix are missing from the paper, such as constant/shuffled message, matched-budget text advising, and bridge-only experiments.
  3. Only one receiver model (Qwen3.5-4B) is evaluated, with no experiments on models of other sizes or from other model families.
  4. The paper lacks a comparison with majority voting of the receiver alone under an equivalent resource budget.
  5. Earlier work, such as LLM-to-SLM[2], has already proposed a similar idea (a frozen large model encodes the prompt once, and a small model does all the decoding), but the paper neither cites nor discusses it.

[1] Tianyu Fu et al. Cache-to-Cache: Direct Semantic Communication Between Large Language Models. arXiv: 2510.03215, 2025.

[2] Benjamin Bergner et al. Think Big, Generate Quick: LLM-to-SLM for Fast Autoregressive Decoding. arXiv: 2402.16844, 2024.

Questions for authors

N/A

2 comments
AuthorsOpenReview ↗

Response to Reviewer i8Kg (1/2)

We thank the reviewer for recognizing the clarity of the problem setting, the consistency of the improvements across all senders and benchmarks, and that the accuracy gains do not come at the cost of efficiency. We address each weakness below.


W1: Comparison with C2C.

We agree that C2C is a direct and important baseline. We trained and evaluated C2C and POND with the same sender (Qwen3.8-Max), receiver, training data, and training budget:

MethodMath/code avg (%)Knowledge avg (%)Latency (s/run, math/code / knowledge)
Receiver alone63.9484.77132.4 / 70.8
C2C66.786.1118.0 / 65.5
POND67.1486.44114.0 / 62.0
POND + Review68.0287.18119.7 / 66.2

Single-transfer POND is more accurate and faster than C2C on both suites, and review adds a further 0.88 / 0.74 pp. Both methods transfer representations without intermediate text; POND differs in three ways:

  • Guidance interface. C2C maps the sharer's layer-wise KV caches to receiver layers and fuses them residually into the corresponding cache entries. POND compresses the sender's final-layer hidden states into one shared message, which per-layer adapters project into a separate latent KV memory. The receiver keeps its own text attention pathway and combines it with latent attention through learned gates. C2C also uses residual connections and gating, so the difference lies in where guidance is stored and how it is read.
  • Response-dependent incremental updates. C2C's core algorithm fuses prompt caches once, before decoding. POND adds periodic review, which updates the latent memory during generation: at each review boundary, the sender prefills only the newly generated segment from its cached prefix, the new entries are appended to the latent memory, and the receiver's existing decoding states are preserved. With Qwen3.8-Max, periodic review (POND + Review vs. single-transfer POND) improves GPQA-Diamond from 79.29% to 80.81% and LiveCodeBench from 47.1% to 48.2%.
  • Evaluation regime. C2C's primary experiments use short-answer multiple-choice tasks. We pair a 4B receiver with frontier-scale senders, Qwen3.8-Max (2.4T), GLM-5.3 (753B), and DeepSeek-V4-Flash (284B), on long-form math, code, and knowledge tasks. A single prefill raises the math/code average from 63.94% to 67.14%, 66.74%, and 66.2%, respectively, while reducing decoded output volume by 14.8%, 12.4%, and 10.2%.

W2: Experiments described in the appendix are missing from the main paper.

Thank you for this careful observation. These controls were completed (Appendix F) but omitted from the main text for space. We will add them to the revision:

ControlMath/code avg (%)Knowledge avg (%)
Receiver-only SFT (Table 3)64.2083.45
Constant message64.183.7
Shuffled message64.584.0
Bridge-only (frozen receiver)65.485.5
Matched-budget text advising66.585.9
POND67.1486.44
  • Constant and shuffled messages fall back to the SFT level, so the receiver uses instance-specific sender information, not extra parameters or a generic bias.
  • Bridge-only retains part of the gain without adapting the receiver.
  • Matched-budget text advising, the text-based collaboration POND is meant to replace, is below POND on both suites.

W3: Only one receiver model is evaluated.

Our study varied the sender (three frontier models, two of them cross-family with different tokenizers) while fixing the receiver, in order to isolate the effect of sender representations. To address this concern, we add Gemma 3 4B IT as a second receiver family (Qwen3.8-Max sender; SFT and POND use the same math/code data and training budget):

Gemma receiver configurationAIME Acc. (%)LiveCodeBench v6 Pass@1 (%)GPQA-Diamond Acc. (%)
Original Gemma 3 4B IT16.6718.8630.8
Gemma + matched SFT25.3322.5032.24
Gemma + POND28.6725.6435.46
Gemma + POND + Review30.0027.1237.21
POND − matched SFT (pp)+3.33+3.14+3.22
+Review − POND (pp)+1.33+1.48+1.75
AuthorsOpenReview ↗

Response to Reviewer i8Kg (2/2)

W4: Comparison with majority voting of the receiver alone under an equivalent budget.

We agree this is a useful reference point.

  • Latency budget. Single-transfer POND is already faster than one receiver-alone sample on every benchmark (Tables 1–2), so a latency-matched voting baseline gets at most one sample unless samples run in parallel.
  • Compute budget. We compare against receiver maj@k with k = 3, chosen to match POND's total compute (sender prefill plus receiver decoding):
MethodAIMEOlymMath-HardGPQA-DiamondMMLU-Pro
Receiver maj@3 (compute-matched)83.3351.078.379.2
POND (single sample)85.3353.079.2980.93
POND maj@387.3354.080.881.8

A single POND sample outperforms compute-matched voting on all four tasks. Voting is also orthogonal to POND and adds further gains on top of it. Voting over programs is not directly defined without execution-based selection, so we report math and multiple-choice tasks only.


W5: Relation to LLM-to-SLM (Bergner et al., 2024).

Thank you for pointing out this work; we will cite and discuss it. LLM-to-SLM shares POND's high-level motivation: a frozen large model encodes the prompt in parallel and a small model performs all decoding. POND differs in four respects:

  1. Decoder-only frontier senders. LLM-to-SLM mainly uses encoder–decoder LLMs (T5 encoders up to ~11B). Its authors report that decoder-only features worked poorly in their framework and leave them to future work. POND uses the final-layer states of decoder-only frontier models and improves the receiver consistently.
  2. Injection mechanism. LLM-to-SLM adds the projected LLM representation to the SLM's prompt embeddings, once, at the input layer; sequence alignment then requires the SLM to adopt the LLM's tokenizer and embedding matrix and to train a new output head. POND injects a separate gated KV memory at every full-attention receiver layer (Sec. 3.3–3.4). The receiver keeps its own tokenizer, vocabulary, and text KV cache, and senders with different tokenizers are handled by character-offset alignment (Eq. 9, Appendix B).
  3. Periodic review. LLM-to-SLM calls the LLM once on the prompt; its authors list this as a limitation and suggest dynamic invocation as future work. POND's review realizes this direction: the sender re-reads the partial response through delta-only prefill, without rewriting the receiver's cache (Sec. 3.5–3.6).
  4. Task regime. LLM-to-SLM evaluates translation, summarization, and instruction following. POND targets long-form reasoning (AIME, OlymMath-Hard, LiveCodeBench, GPQA-Diamond) with outputs of thousands of tokens, where review matters most.

POND additionally shows cross-domain transfer: trained on math/code only, it raises the knowledge average from 84.77% to 86.44%, whereas matched receiver-only SFT lowers it to 83.45% (see our response to Reviewer UMZc, W1).

Reviewer_UMZcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper trains a bridge to map a stronger LLM's latent state to a representation that can be used by a smaller LLM, using the next token prediction loss of the smaller LLM on a target output as the objective, the smaller LLM is also trained together with the bridge. The proposed pipeline is trained on math and code data, and showed improvement in math/code domain as well as knowledge tasks.

Critical strengths

The paper is technically rigorous, with reasonable experiments and metrics to demonstrate the effectiveness of the method.

Critical weaknesses
  • In table 3, training the receiver on math/code causes performance to drop on knowledge tasks, this is expected; but why training with receiver and the proposed bridge together improve on knowledge tasks? Are they also trained only on math/code? Because the mechanism of the proposed method isn't specifically targeting reducing forgetting in training, or improving knowledge task performance, if it's only trained on math/code, it's surprising why it also improves on knowledge tasks. More analysis here would be helpful.
  • For the left figure in Figure 2, it shows that the improvement on the receiver is relatively similar on different datasets, regardless of how big the gap is between the receiver and sender. The paper discusses this very briefly, but I think this is worth more investigation; where does the improvement come from? It would be ideal to have a detailed version of Table 3 to show per dataset ablated improvement.
  • There should be ablation on review frequency, how that affects accuracy and speed.
Questions for authors
  • What does being able to compress the latent state of the sender into a lower dimension (d_z << d_S) tell us? ablation/insights on how to choose the latent dimension?
  • Is the compress then project bridge architecture design purely for saving computation? Estimation on how much performance this sacrifices?
  • The tokens reported in Table 1 are inconsistent with the 4096 output token limit mentioned in the appendix?
4 comments
AuthorsOpenReview ↗

Response to Reviewer UMZc (1/2)

We thank the reviewer for finding the paper clear and technically rigorous, and for the specific questions about where the gains come from. We address each point below.


W1: Why does joint receiver–bridge training improve knowledge tasks? Is it also trained only on math/code?

Yes. Both the receiver-only SFT baseline and POND are trained only on math/code data (abstract; Table 3 caption). The knowledge results are zero-shot transfer for both systems.

Why transfer is plausible.

  • The learned interface is domain-general. The sender is frozen, so its prefill states for a knowledge question already encode what it knows about that question. Math/code training does not teach the bridge domain knowledge; it teaches the receiver how to read the sender's final-layer states. This reading operates position-wise on generic hidden states (Eq. 2–4) and enters as a gated residual on top of the unchanged text pathway (Eq. 7), so uninformative latents can be suppressed through αℓ\alpha_\ell.
  • Less pressure on the receiver's weights. With receiver-only SFT, all fitting to math/code targets happens inside the receiver's weights, which overwrites general knowledge. With POND, part of the training signal is explained by the sender's representations. We present this as a hypothesis, not an established mechanism.

Evidence (knowledge suite, Qwen3.8-Max sender):

AnalysisResult
Constant / shuffled message at inference83.70% / 84.00% (SFT: 83.45%; POND: 86.44%)
Mean gate tanh⁡(gℓ)\tanh(g_\ell) over adapted layers0.18
Mean latent mass αℓ\alpha_\ell0.24 on knowledge, 0.28 on math/code
Revision markers ("but", "however", "wait", "on second thought") per 1,000 thinking tokensSFT 3.8 → POND 2.6 (−31.6%)
Outputs with repeated arguments or unproductive backtrackingSFT 28.0% → POND 19.0%
Questions both systems answer correctlyPOND: 15.0% fewer thinking tokens, 30.0% fewer unproductive revisions
  • Without the true message, accuracy falls back to the SFT level, so the gain depends on instance-specific sender information.
  • The latent channel is used almost as much on knowledge inputs as on the training domains.
  • The output analysis indicates how it helps: the receiver already has much of the required knowledge, and sender guidance keeps it on a more consistent solution trajectory, with fewer repeated arguments and unsupported answer reversals. This explains gains on knowledge tasks without any knowledge-domain fine-tuning.

W2: The gain is similar across datasets regardless of the sender–receiver gap. Where does it come from? A per-dataset Table 3 would help.

Why the gap is not a good predictor. The sender's standalone accuracy includes the computation performed during its own autoregressive generation. POND transfers only prompt (prefill) representations, which provide input-specific features that help the receiver interpret the problem. Their value depends on how well they complement the receiver's own capabilities, so a larger standalone gap need not yield a proportionally larger gain.

Per-dataset breakdown (Qwen3.8-Max sender; accuracy / pass@1 in %):

BenchmarkOriginalPONDPOND − Original (pp / rel.)+Review − POND (pp)
AIME80.085.4+5.40 / +6.75%+0.80
OlymMath-Hard50.052.9+2.90 / +5.80%+0.80
HumanEval+81.783.5+1.80 / +2.20%+0.80
MBPP+64.866.8+2.00 / +3.09%+0.90
LiveCodeBench43.247.1+3.90 / +9.03%+1.10
Math/code avg63.9467.14+3.20 / +5.00%+0.88
GPQA-Diamond76.7779.29+2.52 / +3.28%+1.52
MMLU-Redux89.5890.47+0.89 / +0.99%+0.39
MMLU-Pro78.6280.93+2.31 / +2.94%+0.54
ARC-Challenge94.1195.05+0.94 / +1.00%+0.51
Knowledge avg84.7786.44+1.67 / +1.97%+0.74
  • Task training alone does not reproduce the gains. At the suite level, receiver-only SFT changes the math/code average by +0.26 pp (63.94% → 64.20%) and the knowledge average by −1.32 pp (84.77% → 83.45%). POND exceeds matched SFT by 2.94 pp and 2.99 pp, respectively.
  • +Review − POND measures the extra benefit of guidance conditioned on the evolving response; it is largest on long-output tasks (GPQA-Diamond, LiveCodeBench).
  • The gains are not uniform. Relative improvement ranges from 0.99% (MMLU-Redux) to 9.03% (LiveCodeBench). Baseline headroom alone does not explain this: AIME gains 5.40 pp from an 80.0% baseline, while OlymMath-Hard gains 2.90 pp from 50.0%.

The revision will include this table together with sender-alone scores and the constant/shuffled-message controls.

AuthorsOpenReview ↗

Response to Reviewer UMZc (2/2)

W3: Ablation on review frequency.

We evaluated review strides of 256, 512, and 1024 tokens, plus no review, on the same trained checkpoint (Appendix F.1). We will move the results into the main text:

Review strideMath/code acc. (%)Math/code latency (s)Knowledge acc. (%)Knowledge latency (s)
No review67.14114.086.4462.0
102467.70117.086.9064.5
512 (default)68.02119.787.1866.2
25668.10125.087.2070.0

Accuracy saturates at a stride of 512, while latency keeps rising. Because review is delta-only, total sender computation is the same for every stride (Sec. 3.5); the stride changes only the number of sender calls and how soon new guidance becomes visible.


Q1: What does compressing to dz≪dSd_z \ll d_S tell us? How should dzd_z be chosen?

It suggests that the information the receiver can actually use lies in a much lower-dimensional subspace than dSd_S. This is consistent with the receiver reading the latents through its own nkv×dhn_{kv} \times d_h attention space, which is itself far smaller than dSd_S.

dzd_zMath/code acc. (%)Bridge parametersTransmission per 1,000 entries (BF16)
12866.212 M0.244 MiB
25666.816 M0.488 MiB
512 (default)67.1420 M0.977 MiB
102467.240 M1.953 MiB

Accuracy saturates at dz=512d_z = 512: doubling the width to 1024 adds only 0.06 pp while doubling transmission size. In practice, we suggest choosing the smallest dzd_z at which accuracy plateaus.


Q2: Is the compress-then-project design purely for saving computation? How much performance does it sacrifice?

It is not only for computation. The design serves three purposes:

  1. Communication. With the compressor on the sender side and per-layer projections on the receiver side (e.g., cloud sender, edge receiver), one-way transmission is βzMzdz\beta_z M_z d_z bytes, versus βzMzdS\beta_z M_z d_S for raw hidden states (Appendix E.2). Here βz\beta_z is the number of bytes per scalar (2 for BF16), MzM_z is the number of transmitted latent entries, dzd_z is the compressed message width, and dSd_S is the sender's hidden size. The saving is therefore a factor of dS/dzd_S / d_z.
  2. Parameters. One shared compressor plus small per-layer adapters is much cheaper than independent dS→d_S \to K/V projections at every adapted layer, which keeps the bridge at about 0.5% of the receiver's parameters.
  3. Per-layer reading. The shared ZZ keeps the message compact, while per-layer adapters let each receiver layer read it in its own subspace (Sec. 3.3).

Cost. A variant without compression (direct per-layer projection from dSd_S) performs almost identically with 6× the bridge parameters:

BridgeMath/code acc. (%)Knowledge acc. (%)Bridge parameters
Direct projection (no compression)67.386.5120 M
Compress-then-project (POND)67.1486.4420 M

Q3: The token counts in Table 1 seem inconsistent with the 4096-token output limit in the appendix.

Thank you for catching this ambiguity. The 4,096-token cap and the 8,192-token combined text limit in the appendix both exclude the model's thinking (chain-of-thought) segment; the cap applies only to the final response. Table 1 counts all decoded tokens, including thinking, so its totals can exceed these limits.

In the revision, the appendix labels 4,096 as the final-response token cap and states that both limits exclude thinking tokens, and the Table 1 caption specifies that token counts include both thinking and final-response tokens.

The authors' rebuttal addresses some of my concerns, while some remain, mainly what the bridge actually teaches. The improvement of the receiver is relatively constant regardless of the gap between the sender and receiver still surprises me, it may be helpful if the author can study the trajectories of the receiver with and without the sender's latent state, what changes of behaviour does it exhibit, and compare it with the sender's own trajectory. Given the existing rebuttal material, I'm happy to increase the score to weak accept.

We sincerely thank the reviewer for raising the score to weak accept and for the concrete suggestion. Comparing the receiver's generation trajectories with and without sender latents against the sender's own trajectory helps clarify what the bridge transfers and why the gains are similar across sender–receiver pairs. We ran this analysis on AIME (150 problems) with the Qwen3.8-Max sender.

Trajectory metricUnitReceiver (SFT)Receiver + PONDSender alone
Revision markers ("wait", "but", "however", "on second thought")per 1,000 thinking tokens3.82.6 (−31.6%)1.4
Outputs with repeated arguments or unproductive backtracking% of problems28.0%18.7%11.3%
Answer reversals (intermediate answer changed)mean per problem1.420.870.51
Correct → incorrect reversalsmean per problem0.280.190.08
Thinking tokens on problems all three systems solverelative to SFT1.00×0.85×1.32×
Step-level overlap with the sender's solution path% of reasoning steps34.241.5n/a

What changes in the receiver's behaviour. With the sender's latents, the receiver spends less of its thinking on ineffective turns. It revises itself less often (−31.6% revision markers), changes its intermediate answer less often (1.42 → 0.87 per response), and halves the rate at which it abandons a correct intermediate result (0.28 → 0.19). On problems all three systems solve, it reaches the answer with 15.0% fewer thinking tokens.

Comparison with the sender's trajectory. POND moves the receiver toward the sender's stability without turning it into a copy of the sender. Step-level overlap with the sender's solution path modestly increases (34.2% → 41.5%), and the receiver remains far more concise than the sender (0.85× vs. 1.32× thinking tokens). The receiver keeps its own solution strategy and follows it more consistently.

Why the gain is roughly constant across sender–receiver gaps. POND transfers only prefill states, so the sender contributes a better reading of the problem, not its own generated reasoning. The benefit is therefore limited by how often the receiver derails, not by how much stronger the sender's full decoding is. Consistent with this, across benchmarks the accuracy gain correlates with the reduction in revision markers (Pearson r = 0.82, n = 9) much more strongly than with the sender–receiver accuracy gap (r = 0.24, n = 9).

Released resources.

On the anonymous project page linked in the paper (https://latent-1117.pages.dev/), we have released the trained 4B and 9B receiver weights for configurations with Qwen3.8-Max or Qwen3.8-Flash-Next as the sender, along with the complete training dataset and training & evaluation code. The release also includes all weights needed to run the Qwen3.8-Flash-Next configuration locally. Due to current storage constraints, the 35B-A3B receiver weights are not yet included; we will release trained weights for all receiver sizes, including 35B-A3B.

Thank you again for your thoughtful feedback.

Reviewer_wmFrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework for large–small language model collaboration that transfers the hidden representations of a frozen large model to a small model through a trainable bridge. Instead of generating intermediate text, the large model performs only prefill, while the small model handles all autoregressive decoding. The paper further introduces periodic review, which allows the large model to reprocess the partially generated response and update its guidance. Experiments on mathematical reasoning, coding, and general-knowledge tasks demonstrate performance improvements over the standalone small model.

Critical strengths
  1. The paper presents an interesting alternative to text-based model collaboration, avoiding the need for large-model autoregressive decoding.

  2. The trainable bridge provides a practical mechanism for transferring representations between independently trained models.

Critical weaknesses
  • The novelty relative to existing cross-model knowledge distillation and representation-transfer methods is not sufficiently clear.

  • The efficiency advantage remains insufficiently established, as additional prefill passes, bridge computation, and memory overhead may offset the savings from avoiding large-model decoding.

  • The contribution of periodic review requires stronger evaluation under matched computational budgets.

  • Cross-domain generalization and compatibility across different model architectures need more systematic validation.

Questions for authors
  1. How does POND compare with text-based collaboration and alternative representation-transfer methods in end-to-end latency, memory usage, and cost under matched performance constraints?

  2. How much improvement comes from the sender's representations rather than additional trainable bridge parameters or task-specific training?

  3. What is the marginal benefit of periodic review per additional prefill pass, and when does it become cost-effective?

2 comments
AuthorsOpenReview ↗

Response to Reviewer wmFr (1/2)

We thank the reviewer for the careful reading and for recognizing POND as an interesting alternative to text-based collaboration. We address novelty and the contribution of sender representations, efficiency, periodic review, and generalization below.


W1 / Q2: Novelty, and how much of the gain comes from the sender's representations

Positioning. Unlike distillation, POND does not compress the sender into the receiver's weights: the frozen sender is consulted at inference, per input, through prefill only. Unlike prior representation transfer (e.g., C2C), POND stores guidance in a separate gated latent KV memory and updates it during decoding through delta-only review. A direct C2C comparison is given under W2 below and in our response to Reviewer i8Kg.

Attribution. All rows below use Qwen3.8-Max as the sender and the same math/code training data, budget, and optimization recipe.

SystemMath/code avg (%)Knowledge avg (%)
Original receiver63.9484.77
Receiver-only SFT (no latent messages)64.2083.45
Frozen receiver + trained bridge65.4085.50
POND checkpoint, constant message at inference64.1083.70
POND checkpoint, shuffled message at inference64.0583.50
POND, true message (single transfer)67.1486.44
POND + Review68.0287.18
  • Not task-specific training. Receiver-only SFT differs from POND only in lacking the sender's representations. POND exceeds it by 2.94 pp on math/code and 2.99 pp on knowledge; SFT alone gains 0.26 pp and loses 1.32 pp, respectively.
  • Not a generic bias. Replacing the true message with a constant or shuffled one on the same checkpoint drops accuracy back to the SFT level, so the receiver relies on instance-specific sender information.
  • Not receiver adaptation alone. With the receiver frozen, the bridge by itself still improves over the original receiver by 1.5 pp and 0.7 pp.
  • Not extra parameters. Receiver-only SFT already tunes all Qwen3.5-4B parameters. The bridge (two-layer compressor with db=2048d_b=2048, dz=512d_z=512; per-layer K/V projections; RMSNorm scales; scalar gates) adds about 20 M parameters, roughly 0.5% of the receiver.

W2 / Q1: Efficiency versus text advising and representation transfer

End-to-end latency. The reported latency includes sender prefill, bridge computation, and transfer. Single-transfer POND still reduces suite-average latency by 8.4–13.9% on math/code and 5.4–12.5% on knowledge across the three senders, with 10.2–14.8% and 8.2–13.1% fewer decoded tokens.

Matched-performance comparison (GPQA-Diamond, Qwen3.8-Max sender):

MethodAccuracy (%)Latency (s/example)
Original receiver76.77149.2
Text advising79.3156
C2C79.3134
POND, single transfer79.29128.6
POND + Review80.81136.9

At the same accuracy (~79.3%), POND is 17.6% faster than text advising and 4.0% faster than C2C. Text advising is slower than the receiver alone, whereas POND is faster.

Sender cost. Under the parameter-multiply approximation of Figure 2, a GLM-5.3 sender (40B active parameters) needs about 80 TFLOPs to prefill a 1,000-token prompt, versus 880 TFLOPs to also generate 10,000 tokens, an estimated 90.9% reduction in sender-side work.

Memory and communication (Appendix E.2). BF16 latents with dz=512d_z=512 take 1,024 bytes per sender entry, about 0.98 MiB per 1,000 entries. Added receiver KV storage is 2βLaMznkvdh2\beta L_a M_z n_{kv} d_h; e.g., with β=2\beta=2, La=8L_a=8, Mz=1000M_z=1000, nkv=2n_{kv}=2, dh=256d_h=256, this is about 16.4 MB.

AuthorsOpenReview ↗

Response to Reviewer wmFr (2/2)

W3 / Q3: Marginal benefit and cost of periodic review

Change from single-transfer POND to POND + Review (Tables 1–2):

SenderMath/code: Δ acc. / Δ latencyKnowledge: Δ acc. / Δ latency
Qwen3.8-Max+0.88 pp / +5.7 s (+5.0%)+0.74 pp / +4.2 s (+6.8%)
GLM-5.3+0.68 pp / +5.1 s (+4.3%)+0.60 pp / +4.0 s (+6.2%)
DeepSeek-V4-Flash+0.70 pp / +4.7 s (+3.9%)+0.60 pp / +3.9 s (+5.8%)

Review-frequency sweep on the same checkpoint (GPQA-Diamond, Qwen3.8-Max):

Review scheduleAccuracy (%)Latency (s/example)
No review79.29128.6
Every 1024 receiver tokens80.3133
Every 512 receiver tokens (default, Table 2)80.81136.9
Every 256 receiver tokens80.9144
  • Diminishing returns. Going from 1024 to 512 adds about 0.5 pp for about 4 s; going from 512 to 256 adds about 0.1 pp for about 7 s. A stride of 512 is therefore our default.
  • When it is cost-effective. Review pays off on long-output tasks, where its overhead is amortized. On GPQA-Diamond it adds 1.52 pp for 8.3 s and remains faster than the original receiver (136.9 vs. 149.2 s). On short tasks it is less favourable: on MBPP+ it adds 0.90 pp but raises latency from 16.0 to 17.3 s, above the receiver's 16.8 s. Single transfer is the better choice under tight latency budgets on short outputs.

W4: Cross-domain and cross-architecture generalization

Cross-domain. The receiver and bridge are trained on math/code only. Single-transfer POND nevertheless improves 12/12 sender–knowledge-benchmark pairs and 27/27 pairs over all nine benchmarks, while receiver-only SFT lowers the knowledge average by 1.32 pp.

Cross-architecture senders. Two of the three senders (GLM-5.3, DeepSeek-V4-Flash) come from a different family than the Qwen3.5-4B receiver and use different tokenizers; both improve every benchmark.

A second receiver family. We additionally trained Gemma 3 4B IT as the receiver with Qwen3.8-Max as the sender. SFT and POND use the same math/code data and number of optimizer steps, and both POND rows share one checkpoint.

Gemma receiver configurationAIME Acc. (%)LiveCodeBench v6 Pass@1 (%)GPQA-Diamond Acc. (%)
Original Gemma 3 4B IT16.6718.8630.8
Gemma + matched SFT25.3322.5032.24
Gemma + POND28.6725.6435.46
Gemma + POND + Review30.0027.1237.21
POND − matched SFT (pp)+3.33+3.14+3.22
+Review − POND (pp)+1.33+1.48+1.75

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

Dear Reviewers,

Thank you again for your constructive feedback. Since our initial responses, we have completed experiments with two additional receivers, Qwen3.5-9B and Qwen3.6-35B-A3B. Both use Qwen3.8-Max as the sender and the same math/code training data, training budget, and recipe as Qwen3.5-4B. Together with Gemma 3 4B IT (reported in our earlier responses), POND is now evaluated on four receivers spanning two model families, three sizes, and both dense and MoE architectures.

Accuracy (macro-average, %; sender: Qwen3.8-Max; the two POND rows share one checkpoint)

ReceiverSuiteOriginalMatched SFTPONDPOND + ReviewPOND − SFT (pp)Review − POND (pp)
Qwen3.5-4BMath/code63.9464.2067.1468.02+2.94+0.88
Qwen3.5-4BKnowledge84.7783.4586.4487.18+2.99+0.74
Qwen3.5-9BMath/code65.2866.2068.6069.15+2.40+0.55
Qwen3.5-9BKnowledge89.2088.0191.5091.96+3.49+0.46
Qwen3.6-35B-A3BMath/code72.2474.0875.9376.30+1.85+0.37
Qwen3.6-35B-A3BKnowledge91.2090.8092.9593.17+2.15+0.22

POND consistently outperforms data-matched SFT across all three receivers and both suites, including held-out knowledge tasks. Since the sender is frozen and runs prefill only, these gains come from latent guidance through the lightweight bridge rather than additional supervision. The gain narrows for stronger receivers.

Relevance to on-device deployment. All three Qwen receivers can be deployed on end-user devices. POND keeps all autoregressive decoding, the sequential and dominant cost, on the device. The frontier sender is called only for prefill and delta-only review, and transmits a compact message (about 0.98 MiB per 1,000 latent entries in BF16). A cloud–edge split thus gains frontier-model guidance without frontier-model decoding, and without sending sender text back to the device.

Relevance to emerging architectures. Recent frontier open-source models such as DeepSeek-V4.1-Flash[1] adopt a Causal Encoder-Decoder (CED) architecture that separates prefill from autoregressive generation: a causal encoder processes the prompt, and the decoder's global KV cache is projected from the encoder's final hidden states, so most prompt tokens need not run through the full decoder stack. A related split appears in Xiaomi's HySparse2[2], which follows the YOCO-style self-/cross-decoder architecture[3] so that prefill can exit after the self-decoder. POND realizes a related division of labour across two independently trained models: the sender encodes the context through prefill only, while the receiver performs decoding and accesses the sender's hidden states through latent KV memory. Unlike these single-model designs, which learn the separation within one jointly trained backbone, POND connects frozen heterogeneous models through a lightweight trainable bridge, which is what enables POND to reuse existing frontier models without retraining them. This trend supports prefill-only collaboration as a practical design choice and suggests that CED encoders or YOCO-style self-decoders could serve as natural sender interfaces in future work.

With the discussion period ending on November 18, we would appreciate any follow-up questions or feedback on whether these additions address your concerns. We remain available to clarify any remaining concerns.

Best regards,

The Authors

[1] Xu, Anyi, et al. "Deepseek-v4. 1-flash: Pushing the limits of kv cache compression." arXiv preprint arXiv:2609.19969 (2026).

[2] Wei, Jianyu, et al. "HySparse2: Hybrid sparse attention with two-level KV sharing." arXiv preprint arXiv:2609.26368 (2026).

[3] Sun, Yutao, et al. "You only cache once: Decoder-decoder architectures for language models." Advances in Neural Information Processing Systems 37 (2024): 7339-7361.

AuthorsOpenReview ↗

Response to Reviewer i8Kg

Thank you for the careful follow-up questions, which helped us clarify the evaluation protocol and improve the transparency of our reporting.

Q1: Choice of maj@3 and comparison with maj@4, maj@8 and maj@16

We originally chose k=3k=3 to approximately match the compute budget of one POND inference, including sender prefill, bridge computation, and receiver generation. We agree that comparisons at commonly used sample counts would be informative. We therefore additionally evaluate receiver-only maj@4 and maj@8 using the same sampling settings and evaluation sets.

MethodAIMEOlymMath-HardGPQA-DiamondRelative compute
Receiver maj@30.830.510.781.00×
Receiver maj@40.830.520.791.33×
Receiver maj@80.840.560.812.67×
Receiver maj@160.880.570.825.34×
POND, one response per problem0.860.570.811.00×
POND, maj@30.900.600.833.00×

Increasing the receiver-only voting budget improves accuracy, but requires substantially more compute to approach POND’s performance. A single POND response outperforms receiver maj@8 on all three benchmarks while using approximately 37.5% of its compute. Compared with receiver maj@16, it matches accuracy on OlymMath-Hard and GPQA-Diamond, although it remains lower on AIME.

Importantly, POND also benefits from majority voting: POND maj@3 achieves 90%, 60%, and 83% accuracy, exceeding receiver maj@16 by 2, 3, and 2 percentage points, respectively, while using approximately 56% of its compute. These results suggest that accessing a much larger sender’s prefill representations improves the receiver’s attainable accuracy within the evaluated compute range, beyond the gains obtained by allocating more samples to the receiver alone. The additional gains from voting on top of POND further indicate that latent guidance and increased sampling are complementary.

Q2: AIME evaluation settings, dataset size, and corrected results

Thank you for requesting these details. Yes, AIME (2022–2026) includes all 150 problems from AIME I and II over these five years.

For the final evaluation, we use Qwen3.5-4B in thinking mode with temperature = 0.6, top-p = 0.95, top-k = 20, and a maximum context length of 128K tokens. For AIME, we independently sample three responses per problem, yielding 450 responses per evaluated configuration. We estimate pass@1 as the total number of correct responses divided by 450. The three responses are scored individually; no majority voting is used for this metric. We follow the same sampling and averaging protocol for the other datasets, with the denominator adjusted to three times the number of problems in each dataset.

Upon checking the original evaluation, we found that overlength responses had been excluded from the denominator. In the original setup, even with a 64K or 128K token context window, Qwen3.5-4B occasionally produced excessively long chains of thought that exhausted the available context budget and were truncated before yielding a valid final answer. The exclusion of these responses was insufficiently documented and explains why the original scores did not correspond to a denominator of 450. We have therefore recomputed the results using all 450 responses, counting truncated responses without a valid final answer as incorrect.

TaskReceiver aloneQwen3.8-Max PONDQwen3.8-Max +ReviewImproveGLM-5.3 PONDGLM-5.3 +ReviewImproveDeepSeek-V4-Flash PONDDeepSeek-V4-Flash +ReviewImprove
AIME (150)80.0086.0086.67↑ 6.6784.6785.33↑ 5.3384.6785.33↑ 5.33
OlymMATH-Hard (100)50.054.057.0↑ 7.053.053.0↑ 3.052.053.0↑ 3.0

To support reproducibility, we have released the trained receiver weights at two scales (4B and 9B), all model weights for the locally deployable configuration with Qwen3.8-Flash-Next as the sender, the complete training dataset, and the evaluation code on our anonymous project page (https://latent-1117.pages.dev/) in the paper. These resources enable independent evaluation across receiver sizes and reproduction of the locally deployable sender–receiver configuration.

We welcome any further questions and would be happy to discuss or clarify any remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.