acceptodds
Withdrawn from ICLR 2027

Batch Multiplexing Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Autoregressive decoding repeatedly applies the same model to many independent sequences. We study batch multiplexing, a small change to a decoder-only trans- former that lets several sequences share part of an attention computation while still producing one next-token distribution per sequence. In selected middle layers, a learned multiplexer combines Ksame-position hidden states into one attention in- put. An ordinary attention block runs once per group, and a demultiplexer returns its update to the original sequences before each sequence runs its own MLP. We compare a family of depth-scaled batch-multiplexed models against nanochat us- ing strictly controlled protocols. Across model depths and training horizons, batch multiplexing improves the validation-quality/decode-throughput frontier, which suggests that it is a promising direction for further exploration.

Primary area:
foundation or frontier models, including LLMs
Keywords:
batch multiplexing, autoregressive decoding, attention sharing, transformer inference

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_3TNTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes to process the attention QKV of multiple sequences in a batch as if it were one sequence. They call this batch multiplexing.

Critical strengths

They claim to increase throughput.

Critical weaknesses
  • If I understand correctly, I fundamentally disagree with the idea to concatenate all the sequences together and then share the attention computation over an effective sequence of length K∗TK * T. Aside from leaking information between potentially unrelated sequences, I think this would greatly increase the computational cost with O(K2)O(K^2) scaling.
  • There's also the issue that there are no standard benchmarks here, making it hard to empirically assess whether this method would work. But just based on the idea, I don't think that this kind of thing is usable at scale.
Questions for authors

How do you prevent leakage between sequences when doing this combined attention? How do you mitigate the O(K2)O(K^2) cost scaling?

Reviewer_ZmetOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces batch multiplexing for decoder-only transformers, a method where a learned multiplexer combines hidden states from KK independent sequences into a single shared representation across selected middle layers. Consequently, self-attention and KV caching are executed only once per group before a demultiplexer distributes the updates back to the individual sequences. Across matched depth and training horizons, sharing attention across grouped sequences significantly improves decoding throughput for a given language modeling quality compared to standard transformer baselines.

Critical strengths

The paper explores a novel and interesting design axis, cross-sequence attention sharing, to improve inference efficiency. The empirical evaluation is solid, demonstrating significant decoding throughput improvements over the baseline family across multiple model depths and training token budgets. Furthermore, the supplemental analyses on slot counts, write-weight distributions, and architecture/dataset transferability are reasonable and informative.

Critical weaknesses

The motivation for this specific efficiency axis is not fully compelling. Numerous established paradigms already exist to increase decoding throughput: Mixture-of-Experts (MoE), sparse attention, KV cache compression, and quantization. It is unclear why multiplexing across independent requests is preferable. From an information-theoretic standpoint, compressing independent sequences is counterintuitive, as compression is more effective when exploiting redundancy across dependent sequences.

Furthermore, the paper overlooks several severe limitations regarding deployment and evaluation. Typical serving systems dynamically adjust batch sizes to trade off throughput and interactivity (latency), whereas this method fixes the slot size KK during training. When active requests do not fill all KK slots, sequences must be duplicated to populate the group, introducing redundant computation. In a multi-user serving environment, combining hidden states across distinct users also introduces major cross-request privacy concerns. While such a trade-off might be justifiable if batch sharing improved peak model quality, the results show that multiplexing strictly degrades quality at a fixed model size and only improves throughput. Finally, the evaluation relies exclusively on likelihood-based metrics (validation BPB and the DCLM CORE suite) without measuring open-ended generative performance, leaving it unclear whether the learned distributions generalize robustly during autoregressive sampling.

Questions for authors
  1. Can the batch or group size be dynamically varied at inference time under this framework, and how does the throughput-latency trade-off compare against a vanilla Transformer when sweeping across batch sizes?

  2. To what extent are grouped sequences truly independent during training and evaluation? Specifically, is the data shuffled, or are queries in the CORE evaluation batched by task, and how does mutual information or similarity among grouped sequences impact performance?

  3. Line 377 claims that "Together with Figure 4, this shows that the weights are structured: they specialize across channel ranges and consistently give harder tokens greater influence." What specific aspect of Figure 4 demonstrates that harder tokens receive greater influence?

Reviewer_ezfLOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes batch multiplexing, a decoder-only transformer variant that shares attention across independent sequences. In the middle blocks (the outermost blocks stay ordinary), a learned multiplexer combines the same-position hidden states of K sequences into one attention input using per-channel-range softmax weights. One causal attention runs per group, and a demultiplexer returns a range-wise gated projection of its output to each sequence. Residual streams, MLPs and the LM head remain per sequence, and middle layers keep one KV-cache row per group. At inference, groups advance in lockstep; idle slots are filled with replicated requests or a finished sequence's recycled tokens.

Critical strengths

Clear idea on a new axis.

Bringing data multiplexing to autoregressive decoding by sharing only attention and its KV cache, while keeping per-sequence residual streams, MLPs and LM head, is simple, well motivated, and distinct from head, latent, or sequence-axis KV reduction.

Critical weaknesses
  1. Cross-request coupling is unmeasured.

By design, a request’s predictions depend on which requests share its group. The paper notes that filler slots can perturb group-mates and leaves privacy to future work, but never measures sensitivity to group composition. Re-scoring fixed sequences under different groupings (replicated, cross-domain, degenerate) would quantify this variance.

  1. Unclear rationale for coupling unrelated requests. Independent requests share no content, so there is no clear reason for them to interact, and the paper does not argue why attention across them should be redundant. Mixing their states makes each request’s output depend on which other requests happen to share its group, an odd property, since a model’s predictions should depend only on its own context. The paper’s own results show the interaction only costs quality (K = 1 matches the baseline, K ≥ 2 is worse): it is purely a speed trade-off. Correlated groups (e.g., parallel samples from one prompt) would be better motivated but are untested.
Questions for authors

Why should unrelated requests interact at all? Is there measurable redundancy across independent sequences that the shared attention exploits, or is sharing purely a reduction in attention capacity? How does the method behave when grouped sequences are correlated (e.g., several samples from one prompt)?

Comments on the paper

1 comment

Thank you for the reviews. We would like to clarify five points.

  1. Motivation and contribution. Different sequences may benefit from attention to different degrees at the same token position. Batch multiplexing uses this imbalance to amortize attention across requests with unrelated histories. It combines states at the same position, reducing the number of attention rows from BB to B/KB/K while keeping their length TT. Figure 4 shows that the learned weights are nonuniform and that selectivity varies by layer. Figure 5 shows that tokens with higher write weights also have higher loss when scored independently by nanochat. To our knowledge, this is the first learned attention-sharing mechanism of this kind for autoregressive requests with unrelated histories.

  2. Dynamic group size. The number of groups can vary at inference, but KK is fixed for a trained checkpoint. Our current implementation fills partial groups with duplicates. This limits its ability to use compute efficiently as serving load changes and requires further design.

  3. Privacy. Each sequence is trained to predict its own continuation, and a text-only API returns only that sequence’s generated text. In a controlled test on an earlier (K=8) checkpoint, we changed all seven other slots to cue a competing answer. The target’s logit margin moved only from 0.213 to 0.198, and the answer specified by its own prompt remained the top prediction.

  4. Group composition. The reported DCLM CORE evaluation packs distinct prompt rows within each task. In earlier experiments, replicating each prompt across slots gave higher CORE than sequential packing for both checkpoints tested. On two earlier (K=4) checkpoints, replicating each prompt across slots increased CORE from 0.249 to 0.263 at depth 18 and from 0.262 to 0.283 at depth 26, compared with sequential packing.

  5. Baseline choice. Nanochat’s recipe was fitted for training compute and quality and may yield a different optimum when decode throughput is part of the objective. Whether reducing the attention head count gives a better quality–throughput trade-off requires training and timing those variants under the same protocol.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.