- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes to process the attention QKV of multiple sequences in a batch as if it were one sequence. They call this batch multiplexing.
Critical strengths
They claim to increase throughput.
Critical weaknesses
- If I understand correctly, I fundamentally disagree with the idea to concatenate all the sequences together and then share the attention computation over an effective sequence of length . Aside from leaking information between potentially unrelated sequences, I think this would greatly increase the computational cost with scaling.
- There's also the issue that there are no standard benchmarks here, making it hard to empirically assess whether this method would work. But just based on the idea, I don't think that this kind of thing is usable at scale.
Questions for authors
How do you prevent leakage between sequences when doing this combined attention? How do you mitigate the cost scaling?
Thank you for the reviews. We would like to clarify five points.
Motivation and contribution. Different sequences may benefit from attention to different degrees at the same token position. Batch multiplexing uses this imbalance to amortize attention across requests with unrelated histories. It combines states at the same position, reducing the number of attention rows from B to B/K while keeping their length T. Figure 4 shows that the learned weights are nonuniform and that selectivity varies by layer. Figure 5 shows that tokens with higher write weights also have higher loss when scored independently by nanochat. To our knowledge, this is the first learned attention-sharing mechanism of this kind for autoregressive requests with unrelated histories.
Dynamic group size. The number of groups can vary at inference, but K is fixed for a trained checkpoint. Our current implementation fills partial groups with duplicates. This limits its ability to use compute efficiently as serving load changes and requires further design.
Privacy. Each sequence is trained to predict its own continuation, and a text-only API returns only that sequence’s generated text. In a controlled test on an earlier (K=8) checkpoint, we changed all seven other slots to cue a competing answer. The target’s logit margin moved only from 0.213 to 0.198, and the answer specified by its own prompt remained the top prediction.
Group composition. The reported DCLM CORE evaluation packs distinct prompt rows within each task. In earlier experiments, replicating each prompt across slots gave higher CORE than sequential packing for both checkpoints tested. On two earlier (K=4) checkpoints, replicating each prompt across slots increased CORE from 0.249 to 0.263 at depth 18 and from 0.262 to 0.283 at depth 26, compared with sequential packing.
Baseline choice. Nanochat’s recipe was fitted for training compute and quality and may yield a different optimum when decode throughput is part of the objective. Whether reducing the attention head count gives a better quality–throughput trade-off requires training and timing those variants under the same protocol.