- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PAIR, a multi-model approach for efficient long-context inference. PAIR uses a smaller Reader to process the full context, while a larger Reasoner only sees the question and generated sequence. At each decoding step, their logits are combined using a weighted sum, and the resulting next token is appended to a shared generated sequence, which serves as the communication channel between the two models. This allows context-specific evidence extracted by the Reader to become indirectly available to the Reasoner for subsequent reasoning. The models are further adapted through role-specific fine-tuning for collaborative generation.
Critical strengths
-
Simple and interpretable method : PAIR introduces a simple and intuitive decomposition of long-context inference: a smaller Reader retains the full context, while a larger Reasoner operates only on the question and generated sequence, with their predictions combined through logit-level fusion. Despite its simplicity, the method achieves a strong accuracy–efficiency trade-off, retaining most of the full-context large model’s accuracy while substantially reducing KV-cache memory and improving throughput.
-
Empirical validation across different serving regimes and model configurations : The evaluation goes beyond theoretical KV-memory savings and includes end-to-end throughput, matched-batch comparisons, scaling with context length and batch size, and two-GPU branch-parallel execution. The authors also evaluate multiple Reader–Reasoner size ratios and three model families, providing useful evidence that the observed accuracy preservation is not specific to a single model pair.
Critical weaknesses
-
The main result does not fully cover closely related input-compression approaches. While the paper identifies CoGenesis as the closest architectural precedent to PAIR, the distinction is primarily discussed in terms of motivation and deployment setting, and no direct empirical comparison is provided. Similarly, input-compression methods such as LLMLingua/LongLLMLingua are discussed in related work but are absent from the empirical comparison. Comparing against sequential context compression followed by large-model inference would provide a broader assessment of PAIR’s end-to-end accuracy–efficiency trade-off.
-
Although Appendix G.1 explicitly distinguishes resident KV-cache storage from peak GPU memory, the paper does not report total serving-memory usage, including the additional Reader weights. Reporting this at matched batch sizes across context lengths would clarify the practical memory savings.
Questions for authors
- Could you provide direct empirical comparisons with CoGenesis and with input-compression methods such as LLMLingua/LongLLMLingua followed by large-model inference?
- How much total GPU memory does PAIR require relative to the full-context baseline across different context lengths?
4 comments
Response to W2/Q2
W2 / Q2. Peak GPU memory
W2. Although Appendix G.1 explicitly distinguishes resident KV-cache storage from peak GPU memory, the paper does not report total serving-memory usage, including the additional Reader weights. Reporting this at matched batch sizes across context lengths would clarify the practical memory savings.
Q2. How much total GPU memory does PAIR require relative to the full-context baseline across different context lengths?
Peak GPU memory depends on batch size, context length, and the stage of inference at which the maximum allocation occurs. Therefore, our original evaluation compares end-to-end throughput at the largest tested feasible batch size under the same GPU memory budget. This accounts for PAIR's additional Reader weights while measuring the practical benefit of its lower per-sequence memory requirements.
To make the memory crossover explicit, we additionally measured peak GPU memory at matched batch sizes across four context lengths. We used the HotpotQA batch-scaling workload with 40, 80, 120, and 160 documents (approximately 6.0K, 11.6K, 17.1K, and 22.9K tokens) on a single GH200 GPU with 95.08 GiB of usable memory. Both methods use the FKV backend, with bf16 model weights and bf16 KV caches. LoRA adapters are merged into the model weights.
We report the maximum PyTorch-allocated memory (torch.cuda.max_memory_allocated()) observed during the complete 96-question workload, including prefill, question processing, CUDA Graph operations, and decoding. The measurements include model weights, KV caches, and temporary allocations. The peak occurs during context prefill, when preallocated KV buffers coexist with temporary activations.
The largest tested feasible batch sizes for 40, 80, 120, and 160 documents were 10, 6, 4, and 3 for the full-context 32B, compared with 16, 10, 6, and 6 for PAIR. These are not necessarily the maximum supported batch sizes; PAIR did not encounter OOM at its largest tested batches.
| Docs | Batch | 32B (GiB) | PAIR (GiB) |
|---|---|---|---|
| 40 | 2 | 67.5 | 78.5 |
| 40 | 8 | 87.0 | 84.7 |
| 80 | 2 | 71.5 | 79.3 |
| 80 | 6 | 93.5 | 85.1 |
| 120 | 2 | 75.8 | 80.1 |
| 120 | 4 | 91.6 | 83.7 |
| 160 | 2 | 79.8 | 80.9 |
| 160 | 3 | 91.9 | 84.5 |
Immediately after model loading, the full-context 32B occupies 61.027 GiB, compared with 75.244 GiB for PAIR, reflecting 14.217 GiB of additional Reader weights. This makes PAIR more memory-intensive at small batch sizes. Linear fits to the batch-sweep measurements indicate that the full-context 32B requires approximately 2.8–4.0× more incremental memory per sequence than PAIR. The estimated crossover batch sizes are approximately 7, 4, 3, and 2 for 40, 80, 120, and 160 documents, respectively. Consequently, PAIR supports larger tested batches under the same GPU memory budget, explaining the throughput advantage reported in our original evaluation.
Thank you for the additional comparisons and memory measurements. The baseline comparisons and memory crossover analysis help me better understand where PAIR offers practical benefits. It would be helpful to include these results in the revised paper.
Appreciate your constructive feedback. We have uploaded a revised manuscript with the KL-based fusion-weight selection procedure in Section 4.3, additional training details in Appendix B, fusion-weight calibration results in Appendix C, and matched-batch peak GPU memory measurements and memory crossover analysis in Appendix G. We have also updated the evaluation results to reflect the KL-selected fusion weights.
Response to W1/Q1
W1 / Q1. CoGenesis and input compression
W1. The main result does not fully cover closely related input-compression approaches. While the paper identifies CoGenesis as the closest architectural precedent to PAIR, the distinction is primarily discussed in terms of motivation and deployment setting, and no direct empirical comparison is provided. Similarly, input-compression methods such as LLMLingua/LongLLMLingua are discussed in related work but are absent from the empirical comparison. Comparing against sequential context compression followed by large-model inference would provide a broader assessment of PAIR's end-to-end accuracy--efficiency trade-off.
Q1. Could you provide direct empirical comparisons with CoGenesis and with input-compression methods such as LLMLingua/LongLLMLingua followed by large-model inference?
We compare PAIR against three alternatives: Sequential, CoGenesis, and LongLLMLingua. In Sequential, the adapted Reader generates a complete response, including reasoning and the answer, which is then passed to the unadapted 32B for final answering. LongLLMLingua uses the base 7B to compress the context to approximately 25% of its tokens before the unadapted 32B generates an answer.
We also evaluate a CoGenesis-style learned fusion gate as an alternative to PAIR's fixed fusion. Since CoGenesis was developed for personalized generation and does not specify a training setup for long-context QA, we use PAIR's S-solo-adapted Reader and explicit reasoning step. We train the fusion gate on the L-fusion corpus using the same distillation procedure as PAIR. This comparison evaluates a CoGenesis-style fusion approach adapted to long-context QA, rather than the original CoGenesis training setup.
PAIR achieves higher F1 than Sequential and LongLLMLingua on all four benchmarks. It also outperforms CoGenesis on three benchmarks and trails by only 0.001 F1 on LoCoMo.
We additionally compare end-to-end throughput and time to first token (TTFT) on 96 HotpotQA questions using one GPU, with both models loaded for methods that require them.
Sequential achieves similar throughput to PAIR, but its TTFT is 2.2 times higher because the 32B must wait for the Reader to finish generating its response. LongLLMLingua incurs an even larger delay from context compression. The TTFT values for Sequential and LongLLMLingua are calculated from their measured stage timings.