acceptodds
Under review as a conference paper at ICLR 2027

Combining Document LoRA and KV Cache for Long-Context Question Answering

Abstract

Document LoRA encodes information from a source document through low-rank adaptation (LoRA), which trains small weight-update matrices while keeping the base model fixed. These document-specific parameters provide a form of memory alongside the key-value (KV) cache. We observe that the KV cache and document LoRA differ in the types of questions they answer correctly, motivating their combination for long-context question answering. Our approach first applies KV cache eviction, then fine-tunes the model on synthetic question-answer pairs from the source document using LoRA. Training updates only the low-rank matrices in the feed-forward projections while keeping the compressed KV cache in context. When answering a question, the model uses both the document LoRA and the compressed KV cache. On 2WikiMultihopQA, answering with both memories outperforms using the selected KV cache alone or the same learned adapter alone across all four models. The overall F1 gains reach 25.0 and 9.0 points, respectively. We expect this approach to improve efficiency for repeated QA over a shared document and, with chunked processing, for documents that exceed the model's context window.

Primary area:
generative models
Keywords:
long-context question answering, KV cache eviction, document memory, low-rank adaptation, parameter-efficient fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_V5xEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a hybrid approach of combining kv cache eviction along with per-document LoRA to improve long context performance of LLMs. It identifies the complementary nature of both mechanisms by conducting a clean ablation and clearly motivates the proposed approach. The paper demonstrates the effectiveness of the proposed approach on the 2WikiMultihopQA benchmark.

Critical strengths

The central idea behind the proposed hybrid approach is well-motivated, and the experiments are well-designed and validate it well. The question-type analysis in Section 2 gives a concrete reason to combine the two memories, and Figure 3 confirms that reason within a clean experiment setup. Algorithm-wise, training the adapter with the compressed cache in context is an interesting idea. Figure 4(d) isolates it with a well-matched control setting. Both conditions answer with the same 90%-evicted cache, and only the training context differs, which separates the contribution from Zuo et al. (2026), who already studied document LoRA under KV eviction.

Critical weaknesses

The main weakness lies in the main evaluation results in Section 4. The results come from one benchmark 2WikiMultihopQA which averages about 5k words, which is not truly a long-context setting under today’s standards. Moreover, each document has a single question, so reuse across questions, which is the stated motivation, is never truly tested. Compactor, the selector the method itself uses, is missing from Table 2, even though it is the strongest eviction baseline in Figure 4(a).

Questions for authors

Please see the weaknesses listed above

Reviewer_KMZGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper combines a compressed KV cache with a document-specific LoRA for long-context QA. It first evicts the cache, then trains an FFN LoRA on synthetic QA with the compressed cache in context, and uses both at answer time. On 2WikiMultihopQA documents and four models, it reports 3.7–17.0 F1 gains over Full KV while using about 30% of the memory.

Critical strengths
  • [S1] Training the adapter with the answering cache is supported by a clean ablation.

Figure 4(d) changes only the cache seen during training and gives consistent gains of 6–8 F1 across three ranks.

Critical weaknesses
  • [W1] The gains over Full KV likely come from answer format.

Comparison and bridge-comparison questions are two-way choices, so a short guess already earns about 50 F1. The adapter alone scores around 50 on them, while Full KV is mostly well below 50. This suggests QA-SFT mainly teaches short answers, and without a document-agnostic adapter control or EM/accuracy, thus making the main claim not supported.

  • [W2] Incremental contribution and VERY limited evaluation.

Previous works already study document LoRA combined with an evicted KV cache. The experiments use one benchmark with ~8k-token documents and one question per document, so the claimed reuse benefits and the beyond-context-window setting are never tested.

  • [W3] The paper overall is at a low quality.

Given the novelty, experiments, the discussion on related works and presentation is very limited in this work, I feel this work's quality is way below the ICLR standard. Also, much content looks like AI generated.

Questions for authors
  1. How much of the gain does a document-agnostic adapter (same recipe, QA from other documents) recover when used with the compressed KV? Do the conclusions hold under EM or accuracy?

  2. What is the per-document cost of QA synthesis and training, and after how many questions does it break even?

Reviewer_ShEyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Summary

  • They propose a method for context compression that first prefills the document, then evicts KV entries and fine-tunes a low-rank adapter on synthetic QA with the compressed document KV cache available. On the QA benchmark they evaluate, the combined method achieves better overall performance than KV eviction alone or answering with the trained adapter alone.
Critical strengths

Strength

  • They demonstrate complementary effects between KV cache eviction and document LoRA fine-tuning on synthetic QA. Their combined method outperforms either memory alone in overall QA performance across four models on one benchmark.
Critical weaknesses

Weakness

  • I believe their methodological novelty and contributions over the closest prior work [1] are limited. Specifically, Zuo et al. [1] already combine document LoRA with compressed KV, and Back et al. [2] also report complementary effects between document LoRA and external context. Their methodological change is keeping the selected KV cache available during document-specific LoRA training. Figure 4(d) supports this choice, but the comparison includes only one model and 48 documents, rather than the main experimental setting. I believe they should include the [1]-style LoRA + compressed KV approach as a main baseline and extend the comparison to all four models and 196 documents with rank-32 adapters.
  • Comparing against KV eviction methods without fine-tuning in the main table (Table 2) does not isolate the proposed method’s benefit, since it uses additional training. Indeed, the proposed method even achieves better scores than the full KV baseline. A reasonable baseline would be fine-tuning the model with the full document, then applying existing KV-cache eviction methods on top of the fine-tuned model. The authors should include this baseline with matched training data and adapter capacity and report the computational costs. Comparisons should also match total stored memory.
  • The paper’s target domain is narrow, focusing on document-level QA. The evaluation only covers one QA benchmark, and it remains unclear whether the proposed strategy generalizes to other datasets or downstream tasks such as coding and summarization.
  • The authors should also report the practical cost of compression, including QA generation and LoRA training, in terms of GPU hours and peak memory. They mention these upfront costs but do not quantify them or the practical savings from repeated queries after document compression.

[1] Zuo, C., Wang, L., Jurayj, W., Fleshman, W., & Van Durme, B. (2026). Rethinking LoRA Memory Through the Lens of KV Cache Compression. arXiv preprint arXiv:2606.05698.

[2] Back, S., Lee, D., Kang, N., Lee, T., Hong, S. K., Gwon, Y., & Ahn, S. (2026). Understanding lora as knowledge memory: An empirical analysis. arXiv preprint arXiv:2603.01097.

Questions for authors

See the weakness section please.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.