acceptodds
Under review as a conference paper at ICLR 2027

MILO: Efficient Many-shot In-Context Learning with Block-wise Low-rank Compression

Abstract

Many-shot in-context learning (ICL) enables large language models (LLMs) to adapt to complex tasks by conditioning on thousands of demonstration examples, but this paradigm shifts the inference efficiency bottleneck to the key-value (KV) cache memory. Due to the linear scaling behavior of the KV cache, storing these intermediate tensors has become a paramount challenge for both online serving and on-device deployment. To address this issue, we propose a novel compression framework, termed MILO, that exploits the low-rank redundancy inherent in many-shot contexts. Specifically, MILO features a block-wise low-rank compression strategy that compresses the KV cache at the block granularity, where each block contains multiple many-shot examples. Furthermore, to handle the heterogeneous context density across different blocks, MILO dynamically allocates rank budgets based on the information entropy, preserving the fidelity of critical blocks while aggressively compressing redundant ones. Experimental results on Qwen2.5 models demonstrate that our method achieves up to 50% reduction in KV cache memory and 1.8 throughput improvement, with negligible performance degradation on classification and reasoning benchmarks, significantly outperforming prior baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
in-context learning, large language models, model compression

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_8ccxOpenReview ↗
Rating
1
Confidence
4
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes the MILO compression framework, which features a block-wise compression approach that compresses many-shot examples' KV cache at block granularity. The framework dynamically allocates budgets based on entropy to avoid overcompression. Experiments on Qwen2.5 with ICL benchmarks demonstrate that the proposed MILO reduces up to half storage and improves 1.8x efficiency while the accuracy loss is negligible.

Critical strengths
  • The motivation of this paper is well-defined. Two challenges have been identified: the compression granularity and the information density within a KV cache. The novelty appears to be appropriately directed toward addressing these issues (block-wise and dynamic rank).
  • The presentation of this paper appears to be clear and logical, making it easy to follow and understand.
  • The experimental part is clear in showing the performance of the proposed method compared to two baselines and full cache. Each illustration is direct and easy to understand.
Critical weaknesses

I have two critical concerns about this paper. If these can be properly addressed, I would like to recommend accepting the paper.

  • The testing model is too small and out of date. For example, Qwen now has the newest Qwen3.8 models released. It is necessary to prove the approaches effective from the new up-to-date architectures. And it is necessary to prove the approach is scalable. And the computing resources are limited, I recommend scaling this test to 20B-level models.
  • Only one model series is tested in this paper. This may hinder the demonstration of generalization ability for this approach. Although I saw many KV cache papers testing on one model family, I strongly believe this is not appropriate or rigorous due to the performance stability nature of LLM evaluation.
Questions for authors

I don't have any more questions for the authors. The presentation of the paper is very good. I sincerely hope the experiments can be substantially enhanced.

2 comments

Dear Reviewer 8ccx,

Thank you for your valuable and constructive comments. To address your concerns about the experimental scope, we have added additional experiments on Ministral-3-8B and Ministral-3-14B and extended the number of shots to 1024, with the results summarized below. We have also updated Table 1 and Table 3 in the manuscript.

ModelMethodAvg.
Ministral-3-8BFull Cache0.809
ASVD0.766
Palu0.766
MILO0.782
Ministral-3-14BFull Cache0.827
ASVD0.758
Palu0.792
MILO0.799
ModelMethodNo. of ShotsAccuracy
Ministral-3-14BFull Cache10240.847
MILO10240.815
Full Cache20480.856
MILO20480.824
Full Cache40960.871
MILO40960.833
Reviewer_8ccxOpenReview ↗

Response to author's rebuttal.

The concerns are not addressed. The experiments are not comprehensive enough. The work is far below the acceptance threshold.

Reviewer_GA3YOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes MILO, a framework for efficient many-shot ICL using block-wise low-rank KV compression, adaptive rank allocation, and GPU optimizations. Experiments on Qwen2.5-3B/7B report higher accuracy than ASVD and Palu and improved throughput under KV offloading.

Critical strengths
  1. Relevant problem. The paper addresses the substantial KV-cache storage and transfer costs in many-shot ICL.
  2. Clear problem statement and motivation. The paper clearly explains the bottleneck and motivates adaptive compression through spectral differences between keys and values and across demonstration blocks.
Critical weaknesses
  1. The comparative advantage is insufficiently established. Baseline adaptations, preprocessing cost accounting, and matching of retrieved inputs and actual storage budgets are unclear. These gaps make it difficult to attribute the reported gains to MILO’s design.
  2. The complete system benefit remains unverified. Offline costs and amortization, online timing boundaries and latency breakdowns, and peak GPU memory are insufficiently reported, preventing a reliable assessment of MILO’s practical efficiency.
Questions for authors
  1. Rank allocation and motivation: What are the exact spectral metrics and normalization procedures used in Figures 1–2, and how do these observations inform the implemented allocation? Is the rank budget distributed separately or jointly across layers, heads, and K/V? How are differences in block token lengths handled? Beyond uniform and random allocation, what advantage does the proposed rule offer over other spectrum-based allocation strategies?

  2. Baseline coverage and fairness: How were ASVD and Palu adapted, calibrated, and configured? The submission describes ASVD as compressing examples for each incoming query, whereas MILO compresses during encoding; please clarify whether the placement and accounting of preprocessing work are comparable. Do all methods use identical retrieved examples, ordering, and token budgets? Please report actual storage and transferred-byte counts, since a shared 50% rank budget does not establish comparable storage costs. A comparison with xKV [1], or an explanation of its applicability, would strengthen the evaluation. Table 2 should also distinguish the benefits of combining MILO with quantization from those of standalone quantization at comparable storage budgets.

[1] Chi-Chih Chang et al. xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extraction. ICML, 2026.

  1. Encoding protocol and amortization: What are the demonstration-pool sizes, total token counts, dataset splits, evaluation-query counts, and block construction rules? Please report construction time and storage for each method, separating common KV pre-encoding costs from additional compression and indexing costs. How many queries must reuse the cache to amortize these costs?

  2. Online performance and memory: Which stages are included in Figure 4’s end-to-end throughput measurement? Please provide request latency and a breakdown of retrieval, CPU–GPU transfer, reconstruction, and model execution, explaining any overlap. Figure 5(c)’s kernel optimization ablation does not provide this complete pipeline breakdown. Please also report measured peak GPU memory for each method, including reconstruction buffers and temporary workspace, rather than only KV-reduction ratios.

  3. Retrieval and cache reuse: Does BM25 retrieve examples or blocks, and how does its retrieval budget relate to the reported shot count? How are selected examples mapped to blocks, and are unselected examples loaded with them? If a fixed retrieval count is enforced, how are low-relevance matches handled when too few relevant examples or blocks are available? How are attention dependencies and positional encodings handled when cached blocks are selected or reordered?

  4. Robustness and generality: How sensitive are MILO’s benefits to the candidate-pool size and retrieval budget? The evaluation lacks ablations with alternative retrievers. What evidence supports applicability beyond Qwen2.5-3B/7B and the MathQA throughput setting?

  5. Accuracy trade-offs: How should the claim of “negligible” degradation be reconciled with Table 1? Average accuracy decreases by 3.0 and 2.8 percentage points for the 3B and 7B models, respectively; individual losses include 32.8% to 27.6% on MathQA for 3B and 68.4% to 63.2% on SVAMP for 7B. Please contextualize these losses relative to memory and latency savings and report variability across demonstration sampling or ordering.

2 comments

Dear Reviewer GA3Y,

Thank you for your valuable and constructive comments. Below, we clarify our methodology regarding rank allocation (Q1), baseline comparison (W1, Q2), end-to-end system evaluation (W2, Q4), encoding costs (Q3), impact of the retriever (Q5), generalization (Q6), and accuracy trade-offs (Q7).

Q1: Methodology Regarding Rank Allocation.

  • MILO formulates rank selection as a global resource allocation problem bounded by a fixed total rank budget. We dynamically assign ranks based on block-level information entropy, calculating the marginal information gain of singular components during SVD. We have added additional results to compare with Spectral-based rank allocation in Figure 6 (c).

W1, Q2: Baseline Comparisons.

  • We specifically benchmark MILO against state-of-the-art low-rank methods. We do recognize that the more recent work xKV [1] compresses the KV cache by grouping layers; therefore, it is orthogonal to our approaches. However, all prior approaches operate under the assumption of a single, continuous, and coherent context, while MILO aims to address the KV compression problem explicitly for many-shot in-context learning tasks.
  • For a fair comparison, we reproduce each method under the same many-shot settings, where the KV cache is compressed offline and reconstructed during online inference. We have added additional text to explain the methodology differences in lines 112-113 and lines 122-125 in Section 2.

W2, Q4: End-to-End System Evaluation.

  • We report the system performance in a KV cache offload setting, where MILO can significantly reduce the I/O transfer latency for end-to-end system latency. We have added additional results to showcase the improvement of MILO in Figure 6 (d).

Q3: Encoding Costs.

  • To prevent ambiguity, we have updated our terminology from "pre-encoding" to "offline compression" throughout the manuscript (e.g., Figure 3, Algorithm 1).

  • Unlike dynamic, on-the-fly compression methods, MILO performs SVD on the static many-shot demonstration prompt entirely offline. To clarify our execution pipeline, the computationally expensive SVD factorization occurs entirely during an offline compression stage. Because the many-shot demonstration context is static, this incurs only a one-time computation cost. During online inference, the system solely executes lightweight, parallel low-rank reconstruction matrix multiplications before the RoPE application and sparse attention. Therefore, the offline encoding overhead is completely amortized across all subsequent user queries.

Q5: Impact of Retriever.

  • We have added additional results of MILO's robustness against varying retrieval mechanisms by testing BM25 (lexical), Dense (semantic), and Hybrid selection strategies in Figure 6 (d), with results summarized below.
Number of ShotsFP16BM25DenseHybrid
640.7210.7080.7020.707
1280.7330.7160.7190.722
2560.7430.7210.7170.721
5120.7530.7250.7210.723

MILO maintains consistent accuracy performance across all selection scenarios. This confirms that our block-wise low-rank approximation preserves sufficient fidelity to support accurate KV segment fetching regardless of the underlying retrieval architecture.

Q6: Model Generalization.

We have added additional results on Ministral-3-8B/14B architectures in Table 1 on page 8. Furthermore, we tested MILO's ability to scale with the number of shots in Table 3 on page 10. MILO scales robustly up to thousands of many-shot examples, maintaining competitive accuracy.

ModelMethodAvg.
Ministral-3-8BFull Cache0.809
ASVD0.766
Palu0.766
MILO0.782
Ministral-3-14BFull Cache0.827
ASVD0.758
Palu0.792
MILO0.799
ModelMethodNo. of ShotsAccuracy
Ministral-3-14BFull Cache10240.847
MILO10240.815
Full Cache20480.856
MILO20480.824
Full Cache40960.871
MILO40960.833

Q7: Accuracy Trade-offs.

  • We thank you for the reviewer's detailed comment. We have changed the term 'negligible' to 'minimal' throughout the manuscript to accurately reflect the effectiveness of our approach.

[1] Chi-Chih Chang et al. xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extraction. ICML, 2026.

I thank the authors for their time and effort and appreciate the additional experiments and clarifications. However, my main concerns remain insufficiently addressed; I therefore maintain my current rating.

Reviewer_3QnFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

To address the KV memory bottleneck in multi-example ICL, this paper proposes the MILO, which integrates block-wise low-rank compression, spectral entropy-based dynamic rank allocation, and optimizations via Triton fused kernels. Experiments on Qwen2.5 demonstrate that MILO reduces memory usage by 50% and increases throughput by 1.8x.

Critical strengths
  • The method balances the capture of cross-sample redundancy with reconstruction efficiency.
  • Dynamic rank allocation effectively preserves information in high-density blocks, minimizing precision loss.
  • Operator fusion and multi-stream optimization translate VRAM savings into actual speedups; are orthogonal to quantization.
Critical weaknesses
  • The method proposed in the paper has limited impact.
  • It is not explained how positional encodings are recalculated during retrieval and concatenation following Pre-RoPE compression.
  • There is no assessment of the computational cost associated with SVD during the pre-encoding stage.
  • It is not specified whether the pre-encoding is performed independently for each block or globally.
  • Comparisons with specialized baselines, such as DBSA, are missing.
  • Experiments are limited to 512-shot scenarios and a 7B model.
Questions for authors
  • How are positional encodings for Pre-RoPE Keys dynamically recomputed after retrieving concatenated blocks? Are these pre-encodings computed independently for each block, or are they global? Also provide the computational overhead for the SVD process.
  • How does the method perform with thousands of shots with larger models and when compared against DBSA?
1 comment

Dear Reviewer 3QnF,

Thank you for your constructive feedback. Below, we clarify our core contributions (W1), our methodology regarding RoPE (W2, Q1), our offline compression pipeline and SVD computational overheads (W3, W4, Q1), and our baseline comparisons (W5, W6, Q2).

W1: Limited Impact.

  • While we acknowledge the existence of prior low-rank KV cache compression methods, these approaches operate under the assumption of a single, continuous, and coherent context (intra-context compression). In contrast, MILO explicitly exploits the cross-example semantic redundancy inherent in many-shot demonstrations. Consequently, our approach introduces a fundamentally different paradigm: inter-context compression across discrete example blocks, which yields significantly higher memory reduction in multi-shot serving scenarios.

W2/Q1: RoPE Calculation.

  • We appreciate the opportunity to clarify this architectural detail. During online inference, the positional encodings (RoPE) are applied to the pre-RoPE keys after they are fully reconstructed from their low-rank representations, immediately before the sparse attention operation. We have explicitly formalized this execution sequence in the revised Algorithm 1 (Page 6, highlighted in red).

W3/W4/Q1: Compression Pipeline and SVD Overheads.

  • To prevent ambiguity, we have updated our terminology from "pre-encoding" to "offline compression" throughout the manuscript (e.g., Figure 3, Algorithm 1). Unlike dynamic, on-the-fly compression methods, MILO performs SVD on the static many-shot demonstration prompt entirely offline. The many-shot KV cache is partitioned into discrete blocks, and our entropy-driven dynamic rank allocation is applied to each block. Because this heavy SVD computation is pushed to an offline preprocessing stage, its cost is entirely amortized across all subsequent user queries. The only computational overhead introduced on the critical path during online inference is the lightweight low-rank reconstruction matrix multiplication.

W5/W6/Q2: Baseline Comparisons and Generalization.

  • We note that DBSA is primarily a compute-bound optimization leveraging sparse attention, whereas MILO directly targets the memory-bound capacity constraints of the KV cache via low-rank parameterization. As such, our framework is completely orthogonal and complementary to sparsity-driven techniques.
  • Furthermore, to validate the generalizability of our system across varied model architectures and context scales, we have incorporated new experiments using the Ministral-3-8B and Ministral-3-14B models, successfully scaling up to 4096-shot demonstrations. The updated throughput and accuracy metrics are summarized in the table below.
ModelMethodAvg.
Ministral-3-8BFull Cache0.809
ASVD0.766
Palu0.766
MILO0.782
Ministral-3-14BFull Cache0.827
ASVD0.758
Palu0.792
MILO0.799
ModelMethodNo. of ShotsAccuracy
Ministral-3-14BFull Cache10240.847
MILO10240.815
Full Cache20480.856
MILO20480.824
Full Cache40960.871
MILO40960.833

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Reviewers and Area Chairs,

We sincerely thank all reviewers for their thoughtful and constructive feedback. Based on your valuable suggestions, we have revised our manuscript to clarify our methodology, refine our terminology, and significantly expand our experimental evaluations. Below is a summary of the key updates and clarifications addressing the common themes raised across the reviews.

1. Core Contribution, Novelty, and Baseline Comparisons.

  • Inter-Context vs. Intra-Context Compression: While prior low-rank KV cache compression approaches assume a single, continuous, and coherent context (intra-context), MILO explicitly exploits the cross-example semantic redundancy inherent in many-shot demonstrations. This introduces a fundamentally different paradigm: inter-context compression across discrete example blocks, yielding significantly higher memory reduction in multi-shot serving scenarios.
  • Orthogonality to Existing Methods: Our framework is complementary to recent advancements. DBSA is primarily a compute-bound optimization leveraging sparse attention, whereas MILO directly targets the memory-bound capacity constraints of the KV cache via low-rank parameterization. Similarly, while recent work like xKV [1] compresses the KV cache by grouping layers, MILO uniquely addresses KV compression explicitly across many-shot examples.

2. Execution Pipeline, Overheads, and RoPE Calculation To prevent ambiguity regarding computational costs (raised by Reviewers 3QnF and GA3Y), we have updated our terminology from "pre-encoding" to "offline compression" throughout the manuscript (e.g., Figure 3, Algorithm 1).

  • Amortized SVD Cost: Unlike dynamic, on-the-fly compression methods, MILO performs the computationally heavy SVD factorization on the static many-shot demonstration prompt entirely offline. Because the many-shot context is static, this incurs only a one-time computation cost that is completely amortized across all subsequent user queries.
  • Online Efficiency & RoPE: During online inference, the system solely executes lightweight, parallel low-rank reconstruction matrix multiplications. We have clarified in the revised Algorithm 1 that positional encodings (RoPE) are applied to the pre-RoPE keys after they are fully reconstructed, immediately before the sparse attention operation.

3. Dynamic Rank Allocation & System Evaluation

  • MILO formulates rank selection as a global resource allocation problem bounded by a fixed total rank budget. We dynamically assign ranks based on block-level information entropy, calculating the marginal information gain of singular components during SVD. We have added new results (Figure 6c) comparing this to Spectral-based rank allocation.
  • Furthermore, we evaluated end-to-end system performance in a KV cache offload setting, demonstrating that MILO significantly reduces I/O transfer latency.

4. Robustness to Retriever Selection

  • Addressing Reviewer GA3Y's question on retrieval mechanisms, we evaluated MILO's robustness by testing BM25 (lexical), Dense (semantic), and Hybrid selection strategies. MILO maintains consistent accuracy across all scenarios, confirming that our block-wise low-rank approximation preserves sufficient fidelity for accurate KV segment fetching regardless of the underlying retriever.

5. Model Generalization and Scaling

  • To address concerns regarding experimental scope (raised by Reviewers GA3Y and 8ccx), we incorporated new experiments using the Ministral-3-8B and Ministral-3-14B architectures. Furthermore, we tested MILO's ability to scale with context length, successfully scaling up to 4,096-shot demonstrations while maintaining competitive accuracy against the full cache baselines.

6. Typo

  • We have also revised the manuscript's text to change the term "negligible" to "minimal" when discussing accuracy trade-offs, accurately reflecting our empirical observations.

[1] Chi-Chih Chang et al. xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extraction. ICML, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.