acceptodds
Under review as a conference paper at ICLR 2027

MANTA: Information-Adaptive Context Compression via Matryoshka Tokens and Tolerance-Conditioned Allocation

Abstract

Context compression enables large language models to process long inputs more efficiently by encoding the original context into compact, reusable representations. However, most existing methods fix a single compression ratio at training time, while a few support several predefined ratios. Either way, every context of a given length receives the same allocation, regardless of its information content, resulting in inefficient resource utilization. To address this, we introduce information-adaptive context compression: the allocation grows for information-dense contexts and shrinks for redundant ones. We formalize this through a criterion we call the minimal sufficient allocation: the smallest allocation whose excess loss, measured against the full allocation, remains below a global tolerance. We realize this criterion in MANTA, which exposes the compression ratio as a continuous inference-time condition, built from two components: (1) Matryoshka tokens, a compact nested representation of each chunk with valid arbitrary-length prefixes; (2) a lightweight Elastic Allocator, conditioned on the tolerance, that decides how many Matryoshka tokens each chunk retains from the excess loss its content would incur. MANTA consistently outperforms existing context compression methods across different compression ratios, improving average token-F1 by 9.3% relative in domain and 5.4% out of domain over the strongest compression-matched baseline. Further analyses attribute the improvements to information-adaptive compression, where the allocator flexibly distributes Matryoshka tokens according to information density.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Context Compression, Long-Context LLMs, Adaptive Inference, Nested Representations

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_pFqGOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

MANTA introduces information-adaptive context compression using nested Matryoshka token prefixes and a tolerance-conditioned Elastic Allocator. A per-chunk allocation target is defined by the smallest prefix whose excess loss stays below a tolerance, allowing one model to operate at multiple compression levels. The paper evaluates MRQA in- and out-of-domain splits, several decoder scales and backbone families, and needle-QA at longer context lengths.

Critical strengths

The central design is coherent: nested prefixes make arbitrary-length allocation possible, while conditioning the allocator on a tolerance gives the model a direct inference-time control. The uniform-versus-adaptive comparison and the no-allocator/naive-allocator ablations isolate the value of differentiated allocation better than many compression papers do. Results are reported on both in-domain and out-of-domain MRQA data, across multiple model scales, another backbone family, and contexts longer than those used in training. The gains over the listed compression baselines are consistent enough to make the idea promising.

Critical weaknesses

The comparison should be made more carefully before the method can support a stronger acceptance recommendation. MANTA is reported at nominal ratios such as 65x while baselines are shown at 64x or nearby separately trained ratios; realized retained-token counts and total encoder/decoder cost are not reported. The per-chunk oracle target is computed through repeated loss evaluations while other chunks use the allocator’s own predictions, so the paper should quantify the resulting training/inference mismatch and compare against a strong adaptive baseline under the same protocol. The absolute out-of-domain performance remains substantially below the uncompressed reference, and the long-context needle-QA experiment is not enough by itself to establish robustness to natural long documents.

The related-work coverage is incomplete. The paper should also discuss the following directly relevant works:

  • Perception Compressor: A Training-Free Prompt Compression Framework in Long Context Scenarios — Findings of NAACL 2025
  • GMSA: Enhancing Context Compression via Group Merging and Layer Semantic Alignment — ACL 2026
  • RECOMP: Improving Retrieval-Augmented Language Models with Compression and Selective Augmentation — EMNLP 2023
  • Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inference — AAAI 2025
  • AttentionRAG: Attention-Guided Context Pruning in Retrieval-Augmented Generation — arXiv, 2025

These missing references would help position MANTA against recent adaptive, sentence-level, training-free, and retrieval-oriented compression methods.

Questions for authors

Please add comparisons at identical realized token budgets and report wall-clock latency, peak memory, and encoder cost. How much does performance change when the allocation target is constructed with a fixed background allocation rather than the allocator’s own predictions? A question-disjoint additional dataset and a stronger adaptive baseline would help establish whether the gains reflect the proposed allocation mechanism rather than the particular MRQA training recipe.

4 comments
AuthorsOpenReview ↗

Response Part I

Dear Reviewer pFqG,

We thank the reviewer for the careful and constructive comments. We address each concern below, and the additional experiments and discussions have been incorporated into the revised paper.

Q1: Nominal compression ratio, actual allocation, and end-to-end cost.

We thank the reviewer for highlighting the need to distinguish nominal compression ratios from realized allocations and to report end-to-end costs.

1. Realized token allocation and accuracy.

MANTA retains fewer tokens than SAC and ICAE while achieving higher IID F1 under the reported realized allocation.

MANTA compresses adaptively according to the content of the text, the compression ratio cannot be fixed in advance. Instead, MANTA's Elastic Allocator assigns Matryoshka tokens to each chunk based on its local informativeness and ϵ\epsilon. The table below reports the actual allocation and results on MRQA:

MethodActual retained tokensRatioIID F1IID EMOOD F1OOD EM
MANTA11.8446.53×55.0443.7440.3131.17
SAC12.7643.18×52.0140.9340.1030.59
ICAE12.7643.18×45.7034.8534.2625.78

MANTA retains only 11.84 tokens on average. Despite compressing more tightly than SAC and ICAE (46.53× vs. 43.18×), it improves IID F1 by +3.03 (5.8% relative) over SAC and +9.34 (20.4% relative) over ICAE. Thus, the realized-allocation results support the claim that adaptive allocation identifies high-information-density regions and avoids wasting budget on redundant background content.

2. End-to-end latency, encoder cost, and memory.

MANTA's encode cost is comparable to the baselines, and its wall-clock latency is lower than full context and ICAE, although SAC is faster in this measurement.

We also directly measured request-level latency and memory on the same pool under identical hardware:

MethodRateEncode (ms)Wall (ms)Peak (GB)
Full context1×16.721,474.015.152
MANTA65×26.381,226.855.168
SAC64×26.561,146.515.165
ICAE64×27.151,991.802.638

These request-level measurements confirm that MANTA achieves its accuracy advantage with a tighter retained-token budget and low end-to-end overhead, while the latency comparison also makes clear that SAC remains faster in this measurement.

AuthorsOpenReview ↗

Response Part II

Q2: Offline oracle labels, deployment-time allocation, and fixed background.

We thank the reviewer for drawing attention to the consistency between oracle-target construction during training and allocation at deployment.

1. Offline oracle supervision and deployment-time allocation.

The oracle is used only as offline supervision. At deployment, the Elastic Allocator uses only the context and ϵ\epsilon and produces the allocation in a single run.

The oracle uses the gold answer only during training or diagnosis to construct a supervision signal specifying how many Matryoshka tokens each chunk should retain. At deployment, the Elastic Allocator reads only the context and ϵ\epsilon; it does not read the question or the answer, nor does it scan candidate allocations. We verify the consistency between deployment-time prediction and the offline oracle with ϵ=0.01\epsilon=0.01:

Data distributionPearson rrk-MAE
IID0.961.32
OOD0.881.94

The Pearson rr and k-MAE values support the consistency of the deployment prediction with the offline oracle on both IID and OOD data. Therefore, the oracle is an offline supervision tool rather than a deployment-time input.

2. Predicted-background and fixed-background oracle targets.

Fixed-background training causes only the reported limited changes relative to predicted-background training, so the conclusion does not depend on using the allocator's own predictions as the background target.

We directly compare the two oracle-target constructions with the same ϵ=0.2\epsilon=0.2 for both models. The predicted-background target uses the allocator's own prediction for the other chunks. The fixed-background target instead sets those chunks to the maximum k=32k=32 during oracle construction. The table reports the absolute evaluation of both trained models:

Target constructionIID F1IID EMOOD F1OOD EM
Predicted-background55.8845.4339.8731.29
Fixed-background54.9744.5639.7830.89

At the same ϵ\epsilon, our predicted-background training outperforms fixed-background training by +0.91/+0.87 in IID F1/EM and by +0.09/+0.40 in OOD F1/EM. This matched ablation supports the conclusion that the result does not depend on using the allocator's own predictions as the background target.

3. Absolute OOD performance across compression ratios.

MANTA's absolute OOD performance degrades smoothly with compression rather than remaining far below the uncompressed reference, and closely approaches it at low compression ratios.

The statement that absolute OOD performance is below the uncompressed reference is most relevant at aggressive compression ratios, not throughout the operating curve. The 1B dense Full-FT reference has OOD token-F1 66.59. On the same full-MRQA protocol, MANTA gives:

MethodCompression ratioIID F1OOD F1OOD recovery
Full-FT1.00 ×\times81.7366.59100.0%
MANTA, ϵ=0.01\epsilon=0.0110.30 ×\times70.7755.1182.8%
MANTA, ϵ=0.05\epsilon=0.0513.07 ×\times69.1953.6680.6%
MANTA, ϵ=0.10\epsilon=0.1015.09 ×\times67.2252.0378.1%

At the least-compressed evaluated adaptive point, MANTA reaches 55.11 absolute OOD token-F1 and recovers 82.8% of the uncompressed reference. As allocation becomes more aggressive, the expected gap grows; this is a compression-quality trade-off rather than evidence that MANTA has intrinsically low absolute OOD performance.

AuthorsOpenReview ↗

Response Part III

Q3: Question-disjoint transfer to natural long documents with a strong adaptive baseline.

We thank the reviewer for the valuable suggestion to evaluate question-disjoint data and natural long documents.

MANTA transfers to the audited question-disjoint LongBench suite and outperforms both the fixed-allocation SAC baseline and the strong question-aware adaptive Perception Compressor.

To evaluate transfer to natural long documents without conflating it with the MRQA training pool, we evaluated an 8B LongBench suite containing Qasper, 2WikiMQA, and MuSiQue. We audited the selected IDs against all MRQA fine-tuning examples and found zero exact overlap in both questions and contexts after lowercasing and whitespace normalization. This establishes normalized exact-match disjointness only, and does not rule out semantic overlap or pretraining contamination. To address the request for a stronger adaptive comparison, we added Perception Compressor (PC), a training-free, question-aware adaptive baseline:

MethodCompression ratioQasper2WikiMQAMuSiQueAverage F1
Full-FT1.00×22.9943.2636.9934.41
SAC63.97×14.2423.4510.4816.06
MANTA68.14×18.4427.4116.6220.82
Perception Compressor59.08×6.9815.7113.2411.98

On this audited suite, MANTA reaches a three-task macro F1 of 20.82, exceeding SAC by 4.76 points (29.6% relative) and PC by 8.84 points (73.8% relative). MANTA also operates at a higher realized compression ratio than both SAC and PC.

Q4: Related work—positioning of the representation contract and comparison with related work.

We sincerely thank the reviewer for providing these important references and note that we have added all of them to the Related Work section of the revised manuscript and discuss them in depth. Although these state-of-the-art methods all aim to reduce inference-time context overhead, MANTA differs from existing work in its underlying Representation Contract. Perception Compressor and AttentionRAG depend on a specific question input and perform query-aware text pruning; RECOMP and CPC focus on filtering retrieval results or on sentence-level hard-text compression; and GMSA focuses on organizing the internal compression flow through group merging and layer alignment. MANTA does not rely on any prior question or specific retrieval path. With only a context-only Allocator, it produces a high-information-density soft compressed representation in a single pass. This representation can be used directly as a general-purpose key-value state and reused by the downstream model across arbitrarily many subsequent questions. Thus, MANTA shifts the design point from "question-bound text pruning" to a "multi-question reusable soft-representation contract."

Dear Reviewer, The authors have provided their responses to the raised concerns and revisions. Please review their responses and join the discussion before the end of the author-review discussion phase (Nov 18, AoE). Thanks.

Best, AC

Reviewer_g5XKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MANTA, a novel information-adaptive context compression framework. The authors argue that uniform compression ratios are suboptimal due to varying information density across context chunks. Building upon existing query-independent context compression methods, MANTA proposes the concept of "minimal sufficient allocation". It achieves this by integrating a Nesting Objective, which forces the compressed KV representations to be front-loaded (Matryoshka tokens), and an Elastic Allocator that dynamically predicts the optimal prefix length for each chunk based on the content and a global tolerance ϵ. Experiments demonstrate that MANTA outperforms dedicated per-ratio baselines across various compression ratios and model scales.

Critical strengths
  1. Clear Motivation: The paper clearly identifies the limitation of uniform compression ratios in existing query-independent context compression methods and proposes a well-motivated solution (information-adaptive compression).

  2. Extensive Experiments: The empirical evaluation is comprehensive, covering multiple model scales, different backbone families, and long-context scenarios (up to 32K tokens).

Critical weaknesses
  1. Confusing Ablation Setup in Table 6: The ablation settings in Table 6 are highly confusing. I noticed that the results of MANTA (Matryoshka ✗, Allocator ✗) are exactly the same as the SAC baseline in Table 1. Assuming SAC is trained strictly at a 64x ratio (8 tokens per 512-token chunk), but MANTA is trained on a ladder of {1, 2, 4, 8, 16, 32}, were both evaluated at 64x compression during decoding? A detailed breakdown of what exactly was changed or removed in each row of Table 6 is necessary.

  2. Missing Results at Lower Compression Ratios: Although MANTA claims to cover a compression range from 16x to 512x, the main experiments primarily report results at 64x and above. Performance at lower compression ratios is missing from the main text, which is important for a complete evaluation.

  3. The novelty in Section 3.3 might be highlighted more effectively: Aside from the prefix aspect, the compressor implementation in Section 3.3 (Matryoshka Token Encoding) seems largely identical to SAC. It may be worth considering moving this implementation detail to the Preliminaries or explicitly noting "following SAC," which would help better foreground the paper's actual novel contribution (adaptive compression).

Questions for authors
  1. In Table 6, what exactly do Matryoshka ✗ and Allocator ✗ represent? Specifically, how does this setup differ from both the full MANTA and the SAC baseline? Since the effectiveness of "adaptive compression" is the core claim of this paper, Table 6 is the most critical evidence. Please clarify the exact experimental protocol for this table.

  2. Since MANTA's maximum token allocation is 32 per chunk (which corresponds to 16x compression for a 512-token chunk), how does MANTA using all 32 Matryoshka tokens compare against SAC trained and inferred specifically at 16x? What about 4x?

5 comments
AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer g5XK,

We thank the reviewer for the constructive suggestions and careful reading. We respond to each point in order below, and all additional experiments and clarifications have been incorporated into the revised manuscript.

Q1 (W1 & Q1): Setup of the component ablation in Table 6.

We thank the reviewer for asking us to clarify the Table 6 ablation. All three rows share the same setup and differ only in which components are enabled. The Matryoshka ✗, Allocator ✗ row is the SAC-equivalent baseline with a fixed budget of k=8k=8. The Matryoshka ✓, Allocator ✗ row adds only the Matryoshka token and keeps k=8k=8. The Matryoshka ✓, Allocator ✓ row is full MANTA with adaptive allocation. Adding the Matryoshka token alone improves IID F1 by +1.49+1.49. Adding the Elastic Allocator on top improves IID F1 by a further +3.37+3.37 and OOD F1 by +2.59+2.59, while the average budget (mean-k≈7.84k\approx 7.84) stays below the fixed k=8k=8. This is because the Matryoshka token lets prefixes of different lengths share one compressed representation, and the Elastic Allocator then picks each prefix length based on the context and tolerance ϵ\epsilon, avoiding the redundancy or truncation of a fixed length. Together, the two components make multi-granularity representation effective.

Matryoshka tokenElastic AllocatorAllocationIID F1IID EMOOD F1OOD EM
✗✗k=8k=852.0140.9340.1030.59
✓✗k=8k=853.5042.4139.6630.58
✓✓mean-k≈7.84k\approx 7.8456.8745.1742.2532.92

Q2 (W2 & Q2): Performance at lower compression ratios and the full operating curve.

We thank the reviewer for highlighting the importance of evaluating the lower-compression part of MANTA's operating range.

1. MANTA's performance at lower compression ratios.

MANTA outperforms the corresponding SAC settings at all three lower-compression operating points reported below, including the comparison against SAC at 4×. The table compares MANTA with SAC at three adaptive operating points; each metric column reports the difference Δ=MANTA−SAC\Delta=\text{MANTA}-\text{SAC}.

The actual average compression ratios of MANTA at the three operating points (6.77×, 10.30×, 16.54×) are all higher than those of the corresponding SAC settings (4×, 8×, 16×). At the low-ratio operating point, MANTA's ratio is approximately 70% higher, so this constitutes a stricter cross-setting comparison for MANTA.

MANTA compression ratioSAC compression ratioΔ\Delta IID F1Δ\Delta IID EMΔ\Delta OOD F1Δ\Delta OOD EM
6.77×4×+0.45+0.01+0.86+0.17
10.30×8×+3.68+4.32+2.98+2.46
16.54×16×+4.50+4.98+2.70+2.77

At the low-ratio operating point, MANTA outperforms SAC despite compressing more aggressively. At the medium and high ratios, the advantage widens markedly, with IID/OOD F1 leading by up to +4.50+4.50/+2.98+2.98, respectively. These results support MANTA's effectiveness throughout the lower-compression part of the operating curve.

2. Matched comparison at 16×.

At a matched 16 ×\times compression ratio, the separately retrained low-rate MANTA using its full k=32 allocation outperforms the separately trained SAC-16 ×\times model. The low-rate MANTA achieves 68.34/57.36 IID token-F1/EM and 50.12/40.34 OOD, compared with 61.39/49.29 and 47.55/36.90 for SAC-16 ×\times.

We believe this advantage arises because a higher compression ratio imposes a more severe information bottleneck, under which uniform compression readily discards critical local semantics, whereas the Elastic Allocator can direct the limited token budget toward chunks with higher information density. Together with the lower-ratio comparisons above, the matched result supports the conclusion that MANTA provides a continuous adaptive operating curve without requiring a separate compressor for each compression level.

Q3 (W3): Attribution of the SAC substrate and MANTA's innovations in Section 3.3.

We thank the reviewer for the careful reading. The basic selected-context-token KV encoding in Section 3.3 follows SAC and is not a contribution of MANTA. We will move it to the Preliminaries and explicitly credit SAC. MANTA's contributions are built on top of this encoding: (1) a nested prefix ladder, in which different compression levels share one compressed representation, (2) a context-only Elastic Allocator governed by the tolerance ϵ\epsilon, which selects prefix lengths according to content, and (3) a dynamic-ϵ\epsilon rule for minimal-sufficient allocation per chunk. Unlike SAC's fixed-ratio uniform truncation, MANTA makes adaptive, content-dependent allocation the core of the method. We will revise Section 3.3 to state this boundary clearly.

AuthorsOpenReview ↗

Follow-up on Our Responses to Your Concerns

Dear Reviewer g5XK,

Thank you for your constructive feedback. In response to your concerns regarding the ablation setup, performance at lower compression ratios, and the clarity of our contributions, we have provided additional experiments and clarifications, which we hope can help address your concerns.

We would greatly appreciate it if you could take a moment to review our responses and let us know whether they adequately address your concerns. We would be happy to provide further clarification if needed.

Thank you again for your time and consideration.

Dear Reviewer, The authors have provided their responses to the raised concerns and revisions. Please review their responses and join the discussion before the end of the author-review discussion phase (Nov 18, AoE). Thanks.

Best, AC

Reviewer_N9iFOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MANTA, an information-adaptive context compressor that replaces a fixed allocation with the smallest per-chunk Matryoshka-token prefix predicted to satisfy a global excess-loss tolerance. A LoRA-adapted encoder learns nested prefixes, while a lightweight tolerance-conditioned Elastic Allocator selects an integer allocation from chunk content without using the downstream query. Across MRQA, needle-QA, multiple compression ratios, model scales, and two backbone families, MANTA generally improves over separately trained fixed-ratio compressors at comparable or slightly higher compression. The idea is useful and the empirical coverage is substantial, but the central reusable, query-independent claim is not yet fully established because the allocator is supervised by answer-conditioned decoder loss, the required per-chunk loss monotonicity is assumed rather than measured, and practical end-to-end efficiency is not reported.

Critical strengths
  1. The paper identifies a real limitation of fixed-ratio compression and gives it a precise operational formulation. The minimal sufficient allocation turns a vague notion of information density into a per-chunk decision controlled by one global tolerance.

  2. The architecture is coherent and practically attractive. Matryoshka prefixes let one encoder expose many operating points, and the Elastic Allocator converts the tolerance into content-dependent allocations without retraining a separate compressor for every ratio.

  3. The evaluation is broad for a context-compression paper, which includes full MRQA in-domain and out-of-domain splits across multi-scale backbones.

Critical weaknesses
  1. The query-independent deployment claim is not aligned with the supervision used to learn allocation. Equation 6 defines the oracle from decoder loss on the gold answer, so the minimal sufficient allocation can change when the question or answer changes even if the context chunk is identical.

  2. The monotonic-loss assumption underlying the oracle and fidelity metric is not established. Section 3.1 assumes that per-chunk decoder loss decreases with allocation, while the nesting objective supervises several prefixes but does not mathematically enforce monotonic loss for every chunk and answer.

Questions for authors
  1. How often does the same context or chunk receive different oracle allocations across its different questions, and how does a single context-only allocator resolve those conflicting targets? Please evaluate one compressed representation reused across multiple questions from the same context.

  2. What fraction of per-example, per-chunk loss curves violate monotonicity over k?

  3. Can the authors provide end-to-end Pareto curves covering encoding time, decoding latency, throughput, peak memory, calibration cost, and different numbers of queries per stored context, together with multi-seed uncertainty and broader long-context evaluations?

6 comments
AuthorsOpenReview ↗

Response Part I

Dear Reviewer N9iF,

We thank the reviewer for these insightful and incisive suggestions. We respond below to each sub-question in turn.

Q1 (W1 & Q1): Query-independent deployment and multi-question reuse.

We thank the reviewer for highlighting the tension between question-conditioned oracle allocations and a context-only allocator.

1. Variation in oracle allocations across questions.

Although 54.34% of samples receive different oracle kk values across questions, this does not create deployment conflicts: Stage 1 learns query- independent context compression, and Stage 2 only aligns it with QA supervision.

The practical impact of this variation is limited because the core text-compression capability is learned in Stage 1 pretraining: from the context content alone, MANTA learns which information is intrinsically important and how to preserve it in a nested compressed representation. Stage 2 fine-tuning only aligns this already-learned compression behavior with downstream QA and calibrates it using the oracle signal; it does not need to reproduce or memorize every question-specific allocation.

At deployment, the Elastic Allocator receives only the context content and the global tolerance ϵ\epsilon, and remains independent of any specific question or answer. It therefore produces one allocation vector and one compressed representation per context. We evaluate this directly on a multi-question context dataset by measuring the mean absolute error between the predicted kk and the average oracle kk:

Data distributionMANTASAC (k=8)SAC (k=16)
IID2.435.1812.46
OOD2.035.7012.89

The low prediction error shows that the context-only allocator captures shared information density rather than fitting each question-specific label separately. MANTA's error is 2.43 on IID and 2.03 on OOD, compared with 5.18/12.46 for the fixed-allocation SAC baselines on IID and 5.70/12.89 on OOD.

2. Reuse of one compressed representation across multiple questions.

MANTA remains effective when one encoded representation is reused, gaining +6.53 token-F1 and +6.37 EM over SAC on IID and retaining advantages of +0.35 token-F1 and +1.43 EM on OOD.

We further evaluate whether this compromise is useful in practice by reusing a single encoded representation across 31,686 questions:

MethodIID token-F1IID EMOOD token-F1OOD EM
MANTA50.2237.1740.3631.38
SAC43.6930.8140.0929.95
ICAE34.8222.3331.8022.81

These results confirm that query-dependent oracle variation can be used for training alignment while the resulting context-only representation remains effective across multiple questions.

Q2 (W2 & Q2): Monotonicity of the oracle loss.

We thank the reviewer for raising this important point.

Strict pointwise monotonicity does not hold: only 14.3% of chunks and 10.8% of examples are strictly monotonic. Throughout, we define and interpret monotonicity as a trend that emerges as kk increases, rather than as strict monotonicity for every chunk or example. Under this definition, the learned behavior achieves a monotonic-trend score of 67.6%.

The Matryoshka objective trains every prefix to be useful, but it does not mathematically enforce a decrease in decoder loss at every single step. Accordingly, we do not assume strict monotonicity. Instead, for each example--chunk curve, we take the length of its longest non-increasing subsequence divided by the total length, average over chunks within each example, and then macro-average over examples. This monotonic-trend score captures the dominant non-increasing direction while allowing local reversals.

We compute the score over the overall, IID, and OOD sets:

ScopeExamplesMonotonic-trend score
Overall67,85467.61%
IID58,22167.22%
OOD9,63369.95%

The consistent dominant non-increasing trend supports using nested prefixes as a practically smooth ladder and supports the oracle and fidelity analyses without overstating strict monotonicity.

AuthorsOpenReview ↗

Response Part II

Q3: End-to-end cost, multi-ϵ\epsilon Pareto curve, multi-seed stability, and long-context evaluation.

We thank the reviewer for the careful and comprehensive suggestions.

1. End-to-end cost across compression operating points.

We provide measurements for encoding time, allocation time, decoding throughput, KV-cache memory, and peak memory at the three compression-ratio operating points used in the main experiments.

On a single RTX 4090 GPU, we evaluated the end-to-end system overhead:

Method + RatioEncode (ms)Allocation (ms)Decode (output/s)KV (MB)Peak (GB)
Full-FT 1×29.320.0088.9717.265.41
ICAE 64×34.410.0061.221.2013.361
SAC 64×29.140.0087.950.2785.363
MANTA 65×26.384.1886.810.2265.366
ICAE 85×27.660.0073.900.3732.637
SAC 85×26.350.00126.690.2145.165
MANTA 91×26.803.12122.520.1535.168
ICAE 128×27.260.0073.020.2602.637
SAC 128×26.560.00124.620.1515.165
MANTA 134×26.943.12123.940.1035.168

MANTA adds only 3.12–4.18 ms of allocation overhead and achieves the lowest KV-cache footprint at all compression levels, with KV memory relatively further reduced by 18.7%–31.8% compared with SAC at similar or higher compression ratios.

2. Latency as the number of queries per stored context increases.

The one-time encoding and allocation cost is amortized as more questions reuse the same stored representation.

To evaluate different query loads, we selected 12 contexts containing at least 16 questions each. For each context, MANTA encoded and allocated the context once, stored the resulting representation, and then answered QQ questions sequentially:

Queries per stored contextMANTA adaptive (ms)SAC (k=8) (ms)
179.7588.62
2126.92134.55
4240.87246.43
8441.85434.45
16853.34817.13

This verifies that the deployment design supports different query loads for a single stored context while amortizing the one-time encoding and allocation cost across queries.

3. One-time calibration cost per checkpoint.

Calibration is a one-time cost per trained model: a single ϵ\epsilon sweep took 90.79 s on one RTX 4090. This cost is amortized over all later deployments of the model, so calibration remains a fixed, one-off cost rather than a per-query cost.

4. Transfer to natural long-context QA tasks.

MANTA transfers beyond the MRQA setting, achieving a three-task macro F1 of 20.82 at a 67.00× compression ratio on the evaluated long-context tasks.

In response to the request for broader long-context evaluation, we evaluated three natural long-document QA tasks under the 8B setting. Dense Full-FT serves as the uncompressed upper-bound reference, and the compression ratios of SAC and MANTA are listed according to the actual evaluation configurations:

MethodCompression ratioQasper2WikiMQAMuSiQueAverage F1
Full-FT1.00×22.9943.2636.9934.41
SAC64.00×14.2423.4510.4816.06
MANTA67.00×18.4427.4116.6220.82

MANTA's three-task macro F1 exceeds SAC by 4.76 points, indicating that the advantage is not confined to the MRQA setting.

5. Multi-seed uncertainty and training stability.

The three independently trained seeds show no substantial fluctuation, with IID sample standard deviations of only 1.0–1.2 and OOD sample standard deviations of only 1.8–2.1.

Finally, to verify training stability, we evaluated three independently trained random seeds on the MRQA subset:

Training seedIID token-F1IID EMOOD token-F1OOD EM
Run170.158.955.944.2
Run268.957.157.346.7
Run370.959.359.448.5
Mean ± sample SD70.0 ± 1.058.4 ± 1.257.5 ± 1.846.5 ± 2.1

These results indicate that MANTA's improvements are robust and do not stem from randomness.

AuthorsOpenReview ↗

Follow-up on Our Responses to Your Concerns

Dear Reviewer N9iF,

Thank you for your constructive feedback. In response to your concerns regarding query-independent compression, loss monotonicity, and practical efficiency, we have conducted additional experiments and analyses, which we hope can help address your concerns.

We would greatly appreciate it if you could take a moment to review our responses and let us know whether they adequately address your concerns. We would be happy to provide further clarification if needed.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.