acceptodds
Under review as a conference paper at ICLR 2027

AdaMem: Distortion-Guided Adaptive Context Compression

Abstract

Soft context compression improves long-context inference efficiency by replacing raw contexts with compact continuous memory representations. Existing methods, however, either use fixed compression budgets or adapt them according to indirect signals such as relevance, redundancy, or information density, without explicitly modeling the quality degradation induced by different compression ratios. We formulate context compression as a quality-constrained memory allocation problem and propose AdaMem, a distortion-guided framework that adaptively allocates memory budgets to individual context chunks according to their predicted compression sensitivity. Specifically, AdaMem employs a shared multi-ratio compressor to provide multiple compression budgets. Based on prefix representations, it predicts the distortion of each candidate ratio and selects the most aggressive ratio that satisfies a local predicted-distortion threshold. AdaMem further enables ratio selection before memory construction while reusing the prefix KV cache. Experiments on MRQA show that AdaMem consistently improves the quality–compression trade-off over fixed-ratio and adaptive baselines, reducing exported memory representations by 73.18% in-domain and 62.48% out-of-domain relative to fixed \(5\times\) compression, with consistent gains across 1B–8B model scales.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Soft Prompt Compression, Adaptive Compression, Distortion-Guided Routing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Q1seOpenReview ↗
Rating
2(−1)
Confidence
3(+1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

AdaMem formulates adaptive soft-context compression as per-chunk memory allocation guided by predicted compression distortion. A shared compressor supports 5×, 10×, 15×, and 34× output budgets. A lightweight predictor uses query-conditioned prefix states to estimate the answer-NLL change associated with each ratio, after which pathwise routing chooses the most aggressive ratio below a user-specified tolerance. Prefix KV reuse permits routing before construction of the exported memory. Experiments on MRQA and LongBench evaluate in-domain, out-of-domain, and cross-scale behavior.

Critical strengths
  1. The paper makes a clear and consequential distinction between content importance and compression sensitivity. Predicting ratio-specific distortion is better aligned with the allocation decision than relying only on relevance or information-density proxies.

  2. The evaluation is unusually careful. It includes same-compressor fixed-ratio controls, tolerance sweeps, in-domain and out-of-domain tests, full-context LongBench transfer, split-versus-joint validation, multiple model scales, and oracle analyses that separate the quality of the distortion signal from predictor error.

Critical weaknesses
  1. The strongest headline comparison combines gains from the shared compressor with gains from adaptive routing. In the controlled same-compressor comparison, routing mainly matches fixed 15× quality at 18.65× in-domain and fixed 10× quality at 13.33× out-of-domain. These are useful but materially smaller gains than the comparison against separately trained baseline systems. The paper should more clearly separate architecture/training improvements from allocation improvements.

  2. The reported efficiency measure counts only exported memory representations. Every candidate ratio still processes the same Kmax⁡=102K_{\max}=102 memory-token bank in the compressor; routing changes only the suffix exported to the downstream model. Therefore, the method saves downstream context and KV memory but apparently does not reduce compressor computation. End-to-end latency, peak memory, and throughput are needed to establish the practical efficiency benefit.

Questions for authors
  1. What are the end-to-end latency, peak-memory, and throughput results for fixed 5×, fixed 15×, and AdaMem, including prefix encoding, predictor execution, memory construction, and downstream generation?

  2. How well calibrated are predicted Δ\DeltaNLL values across ratios and domains? Please report per-ratio prediction errors or reliability curves and connect them to routing errors.

  3. Can the authors compare adaptive methods at matched output ratios using the same compressor or otherwise control for differences in compressor architecture and training data?

  4. Does independently constraining each chunk fail when several individually safe compression decisions interact adversely? The gap between the chunk oracle and exact task oracle suggests this may be important.

6 comments

Thank you for the suggestions on attribution, complete-pipeline costs, and distortion prediction. We provide same-compressor budget-matched controls, latency and throughput measurements, cross-domain calibration, and multi-chunk interaction diagnostics, each organized around the design, results, and analysis.

W1 / Q3. Allocation with the same compressor and matched budgets.

To separate compressor quality from routing, we fix the compressor and decoder checkpoints, prompts, and candidate ratios {5, 10, 15, 34} on all 67,854 official MRQA question–answer (QA) examples. The predictor is trained only on ID MRQA examples. We use the manuscript's first-reference ROUGE-1 F1/EM and equally weighted averages over the six datasets in each split.

Fixed15 and Fixed10 provide quality–budget references for ID and OOD, respectively. We then apply two budget controls: dataset-level controls preserve AdaMem's ratio counts and total exported vectors, while per-request controls additionally fix each example's vector budget and vary only the allocation across chunks. Random uses 10 routing seeds; BM25 uses query relevance. The DAST-style and COMI-style baselines adapt allocation based on attention plus causal-context NLL and relevance minus redundancy, respectively, to the same compressor.

Below, Δ=AdaMem−control. Confidence intervals use 10,000 paired context-cluster resamples within each dataset, with additional routing-seed resampling for Random, conditional on frozen checkpoints. Intervals are unadjusted; * denotes Holm-adjusted p<0.05 across eight ID/OOD F1 comparisons.

Quality and exported budgets relative to fixed-ratio references are:

SplitFixed reference: F1/EMAdaMem: F1/EMNominal ratio, fixed→adaptiveReduction in exported vectors (%)
IDFixed15: 66.31/49.6466.24/49.3415× → 18.65×19.55%
OODFixed10: 49.11/33.9249.03/33.9410× → 13.33×24.96%

Dataset-level budget-matched results are:

RouterID F1/EMOOD F1/EMΔF1 ID [95% CI]ΔF1 OOD [95% CI]
AdaMem66.24/49.3449.03/33.94——
Random (10 seeds)65.43/48.7248.61/33.57+0.81 [0.65, 0.98]*+0.42 [−0.02, 0.87]
BM2565.52/48.9249.01/33.80+0.72 [0.52, 0.92]*+0.02 [−0.53, 0.57]
DAST-style65.84/49.1649.25/34.30+0.41 [0.23, 0.58]*−0.21 [−0.75, 0.32]
COMI-style65.57/48.9248.81/33.66+0.67 [0.47, 0.86]*+0.22 [−0.35, 0.83]

Per-request budget-matched results are:

Per-request allocationID F1/EMOOD F1/EM
Random (10 seeds)66.06/49.1548.98/33.88
BM2566.27/49.3349.12/33.99
DAST-style66.29/49.3749.01/33.96
COMI-style66.09/49.1949.06/33.91

With the same compressor, AdaMem exports 19.55% fewer vectors than Fixed15 on ID and 24.96% fewer than Fixed10 on OOD, with F1 point estimates of 66.24 versus 66.31 and 49.03 versus 49.11, respectively. The budget-matched controls further identify ID gains of 0.41–0.81 F1 points over the four routing baselines, all significant after Holm correction; OOD superiority is not statistically established. Per-request scores are closer, indicating a contribution from budget allocation across requests, while single-chunk requests have no allocation freedom. We will report compressor quality and routing-specific allocation benefits separately and describe the fixed-ratio comparisons through their observed quality–budget operating points.

W2 / Q1. Complete-pipeline latency, memory, and throughput.

To complement exported-representation counts with deployment measurements, we evaluate 1,200 stratified MRQA and 100 untruncated LongBench requests on an RTX 3090 using BF16/SDPA, request batch size 1, five repeats, and population weighting. At every compression ratio, the compressor encodes 102 physical memory slots per chunk and exports the selected suffix. Full context uses the same base decoder. Peak memory is the maximum total allocated GPU memory observed over the measured requests.

Latency includes tokenization and transfer, prefix encoding, prediction and routing, memory construction, decoder prefill and generation, and detokenization, excluding model loading and scoring. The EOS setting stops at EOS or the output-length limit: 128 tokens for MRQA and the original task-specific limits for LongBench. Forced32 generates exactly 32 output tokens.

Throughput is computed from complete-run logs. Within each benchmark, Fixed5, Fixed15, and AdaMem share the checkpoint, request order, and physical RTX 3090 GPU. After excluding five warm-up requests, each of five timed passes processes 1,200 MRQA or 100 LongBench requests, computing 32 output tokens per request. N/T uses the actual request lists without population weighting; timing includes the complete pipeline, checks and I/O, and MRQA scoring, excluding loading, warm-up, and separate profiling. Its timing boundary therefore differs from the weighted inference latency. The five passes repeat serial processing at concurrency 1 on the same device. Stage times are measured through separate synchronized profiling to examine module costs and are not summed to reconstruct E2E latency.

Latency and peak memory are:

BenchmarkMethodEOS E2E (ms)EOS TTFT (ms)Forced32 E2E (ms)Peak GPU memory (GiB)
MRQAFull context2129.4238.13759.002.498
MRQAFixed5197.94116.95863.924.695
MRQAFixed15193.39116.47868.254.695
MRQAFixed34201.03117.18866.304.695
MRQAAdaMem202.60121.24866.624.733
LongBenchFull context2339.16820.861564.028.443
LongBenchFixed51751.601600.212346.105.877
LongBenchFixed151637.811543.702280.665.055
LongBenchFixed341600.401512.652278.614.827
LongBenchAdaMem1683.561594.372329.254.832

Forced32 serial throughput is:

BenchmarkMethodCompleted NMeasured T, sN/T, requests/s
MRQAFixed56,0005131.01.1694
MRQAFixed156,0005163.11.1621
MRQAAdaMem6,0005150.61.1649
LongBenchFixed55001155.60.4327
LongBenchFixed155001123.80.4449
LongBenchAdaMem5001147.50.4357

Separate stage measurements are:

AdaMem, separate profilingPrefix encoding (ms)Memory encoding (ms)Predictor (ms)
MRQA43.7146.634.81
LongBench703.03744.9377.46

On LongBench, AdaMem reduces peak memory from Fixed5's 5.877 to 4.832 GiB and EOS E2E latency from 1751.60 to 1683.56 ms. Its serial throughput is 1.1649 requests/s on MRQA and 0.4357 requests/s on LongBench, close to the fixed-ratio point estimates. Separate stage measurements identify prefix and memory encoding as the main costs, with a smaller predictor cost, suggesting targets for implementation optimization. These results characterize the complete-pipeline costs accompanying representation-budget compression, with the long-context peak-memory reduction particularly clear.

Q2. Calibration and routing errors across ratios and domains.

To examine how prediction errors affect routing, we use 3,000 ID and 3,000 OOD QA examples balanced by dataset and length, covering 8,668 chunks. Labels measure answer-NLL changes when one chunk's ratio changes and all others remain at Fixed5. Acceptance requires the distortion at every candidate ratio up to the tested ratio to be at most T=ln(1.05), matching Pathwise routing. We report per-ratio MAE, signed bias (predicted minus observed), acceptance precision and recall, and over-/under-compression relative to the measured-local selector.

We also compare the raw predictor, bias correction, and a q95 residual-margin rule. After excluding 31 calibration examples sharing 27 official-evaluation contexts, we fit rules on the remaining 2,969 internal ID examples and freeze them for both domains. Joint-NLL violations occur when measured complete-route NLL changes exceed the threshold; exported-vector savings are relative to Fixed5, with τ=5% throughout.

Per-ratio errors and acceptance are:

RatioMAE ID/OOD, nats/tokenBias ID/OOD (predicted−observed)ID acceptance precision/recall %OOD precision/recall %
10×0.0939 / 0.1658−0.0082 / +0.002182.47 / 90.4270.46 / 80.39
15×0.1217 / 0.2091−0.0279 / −0.007474.93 / 90.6661.02 / 75.08
34×0.1762 / 0.2881−0.0650 / −0.036966.70 / 88.7552.99 / 64.21

Routing errors relative to the measured-local selector are:

Relative to local distortion selectorOver-compression %Under-compression %
ID29.2211.34
OOD35.9622.74

Calibration results are:

Rule, τ=5%Joint-NLL violations ID/OOD %Exported-vector savings vs Fixed5, ID/OOD %
Raw predictor29.77 / 29.1369.84 / 60.81
Context-clean bias correction19.47 / 22.2052.46 / 40.50
q95 residual margin (all Fixed5)0 / 00 / 0

Underestimation at higher ratios accompanies over-compression, identifying false acceptance as a calibration priority. Bias correction fitted only on ID data reduces joint violations from 29.77% to 19.47% on ID and from 29.13% to 22.20% on OOD, while retaining 52.46% and 40.50% exported-vector savings. The more conservative q95 rule returns all examples to Fixed5, producing zero violations and zero savings. Bias correction therefore offers a more practical operating point between joint-distortion control and compression savings and demonstrates the effect of applying ID calibration to OOD examples.

Q4. Interactions between individually acceptable chunk decisions.

To distinguish accumulated marginal distortion from interactions, we define complete-route distortion D(R)=L(R)−L(all5), accumulated marginals A(R)=Σᵢdᵢ(rᵢ), and nonadditivity I(R)=D(R)−A(R), with threshold T=ln(1.05). The first experiment evaluates the previously analyzed evaluation set, disjoint from predictor training. We also select 1,188 examples whose original learned routes satisfy every measured local constraint, keeping those routes unchanged. Confidence intervals are estimated by bootstrapping contexts.

The second experiment exhaustively evaluates 13,552 routes on 400 distinct official contexts. ID contains 95 two-chunk, 94 three-chunk, and 11 four-chunk cases; OOD contains only two-chunk cases. The joint oracle minimizes exported-vector count subject to measured D≤T. We also fix each example's AdaMem-selected vector budget M and test whether any allocation satisfies the joint threshold.

Joint violations in the multi-chunk experiment are:

Selector / evaluation subsetJoint violations / QA examplesRate [95% CI]
True-label local Pathwise selector204 / 2,3438.71% [7.52%, 9.85%]
Learned router, all multi-chunk QA examples842 / 2,343—
Learned router, all true local constraints satisfied95 / 1,1888.00% [6.40%, 9.54%]

Of the 204 violations under the true-local selector, 121 have A≤T, directly identifying a nonadditive contribution to threshold crossings.

The exhaustive diagnostic gives:

DomainContextsJoint-NLL violations under the true-label local selectorCases with no feasible allocation at AdaMem's budget M
ID20021/20066/200
OOD2009/20064/200

With measured local labels, joint distortion remains within the threshold for most examples, but individual acceptability does not imply complete-route acceptability. Exhaustive evaluation also finds learned budgets for which no reallocation meets the joint threshold. These diagnostics motivate improving chunk allocation and, where needed, jointly adjusting the total budget and its distribution, providing evidence for interaction-aware routing or joint calibration. Generated-answer quality remains directly evaluated by F1; satisfying a joint NLL constraint alone does not guarantee F1 preservation.

Reviewer_G1XuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AdaMem, an adaptive soft context-compression framework that selects a per-chunk compression ratio from a discrete set {5×, 10×, 15×, 34×} using a predicted "distortion": the change in teacher-forced reference-answer NLL when only that chunk's ratio is changed relative to 5×. The system has three parts. (i) A single shared multi-ratio compressor (Llama-3.2-1B + LoRA) with a 102-token memory bank, where the ratio determines the logical RoPE position layout and how many trailing memory tokens are exported; it is trained with a ratio curriculum. (ii) A lightweight cross-attention predictor that reads the query-conditioned prefix states and regresses ΔNLL for the three non-reference ratios. (iii) A "Pathwise" rule that picks the largest ratio whose running-max predicted distortion stays below ln(1+τ). Because prefix tokens do not attend to memory tokens, the prefix can be encoded and cached before the ratio is chosen ("split execution"). On MRQA, AdaMem at τ=5% reaches 66.24/49.34 ID F1/EM at a nominal 18.65× and 49.03/33.94 OOD at 13.33×. These numbers exceed reproduced fixed-5× SAC while exporting 73%/62% fewer memory vectors. The paper also reports a τ sweep, 1B–8B scaling, zero-shot transfer to five LongBench QA sets, an attention-topology ablation, Split-vs-Joint consistency and non-inferiority tests, and oracle routing analyses.

Critical strengths

1.A clear, controllable formulation with a useful knob. Casting ratio selection as predicting a per-ratio ΔNLL profile and thresholding it, rather than classifying a ratio or redistributing a fixed global budget, is clean. It lets τ be changed at inference without retraining the predictor. The τ sweep (Table 4) shows a smooth trade-off from the same checkpoint. For example, τ=1% reaches 8.54× ID with only −0.30 F1 relative to the same compressor at fixed 5×, which is a practically attractive operating point.

2.Transparent and largely well-controlled internal analyses. The paper reports same-compressor fixed-ratio comparisons (Table 3), per-dataset breakdowns including where the method loses (TriviaQA, DROP, MuSiQue), chunk-level routing distributions, and oracle headroom analyses. It also frames the tolerance as a local surrogate rather than a downstream guarantee. This candour makes the contribution's real size assessable, which is also why my main concern below can be stated precisely.

Critical weaknesses

1.The headline improvements over baselines are mostly attributable to the compressor, not to distortion-guided routing, and the routing gain itself is small and not compared against appropriate routing baselines.

a.Same compressor, fixed 5×: 67.53/51.09 ID, 50.14/34.45 OOD (Table 3). Already beats every compressed baseline in Table 1, e.g. SAC 5× at 63.57/46.95 ID, 47.90/32.13 OOD. Adaptive routing (66.24/49.34 ID, 49.03/33.94 OOD) is below the paper's own fixed 5×. Table 11 reports a ratio-averaged F1 main effect of +4.53 points for bidirectional prefix encoding, averaged over the memory-attention condition. b.The paper lacks same-compressor, budget-matched routing baselines such as random routing, relevance heuristics, and DAST/COMI-style allocation. c.Cleanest control is Table 6 (5K holdout, same checkpoint, example-averaged): predicted router 69.70/53.78 is below fixed 5× at 70.78/55.66.

2.The efficiency benefit is asserted via exported-vector counts only, and may not hold end to end.

Encoder processes all 102 memory tokens plus the full prefix for every chunk (Sec. 3.4), the predictor adds overhead, the compressor and decoder use the same-size backbone, and query conditioning prevents reuse across queries. The holdout averages ~1.6 chunks/example (Table 19: 8,024/5,000); this count alone does not establish compute parity, but exported-vector savings likewise do not establish end-to-end savings. Full-FT leads by 5.27 F1 on ID (71.51 vs. 66.24) and 3.48 on OOD (52.51 vs. 49.03), although it is not a same-checkpoint control. No latency, FLOPs, or throughput is reported.

Questions for authors

1.Can you report same-compressor, budget-matched routing baselines, with seeds and CIs on ID/OOD averages? Specifically: (a) random routing with the same ratio histogram, (b) a query-relevance or attention-based heuristic router, and (c) DAST- or COMI-style allocation applied to your compressor. If AdaMem beats these by a clear margin, I would raise my score. 2.Please report end-to-end cost (wall-clock latency / FLOPs / peak memory) for full context, fixed-5×, fixed-34×, and AdaMem, including compressor and predictor time, on both MRQA and LongBench. Is there a deployment scenario where query-conditioned memories are amortised? 3.How accurate is the predictor itself (e.g., MAE / Spearman per ratio, and precision/recall of the "acceptable at τ" decision on held-out chunks)? The 4.21-F1 gap to the Chunk Distortion Oracle indicates substantial predictor headroom, although it is not itself a predictor-accuracy metric. Which errors (over- vs. under-compression) dominate? 4.On LongBench, AdaMem reaches nominal ratios 4.3–6.6 times the fixed-5× ratio (76.64%–84.88% fewer exported representations) yet scores higher on four of five datasets. What do fixed 15× and fixed 34× achieve there? If shorter inputs alone improve F1 for this 1B decoder, the comparison in Table 2 does not isolate routing. Please also add at least one external baseline and significance tests (N=150–200 per set).

Suggestions: 1.Both oracles select routes using the reference answer. Gains of the Chunk Distortion Oracle over fixed 5× (73.91 vs. 70.78) therefore partly reflect label-informed selection among alternatives, so I would soften the claim that measured distortions "support effective nonuniform allocation". The Exact Task Oracle (Table 6, F.1) goes further, searching routes to match or exceed fixed-5× reference-answer F1 on the same examples being scored. 2.Under exact arithmetic and the mask in Appendix B.2 (prefix never attends to memory), Split and Joint execution should be mathematically identical. The non-inferiority study therefore primarily validates implementation and finite-precision consistency. A one-line argument would clarify this, and "split execution" should be framed as an execution-scheduling mechanism rather than a separate quality contribution.

5 comments

We greatly appreciate your valuable feedback. We provide same-compressor budget-matched controls, complete-pipeline measurements, predictor diagnostics, and LongBench significance tests. Each response presents the design, results, and analysis, followed by clarification of the oracle and Split interpretations.

W1/Q1. Attribution and budget-matched routing.

To separate compressor improvements from allocation effects, we fix the compressor and decoder checkpoints, candidate ratios {5, 10, 15, 34}, and prompts, and compare routers on 58,221 ID and 9,633 OOD question–answer (QA) examples. The distortion predictor is trained only on ID MRQA examples.

We use two controls. Dataset-level controls preserve AdaMem's ratio counts and total exported vectors within each dataset, testing overall budget allocation. Per-request controls additionally fix each example's vector budget and change only its allocation across chunks. Random uses 10 routing seeds, and BM25 uses query–chunk relevance. The DAST-style and COMI-style baselines adapt allocation based on attention plus causal-context NLL and relevance minus redundancy, respectively, to the same compressor.

MRQA uses the manuscript's first-reference ROUGE-1 F1/EM and equally weighted averages over the six datasets in each split. Below, Δ=AdaMem−control is computed before rounding. Confidence intervals use 10,000 paired context-cluster resamples within each dataset, with additional seed resampling for Random, conditional on the trained checkpoints. Intervals are unadjusted; * denotes Holm-adjusted p<0.05 across eight ID/OOD F1 tests.

The dataset-level budget-matched results are:

RouterID F1/EMOOD F1/EMΔF1 ID [95% CI]ΔF1 OOD [95% CI]
AdaMem66.24/49.3449.03/33.94——
Random (10 seeds)65.43/48.7248.61/33.57+0.81 [0.65, 0.98]*+0.42 [−0.02, 0.87]
BM2565.52/48.9249.01/33.80+0.72 [0.52, 0.92]*+0.02 [−0.53, 0.57]
DAST-style65.84/49.1649.25/34.30+0.41 [0.23, 0.58]*−0.21 [−0.75, 0.32]
COMI-style65.57/48.9248.81/33.66+0.67 [0.47, 0.86]*+0.22 [−0.35, 0.83]

With each request's budget additionally fixed, the results are:

Per-request allocationID F1/EMOOD F1/EM
Random (10 seeds)66.06/49.1548.98/33.88
BM2566.27/49.3349.12/33.99
DAST-style66.29/49.3749.01/33.96
COMI-style66.09/49.1949.06/33.91

At the same dataset budget, AdaMem improves ID F1 by 0.41–0.81 points over the four allocation baselines, with all four gains significant after Holm correction. This directly identifies a contribution from routing. Scores are closer when each request's budget is fixed, indicating that the gains also involve budget allocation across requests; single-chunk requests have no allocation freedom. OOD differences are not statistically significant. We will attribute most of the overall Table 1 quality improvement to the shared compressor and report the routing-specific ID gains through these budget-matched controls.

W2/Q2. Complete-pipeline cost and deployment measurements.

To complement the paper's exported-representation counts with deployment costs, we measure 1,200 MRQA and 100 LongBench requests on an RTX 3090 using BF16/SDPA and request batch size 1, with five repeats and population weighting. LongBench inputs are complete and untruncated, with no OOM-based exclusions. The full-context control uses the same base decoder.

E2E timing includes tokenization and transfer, compression and prediction, prefill, generation, and detokenization, excluding model loading and scoring. The EOS setting stops at EOS or the output-length limit: 128 tokens for MRQA and the original task-specific limits for LongBench. Forced32 generates exactly 32 output tokens. All ratios process the full bank of 102 memory slots. Peak memory is the maximum allocated GPU memory. FLOPs estimate the dominant matrix multiplications at 1 MAC=2 FLOPs, including LoRA and the predictor but excluding elementwise operations, transfers, and cache traffic.

Stage costs are measured through separate synchronized profiling and are not added to reconstruct E2E latency. These cost measurements use bidirectional prefixes; the Q4 quality controls use causal prefixes and are reported separately.

BenchmarkMethodEOS E2E (ms)EOS TTFT (ms)Forced32 E2E (ms)Peak GPU memory (GiB)Forced32 compute (TFLOPs)
MRQAFull context2129.4238.13759.002.4981.233
MRQAFixed5197.94116.95863.924.6951.976
MRQAFixed15193.39116.47868.254.6951.758
MRQAFixed34201.03117.18866.304.6951.697
MRQAAdaMem202.60121.24866.624.7331.742
LongBenchFull context2339.16820.861564.028.44344.942
LongBenchFixed51751.601600.212346.105.87740.649
LongBenchFixed151637.811543.702280.665.05536.498
LongBenchFixed341600.401512.652278.614.82735.465
LongBenchAdaMem1683.561594.372329.254.83235.573

Separate stage measurements for AdaMem are:

AdaMem, separate profilingPrefix msMemory encoding msPredictor ms
MRQA43.7146.634.81
LongBench703.03744.9377.46

On LongBench, relative to Fixed5, AdaMem reduces peak memory from 5.877 to 4.832 GiB, Forced32 compute from 40.649 to 35.573 TFLOPs, and EOS E2E latency from 1751.60 to 1683.56 ms. MRQA E2E latency remains close to the fixed-ratio settings. These measurements characterize the practical costs accompanying fewer exported representations: lower peak memory and matrix-multiplication compute are observed on long contexts, while latency also depends on prefix encoding, memory construction, and generation settings.

Q3. Predictor accuracy and dominant routing errors.

To evaluate the predictor directly, we use a diagnostic sample balanced by dataset and length, containing 3,000 ID and 3,000 OOD QA examples and all associated chunks. MAE and Spearman correlation use raw ΔNLL; acceptance follows the Pathwise cumulative threshold at τ=5%. Summaries are computed over chunks in this balanced diagnostic sample.

We also compare learned routes with the selector using measured local distortions. Over-compression selects a higher compression ratio than that selector; under-compression selects a lower ratio. This identifies the dominant direction of allocation errors.

RatioMAE ID/OOD (nats/token)Spearman ID/OODID precision/recall %OOD precision/recall %
10×0.0939 / 0.16580.146 / 0.03482.47 / 90.4270.46 / 80.39
15×0.1217 / 0.20910.225 / 0.02574.93 / 90.6661.02 / 75.08
34×0.1762 / 0.28810.308 / 0.08466.70 / 88.7552.99 / 64.21
Relative to local distortion selectorOver-compression %Under-compression %
ID29.2211.34
OOD35.9622.74

At 10×, acceptance recall reaches 90.42% ID and 80.39% OOD, identifying many chunks that meet the local distortion threshold. MAE increases and acceptance precision declines at higher ratios; OOD ranking and acceptance recall are weaker than on ID. Over-compression is more frequent than under-compression relative to the measured-local selector. These diagnostics prioritize controlling false acceptance at high ratios and improving cross-domain calibration.

Q4. LongBench fixed-ratio controls and significance.

To evaluate adaptive allocation on long-context tasks, we compare Fixed5, Fixed15, Fixed34, and AdaMem on all 950 official examples. Fixed and adaptive settings share the compressor, prompts, and generation settings and use causal prefixes. We use official normalized multi-answer token F1 and macro-average across tasks. COMI retains its native modules and decoder adaptation at its trained 5× and 15× ratios, providing system-level comparisons.

Confidence intervals use 10,000 paired whole-context bootstrap resamples, and we report permutation-test p values. Holm correction is applied separately to 25 task comparisons, 15 fixed-only task comparisons, and five macro comparisons. NarrativeQA's 200 QA examples share 20 contexts; resampling preserves this dependence. All differences are computed before rounding.

Per-task F1 and macro averages are:

DatasetNContextsFixed5Fixed15Fixed34AdaMemCOMI5COMI15
2WikiMultiHopQA20020022.0122.7420.2424.0521.4322.51
MultiFieldQA-en15011219.5216.7517.4520.5414.9211.03
MuSiQue20020013.1510.1411.3210.429.988.51
NarrativeQA2002011.5011.3811.6512.386.404.94
Qasper20014812.7515.2216.1615.829.2612.42
Task macro95068015.7915.2515.3616.6412.4011.88

Macro-average differences and significance tests are:

ControlΔF1 [95% CI]Permutation pHolm p (5 macro tests)
Fixed5+0.85 [−0.83, 2.55]0.31650.3165
Fixed15+1.40 [−0.25, 3.06]0.10090.2018
Fixed34+1.28 [−0.02, 2.59]0.05840.1752
COMI5+4.24 [2.37, 6.18]0.00010.0005
COMI15+4.76 [2.79, 6.79]0.00010.0005

The per-task paired tests below use Δ=AdaMem−control:

DatasetControlΔF1 [95% CI]Permutation pHolm p (25 task tests)Holm p (15 fixed-only tests)
2WikiMultiHopQAFixed5+2.04 [−2.87, 7.03]0.42301.00001.0000
2WikiMultiHopQAFixed15+1.31 [−3.31, 5.96]0.58221.00001.0000
2WikiMultiHopQAFixed34+3.82 [0.01, 7.74]0.05480.98630.7123
2WikiMultiHopQACOMI5+2.62 [−3.39, 8.54]0.39521.0000—
2WikiMultiHopQACOMI15+1.54 [−4.49, 7.49]0.61931.0000—
MultiFieldQA-enFixed5+1.02 [−1.98, 4.06]0.51021.00001.0000
MultiFieldQA-enFixed15+3.79 [0.35, 7.24]0.02810.53790.4035
MultiFieldQA-enFixed34+3.09 [0.52, 6.01]0.02690.53790.4035
MultiFieldQA-enCOMI5+5.62 [1.90, 9.36]0.00340.0714—
MultiFieldQA-enCOMI15+9.51 [5.79, 13.36]0.00010.0025—
MuSiQueFixed5−2.73 [−6.97, 1.54]0.21451.00001.0000
MuSiQueFixed15+0.28 [−3.22, 3.76]0.87381.00001.0000
MuSiQueFixed34−0.90 [−3.47, 1.73]0.50091.00001.0000
MuSiQueCOMI5+0.44 [−3.40, 4.33]0.83641.0000—
MuSiQueCOMI15+1.91 [−1.89, 5.91]0.34181.0000—
NarrativeQAFixed5+0.87 [−1.72, 3.33]0.51411.00001.0000
NarrativeQAFixed15+1.00 [−2.21, 4.14]0.56671.00001.0000
NarrativeQAFixed34+0.72 [−1.77, 3.62]0.67231.00001.0000
NarrativeQACOMI5+5.98 [2.89, 9.95]0.00040.0096—
NarrativeQACOMI15+7.44 [3.25, 12.15]0.00150.0330—
QasperFixed5+3.07 [−0.19, 6.38]0.06711.00000.8051
QasperFixed15+0.60 [−2.90, 4.21]0.74091.00001.0000
QasperFixed34−0.34 [−2.80, 2.10]0.79601.00001.0000
QasperCOMI5+6.56 [2.83, 10.45]0.00080.0184—
QasperCOMI15+3.40 [−0.37, 7.27]0.08551.0000—

AdaMem's macro F1 of 16.64 exceeds the point estimates of all three fixed-ratio settings; these fixed-ratio differences are not statistically significant. In the system-level comparisons, gains over COMI5 and COMI15 are 4.24 and 4.76 F1 points, both with Holm p=0.0005. Per-task improvements on MultiFieldQA-en over COMI15 and on NarrativeQA over both COMI settings also pass the corresponding Holm tests. Fixed5 remains stronger on MuSiQue, showing variation in performance across tasks. We will present fixed-ratio ablations and COMI system comparisons separately so that aggregate gains, task variation, and significance are clearly connected.

Reviewer_mmhMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AdaMem, which is a distortion-guided adaptive memory framework for context compression. It observes that chunks vary widely in compression sensitivity, so it predicts each chunk's answer-NLL change at candidate ratios (5×, 10×, 15×, 34×) from query-conditioned prefix states, then routes each chunk to the largest ratio whose predicted distortion stays within a user tolerance.

Critical strengths
  1. The paper identifies chunk-level heterogeneity in compression sensitivity, formalizes per-chunk ratio selection under a local distortion constraint, and reports oracle gaps that quantify remaining headroom rather than obscuring it.
  2. AdaMem beats the strongest reproduced soft-compression baseline in F1 and EM.
Critical weaknesses
  1. Distortion labels are measured by perturbing one chunk at a time, ignoring cross-chunk interference when multiple chunks are compressed jointly.
  2. AdaMem's routing relies on a local answer-NLL surrogate that does not guarantee downstream quality.
  3. The predictor is trained on reference-answer NLL labels requiring gold answers, limiting applicability to settings where supervision is scarce.
Questions for authors
  1. How do you split chunks from the text? (any metrics?)
4 comments

Thank you for recognizing the differences in compression sensitivity across chunks and the headroom identified by the oracle analysis. We provide additional experiments on joint compression, downstream allocation, and answer-supervision budgets, followed by the chunking implementation.

Evaluation scope. The experiments use the previously analyzed evaluation set of 5,000 in-domain (ID) QA examples. This set shares no contexts with the predictor training data and no QA examples with the compressor SFT data.

W1. Cross-chunk interference.

To test how single-chunk distortion relates to joint compression, we evaluated 16,624 distinct routes on 2,343 multi-chunk QA examples. To obtain each local label, we change one chunk's ratio from 5× to a candidate ratio while keeping all other chunks at 5×. For a complete route R, we define joint distortion D=L(R)−L(all5), accumulated marginal distortion A=Σᵢdᵢ(rᵢ), and the nonadditive component I=D−A, with threshold T=ln(1.05).

We compare the local Pathwise selector using measured labels, the learned router, and a subset of 1,188 examples whose learned routes satisfy every local constraint under measured marginal distortions. This subset is selected offline, with the original routes unchanged. Confidence intervals are estimated by bootstrapping contexts.

Selector / evaluation subsetJoint-NLL exceedances (D>T) / QA examplesRate [95% context-bootstrap CI]
True-label local Pathwise selector204 / 2,3438.71% [7.52%, 9.85%]
Learned router, all multi-chunk QA examples842 / 2,34335.94%
Learned router, all true local constraints satisfied95 / 1,1888.00% [6.40%, 9.54%]

Among the 204 joint-threshold exceedances under the true-label selector, 121 have A≤T: accumulated marginal distortion alone remains within the threshold.

With measured local labels, joint distortion remains within the threshold for most multi-chunk examples; the 121 cases above directly identify a contribution from cross-chunk nonadditivity. This diagnostic separates marginal accumulation from interactions and provides evidence for improving joint routing.

W2. Answer NLL and downstream allocation benefits.

To test whether NLL-guided allocation improves answer quality, we hold the compressor fixed and preserve each QA example's multiset of assigned compression ratios and exported-memory budget. The multiset preserves how often each ratio is used. The control uniformly permutes these ratios across chunks within the example, and we compute its exact expected score.

Route selection does not use reference answers. We report both the full evaluation set and the 413 examples with allocation freedom: multiple enumerated routes share their original AdaMem budget, which in this evaluation set is equivalent to the learned route using at least two distinct ratios. Subset selection uses route structure, without F1/EM filtering. We also report normalized multi-answer QA-F1 alongside the original scoring protocol.

Evaluation subsetMetricAdaMem−permutation, points [95% CI]
All 5,000 QA examplesFirst-reference ROUGE-1 F1+0.252 [0.109, 0.393]
413 QA examples with allocation freedomFirst-reference ROUGE-1 F1+3.046 [1.381, 4.841]
All 5,000 QA examplesNormalized multi-answer QA-F1+0.175 [0.023, 0.310]

With the compressor and each example's budget fixed, AdaMem improves ROUGE-1 F1 by 0.252 points over the full evaluation set and by 3.046 points where reallocation is possible. The positive normalized multi-answer QA-F1 result also shows a benefit beyond first-reference scoring. These direct quality comparisons on ID examples support NLL as a trainable allocation signal and complement the budget-matched controls on the full MRQA evaluation.

W3. Additional answer supervision.

To assess the router's need for additional human-annotated answers, we vary the gold-answer budget and add a router supervised entirely by pseudo answers. All conditions use the same frozen compressor, predictor architecture, evaluation set, and tolerance τ=0.05. Human-answer budgets include training and validation examples; calibration uses no examples with gold answers.

The frozen 5× model generates pseudo answers for the same 5000 training QA examples used in the gold-answer condition. Both distortion targets and early stopping use pseudo answers. All conditions retain the compressor's existing SFT supervision from 320K QA examples, so the comparison concerns additional human answers for the router.

Router labelsQA examples requiring additional human-annotated answersROUGE-1 F1Raw EMReduction in exported memory representations relative to Fixed5 (%)
Gold1,00070.21554.52053.86%
Gold5,00069.94854.24062.90%
Gold10,00069.84454.16068.28%
Gold24,00069.69853.78071.82%
Pseudo answers069.77854.06066.49%

For the two routers trained and validated on the same 5K examples, the evaluation differences between pseudo-answer and gold-answer supervision are:

Pseudo−gold using the same 5K training and validation QA examplesDifference [95% CI]
ROUGE-1 F1, points−0.170 [−0.561, +0.229]
Memory saving, percentage points+3.594 [3.024, 4.170]

Without additional human-annotated answers, the pseudo-answer variant achieves 69.778 ROUGE-1 F1 and 54.060 raw EM while exporting 66.49% fewer memory representations than Fixed5. Relative to the 5K gold-answer condition, its F1 point estimate differs by −0.170 points and its representation savings increase by 3.594 percentage points. This provides an option with lower annotation cost for the quality–budget trade-off at the same tolerance. At deployment, the router still uses only context and question features.

Q1. Chunk splitting.

We tokenize each context with the backbone tokenizer and split it by token position into consecutive, non-overlapping chunks of up to 510 tokens. This follows SAC's sub-context length in Section 4.1 and keeps chunk granularity consistent in the comparison. No semantic-similarity metric or sentence-boundary score determines the boundaries.Full chunks contain 510 tokens. A shorter final chunk is retained at its actual length without padding. Nominal ratios of 5×, 10×, 15×, and 34× export 102, 51, 34, and 15 memory representations per chunk, respectively, including the shorter final chunk. Position-based chunking is reproducible and allows full chunks of equal length to be batched for parallel prefill. As future work, we plan to investigate 512-token chunks with power-of-two compression ratios: for example, {4, 8, 16, 32} would produce {128, 64, 32, 16} memory representations per full chunk, forming a logarithmically spaced hierarchy of budgets.

Dear Reviewer, The authors have provided their responses to the raised concerns and revisions. Please review their responses and join the discussion before the end of the author-review discussion phase (Nov 18, AoE). Thanks.

Best, AC

Thanks for your response.

I have read through your additional results. After consideration, I decided to maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.