- Rating
- 2(−1)
- Confidence
- 3(+1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
AdaMem formulates adaptive soft-context compression as per-chunk memory allocation guided by predicted compression distortion. A shared compressor supports 5×, 10×, 15×, and 34× output budgets. A lightweight predictor uses query-conditioned prefix states to estimate the answer-NLL change associated with each ratio, after which pathwise routing chooses the most aggressive ratio below a user-specified tolerance. Prefix KV reuse permits routing before construction of the exported memory. Experiments on MRQA and LongBench evaluate in-domain, out-of-domain, and cross-scale behavior.
Critical strengths
-
The paper makes a clear and consequential distinction between content importance and compression sensitivity. Predicting ratio-specific distortion is better aligned with the allocation decision than relying only on relevance or information-density proxies.
-
The evaluation is unusually careful. It includes same-compressor fixed-ratio controls, tolerance sweeps, in-domain and out-of-domain tests, full-context LongBench transfer, split-versus-joint validation, multiple model scales, and oracle analyses that separate the quality of the distortion signal from predictor error.
Critical weaknesses
-
The strongest headline comparison combines gains from the shared compressor with gains from adaptive routing. In the controlled same-compressor comparison, routing mainly matches fixed 15× quality at 18.65× in-domain and fixed 10× quality at 13.33× out-of-domain. These are useful but materially smaller gains than the comparison against separately trained baseline systems. The paper should more clearly separate architecture/training improvements from allocation improvements.
-
The reported efficiency measure counts only exported memory representations. Every candidate ratio still processes the same memory-token bank in the compressor; routing changes only the suffix exported to the downstream model. Therefore, the method saves downstream context and KV memory but apparently does not reduce compressor computation. End-to-end latency, peak memory, and throughput are needed to establish the practical efficiency benefit.
Questions for authors
-
What are the end-to-end latency, peak-memory, and throughput results for fixed 5×, fixed 15×, and AdaMem, including prefix encoding, predictor execution, memory construction, and downstream generation?
-
How well calibrated are predicted NLL values across ratios and domains? Please report per-ratio prediction errors or reliability curves and connect them to routing errors.
-
Can the authors compare adaptive methods at matched output ratios using the same compressor or otherwise control for differences in compressor architecture and training data?
-
Does independently constraining each chunk fail when several individually safe compression decisions interact adversely? The gap between the chunk oracle and exact task oracle suggests this may be important.
6 comments
W2 / Q1. Complete-pipeline latency, memory, and throughput.
To complement exported-representation counts with deployment measurements, we evaluate 1,200 stratified MRQA and 100 untruncated LongBench requests on an RTX 3090 using BF16/SDPA, request batch size 1, five repeats, and population weighting. At every compression ratio, the compressor encodes 102 physical memory slots per chunk and exports the selected suffix. Full context uses the same base decoder. Peak memory is the maximum total allocated GPU memory observed over the measured requests.
Latency includes tokenization and transfer, prefix encoding, prediction and routing, memory construction, decoder prefill and generation, and detokenization, excluding model loading and scoring. The EOS setting stops at EOS or the output-length limit: 128 tokens for MRQA and the original task-specific limits for LongBench. Forced32 generates exactly 32 output tokens.
Throughput is computed from complete-run logs. Within each benchmark, Fixed5, Fixed15, and AdaMem share the checkpoint, request order, and physical RTX 3090 GPU. After excluding five warm-up requests, each of five timed passes processes 1,200 MRQA or 100 LongBench requests, computing 32 output tokens per request. N/T uses the actual request lists without population weighting; timing includes the complete pipeline, checks and I/O, and MRQA scoring, excluding loading, warm-up, and separate profiling. Its timing boundary therefore differs from the weighted inference latency. The five passes repeat serial processing at concurrency 1 on the same device. Stage times are measured through separate synchronized profiling to examine module costs and are not summed to reconstruct E2E latency.
Latency and peak memory are:
| Benchmark | Method | EOS E2E (ms) | EOS TTFT (ms) | Forced32 E2E (ms) | Peak GPU memory (GiB) |
|---|---|---|---|---|---|
| MRQA | Full context | 2129.42 | 38.13 | 759.00 | 2.498 |
| MRQA | Fixed5 | 197.94 | 116.95 | 863.92 | 4.695 |
| MRQA | Fixed15 | 193.39 | 116.47 | 868.25 | 4.695 |
| MRQA | Fixed34 | 201.03 | 117.18 | 866.30 | 4.695 |
| MRQA | AdaMem | 202.60 | 121.24 | 866.62 | 4.733 |
| LongBench | Full context | 2339.16 | 820.86 | 1564.02 | 8.443 |
| LongBench | Fixed5 | 1751.60 | 1600.21 | 2346.10 | 5.877 |
| LongBench | Fixed15 | 1637.81 | 1543.70 | 2280.66 | 5.055 |
| LongBench | Fixed34 | 1600.40 | 1512.65 | 2278.61 | 4.827 |
| LongBench | AdaMem | 1683.56 | 1594.37 | 2329.25 | 4.832 |
Forced32 serial throughput is:
| Benchmark | Method | Completed N | Measured T, s | N/T, requests/s |
|---|---|---|---|---|
| MRQA | Fixed5 | 6,000 | 5131.0 | 1.1694 |
| MRQA | Fixed15 | 6,000 | 5163.1 | 1.1621 |
| MRQA | AdaMem | 6,000 | 5150.6 | 1.1649 |
| LongBench | Fixed5 | 500 | 1155.6 | 0.4327 |
| LongBench | Fixed15 | 500 | 1123.8 | 0.4449 |
| LongBench | AdaMem | 500 | 1147.5 | 0.4357 |
Separate stage measurements are:
| AdaMem, separate profiling | Prefix encoding (ms) | Memory encoding (ms) | Predictor (ms) |
|---|---|---|---|
| MRQA | 43.71 | 46.63 | 4.81 |
| LongBench | 703.03 | 744.93 | 77.46 |
On LongBench, AdaMem reduces peak memory from Fixed5's 5.877 to 4.832 GiB and EOS E2E latency from 1751.60 to 1683.56 ms. Its serial throughput is 1.1649 requests/s on MRQA and 0.4357 requests/s on LongBench, close to the fixed-ratio point estimates. Separate stage measurements identify prefix and memory encoding as the main costs, with a smaller predictor cost, suggesting targets for implementation optimization. These results characterize the complete-pipeline costs accompanying representation-budget compression, with the long-context peak-memory reduction particularly clear.
Q2. Calibration and routing errors across ratios and domains.
To examine how prediction errors affect routing, we use 3,000 ID and 3,000 OOD QA examples balanced by dataset and length, covering 8,668 chunks. Labels measure answer-NLL changes when one chunk's ratio changes and all others remain at Fixed5. Acceptance requires the distortion at every candidate ratio up to the tested ratio to be at most T=ln(1.05), matching Pathwise routing. We report per-ratio MAE, signed bias (predicted minus observed), acceptance precision and recall, and over-/under-compression relative to the measured-local selector.
We also compare the raw predictor, bias correction, and a q95 residual-margin rule. After excluding 31 calibration examples sharing 27 official-evaluation contexts, we fit rules on the remaining 2,969 internal ID examples and freeze them for both domains. Joint-NLL violations occur when measured complete-route NLL changes exceed the threshold; exported-vector savings are relative to Fixed5, with τ=5% throughout.
Per-ratio errors and acceptance are:
| Ratio | MAE ID/OOD, nats/token | Bias ID/OOD (predicted−observed) | ID acceptance precision/recall % | OOD precision/recall % |
|---|---|---|---|---|
| 10× | 0.0939 / 0.1658 | −0.0082 / +0.0021 | 82.47 / 90.42 | 70.46 / 80.39 |
| 15× | 0.1217 / 0.2091 | −0.0279 / −0.0074 | 74.93 / 90.66 | 61.02 / 75.08 |
| 34× | 0.1762 / 0.2881 | −0.0650 / −0.0369 | 66.70 / 88.75 | 52.99 / 64.21 |
Routing errors relative to the measured-local selector are:
| Relative to local distortion selector | Over-compression % | Under-compression % |
|---|---|---|
| ID | 29.22 | 11.34 |
| OOD | 35.96 | 22.74 |
Calibration results are:
| Rule, τ=5% | Joint-NLL violations ID/OOD % | Exported-vector savings vs Fixed5, ID/OOD % |
|---|---|---|
| Raw predictor | 29.77 / 29.13 | 69.84 / 60.81 |
| Context-clean bias correction | 19.47 / 22.20 | 52.46 / 40.50 |
| q95 residual margin (all Fixed5) | 0 / 0 | 0 / 0 |
Underestimation at higher ratios accompanies over-compression, identifying false acceptance as a calibration priority. Bias correction fitted only on ID data reduces joint violations from 29.77% to 19.47% on ID and from 29.13% to 22.20% on OOD, while retaining 52.46% and 40.50% exported-vector savings. The more conservative q95 rule returns all examples to Fixed5, producing zero violations and zero savings. Bias correction therefore offers a more practical operating point between joint-distortion control and compression savings and demonstrates the effect of applying ID calibration to OOD examples.
Q4. Interactions between individually acceptable chunk decisions.
To distinguish accumulated marginal distortion from interactions, we define complete-route distortion D(R)=L(R)−L(all5), accumulated marginals A(R)=Σᵢdᵢ(rᵢ), and nonadditivity I(R)=D(R)−A(R), with threshold T=ln(1.05). The first experiment evaluates the previously analyzed evaluation set, disjoint from predictor training. We also select 1,188 examples whose original learned routes satisfy every measured local constraint, keeping those routes unchanged. Confidence intervals are estimated by bootstrapping contexts.
The second experiment exhaustively evaluates 13,552 routes on 400 distinct official contexts. ID contains 95 two-chunk, 94 three-chunk, and 11 four-chunk cases; OOD contains only two-chunk cases. The joint oracle minimizes exported-vector count subject to measured D≤T. We also fix each example's AdaMem-selected vector budget M and test whether any allocation satisfies the joint threshold.
Joint violations in the multi-chunk experiment are:
| Selector / evaluation subset | Joint violations / QA examples | Rate [95% CI] |
|---|---|---|
| True-label local Pathwise selector | 204 / 2,343 | 8.71% [7.52%, 9.85%] |
| Learned router, all multi-chunk QA examples | 842 / 2,343 | — |
| Learned router, all true local constraints satisfied | 95 / 1,188 | 8.00% [6.40%, 9.54%] |
Of the 204 violations under the true-local selector, 121 have A≤T, directly identifying a nonadditive contribution to threshold crossings.
The exhaustive diagnostic gives:
| Domain | Contexts | Joint-NLL violations under the true-label local selector | Cases with no feasible allocation at AdaMem's budget M |
|---|---|---|---|
| ID | 200 | 21/200 | 66/200 |
| OOD | 200 | 9/200 | 64/200 |
With measured local labels, joint distortion remains within the threshold for most examples, but individual acceptability does not imply complete-route acceptability. Exhaustive evaluation also finds learned budgets for which no reallocation meets the joint threshold. These diagnostics motivate improving chunk allocation and, where needed, jointly adjusting the total budget and its distribution, providing evidence for interaction-aware routing or joint calibration. Generated-answer quality remains directly evaluated by F1; satisfying a joint NLL constraint alone does not guarantee F1 preservation.
Thank you for the suggestions on attribution, complete-pipeline costs, and distortion prediction. We provide same-compressor budget-matched controls, latency and throughput measurements, cross-domain calibration, and multi-chunk interaction diagnostics, each organized around the design, results, and analysis.
W1 / Q3. Allocation with the same compressor and matched budgets.
To separate compressor quality from routing, we fix the compressor and decoder checkpoints, prompts, and candidate ratios {5, 10, 15, 34} on all 67,854 official MRQA question–answer (QA) examples. The predictor is trained only on ID MRQA examples. We use the manuscript's first-reference ROUGE-1 F1/EM and equally weighted averages over the six datasets in each split.
Fixed15 and Fixed10 provide quality–budget references for ID and OOD, respectively. We then apply two budget controls: dataset-level controls preserve AdaMem's ratio counts and total exported vectors, while per-request controls additionally fix each example's vector budget and vary only the allocation across chunks. Random uses 10 routing seeds; BM25 uses query relevance. The DAST-style and COMI-style baselines adapt allocation based on attention plus causal-context NLL and relevance minus redundancy, respectively, to the same compressor.
Below, Δ=AdaMem−control. Confidence intervals use 10,000 paired context-cluster resamples within each dataset, with additional routing-seed resampling for Random, conditional on frozen checkpoints. Intervals are unadjusted; * denotes Holm-adjusted p<0.05 across eight ID/OOD F1 comparisons.
Quality and exported budgets relative to fixed-ratio references are:
Dataset-level budget-matched results are:
Per-request budget-matched results are:
With the same compressor, AdaMem exports 19.55% fewer vectors than Fixed15 on ID and 24.96% fewer than Fixed10 on OOD, with F1 point estimates of 66.24 versus 66.31 and 49.03 versus 49.11, respectively. The budget-matched controls further identify ID gains of 0.41–0.81 F1 points over the four routing baselines, all significant after Holm correction; OOD superiority is not statistically established. Per-request scores are closer, indicating a contribution from budget allocation across requests, while single-chunk requests have no allocation freedom. We will report compressor quality and routing-specific allocation benefits separately and describe the fixed-ratio comparisons through their observed quality–budget operating points.