- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
LMI is a learned hard-context compressor that partitions a long context into windows, appends latent tokens, and exchanges latent information across windows. It then predicts token-level BIOE labels, aggregates these into sentence-level importance scores, and retains complete sentences under a target budget. The system is trained using QA data and automatically mined summarization supervision, with a 7B compressor and a distilled 0.5B variant. Evaluation covers synthetic retrieval, LoCoMo, LongBench, serving efficiency, and a 51-issue SWE-bench agent experiment.
Critical strengths
-
The paper tackles a genuine limitation of window-independent compression: evidence needed to judge relevance can be distributed across distant windows. The large improvement on the interaction-sensitive synthetic task and the explicit interaction ablation support the usefulness of some form of cross-window communication.
-
Retaining complete sentences is a sensible practical design that avoids fragmented prompts. The paper also attempts unusually deployment-oriented evaluation, including a smaller distilled compressor, serving measurements, and an agent pipeline.
Critical weaknesses
-
The cross-window mechanism is not specified clearly enough to establish either its behavior or complexity. In the early layers, corresponding latent slots are averaged across all windows and the same average is broadcast back, which removes window identity. The paper then states that the final quarter uses bidirectional interaction among latent memories, but does not give the precise attention mask or complexity. If all latent tokens attend globally, this component is quadratic in the number of windows; if it is only mean aggregation, the description of continuous bidirectional interaction is misleading. The claimed near-linear complexity therefore cannot be verified from the current specification.
-
Several decisive empirical results lack controls needed for a reliable conclusion. The strongest retrieval result uses a newly constructed benchmark without sufficient dataset-size or uncertainty information; compression budgets and operating-point selection are not fully described; the closest CPRS baseline uses different training data; and the SWE-bench result contains only 51 unspecified issues, where the reported gain is four additional successes without statistical uncertainty. These limitations make it difficult to attribute the gains specifically to LMI or assess generalization.
Questions for authors
-
Please provide exact tensor shapes and attention masks for every interaction layer. Do latent tokens retain window identity after mean broadcasting, and does the final bidirectional stage attend over all latent tokens? What is its exact asymptotic and measured cost?
-
How many examples are in each custom needle benchmark, how are train/test entities and templates separated, and are the differences statistically significant?
-
How were the 51 SWE-bench issues selected? Please report the full issue list, paired outcomes, confidence intervals or a paired significance test, and results on an established SWE-bench subset.
-
Can the authors compare LMI with CPRS under identical training data, labels, sentence-selection policy, and compression ratio? The existing system-level comparison does not isolate the contribution of latent interaction.
-
How are total time and average task latency in Table 4 related under 32 concurrent requests, and do the results hold across context lengths and generators?
8 comments
Response to Questions
Response to Question 1
- We have added a detailed explanation of the model architecture to the revised manuscript (Appendix A.1.1, lines 724–750). To specify the tensor shapes, consider the 7B model with latent length , window length , and a 32,768 document (M=4 windows). Each window’s input sequence is [query; X_m ∈ R^{8K×d}; Z_m ∈ R^{48×d}] and the four windows are prefilled in parallel along the batch dimension. The details are as follows:
- First three-quarters of the layers (layers 0–20): Each layer uses a standard, within-window lower-triangular causal attention mask of shape , with no attention between windows. The latent tokens causally absorb information from the query and context in their respective windows.
- Exchange layers (layers 3, 7, 11, 15, and 19): The attention mask is unchanged. After attention computation, cross-window pooling is applied: Z̄ = mean_m(Z_m) ∈ R^{48×d} That is, the (M×48) latent-token sequences are averaged across windows to produce one (1×48) sequence, which is then broadcast back to and replaces the latent tokens in each window. This operation involves no attention computation or parameters and has a cost of .
- Final quarter of the layers (layers 21–27): The attention mask switches to bidirectional attention within each window, with the same shape. Thus, [query; context; latent] attends bidirectionally within the window, while attention between windows remains zero. After the attention computation in every layer, the latent tokens again undergo the “average-pooling and broadcast-replacement” operation.
- Latent tokens retain their window identity after mean broadcasting. No cross-window attention is computed at any point in the model. The bidirectional attention in the final quarter of the layers is restricted to each window: latent tokens exchange information globally through pooling and broadcasting, then pass that information to all context tokens in their window through within-window bidirectional attention.The attention cost per window and layer is , treating and as fixed constants. Across windows, this gives . The pooling cost per layer is . Thus, the total complexity is . We have added the detailed complexity derivation to the revised manuscript (Appendix A.1.2, lines 752–829).
Response to Question 2
The LMI model was trained only on QA and summarization datasets; it was not specifically trained on Needle-in-a-Haystack tasks. We constructed two test sets from open-source corpora: Needle5_in_Haystack_32K and Needle5_in_Haystack_32K_atc, containing 43 and 100 examples, respectively. These examples were used exclusively for testing and were not included in the training data. For the ablation experiments with different numbers of windows, we also constructed non-ATC test sets at different lengths, each containing 43 examples, and ATC test sets at different lengths, each containing 100 examples (see the newly added §4.5.3, lines 505–519, and Tables 12 and 13 in Appendix A.8; lines 1111–1118 and 1124–1132, respectively).
Response to Question 3
The SWE-bench test cases used in our paper come from the open-source Hugging Face dataset princeton-nlp/SWE-bench_bm25_40K. To limit evaluation time, we selected cases in the test subset whose IDs begin with django__django-16[0-4], for a total of 51 cases (i.e., cases from the Django 1.60–1.64 series). We did not apply any further targeted selection. Our evaluation uses a concurrent, end-to-end, service-based setup. Competition for computing resources can introduce per-instance variation unrelated to the compression method itself. Therefore, pass or fail on an individual case is not a stable measure of method performance under concurrent execution.
Response to Question 4
We have added ablation experiments that hold the training dataset constant and vary only the model architecture (see lines 465–471 and Tables 6 and 7 in the revised manuscript; lines 473–478 and 487–493, respectively). The results show that cross-window latent-token interaction delivers substantial and consistent improvements to the LMI model.
Response to Questions
Response to Question 5
We have added results for MiniMax-M2.7 and Qwen2.5-14B-Instruct at a 64K context length (see Table 4, lines 409–412 of the revised manuscript). Total time is measured as the time required to complete all tasks. Average per-task latency is measured separately for each task and includes its time spent waiting in the queue. For example, when running MiniMax-M2.7 on 64K input sequences, the baseline at concurrency 32 typically sustains about 15 active tasks, with the remaining 17 waiting in the queue. With LMI compression before inference, all 32 concurrent slots can be used. This illustrates both that LMI can increase effective concurrency and that there is no simple analytical relationship between the total time and average per-task latency reported in Table 4.
Thanks
We sincerely thank you for your valuable questions and constructive feedback. Your comments have helped us further improve our work, and we hope our responses have clarified our study.
Response to Weaknesses
Response to Weakness 1
We have added a detailed explanation of the model architecture to the revised manuscript (Appendix A.1.1, lines 724–750). Your observation that “latent tokens retain window identity after mean broadcasting” is correct. The purpose of pooling is to inject global information; window-specific information is restored in the next layer, as each window’s latent tokens continue to interact through attention with that window’s context.
No cross-window attention is computed at any point in the model. The bidirectional attention in the final quarter of the layers is performed only within each window. Latent tokens in each window exchange information globally through pooling and broadcasting, then pass that information to all context tokens in their window through within-window bidirectional attention.
We also commit to releasing our model-training code, dataset-construction code, and all corpora used in our experiments.
Response to Weakness 2
To evaluate the contribution of the LMI architecture, we have added ablation experiments on cross-window latent-token interaction in both the Needle-in-a-Haystack and QA settings (see lines 465–471 and Tables 6 and 7 in the revised manuscript; lines 473–478 and 487–493, respectively). The results empirically demonstrate that this interaction mechanism delivers substantial and consistent improvements to the LMI model.