acceptodds
Under review as a conference paper at ICLR 2027

LMI: LATENT MEMORY INTERACTION FOR EFFICIENT AND FAITHFUL LONG-CONTEXT COMPRESSION IN LLMS

Abstract

Large language models (LLMs) have demonstrated strong long-context understanding, yet scaling context length remains challenging due to the quadratic complexity of self-attention and the linear growth of the key-value (KV) cache. Existing compression approaches—token pruning, pooling, and hierarchical summarization—often suffer from irreversible information loss and limited global interaction among compressed representations. We propose LMI (Latent Memory Interaction), a window-parallel framework that partitions long sequences into context windows and introduces learnable latent tokens that aggregate and exchange information across windows. Unlike methods that independently compress each segment, LMI dynamically refines latent representations through cross-window interaction, preserving local detail and global semantic dependencies while maintaining near-linear complexity and compatibility with existing LLM architectures. Extensive experiments on long-context benchmarks demonstrate that LMI achieves competitive performance against existing token-reduction and compression methods; on a 128K-token, 32-request serving workload, LMI reduces end-to-end inference time by 8.7 relative to full-context inference, with the one-time compression overhead more than offset by the reduction in generation cost.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long-context Compression, Latent memory, Semantic-aware

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_8LMxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LMI is a learned hard-context compressor that partitions a long context into windows, appends latent tokens, and exchanges latent information across windows. It then predicts token-level BIOE labels, aggregates these into sentence-level importance scores, and retains complete sentences under a target budget. The system is trained using QA data and automatically mined summarization supervision, with a 7B compressor and a distilled 0.5B variant. Evaluation covers synthetic retrieval, LoCoMo, LongBench, serving efficiency, and a 51-issue SWE-bench agent experiment.

Critical strengths
  1. The paper tackles a genuine limitation of window-independent compression: evidence needed to judge relevance can be distributed across distant windows. The large improvement on the interaction-sensitive synthetic task and the explicit interaction ablation support the usefulness of some form of cross-window communication.

  2. Retaining complete sentences is a sensible practical design that avoids fragmented prompts. The paper also attempts unusually deployment-oriented evaluation, including a smaller distilled compressor, serving measurements, and an agent pipeline.

Critical weaknesses
  1. The cross-window mechanism is not specified clearly enough to establish either its behavior or complexity. In the early layers, corresponding latent slots are averaged across all windows and the same average is broadcast back, which removes window identity. The paper then states that the final quarter uses bidirectional interaction among latent memories, but does not give the precise attention mask or complexity. If all MRMR latent tokens attend globally, this component is quadratic in the number of windows; if it is only mean aggregation, the description of continuous bidirectional interaction is misleading. The claimed near-linear complexity therefore cannot be verified from the current specification.

  2. Several decisive empirical results lack controls needed for a reliable conclusion. The strongest retrieval result uses a newly constructed benchmark without sufficient dataset-size or uncertainty information; compression budgets and operating-point selection are not fully described; the closest CPRS baseline uses different training data; and the SWE-bench result contains only 51 unspecified issues, where the reported gain is four additional successes without statistical uncertainty. These limitations make it difficult to attribute the gains specifically to LMI or assess generalization.

Questions for authors
  1. Please provide exact tensor shapes and attention masks for every interaction layer. Do latent tokens retain window identity after mean broadcasting, and does the final bidirectional stage attend over all MRMR latent tokens? What is its exact asymptotic and measured cost?

  2. How many examples are in each custom needle benchmark, how are train/test entities and templates separated, and are the differences statistically significant?

  3. How were the 51 SWE-bench issues selected? Please report the full issue list, paired outcomes, confidence intervals or a paired significance test, and results on an established SWE-bench subset.

  4. Can the authors compare LMI with CPRS under identical training data, labels, sentence-selection policy, and compression ratio? The existing system-level comparison does not isolate the contribution of latent interaction.

  5. How are total time and average task latency in Table 4 related under 32 concurrent requests, and do the results hold across context lengths and generators?

8 comments
AuthorsOpenReview ↗

Response to Weaknesses

Response to Weakness 1

We have added a detailed explanation of the model architecture to the revised manuscript (Appendix A.1.1, lines 724–750). Your observation that “latent tokens retain window identity after mean broadcasting” is correct. The purpose of pooling is to inject global information; window-specific information is restored in the next layer, as each window’s latent tokens continue to interact through attention with that window’s context.

No cross-window attention is computed at any point in the model. The bidirectional attention in the final quarter of the layers is performed only within each window. Latent tokens in each window exchange information globally through pooling and broadcasting, then pass that information to all context tokens in their window through within-window bidirectional attention.

We also commit to releasing our model-training code, dataset-construction code, and all corpora used in our experiments.

Response to Weakness 2

To evaluate the contribution of the LMI architecture, we have added ablation experiments on cross-window latent-token interaction in both the Needle-in-a-Haystack and QA settings (see lines 465–471 and Tables 6 and 7 in the revised manuscript; lines 473–478 and 487–493, respectively). The results empirically demonstrate that this interaction mechanism delivers substantial and consistent improvements to the LMI model.

AuthorsOpenReview ↗

Response to Questions

Response to Question 1

  1. We have added a detailed explanation of the model architecture to the revised manuscript (Appendix A.1.1, lines 724–750). To specify the tensor shapes, consider the 7B model with latent length R=48R=48, window length K=8192K=8192, and a 32,768 document (M=4 windows). Each window’s input sequence is [query; X_m ∈ R^{8K×d}; Z_m ∈ R^{48×d}] and the four windows are prefilled in parallel along the batch dimension. The details are as follows:
    1. First three-quarters of the layers (layers 0–20): Each layer uses a standard, within-window lower-triangular causal attention mask of shape (∣q∣+8K+48)×(∣q∣+8K+48)(|q|+8K+48)×(|q|+8K+48), with no attention between windows. The latent tokens causally absorb information from the query and context in their respective windows.
    2. Exchange layers (layers 3, 7, 11, 15, and 19): The attention mask is unchanged. After attention computation, cross-window pooling is applied: Z̄ = mean_m(Z_m) ∈ R^{48×d} That is, the (M×48) latent-token sequences are averaged across windows to produce one (1×48) sequence, which is then broadcast back to and replaces the latent tokens in each window. This operation involves no attention computation or parameters and has a cost of O(MR)O(MR).
    3. Final quarter of the layers (layers 21–27): The attention mask switches to bidirectional attention within each window, with the same shape. Thus, [query; context; latent] attends bidirectionally within the window, while attention between windows remains zero. After the attention computation in every layer, the latent tokens again undergo the “average-pooling and broadcast-replacement” operation.
  2. Latent tokens retain their window identity after mean broadcasting. No cross-window attention is computed at any point in the model. The bidirectional attention in the final quarter of the layers is restricted to each window: latent tokens exchange information globally through pooling and broadcasting, then pass that information to all context tokens in their window through within-window bidirectional attention.The attention cost per window and layer is O((K+R)2)=O(K2)O((K+R)^2)=O(K^2), treating KK and RR as fixed constants. Across MM windows, this gives O(MK2)=O(NK)O(MK^2)=O(NK). The pooling cost per layer is O(MR)=O(NR/K)O(MR)=O(NR/K). Thus, the total complexity is O(NK)+O(NR/K)O(NK)+O(NR/K). We have added the detailed complexity derivation to the revised manuscript (Appendix A.1.2, lines 752–829).

Response to Question 2

The LMI model was trained only on QA and summarization datasets; it was not specifically trained on Needle-in-a-Haystack tasks. We constructed two test sets from open-source corpora: Needle5_in_Haystack_32K and Needle5_in_Haystack_32K_atc, containing 43 and 100 examples, respectively. These examples were used exclusively for testing and were not included in the training data. For the ablation experiments with different numbers of windows, we also constructed non-ATC test sets at different lengths, each containing 43 examples, and ATC test sets at different lengths, each containing 100 examples (see the newly added §4.5.3, lines 505–519, and Tables 12 and 13 in Appendix A.8; lines 1111–1118 and 1124–1132, respectively).

Response to Question 3

The SWE-bench test cases used in our paper come from the open-source Hugging Face dataset princeton-nlp/SWE-bench_bm25_40K. To limit evaluation time, we selected cases in the test subset whose IDs begin with django__django-16[0-4], for a total of 51 cases (i.e., cases from the Django 1.60–1.64 series). We did not apply any further targeted selection. Our evaluation uses a concurrent, end-to-end, service-based setup. Competition for computing resources can introduce per-instance variation unrelated to the compression method itself. Therefore, pass or fail on an individual case is not a stable measure of method performance under concurrent execution.

Response to Question 4

We have added ablation experiments that hold the training dataset constant and vary only the model architecture (see lines 465–471 and Tables 6 and 7 in the revised manuscript; lines 473–478 and 487–493, respectively). The results show that cross-window latent-token interaction delivers substantial and consistent improvements to the LMI model.

AuthorsOpenReview ↗

Response to Questions

Response to Question 5

We have added results for MiniMax-M2.7 and Qwen2.5-14B-Instruct at a 64K context length (see Table 4, lines 409–412 of the revised manuscript). Total time is measured as the time required to complete all tasks. Average per-task latency is measured separately for each task and includes its time spent waiting in the queue. For example, when running MiniMax-M2.7 on 64K input sequences, the baseline at concurrency 32 typically sustains about 15 active tasks, with the remaining 17 waiting in the queue. With LMI compression before inference, all 32 concurrent slots can be used. This illustrates both that LMI can increase effective concurrency and that there is no simple analytical relationship between the total time and average per-task latency reported in Table 4.

Thanks

We sincerely thank you for your valuable questions and constructive feedback. Your comments have helped us further improve our work, and we hope our responses have clarified our study.

Reviewer_bG8LOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LMI is a learned long-context compressor: split the input into chunks, attach 48 latent tokens per chunk, let the latents interact across chunks (mean-and-broadcast plus bidirectional attention), score tokens with a BIOE classifier, and keep the top sentences. Supervision comes from QA data plus a perplexity-guided mining procedure, and a 7B compressor is distilled to 0.5B. Evaluated on needle retrieval (including a new self-built ATC setting), LoCoMo, LongBench, serving efficiency, and a small SWE-bench agent run.

Critical strengths

The cross-window interaction ablation is the strongest evidence in the paper (ATC recall 60.6 → 34.6 without it). Gains on non-synthetic settings (LongBench, LoCoMo) are consistent if modest, and the efficiency numbers are real wall-clock measurements.

Critical weaknesses

The biggest gains are on the authors' own ATC benchmark, where CPRS collapses below plain RAG — hard to tell how much of that generalizes, and there's no word on whether ATC will be released. Missing LLMLingua-2 / Activation Beacon baselines, no error bars anywhere (the SWE-bench claim rests on 51 issues), and the BIOE setup is under-justified: the score uses only p_I − p_O, so it's unclear what the B/E heads do.

Questions for authors

Will ATC be released, and does the gain hold on an independently built multi-hop set? Why were LLMLingua-2 and Activation Beacon left out? What do the B/E heads contribute?

3 comments
AuthorsOpenReview ↗

Response to Weaknesses and Questions

Response to Weaknesses

Response to Weakness 1

  1. We construct the ATC data as follows. First, we randomly sample person names from a name list. We then select one of several predefined templates and use it, together with the sampled names, to generate a family-relation chain. Next, we split the chain into multiple independent facts, or “needles,” and randomly insert these needles into haystack text drawn from the open-source Paul Graham Essays corpus. This produces long contexts of predefined lengths. We plan to release the dataset-construction code and all corpora used.

    We designed the dataset this way to create a challenging retrieval task in which some needles require logical reasoning to retrieve, rather than being retrievable directly from the query alone. Several open-source needle-in-a-haystack datasets are available, including MRCR. However, we consider these datasets to focus primarily on retrieving needles through semantic similarity, whereas our architecture is designed to retrieve needles in settings that require complex, chained reasoning. We therefore believe that a needle-in-a-haystack dataset constructed from the open-source corpus used by the Ancestral Trace Challenge (ATC) better reflects the motivation behind our model design. We commit to releasing all construction code and source corpora related to the ACT evaluation dataset.

  2. Regarding your observation that CPRS performs worse than RAG on the multi-needle task, we offer the following possible explanation. The relevant passages in our multi-needle task may be relatively short. Prior work has shown that language models may use relevant information less reliably in long contexts, particularly when it is surrounded by distractors, and that shorter relevant passages can be more difficult to retrieve from such contexts (Liu et al., “Lost in the Middle: How Language Models Use Long Contexts,” Transactions of the Association for Computational Linguistics, 2024). This may make the task more challenging for CPRS, whereas RAG’s localized retrieval may help it identify these short, relevant passages more effectively.

  3. We selected baselines according to the following criteria: they should be hard-compression methods that, like LMI, produce readable text; they should not require changes to the downstream model’s front end; and they should be closely related to our setting. Under these criteria, LLMLingua-2 differs from LMI primarily in its scorer and sentence-level structure, and our controlled comparison with CPRS addresses this distinction. The LLMLingua-2 paper evaluates quality preservation at compression ratios of 2–5×. Its token-level retention decisions use only local-window context and do not provide the cross-window global information exchange that LMI is designed to support. Our experimental settings typically require compression ratios above 10× and cross-window information exchange, which fall outside the range evaluated in that work. Nevertheless, we agree that a direct comparison would be valuable and commit to adding evaluation results for LLMLingua-2 and our method under matched compression-ratio settings in the revised manuscript. Activation Beacon uses representation-level soft compression, converting the context into continuous KV representations rather than readable text. Its use also requires a compatible downstream front end. Therefore, it cannot be compared with LMI under the same conditions.

  4. Regarding the error bars, the results in Table 3 are averages over five runs. For Table 5, we conducted multiple comparative runs to account for random variation. The number of tasks completed by LMI ranged from 36 to 38, while the baseline ranged from 32 to 34. We selected a run with a mid-range result for each method to report in the table. We commit to adding the remaining statistical information to the appendix in a subsequent revision.

  5. Thank you for pointing out the role of the B/E labels. This was an oversight on our part. In the initial submission, we stated only that “the B and E probabilities are primarily used to capture the structural boundaries of retained semantic units and are therefore not directly involved in the scalar importance score” (lines 252–253), but did not explain in the main text how the compression ratio is determined, which is directly related to the B/E labels. We have now clarified the role of the B/E head in the revised manuscript (see lines 270–271 of the revised manuscript): when the compression rate is unspecified, the number of sentences retrieved is determined by the number of B/E label pairs.

Response to Questions

Response to Question 1

Please refer to our response to Weakness 1.

Thanks

We sincerely thank you for your valuable questions and constructive feedback. Your comments have helped us further improve our work, and we hope our responses have clarified our study.

AuthorsOpenReview ↗

Additional comparative baseline

We thank the reviewer for the suggestion on adding additional comparative baselines raised in the first-round comments. As we promised in our response to Critical Weakness 1, we have recently completed the comparison experiments with LLMLingua-2 as an additional baseline.

1. LLMLingua-2 Comparative Experiments

Experimental Setup: The post-compression token budget is set to a 5:1 compression ratio, the maximum ratio validated in the LLMLingua-2 paper. This represents a lower compression ratio than that of LMI, i.e., LLMLingua-2 retains more tokens than LMI under this configuration. The downstream model and input settings are identical to those in Table 3 of the main paper. The experimental results are presented as follows:

Generation ModelCompression ModelNQDUVCMusiQMPCTQAVG
Qwen2.5-7B-InstructLLMLingua-211.5019.5714.0717.7420.952.1685.5722.44
Qwen2.5-7B-InstructLMI-Distill-0.5B (Ours)25.2227.9014.2331.2623.494.8686.8629.36
Qwen2.5-14B-InstructLLMLingua-212.2219.8213.7618.8620.931.6487.7522.94
Qwen2.5-14B-InstructLMI-Distill-0.5B (Ours)26.8226.4013.4333.1523.133.2686.8829.38
MiniMax-M2.7LLMLingua-214.2216.9313.6611.6918.661.6689.3521.50
MiniMax-M2.7LMI-Distill-0.5B (Ours)27.5223.9913.9435.0222.411.8287.7129.28

Conclusions: Even under a more favorable compression ratio setting, LLMLingua-2 still underperforms our proposed LMI-Distill-0.5B model on the LongBench v1 benchmark. This performance gap can be primarily attributed to its query-agnostic design — that is, LLMLingua-2 does not condition on the query when compressing long texts, and its compression decisions are independent of the target question.

Thank you for the detailed rebuttal. I have read your response and the additional results. My overall assessment and rating remain unchanged.

Reviewer_bjroOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LMI compresses long contexts by processing chunks in parallel and exchanging information through learnable latent memories. It then selects complete sentences for the downstream model. The paper also develops a distilled 0.5B compressor and evaluates retrieval, question answering, serving efficiency, and a coding-agent application. The results show competitive average QA performance with substantial context reduction and an 8.7-fold speedup in the reported 128K serving workload.

Critical strengths
  1. Cross-chunk communication is a sensible way to improve window-based compression. The matched ablation in Table 7 provides convincing support in the tested setting: removing interaction reduces recall from 60.60 to 34.59 and requires more retained sentences. This is a useful result for tasks whose evidence is spread across the context.

  2. The method has practical appeal because it produces ordinary text that existing generators can consume. Table 3 shows average QA performance close to full context while retaining much less text, and the serving measurements include compression overhead. The distilled model and coding-agent experiment further strengthen the practical case.

Critical weaknesses
  1. I had difficulty following how the shared memories influence the final token-selection scores. The memories are appended after the context tokens, and the early layers remain causal. The final stage is described as bidirectional interaction among memories, but the connection back to the context tokens is unclear. Since this is central to the method, a precise attention-mask description or short pseudocode would make both the mechanism and its computational cost much easier to assess.

  2. The quality and speed results are measured in different settings, which makes the practical trade-off difficult to judge. Table 3 evaluates QA with Qwen2.5, while the 8.7-fold speedup in Table 4 uses MiniMax-M2.7 at 128K with 10:1 compression. Appendix A.1 acknowledges this difference. A quality measurement for the timed configuration would make the headline result considerably more informative, especially given the paper's emphasis on faithful compression.

Questions for authors
  1. Could you clarify the attention pattern in the final quarter of layers and show how cross-chunk memory reaches the token representations used for selection?

  2. Do you have task-quality or evidence-retention results for the same model, inputs, and compression setting used in Table 4? This would help establish the quality-speed trade-off at the reported operating point.

2 comments
AuthorsOpenReview ↗

Response to Weaknesses and Questions

Response to Weaknesses

Response to Weakness 1

To clarify our approach, we have added a detailed explanation of the model architecture to the revised manuscript (Appendix A.1.1, lines 724–750). In brief, taking the 7B model as an example, the latent tokens causally absorb information from their respective windows in the first three-quarters of the layers. At the exchange points {3,7,11,15,19}\{3, 7, 11, 15, 19\}, they receive global information through cross-window pooling. In the final quarter of the layers, bidirectional attention within each window allows every context token to attend to the 48 latent tokens in that window, which carry global information.

We also commit to releasing our model-training code, dataset-construction code, and the corpora used in our experiments.

Response to Weakness 2

We have added a comparison on LongBench-v1 using MiniMax-M2.7(see the newly added results in Table 3, lines 395–399 of the revised manuscript). The results are consistent with our comparisons on LongBench-v1 using Qwen2.5-7B-Instruct and Qwen2.5-14B-Instruct. However, due to time constraints, we have not completed the experiments analyzing the trade-off between efficiency and accuracy. We will include these experiments in the final version of the paper.

Response to Questions

Response to Question 1

As noted in our response to Weakness 1, we have added a detailed explanation of the model architecture to the revised manuscript (Appendix A.1.1, lines 724–750). In brief, the attention pattern in the final quarter of the layers is bidirectional within each window, with no attention between windows. Information is communicated across windows as follows: at each exchange point, pooling combines the latent tokens from all windows to form shared global information, which is then broadcast back to each window. Within each window, bidirectional attention allows the context tokens to attend to the sequence of 48 latent tokens carrying global information.

A pooling-and-broadcast operation is performed after the bidirectional attention in every layer of the final quarter, resulting in more frequent communication than in the earlier layers, where this operation occurs once every four layers.

Response to Question 2

The experiments reported in Table 4 are stress tests of model performance on random sequences with fixed input and output lengths. The quality of the generated outputs was not evaluated for accuracy. For reference, we have added an accuracy evaluation on LongBench-v1 comparing MiniMax-M2.7 before and after compression. Please see our response to Weakness 2 for the detailed results (newly added results in Table 3, lines 395–399 of the revised manuscript).

Thanks

We sincerely thank you for your valuable questions and constructive feedback. Your comments have helped us further improve our work, and we hope our responses have clarified our study.

Dear Reviewer,

Thank you for your thorough and insightful comments during the initial review round.

We have prepared point-by-point responses and conducted supplementary experiments to address all your concerns, as well as all feedback from other reviewers. All additional results align with the core conclusions of our manuscript: the multi-hop retrieval gains enabled by cross-window latent interactions, and the efficiency advantages of prefill-style compression, deliver consistent and notable improvements across all evaluated scenarios.

We would like to bring to your attention that the discussion phase will conclude in less than 8 hours. Should you have any remaining unresolved concerns or require further experimental validation, we sincerely ask that you inform us within the remaining time window. Late feedback may leave us with insufficient time to conduct additional revisions or experiments, an outcome we very much wish to avoid. If your concerns have been fully addressed, we would greatly appreciate your consideration of our responses in your final rating.

We attach great importance to this conference and have full confidence in the value of our work. We look forward to your further feedback.

Best regards, The Authors

Reviewer_WUrdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LMI, a compressor that shortens long inputs for an LLM. Each window gets a few learned memory tokens that are averaged across windows and shared back, the best-scoring sentences are kept as plain text, and a large model is distilled into a small one. The paper reports large gains over CPRS on its own NeedleBench-style chain task, LongBench scores close to full context, and a large serving speedup.

Critical strengths
  1. The research problem is real. Compressing each window on its own can lose evidence that matters only because of another window. Sharing a small memory across windows is a simple, cheap fix, and because the output is plain text, it works with any LLM without changes. The idea addresses a real limitation of window-parallel compression and is easy to adopt, so it could be useful well beyond this paper if its benefit holds up.
Critical weaknesses
  1. Section 2.1 names sentence-level selection as one of LMI's two differences from CPRS, but CPRS's report already shows that CPRS can select sentences and is trained to do so, and LMI's labels reuse known ideas such as RECOMP's likelihood-based sentence scoring. That leaves sharing memory across windows as the clearly new part. Yet its only controlled test is on the authors' own chain task, every setting in Table 1 uses just four windows, and the paper does not explain how the exchange changes which sentences are kept. The paper's value rests on this one component, and the evidence does not yet show that this component, rather than other differences from CPRS, is what helps beyond one synthetic setting.
  2. The paper doesn't say how a needle counts as retrieved. In Figure 5, LMI is credited with a needle from the fragment "Corey Rodriguez's paternal", which drops "Mary Meadows" and "grandfather". With whole-needle matching, that example gives 4/5 rather than the reported 5/5. Recall is the main evidence for the paper's headline result, so without a stated matching rule and whole-chain recall, it is unclear how much usable evidence the compression actually keeps.
Questions for authors
  1. How is a needle counted as retrieved in Table 1, and how are compressed spans matched to LoCoMo's evidence turns in Table 2? Is Figure 5 abbreviated, and was sentence selection on when it was produced?
  2. The paper gives CPRS's chunk sizes and compression ratios, but not its control prompt or output granularity. Which prompt and granularity did you use in Tables 1–3 and Figure 5, and was it the released CPRS checkpoint? CPRS documents a sentence-extraction mode, so this affects how the comparison should be read.
  3. §3.2 says the last quarter of layers is bidirectional, describing it both as bidirectional blocks and as interaction "among latent memories". Is this attention within each window, or across all windows' memory tokens? And what is the 0.5B layer schedule? Table 6 lists one schedule for both models, but it needs at least 27 layers and Qwen2-0.5B has 24.
6 comments
AuthorsOpenReview ↗

Response to Weaknesses

Response to Weakness 1

  1. In §2.1, our intended point was that the main difference between our method and CPRS is not sentence-level selection itself, but our distinctive post-processing method, which converts token-level scores into sentence-level scores. CPRS uses B/E labels to determine sentence boundaries. Figure 5 in the submitted version shows CPRS’s original post-processing results under different model architectures. Some of the sentences retrieved by CPRS are incomplete, which motivated us to develop a new post-processing method. Although LMI and RECOMP both use likelihood-based sentence scoring, RECOMP is a reranker-style model that scores each query–sentence pair as a whole. LMI differs in that it scores tokens, thereby accounting for both global sentence semantics and the importance of local keywords. Overall, we acknowledge that the idea of sentence-level selection itself is not novel to our work. Our contribution is a new post-processing scoring method intended to address the limitations of CPRS. We also acknowledge that this contribution is less novel than our model architecture and data-flywheel training pipeline.

  2. Regarding cross-window shared memory, we added ablation experiments that hold the training dataset constant and vary only the model architecture (see lines 465–471 and Tables 6 and 7 in the revised manuscript; lines 473–478 and 487–493, respectively). The results show that cross-window latent-token interaction delivers substantial and consistent improvements to the LMI model. We also added ablation experiments with different numbers of windows (see the new §4.5.3, lines 505–519, and Tables 12 and 13 in Appendix A.7; lines 1111–1118 and 1124–1132, respectively). The results show that the gains from the proposed interaction mechanism persist as the number of windows increases, suggesting that LMI is robust to changes in window count. In addition, Figure 5 in the revised manuscript now includes details of CPRS’s original B/I/O/E-based post-processing algorithm, as well as the sentence-level scores produced by our semantic-aware structured post-processing method. These additions clarify the differences between the two post-processing approaches.

Response to Weakness 2

  1. We agree that the submitted version did not clearly explain how needle recall was calculated. We provide a detailed explanation of the needle-recall calculation for Table 1 in our response to Question 1. We have also updated Figure 5 in the revised manuscript (see lines 993–1017) to address your questions about the figure. In the submitted version, Figure 5 showed results obtained using CPRS’s original post-processing method across different model architectures, so that the effect of the model architecture could be isolated while keeping the post-processing method constant. When calculating the number of retrieved items in Figure 5, we applied sentence completion: if any part of a sentence was retrieved, the entire sentence was counted as retrieved. We have added the recall results and sentence-level scoring details for our post-processing method to Figure 5 to facilitate comparison. The advantages of our post-processing algorithm are described in lines 262–266 of the revised manuscript.
AuthorsOpenReview ↗

Response to Questions

Response to Question 1

We have added detailed descriptions of the retrieval procedures used by each method to the revised manuscript (see the newly added Appendix A.2, lines 853–863). For Table 1, RAG uses the GTE model and retrieves sentences with relevance scores above 0.6. Provence uses a binary classification head to estimate the probability that each token is retained or discarded. It then applies softmax to obtain each token’s retention score and averages the token-level scores within each sentence to calculate a sentence-level retention score. Sentences with scores above 0.5 are selected. CPRS uses its native post-processing method. LMI uses our semantic-aware structured post-processing method to derive token-level importance scores from the I/O labels. It then calculates and ranks sentence-level importance scores according to Equation (5). When the compression rate is unspecified, the number of sentences retrieved is determined by the number of B/E label pairs. Table 2 uses a top-k value of 50. For all methods, sentence completion is applied when calculating recall: if any part of a sentence is retrieved, the entire sentence is counted as retrieved. The rationale for the selection in Figure 5 is explained in our response to Weakness 2.

Response to Question 2

We used the settings provided in the official CPRS release. The CPRS model weights were downloaded from [https://www.modelscope.cn/models/iic/QwenLong-CPRS-7B/files]. We have added the prompts used for CPRS and LMI to Appendix A.4 of the revised manuscript (lines 964–971). The two methods use the same prompts: For Needle-in-a-Haystack tasks: CPRS_PROMPT="You are an expert for information extraction, your task is to extract the 'needles' in the format of 'One of the special magic {type_needle_v} for {key} is: {value}.' from the documents to answer user's question.\N## tagging rule:\n- tag the needles with 'needle'" For QA tasks: CPRS_PROMPT="You are an expert for information extraction, your task is to extract some sentences from the documents as the supporting facts of the user's question.\n## tagging rule:\n- tag the supporting facts with 'fact'"

CPRS’s native post-processing method uses the B and E labels to identify sentence boundaries. In our CPRS experiments, we used the post-processing code provided by CPRS.

Response to Question 3

This memory is confined to each window. We provide a detailed description of the model architecture in the newly added Appendix A.1.1 of the revised manuscript (lines 724–750), which we hope will clarify this mechanism. In brief, in the final quarter of the model layers, each window’s sequence is [query; 8K context; 48 latent tokens]. Full bidirectional attention is computed within each window over the context tokens and that window’s latent tokens; no attention is computed across different windows. All cross-window communication is handled by the latent-token “average-pooling–broadcast-and-replacement” operation.

The initial submission described the interaction-layer configuration for the 7B model in detail (Appendix A.4, Table 8, line 978). In the revised manuscript, we have also added the interaction-layer configuration for the 0.5B model (Appendix A.4, Table 8, line 979).

We sincerely thank you for your valuable questions and constructive feedback. Your comments have helped us further improve our work, and we hope our responses have clarified our study.

Comments on the paper

1 comment
AuthorsOpenReview ↗

The revised manuscript for the rebuttal stage has been submitted.

We have revised the manuscript during the rebuttal period in response to all reviewers’ comments. The revised manuscript is available for download on this page.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.