- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces "Semantic Consistence Context Compression" (SeCo), a method to overcome position bias in the compression of contexts to shorten sequences, by basing compression instead on semantic relevance. Rather than using position, which depends on physical proximity of tokens, tokens are instead grouped into semantic groups, which might include tokens that are relatively far apart in the input. The authors demonstrate improved performance in terms of EM and F1 as well as improved latency.
Critical strengths
-
Introduces a novel semantic-grouping-based method to compress context.
-
Outperforms standard compression approaches in terms of score and latency on a variety of benchmarks.
Critical weaknesses
- The paper does not apparently report scores on various RULER tasks such as NIAH, QA and so forth, which might help elucidate further how well the model helps with long contexts.
Questions for authors
If I am understanding correctly, this approach should substantially improve extension to contexts much longer than the training context. This case is problematic for models that rely on position encoding, but a model that groups tokens semantically should be much less sensitive to differences between the lengths observed in training and testing. If so, it would be interesting to see how robust it is to increased context length.
4 comments
Response to Reviewer AZgr — W1 & Q1 (Part 2 of 2)
Continued from “Response to Reviewer AZgr — W1 & Q1 (Part 1 of 2).”
Table 2. RULER QA scores at 16× and 32× compression. Values are mean scores assigned by Qwen3.8-27B. Column headers indicate input lengths before compression (1K = 1,024 tokens). Bold indicates the highest score among the three methods for each task, compression ratio, and input length.
| Task / compression ratio | Method | 4K | 8K | 16K | 32K |
|---|---|---|---|---|---|
| SQuAD / 16× | SeCo | 50.00 | 52.50 | 52.90 | 43.87 |
| COMI | 37.74 | 40.62 | 41.94 | 43.23 | |
| RAM | 38.06 | 35.62 | 39.35 | 40.00 | |
| HotpotQA / 16× | SeCo | 55.94 | 54.52 | 54.52 | 50.97 |
| COMI | 44.38 | 42.58 | 41.29 | 43.23 | |
| RAM | 40.31 | 47.42 | 45.81 | 36.13 | |
| SQuAD / 32× | SeCo | 52.26 | 51.56 | 50.97 | 47.74 |
| COMI | 40.97 | 37.19 | 38.06 | 34.84 | |
| RAM | 43.23 | 41.56 | 31.61 | 42.90 | |
| HotpotQA / 32× | SeCo | 55.62 | 55.16 | 55.48 | 50.97 |
| COMI | 42.50 | 40.65 | 37.10 | 40.65 | |
| RAM | 45.00 | 48.39 | 30.32 | 28.39 |
Findings. SeCo achieves the highest NIAH score in 96 of the 110 input-length–needle-depth combinations, including four ties. It also outperforms both baselines in all 16 RULER QA settings. At 32K input length, for example, SeCo scores 50.97 on HotpotQA at both 16× and 32× compression, compared with 43.23 and 40.65 for COMI and 36.13 and 28.39 for RAM, respectively.
Generalization beyond the fine-tuning length. All three methods were fine-tuned with inputs of at most 16K tokens, whereas test inputs reached 32K tokens—twice the training length limit. The NIAH settings above 16K and the 32K RULER QA settings therefore directly evaluate generalization beyond the fine-tuning length. SeCo’s favorable performance in these settings provides empirical evidence of length generalization on the evaluated tasks.
A plausible explanation is SeCo’s semantic grouping design. Tokens are assigned to centers according to similarity in semantic space, allowing tokens that are far apart in the sequence to contribute to the same compressed representation. Global query-guided center selection also allows centers to be drawn from relevant regions throughout the context. These properties can help SeCo accommodate longer contexts with additional distractor text.
These results support SeCo’s generalization from a 16K fine-tuning limit to test inputs of up to 32K tokens under the evaluated tasks and scoring protocol. We have added the NIAH and RULER QA experiments to Appendix H of the revised manuscript.
References for Parts 1–2
[1] Tang et al. COMI: Coarse-to-fine Context Compression via Marginal Information Gain. ICLR, 2026.
[2] Tang et al. Read As Human: Compressing Context via Parallelizable Close Reading and Skimming. ACL (1), pp. 28393–28406, 2026.
[3] Hsieh et al. RULER: What’s the Real Context Size of Your Long-Context Language Models? CoRR, abs/2404.06654, 2024.
Response to author response
Thank you for taking my suggestion and trying these additional experiments. The results look encouraging and support the contribution of the approach.
The overall scores for ICLR this year seem to be skewed downwards, so I still think 3 is an appropriate score.
Thank you for your response
Thank you for your careful review and insightful suggestions, which have greatly enhanced the quality of the paper. Many thanks again!
Response to Reviewer AZgr — W1 & Q1 (Part 1 of 2)
Thank you for your positive assessment of SeCo’s semantic grouping approach and its task performance and latency results. Following your suggestion, we evaluated SeCo on NIAH and RULER QA [3] to examine its behavior on contexts longer than those seen during fine-tuning.
Experimental setup. We compared SeCo, COMI [1], and RAM [2] using the same Llama-3.2-1B-Instruct backbone and a shared training input-length limit of 16K tokens.
For both evaluations, we used Qwen3.8-27B to score the generated responses. All reported values are mean judge-assigned scores.
Table 1. NIAH scores at 16× compression. Column headers indicate input lengths before compression (1K = 1,024 tokens). Bold indicates the highest score among the three methods at each input length and needle depth, including ties.
Continued in “Response to Reviewer AZgr — W1 & Q1 (Part 2 of 2),” which presents the RULER QA results, discusses generalization, and lists the references for both parts.