acceptodds
Under review as a conference paper at ICLR 2027

Beyond Position Bias: Shifting Context Compression from Position-Driven to Semantic-Driven

Abstract

Large Language Models (LLMs) have demonstrated exceptional performance across diverse tasks. However, their deployment in long-context scenarios faces high computational overhead and information redundancy. While soft prompt compression has emerged as a promising way to mitigate these costs by compressing sequences into compact embeddings, existing paradigms remain fundamentally constrained by position bias: they primarily rely on learnable tokens insertion at fixed positions or group tokens according to their physical token layout, thereby inducing performance instability and semantic fragmentation. To overcome this bottleneck, we propose Semantic Consistency Context Compression (SeCo), a method that shifts context compression from position-driven to semantic-driven. Rather than constraint by physical token layout, SeCo dynamically anchors compression directly in the semantic space by selecting query-relevant tokens as semantic centers and aggregating remaining tokens via consistency-weighted merging. This design inherently preserves semantic consistency while eliminating position bias. Extensive experiments on 14 benchmarks across two backbone models demonstrate that SeCo consistently shows superiority in downstream tasks, inference latency, and out-of-domain robustness. The code is available at https://anonymous.4open.science/r/SeCo-52YH

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Context Compression

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AZgrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces "Semantic Consistence Context Compression" (SeCo), a method to overcome position bias in the compression of contexts to shorten sequences, by basing compression instead on semantic relevance. Rather than using position, which depends on physical proximity of tokens, tokens are instead grouped into semantic groups, which might include tokens that are relatively far apart in the input. The authors demonstrate improved performance in terms of EM and F1 as well as improved latency.

Critical strengths
  • Introduces a novel semantic-grouping-based method to compress context.

  • Outperforms standard compression approaches in terms of score and latency on a variety of benchmarks.

Critical weaknesses
  • The paper does not apparently report scores on various RULER tasks such as NIAH, QA and so forth, which might help elucidate further how well the model helps with long contexts.
Questions for authors

If I am understanding correctly, this approach should substantially improve extension to contexts much longer than the training context. This case is problematic for models that rely on position encoding, but a model that groups tokens semantically should be much less sensitive to differences between the lengths observed in training and testing. If so, it would be interesting to see how robust it is to increased context length.

4 comments
AuthorsOpenReview ↗

Response to Reviewer AZgr — W1 & Q1 (Part 1 of 2)

Thank you for your positive assessment of SeCo’s semantic grouping approach and its task performance and latency results. Following your suggestion, we evaluated SeCo on NIAH and RULER QA [3] to examine its behavior on contexts longer than those seen during fine-tuning.

W1 & Q1. RULER evaluation and generalization beyond the fine-tuning length

Experimental setup. We compared SeCo, COMI [1], and RAM [2] using the same Llama-3.2-1B-Instruct backbone and a shared training input-length limit of 16K tokens.

  • NIAH: We evaluated all three methods at 16× compression, using ten input lengths from 2K to 32K tokens and eleven needle depths from 0% to 100% in increments of 10 percentage points.
  • RULER QA: We evaluated the SQuAD and HotpotQA tasks at 16× and 32× compression, with input lengths of 4K, 8K, 16K, and 32K tokens.

For both evaluations, we used Qwen3.8-27B to score the generated responses. All reported values are mean judge-assigned scores.

Table 1. NIAH scores at 16× compression. Column headers indicate input lengths before compression (1K = 1,024 tokens). Bold indicates the highest score among the three methods at each input length and needle depth, including ties.

Needle depth (%)Method2K5K9K12K15K19K22K25K29K32K
0SeCo48504648465049494849
RAM43403836363834323434
COMI46504850484846484748
10SeCo53445457565955565450
RAM60423638364038383442
COMI45445247424542424236
20SeCo46505759555357515446
RAM66364038364240383644
COMI47484046444444383642
30SeCo50545655505756505750
RAM54474040404442403838
COMI42385050454446504444
40SeCo52505352475457545250
RAM57384244343440383240
COMI50425042464444444246
50SeCo49525747645954525346
RAM54384046363842343842
COMI48484456464236424238
60SeCo49595949485961545250
RAM42474046464244443834
COMI50464250454250424246
70SeCo51505957545753505254
RAM45404042403844323234
COMI44443842485248464348
80SeCo56545554485757545454
RAM48383844384244343236
COMI54474442454442444636
90SeCo49506053525456525253
RAM38424538404242443834
COMI50515042463848464448
100SeCo57645557535958525458
RAM43423636384036383236
COMI52505454565254525152

Continued in “Response to Reviewer AZgr — W1 & Q1 (Part 2 of 2),” which presents the RULER QA results, discusses generalization, and lists the references for both parts.

AuthorsOpenReview ↗

Response to Reviewer AZgr — W1 & Q1 (Part 2 of 2)

Continued from “Response to Reviewer AZgr — W1 & Q1 (Part 1 of 2).”

Table 2. RULER QA scores at 16× and 32× compression. Values are mean scores assigned by Qwen3.8-27B. Column headers indicate input lengths before compression (1K = 1,024 tokens). Bold indicates the highest score among the three methods for each task, compression ratio, and input length.

Task / compression ratioMethod4K8K16K32K
SQuAD / 16×SeCo50.0052.5052.9043.87
COMI37.7440.6241.9443.23
RAM38.0635.6239.3540.00
HotpotQA / 16×SeCo55.9454.5254.5250.97
COMI44.3842.5841.2943.23
RAM40.3147.4245.8136.13
SQuAD / 32×SeCo52.2651.5650.9747.74
COMI40.9737.1938.0634.84
RAM43.2341.5631.6142.90
HotpotQA / 32×SeCo55.6255.1655.4850.97
COMI42.5040.6537.1040.65
RAM45.0048.3930.3228.39

Findings. SeCo achieves the highest NIAH score in 96 of the 110 input-length–needle-depth combinations, including four ties. It also outperforms both baselines in all 16 RULER QA settings. At 32K input length, for example, SeCo scores 50.97 on HotpotQA at both 16× and 32× compression, compared with 43.23 and 40.65 for COMI and 36.13 and 28.39 for RAM, respectively.

Generalization beyond the fine-tuning length. All three methods were fine-tuned with inputs of at most 16K tokens, whereas test inputs reached 32K tokens—twice the training length limit. The NIAH settings above 16K and the 32K RULER QA settings therefore directly evaluate generalization beyond the fine-tuning length. SeCo’s favorable performance in these settings provides empirical evidence of length generalization on the evaluated tasks.

A plausible explanation is SeCo’s semantic grouping design. Tokens are assigned to centers according to similarity in semantic space, allowing tokens that are far apart in the sequence to contribute to the same compressed representation. Global query-guided center selection also allows centers to be drawn from relevant regions throughout the context. These properties can help SeCo accommodate longer contexts with additional distractor text.

These results support SeCo’s generalization from a 16K fine-tuning limit to test inputs of up to 32K tokens under the evaluated tasks and scoring protocol. We have added the NIAH and RULER QA experiments to Appendix H of the revised manuscript.

References for Parts 1–2

[1] Tang et al. COMI: Coarse-to-fine Context Compression via Marginal Information Gain. ICLR, 2026.

[2] Tang et al. Read As Human: Compressing Context via Parallelizable Close Reading and Skimming. ACL (1), pp. 28393–28406, 2026.

[3] Hsieh et al. RULER: What’s the Real Context Size of Your Long-Context Language Models? CoRR, abs/2404.06654, 2024.

Reviewer_AZgrOpenReview ↗

Response to author response

Thank you for taking my suggestion and trying these additional experiments. The results look encouraging and support the contribution of the approach.

The overall scores for ICLR this year seem to be skewed downwards, so I still think 3 is an appropriate score.

AuthorsOpenReview ↗

Thank you for your response

Thank you for your careful review and insightful suggestions, which have greatly enhanced the quality of the paper. Many thanks again!

Reviewer_gc7yOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper propose a new method to do prompt compression. The main issue of exsiting works, as the author points out, is to enforce the compression over consecutive tokens on a fixed intervals. The author instead suggests clustering context tokens in the semantic space and merge the tokens within each cluster into a single embedding. The authors believe that this clustering will result in better compression and lead to higher downstream performance. The proposed method is query-dependent and it follows a very similar structure than existing works: COMI and ROM. The author then evaluate the performance on the methods against several baselines with LLaMA3.2-1B and Qwen3-4B-Instruct under 16x and 32x compression. The evaluation result on a few widely used benchmark show that the method largely outperform all existing methods. An additional latency evaluation shows the method is also the fastest across all methods.

Critical strengths

The critical strength is the method beating all baseline methods by a very large margin as shown in table-1. Otherwise the paper is well-written and easy to follow.

Critical weaknesses

I have some concerns about the results in Table 1.

Table-1 includes 6 baseline, in which 4 of them are query-agnostic (ICAE, SAC, 500xDPL, 500x EPL) and 2 of them are query-dependent (RAM and COMI). Comparing against query-agnostic is not fair because their compression is not query-aware so the quality must be worse. However, since their compression is not query-aware, this allows them to reuse the same compression across multi-round of conversation. In comparaision, methods like RAM and COMI will need to rerun the compression every round, which will not scale to very long conversation,

I checked the original papers of the two baselines, RAM (https://arxiv.org/abs/2602.01840) and COMI. The Table 1 of RAM has a similar setup as this paper, where the model is Qwen3-4B-Instruct and the tasks are HotpotQA , 2WikiMQA, NarrativeQA etc. The main difference is that the compression ratio tested in the RAM paper was 4 and 8 but 16 and 32 in this paper. While the results between the two works are not directly comparable. The huge performance gap between RAM reported in the RAM paper and the RAM reported in this paper is suspicious.

If you look at the table 1 in the RAM paper, RAM achieved excellence performance there but performs very poorly in this paper.

In particular, RAM scored 37.9/49.15 on HotpotQA with Qwen3-4B under 8x compression but only 18.86/31.75 on HotpotQA with Qwen3-4B under 16x compression.

Similarly, COMI can achieve 45.11/50.52 on HotpotQA with Qwen2-7B under 16x compression but only 13.83/25.12 on HotpotQA with Qwen3-4B under 16x compression.

While the above numbers are not directly comparable, the difference being so big is already suspicous enough. I wonder whether the baselines were configured optimally, as their performance seems lower than expected.

Questions for authors

To address my concern over the Table-1, I hope the author could provide additional experiments to evaluate SeCO under the exact same setup as in the Table-1 of RAM (https://arxiv.org/abs/2602.01840) and Table-1 of COMI (https://arxiv.org/abs/2602.01719) and compare against the results listed there. If SeCO consistenly overperforms the two baselines, I will change my review to accept.

For the method side, the clustering is based on similarity which will very likely result in imbalance between clusters. However, all the cluster of tokens are merged into an embedding of the same size, which creates a tension. I wonder if this will result in reduced performance ?

8 comments
AuthorsOpenReview ↗

Response to Reviewer gc7y — W1 & Q1: Cross-paper score differences and comparisons under the original settings (Part 1 of 2)

We sincerely thank you for your constructive and insightful comments, which have helped us improve our manuscript. Below, we respond to each point in detail.

W1 & Q1. Cross-paper score differences and comparisons under the original settings

Following your suggestion, we evaluated SeCo using the training and evaluation settings used for Table 1 in the original RAM [1] and COMI [2] papers, respectively. We compared SeCo’s performance under each setting with the results reported in the corresponding paper. These comparisons have been added to Appendix I of the revised manuscript.

This part presents the experimental settings and the RAM comparison. Part 2 presents the COMI comparison, the overall findings, a clarification of the comparison in our own Table 1, and the references for both parts.

  • RAM. We truncated training inputs to 20K tokens and trained SeCo with a batch size of 32 and a learning rate of 1 × 10⁻⁵, using linear learning rate decay. For each training example, we randomly sampled a compression constraint from {2×, 4×, 8×, 16×, 32×}. We limited NarrativeQA test inputs to 32K tokens and report results at compression ratios of 4× and 8×.
  • COMI. We limited both training and test inputs to 32K tokens and trained SeCo with a batch size of 64 and a learning rate of 1 × 10⁻⁵, using linear learning rate decay. Following COMI’s training protocol, we randomly sampled a compression constraint from {16×, 32×} for each training example and report results at compression ratios of 16× and 32×.

For the four QA tasks, each cell reports EM / F1; for MultiNews, only F1 is reported. AVG reports the mean EM across the four QA tasks and the mean F1 across all five tasks. Bold indicates the higher score for each metric under the same backbone and compression constraint.

Results under the settings used for Table 1 in the original RAM paper [1]

LLaMA-3.1-8B-Instruct

ConstraintMethodNaturalQuestions2WikiMQAHotpotQANarrativeQAMultiNewsAVG
4×RAM65.35 / 59.2248.66 / 54.8946.24 / 59.1319.64 / 32.4729.2144.97 / 46.98
4×SeCo66.00 / 69.1657.50 / 67.7851.50 / 69.9825.37 / 38.5231.3150.09 / 55.35
8×RAM62.41 / 57.1439.63 / 45.2438.82 / 50.8018.80 / 31.5726.1139.92 / 42.17
8×SeCo62.19 / 67.9152.00 / 61.3748.05 / 67.4324.12 / 36.7728.2046.59 / 52.34

Qwen3-4B-Instruct

ConstraintMethodNaturalQuestions2WikiMQAHotpotQANarrativeQAMultiNewsAVG
4×RAM66.59 / 59.9750.39 / 56.4746.37 / 59.2819.45 / 31.9232.0745.70 / 47.94
4×SeCo66.43 / 69.5755.80 / 61.9748.00 / 66.3725.68 / 37.8435.1348.98 / 54.18
8×RAM61.09 / 56.1540.24 / 45.6637.90 / 49.1519.92 / 31.2228.5039.79 / 42.14
8×SeCo63.19 / 66.3143.60 / 58.6447.60 / 66.1024.39 / 36.5631.4244.70 / 51.81

Continued in “Response to Reviewer gc7y — W1 & Q1 (Part 2 of 2),” which also includes the references for both parts.

AuthorsOpenReview ↗

Response to Reviewer gc7y — W1 & Q1: Cross-paper score differences and comparisons under the original settings (Part 2 of 2)

Continued from “Response to Reviewer gc7y — W1 & Q1 (Part 1 of 2).”

Results under the settings used for Table 1 in the original COMI paper [2]

For the four QA tasks, each cell reports EM / F1; for MultiNews, only F1 is reported. AVG reports the mean EM across the four QA tasks and the mean F1 across all five tasks. Bold indicates the higher score for each metric under the same backbone and compression constraint.

LLaMA-2-7B-Chat

ConstraintMethodNaturalQuestions2WikiMQAHotpotQANarrativeQAMultiNewsAVG
16×COMI22.75 / 27.3241.43 / 48.7436.60 / 49.508.31 / 20.3227.3027.27 / 34.64
16×SeCo26.67 / 31.4045.09 / 52.3240.42 / 53.3711.48 / 23.6330.5930.92 / 38.26
32×COMI16.99 / 23.9437.34 / 43.8531.46 / 43.326.83 / 15.9024.7123.16 / 30.34
32×SeCo20.75 / 27.6841.12 / 47.6135.52 / 47.299.84 / 19.3127.9326.81 / 33.96

Qwen2-7B-Instruct

ConstraintMethodNaturalQuestions2WikiMQAHotpotQANarrativeQAMultiNewsAVG
16×COMI56.31 / 55.5252.13 / 60.4345.11 / 60.5213.25 / 24.6935.8541.70 / 47.40
16×SeCo57.08 / 59.1955.64 / 63.9749.02 / 63.3516.44 / 28.0238.9444.55 / 50.69
32×COMI49.15 / 49.5948.89 / 57.0740.46 / 55.2411.18 / 22.3033.6037.42 / 43.56
32×SeCo51.62 / 54.3848.42 / 59.7144.39 / 58.1314.34 / 25.7436.8139.69 / 46.95

SeCo outperforms the published baselines on the vast majority of metrics. In particular, on HotpotQA—the task you highlighted—SeCo achieves higher EM and F1 than both RAM and COMI across all evaluated backbones and compression constraints under their respective settings.

Fair comparison in Table 1 of the SeCo paper

Differences in training data, training setup, backbone, and compression constraint can affect the comparability of scores across papers. For the comparisons in Table 1 of the SeCo paper, we used the same training data, hyperparameters, evaluation splits, and EM/F1 scoring procedures for SeCo, RAM, and COMI within each combination of backbone and compression constraint. We used the official implementations of COMI and RAM throughout the experiments reported in the SeCo paper. The exact code used for both methods is available in the anonymous repository.

References for Parts 1–2

[1] Tang et al., Read As Human: Compressing Context via Parallelizable Close Reading and Skimming, ACL (1), pp. 28393-28406, 2026.

[2] Tang et al., COMI: Coarse-to-fine Context Compression via Marginal Information Gain, ICLR, 2026.

AuthorsOpenReview ↗

Response to Reviewer gc7y — Q2: Cluster-size imbalance and compression quality

Q2. Effect of unequal cluster sizes on compression quality

Clusters containing different numbers of tokens can each be summarized by a single representation. SeCo assigns tokens to clusters based on their similarity to the cluster center, grouping tokens that convey similar information. Larger clusters may contain more redundant information and thus offer greater scope for compression. During cluster compression, SeCo aggregates tokens using softmax-normalized weights within each cluster (Eqs. 6–9). This normalization helps control the scale of the resulting token representations, supporting numerical stability during compression.

To assess whether greater cluster-size imbalance is associated with lower task performance, we evaluated SeCo on 500 examples from each of Natural Questions (NQ), 2WikiMQA, and MultiNews, using the same examples at 4× and 8× compression. For each example, we quantified cluster-size imbalance using the coefficient of variation of the number of tokens per cluster (standard deviation divided by mean). Within each dataset and compression ratio, we computed the partial Spearman correlation between imbalance and per-example token F1, controlling for context length.

DatasetCompression ratePartial Spearman ρ95% bootstrap CI
NQ4×−0.102[−0.185, −0.011]
NQ8×−0.006[−0.091, +0.082]
2WikiMQA4×−0.025[−0.115, +0.057]
2WikiMQA8×+0.110[+0.026, +0.198]
MultiNews4×+0.051[−0.039, +0.136]
MultiNews8×−0.040[−0.135, +0.045]

The estimated correlations were small across all six settings (|ρ| ≤ 0.110). NQ at 4× compression showed a weak negative association, with a similar estimate when imbalance was measured as the ratio of the largest cluster size to the mean cluster size (ρ = −0.092). In contrast, 2WikiMQA at 8× compression showed a weak positive association. In the remaining four settings, the pointwise confidence intervals for the primary imbalance measure included zero. Across the evaluated datasets and compression ratios, we find no consistent negative association between cluster-size imbalance and downstream task performance. These results suggest that greater cluster-size imbalance is not consistently associated with poorer compression quality, as measured by downstream task performance.

Reviewer_vqrqOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes SeCo, a semantic-driven framework for soft prompt compression that addresses position bias in existing methods. Rather than relying on fixed-position learnable tokens or position-based grouping, SeCo selects query-relevant tokens as semantic centers, assigns remaining tokens based on semantic similarity, and generates compressed representations through consistency-weighted merging. Experiments on 14 benchmarks with two backbone models at 16x and 32x compression constraints demonstrate improvements in downstream performance, inference efficiency, and out-of-domain generalization. Ablation studies further validate the effectiveness of query guidance and semantic consistency merging.

Critical strengths
  1. The method integrates query-guided semantic center selection, semantic assignment, and consistency-weighted merging into a unified framework, effectively preserving task-relevant information while mitigating positional constraints. The overall approach is lightweight and elegant, and effectively improves model performance.
  2. The experimental evaluation is comprehensive. Results across 14 benchmarks and two backbone models demonstrate significant improvements in downstream performance, out-of-domain generalization, and inference efficiency. Extensive ablation studies further validate the effectiveness of each proposed component.
Critical weaknesses
  1. The ablation studies validate the effectiveness of the main components, but I would be interested in further analysis of how different semantic center selection strategies affect compression performance. This would further strengthen the empirical justification for the proposed design.
  2. There is a minor grammatical error in Section 3: "within each compressed tokens" should be corrected to "within each compressed token."
Questions for authors

Seek weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer vqrq — W1 (Part 1 of 2)

We sincerely thank you for your constructive feedback and for recognizing the effectiveness of our main components. Your suggestion prompted us to further investigate semantic center selection.

W1. Effects of semantic center selection strategies on compression performance

We conducted additional ablations comparing SeCo’s global query-guided center selection with four alternatives. We evaluated all five strategies using LLaMA-3.2-1B-Instruct and Qwen3-4B-Instruct-2507 at 16× and 32× compression, on six in-domain (ID) and six out-of-domain (OOD) datasets. These experiments have been added to Appendix J of the revised manuscript.

Experimental setup. All strategies select the same number of centers, KK, from the valid context tokens, with KK determined by Eq. (2):

  • SeCo (Global Semantic Top-K): Select the KK tokens with the highest query-relevance scores across the entire context.
  • Random (Ablation-1): Uniformly sample KK distinct context positions without replacement.
  • Uniform Stride (Ablation-2): Select KK distinct context positions at approximately equal intervals.
  • Stratified Random (Ablation-3): Divide the context into KK contiguous blocks of approximately equal length and randomly select one token from each block.
  • Block-wise Semantic (Ablation-4): Use the same block partition as Stratified Random and select the token with the highest query-relevance score in each block.

Only center selection is changed; all variants retain SeCo’s subsequent semantic assignment and consistency merging. Non-center tokens are assigned to their most similar centers and incorporated into the compressed representations through weighted aggregation. For Block-wise Semantic, the blocks constrain center selection only: subsequent token assignment remains global and can cross block boundaries.

Results. Each entry reports aggregate EM / F1 over the corresponding ID or OOD datasets. Bold indicates the highest score for each metric in each column.

LLaMA-3.2-1B-Instruct

Method16× ID16× OOD32× ID32× OOD
SeCo52.89 / 68.1640.20 / 55.3352.27 / 67.8539.72 / 54.94
Random46.73 / 63.0337.22 / 52.6047.16 / 63.5136.40 / 52.63
Uniform Stride43.04 / 57.4433.44 / 47.4043.83 / 57.6029.10 / 42.35
Stratified Random46.80 / 63.0637.02 / 52.6246.27 / 62.8536.57 / 52.58
Block-wise Semantic46.27 / 62.6736.09 / 51.6745.73 / 62.5635.34 / 51.51

Qwen3-4B-Instruct-2507

Method16× ID16× OOD32× ID32× OOD
SeCo57.88 / 72.5048.69 / 63.3053.49 / 68.4640.29 / 56.54
Random36.92 / 55.7431.23 / 48.7839.03 / 57.1932.87 / 50.30
Uniform Stride30.96 / 50.5225.38 / 44.0636.54 / 54.9729.68 / 47.12
Stratified Random41.74 / 55.9335.93 / 49.2435.79 / 54.2729.16 / 46.93
Block-wise Semantic35.31 / 53.5729.11 / 46.2536.52 / 54.7229.91 / 47.44

Continued in “Response to Reviewer vqrq — W1 (Part 2 of 2) and W2,” which discusses the findings and addresses W2.

AuthorsOpenReview ↗

Response to Reviewer vqrq — W1 (Part 2 of 2) and W2

Continued from “Response to Reviewer vqrq — W1 (Part 1 of 2).”

W1. Effects of semantic center selection strategies on compression performance (continued)

The ablations support the proposed center-selection strategy in three respects:

  • Positional coverage alone does not match SeCo’s performance. Random, Uniform Stride, and Stratified Random all yield lower aggregate EM and F1 than SeCo in every evaluated setting. For example, with LLaMA-3.2-1B-Instruct at 16× compression, SeCo achieves ID/OOD F1 scores of 68.16/55.33, compared with 57.44/47.40 for Uniform Stride and 63.06/52.62 for Stratified Random. Thus, distributing centers evenly across the context is insufficient to match SeCo in these experiments.

  • Global selection remains beneficial when both strategies use query relevance. Block-wise Semantic retains query-guided selection but requires exactly one center per block. At 32× compression, SeCo exceeds this variant by 6.54 EM / 5.29 F1 points on ID and 4.38 EM / 3.43 F1 points on OOD with LLaMA-3.2-1B-Instruct. With Qwen3-4B-Instruct-2507, the corresponding gains are 16.97 EM / 13.74 F1 points on ID and 10.38 EM / 9.10 F1 points on OOD. This comparison supports allowing centers to be selected globally according to query relevance, without enforcing a fixed quota for each block.

  • The aggregate advantage holds across both backbones and compression ratios. SeCo achieves the highest aggregate EM and F1 in all eight combinations of backbone, compression ratio, and ID/OOD split. Relative to the strongest alternative in each setting, its aggregate F1 gains range from 2.31 to 5.10 points for LLaMA-3.2-1B-Instruct and 6.24 to 16.57 points for Qwen3-4B-Instruct-2507.

Together, these results provide empirical support for global query-guided center selection under the evaluated settings. The comparisons with Random and the coverage-based alternatives support the value of query guidance, while the comparison with Block-wise Semantic supports the flexibility of global selection: query guidance within fixed blocks does not close the observed performance gap.

W2. Grammatical number error in Section 3

Thank you for pointing out this error. We will replace “within each compressed tokens” with “within each compressed token” in Section 3.

Thank you for the detailed response and the additional experiments. My concerns have been substantially addressed, and I have no further questions. I will keep my current rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.